Enroot: Difference between revisions

From Leo's Notes
This page was last edited on 17 December 2021, at 21:39.
No edit summary
No edit summary
Line 1: Line 1:
Enroot is a tool built by NVIDIA to run containers on Linux. Enroot is targeted for HPC environments with integration with the Slurm scheduler, but can also be used as a standalone tool to run containers as an unprivileged user. This is similar to Singularity, but with the added benefit of allowing users to read/write in the container and also to appear as a root user within the container environment.
Enroot is a tool built by NVIDIA to run containers on Linux. Enroot is targeted for HPC environments with integration with the Slurm scheduler, but can also be used as a standalone tool to run containers as an unprivileged user. This is similar to Singularity, but with the added benefit of allowing users to read/write in the container and also to appear as a root user within the container environment.


==Compiling and installing enroot==
== Usage cheat sheet ==
At the time of writing, there is no precompiled EL8-based version of Enroot from the releases page. Fortunately, the enroot RPM packages can be built relatively easily. On a RHEL8-based system, this should build the RPMs.
{{Highlight
| code = # yum install -y rpm-build rpmlint libtool git libcap libtool automake jq squashfs-tools parallel fuse-overlayfs pigz squashfuse
# cd /
 
# git clone --recurse-submodules https://github.com/NVIDIA/enroot.git
# cd /enroot
 
## for hardened mode, use: make rpm PACKAGE=enroot-hardened
# CPPFLAGS="-DALLOW_SPECULATION -DINHERIT_FDS" make rpm
# ls /enroot/pkg/rpm/RPMS/x86_64/enroot*rpm
| lang = terminal
}}
If desired, you can also build the enroot-hardened version of the package. The hardened version contains security mitigations against spectre-based exploits.
 
When installing Enroot, you can choose to use either the standard or the <code>+caps</code> version. The <code>+caps</code> version also contains additional capabilities for unprivileged users to import and convert container images which is handy for a HPC-based environment.
 
===Configuration===
Enroot has a configuration directory at <code>/etc/enroot</code>. The main configuration file is at <code>/etc/enroot/enroot.conf</code>. All settings defined in this configuration file can be overridden by defining an environment variable with the same name.
 
A few parameters of note are listed below.
 
*<code>ENROOT_MOUNT_HOME</code> - set this to yes will make Enroot always mount the user's home directory.
*<code>ENROOT_SQUASH_OPTIONS='-comp lz4 -noD'</code> - Some systems don't support <code>lzo</code> (the default) which requires you to override it to <code>lz4</code>.
 
== Quick usage ==
{| class="wikitable"
{| class="wikitable"
!Task
!Task
Line 58: Line 32:
|}
|}


== Other notes ==
==Compiling and installing enroot==
At the time of writing, there is no precompiled EL8-based version of Enroot from the releases page. Fortunately, the enroot RPM packages can be built relatively easily. On a RHEL8-based system, this should build the RPMs.
{{Highlight
| code = # yum install -y rpm-build rpmlint libtool git libcap libtool automake jq squashfs-tools parallel fuse-overlayfs pigz squashfuse
# cd /
 
# git clone --recurse-submodules https://github.com/NVIDIA/enroot.git
# cd /enroot
 
## for hardened mode, use: make rpm PACKAGE=enroot-hardened
# CPPFLAGS="-DALLOW_SPECULATION -DINHERIT_FDS" make rpm
# ls /enroot/pkg/rpm/RPMS/x86_64/enroot*rpm
| lang = terminal
}}
If desired, you can also build the enroot-hardened version of the package. The hardened version contains security mitigations against spectre-based exploits.
 
When installing Enroot, you can choose to use either the standard or the <code>+caps</code> version. The <code>+caps</code> version also contains additional capabilities for unprivileged users to import and convert container images which is handy for a HPC-based environment.
 
After installing Enroot, you may wish to use the enroot-check script ([https://github.com/NVIDIA/enroot/releases available from their GitHub release page]) to verify whether your system is set up appropriately.
 
===Configuration===
Enroot has a configuration directory at <code>/etc/enroot</code>. The main configuration file is at <code>/etc/enroot/enroot.conf</code>. All settings defined in this configuration file can be overridden by defining an environment variable with the same name.
 
A few parameters of note are listed below.
 
*<code>ENROOT_MOUNT_HOME</code> - set this to yes will make Enroot always mount the user's home directory.
*<code>ENROOT_SQUASH_OPTIONS='-comp lz4 -noD'</code> - Some systems don't support <code>lzo</code> (the default) which requires you to override it to <code>lz4</code>.
 
=== GPU support using libnvidia-container ===
Enroot comes with GPU support using <code>libnvidia-container</code>. The precompiled packages and installation instructions can be found at https://nvidia.github.io/libnvidia-container/.


* There is Slurm integration using the Pyxis plugin. See more on my [[Slurm#Pyxis Plugin|Slurm]] page.
After installing the libnvidia-container package, run the enroot-check script ([https://github.com/NVIDIA/enroot/releases available from their GitHub release page]) to verify whether your system is set up appropriately. If everything's working, you should see:
* Becoming 'root' within the image (using the --root option) basically fakes your uid/gid to 0. There are consequences doing this that you should be aware of:
{{Highlight
** The user will be 'root' since their uid is 0. This might not be desirable. I have mangled <code>/etc/passwd</code> so that the user still appears as themself with this script:  
| code = # ./enroot-check_3.4.0_x86_64.run --verify
...
Extra packages:
 
nvidia-container-cli            : OK
| lang = terminal
}}
 
==Other notes==
 
*There is Slurm integration using the Pyxis plugin. See more on my [[Slurm#Pyxis Plugin|Slurm]] page.
*Becoming 'root' within the image (using the <code>--root</code> option) basically fakes your uid/gid to 0. There are consequences doing this that you should be aware of:
**The user will be 'root' since their uid is 0. This might not be desirable. I have mangled <code>/etc/passwd</code> so that the user still appears as themselves with this script:
{{Highlight
{{Highlight
| code = if [ $(id -u) -eq 0 ] ; then
| code = if [ $(id -u) -eq 0 ] ; then
Line 72: Line 87:
| lang = bash
| lang = bash
}}
}}
** The uid/gid cannot change once the container has started. <code>sudo</code>/<code>su</code> has no effect.
 
** All file mounts not owned by the user will appear as nobody/nogroup.
**The uid/gid cannot change once the container has started. <code>sudo</code>/<code>su</code> has no effect.
**All file mounts not owned by the user will appear as nobody/nogroup.
*
*

Revision as of 21:39, 17 December 2021

Enroot is a tool built by NVIDIA to run containers on Linux. Enroot is targeted for HPC environments with integration with the Slurm scheduler, but can also be used as a standalone tool to run containers as an unprivileged user. This is similar to Singularity, but with the added benefit of allowing users to read/write in the container and also to appear as a root user within the container environment.

Usage cheat sheet

Task Command
Import a new container from Docker Hub enroot import docker://centos:8
Create an instance of a container enroot create --name centos centos+8.sqsh
Destroy an instance enroot remove centos
Convert an image from a local docker daemon

You may override the squashfs options if desired.

enroot import dockerd://centos:8

ENROOT_SQUASH_OPTIONS='-comp lz4 -noD' enroot import dockerd://container:latest

Run an enroot image enroot start --mount /tmp:tmp /images/fedora-mate.sqsh bash
Run an enroot image as root enroot start --rw --root --mount /tmp:tmp /images/fedora-mate.sqsh bash
Enter an existing Enroot container enroot exec $pid bash

Where $pid is the PID of enroot start.

Compiling and installing enroot

At the time of writing, there is no precompiled EL8-based version of Enroot from the releases page. Fortunately, the enroot RPM packages can be built relatively easily. On a RHEL8-based system, this should build the RPMs.

# yum install -y rpm-build rpmlint libtool git libcap libtool automake jq squashfs-tools parallel fuse-overlayfs pigz squashfuse
# cd /

# git clone --recurse-submodules https://github.com/NVIDIA/enroot.git
# cd /enroot

## for hardened mode, use: make rpm PACKAGE=enroot-hardened
# CPPFLAGS="-DALLOW_SPECULATION -DINHERIT_FDS" make rpm
# ls /enroot/pkg/rpm/RPMS/x86_64/enroot*rpm

If desired, you can also build the enroot-hardened version of the package. The hardened version contains security mitigations against spectre-based exploits.

When installing Enroot, you can choose to use either the standard or the +caps version. The +caps version also contains additional capabilities for unprivileged users to import and convert container images which is handy for a HPC-based environment.

After installing Enroot, you may wish to use the enroot-check script (available from their GitHub release page) to verify whether your system is set up appropriately.

Configuration

Enroot has a configuration directory at /etc/enroot. The main configuration file is at /etc/enroot/enroot.conf. All settings defined in this configuration file can be overridden by defining an environment variable with the same name.

A few parameters of note are listed below.

  • ENROOT_MOUNT_HOME - set this to yes will make Enroot always mount the user's home directory.
  • ENROOT_SQUASH_OPTIONS='-comp lz4 -noD' - Some systems don't support lzo (the default) which requires you to override it to lz4.

GPU support using libnvidia-container

Enroot comes with GPU support using libnvidia-container. The precompiled packages and installation instructions can be found at https://nvidia.github.io/libnvidia-container/.

After installing the libnvidia-container package, run the enroot-check script (available from their GitHub release page) to verify whether your system is set up appropriately. If everything's working, you should see:

# ./enroot-check_3.4.0_x86_64.run --verify 
...
Extra packages:

nvidia-container-cli             : OK

Other notes

  • There is Slurm integration using the Pyxis plugin. See more on my Slurm page.
  • Becoming 'root' within the image (using the --root option) basically fakes your uid/gid to 0. There are consequences doing this that you should be aware of:
    • The user will be 'root' since their uid is 0. This might not be desirable. I have mangled /etc/passwd so that the user still appears as themselves with this script:
if [ $(id -u) -eq 0 ] ; then
  ActualUsername=$(basename $HOME)
  echo "I see you want to be root. Updating user $ActualUsername to root"
  sed -i -e "s/^\($ActualUsername:[^:]\):[0-9]*:[0-9]*:/\1:0:0:/" /etc/passwd
  sed -i "/^root/d" /etc/passwd
fi
    • The uid/gid cannot change once the container has started. sudo/su has no effect.
    • All file mounts not owned by the user will appear as nobody/nogroup.