Enroot

Enroot is a tool built by NVIDIA to run containers on Linux. Enroot is targeted for HPC environments with integration with the Slurm scheduler, but can also be used as a standalone tool to run containers as an unprivileged user. This is similar to Singularity, but with the added benefit of allowing users to read/write in the container and also to appear as a root user within the container environment.

Usage cheat sheet

Task	Command
Import a new container from Docker Hub	enroot import docker://centos:8
Create an instance of a container	enroot create --name centos centos+8.sqsh
Destroy an instance	enroot remove centos
Convert an image from a local docker daemon You may override the squashfs options if desired.	enroot import dockerd://centos:8 ENROOT_SQUASH_OPTIONS='-comp lz4 -noD' enroot import dockerd://container:latest
Run an enroot image	enroot start --mount /tmp:tmp /images/fedora-mate.sqsh bash
Run an enroot image as root	enroot start --rw --root --mount /tmp:tmp /images/fedora-mate.sqsh bash
Enter an existing Enroot container	enroot exec $pid bash Where $pid is the PID of enroot start.

Compiling and installing enroot

At the time of writing, there is no precompiled EL8-based version of Enroot from the releases page. Fortunately, the enroot RPM packages can be built relatively easily. On a RHEL8-based system, this should build the RPMs.

# yum install -y rpm-build rpmlint libtool git libcap libtool automake jq squashfs-tools parallel fuse-overlayfs pigz squashfuse
# cd /

# git clone --recurse-submodules https://github.com/NVIDIA/enroot.git
# cd /enroot

## for hardened mode, use: make rpm PACKAGE=enroot-hardened
# CPPFLAGS="-DALLOW_SPECULATION -DINHERIT_FDS" make rpm
# ls /enroot/pkg/rpm/RPMS/x86_64/enroot*rpm

If desired, you can also build the enroot-hardened version of the package. The hardened version contains security mitigations against spectre-based exploits.

When installing Enroot, you can choose to use either the standard or the +caps version. The +caps version also contains additional capabilities for unprivileged users to import and convert container images which is handy for a HPC-based environment.

After installing Enroot, you may wish to use the enroot-check script (available from their GitHub release page) to verify whether your system is set up appropriately. You'll likely see some issues regarding glilbc. You can verify your glibc version by running ldd --version.

# ./enroot-check_3.4.0_x86_64.run --verify 
...
Kernel configuration:

CONFIG_NAMESPACES                 : OK
CONFIG_USER_NS                    : OK
CONFIG_SECCOMP_FILTER             : OK
CONFIG_OVERLAY_FS                 : OK (module)
CONFIG_X86_VSYSCALL_EMULATION     : OK
CONFIG_VSYSCALL_EMULATE           : KO (required if glibc <= 2.13)
CONFIG_VSYSCALL_NATIVE            : KO (required if glibc <= 2.13)
...

# ldd --version
ldd (GNU libc) 2.28

Configuration

Enroot has a configuration directory at /etc/enroot. The main configuration file is at /etc/enroot/enroot.conf. All settings defined in this configuration file can be overridden by defining an environment variable with the same name.

A few parameters of note are listed below.

• ENROOT_MOUNT_HOME - set this to yes will make Enroot always mount the user's home directory.
• ENROOT_SQUASH_OPTIONS='-comp lz4 -noD' - Some systems don't support lzo (the default) which requires you to override it to lz4.

GPU support using libnvidia-container

Enroot comes with GPU support using libnvidia-container. The precompiled packages and installation instructions can be found at https://nvidia.github.io/libnvidia-container/.

After installing the libnvidia-container package, run the enroot-check script (available from their GitHub release page) to verify whether your system is set up appropriately. If everything's working, you should see:

# ./enroot-check_3.4.0_x86_64.run --verify 
...
Extra packages:

nvidia-container-cli             : OK

Other notes

• There is Slurm integration using the Pyxis plugin. See more on my Slurm page.
• Becoming 'root' within the image (using the --root option) basically fakes your uid/gid to 0. There are consequences doing this that you should be aware of:
  • The user will be 'root' since their uid is 0. This might not be desirable. I have mangled /etc/passwd so that the user still appears as themselves with this script:

if [ $(id -u) -eq 0 ] ; then
  ActualUsername=$(basename $HOME)
  echo "I see you want to be root. Updating user $ActualUsername to root"
  sed -i -e "s/^\($ActualUsername:[^:]\):[0-9]*:[0-9]*:/\1:0:0:/" /etc/passwd
  sed -i "/^root/d" /etc/passwd
fi

• • The uid/gid cannot change once the container has started. sudo/su has no effect.
  • All file mounts not owned by the user will appear as nobody/nogroup.