NixOS inside LXC on Proxmox

From Leo's Notes
Last edited on 13 July 2023, at 05:44.

This page will go over how you can set up and run NixOS in LXC on Proxmox.

The instructions here are based on the following resources:

Guide

Step 1: Obtain the container tarball

Find and download a recently generated NixOS container tarball from https://hydra.nixos.org/job/nixos/trunk-combined/nixos.containerTarball.x86_64-linux. Place the .tar.xz archive in your CT Volumes store in Proxmox (which is typically located under /var/lib/vz/template/cache/)

Step 2: Create the container

In Proxmox shell, create the container with the following command: 

# pct create 300 --arch amd64 --description nixos --ostype unmanaged \
  --net0 name=eth0  --storage local-lvm --unprivileged 1 \
  local:vztmpl/nixos-system-x86_64-linux.tar.xz

Open PVE and enable nesting. This is required by Nix. Not enabling nesting would cause the nix-daemon to have issues remounting /nix/store or setting up namespaces.

You may optionally adjust the size of the storage if desired (it defaults to 4GB which may not be enough). You may also want to change the other resource allocations before starting the container.

Step 3: Start the container and configure it

Start the CT. The console will be blank. We'll fix this shortly. However, to connect to our container in the current state, we'll have to use the Proxmox shell and run: 

# lxc-attach --name 300

Note that because we dropped into the container without any of the environment variables set, nothing other than your shell will work. To fix this, update your path with the NixOS bin path and start bash: 

sh-5.2# PATH=$PATH:/run/current-system/sw/bin/
sh-5.2# bash
[root@nixos:~]#

That's better! Next, we'll fix the blank console. Edit the /etc/nixos/configuration.nix such that the getty on tty1 works. Add in the following lines to configuration.nix: 

# Supress systemd units that don't work because of LXC
  systemd.suppressedSystemUnits = [
    "dev-mqueue.mount"
    "sys-kernel-debug.mount"
    "sys-fs-fuse-connections.mount"
  ];

  # start tty0 on serial console
  systemd.services."getty@tty1" = {
    enable = lib.mkForce true;
    wantedBy = [ "getty.target" ]; # to start at boot
    serviceConfig.Restart = "always"; # restart when session is closed
  };

  environment.systemPackages = with pkgs; [
    vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
    binutils
  ];

Then run nixos-rebuild switch to update.

Troubleshooting

Trouble with nix-channel --update

If you did not nesting in the CT options, you will get: unexpected Nix daemon error: error: remounting /nix/store writable: Permission denied.