Proxmox

Proxmox Virtual Environment (VE) is an open source virtualization management software that allows deployment and management of KVM-based virtual machines and LXC containers. The platform itself run on a Debian based distribution and supports filesystems such as ZFS and Ceph.

In terms of functionality, Proxmox VE is similar to VMware ESXi with some overlap with vSphere. You can have a cluster of Proxmox servers controlled and managed through a singular web console, command line tools, or via the provided REST API.

Installation

Installation is as simple as installing any other Linux distribution. Download the Proxmox iso image from https://www.proxmox.com/en/downloads. After installation, you should be able to access the web console via HTTPS on port 8006.

Tasks

Remove the subscription nag

If your Proxmox server has no active subscription, you will be nagged every time you log in to the web interface. This can be disabled by running the following as root in the server's console:

sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

See also, the ansible role at https://github.com/FuzzyMistborn/ansible-role-proxmox-nag-removal/blob/master/tasks/remove-nag.yml

Make a VM automatically start at boot

You can enable or disable a VM from starting with the Proxmox host as well as configure the order in which VMs are started.

The setting is located under the VM or container's 'Options' pane.

Proxmox start VM or container at boot

Adding additional LXC templates

Proxmox officially supports about a dozen different Linux distributions and provides up to date LXC templates through their repositories. These LXC templates can be downloaded using the pveam tool or through the Proxmox web interface.

Web interface

Download additional Proxmox CT Templates via the web interface

Go to your local storage and click on CT Templates. Click on the Templates button to see available templates.

pveam tool

The Proxmox VE Appliance Manager (pvadm) tool is available to manage container templates from Proxmox's repository. More information at https://pve.proxmox.com/pve-docs/pveam.1.html.

Run pveam available to list all available templates, then run pveadm download $storage $template to download a template to a storage pool. For example:

root@server:~# pveam available
mail            proxmox-mailgateway-6.4-standard_6.4-1_amd64.tar.gz
mail            proxmox-mailgateway-7.0-standard_7.0-1_amd64.tar.gz
system          almalinux-8-default_20210928_amd64.tar.xz
system          alpine-3.12-default_20200823_amd64.tar.xz
system          alpine-3.13-default_20210419_amd64.tar.xz
system          alpine-3.14-default_20210623_amd64.tar.xz
system          alpine-3.15-default_20211202_amd64.tar.xz
system          archlinux-base_20210420-1_amd64.tar.gz

root@server:~# pveam download local fedora-35-default_20211111_amd64.tar.xz
downloading http://download.proxmox.com/images/system/fedora-35-default_20211111_amd64.tar.xz to /var/lib/vz/template/cache/fedora-35-default_20211111_amd64.tar.xz
--2021-12-29 15:17:28--  http://download.proxmox.com/images/system/fedora-35-default_20211111_amd64.tar.xz
Resolving download.proxmox.com (download.proxmox.com)... 144.217.225.162, 2607:5300:203:7dc2::162
Connecting to download.proxmox.com (download.proxmox.com)|144.217.225.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 89702020 (86M) [application/octet-stream]
Saving to: '/var/lib/vz/template/cache/fedora-35-default_20211111_amd64.tar.xz.tmp.1053836'
     0K ........ ........ ........ ........ 37% 2.59M 21s
 32768K ........ ........ ........ ........ 74% 13.6M 5s
 65536K ........ ........ .....            100% 20.5M=16s
2021-12-29 15:17:44 (5.42 MB/s) - '/var/lib/vz/template/cache/fedora-35-default_20211111_amd64.tar.xz.tmp.1053836' saved [89702020/89702020]
calculating checksum...OK, checksum verified
download of 'http://download.proxmox.com/images/system/fedora-35-default_20211111_amd64.tar.xz' to '/var/lib/vz/template/cache/fedora-35-default_20211111_amd64.tar.xz' finished

Integrating Terraform with Proxmox

You can use Terraform to manage VMs on Proxmox. More information on how to set this up on the Terraform page.

Install SSL certificates

Proxmox will generate a self signed SSL certificate for its management interfaces. To install your own, log in to Proxmox and copy your SSL private key and certificates to /etc/pve/nodes/$node/pve-ssl.{pem,key}.

The pve-ssl.pem file should contain your full certificate chain. This can be generated by concatenating your primary certificate, followed by any intermediate certificates.

The pve-ssl.key file should contain your private key without a password.

Restart the pveproxy service to apply. If you have any problems, check the service's status with systemctl status pveproxy.

# cp fullchain.pem /etc/pve/nodes/<node>/pve-ssl.pem
# cp private-key.pem /etc/pve/nodes/<node>/pve-ssl.key
# systemctl restart pveproxy

Note: Changing the SSL certificates prevented my SPICE clients from connecting. I had to revert back to the self generated certificates for them to work again.

ZFS memory management

By default, ZFS's ARC uses 50% of the host's memory. If you are running VMs using a ZFS storage pool that exceeds 50% of your system's available memory, you will need to tweak the amount of memory assigned to ARC to prevent out of memory issues.

The minimum memory assigned to ARC as per Proxmox's documentation is 2GB + 1GB per TB of storage.

To change the ARC max size to 3GB: echo "$[3 * 1024*1024*1024]" > /sys/module/zfs/parameters/zfs_arc_max

To make this permanent, edit /etc/modprobe.d/zfs.conf and add options zfs zfs_arc_max=3221225472

VNC to guest VMs

Edit /etc/pve/local/qemu-server/<vmid>.conf and add a line containing additional arguments to KVM defining the VNC listen address and password. You may choose to have the VM serve VNC on a specific port or through a unix socket by adding one of the following lines:

## Listen on an port (didn't work for me)
args: -object secret,id=vncpass,data=password -vnc 0.0.0.0:5915,password-secret=vncpass

## Listen on a socket (worked for me)
args: -object secret,id=vncpass,data=password -vnc unix:/var/run/qemu-server/115-secondary.vnc,password-secret=vncpass

Note that Proxmox will create a VNC socket for its own use at /var/run/qemu-server/115.vnc. The password for this is set by Proxmox and supposedly can be changed with the qm and vnc_password commands, but that didn't work for me (using Proxmox 7.1).

If you listen on a socket, you could use something like socat to expose it when necessary (so the port isn't always opened) by running:

# socat tcp4-listen:5915,fork,reuseaddr unix-connect:/var/run/qemu-server/115-secondary.vnc

Attach a disk from another VM

To attach a disk to another VM, detach it first, then go to 'Disk Action' -> 'Reassign Disk'.

Manual process below: (before the reassign disk was a feature)

If you are using ZFS for the VM disks, you can attach a disk from the source VM to another target VM by renaming the disk's VM ID to the target VM. You can run zfs list on the Proxmox server to see what disks are available. Once the disk is renamed, the disk can be made visible as an unused disk by running qm rescan to rescan and update the VM configs. From the Proxmox console, you should then be able to attach the unused disk under the hardware page.

For example, to attach a VM disk which was originally connected to VM 101 to VM 102, I had to rename the disk vm-101-disk-0 to vm-102-disk-1 by running: zfs rename data/vm-101-disk-0 data/vm-102-disk-1.

Renumber VM IDs

Use the following script to renumber a VM's ID. This script supports VMs that are stored on both LVM and ZFS. Power off the VM before proceeding. Proxmox should be able to see the VM being renamed automatically.

Be aware that renumbering a VM may cause CloudInit to re-run again as it may think it's on a new instance. This could have unintended side effects such as having your system's SSH host keys wiped and accounts' passwords reset.

#!/bin/bash
Old="$1"
New="$2"

usage(){
        echo "Usage: $0 old-vmid new-vmid"
        exit 2
}

if [ -z "$Old" ] || ! expr "$Old" : '^[0-9]\+$' >/dev/null ; then
        echo "Error: Invalid old-vimd"
        usage

        if [ ! -f /etc/pve/qemu-server/$Old.conf ] ; then
                echo "Error: $Old is not a valid VM ID"
                exit 1
        fi
fi
if [ -z "$New" ] || ! expr "$New" : '^[0-9]\+$' >/dev/null ; then
        echo "Error: Invalid new-vimd"
        usage
fi

echo "Found these disks in the config:"
cat /etc/pve/qemu-server/$Old.conf | grep -i -- vm-$Old- | awk -F, '{print $1}' | awk '{print $2}'

echo "Found these disks on LVM:"
lvs --noheadings -o lv_name,vg_name | grep "vm-$Old-"

echo "Found these disks on ZFS:"
zfs list | grep "vm-$Old-" | awk '{print $1}'


echo "Will execute the following:"
echo
lvs --noheadings -o lv_name,vg_name | grep "vm-$Old-" | while read lv vg ; do
        echo lvrename $vg/$lv $vg/$(echo $lv | sed "s/$Old/$New/g")
done
zfs list | grep "vm-$Old-" | awk '{print $1}' | while read i ; do
        echo zfs rename $i $(echo $i | sed "s/$Old/$New/g")
done
echo sed -i "s/$Old/$New/g" /etc/pve/qemu-server/$Old.conf
echo mv /etc/pve/qemu-server/$Old.conf /etc/pve/qemu-server/$New.conf


echo
read -p "Proceed with the rename? (y/n) " answer
case $answer in
  [yY]* ) echo "Proceeding...";;
  * ) echo "Aborting..."; exit;;
esac

echo "Task: Rename LVS data volumes"
lvs --noheadings -o lv_name,vg_name | grep "vm-$Old-" | while read lv vg ; do
        lvrename $vg/$lv $vg/$(echo $lv | sed "s/$Old/$New/g")
done

echo "Task: Rename ZFS datasets"
zfs list | grep "vm-$Old-" | awk '{print $1}' | while read i ; do
        zfs rename $i $(echo $i | sed "s/$Old/$New/g")
done

echo "Renumbering IDs"
sed -i "s/$Old/$New/g" /etc/pve/qemu-server/$Old.conf
mv /etc/pve/qemu-server/$Old.conf /etc/pve/qemu-server/$New.conf

Troubleshooting

LXC containers does not start networking

I have a Fedora based LXC container which takes a long time to start up. After the tty does start, the only network adapter isn't up. systemd-networkd is also in a a failed state with a bad exit code and status 226/namespace.

This was resolved by enabling nesting on this container with the following commands:

# pct set $CTID -features nesting=1 
# pct reboot $CTID

PCIe passthrough fails due to platform RMRR

This is covered in more details at HP_DL380_page on PCIe_passthrough_failing. Basically, the workaround for this is to install a patched kernel that doesn't respect the RMRR restrictions.

If you want to build your own patched kernel, you might be interested in this repo: https://github.com/sarrchri/relax-intel-rmrr