Proxmox - Revision history

Leo: PCIe bus error resolved

2025-08-11T20:48:42Z

PCIe bus error resolved

← Older revision		Revision as of 20:48, 11 August 2025
Line 547:		Line 547:
	\| lang = terminal		\| lang = terminal
	}}		}}
	~~There are some potential solutions that involve~~ adding kernel arguments to <code>/etc/default/grub</code>, then ~~run~~ <code>update-grub</code>~~, and then reboot~~.		One possible fix is to disable PCIe Active State Power Management by adding <code>pcie_aspm=off</code> to the kernel boot arguments in <code>/etc/default/grub</code>, then apply it to grub by running <code>update-grub</code> before rebooting.

	~~==== What didn't help ====~~		A side note: adding <code>acpi_enforce_resources=no</code> to the kernel args and rebooting did not help. There are also some suggestions to add <code>pci=noaer</code> which will turn off error reporting. I did no try this after disabling ASPM.{{Navbox Linux}}

	* Adding <code>acpi_enforce_resources=no</code> to the kernel args and rebooting did not help.

	~~What~~ to ~~try instead:~~

	* Try <code>~~pcie_aspm~~=~~off~~</code> ~~which will disable PCIe Active State Power Management.~~
	* Try pci=noaer which will turn off error reporting
	{{Navbox Linux}}
	[[Category:Linux]]		[[Category:Linux]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]

Leo: /* Troubleshooting */

2025-08-03T04:27:00Z

Troubleshooting

@@ Line 527: / Line 527: @@
    post-up /sbin/ethtool -K eno1 tso off gso off
 | lang = terminal
-}}{{Navbox Linux}}
+}}
 [[Category:Linux]]
 [[Category:LinuxUtilities]]

Leo: editing

2023-08-18T20:24:55Z

editing

Show changes

Leo at 06:16, 18 August 2023

2023-08-18T06:16:45Z

@@ Line 446: / Line 446: @@
 A reboot of the Proxmox server did not help.
-Fix: I re-installed the NVIDIA drivers but also answered 'yes' for the 32bit libraries to be installed. Perhaps that was the fix, or the act of re-installing the driver recompiled the driver on the current running kernel which fixed the issue.{{Navbox Linux}}
+Fix: I re-installed the NVIDIA drivers but also answered 'yes' for the 32bit libraries to be installed. Perhaps that was the fix, or the act of re-installing the driver recompiled the driver on the current running kernel which fixed the issue.
 [[Category:Linux]]
 [[Category:LinuxUtilities]]

Leo: /* Remove a single node cluster */

2023-08-17T22:04:11Z

Remove a single node cluster

← Older revision		Revision as of 22:04, 17 August 2023
Line 187:		Line 187:
	}}		}}

	==== Remove a single node cluster ====		=== Remove a single node cluster ===
	If you created a single node cluster and want to undo it, run:		If you created a single node cluster and want to undo it, run:
	{{Highlight		{{Highlight

Leo: /* Setup LXC with NVIDIA GPU */

2023-08-08T17:05:06Z

Setup LXC with NVIDIA GPU

← Older revision		Revision as of 17:05, 8 August 2023
Line 349:		Line 349:
	After doing a NixOS rebuild, docker works and docker containers with the nvidia runtime also works.		After doing a NixOS rebuild, docker works and docker containers with the nvidia runtime also works.

	==== Troubleshooting ====		==== Troubleshooting NVIDIA LXC issues ====

			===== Missing nvidia-uvm =====
	You also need the cuda device <code>nvidia-uvm</code> passed through.		You also need the cuda device <code>nvidia-uvm</code> passed through.

Line 366:		Line 368:
	}}		}}

	===== The fix =====		====== The fix ======
	If the device <code>/dev/nvidia-uvm</code> and <code>/dev/nvidia-uvm-tools</code> isn't available on the PVE (ie. it's missing), then you will have to run this:		If the device <code>/dev/nvidia-uvm</code> and <code>/dev/nvidia-uvm-tools</code> isn't available on the PVE (ie. it's missing), then you will have to run this:
	{{Highlight		{{Highlight
Line 394:		Line 396:
	\| lang = text		\| lang = text
	}}		}}

			==== Randomly losing NVIDIA device access ====
			Whenever you change a LXC setting (such as memory or CPU change), it will break access to the NVIDIA GPUs from within the LXC. Any existing processes with the NVIDIA GPU opened will continue to work, but any new processes trying to access the GPU will fail. The only fix is to restart the LXC.

	=== Replace a failed ZFS mirror boot device ===		=== Replace a failed ZFS mirror boot device ===

Leo: /* Run Docker in LXC */

2023-07-27T05:39:46Z

Run Docker in LXC

← Older revision		Revision as of 05:39, 27 July 2023
Line 217:		Line 217:
	\|OverlayFS		\|OverlayFS
	\|ZFS backing filesystem for <code>/var/lib/docker</code> should have no issues.		\|ZFS backing filesystem for <code>/var/lib/docker</code> should have no issues.
	\|<code>/var/lib/docker</code> should use ~~a extfs~~ (or some non-ZFS filesystem, more on this below.)		\|<code>/var/lib/docker</code> should use EXT or XFS (or some non-ZFS filesystem, more on this below.)
	\|}		\|}
	It's recommended that you do not run Docker in LXC unless you have a good reason such as: a) hardware passthrough on systems that cannot support it through VMs, or b) tight memory constraints where containerized workloads benefit from shared memory.		It's recommended that you do not run Docker in LXC unless you have a good reason such as: a) hardware passthrough on systems that cannot support it through VMs, or b) tight memory constraints where containerized workloads benefit from shared memory.

	~~To run unprivileged LXC~~ using a ZFS backed storage, you will have to ~~create~~ a ~~separate volume~~ for <code>/var/lib/docker</code> ~~that uses something else such as~~ a ~~LVM backed storage or create a ZFS volume that uses~~ a different filesystem (see https://du.nkel.dev/blog/2021-03-25_proxmox_docker/). This issue manifests itself as issues when pulling some (but not all) container images with errors similar to:		==== /var/lib/docker issues when using ZFS backed storage ====
			Docker uses overlayfs to handle the container layers. On ZFS backed storage, it requires additional privileges to allow it to work properly. For LXC that are unprivileged, certain docker image pulls may fail if the image data is stored on ZFS storage. To work around this issue, you will have to use a different filesystem for <code>/var/lib/docker</code> by making a zvol and then formatting it with a different filesystem (see https://du.nkel.dev/blog/2021-03-25_proxmox_docker/). Alternatively, put the <code>/var/lib/docker</code> volume on a non-ZFS backed storage such as a LVM backed storage.

			This issue manifests itself as issues when pulling some (but not all) container images with errors similar to:

	{{Highlight		{{Highlight
Line 240:		Line 243:
	}}		}}

	Switching <code>/var/lib/docker</code> to a ext backed filesystem fixed the issue.		Switching <code>/var/lib/docker</code> to a ext backed filesystem fixed the issue. This can be done with the following steps:

			# Create the zvol: <code>zfs create -V 8G rpool/data/ctvol-111-disk-1</code>
			# Format the zvol: <code>mkfs.ext4 /dev/zvol/rpool/data/ctvol-111-disk-1</code>
			# Edit the LXC config under <code>/etc/pve/nodes/pve/lxc/111.conf</code> and add the following mountpoint config:
			#* <code>mp0: /dev/zvol/rpool/data/ctvol-111-disk-1,mp=/var/lib/docker</code>

	=== Install NVIDIA driver ===		=== Install NVIDIA driver ===

Leo: /* Troubleshooting */ Segfaulting VirGL

2023-07-26T17:21:57Z

Troubleshooting: Segfaulting VirGL

← Older revision		Revision as of 17:21, 26 July 2023
Line 422:		Line 422:
	This is covered in more details at [[HP DL380 G6#PCIe passthrough fails due to platform RMRR requirement\|HP_DL380_page on PCIe_passthrough_failing]]. Basically, the workaround for this is to install a patched kernel that doesn't respect the RMRR restrictions.		This is covered in more details at [[HP DL380 G6#PCIe passthrough fails due to platform RMRR requirement\|HP_DL380_page on PCIe_passthrough_failing]]. Basically, the workaround for this is to install a patched kernel that doesn't respect the RMRR restrictions.

	If you want to build your own patched kernel, you might be interested in this repo: https://github.com/sarrchri/relax-intel-rmrr<~~nowiki/~~>{{Navbox Linux}}		If you want to build your own patched kernel, you might be interested in this repo: https://github.com/sarrchri/relax-intel-rmrr

			=== NVIDIA driver crashes with KVM using VirGL GPU ===
			On Proxmox 7.4 with NVIDIA driver version 535.54.03 installed, starting a virtual machine with a VirGL GPU device results in KVM segfaulting:
			{{Highlight
			\| code = [Wed Jul 26 11:13:38 2023] kvm[559117]: segfault at 0 ip 00007ff13a39ca23 sp 00007ffd4eb76450 error 4 in libnvidia-eglcore.so.535.54.03[7ff1398bd000+164e000]
			[Wed Jul 26 11:13:38 2023] Code: 48 8b 45 10 44 8b a0 04 0d 00 00 e9 44 fe ff ff 0f 1f 80 00 00 00 00 48 8b 77 08 48 89 df e8 e4 4d 78 ff 48 8b 7d 10 48 89 de <48> 8b 07 ff 90 98 01 00 00 48 8b 43 38 e9 f2 fd ff ff 0f 1f 00 48
			\| lang = text
			}}
			A reboot of the Proxmox server did not help.

			Fix: I re-installed the NVIDIA drivers but also answered 'yes' for the 32bit libraries to be installed. Perhaps that was the fix, or the act of re-installing the driver recompiled the driver on the current running kernel which fixed the issue.{{Navbox Linux}}
	[[Category:Linux]]		[[Category:Linux]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]

Leo: /* Install NVIDIA driver */

2023-07-26T07:49:46Z

Install NVIDIA driver

← Older revision		Revision as of 07:49, 26 July 2023
Line 249:		Line 249:
	}}		}}
	As part of the installation, the driver will ensure nouveau is disabled. It'll blacklist the nouveau kernel module for you but you will have to reboot (or rmmod it) before running the installer again.		As part of the installation, the driver will ensure nouveau is disabled. It'll blacklist the nouveau kernel module for you but you will have to reboot (or rmmod it) before running the installer again.

			Once you have the NVIDIA drivers installed, to hardware accelerate your VM:

			* Set the graphics card of the VM to VirGL GPU
			* Install the needed dependencies: <code>apt install libgl1 libegl1</code>

	=== Setup LXC with NVIDIA GPU ===		=== Setup LXC with NVIDIA GPU ===

Leo at 07:22, 26 July 2023

2023-07-26T07:22:47Z

← Older revision		Revision as of 07:22, 26 July 2023
Line 241:		Line 241:

	Switching <code>/var/lib/docker</code> to a ext backed filesystem fixed the issue.		Switching <code>/var/lib/docker</code> to a ext backed filesystem fixed the issue.

			=== Install NVIDIA driver ===
			The NVIDIA Linux driver requires the following dependencies:
			{{Highlight
			\| code = # apt install gcc make pve-headers
			\| lang = terminal
			}}
			As part of the installation, the driver will ensure nouveau is disabled. It'll blacklist the nouveau kernel module for you but you will have to reboot (or rmmod it) before running the installer again.

	=== Setup LXC with NVIDIA GPU ===		=== Setup LXC with NVIDIA GPU ===
Line 389:		Line 397:
	# Then reinstall grub: <code>proxmox-boot-tool init /dev/sda2</code>		# Then reinstall grub: <code>proxmox-boot-tool init /dev/sda2</code>
	See Proxmox's sysadmin documentation on this topic at: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_setup		See Proxmox's sysadmin documentation on this topic at: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_setup

			=== Migrating VMs from a Proxmox OS disk ===
			I wanted to reinstall Proxmox on another disk on the same machine which required me migrating VMs from the old install to the new install. The easiest way to accomplish this from what I can tell is to tar up /etc/pve from the old disk and restoring it to the new install. What you want to restore are the lxc and qemu-server .conf files.

			You may also want to restore the rrd databases under /var/lib/rrdcached/db

	==Troubleshooting==		==Troubleshooting==