GPU Passthrough

All PVE nodes must be configured for LXC/VM migration support.

Host (PVE)

Kernel Options

AMDIntel

/etc/default/grub

0644 root:root

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

/etc/modules-load.d/video.conf

0644 root:root

# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
vfio
vfio_iommu_type1
vfio_pci

/etc/default/grub

0644 root:root

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt i915.enable_gvt=1"

/etc/modules-load.d/video.conf

0644 root:root

# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
vfio
vfio_iommu_type1
vfio_pci
kvmgt

# Update GRUB, reboot, and confirm GPU reported.
update-grub
reboot

dmesg | grep -e DMAR -e IOMMU  # IOMMU enabled w. graphics passthrough.
lspci -nnv | grep -i vga  # PCI device reported.

# Confirm kernel booted with para-virtualization enabled.
cat /proc/cmdline

# Confirm GPU is associated with kernel driver.
lspci -k | grep -A 3 -i "VGA"

Map non-root users to GPU

Unprivileged containers require other R/W permissions on GPU.

# Get PVE GPU groups and major device ID
getent group video
> video:x:44:root  # PVE GID: 44
getent group render
> render:x:993:root  # PVE GID: 993

ls -l /dev/dri
> crw-rw---- 1 root video  226,   0 May 12 21:54 card1  # Major ID 226.
> crw-rw---- 1 root render 226, 128 May 12 21:54 renderD128  # Major ID 226.

/etc/subgid

0644 root:root

# Enable root mapping of PVE render, video groups for unprivileged
# containers. ALWAYS confirm group ID's as they may change between major OS
# versions.
root:44:1
root:993:1

/etc/udev/rules.d/59-igpu-passthrough.rules

0644 root:root

# Map others R/W to GPU on boot.
KERNEL=="renderD128", MODE="0666"
KERNEL=="card1", MODE="0666"

Container configuration

# Add root user to render, video groups.
usermod -aG render,video root

# Get LXC GPU groups and major device ID
getent group video
> video:x:44:root  # LXC GID: 44
getent group render
> render:x:992:root  # LXC GID: 992

Map LXC groups to PVE

On PVE Host

/etc/pve/lxc/{ID}.conf

0644 root:root

# Map major device ID to LXC container.
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file,mode=0666
# Map LXC render,video groups to PVE render,video groups.
#
# Generate base mapping with: https://github.com/ddimick/proxmox-lxc-idmapper
#
#   run.py :{LXC GID}=:{PVE GID}
#
#   run.py :44=:44 :992=:993
lxc.idmap: u 0 100000 65536   # UID 0-65536 (LXC) to 100000-165535 (PVE).
lxc.idmap: g 0 100000 44      # GID 0-43 (LXC) to 100000-100043 (PVE).
lxc.idmap: g 44 44 1          # GID 44 same on LXC/PVE.
lxc.idmap: g 45 100045 947    # GID 45-992 (LXC) to 100045-100992 (PVE).
lxc.idmap: g 992 993 1        # GID 992 (LXC) to 993 (PVE).
lxc.idmap: g 994 100994 64543 # GID 994-65535 (LXC) to 100994-165535 (PVE).

# Restart LXC container and verify device appears with video, render groups.
ls -l /dev/dri

Reference¹²³