Table of Contents

Ubuntu - GPU - GPU pass-through - Configuring the host

Verify the host environment supports GPU pass-through

TODO:

Enable IOMMU

IOMMU is usually disabled by default.

For AMD-based hosts:

/etc/default/grub
GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on rd.driver.pre=vfio-pci"

For Intel-based hosts:

/etc/default/grub
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt rd.driver.pre=vfio-pci"

Re-generate the main GRUB 2 configuration file

The GRUB 2 configuration file is /boot/grub2/grub.cfg. 

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot

Verify that IOMMU is enabled

sudo dmesg | grep -e DMAR -e IOMMU

returns: 

[    2.172228] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    2.176588] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.176924] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    3.875165] AMD-Vi: AMD IOMMUv2 loaded and initialized

Blacklist the Nouveau driver

Prevent the host OS from loading the built-in nouveau driver.

Create the file /etc/modprobe.d/60-blacklist-nouveau.conf:

/etc/modprobe.d/60-blacklist-nouveau.conf
blacklist nouveau

Configure VFIO and isolate the GPU used for pass-through

Find the card vendor and model IDs. 

lspci -k | grep VGA

returns: 

0c:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] (rev c8)

NOTE: This shows the Bus number is 0c:00.0.

Get the device ID

Using the Bus number identified earlier. 

lspci -nn | grep 0c:00.0

returns: 

0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)

NOTE: This shows the ID is 1002:744c.

Create a module VFIO file

Create the file /etc/modprobe.d/vfio.conf:

/etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:744c

NOTE: Verify that the card does not need an extra ids= parameter.

  • For some cards, the audio device must be specified too, so that ID must also be added to the list, otherwise the card will be unusable.

Load the VFIO driver

There are various ways you can load the VFIO driver:

Including the driver in the initrd file

Create the file /etc/dracut.conf.d/gpu-passthrough.conf: 

    add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

NOTE: Mind the leading whitespace.

Re-generate the initrd file: 

sudo dracut --force /boot/initrd $(uname -r)

Adding the driver to the list of auto-loaded modules

Create the file /etc/modules-load.d/vfio-pci.conf:

/etc/modules-load.d/vfio-pci.conf
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

Load the driver manually

To load the driver manually at runtime: 

sudo modprobe vfio-pci

Disable MSR for Microsoft Windows guests

For Microsoft Windows guests, it is recommended to disable MSR (model-specific register) to avoid the guest crashing.

Create the file /etc/modprobe.d/kvm.conf:

/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

Install UEFI firmware

For proper GPU Pass-Through functionality, the host needs to boot using UEFI firmware.

Install the qemu-ovmf package if not already installed: 

sudo apt install qemu-ovmf

Reboot the host machine

For most of the changes in the above steps to take effect, the host machine needs to be rebooted. 

sudo shutdown -r now