[Fail] IOMMU and VFIO GPU Passthrough on a HP Zbook 14 laptop with dual graphics

(RAW, unedited dump of my notes that ends in a glorious failure) Goals for today:

  • IOMMU groups on Zbook 14 G1 + Slackware 14.2
  • GPU passthrough for a second graphics card (some kind of a Radeon)
  • (maybe) Looking Glass to copy framebuffer to the Intel GPU

We are going to check the iommu capabilities of my laptop first, so: add to the lilo append line:

append " intel_iommu=on"

My line looks like:

append=" vt.default_utf8=1 intel_iommu=on"

(After this step and a restart my WiFi connection required re-entering password 😛 )

Check the ls-iommu.sh to see the groups, in my case:

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Haswell-ULT DRAM Controller [8086:0a04] (rev 0b)
IOMMU Group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation Haswell-ULT Integrated Graphics Controller [8086:0a16] (rev 0b)
IOMMU Group 10 00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 6 [8086:9c1a] (rev e4)
IOMMU Group 11 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series USB EHCI #1 [8086:9c26] (rev 04)
IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation 8 Series LPC Controller [8086:9c43] (rev 04)
IOMMU Group 12 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series SATA Controller 1 [AHCI mode] [8086:9c03] (rev 04)
IOMMU Group 12 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series SMBus Controller [8086:9c22] (rev 04)
IOMMU Group 13 02:00.0 Network controller [0280]: Intel Corporation Wireless 7260 [8086:08b1] (rev 73)
IOMMU Group 14 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Mars [Radeon HD 8730M] [1002:6601] (rev ff)
IOMMU Group 14 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] (rev ff)
IOMMU Group 15 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader [10ec:5227] (rev 01)
IOMMU Group 2 00:03.0 Audio device [0403]: Intel Corporation Haswell-ULT HD Audio Controller [8086:0a0c] (rev 0b)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 8 Series USB xHCI HC [8086:9c31] (rev 04)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 8 Series HECI #0 [8086:9c3a] (rev 04)
IOMMU Group 4 00:16.3 Serial controller [0700]: Intel Corporation 8 Series HECI KT [8086:9c3d] (rev 04)
IOMMU Group 5 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I218-LM [8086:155a] (rev 04)
IOMMU Group 6 00:1b.0 Audio device [0403]: Intel Corporation 8 Series HD Audio Controller [8086:9c20] (rev 04)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 1 [8086:9c10] (rev e4)
IOMMU Group 8 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 4 [8086:9c16] (rev e4)
IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 5 [8086:9c18] (rev e4)

Graphics are separated and I can give one the ethernet controllers to the virtual machine later on.

Check the drivers loaded for the devices:

lspci -nnk

Check if your kernel currently uses CONFIG_VFIO_PCI_VGA: (zgrep for grep in an gz archive)

zgrep CONFIG_VFIO_PCI_VGA /proc/config.gz

or check your /boot/config-* files, but as of 14.2 they are not set

Recompile the kernel to use CONFIG_VFIO_PCI_VGA:

cd /usr/src/linux-4.4.14
mv .config .config.default
cp /boot/config-huge-4.4.14 /usr/src/linux-4.4.14/.config

make menuconfig

Device Drivers – VFIO Non-Privileged userspace driver framework – VFIO PCI support for VGA devices
General Setup – Local Version – “.anil”

make bzImage
make modules

Create a new copy (cp could preserve some attribs/sparse info, cat is more … pure? honestly, the manual says so)

cat arch/x86/boot/bzImage > /boot/vmlinuz-4.4.14.anil

Compile modules:

make modules_install

Lilo:

image = /boot/vmliuz-4.4.14.anil
    root = /dev/sda1
    label = LinuxVFIO
    read-only

Reboot and check:

bash-4.3$ zgrep VFIO /proc/config.gz 
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_KVM_VFIO=y

Ok, good to go.

I want to forward those two first:
IOMMU Group 14 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Mars [Radeon HD 8730M] [1002:6601] (rev ff)
IOMMU Group 14 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] (rev ff)

so the id are 1002:6601 and 1002:aab0, but lspci -nnk shows the audio to be loaded using snd_hda_intel, so maybe … I don’t need it yet? GPU is enough for now.

Now configure the modprobe, create a file /etc/modprobe.d/vfio.conf and put:

blacklist radeon
options vfio-pci ids=1002:6601

I blacklisted radeon driver just in case so that it can’t conflict in any way.

After a reboot I tried modprobe vfio-pci and after lspci -nnk got:

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Mars [Radeon HD 8730M] [1002:6601]
        Subsystem: Hewlett-Packard Company FirePro M4100 [103c:2100]
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

Ok, seems nice. Now it’s time to install the missing software:

Virtualization packages

SMARTCARD support (just in case, you can skip it)

groupadd -g 257 pcscd
useradd -u 257 -g pcscd -d /var/run/pcscd -s /bin/false pcscd

pcsc-lite
libcacard

Spice

celt051
six
pyparsing
spice-protocol
spice

vala
spice-gtk

usbredir

usbredir

device-tree-compiler

(for ARM support)

device-tree-compiler

qemu

qemu

libvirt

urlgrabber
yajl
libvirt
libvirt-glib
libvirt-python
libosinfo

START IT

/etc/rc.d/rc.libvirt start

grab win10 image: (from the official Microsoft website)

grab driver for accessing virtual hard drive:

https://docs.fedoraproject.org/quick-docs/en-US/creating-windows-virtual-machines-using-virtio-drivers.html#Direct_download
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.141-1/virtio-win-0.1.141.iso

get the bios: (the name changes every day, edk2.git-ovmf-x64)

https://www.kraxel.org/repos/jenkins/edk2/
https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20180413.57.gd4ccf26491.noarch.rpm
rpm2tgz edk2.git-ovmf-x64-0-20180413.57.gd4ccf26491.noarch.rpm
installpkg edk2.git-ovmf-x64-0-20180413.57.gd4ccf26491.noarch.tgz

download kvm-install.sh, readjust file name

modprobe kvm-intel

readjust number of bounded cpu/threads !!!!

modprobe vfio-pci

test without vfio

test1-install.sh

#!/bin/sh

INSTALLFILE=test1.qcow2
FILESIZE=40G

INSTALLCD=/home/anil/iommu/Win10_1709_Polish_x64.iso
# if you use a hardware CD-ROM drive, check for the device. In most cases it's /dev/sr0
#INSTALLCD=/dev/sr0

DRIVERCD=/home/anil/iommu/virtio-win-0.1.141.iso

# PCI address of the passtrough devices
# DEVICE1="03:00.0"
# DEVICE2="01:00.1"

# create installation file if not exist
if [ ! -e $INSTALLFILE ]; then
    qemu-img create -f qcow2 $INSTALLFILE $FILESIZE
fi

#QEMU_PA_SAMPLES=4096 QEMU_AUDIO_DRV=pa \
qemu-system-x86_64 \
-bios /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd \
-cpu host \
-device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=3 \
-device usb-kbd \
-device usb-tablet \
-device virtio-blk-pci,scsi=off,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 \
-device virtio-net-pci,netdev=user.0,mac=52:54:00:a0:66:43 \
-drive file=$DRIVERCD,if=none,id=drive-ide0-1-0,readonly=on,format=raw \
-drive file=$INSTALLCD,if=none,id=drive-ide0-0-0,readonly=on,format=raw \
-drive file=$INSTALLFILE,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe \
-netdev user,id=user.0 \
-machine pc-i440fx-2.1,accel=kvm \
-enable-kvm \
-m 4096 \
-rtc base=localtime,driftfix=slew \
-soundhw hda \
-smp 1,sockets=1,cores=1,threads=1 \
-usb \
-vga qxl \

when creating partitions choose driver:

E:\viostor\w10\amd64

it will be displayed as

Red Hat VirtIO SCSI controller

OK, the install should be ready, now time for spice: (add it to a qemu .sh script)

-spice port=5900,addr=127.0.0.1,disable-ticketing

and launch

spicy

and connect to localhost at 5900

VFIO test 1: …shit

I forwarded only 03:00.0, the AMD GPU, to the host, without the sound card
Seems like I have to – they are in the same group. But the audio card uses snd_hda_intel driver, so I have to blacklist it
during boot and load it later only for the Intel soundcard

Btw. my error was:

qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=0x8.0x0,multifunction=on: vfio error: 0000:03:00.0: group 14 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Now to get rid of it:

file /etc/modprobe.d/snd_hda_intel.conf

blacklist snd_hda_intel

file /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:6601,1002:aab0

Now modify qemu:

-device vfio-pci,host=$DEVICE1,addr=0x8.0x0,multifunction=on,x-vga=on \
-device vfio-pci,host=$DEVICE2,addr=0x8.0x1 \

Warning: if you add machine …,iommu=on you will get:
qemu-system-x86_64: Property ‘.iommu’ not found

memory error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

dmesg:
[ 2197.603335] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 2197.603562] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[ 2198.944507] vfio-pci 0000:03:00.0: Invalid ROM contents
[ 2199.095967] kvm: zapping shadow pages for mmio generation wraparound
[ 2199.099047] kvm: zapping shadow pages for mmio generation wraparound

and the screen looks weird (ctrl-alt-f2 to a console and back to f7 solves it)

amd firepro m4100 driver

I downloaded the driver from the AMD website in .zip, unpacked it into amd/ folder and did
mkisofs -o amd_firepro.iso amd/

… and so far I’m stuck on a wrong BIOS for the graphics card, I even tried modifying some dumped BIOSs (but it looks like my graphics has no UEFI-compatible BIOS, at least GPU-Z tells me that) anyone wants to continue from here?