(RAW, unedited dump of my notes that ends in a glorious failure) Goals for today:
- IOMMU groups on Zbook 14 G1 + Slackware 14.2
- GPU passthrough for a second graphics card (some kind of a Radeon)
- (maybe) Looking Glass to copy framebuffer to the Intel GPU
We are going to check the iommu capabilities of my laptop first, so: add to the lilo append line:
append " intel_iommu=on"
My line looks like:
append=" vt.default_utf8=1 intel_iommu=on"
(After this step and a restart my WiFi connection required re-entering password 😛 )
Check the ls-iommu.sh to see the groups, in my case:
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Haswell-ULT DRAM Controller [8086:0a04] (rev 0b) IOMMU Group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation Haswell-ULT Integrated Graphics Controller [8086:0a16] (rev 0b) IOMMU Group 10 00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 6 [8086:9c1a] (rev e4) IOMMU Group 11 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series USB EHCI #1 [8086:9c26] (rev 04) IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation 8 Series LPC Controller [8086:9c43] (rev 04) IOMMU Group 12 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series SATA Controller 1 [AHCI mode] [8086:9c03] (rev 04) IOMMU Group 12 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series SMBus Controller [8086:9c22] (rev 04) IOMMU Group 13 02:00.0 Network controller [0280]: Intel Corporation Wireless 7260 [8086:08b1] (rev 73) IOMMU Group 14 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Mars [Radeon HD 8730M] [1002:6601] (rev ff) IOMMU Group 14 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] (rev ff) IOMMU Group 15 04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader [10ec:5227] (rev 01) IOMMU Group 2 00:03.0 Audio device [0403]: Intel Corporation Haswell-ULT HD Audio Controller [8086:0a0c] (rev 0b) IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 8 Series USB xHCI HC [8086:9c31] (rev 04) IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 8 Series HECI #0 [8086:9c3a] (rev 04) IOMMU Group 4 00:16.3 Serial controller [0700]: Intel Corporation 8 Series HECI KT [8086:9c3d] (rev 04) IOMMU Group 5 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I218-LM [8086:155a] (rev 04) IOMMU Group 6 00:1b.0 Audio device [0403]: Intel Corporation 8 Series HD Audio Controller [8086:9c20] (rev 04) IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 1 [8086:9c10] (rev e4) IOMMU Group 8 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 4 [8086:9c16] (rev e4) IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series PCI Express Root Port 5 [8086:9c18] (rev e4)
Graphics are separated and I can give one the ethernet controllers to the virtual machine later on.
Check the drivers loaded for the devices:
lspci -nnk
Check if your kernel currently uses CONFIG_VFIO_PCI_VGA: (zgrep for grep in an gz archive)
zgrep CONFIG_VFIO_PCI_VGA /proc/config.gz
or check your /boot/config-* files, but as of 14.2 they are not set
Recompile the kernel to use CONFIG_VFIO_PCI_VGA:
cd /usr/src/linux-4.4.14 mv .config .config.default cp /boot/config-huge-4.4.14 /usr/src/linux-4.4.14/.config
make menuconfig
Device Drivers – VFIO Non-Privileged userspace driver framework – VFIO PCI support for VGA devices
General Setup – Local Version – “.anil”
make bzImage make modules
Create a new copy (cp could preserve some attribs/sparse info, cat is more … pure? honestly, the manual says so)
cat arch/x86/boot/bzImage > /boot/vmlinuz-4.4.14.anil
Compile modules:
make modules_install
Lilo:
image = /boot/vmliuz-4.4.14.anil
root = /dev/sda1
label = LinuxVFIO
read-only
Reboot and check:
bash-4.3$ zgrep VFIO /proc/config.gz CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO_VIRQFD=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_VFIO_PCI_MMAP=y CONFIG_VFIO_PCI_INTX=y CONFIG_KVM_VFIO=y
Ok, good to go.
I want to forward those two first:
IOMMU Group 14 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Mars [Radeon HD 8730M] [1002:6601] (rev ff)
IOMMU Group 14 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] (rev ff)
so the id are 1002:6601 and 1002:aab0, but lspci -nnk shows the audio to be loaded using snd_hda_intel, so maybe … I don’t need it yet? GPU is enough for now.
Now configure the modprobe, create a file /etc/modprobe.d/vfio.conf and put:
blacklist radeon options vfio-pci ids=1002:6601
I blacklisted radeon driver just in case so that it can’t conflict in any way.
After a reboot I tried modprobe vfio-pci and after lspci -nnk got:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Mars [Radeon HD 8730M] [1002:6601]
Subsystem: Hewlett-Packard Company FirePro M4100 [103c:2100]
Kernel driver in use: vfio-pci
Kernel modules: radeon
Ok, seems nice. Now it’s time to install the missing software:
Virtualization packages
SMARTCARD support (just in case, you can skip it)
groupadd -g 257 pcscd useradd -u 257 -g pcscd -d /var/run/pcscd -s /bin/false pcscd
pcsc-lite libcacard
Spice
celt051 six pyparsing spice-protocol spice
vala spice-gtk
usbredir
usbredir
device-tree-compiler
(for ARM support)
device-tree-compiler
qemu
qemu
libvirt
urlgrabber yajl libvirt libvirt-glib libvirt-python libosinfo
START IT
/etc/rc.d/rc.libvirt start
grab win10 image: (from the official Microsoft website)
grab driver for accessing virtual hard drive:
https://docs.fedoraproject.org/quick-docs/en-US/creating-windows-virtual-machines-using-virtio-drivers.html#Direct_download https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.141-1/virtio-win-0.1.141.iso
get the bios: (the name changes every day, edk2.git-ovmf-x64)
https://www.kraxel.org/repos/jenkins/edk2/ https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20180413.57.gd4ccf26491.noarch.rpm rpm2tgz edk2.git-ovmf-x64-0-20180413.57.gd4ccf26491.noarch.rpm installpkg edk2.git-ovmf-x64-0-20180413.57.gd4ccf26491.noarch.tgz
download kvm-install.sh, readjust file name
modprobe kvm-intel
readjust number of bounded cpu/threads !!!!
modprobe vfio-pci
test without vfio
test1-install.sh
#!/bin/sh
INSTALLFILE=test1.qcow2
FILESIZE=40G
INSTALLCD=/home/anil/iommu/Win10_1709_Polish_x64.iso
# if you use a hardware CD-ROM drive, check for the device. In most cases it's /dev/sr0
#INSTALLCD=/dev/sr0
DRIVERCD=/home/anil/iommu/virtio-win-0.1.141.iso
# PCI address of the passtrough devices
# DEVICE1="03:00.0"
# DEVICE2="01:00.1"
# create installation file if not exist
if [ ! -e $INSTALLFILE ]; then
qemu-img create -f qcow2 $INSTALLFILE $FILESIZE
fi
#QEMU_PA_SAMPLES=4096 QEMU_AUDIO_DRV=pa \
qemu-system-x86_64 \
-bios /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd \
-cpu host \
-device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=3 \
-device usb-kbd \
-device usb-tablet \
-device virtio-blk-pci,scsi=off,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 \
-device virtio-net-pci,netdev=user.0,mac=52:54:00:a0:66:43 \
-drive file=$DRIVERCD,if=none,id=drive-ide0-1-0,readonly=on,format=raw \
-drive file=$INSTALLCD,if=none,id=drive-ide0-0-0,readonly=on,format=raw \
-drive file=$INSTALLFILE,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe \
-netdev user,id=user.0 \
-machine pc-i440fx-2.1,accel=kvm \
-enable-kvm \
-m 4096 \
-rtc base=localtime,driftfix=slew \
-soundhw hda \
-smp 1,sockets=1,cores=1,threads=1 \
-usb \
-vga qxl \
when creating partitions choose driver:
E:\viostor\w10\amd64
it will be displayed as
Red Hat VirtIO SCSI controller
OK, the install should be ready, now time for spice: (add it to a qemu .sh script)
-spice port=5900,addr=127.0.0.1,disable-ticketing
and launch
spicy
and connect to localhost at 5900
VFIO test 1: …shit
I forwarded only 03:00.0, the AMD GPU, to the host, without the sound card
Seems like I have to – they are in the same group. But the audio card uses snd_hda_intel driver, so I have to blacklist it
during boot and load it later only for the Intel soundcard
Btw. my error was:
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=0x8.0x0,multifunction=on: vfio error: 0000:03:00.0: group 14 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
Now to get rid of it:
file /etc/modprobe.d/snd_hda_intel.conf
blacklist snd_hda_intel
file /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:6601,1002:aab0
Now modify qemu:
-device vfio-pci,host=$DEVICE1,addr=0x8.0x0,multifunction=on,x-vga=on \ -device vfio-pci,host=$DEVICE2,addr=0x8.0x1 \
Warning: if you add machine …,iommu=on you will get:
qemu-system-x86_64: Property ‘.iommu’ not found
memory error:
qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=
dmesg:
[ 2197.603335] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 2197.603562] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[ 2198.944507] vfio-pci 0000:03:00.0: Invalid ROM contents
[ 2199.095967] kvm: zapping shadow pages for mmio generation wraparound
[ 2199.099047] kvm: zapping shadow pages for mmio generation wraparound
and the screen looks weird (ctrl-alt-f2 to a console and back to f7 solves it)
amd firepro m4100 driver
I downloaded the driver from the AMD website in .zip, unpacked it into amd/ folder and did
mkisofs -o amd_firepro.iso amd/
… and so far I’m stuck on a wrong BIOS for the graphics card, I even tried modifying some dumped BIOSs (but it looks like my graphics has no UEFI-compatible BIOS, at least GPU-Z tells me that) anyone wants to continue from here?