Are NVIDIA GPU manual driver unloading and loading with qemu hooks still needed?

[u/u0_a321]

Hello, Can some help me with this question?

I have an NVIDIA GPU that I pass through to a Windows 11 VM using QEMU/KVM and libvirt.

I used to have libvirt hooks that stopped the display manager, unloaded the NVIDIA modules, manually detached the GPU with `virsh nodedev-detach`, and then loaded `vfio-pci`.

I recently removed those hooks to see if they were still needed. Surprisingly, passthrough still works fine. When the VM starts, the GPU is used by the VM, and when the VM shuts down, the NVIDIA driver works normally on the host again.

My GPU is configured as a managed PCI device in libvirt (`managed="yes"`).

So my question is: Are these manual NVIDIA driver unload / PCI detach hooks still needed with current QEMU/KVM + libvirt, or does libvirt handle this automatically now?

I'm mainly wondering if there is a reason to keep the old hooks even though everything seems to work without them.

[u/Max-P]

I have never needed those hooks. AFAIK they had more to do with Xorg crashing when a GPU is removed (even if it's not even using it) than the drivers themselves. In the age of Wayland it's basically useless, the compositors handle this just fine. At least KWin does, it's even a tested use case because one of the developers use VFIO and thus wants it to work correctly.

[u/u0_a321 → u/Max-P]

Ok thank you. So that means I no longer have to keep using the hooks.

[u/five35]

I only discovered managed="yes" a few days ago and I'm delighted by how well it works. For me, it even works better, as I no longer need to worry about e.g. …/bind vs. …/new_id or tracking what the original driver was. IMO, letting libvirt handle driver bindings is miles better than continuing to use hooks.

The only wrinkle I ran into was KWin grabbing the card when libvirt released it back to the host. Whatever libvirt does to rebind hardware back to the original drivers, KWin seems to think it means "free real estate".

The solution was simple, though — just set environment variables KWIN_DRM_DEVICES=/dev/dri/by-path/pci-…-card and KWIN_RENDER_NODES=/dev/dri/by-path/pci-…-render for my integrated graphics. After that, KWin leaves my NVIDIA card alone and I can freely switch between native Linux gaming and passthrough Windows gaming.

[u/u0_a321 → u/five35]

Oh nice. I use Hyprland, and I'm yet to have any issues without the hooks. But what I'm concerned about now(I didn't think about this when I tried passthroughing without hooks) is, what happens when I start the VM, but some process is using the GPU? Does the VM fail to start?

[u/five35 → u/u0_a321]

To be honest, I'd been putting off testing this because getting it wrong with hooks caused problems for me which could only be solved with a hard reboot. 😅

And it turns out that… yeah, libvirt's rebinding causes the same problems. I don't know whether it's something to do with the hardware or something in the nvidia-open driver I'm using, but trying to unbind the GPU while it's in use causes some process within in the kernel to hang. The system in general stays usable, but I can't use the dGPU for anything until I give my power button the Hug of Death.

So it seems you still need either something which checks for that before starting your VM or a hook which exists just to cause an error if the card is in use.

I start my VM using the following script, which uses lsof to make sure the relevant devices aren't in use.

```

!/bin/sh

if ! lsof -t /dev/dri/by-path/pci-0000:01:00.0-* > /dev/null; then

&2 echo "Cannot start; the dGPU is in use."

exit 1 fi

virsh start virtual-cyclone ```

[u/u0_a321 → u/five35]

For my hooks, I always stop the display manager sddm first, then detach the gpu, unload kernel modules, then bind it to vfio pci, and start the display manager again. But after a vm shutdown, indo everything in reverse, except restart the display manager. And it works. I can share my hooks if needed.