# Passing Two RTX 3090s Into a Windows 11 VM on Proxmox ### \# Passing Two RTX 3090s Into a Windows 11 VM on Proxmox (and the One Line That Actually Fixed It) I added a GPU node to my Proxmox cluster — a Xeon E5-2696 v4 box with \*\*two RTX 3090s\*\* — and set out to pass both cards into a Windows 11 VM (`ai-win`, VMID 142) for CUDA/ML work. Here's the full run: what broke, what the misleading symptoms pointed at, and the fix that actually mattered. ### \## Baseline first Before any GPU work, I brought the node to cluster parity (it shipped on an older PVE): \- Disabled the enterprise apt repos (the ones that spam `401` without a subscription key), added `pve-no-subscription`. \- `apt dist-upgrade`: PVE \*\*8.0.3 → 8.4.19\*\*, kernel \*\*6.2 → 6.8\*\*. \- Patched the subscription nag, confirmed timezone/NTP. \- Rebooted, verified the node rejoined the cluster quorate. Clean starting point. Now the GPUs. ```markdown ## Host-side passthrough prep Standard vfio setup: ```bash # /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" # /etc/modules vfio vfio_iommu_type1 vfio_pci # /etc/modprobe.d/blacklist-gpu.conf blacklist nouveau blacklist nvidiafb blacklist nvidia # /etc/modprobe.d/vfio.conf (both 3090s share the same device IDs) options vfio-pci ids=10de:2204,10de:1aef ``` `update-initramfs -u -k all`, reboot, verify: ``` DMAR: IOMMU enabled 01:00.0 ... Kernel driver in use: vfio-pci 03:00.0 ... Kernel driver in use: vfio-pci ``` IOMMU groups came out clean — each GPU in its own group, no ACS-override hack needed. So far textbook. --- ## Issue 1 — `x-vga=1` locked me out of the VM I attached the first GPU the "gaming VM" way: ```bash qm set 142 -hostpci0 0000:01:00,pcie=1,x-vga=1 ``` VM booted... and the console went black. No RDP either (the guest had no network preconfigured). Total lockout. **Root cause:** `x-vga=1` tells QEMU *"this passthrough GPU **is** the primary display."* It tears down the emulated VGA adapter. On a headless box with no other way in, you've cut the only branch you're sitting on. **Fix:** drop `x-vga=1`, keep an emulated display alongside the passthrough: ```bash qm set 142 -hostpci0 0000:01:00,pcie=1 qm set 142 -vga std ``` **Lesson:** `x-vga=1` is only for a VM driving a *physical monitor*. For a headless compute VM, never use it — you want `vga: std` so the noVNC console always works. The GPU still passes through fine for CUDA. --- ## Issue 2 — the boot that hung forever (the red herring) With one GPU attached (no `x-vga`), the VM still wouldn't come up: one CPU core pinned at 100%, black screen, no network — for five minutes straight. Not a crash, not idle. Stuck. The internet's first answer for "3090 passthrough hangs" is **BIOS: enable Above 4G Decoding / Resizable BAR** (a 24 GB card needs a >4 GB memory window). So I rebooted into firmware and enabled Above 4G Decoding, Resizable BAR, SR-IOV, VT-d. **It changed nothing.** The VM hung exactly the same way. I verified the BIOS settings *were* taking effect — the card's BAR1 now mapped above the 4 GB line: ``` Region 1: Memory at 383000000000 (64-bit, prefetchable) [size=32G] ``` 32 GB window, mapped high. Above 4G was working perfectly. **It was never the problem.** Easy to burn an hour here chasing the popular answer instead of reading your own logs. --- ## Issue 3 — the actual root cause, in the host logs Instead of guessing, I cleared the kernel ring buffer, booted the VM, and read `dmesg`: ``` vfio-pci 0000:01:00.0: No more image in the PCI ROM ``` **That's the real fault.** The guest can't read the GPU's vBIOS, so Windows hangs initializing the card. Ampere cards frequently need the vBIOS handed to them explicitly via a **romfile** rather than read live from the card. Dump the vBIOS on the host (GPUs are vfio-bound, so the host driver isn't holding them): ```bash echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom cat /sys/bus/pci/devices/0000:01:00.0/rom > /usr/share/kvm/gpu-01.rom echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom ``` A valid ROM starts with the `55 aa` signature: ``` 000000 55 aa 7a eb ... # good ``` Attach with the romfile: ```bash qm set 142 -hostpci0 0000:01:00,pcie=1,romfile=gpu-01.rom ``` Boot. Windows reached the desktop, network up, **zero `PCI ROM` errors** in `dmesg`. Fixed. --- ## Issue 4 — the second card wouldn't dump Live-dumping the second 3090's vBIOS returned **0 bytes** — a common quirk (only one card can be read at a time depending on VGA arbitration). The two cards were different brands (ASUS + MSI) but the **same GA102 die**. So I tried the first card's ROM for both: ```bash qm set 142 -hostpci1 0000:03:00,pcie=1,romfile=gpu-01.rom ``` Booted clean with both GPUs. Same-die vBIOS is close enough — no need to hunt the exact MSI ROM on TechPowerUp. --- ## Issue 5 — self-inflicted: corrupting Windows with hard stops The VM had **no QEMU guest agent**, so `qm shutdown` (which asks Windows nicely over ACPI) just hung. I fell back to `qm stop` (pull the plug) — repeatedly, mid-boot, while iterating on the config. Predictable result: Windows dropped into a **"Looking for solutions..." repair loop**. That wasn't the GPU — that was me yanking power during boot. **Lessons:** - No guest agent → `qm shutdown` hangs; `qm stop` is the fallback, but **never mid-boot**. - **Snapshot before you start poking.** Once the VM was healthy, I took one immediately: ```bash qm snapshot 142 pre_gpu_passthrough --description "clean Win11, healthy, pre-passthrough" ``` Every risky attempt after that had a one-command rollback. --- ## Debugging a headless VM with no network When the guest gives you nothing — no agent, no ping, no RDP — you can still see its screen straight from the hypervisor: ```bash qm monitor 142 <<< "screendump /tmp/142.ppm" # copy off the host, convert locally convert /tmp/142.ppm /tmp/142.png ``` That screenshot is what turned "it's broken" into "it's at the Windows lock screen" vs. "it's in a repair loop" — completely different problems, indistinguishable from a black VNC window. --- ## Final working config ``` bios: ovmf machine: pc-q35-9.0 vga: std hostpci0: 0000:01:00,pcie=1,romfile=gpu-01.rom hostpci1: 0000:03:00,pcie=1,romfile=gpu-01.rom ``` Both RTX 3090s passed through, Windows 11 boots to desktop, console stays reachable. Last step is installing the NVIDIA Studio driver inside the guest — until then the cards show as "Basic Display Adapter." ## Takeaways 1. **Read your own `dmesg` before applying the popular fix.** The internet said "BIOS Above 4G." The logs said `No image in the PCI ROM`. The logs were right. 2. **`romfile=` is the real fix for RTX 3090 passthrough**, not BIOS BARs (those just need to be on, which they usually already are). 3. **Never `x-vga=1` on a headless compute VM** — it removes your only console. 4. **Snapshot before experimenting**, and never hard-stop a VM mid-boot. 5. **`screendump` from the QEMU monitor** is your eyes when the guest is otherwise dark.