Skip to main content

Passing Two RTX 3090s Into a Windows 11 VM on Proxmox

# Passing Two RTX 3090s Into a Windows 11 VM on Proxmox (and the One Line That Actually Fixed It)

I added a GPU node to my Proxmox cluster — a Xeon E5-2696 v4 box with **two RTX 3090s** — and set out to pass both cards into a Windows 11 VM (`ai-win`, VMID 142) for CUDA/ML work. Here's the full run: what broke, what the misleading symptoms pointed at, and the fix that actually mattered.

## Baseline first

Before any GPU work, I brought the node to cluster parity (it shipped on an older PVE):

- Disabled the enterprise apt repos (the ones that spam `401` without a subscription key), added `pve-no-subscription`.
- `apt dist-upgrade`: PVE **8.0.3 → 8.4.19**, kernel **6.2 → 6.8**.
- Patched the subscription nag, confirmed timezone/NTP.
- Rebooted, verified the node rejoined the cluster quorate.

Clean starting point. Now the GPUs.



## Host-side passthrough prep

Standard vfio setup:

```bash
# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# /etc/modules
vfio
vfio_iommu_type1
vfio_pci

# /etc/modprobe.d/blacklist-gpu.conf
blacklist nouveau
blacklist nvidiafb
blacklist nvidia

# /etc/modprobe.d/vfio.conf  (both 3090s share the same device IDs)
options vfio-pci ids=10de:2204,10de:1aef

update-initramfs -u -k all, reboot, verify:

DMAR: IOMMU enabled
01:00.0 ... Kernel driver in use: vfio-pci
03:00.0 ... Kernel driver in use: vfio-pci

IOMMU groups came out clean — each GPU in its own group, no ACS-override hack needed. So far textbook.

Issue 1 — x-vga=1 locked me out of the VM

I attached the first GPU the "gaming VM" way:

qm set 142 -hostpci0 0000:01:00,pcie=1,x-vga=1

VM booted... and the console went black. No RDP either (the guest had no network preconfigured). Total lockout.

Root cause: x-vga=1 tells QEMU "this passthrough GPU is the primary display." It tears down the emulated VGA adapter. On a headless box with no other way in, you've cut the only branch you're sitting on.

Fix: drop x-vga=1, keep an emulated display alongside the passthrough:

qm set 142 -hostpci0 0000:01:00,pcie=1
qm set 142 -vga std

Lesson: x-vga=1 is only for a VM driving a physical monitor. For a headless compute VM, never use it — you want vga: std so the noVNC console always works. The GPU still passes through fine for CUDA.

Issue 2 — the boot that hung forever (the red herring)

With one GPU attached (no x-vga), the VM still wouldn't come up: one CPU core pinned at 100%, black screen, no network — for five minutes straight. Not a crash, not idle. Stuck.

The internet's first answer for "3090 passthrough hangs" is BIOS: enable Above 4G Decoding / Resizable BAR (a 24 GB card needs a >4 GB memory window). So I rebooted into firmware and enabled Above 4G Decoding, Resizable BAR, SR-IOV, VT-d.

It changed nothing. The VM hung exactly the same way.

I verified the BIOS settings were taking effect — the card's BAR1 now mapped above the 4 GB line:

Region 1: Memory at 383000000000 (64-bit, prefetchable) [size=32G]

32 GB window, mapped high. Above 4G was working perfectly. It was never the problem. Easy to burn an hour here chasing the popular answer instead of reading your own logs.

Issue 3 — the actual root cause, in the host logs

Instead of guessing, I cleared the kernel ring buffer, booted the VM, and read dmesg:

vfio-pci 0000:01:00.0: No more image in the PCI ROM

That's the real fault. The guest can't read the GPU's vBIOS, so Windows hangs initializing the card. Ampere cards frequently need the vBIOS handed to them explicitly via a romfile rather than read live from the card.

Dump the vBIOS on the host (GPUs are vfio-bound, so the host driver isn't holding them):

echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom
cat /sys/bus/pci/devices/0000:01:00.0/rom > /usr/share/kvm/gpu-01.rom
echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

A valid ROM starts with the 55 aa signature:

000000  55 aa 7a eb ...   # good

Attach with the romfile:

qm set 142 -hostpci0 0000:01:00,pcie=1,romfile=gpu-01.rom

Boot. Windows reached the desktop, network up, zero PCI ROM errors in dmesg. Fixed.

Issue 4 — the second card wouldn't dump

Live-dumping the second 3090's vBIOS returned 0 bytes — a common quirk (only one card can be read at a time depending on VGA arbitration).

The two cards were different brands (ASUS + MSI) but the same GA102 die. So I tried the first card's ROM for both:

qm set 142 -hostpci1 0000:03:00,pcie=1,romfile=gpu-01.rom

Booted clean with both GPUs. Same-die vBIOS is close enough — no need to hunt the exact MSI ROM on TechPowerUp.

Issue 5 — self-inflicted: corrupting Windows with hard stops

The VM had no QEMU guest agent, so qm shutdown (which asks Windows nicely over ACPI) just hung. I fell back to qm stop (pull the plug) — repeatedly, mid-boot, while iterating on the config.

Predictable result: Windows dropped into a "Looking for solutions..." repair loop. That wasn't the GPU — that was me yanking power during boot.

Lessons:

  • No guest agent → qm shutdown hangs; qm stop is the fallback, but never mid-boot.
  • Snapshot before you start poking. Once the VM was healthy, I took one immediately: 
    qm snapshot 142 pre_gpu_passthrough --description "clean Win11, healthy, pre-passthrough"
    Every risky attempt after that had a one-command rollback.

Debugging a headless VM with no network

When the guest gives you nothing — no agent, no ping, no RDP — you can still see its screen straight from the hypervisor:

qm monitor 142 <<< "screendump /tmp/142.ppm"
# copy off the host, convert locally
convert /tmp/142.ppm /tmp/142.png

That screenshot is what turned "it's broken" into "it's at the Windows lock screen" vs. "it's in a repair loop" — completely different problems, indistinguishable from a black VNC window.

Final working config

bios: ovmf
machine: pc-q35-9.0
vga: std
hostpci0: 0000:01:00,pcie=1,romfile=gpu-01.rom
hostpci1: 0000:03:00,pcie=1,romfile=gpu-01.rom

Both RTX 3090s passed through, Windows 11 boots to desktop, console stays reachable. Last step is installing the NVIDIA Studio driver inside the guest — until then the cards show as "Basic Display Adapter."

Takeaways

  1. Read your own dmesg before applying the popular fix. The internet said "BIOS Above 4G." The logs said No image in the PCI ROM. The logs were right.
  2. romfile= is the real fix for RTX 3090 passthrough, not BIOS BARs (those just need to be on, which they usually already are).
  3. Never x-vga=1 on a headless compute VM — it removes your only console.
  4. Snapshot before experimenting, and never hard-stop a VM mid-boot.
  5. screendump from the QEMU monitor is your eyes when the guest is otherwise dark.

No comments to display

Back to top