Last time, the RTX PRO 6000 dropped off the PCIe bus once in thirty days, and I had a tidy leading theory — PCIe ASPM L1 failing to wake cleanly under a sudden load spike — with a scheduled motherboard swap to test it. I said I’d hold off on any software mitigation so the swap would be a clean before/after comparison.

That plan lasted about a day. This is the story of how one Xid 79 became five, how the leading theory got tested and killed, how two of my own “confirmed working” fixes turned out to be silently broken, and how the motherboard swap I was counting on as the fix quietly turned into a diagnostic step instead.

Recurrence #2: the one I almost missed

On 2026-08-20 I went back through a full month of kernel logs to double check the “one occurrence in thirty days” claim from the first post, expecting to confirm it. Instead:

Aug 18 06:01:34 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 79, pid=2919, name=VLLM::EngineCor, GPU has fallen off the bus.
Aug 18 06:01:34 ai-box kernel: NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
Aug 18 06:01:34 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (OS Reboot)

A second Xid 79 had happened the very next morning after the first, and nobody had logged it. Same downstream signature — torch.AcceleratorError: CUDA error: unspecified launch failure, clean exit code 0, Restart=on-failure not firing — but a bigger prefill this time (183,138 tokens vs. ~76,433) and a much worse outcome: the box sat dead for eight hours before anyone noticed.

That single missed entry retired the “one occurrence, not a pattern (yet)” line from the first post. This was now 2-for-2, and worse, it meant my monitoring — “check dmesg when something feels off” — wasn’t actually catching these.

I also went looking for anything that would implicate the motherboard’s power delivery specifically: AER errors, correctable/uncorrectable PCIe events, voltage or brownout messages, thermal or ECC events, across 60 days of logs. Nothing. The only kernel-side signal, both times, was the Xid 79/154 pair itself. Whatever this was, it wasn’t leaving a paper trail anywhere except the one line that actually mattered.

Actually testing the ASPM theory

The first post held off on any mitigation to keep the motherboard swap a clean variable. Waiting for a scheduled swap a few days out while the box was silently going down for hours felt like the wrong tradeoff, so I applied the leading mitigation immediately: pcie_aspm=off as a kernel boot parameter.

Verification is where this got interesting:

$ cat /proc/cmdline | grep -o pcie_aspm=off
pcie_aspm=off

$ sudo lspci -vvv -s 01:00.0 | grep -i "LnkCtl\|ASPM"
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

The kernel parameter was active. ASPM was still enabled on the link. Turns out pcie_aspm=off only tells Linux to stop managing ASPM transitions itself — it doesn’t rewrite the link control register the BIOS programs at POST. Whatever the firmware set at boot is what stays in effect once the kernel steps back. On this board, that’s ASPM L1 on, unconditionally, and the kernel param never touches it.

So the mitigation I’d applied hadn’t actually been tested at all — a distinction that mattered a lot in hindsight, because a Xid 79 landed about 32 minutes after the reboot that was supposed to apply it. It neither confirmed nor refuted anything; the real fix had to happen in BIOS.

I wrote up a small decision tree for finding the right BIOS setting on an MSI PRO B650M-A WIFI — the generic AM5 BIOS reference doesn’t itemize a setting literally called “ASPM,” so the plan was: try AMD CBS → PSPP Policy first (the AGESA knob for CPU-attached PCIe root ports, relevant since the 7800X3D gives the GPU direct lanes), fall back to a PCIe sub-system submenu, fall back again to the BIOS’s built-in setting search.

PSPP Policy → Disabled worked on the first try:

$ sudo lspci -vvv -s 01:00.0 | grep -i "LnkCtl\|ASPM\|LnkSta\|LnkCap"
LnkCap:	Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
LnkSta:	Speed 32GT/s, Width x16

ASPM Disabled — the first time the setting had actually taken effect at the hardware level, full Gen5 x16, not downgraded. This was, finally, a real test of the theory.

Recurrence #3: theory ruled out

It didn’t survive the same boot:

Aug 20 17:20:04 ai-box kernel: NVRM: GPU at PCI:0000:01:00: GPU-6d1274a6-54f2-2626-1167-53a6d375e766
Aug 20 17:20:04 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
Aug 20 17:20:04 ai-box kernel: NVRM: krcRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
Aug 20 17:20:04 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (OS Reboot)
Aug 20 17:20:06 ai-box kernel: NVRM: nvGpuOpsReportFatalError: uvm encountered global fatal error 0x60, requiring os reboot to recover.

ASPM was confirmed disabled going in, confirmed disabled again after. It happened anyway. And the traffic pattern broke the theory a second way: the first two crashes both hit during an idle-to-large-prefill-burst transition, the textbook ASPM-wake failure scenario. This one didn’t — over a minute of steady low-throughput decoding beforehand (~27 tok/s generation, flat KV cache usage, no waiting requests), no preceding idle gap, no prefill spike. The load pattern the original theory was partly built on simply wasn’t there.

One more detail: Xid 154’s “GPU recovery action changed… to 0x2 (OS Reboot)” reads like the driver is about to handle it. It isn’t. No shutdown or reboot sequence appeared anywhere in the logs afterward — the GPU stayed dead until I did a hard power cycle. The “recovery action” is advisory, logged for a human to read, not something the driver executes on its own.

With ASPM ruled out, the ranked list flipped: budget-board signal integrity moved to the top by elimination, physical/power stayed exactly where it was (plausible, no evidence either way), and GPU hardware fault stayed last (still no ECC or thermal symptoms across three incidents). Three-for-three, no software mitigation left to try from the original theory, and the 2026-08-24 motherboard swap was still five days out.

Is this RAM, driver, or vLLM?

Before assuming “budget board” by default, I checked the other obvious suspects directly instead of ranking them by vibes.

RAM: ruled out. No EDAC, no MCE events, no OOM kills, no page-allocation failures anywhere in 90 days of kernel logs. A memory fault has a distinct signature — this isn’t it.

Then I went looking for anyone else who’d seen this exact signature, and found two reports that changed the whole shape of the investigation:

  • dbirks/home-k8s#46 — an RTX PRO 6000 Blackwell hitting Xid 79 + Xid 154 under vLLM on driver 610.43.02, the exact version this box runs, reproduced across driver versions 570 through 610 and across motherboards spanning WRX80, WRX90, EPYC, X670E, AM4, and Z490 — including proper workstation-class boards, not just budget consumer ones. Thermal was ruled out there too, with crashes recorded at die temps as low as 28°C. Their working theory: a GSP firmware heartbeat failure after a GC6 idle power-state exit, aggravated by vLLM’s CUDA graph replay under sustained inference.
  • NVIDIA developer forum thread on an RTX 5080 — same Xid 79/154 pair, same driver range, the card moved between two completely different systems (a Z390 board, then an X870/Ryzen board) and the fault followed the card, not the board. PSU rail was measured through multiple faults with no OCP trip — power delivery ruled out directly, not just by absence of log evidence. ASPM was ruled out there too, at the register level, independently matching what I’d just found on my own box.

That second report in particular is hard to argue with: same GPU family, same exact fault pair, survives a full motherboard change, PSU instrumented and clean. If the identical signature reproduces on WRX80/WRX90/EPYC workstation boards with the same card, my B650M-A WIFI theory doesn’t hold up as the explanation — at most it’s a contributing factor, not the root cause.

vLLM: a trigger, not the cause. CUDA graph replay under sustained inference is called out as an aggravating factor for the GSP heartbeat bug in both reports, but the same Xid 79 pattern is also reported under other inference frameworks on this GPU family. vLLM is where the crash surfaces — it’s issuing the CUDA calls when the GPU disappears — not where the fault originates. Worth noting: the third crash’s steady-decode pattern (rather than idle→burst) actually fits a GC6 power-cycling trigger better than my original ASPM-wake hypothesis did — token-by-token decode has frequent micro-idle gaps between kernel launches, which is exactly the kind of repeated power-state cycling the community reports point at.

The config audit, and three mitigations

Since GSP/power-management was now the leading theory, I audited this box’s actual settings against what the community reports implicated:

Setting Found on this box Implicated value
persistence_mode Disabled Non-persistent mode lets the GPU cycle power states between requests
NVreg_DynamicPowerManagement 3 (full dynamic/GC6 cycling) The exact mode named in the GSP heartbeat bug
power.limit 600W (uncapped) Community mitigation: cap to 450W, ~5-15% perf cost

Two of three risk factors were live. I staged all three fixes: nvidia-smi -pm 1 for persistence mode, a modprobe override (NVreg_DynamicPowerManagement=0x00) baked into the initramfs to disable dynamic power management, and held the power cap in reserve as a second-line option since it costs throughput.

Two bugs in my own “confirmed working” mitigations

This is the part I’d rather not have to write up, but it’s the most useful part for anyone else chasing something like this: I confirmed the mitigations were working, twice, and was wrong both times.

The first pass looked clean:

$ sudo nvidia-smi -q | grep "Persistence Mode"
Persistence Mode: Enabled

What I didn’t account for: nvidia-smi -pm 1 is a live, in-memory command. nvidia-persistenced.service — the daemon actually responsible for persistence mode across reboots — was already running with --no-persistence-mode baked into its stock ExecStart. I wrote a systemd drop-in to strip that flag, restarted the service, and confirmed persistence mode was enabled. Correct outcome, wrong reason: I’d checked the symptom right after the live command, not after the drop-in actually took effect.

Recurrence #4: the empty file

Aug 20 22:36:01 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
Aug 20 22:36:01 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (OS Reboot)
Aug 20 22:36:01 ai-box kernel: NVRM: nvGpuOpsReportFatalError: uvm encountered global fatal error 0x60, requiring os reboot to recover.

Same signature, steady decode again (not idle→burst — a third data point against the original trigger pattern), and a hard power cycle again, this time with 17 minutes before a client even noticed (a downstream litellm proxy started throwing httpx.ConnectError at 22:53).

Checking all three mitigations against the live system afterward, rather than trusting the log entry that said they were fine:

Mitigation Status found
BIOS ASPM disable Confirmed still disabled
NVreg_DynamicPowerManagement=0x00 Confirmed still 0
Persistence mode override /etc/systemd/system/nvidia-persistenced.service.d/override.conf — 0 bytes

The tee heredoc I’d used to write that override file had silently written nothing. Only two of the three mitigations had actually been active during the fourth crash. Fixed it properly this time — wrote the file, checked its size before touching the service (97 bytes, not 0), reloaded, restarted, and verified two different ways: systemctl show nvidia-persistenced.service -p ExecStart to confirm the effective command line, and nvidia-smi -q to confirm the resulting state. Not because the first check was the wrong command, but because checking the same thing the same way twice doesn’t catch a bug in the step you didn’t check.

Recurrence #5: 22 minutes into a clean boot

I re-verified all three mitigations independently after a reboot — ASPM disabled, dynamic power management off, persistence mode on, checked at the source each time, and confirmed surviving the reboot itself for the first time. Monitoring resumed for real.

22 minutes later:

Aug 20 23:35:31 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
Aug 20 23:35:31 ai-box kernel: NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (OS Reboot)

This time the log had more detail than any prior occurrence — a run of GSP RPC failures during the teardown, repeated about six times as the kernel tried to free GPU objects post-fault:

NVRM: GPU0 kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior invalidate (status=0x0000000f) ...
NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 35548!
NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0x5c000009; ...
NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573

This is exactly the “below the driver boundary” failure mode described in both community reports: the GPU’s onboard GSP microcontroller stopped answering the driver’s RPC channel entirely. Not new evidence for or against the theory — just the clearest look yet at what’s actually failing.

This was the first occurrence with all three mitigations independently verified active for the entire pre-crash window, across a reboot. It crashed anyway, 22 minutes in.

Where this leaves things

Five occurrences, five different subsets of active mitigations, one recurrence with every known software and BIOS fix genuinely in place at once. That was the bar I’d set myself in the exec plan for when to stop iterating on software — and it’s been cleared.

Ranked candidates now:

  1. Silicon/GSP-firmware defect in this specific card (RMA candidate). Leading. The one clean test — all three mitigations verified active, independently, across a reboot — still crashed within half an hour.
  2. GSP firmware / driver power-management bug affecting this GPU family broadly. Can’t be fully separated from #1 by logs alone. The community reports show the identical signature across many cards and boards; that’s consistent with either “this driver line is broadly bad” or “GSP heartbeat bugs happen to look identical to a silicon fault.” Only an RMA swap or a driver change would tell them apart.
  3. Consumer/budget board (signal integrity). Downgraded twice now — first by ASPM’s own hardware-level test failing, then by the community reports reproducing on workstation-class boards.
  4. Physical — seating, cable, card sag. Unchanged the whole way through. Still plausible, still zero supporting or contradicting evidence, worth a look during the swap regardless.

The motherboard I ordered for entirely unrelated reasons, and which the first post treated as the likely fix, is arriving this weekend anyway. It’s going in — but as a diagnostic step now, not the expected resolution. If Xid 79 survives the swap too, that’s about as strong a case for RMA as logs alone can build, and I’ll open that conversation with the card vendor citing both community reports directly.

The gap that’s mattered the most through all five occurrences isn’t the root cause at all: vllm-coder.service still has no automated recovery from a clean-exit crash, and three of the five outages went completely unnoticed for hours because nothing pages anyone. Whatever the swap decides about the hardware, that’s getting fixed regardless — Restart=always closes part of it, and actual external alerting on the service being down is next.

Next post: whether the new board changes anything, or whether Xid 79 follows the card again — and if it does, how the RMA conversation goes.