The local LLM server had been running clean for over two months — no crashes, no restarts, just vllm-coder.service quietly serving Claude Code and opencode off the RTX PRO 6000. On 2026-08-17 it died mid-request, stayed dead for over two hours, and nobody noticed until I went to use it that evening. This is the timeline, the diagnosis, and the leading theory for a root cause I won’t be able to confirm until next weekend.

What happened

At 19:21:52, vllm-coder.service was serving a long-context request — a ~76,433-token prompt with a 98% prefix cache hit, so the engine had almost the entire context already resident and just needed to process the new tail. Mid-request, the engine hit a fatal error and exited. Cleanly. Exit code 0.

By the time I opened a terminal that evening at 21:43, requests had been failing silently for over two hours. sudo dmesg -T was the first move.

The red herring

The obvious first stop is the service’s own log:

$ journalctl -u vllm-coder.service
...
torch.AcceleratorError: CUDA error: unspecified launch failure
Search for `cudaErrorLaunchFailure` in
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html

unspecified launch failure is CUDA’s catch-all for “something went wrong on the device and I don’t know what.” It’s consistent with a dozen different root causes — a bad kernel launch, memory corruption, a driver bug, an actual hardware fault — and by itself it points at vLLM or PyTorch, not the hardware underneath them. I spent a few minutes down this path assuming it was a software regression before remembering that CUDA errors this generic are usually downstream symptoms, not causes. The real event happened one layer lower, and vLLM’s log doesn’t see that layer.

The real signal

Same timestamp, kernel log instead of the service log:

$ journalctl -k -b -1 --since "2026-08-17 19:21:00"
...
NVRM: GPU at PCI:0000:01:00: GPU-6d1274a6-54f2-2626-1167-53a6d375e766
NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
NVRM: krcRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.

Xid 79 is NVIDIA’s code for the driver losing contact with the physical GPU at the PCIe level. Not an OOM, not a CUDA logic error, not a vLLM bug — the driver sent a command to a device that, from its perspective, was no longer there. The kernel’s UVM module logged the aftermath trying to reclaim GPU memory it could no longer reach:

BUG uvm_gpu_chunk_5: Objects remaining on __kmem_cache_shutdown()

That’s the driver tearing down state for a GPU that vanished out from under it. The torch.AcceleratorError in the vLLM log was just CUDA reporting the same event from the application side — a symptom two layers removed from the cause.

Why auto-recovery didn’t save it

vllm-coder.service has been running with Restart=on-failure since the initial build:

$ systemctl cat vllm-coder.service
...
[Service]
Restart=on-failure
RestartSec=5

Restart=on-failure only fires when the process exits with a non-zero code or dies to a signal. vLLM’s engine caught the fatal CUDA error, logged it, and exited cleanly — exit code 0. As far as systemd was concerned, the service asked to stop and stopped:

$ journalctl -u vllm-coder.service
...
systemd[1]: vllm-coder.service: Deactivated successfully.

“Deactivated successfully” is the most misleading line in this entire incident. Nothing about it triggers a restart, and nothing about it looked wrong from the outside — no crash loop, no error state, just a service that quietly stopped taking requests two seconds after the GPU disappeared.

It wouldn’t have mattered anyway. Xid 79 isn’t a software-recoverable state — the PCIe link itself needs retraining, and a process restart doesn’t touch the link. The GPU was gone until the physical bus reset, which in practice meant a full reboot:

$ sudo reboot

The box came back at 04:07:03 the next morning (I’d gone to bed rather than babysit a reboot at 22:30), and vllm-coder.service came up clean — model loaded, CUDA graphs captured, serving normally within about a minute:

04:07:15  vllm-coder.service: loading Qwen3.8-27B (51.1 GiB)
04:07:35  model load complete (19.6s)
04:08:06  CUDA graph capture complete, serving

Scope check: has this happened before?

Before speculating on cause, I wanted to know if this was a one-off or a pattern I’d somehow missed:

$ journalctl -k --since "30 days ago" | grep -i xid
NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.

One line. One occurrence, in a month of continuous operation. That rules out a chronically flaky link — whatever’s going on isn’t happening every day, or even every week. It also means I don’t have enough data points to actually diagnose it; everything past this point is informed speculation, not a confirmed root cause.

I also checked whether the link was still healthy after the reboot, to rule out an ongoing physical problem:

$ sudo lspci -vvv -s 01:00.0 | grep -A3 LnkCap
LnkCap: Port #0, Speed 32GT/s, Width x16
LnkSta: Speed 32GT/s (ok), Width x16 (ok)

$ sudo lspci -vvv -s 01:00.0 | grep -i aer -A5
DevSta: CorrErr- UncorrErr- FatalErr- UnsupReq-
CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErrMsg-

Full Gen5 x16, negotiated correctly, no AER error counters incremented. Whatever happened at 19:21:52, it didn’t leave the link in a degraded state — which argues against a persistent signal-integrity problem and toward something transient.

Hardware speculation, ranked

Here’s the platform this GPU lives in:

Component Detail
Motherboard MSI PRO B650M-A WIFI (MS-7D77), BIOS 1.N0 — budget mATX consumer board
CPU AMD Ryzen 7 7800X3D (AM5, direct x16 lanes to the GPU slot)
GPU RTX PRO 6000 Blackwell Workstation Edition, 600W TDP, drawing ~502W at time of inspection
Driver 610.43.02, open-source kernel module

The mismatch is obvious once you say it out loud: a 600W triple-slot workstation card, on a board that was never validated for anything like it. B650M-A WIFI is a fine board for a gaming build. It has never been anyone’s reference platform for a data-center-class GPU pulling half a kilowatt.

The one concrete lead in the logs:

$ sudo lspci -vvv -s 01:00.0 | grep LnkCtl
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

$ cat /sys/module/pcie_aspm/parameters/policy
[default] performance powersave powersupersave

ASPM L1 is a PCIe power-saving state that drops the link’s power draw during idle gaps and retrains it when traffic resumes. It’s a well-documented trigger for exactly this Xid code on consumer motherboards paired with high-power NVIDIA GPUs — the retrain doesn’t always complete cleanly under a sudden load spike, and the driver reports the failure as the GPU vanishing from the bus. The timing fits: the request that killed the service was a 76k-token prefill — a large, sudden burst — arriving after a run of small, low-throughput requests. Idle, then a spike. That’s the textbook ASPM-wake failure pattern.

Ranked by how well the evidence supports each:

  1. PCIe ASPM L1 wake failure. Best fit — matches the known failure signature for this Xid code on this class of hardware, and the load pattern at the moment of the crash (idle-to-burst) is exactly the trigger condition reported elsewhere for this bug.
  2. Consumer/budget board in general. B650M-A WIFI isn’t validated for a card in this power class — thinner signal-integrity margins, no PCIe retimer, compared to an actual workstation board. Could be contributing even if ASPM isn’t the specific trigger.
  3. Physical — seating, cable, card sag. A 600W triple-slot card has real mass and real current draw. Nothing in the logs points here or rules it out; it would need a physical inspection I haven’t done yet.
  4. GPU hardware fault. Least likely. Clean ECC, no thermal symptoms (72°C at inspection), link renegotiated to full Gen5 x16 with no AER errors, and a single occurrence in 30 days of otherwise continuous load. A genuinely faulty card usually doesn’t wait a month and recur once.

To be clear about the confidence level here: one data point in thirty days is not a diagnosis. It’s a ranked list of plausible explanations, ordered by fit, with the top one because it matches a known failure mode down to the specific timing pattern — not because I’ve reproduced it.

Resolution

Immediate fix: sudo reboot. The link retrained, the GPU came back, vllm-coder.service has been serving normally since.

Permanent fix: pending. I’d already ordered a replacement motherboard before this incident — for unrelated reasons, as part of a planned upgrade — and the swap is scheduled for next weekend (~2026-08-24). That means the platform is about to change out from under this theory regardless of what I do now, which changes the calculus on what’s worth testing.

The obvious next test if I weren’t about to swap the board would be pcie_aspm=off as a kernel boot parameter, or disabling ASPM for the GPU specifically in BIOS — a direct test of the leading theory. I’m holding off. Applying it now and then replacing the motherboard a week later would leave me with two changes and no way to tell which one (if either) actually mattered. Better to let the new board be a clean variable.

Open question heading into the swap

If Xid 79 doesn’t recur after the motherboard swap, that’s reasonably strong circumstantial support for “budget consumer board, ASPM or otherwise, wasn’t built for this card” — even without isolating the exact mechanism. If it does recur on new hardware, the ASPM theory gets a lot less interesting and the GPU itself (or something further upstream, like the PSU under sustained high draw) moves up the list.

Either way, I’m not making any config changes before the swap — I want the before/after comparison to mean something. One belt-and-suspenders fix I am making regardless of root cause: vllm-coder.service is getting Restart=always instead of Restart=on-failure. It won’t do anything for Xid 79 itself — no restart policy fixes a dead PCIe link — but it closes the “clean exit doesn’t count as failure” gap for any future fatal-but-exits-0 engine error that would be recoverable with a restart.

Next post, assuming the theory is testable: whether the new board runs clean, or whether Xid 79 follows the GPU to a platform that was actually built for it.