Last time, five Xid 79 occurrences in four days had killed every software and BIOS mitigation I could find, and the motherboard swap I’d been counting on had quietly downgraded from “the fix” to “a diagnostic step.” I’d narrowed it to a likely silicon or GSP-firmware defect, matched by two independent public reports of the identical fault on the same GPU family, and started the RMA paperwork.

Since then: I ran the obvious next test, got a clean result, then figured out why it was clean — which turned out to be the more useful finding — and a Reddit comment pointed at something I hadn’t considered at all. Here’s where it stands.

The obvious next test: does it fail under raw stress?

Every Xid 79 so far had happened under vLLM specifically. Before finalizing the RMA, I wanted to rule out the possibility that vLLM’s CUDA graph replay was itself the trigger, independent of anything hardware-level — so I built gpu-burn, the standard application-agnostic CUDA stress tool, targeting this card’s actual compute capability (COMPUTE=12.0 — the Makefile defaults to a much older architecture and will silently under-target Blackwell if you don’t override it).

Stopped vllm-coder.service to free VRAM, ran a 30-second smoke test clean, then let it run for 4 hours at full tilt — 100% SM utilization, power pinned at the card’s 600W cap, with a live kernel-log watcher for Xid events running the whole time.

Tested 1 GPUs:
	GPU 0: OK

Zero errors, zero Xid events, for 4 straight hours. Temps peaked at 89°C with no thermal throttling. On its face, that looks like good news for the card.

Why the clean result didn’t actually mean much

Before writing this up as “ruled out,” I went back through the telemetry the test itself had recorded, and it undercut the result immediately.

This card’s rated max SM clock is 3090 MHz. The moment the burn test’s sustained 100% load kicked in, the 600W power limiter engaged almost instantly — within one 5-second sample — and held the clock in a 1867–2280 MHz band for the entire 4 hours. Never once within 800 MHz of the rated ceiling.

That makes sense once you think about it: sustained heavy compute load pushes power draw to the cap fast, and once you’re power-limited, the boost algorithm has no room left to explore the top of its clock range. A stress test that maxes out utilization is, paradoxically, one of the least likely ways to see peak transient clock behavior — it forces the card into a conservative, power-throttled state almost immediately.

Every real Xid 79 on this box had happened under a completely different profile: an idle period followed by a sudden large-prefill burst, or steady token-by-token decode with frequent micro-idle gaps between kernel launches. In both cases the GPU is running well under its power cap, which is exactly when the boost algorithm is free to ramp toward its true ceiling on a sudden transition. My clean 4-hour result didn’t clear the card — it just proved the test never went looking where the actual failures live.

A Reddit comment that reframed the whole thing

While digging into that gap, I came across a comment on r/LocalLLaMA describing something structurally similar:

I’ve been working on this same issue for a month or so now. I found the solution. It wasn’t RAM or CPU or PSU or Power limits or anything else people are suggesting. It’s the boost clock. When you hit the GPU with a load, it spikes to a massive core clock speed and causes instability. I tried literally everything pulling my hair out. After this change I have pumped the card so hard my room is hot and no issues. The command below will set the max core clock to 2400mhz. You can experiment with different numbers, I started with 2100 and worked my way up without crashes. I experience the exact same issue with AMD 5950x Precision Boost Overdrive, that took me 2 years to work out >.> Hope this helps someone…

nvidia-smi -lgc 180,2400

Not the same GPU, not the same failure mode necessarily — but the mechanism (a transient boost-clock excursion on a load spike, invisible to thermal/power/RAM diagnostics) lines up with the exact gap the burn test had just exposed. And the fix is close to free to test: no BIOS trip, no reboot, one command, fully reversible.

Testing it properly instead of trusting the command

I’ve been burned twice already in this investigation by trusting a mitigation command’s own “success” output instead of checking the actual system state afterward — an empty override file, a persistence-mode flag reverted by a stock daemon flag I didn’t know was there. So instead of running -lgc and calling it done, I measured the actual clock behavior before and after, sampling clocks.sm at 100ms resolution across an identical idle-to-6-second-burst transition:

State Peak SM clock (idle → burst)
Unlocked (default boost) 2797 MHz
Locked, nvidia-smi -lgc 180,2400 2392 MHz

That’s a real, measured ~400 MHz reduction in peak transient clock, confirmed by direct observation — not by nvidia-smi -q -d CLOCK’s “Applications Clocks” field, which reported “deprecated” for this driver/architecture and would have told me nothing useful on its own.

Applied it live: sudo nvidia-smi -lgc 180,2400. Like persistence mode earlier in this saga, this setting doesn’t survive a reboot on its own, so I staged a systemd oneshot unit to reapply it automatically:

# /etc/systemd/system/nvidia-clock-lock.service
[Unit]
Description=Lock NVIDIA GPU clocks to mitigate Xid 79 boost-clock instability
After=nvidia-persistenced.service
Requires=nvidia-persistenced.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -lgc 180,2400
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enabled and confirmed active immediately. Reboot-persistence itself is not yet verified across an actual reboot — I’m flagging that explicitly, on the principle that “confirmed working” has bitten me before in this exact log.

Monitoring without building anything new

One nice surprise: I didn’t need to stand up new telemetry for this. This box already runs a Prometheus + nvidia_gpu_exporter stack from an earlier monitoring setup, and it’s already scraping nvidia_smi_clocks_current_sm_clock_hz every 15 seconds with 15-day retention. If Xid 79 recurs, I’ll finally have the exact SM clock value at the moment of the crash, cross-referenced against the kernel-log timestamp — a piece of evidence none of the five prior occurrences had, since nobody was watching clock telemetry in real time when they happened.

Where this leaves the RMA

To be clear about what this does and doesn’t change: the RMA case I wrote up doesn’t get weaker. Every previously-known mitigation — BIOS ASPM disable, dynamic power management off, persistence mode on — was verified active in combination and the fault still recurred. This clock lock is a new, different lever, untested by time. If it turns out to hold where the others didn’t, that’s a genuinely useful outcome regardless of what it means for the RMA: either it’s a real fix and the card doesn’t need replacing after all, or it’s one more thing that gets ruled out before the card ships back.

I’m running real inference traffic against it now — the actual workload that’s triggered all five crashes so far, not a synthetic stand-in. Next post: whether the clock cap holds, or whether Xid 79 finds a way through it too.