I switched my homelab coding model from BF16 to NVFP4 yesterday. Across 1,258 graded coding and instruction-following prompts, I couldn’t find a statistically significant quality difference. Single-request decode went from 59 to 133 tok/s, the box now serves about twice the total tokens, and the KV cache more than doubled.

The MTP post ended with me wanting to try a quantized version. The catch with quantization is that “it feels fine” isn’t a test. So before switching anything I built a proper benchmark harness with Claude Code, ran it against both models, and published it: github.com/techeretic/llm-bench.

What I tested

Same box as before: ai-box, one RTX PRO 6000 Blackwell (96 GB), vLLM 0.31, Qwen3.8-27B with MTP speculative decoding (2 draft tokens).

Variant Size on disk Engine Tested for
BF16 (production) 52 GB vLLM speed + quality
Unsloth NVFP4 22 GB vLLM speed + quality
Unsloth UD-Q8_K_XL GGUF   llama.cpp speed
Unsloth UD-Q6_K_XL GGUF   llama.cpp speed
Unsloth UD-Q4_K_XL GGUF   llama.cpp speed

I wanted more on that list. NVIDIA’s own NVFP4, RedHat’s NVFP4, Qwen’s official FP8 and a DFlash2 draft model were all queued up too. Four of my six Hugging Face downloads died on SSL and CDN timeouts, and I didn’t retry them. That’s a follow-up.

flowchart LR
  D["6 downloads"] -->|"4 failed:<br/>SSL / CDN timeouts"| X["dropped for now"]
  D --> U["Unsloth NVFP4<br/>Unsloth GGUF ×3"]
  U --> S["long-context speed<br/>5 variants"]
  S -->|"GGUF prefill<br/>2-3x slower"| Q["full quality suite<br/>BF16 vs NVFP4"]
  Q --> P["production:<br/>NVFP4"]

All the vLLM variants ran with exactly the same server flags (FP8 KV cache, prefix caching, --max-num-seqs 480, MTP n=2). Only the weights changed. Otherwise you’re comparing configs, not quants.

Speed first

Coding agents send big prompts. So the speed test builds prompts from real Python source (vLLM’s own package, as it happens) at 32K, 64K and 125K tokens, puts a random nonce at the start so prefix caching can’t help, and generates 1,500 tokens. Three runs per point.

Line chart of single-request decode speed vs prompt length for five Qwen3.8-27B variants with MTP: at 125K tokens NVFP4 on vLLM does 111 tok/s, Q4_K_XL on llama.cpp 107, Q6_K_XL 90, Q8_K_XL 78 and BF16 on vLLM 61

BF16 holds flat at about 61 tok/s all the way to 125K. Before MTP it did 28 and 27 at 32K and 125K, so the MTP gain holds up at long context too. NVFP4 is roughly twice as fast at every length. The Q4 GGUF is the fastest thing on the chart for short prompts (154 tok/s), but that’s not where my agents live.

The GGUFs lose on the other half of the request: reading the prompt.

Grouped bar chart of time to first token and line chart of prefill rate at 32K, 64K and 125K tokens: at 125K NVFP4 takes 23 s, BF16 35 s and the three GGUFs 54 to 58 s; NVFP4 prefills at 9.7K tok/s at 32K, BF16 at 5.2K, the GGUFs at about 3K

at 125K tokens BF16 NVFP4 Q8_K_XL Q6_K_XL Q4_K_XL
Time to first token 34.7 s 23.2 s 54.2 s 58.1 s 54.3 s
Prefill 3.6K tok/s 5.4K tok/s 2.3K tok/s 2.1K tok/s 2.3K tok/s
Decode 61 tok/s 111 tok/s 78.5 tok/s 90 tok/s 107 tok/s
MTP acceptance length 2.68 2.56 2.54 2.59 2.57

An agent with a 125K context waits close to a minute on llama.cpp before it sees a single token, and it pays that again on every turn that changes the prompt. NVFP4 gets there in 23 seconds. On decode only Q4 beats it, by a few tok/s, and only below 125K. That settled it. The GGUFs didn’t go on to the quality suite.

One nice detail in the last row: MTP acceptance barely moves after quantization. The draft head still guesses about 2.6 tokens per step on NVFP4.

Then quality

This is the part I actually cared about. A quick 12-task check I tried first was too easy to tell the variants apart. So the harness grew five real suites, and it grades everything by running code or by rules, never by eyeballing.

Suite Size Mode Graded by
HumanEval+ / MBPP+ 164 + 378 thinking off, greedy EvalPlus hidden tests
LiveCodeBench v6 175 (43 easy, 52 medium, 80 hard) thinking on, 32K max LCB hidden tests
IFEval 541 thinking off, greedy IFEval rule checkers
Spec writing 6 prompts × 3 samples thinking on, 24K max rule checks + blind pairwise judge
Concurrency 1 / 4 / 16 / 64 / 128 requests thinking off timing

Both models get the identical prompts, and the comparison is paired: for every item, did one model get it right where the other didn’t? That’s what the McNemar test uses, and it’s a lot more sensitive than comparing two percentages.

Dot plot comparing BF16 and NVFP4 pass rates with 95% confidence intervals: HumanEval+ plus 92.1 vs 93.3, MBPP+ plus 77.8 vs 77.2, LiveCodeBench v6 61.7 vs 60.6, IFEval strict 83.2 vs 82.8; no McNemar p-value below 0.18

Every gap is inside the noise. The smallest p-value is 0.18 (LiveCodeBench medium, where BF16 solved 7 problems NVFP4 didn’t and NVFP4 solved 2 BF16 didn’t). Hard went the other way, 3 vs 6. On the 1,258 graded items across HumanEval+, MBPP+, LiveCodeBench and IFEval (strict), the two models disagree on 96, about 8%, and those split 50 for BF16 and 46 for NVFP4.

Two honest caveats. 58 BF16 and 56 NVFP4 LiveCodeBench answers hit the 32K thinking budget before finishing, so LCB here measures “within 32K tokens”. And 175 problems can only rule out big gaps. A 2 to 3 point difference on hard problems wouldn’t show up at this size.

Writing specs

I use this model to write technical specs as much as code, so I wanted that tested too. Six prompts (a config-diff CLI, a job queue, a Postgres failover migration, an LLM rate limiter, resumable uploads, a webhook protocol), each with strict rules: exact section order, a required table, a JSON example, RFC 2119 keywords, a word limit. Three samples per model per prompt.

Both models passed all 117 rule checks. For quality beyond the rules, the 18 pairs were shuffled into anonymous “Spec X / Spec Y” files and judged blind. The judge was Claude, not me, and it wrote all its verdicts down before opening the key.

Grid of 18 blind pairwise spec judgements across six spec topics and three samples: BF16 judged better in 8, NVFP4 in 6, tie in 4; mean score BF16 7.39, NVFP4 7.47

8 to 6 with 4 ties, and NVFP4 actually has the slightly higher mean score (7.47 vs 7.39). No topic went one way three times. NVFP4’s specs were a bit shorter (median 945 words vs 1,019).

What NVFP4 buys

Three line charts over 1 to 128 parallel requests: total output tokens per second peaks at 1,028 for NVFP4 vs 493 for BF16 at 64 requests; per-request decode 133 vs 59 at one request; median time to first token at 64 requests 21 s vs 41 s

  BF16 NVFP4  
One request, decode 59 tok/s 133 tok/s 2.3x
Peak total throughput (64 requests) 493 tok/s 1,028 tok/s 2.1x
Median TTFT at 64 requests 41 s 21 s 2x
KV cache 748K tokens 1.6M tokens 2.1x
Full 262K-context requests that fit 2.85 6.09  

Both models plateau at 64 requests. Past that you just queue. At 128 requests the median BF16 request waits 109 seconds for its first token.

The benchmark itself is a decent real-world measurement, since it’s the same mix of short and long thinking requests at 48 to 64 in flight:

Horizontal bar chart of wall-clock time per benchmark suite: LiveCodeBench 65.0 min on BF16 vs 29.1 on NVFP4 (2.2x), spec writing 5.9 vs 3.5, IFEval 2.9 vs 1.6, MBPP+ 1.4 vs 0.8, HumanEval+ 0.8 vs 0.5

How the harness works

flowchart LR
  subgraph data["inputs"]
    H["HumanEval+ / MBPP+"]
    L["LiveCodeBench v6"]
    I["IFEval"]
    SP["specs.py prompts"]
    C["real Python source<br/>(speed prompts)"]
  end
  data --> G["gen.py<br/>async, resumable"]
  G -->|"OpenAI API<br/>LLM_HOST / LLM_PORT"| M["model under test"]
  G --> O["out/label/*.jsonl"]
  O --> SC["score.py<br/>runs the code / rules"]
  SC --> B["blind.py<br/>X/Y spec pairs"]
  SC --> CP["compare.py<br/>Wilson CI, McNemar"]
  B --> CP
  CP --> CH["charts"]

A few rules I’d keep for any model comparison:

  • Measure from the client. TTFT and decode come from the streaming client. Server counters are only used for things the client can’t see, like MTP acceptance.
  • Same flags for every variant, only the weights change.
  • Grade by execution or rules. EvalPlus and LiveCodeBench run the generated code against hidden tests. IFEval and the spec checks are rule based.
  • Pair everything and report the disagreements, not just two percentages.
  • Anything subjective is judged blind.

The production model runs on the same box, so testing a candidate means downtime:

sequenceDiagram
  participant P as vllm-coder (prod)
  participant C as candidate
  participant B as bench
  B->>P: BF16 suite against live prod (no downtime)
  B->>P: systemctl stop
  B->>C: serve-nvfp4.sh, same flags
  Note over C: first start JIT-compiles FlashInfer FP4 kernels
  B->>C: run-all.sh nvfp4 (36 min)
  B->>C: kill
  B->>P: systemctl start, check :8000 and :4000

Things that bit me

  • NVFP4’s first start compiles FP4 kernels, and that compile wants a lot of RAM. ai-box has 60 GB, and the compile got OOM-killed while llama.cpp was building in parallel. Compile alone, with MAX_JOBS=8.
  • A serve script without the execute bit cost about 10 extra minutes of production downtime. Permission denied in a log nobody was watching. Now everything is launched with bash script.sh.
  • pkill -f over ssh kills itself. The pattern matches the ssh shell’s own command line. Use a bracket pattern like "[a]pi_server".
  • LiveCodeBench ships its hidden tests as pickles. Unpickling a downloaded file can run arbitrary code, so the harness uses an unpickler that refuses every class (the payload is only a JSON string anyway).
  • EvalPlus wants every problem in the file. You can’t grade a 2-problem smoke test, so smoke tests only check that generation works.
  • The model-generated code actually runs on the box during grading. EvalPlus and LCB sandbox it with timeouts and separate processes, but run the scorers from /tmp, not from inside the harness folder.

The first version lived in my homelab repo with hostnames, IPs and the GPU baked into the scripts. For the public repo I stripped all of that. The endpoint is now just LLM_HOST, LLM_PORT and LLM_MODEL, so it works against anything OpenAI-compatible:

LLM_HOST=127.0.0.1 LLM_PORT=8000 LLM_MODEL=my-model \
  CORPUS=/path/to/some/python/package ./run-all.sh my-label

The switch

Production moved to NVFP4 on October 13th. Same served name (qwen3.8-27b), same flags, so OpenCode and the other clients didn’t notice anything except speed. The BF16 weights and the old unit file are still on disk in case I need to roll back.

What I haven’t tested: the GGUFs on quality, NVIDIA’s and RedHat’s NVFP4 builds, and Qwen’s FP8. With the harness sitting there, each of those is now one command and about an hour of GPU time. That’s a much easier sell than a weekend of eyeballing outputs.