The model on my homelab inference box shipped with its own speculative decoding head, and I wasn’t using it. One flag and a restart took single-user decode speed from 28.5 tok/s to between 56 and 68 tok/s. No new download, no change in output quality, and the clients didn’t need to change anything.

I have been insisting on running the full BF16 weights on the Pro 6000, not a quantized version. Recently I noticed that the tokens/sec I was getting was way too low. Two days earlier I had tested Qwen3.8-Flash-Next against the Qwen3.8-27B I run day to day. Flash-Next did about 102 tok/s. The 27B did 28.5. So I started digging into why, and found some pretty juvenile mistakes on my side. Not using the MTP head was the biggest one.

The setup

Quick context if you haven’t read the post about building this box:

  • ai-box: one NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB
  • Model: Qwen3.8-27B, full BF16 weights, 262K context, FP8 KV cache, prefix caching on
  • Server: vLLM 0.31.0 as a systemd service (vllm-coder), localhost only on port 8000
  • In front of it: a small stream-filter proxy on port 4000 for the LAN, and LiteLLM on 4001
  • Clients: mostly OpenCode, plus a few other things that talk OpenAI or Anthropic style APIs

What MTP actually is

MTP stands for Multi-Token Prediction. Some models are trained with one or more extra small layers that try to predict tokens past the next one. Qwen3.8-27B has one of these layers.

vLLM can use that extra layer as a draft model for speculative decoding. The MTP head cheaply guesses the next few tokens, then the full model checks all of them in a single forward pass. Every guess it accepts is a token you got for free. A wrong guess costs very little, because the full model just throws it away and carries on from its own token.

The important part is that the full model still has the final say on every token. So the output follows exactly the same distribution as normal decoding. You are not trading quality for speed.

Why this helps so much on a 27B dense model: decoding is limited by memory bandwidth, not compute. Every token means reading roughly 55 GB of weights from VRAM. If each of those reads produces about 2.3 tokens instead of 1, you get most of that back as speed.

You can also do speculative decoding with a separate small draft model (EAGLE, DFlash and friends). MTP is nicer for a home setup. There’s nothing extra to download, no risk of a tokenizer mismatch, and the head was trained together with the main model, so it guesses right more often.

How to check if your model has it

Look in the model’s config.json. For multimodal Qwen models it’s under text_config:

"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false

Then check model.safetensors.index.json for tensors starting with mtp.. Mine has 15 of them (mtp.fc.weight, mtp.layers.0.* and so on). If the config says there are MTP layers but the weights aren’t there, the flag won’t help you.

The architecture here is Qwen3_5ForConditionalGeneration, and vLLM resolves the draft to Qwen3_5MTP on its own. vLLM 0.31 knows a bunch of MTP variants (qwen3_5_mtp, qwen3_next_mtp, deepseek_mtp, glm4_moe_mtp, gemma4_mtp and more), but you can just say mtp and it picks the right one.

The change

This is the whole diff to /etc/systemd/system/vllm-coder.service:

-  --max-num-seqs 512 \
+  --max-num-seqs 480 \
   ...
-  --served-model-name qwen3.8-27b
+  --served-model-name qwen3.8-27b \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The rest of the flags, for reference: --dtype auto --kv-cache-dtype fp8 --gpu-memory-utilization 0.87 --max-model-len 262144 --attention-backend FLASHINFER --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3.

One thing to know about num_speculative_tokens: 2. The model only has one MTP layer, so to draft two tokens vLLM runs that same layer twice. vLLM warns you about it:

Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer, which may result in lower acceptance rate

That’s true, the second guess is accepted less often. But it still lands 52 to 59% of the time, which is plenty to be worth it.

The --max-num-seqs change wasn’t in my plan. It came from a failed start.

The failed start: not enough Mamba cache blocks

The MTP head loaded fine, then it fell over during CUDA graph capture:

ValueError: max_num_seqs (512) exceeds available Mamba cache blocks (501). Each decode sequence requires one Mamba cache block, so CUDA graph capture cannot proceed.

Mamba on a Qwen model threw me for a second. Qwen3.8 is a hybrid architecture. Some of its layers are Gated DeltaNet (a linear attention design), and the rest are normal full attention. The linear attention layers keep a fixed size recurrent state for every sequence, so each concurrent sequence needs its own “Mamba cache block”. The MTP weights and their state took enough of my 0.87 memory budget that the number of available blocks dropped from at least 512 to 501.

Two ways out: lower --max-num-seqs or raise --gpu-memory-utilization. I went with 480 sequences. Nobody needs 512 concurrent agents at home, and I’d rather not touch the memory budget.

This one is worth remembering because it can’t happen on a pure transformer model. As more hybrid models show up, I think more people will run into it.

The numbers

Startup took about 120 seconds (68.6 s of engine init, 17.8 s of that was torch.compile). Weights plus non-torch memory came to 52.9 GiB, peak activations 3.7 GiB, CUDA graphs 0.9 GiB. That left a KV cache of 747,706 tokens, enough for 2.85 requests at the full 262K context.

For the benchmark I ran a single stream, forced 1,500 output tokens with ignore_eos, turned thinking off, and put a random nonce at the start of each prompt so prefix caching couldn’t help. Times include TTFT, but that was around 0.1 s so it doesn’t move the numbers.

Prompt Before (no MTP) With MTP (2 runs) Speedup
Code: “LRU cache implementation with tests” 28.5 tok/s 67.5 / 66.3 tok/s ~2.35x
Prose: “History of the Roman Empire” 28.5 tok/s 57.2 / 55.9 tok/s ~2.0x

Bar chart of Qwen3.8-27B single-stream decode speed on vLLM: 28.5 tok/s before MTP for both prompts, rising to 66.9 tok/s (2.35x) for code and 56.5 tok/s (1.98x) for prose with MTP speculative decoding

What vLLM reported about the speculative decoding itself:

  • Mean acceptance length was 2.27 to 2.39 tokens per verification step, out of a possible 3
  • The first draft position was accepted 75 to 80% of the time, the second 52 to 59%
  • Overall draft acceptance rate was 64 to 70%
  • My first smoke test through the proxy (write a palindrome function) had 70 drafts, 140 draft tokens, 90 accepted

The part I liked most: code accepts better than prose. Code is predictable. Boilerplate, syntax, test scaffolding, it’s all stuff the little head can guess. That works out nicely, because this box’s main job is serving coding agents.

Keeping an eye on it

vLLM exposes the counters on its metrics endpoint:

curl -s localhost:8000/metrics | grep spec_decode

You get vllm:spec_decode_num_drafts_total, vllm:spec_decode_num_draft_tokens_total, vllm:spec_decode_num_accepted_tokens_total and a per-position vllm:spec_decode_num_accepted_tokens_per_pos_total{position=...}.

There’s also a periodic summary line in the logs with the mean acceptance length and per-position rates:

journalctl -u vllm-coder | grep "SpecDecoding metrics"

Prometheus already scrapes ai-box, so an acceptance rate panel in Grafana is the obvious next step. That’s already a ticket on my board.

What changed for the clients

Nothing. MTP is entirely server side. Same endpoint, same model name (qwen3.8-27b), same context and output limits. I didn’t touch the OpenCode config at all. Responses just come back about twice as fast.

What I haven’t tested yet

Everything above is single-user. A few things I still want to check:

  • Concurrency. Speculative decoding usually helps less as the batch grows, because the GPU stops being memory bound and starts being compute bound. Before MTP I got 105 tok/s total at 4 concurrent streams (about 27 each). I want to rerun 4, 16 and 64 streams with MTP on and off.
  • Long context. Decode was 28 and 27 tok/s at 32K and 125K prompts before. Not measured with MTP yet.
  • Prefill and TTFT. Should be unchanged, but I haven’t checked properly.
  • num_speculative_tokens of 1 and 3. Two is a guess that worked. A proper sweep would show where the trade-off sits.
  • Thinking mode. All of my tests had thinking off. Reasoning text might accept at a different rate.

I’ll write these up in a follow-up post along with the Grafana panel.

If you want to undo it, I kept the old unit file:

sudo cp ~/vllm-coder.service.bak-20261013-premtp /etc/systemd/system/vllm-coder.service
sudo systemctl daemon-reload && sudo systemctl restart vllm-coder

So, Flash-Next or 27B?

Two days ago Flash-Next was about 3.5x faster on decode. Now the gap is closer to 1.5 to 1.8x, and the 27B still wins on accuracy, vision and prefill. That’s a much easier trade to live with, and thus I’m staying on the 27B.

The funny part is that the speedup was sitting in the weights the whole time. I just never asked it to do the thing it was trained to do. Now that I know 28.5 wasn’t the ceiling, I want to see how much faster I can make it go. That might even mean giving up on BF16 and trying my luck with a quantized version. The Unsloth variants of Qwen3.8 are first on my list.