I grafted a speculative decoding head into a 90 GB model file

Two days ago I wrote down that this model could not do speculative decoding, and that fixing it would require two separate upstream changes rather than one: somebody had to export the prediction head into a GGUF, and somebody had to implement the graph that drives it for this architecture.

Both happened within 36 hours. Neither happened at the vendor. Neither is in master. And they were built against each other's assumptions closely enough to almost work, which turned out to be the interesting part.

The result was 27.75 → 43.30 tok/s on code, a 56% gain, for about ninety minutes of work and 1.9 GB of new bytes on disk.

What the model advertised, and what shipped

Qwen3.8-Flash-Next is a 125B/A6B MoE whose card lists an MTP head: "1 layer, trained with multi-steps," 4B parameters. Multi-token prediction is the built-in form of speculative decoding — instead of running a separate small draft model, the target model carries an extra block trained to guess the next few tokens, and verifies its own guesses in a single forward pass.

Every other model on my host runs a drafter. This one could not, and the decisive evidence was not in the metadata but in the converter that shipped with the architecture PR:

# the MTP block is a separate draft head; vLLM drops it too
supports_mtp_export = False
no_mtp = True

Corroborating: no nextn_predict_layers key in any published GGUF, no separate mtp-*.gguf in the repo, and zero nextn references anywhere in the architecture's C++ source. The generic plumbing existed in llama-arch.cpp; nothing drove it.

That is why I called it two problems. A head with no graph is 2.5 GB of dead weight. A graph with no head has nothing to run.

(A methodology note I owe the record: my first "no MTP" call rested on a missing metadata key plus a repo listing I had truncated with head -40. Neither actually ruled anything out. Reading the converter is what settled it. Read the code, not the symptoms.)

Both halves arrived, from strangers, in incompatible forms

piece	who	where	state
the graph	rmonsurate	llama.cpp PR #27836	open, draft
the head	dzannotti	a third-party GGUF repo	published, 2.5 GB Q4_K_M
the export in master	—	—	still nothing

Two strangers, working independently, solved one half each. Which sounds like the story ends here. It does not, because the head was exported for a different pull request.

The head has to live inside the target file

This is the fact that turns a 2.5 GB download into an afternoon.

With --spec-type draft-mtp and no -md flag, llama-server takes a branch that does something unusual — it never loads a second model at all:

} else if (spec_mtp) {
    // no llama_model_load_from_file here -- a second CONTEXT, same weights
    cparams.ctx_type  = LLAMA_CONTEXT_TYPE_MTP;
    cparams.ctx_other = ctx_tgt;
    llama_context * ctx_dft = llama_init_from_model(model_tgt, cparams);

The server source says so outright: "spec_mtp doesn't use load a model internally." The head is a block of the target model — blk.48.*, with nextn_predict_layers = 1 — not a separate drafter the way a conventional draft model is. So it cannot sit in its own file next to the weights. It has to be in them.

Which means merging a new tensor block into a 93.6 GB, three-shard GGUF. merge-mtp-head.py rewrites shard 1's metadata (block_count 48→49, nextn_predict_layers, compress_ratios, split.count, split.tensors.count 1224→1256), hard-links the three untouched shards, and writes the head as a new final shard. ~1.9 GB of new bytes; no re-download of the 90 GB.

The trap: two pull requests disagree about one tensor name

The published head was exported for PR #27739, the earlier MTP attempt. PR #27836 is a different implementation. They agree about everything except where the head's hyper-connection mixer lives:

	#27739 (the head as published)	#27836 (what I build against)
head's mixer	output_hc_norm/down/up (top level)	blk.N.nextn.hc_head_norm/down/up
why	an MTP-only file has no trunk, so the head's mixer becomes the file's mixer	the head lives inside a full model, which already owns the trunk's output_hc_*

Same three tensors. Same bytes. Different name. And both names are correct in their own context — that is what makes it a good trap rather than a bug. In a standalone head file there is no trunk to collide with, so the top-level name is the natural one. Inside a full model, that name is already taken.

The published merge script filters to tensors matching blk.*, which drops all three, and the server dies at load:

GGML_ASSERT(layer.nextn.hc_head_norm && "MTP block missing nextn.hc_head_norm") failed

Renaming them instead of dropping them is the fix. While in there, it is also worth not copying the head's own token_embd and output tensors: #27836 marks nextn.embed_tokens and nextn.shared_head_head as TENSOR_NOT_REQUIRED and falls back to the target's. That saves ~1.5 GB and makes the draft read the exact weights the target verifies against, which is strictly what you want from a drafter.

It works, and the acceptance curve is the whole story

Measured on UD-IQ4_XS, q8_0 KV, temp 0, 300 tokens, single stream:

workload	bare	MTP (n-max 3, p-min 0.75)	acceptance
code	27.75 tok/s	43.30 (+56%)	91.6% (197/215)
prose	28.28 tok/s	33.47 (+18%)	86.5% (154/178)

Re-measured at production geometry (ctx 131072, --parallel 4): 43.18 tok/s, 94.5% cumulative acceptance. The larger context and the extra slots cost single-stream decode nothing.

That two-row table is the mechanism in miniature: acceptance moved 5 points, throughput moved 38. Speculation buys you amortised forward passes, and near the top of the acceptance range that is steeply nonlinear. At 91.6% a depth-3 draft usually lands all three tokens; at 86.5% it usually loses one, and a rejection discards the entire tail behind it. Code is predictable. Prose is not. Anyone quoting "speculative decoding gives about a 2x" is quoting their workload, not the technique.

Three ways I got it wrong

None of the real difficulty was in the tensors.

1. I hard-linked a symlink. The merge script hard-links the untouched shards rather than copying 90 GB. But Hugging Face cache snapshot entries are symlinks into ../../../blobs/<hash>, and os.link() on a symlink duplicates the symlink — relative target and all — which resolves to nothing from the new directory. I produced two 79-byte "shards" and a server that died in 0.4 seconds. os.path.realpath() first. Cheap to fix, and it failed instantly rather than ninety seconds into a load, which is the only reason it cost nothing.

2. My baseline was never a baseline. llama-server accepts per-request speculative.n_max, p_min and type, so the obvious move is to sweep the drafter on a single load instead of restarting for every setting. Six configurations came back within 2% of each other — including "speculative.type": "none", which cheerfully reported 198 drafted tokens.

The give-away was in the echoed settings:

speculative.types = none,draft-mtp

The per-request value appended to the launch-time list rather than replacing it. Every row in that sweep was the same configuration, and "off" was MTP. Had I not read the echo, I would have published a table showing speculative decoding making no difference whatsoever — a confident, well-formatted, entirely fictional result.

A baseline needs a restart. And a check that cannot fail is not a check — which is the third time this year I have shipped an instrument that agreed with me for structural reasons rather than empirical ones.

3. pgrep -f matched the process doing the grepping. Twice. Once killing a PID that was my own shell, once "confirming" a server had stopped when it had not. This exact failure is written down in this very repository, and I walked into it anyway. The fix that actually holds is to stop pattern-matching entirely and have the child write its own PID:

sh -c 'echo $$ > "$PIDFILE"; exec llama-server ...'

A fourth, milder one: nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group. A 90-second model load inside a 2-minute command budget gets killed at the two-minute mark and looks exactly like a crash. setsid is the fix.

Standing caveats

I would rather publish these than a cleaner story:

• PR #27836 is a draft. If a nextn tensor gets renamed before merge, my grafted shard goes stale and the server aborts at load. A checker watches for it; the fix is to re-run the merge.
• Greedy identity is unverified here. The PR author reports temp-0 output byte-identical with the drafter on and off, on Metal. An independent tester on ROCm reports divergence beginning around token 60–120 and attributes it to numerics. I have not measured it on this host. If you need bit-exact reproducibility, turn MTP off.
• The drafter costs ~1.9 GB resident and is pure waste on a binary without the #27836 graph — such a build loads the model happily and logs unused tensor blk.48.nextn.* for every head tensor while running at full bare speed. My start script refuses to enable it in that case rather than serving a silently slower model.
• If the vendor publishes their own MTP export, this all goes away. That is the outcome to watch for, and it would fix the naming problem too, since they would export against whatever actually merges.

Why bother

Because the alternative was waiting. The head existed, the graph existed, and the only thing standing between them was three tensors with the right bytes and the wrong names.

That gap is where most of the practical work in local inference lives right now. The models ship faster than the runtimes support them, the runtimes ship faster than the exports catch up, and the interesting capability is usually sitting in two incompatible halves in two different strangers' repositories. Being able to read a converter, diff two PRs' tensor naming, and rewrite a GGUF header is not exotic skill — it is just the difference between running the model the card describes and running the one that happened to be packaged.

---

Measured on a DGX Spark (GB10 / SM 12.1, 128 GB unified LPDDR5x) with llama-server from llama.cpp master plus PR #27836 and one backport, serving unsloth/Qwen3.8-Flash-Next-GGUF at UD-IQ4_XS. Background: the MTP head is in none of the GGUFs, qwen4exp is not qwen3next, and why tokens per second told me the wrong model was faster. More on the hardware: home lab AI infrastructure.