Tokens per second told me the wrong model was faster

Every local-inference conversation converges on one number. Tokens per second. It is on every model card, every benchmark table, every forum post comparing two quants.

It is also a product of two things, and almost nobody reports the second one. I spent a week believing a 125B model was merely competitive with the 31B sitting next to it on the same box. Decomposed properly, it was doing three times the work per unit of compute — and the raw tok/s number had been actively hiding that.

The number that looked unremarkable

Qwen3.8-Flash-Next is 125B total parameters with 6B activated, plus a 51B n-gram embedding table. On my DGX Spark it decodes at 26.6 tok/s single-stream, no speculative decoding of any kind.

Next to the dense models on the same host, that is a shrug:

model	tok/s
Gemma-4-31B-QAT	29.3
Qwen3.8-Flash-Next (125B-A6B)	26.6
Qwen3.8-27B-INT4	25.5
Muse-Glimmer-30B	20.2

Roughly a wash. A 125B that ties a 31B is fine but not exciting.

Then I noticed that every model in that table except one is running a drafter.

tok/s = forward passes/s x mean acceptance length

Speculative decoding works by having a small, cheap model propose several tokens, then having the real model verify them all in one forward pass. When the draft is accepted, you emitted three or four tokens for the price of one pass.

That means a speculative model's tok/s is not a rate of computation. It is:

tokens/s  =  forward passes/s  x  mean acceptance length (MAL)

MAL is how many tokens survive verification per pass. It is the multiplier, and it is the number that goes missing from every benchmark table. Divide it back out and the same four models look completely different:

model	tok/s	MAL	forwards/s
Gemma-4-31B-QAT (MTP γ=7)	29.3	3.60	8.1
Qwen3.8-27B-INT4 (DFlash2)	25.5	2.96	8.6
Muse-Glimmer-30B (DFlash2)	20.2	3.63	5.6
Qwen3.8-Flash-Next (none)	26.6	1.00	26.6

Per forward pass the A6B is roughly 3x faster than the dense models. Which is precisely what 6B active parameters is supposed to buy you, and exactly what the headline number concealed.

The arithmetic behind it is simple: a dense 27B at 4 bits must read about 13.5 GB of weights to produce one token. The MoE reads about 2.6 GB. Same box, same memory bus, one fifth the traffic.

This matters practically, not just aesthetically. The dense models are near the top of their acceptance range already — there is not much multiplier left to win. The MoE had a multiplier of 1.00 and every bit of it still on the table.

The multiplier is steeply nonlinear near the top

I got to test that directly a few days later, when a working multi-token-prediction path finally existed for this architecture and I grafted the drafter in. Two workloads, same model, same settings, temp 0:

workload	bare	with drafter	acceptance
code	27.75 tok/s	43.30 (+56%)	91.6%
prose	28.28 tok/s	33.47 (+18%)	86.5%

That table is the whole mechanism in miniature. Acceptance moved 5 points. Throughput moved 38.

The reason is that a rejection does not cost you one token, it costs you the tail behind it. With a depth-3 draft, 91.6% per-token acceptance usually lands all three; 86.5% usually loses one, and everything after the loss is discarded. Small changes in acceptance produce large changes in amortization, which is why "speculative decoding gives about a 2x" is not a useful sentence — the answer depends entirely on how predictable your output is. Code is predictable. Prose is not.

What the bandwidth arithmetic says

Once you are counting forward passes, you can ask the real question: how close is this to the hardware ceiling?

At ~2.6 GB read per token against the GB10's ~273 GB/s of memory bandwidth, bandwidth alone would permit roughly 100 tok/s. Measured bare, it does 26.6. That is about a quarter of bandwidth-bound.

The dense models sit far closer to their own ceilings, and the reason is access pattern rather than volume: reading 13.5 GB contiguously is exactly what LPDDR is good at. The MoE reads less and reads it badly.

The specific suspect is the n-gram embedding table. Per token it performs 16 random gathers of about 170 bytes each, scattered across 26.82 GiB. When I profiled the access pattern, 0 of 4.75 million consecutive gathers shared a memory page. Every single one is a fresh page walk. That is a pathological case for a memory controller, and it is roughly the worst possible ratio of bytes-wanted to bytes-paid-for.

So the honest performance statement for this model is not "26.6 tok/s." It is: 26.6 forward passes per second, three times the rate of the dense models it sits beside, at about a quarter of what the memory bus would permit, with the gap owed to gather locality rather than compute. That sentence tells you what to fix. The single number does not.

A merge was worth 28%, and no config changed

One more reason to distrust a bare tok/s figure: it is a property of a build, not of a model.

The architecture support for this model landed as an open pull request, then got squash-merged into llama.cpp master. Same prompt, same parameters, same quant, single stream, before and after:

build	decode
pre-merge branch	20.30 / 20.45 / 20.80 tok/s
master, post-merge	26.32 / 26.61 / 26.63 tok/s

About +28%, consistent across runs, with long-context recall unchanged — both needle retrievals came back byte-identical to the pre-merge control. Not one flag in my start script changed. If I had benchmarked this model a day earlier and published, I would have published a number that was 28% wrong about the same weights on the same hardware.

Keep the number honest

None of this requires special tooling. Four habits cover it:

1. Report MAL alongside tok/s, or report that it is 1.00. A benchmark table mixing speculative and non-speculative entries is comparing two different quantities. llama-server exposes spec_decode_* counters at /metrics; when they are all zero, nothing is drafting and your tok/s is your forward rate.
2. Compute bytes-per-token and divide by your memory bandwidth. It takes thirty seconds and tells you whether you are chasing a 10% tuning win or a 4x structural one.
3. Benchmark the same workload class you actually run. The 56%/18% code-versus-prose split above is not noise, it is the acceptance distribution, and it will not transfer between workloads.
4. Record the build. Not the model, not the quant — the commit. A merge moved mine 28%.

The general form is one I keep relearning in different costumes: a metric that compounds two factors will let you optimize the wrong one indefinitely, and it will feel like progress the entire time. I have made the same class of mistake with an LLM judge and again with a guardrail that measured the wrong noun. It is always the same lesson, and I have never once recognized it in advance.

---

Measured on a DGX Spark (GB10 / SM 12.1, 128 GB unified LPDDR5x, ~273 GB/s) running llama-server from llama.cpp master with unsloth/Qwen3.8-Flash-Next-GGUF at UD-IQ4_XS. More on the hardware in home lab AI infrastructure. Field notes: qwen4exp is not qwen3next and the MTP head is in none of the GGUFs.