The cache hit that answered the wrong question

Prefix caching on my Qwen3.8-Flash-Next server had been off since the day it crashed the GPU. Every request re-read its prompt from the first token: the pipeline's long system preamble, the growing history of every multi-turn conversation, all of it prefilled again each time. On a machine whose single bottleneck is memory bandwidth, that is the most expensive thing you can choose to do twice.

The fixes existed. They were just spread across five open pull requests that nobody had merged. Last night I carried them into my build as overlays, turned the cache on, and wrote a soak to find out whether it was real.

The soak passed on everything I had designed it to catch. Then, in a phase I had added almost as an afterthought, one request wrote the answer to a question it had never been asked.

What a cache hit means on this model

On a plain transformer a prefix-cache hit is bookkeeping. The attention KV blocks for a prompt prefix already exist, the scheduler points the new request at them, prefill starts where the cached blocks end. Nothing has to be reconstructed; it only has to be found.

Flash-Next is not a plain transformer. Thirty-six of its layers are gated delta-net recurrences, and its n-gram embedding layer runs a short convolution with its own state. A recurrence does not have blocks you can point at. It has a state vector at each position, and reusing a prefix means restoring the state the model had reached at the end of it. vLLM handles that with a cache mode called align: every 1,600 tokens, it checkpoints the recurrent state alongside the attention blocks, and a hit restores the checkpoint at the last boundary the new prompt shares, then prefills the remainder.

That is where it broke on the GB10. The engine takes the smallest block size across its cache groups, and one of this model's groups is a small ring buffer for the sparse attention's raw keys. The ring's block dragged the alignment grid down, the restore read a slot nothing had written, and the result was either an illegal memory access or, with a bounds guard in place, a restored state of exactly zero. Greedy answers changed on cache hits. A conversation would come back empty on turn two.

Five pull requests fix the pieces: the scheduler splits chunks on the recurrent grid, the state index is seeded from the right block size, the cache manager honours the drafter's extra block, block tables are indexed by request slot instead of batch row, and the drafter's own cache groups are flagged so they stop poisoning cross-request lookups. All open. I apply them to the image at runtime as a stack of diffs, regenerated per upstream commit, so a git pull that changes a file underneath one of them stops the launch instead of silently mixing versions.

What would prove a hit is real

A metric that says prefix_cache_hits_total went up proves that the scheduler found blocks. It does not prove the model resumed from the right state. Nor does "the answer looks fine": a zeroed recurrence still produces fluent text, because most of the model is attention and the attention blocks are correct. The failure mode is a model that has forgotten what was in the first 1,600 tokens while sounding entirely confident about it.

So the soak plants evidence. A 4,500-token memo, long enough to cross more than one alignment boundary, carries a reconciliation code exactly once, in the middle. Every request against that memo asks for the code and nothing else. If a hit restores real state, the code comes back verbatim. If a hit restores zeros, the model has no way to know it.

Around that core: a cold request followed by the identical request, to confirm a hit and a speedup. A six-turn conversation that grows on the shared memo, where hits should rise every turn and the code must still be recallable at turn seven. Four concurrent streams of different lengths, two rounds, because the original bug was a concurrency bug. And an 11,000-token prompt run twice, cold then hit, for the number that justifies the whole exercise.

It worked

Hits arrived in 1,600-token blocks, as align mode says they should. The identical repeat was faster. The planted code came back exactly on every hit: alone, with a longer suffix appended, at turn seven of the conversation, and on all eight concurrent requests across both rounds. Zero preemptions. Nothing in the engine log.

The 11,208-token prompt went from 5.99 s cold to 0.96 s on a hit, an 84% cut in time to first token. Over the whole soak the cache hit rate settled around 40%.

One assertion failed, and I had expected it to: the 120-token greedy summary differed between the cold run and the hit. That looks like the zero-state bug. It is not. Two hits differ from each other just as much, somewhere between the 110th and 240th character, both coherent summaries that chose different words, while the same test on a 1,200-token memo, short of a single alignment boundary, is byte-identical across a cold run and three hits. That is the sparse attention's top-k selection, which on this GPU hands out result slots in thread-arrival order and breaks ties first-come. It varies from launch to launch on any prompt long enough to engage it, with or without a cache. There is a kernel rewrite for it upstream, in C++, which I cannot carry as an overlay. The cache was innocent of this one, and I moved on.

Then stream 3 answered the wrong question

Round one of the concurrency phase, third run of the night. Four streams sharing the memo. Streams 0 and 2 ask for the code. Streams 1 and 3 ask for a hundred words on section three. Stream 3 returned:

7395-2841-6620

Not a hundred words. The reconciliation code. The exact string its two neighbours had just asked for and it had not.

If you have read the bug reports, that is the shape. A request served from a state that belonged to another request. It is what "zero-state restore" looks like one step further along, where the restored state is not empty but somebody else's. I had four-way concurrency, cache hits in flight, and a pull request in my stack whose description says the fault class it fixes "poisons the recurrent state and never recovers". I stopped and treated it as a leak until proven otherwise.

Proving otherwise

The obvious reproduction did nothing. The same prompt alone, four times, all hits: four proper generations about section three, with the wording drifting between runs the way the top-k nondeterminism drifts. Six more rounds of the identical four-way pattern: twenty-four requests, zero leaks, every code recall exact. Four rounds with a fresh memo per request, so that no hit was possible: also clean.

Then the mechanism argument, which is the part I trust more than the counts. The question that leaked, "reply with the code only", sits at the very end of the recall prompts, well past the last 1,600-token boundary. It is never inside a checkpointed state. A restore can carry the state of the shared memo; it physically cannot carry a suffix that was never checkpointed. And if attention blocks were being mixed up between requests, the recall streams would sometimes get the wrong code, and across roughly sixty of them they never did.

Which left the weakness in my own soak. Every stream shared one memo, so every stream carried the same code. A leaked code and the stream's own code were the same twelve characters. The instrument could not tell the two apart, and a check that cannot distinguish the failure from the normal case passes vacuously. I have written that sentence about my own tooling before, and apparently I needed to write it again.

The fix was a phase where four concurrent streams carry four different memos with four different codes, rotating which slot gets which memo every round, so each slot lands on cache blocks a different slot wrote a round earlier. Any code from another memo appearing in an answer is now unambiguous. Ten rounds, then four more: fifty-six requests, zero foreign codes, every recall exact, every generation about section three.

Stream 3 had produced its own memo's code. The memo says "keep it verbatim". The system prompt says "answer with exactly what is asked and nothing else". A 6B-active model whose sparse attention selects a slightly different set of blocks on each launch will, once in a few dozen tries, resolve that tension the wrong way. It is a nondeterministic sample, not a leak, and the only reason it looked like a leak is that my test had been designed so that the two were indistinguishable.

What stays on

Prefix caching is now the default on that server. The pipeline is registered against it again. The soak lives next to the launcher and runs in about six minutes, and its determinism check now knows the difference between "the hit path disagrees with the cold path", which would be damning, and "the hit path disagrees with itself", which is the kernel. Every prompt it uses is unique per run, because a soak that starts from a warm cache measures a hit and calls it cold, which cost me two false failures before I noticed.

Two things I would say to anyone turning this on for a hybrid model. First, the metric is not the proof. Plant something the model can only know if the restored state is real, and ask for it on every hit. Second, when the thing you are hunting is cross-request contamination, make sure the requests can be told apart. Mine could not, and I spent an hour on a ghost because of it.

That hour was still the right call. The silent failure these systems produce is exactly a confident answer from the wrong state, and the one way to be sure you are not shipping it is to build an instrument that would fail if you were.