I turned off concurrency and the server got faster

There is a default assumption in every inference deployment I have ever set up: more slots is more throughput. --parallel 4 sounds like four times the work. It is right there in the flag name.

On a single GPU it is not just wrong, it can be actively backwards. I measured a 72x drop in prefix-cache reuse going from one slot to four, on the same model, the same host, and the same production workload. Dropping to --parallel 1 — a change I made defensively, to dodge an unrelated crash — turned out to be the largest throughput win available to me, and it cost nothing.

Here is the measurement, and more usefully, the mechanism.

The setup

A DGX Spark (GB10, 128 GB unified memory) running llama-server with Qwen3.8-Flash-Next — a 125B-total / 6B-active MoE, 93.6 GB on disk. The client is my own agentic newsroom pipeline: an editorial stage that issues 66 LLM calls over 38 minutes, mostly long prompts against a stable system preamble.

The important number about that workload is its shape. Over one complete stage run:

metric	value
LLM calls	66
prompt tokens presented	384,119
completion tokens	21,271
prompt : completion	18 : 1

Eighteen tokens in for every token out. That ratio decides everything below. A workload like this is prefill-bound: the expensive part is reading the prompt, not writing the answer.

Prefill is the cost, and it gets worse as it goes

Decode throughput is the number everyone quotes, so it is worth showing what prefill actually costs on a long prompt. Cumulative rate is what the server logs; marginal rate is what each additional 4K block really costs.

tokens processed	elapsed	cumulative	marginal
4,236	6.68 s	634 tok/s	634 tok/s
8,332	13.52 s	616 tok/s	599 tok/s
12,428	23.10 s	538 tok/s	428 tok/s
16,524	33.49 s	493 tok/s	394 tok/s
20,620	45.73 s	451 tok/s	335 tok/s
23,581	55.71 s	423 tok/s	297 tok/s

Marginal prefill more than halves between the first 4K and the last. A 25.6K-token prompt spends about 60 seconds before it emits a single token — roughly 45% of that call's wall clock. Qwen Sparse Attention is meant to blunt exactly this curve. It clearly does not flatten it.

So the highest-value thing you can do to a prefill-bound workload is not prefill. Which is what the prefix cache is for.

The measurement

Two windows of continuous production serving, same model, same context size, same batch geometry, f16 KV. The only difference is the slot count.

metric	`--parallel 4`	`--parallel 1`
prompt tokens presented	26,736	403,992
...actually prefilled	26,419	53,717
...served from cache	317	350,275
prefix-cache reuse	1.2%	86.7%
decode, aggregate	~22.5 tok/s	23.79 tok/s

86.7% of all prompt tokens never touched the GPU. At the measured 503 tok/s prefill rate, that cache avoided roughly 696 seconds — 11.6 minutes of prefill inside a 39-minute window.

And decode was unchanged. 23.79 against ~22.5 tok/s aggregate is noise. Serializing the requests cost nothing measurable and bought back a sixth of the wall clock.

The mechanism is slot affinity

llama-server picks a slot for each incoming request, and it can reuse whatever KV prefix that slot already holds. You can watch it choose:

slot get_availabl: id  0 | task N | selected slot by LCP similarity, f_sim_best = 0.224 (> 0.100 thold)

LCP is longest common prefix. When a slot's resident context shares a prefix with the incoming prompt, that prefix is already computed and gets skipped.

With one slot, every request lands on the slot holding the previous request's context. My pipeline's prompts share a long, stable preamble, so that preamble is resident, matched, and reused — call after call.

With four, requests scatter. When no slot is a strong match, selection falls back to LRU:

slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1

Each request lands on a slot holding some other request's context. The shared preamble is absent, so it gets recomputed from scratch. Four slots did not quadruple throughput. They quartered cache locality, and locality was the whole game.

The other half of the disappointment: on one GPU, slots do not add compute. They schedule access to it. Four concurrent requests completed in 118 s against a 123 s serial estimate — a 4% difference, which is to say none.

I got the diagnosis wrong first, and the wrong version is the instructive one

When I first saw prompt_tokens_cached_total sitting at 317 against 26,736 presented, I wrote this down:

Prefix caching is doing essentially nothing. Every long prompt pays full prefill from scratch. Restructuring prompts so the bulk sits in a stable shared prefix ahead of the variable part is the single largest available win.

Every sentence there is defensible. The measurement was real, the reasoning was sound, and the recommended fix was a genuinely good idea. It was also an answer to a question I had not asked.

I had attributed a 1.2% hit rate to the prompts — concluding my pipeline's requests did not share enough prefix to be cacheable. They shared an enormous prefix. The four-slot configuration was preventing those prefixes from ever being tested, because the request was rarely routed to the slot that held them.

The tell was available the whole time and I did not read it: the log line says selected slot by LRU, not by LCP similarity. LRU selection means the matcher found nothing worth matching. On a workload whose prompts are 90% identical, that should have been the alarm. Instead I read a low cache-hit number and explained it with the first plausible cause, which happened to be the one that put the fault in my prompts instead of my configuration.

This is the same failure I keep writing about: the instrument reported honestly, and I supplied the causal story. A cache-hit rate tells you tokens were recomputed. It does not tell you why, and the difference between "the prompts don't share a prefix" and "the router never gave the prefix a chance" is the difference between a week of prompt engineering and a one-character config change.

What actually generalizes

None of this is specific to qwen4exp, or to llama.cpp, or to unified memory. The shape is:

When a workload is prefill-bound and its prompts share long prefixes, concurrency can be actively counterproductive on a single accelerator. Slots multiply the number of distinct KV states you are trying to keep warm, while the compute available to warm them stays fixed.

Three things worth doing before you raise a slot count:

Measure cache reuse, not throughput. prompt_tokens_cached_total / prompt_tokens_total from /metrics. If that ratio is low on a workload with shared prefixes, adding slots will make it lower.
Read the slot-selection lines, not just the totals. by LCP similarity means the router is doing its job. by LRU on a repetitive workload means it is not.
Check your prompt:completion ratio. At 18:1, decode tuning is nearly irrelevant — I could double my tok/s and move the stage wall-clock by a fifth, because LLM time is only ~45% of the stage to begin with. Prefill avoided is worth several times more than prefill accelerated.

Concurrency is a real tool. It is a tool for saturating hardware that is idle, and a single GPU serving one request at a time is not idle. Mine was busy recomputing a preamble it already had.

Measured on a DGX Spark (GB10 / SM 12.1, 128 GB unified LPDDR5x) running llama-server from llama.cpp master with unsloth/Qwen3.8-Flash-Next-GGUF at UD-IQ4_XS. More on the box in home lab AI infrastructure; the pipeline driving it is the Agentic Daily. Related field notes: RSS is the wrong instrument for memory on a DGX Spark and a quantized KV cache crashes qwen4exp two different ways.