Question the measurement
My LLM judge was flipping a coin a third of the time
What happened when I tested the judge I had trusted for months.
The notebook
Writing
Small thoughts, things I learned the hard way, and ideas that needed more room.
Follow via RSS — every new thought, field note, and essay in one feed.
A few places to start
Follow a fix
Every FP8 GEMM traps on DGX Spark (SM121)
An exact failure, the hardware behind it, and the two-line fix.
Look at the craft
A page transition that breathes
The engineering behind a quieter page transition on this site.
Browse the notebook
13 pieces about Inference & performance
Essay
I grafted a speculative decoding head into a 90 GB model file
The model card advertised a 4B multi-token-prediction head. No published GGUF contained it, and the architecture had no code path to run it. Both halves arrived within 36 hours from two different strangers, in incompatible forms — so I merged the head into the target file myself. It went from 27.75 to 43.30 tok/s, and three of my four mistakes along the way were about verification, not tensors.
Essay
Tokens per second told me the wrong model was faster
A 125B model decoding at 26.6 tok/s looks unremarkable beside a 31B at 29.3 — until you notice the 31B is running a drafter, so its number is a forward-pass rate multiplied by an acceptance length. Per forward pass the big model was 3x faster. Here is how to decompose the number, and what the bandwidth arithmetic says is still on the table.
Essay
I turned off concurrency and the server got faster
Four slots served my agentic pipeline at 1.2% prefix-cache reuse. One slot served the same pipeline at 86.7%, and saved 11.6 minutes of prefill in a 39-minute window. Slots schedule requests; they do not add compute — and on a single GPU they scatter the one thing a prefill-bound workload actually depends on.
Field note
Dropping --kv-unified silently turns a 262144-token server into a 65536-token one
llama-server divides the context size by the slot count unless --kv-unified is set: n_ctx_seq = n_ctx / n_seq_max. There is no error and no warning. The same command line that asks for --ctx-size 262144 --parallel 4 serves 65536 per sequence, and the only evidence is one line in the boot log. Long prompts then fail or shift in ways that look like a model problem rather than a configuration one.
Field note
GGML_ASSERT(layer.nextn.hc_head_norm) — two MTP pull requests disagree about one tensor name
A published Qwen3.8-Flash-Next MTP head loads against PR #27739 and aborts against PR #27836 with "MTP block missing nextn.hc_head_norm". The head is not damaged and no tensor is missing: the same three hyper-connection tensors are exported as top-level output_hc_norm/down/up by one PR and expected as blk.N.nextn.hc_head_norm/down/up by the other, because a standalone head file has no trunk to collide with and a grafted head does. The published merge script filters to blk.* and silently drops all three. Rename them instead of dropping them.
Field note
A per-request speculative.type of "none" still drafted 198 tokens
llama-server accepts per-request speculative.n_max, p_min and type, which makes sweeping a drafter on one load look safe. It is not: the per-request value appends to the launch-time list rather than replacing it, so the echoed settings read speculative.types = none,draft-mtp and every row of a six-configuration sweep was the same configuration. The "off" arm reported 198 drafted tokens. A speculative-decoding baseline needs a server restart, and the echoed settings block is the only thing that reveals it.
Essay
My guardrail checked the config, not the corpus
I have swapped the local model under my newspaper 85 times in four months. That looked like a natural experiment, so I built an instrument to read it — including a gate that refuses comparisons contaminated by time. The gate passed. It was reading my switch log instead of my writing, and the two models it cleared had never once alternated.
Essay
The model ran out of room to think, and every diagnostic guessed wrong
Three times this year a reasoning model spent its whole token budget thinking and handed back empty content under HTTP 200. Three times, the code reading the result invented a different explanation — a parser bug, then an unconstrained grammar. Neither was true, and one of them fired inside a safety check.
Field note
A quantized KV cache crashes qwen4exp on llama.cpp, in two different ways
A quantized KV cache (--cache-type-k q8_0) crashes qwen4exp twice, not once. The first crash is a Hadamard-rotation gate that qwen4exp does not implement; disabling that rotation produces a server that loads, saves 4 GiB, and matches f16 on every quality test run against it, then fails a second, different assert once real concurrent load arrives across all four slots. That second assert looked quantization-specific at first. It is not: the identical assert was later confirmed on f16 too, with no quantization involved, so the real cause is a multi-slot desync in llama_memory_hybrid_idx that q8_0 merely reaches sooner. Verdict: f16, run with --parallel 1 until upstream fixes the desync.
Field note
The Qwen3.8-Flash-Next MTP head was missing from the August 26 GGUFs
On August 26, the converter disabled MTP export and the architecture lacked its inference path. This historical field note now links to the August 28 follow-up, which documents working speculative decoding using a third-party head, a draft PR, and a local GGUF graft.
Field note
Which NVFP4 MoE backend actually works on SM121 (it depends on the model)
One model needs --moe-backend cutlass and is slow under marlin. Another needs VLLM_SCALED_MM_BACKEND=marlin and crashes without it. NVIDIA officially recommends marlin. All three statements are true.
Field note
FlashInfer CUTLASS MoE fails TMA descriptor init on SM121
FlashInfer ships a precompiled fused-MoE library built for SM120. Its TMA descriptors do not initialize on SM121, so concurrent MoE batches crash after hours of clean operation. vLLM cannot patch it.
Field note
Every FP8 GEMM traps on DGX Spark (SM121)
vLLM guards its FP8 CUTLASS kernels on __CUDA_ARCH__ == 1200 and executes a deliberate trap instruction on anything else. DGX Spark is arch 1210, so every FP8 GEMM crashes. Two lines fix it.