Question the measurement
My LLM judge was flipping a coin a third of the time
What happened when I tested the judge I had trusted for months.
The notebook
Writing
Small thoughts, things I learned the hard way, and ideas that needed more room.
Follow via RSS — every new thought, field note, and essay in one feed.
A few places to start
Follow a fix
Every FP8 GEMM traps on DGX Spark (SM121)
An exact failure, the hardware behind it, and the two-line fix.
Look at the craft
A page transition that breathes
The engineering behind a quieter page transition on this site.
Browse the notebook
8 pieces about Agents & tooling
Field note
Dropping --kv-unified silently turns a 262144-token server into a 65536-token one
llama-server divides the context size by the slot count unless --kv-unified is set: n_ctx_seq = n_ctx / n_seq_max. There is no error and no warning. The same command line that asks for --ctx-size 262144 --parallel 4 serves 65536 per sequence, and the only evidence is one line in the boot log. Long prompts then fail or shift in ways that look like a model problem rather than a configuration one.
Essay
My guardrail checked the config, not the corpus
I have swapped the local model under my newspaper 85 times in four months. That looked like a natural experiment, so I built an instrument to read it — including a gate that refuses comparisons contaminated by time. The gate passed. It was reading my switch log instead of my writing, and the two models it cleared had never once alternated.
Essay
My LLM judge was flipping a coin a third of the time
I used a language model to pick the better of two drafts, and trusted it for months. Then I gave it two drafts from an identical configuration and asked it to choose. Here is what measuring a judge’s noise floor costs, and why every A/B result before it was unreadable.
Essay
The model ran out of room to think, and every diagnostic guessed wrong
Three times this year a reasoning model spent its whole token budget thinking and handed back empty content under HTTP 200. Three times, the code reading the result invented a different explanation — a parser bug, then an unconstrained grammar. Neither was true, and one of them fired inside a safety check.
Field note
A pre-flight probe reported an exhausted thinking budget as an unconstrained grammar
An engine-swap safety check asked a reasoning model for a JSON verdict inside a 40-token budget. Reasoning consumed the whole budget, content came back empty, and the probe scored that as a schema that does not bind. The grammar was fine. A three-arm test isolates budget from grammar to show why.
Field note
A reasoning parser with no start/end strings silently disables thinking-budget enforcement
Muse-Glimmer delimits reasoning by channel markers rather than a token pair, so vLLM cannot resolve reasoning token ids, so reasoning_config.enabled stays false, so the budget cap never engages — on a model whose chat template reasons by default.
Field note
Muse-Glimmer returns empty reasoning_content while the parser demonstrably extracts it
Every completion reports reasoning_content: "" and reasoning_tokens: 0, even when a hundred tokens of reasoning were generated. The parser returns the reasoning when called directly. The defect is somewhere between the two, and I have not found it.
Field note
Structured output stalls forever once you register 28 tools
vLLM builds a Lark grammar enumerating every registered tool as an alternative. Past roughly 28 tools the guidance parser gives up, and the model generates tokens that never reach the client.