Question the measurement
My LLM judge was flipping a coin a third of the time
What happened when I tested the judge I had trusted for months.
The notebook
Writing
Small thoughts, things I learned the hard way, and ideas that needed more room.
Follow via RSS — every new thought, field note, and essay in one feed.
A few places to start
Follow a fix
Every FP8 GEMM traps on DGX Spark (SM121)
An exact failure, the hardware behind it, and the two-line fix.
Look at the craft
A page transition that breathes
The engineering behind a quieter page transition on this site.
Browse the notebook
21 pieces
Field note
Downloading one new file from a Hugging Face repo invalidates the path to all the others
Fetching a newly published mmproj moved refs/main to a fresh snapshot directory containing only that one file, while 93.6 GB of weights stayed in the previous snapshot. Any launcher resolving through refs/main — which is the recommended practice, since snapshot hashes change on every re-pull — then fails its own existence check on weights that are plainly on disk. The fix downloads nothing: a second hf download with the original include pattern relinks the existing blobs into the current snapshot.
Field note
Dropping --kv-unified silently turns a 262144-token server into a 65536-token one
llama-server divides the context size by the slot count unless --kv-unified is set: n_ctx_seq = n_ctx / n_seq_max. There is no error and no warning. The same command line that asks for --ctx-size 262144 --parallel 4 serves 65536 per sequence, and the only evidence is one line in the boot log. Long prompts then fail or shift in ways that look like a model problem rather than a configuration one.
Field note
GGML_ASSERT(layer.nextn.hc_head_norm) — two MTP pull requests disagree about one tensor name
A published Qwen3.8-Flash-Next MTP head loads against PR #27739 and aborts against PR #27836 with "MTP block missing nextn.hc_head_norm". The head is not damaged and no tensor is missing: the same three hyper-connection tensors are exported as top-level output_hc_norm/down/up by one PR and expected as blk.N.nextn.hc_head_norm/down/up by the other, because a standalone head file has no trunk to collide with and a grafted head does. The published merge script filters to blk.* and silently drops all three. Rename them instead of dropping them.
Field note
A per-request speculative.type of "none" still drafted 198 tokens
llama-server accepts per-request speculative.n_max, p_min and type, which makes sweeping a drafter on one load look safe. It is not: the per-request value appends to the launch-time list rather than replacing it, so the echoed settings read speculative.types = none,draft-mtp and every row of a six-configuration sweep was the same configuration. The "off" arm reported 198 drafted tokens. A speculative-decoding baseline needs a server restart, and the echoed settings block is the only thing that reveals it.
Field note
pgrep -f matched my own shell, and nohup did not survive the timeout
Two process-management traps that both look like a crashed inference server. pgrep -f matches the command line of the process doing the grepping, so a stop script can kill its own shell and a wait loop can "confirm" a server stopped when it did not. And nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group — so a 90-second model load inside a 2-minute command budget is killed at exactly two minutes and looks identical to a load failure. Have the child write its own PID, and use setsid.
Field note
DefaultOOMScoreAdjust=200 in the systemd user manager kills terminal-launched servers first
A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.
Field note
Detecting which inference engine owns a port: vLLM vs. llama.cpp
Detection that infers the engine from the port number breaks the moment either engine moves. Moving llama.cpp onto the port normally used by vLLM mislabeled it, and downstream code silently wrote null. /props is a clean discriminator: llama-server serves it, and the vLLM OpenAI-compatible server has no such route and returns 404. A second gotcha in the same detour: docker ps --filter publish= does not match a container using host networking.
Field note
A pre-flight probe reported an exhausted thinking budget as an unconstrained grammar
An engine-swap safety check asked a reasoning model for a JSON verdict inside a 40-token budget. Reasoning consumed the whole budget, content came back empty, and the probe scored that as a schema that does not bind. The grammar was fine. A three-arm test isolates budget from grammar to show why.
Field note
A quantized KV cache crashes qwen4exp on llama.cpp, in two different ways
A quantized KV cache (--cache-type-k q8_0) crashes qwen4exp twice, not once. The first crash is a Hadamard-rotation gate that qwen4exp does not implement; disabling that rotation produces a server that loads, saves 4 GiB, and matches f16 on every quality test run against it, then fails a second, different assert once real concurrent load arrives across all four slots. That second assert looked quantization-specific at first. It is not: the identical assert was later confirmed on f16 too, with no quantization involved, so the real cause is a multi-slot desync in llama_memory_hybrid_idx that q8_0 merely reaches sooner. Verdict: f16, run with --parallel 1 until upstream fixes the desync.
Field note
The Qwen3.8-Flash-Next MTP head was missing from the August 26 GGUFs
On August 26, the converter disabled MTP export and the architecture lacked its inference path. This historical field note now links to the August 28 follow-up, which documents working speculative decoding using a third-party head, a draft PR, and a local GGUF graft.
Field note
qwen4exp is not qwen3next: the August 26 llama.cpp loading failure
In my August 26 setup, llama.cpp master could not load the qwen4exp architecture. Building PR #27742 ran the 125B model at 22-27 tok/s without speculative decoding. This dated investigation now links to the August 28 experiment that added a working drafter through a local GGUF graft.
Field note
RSS is the wrong instrument for memory on a DGX Spark
The llama-server process shows about 29 GiB of RSS while about 87 GiB of weights for Qwen3.8-Flash-Next sit in CUDA buffers that RSS never counts, and nvidia-smi reports [N/A] for memory on GB10 because there is no separate GPU pool to report. MemAvailable before and after start is the honest measurement, and that holds for every model on a DGX Spark, not just this one. The idle headroom measured at launch was not production headroom either: the same configuration was later killed by the kernel under real load, with nothing in its own log, and working defaults moved to a smaller context size and ubatch afterward.
Field note
A reasoning parser with no start/end strings silently disables thinking-budget enforcement
Muse-Glimmer delimits reasoning by channel markers rather than a token pair, so vLLM cannot resolve reasoning token ids, so reasoning_config.enabled stays false, so the budget cap never engages — on a model whose chat template reasons by default.
Field note
Muse-Glimmer returns empty reasoning_content while the parser demonstrably extracts it
Every completion reports reasoning_content: "" and reasoning_tokens: 0, even when a hundred tokens of reasoning were generated. The parser returns the reasoning when called directly. The defect is somewhere between the two, and I have not found it.
Field note
cache_kernels.cu fails to build on aarch64 with CUDA 13.0
A KV-cache batching optimization calls cuMemcpyBatchAsync with a signature the CUDA 13.0 aarch64 driver API does not have. The build stops before anything runs.
Field note
Which NVFP4 MoE backend actually works on SM121 (it depends on the model)
One model needs --moe-backend cutlass and is slow under marlin. Another needs VLLM_SCALED_MM_BACKEND=marlin and crashes without it. NVIDIA officially recommends marlin. All three statements are true.
Field note
FlashInfer CUTLASS MoE fails TMA descriptor init on SM121
FlashInfer ships a precompiled fused-MoE library built for SM120. Its TMA descriptors do not initialize on SM121, so concurrent MoE batches crash after hours of clean operation. vLLM cannot patch it.
Field note
Every FP8 GEMM traps on DGX Spark (SM121)
vLLM guards its FP8 CUTLASS kernels on __CUDA_ARCH__ == 1200 and executes a deliberate trap instruction on anything else. DGX Spark is arch 1210, so every FP8 GEMM crashes. Two lines fix it.
Field note
Two ways a DGX Spark inference server dies after hours of clean operation
Mistral NVFP4 traps with an illegal instruction at around ten hours. Qwen3.6 int4 deadlocks silently at around seven, still answering health checks at zero tokens per second. Whether these share a root cause is still open.
Field note
Structured output stalls forever once you register 28 tools
vLLM builds a Lark grammar enumerating every registered tool as an alternative. Past roughly 28 tools the guidance parser gives up, and the model generates tokens that never reach the client.
Field note
IndexError in the multimodal processor when vLLM initializes Mistral Small 4
A change to how multimodal processors handle token inputs crashes Pixtral-architecture models during dummy-input construction, before the server ever accepts a request.