My guardrail checked the config, not the corpus

I change the model under my newspaper constantly. Since April there have been 85 recorded switches across about twenty distinct builds — Gemma-4 at 31B and 26B, Qwen3.6 and Qwen3.8, Ling-3.0, a Laguna, a couple of things called Muse-Glimmer. Some of those swaps are whole families. Some hold the weights fixed and change only the quantization: the same Qwen3.6-35B in four different quants, the same Qwen3.8-27B in Q4, Q8, fp8 and an int4 variant.

The paper kept writing through all of it. 244 long-form pieces since May, every one of them scored by an adversarial critic that reads the draft, argues with it, and records what it made the writer change. Nobody planned this as an experiment. But four months of production prose with the model swapped underneath it, and a critic's verdict attached to every piece, is the kind of dataset you cannot buy. So I built something to read it.

The instrument was straightforward. Parse the switch log into epochs. Walk every published article, pull its frontmatter, join it to the critic's trace by ID. About half the corpus had no generation timestamp in its frontmatter at all, but every one of those had a critic trace, and the trace is timestamped — so the gap closed completely, and all 244 pieces got an exact model attribution rather than a guess.

Then I did the part I was pleased with at the time. I wrote the guardrails before I looked at any numbers.

Two gates, written before the data

The first gate was volume. A model that wrote four pieces does not get a bar on a chart next to one that wrote a hundred, so anything under a threshold renders as insufficient rather than a number. Nothing controversial there.

The second gate was the one I was proud of. An observational study across four months has an obvious problem: the harness didn't hold still. My own improvement agent patches prompts weekly. Persona voice anchors landed in August. The grounding checker didn't exist until July. If model A ran in May and model B ran in August, any difference between them is a difference between two different systems, and attributing it to the model is just wrong.

So I wrote a rule: two models are only comparable if their epochs alternate in both directions — A then B, and somewhere else B then A. A model confined to a window the other never shares is confounded with everything that changed in that window, and the tool should refuse to compare it rather than let me eyeball the difference and believe it.

I ran it. Of all the pairs in four months of switching, exactly one came back clean: the two Gemma-4-31B quantizations, QAT w4a16 against NVFP4. Between them they had written 185 of the 244 pieces. The best-powered pair in the corpus was also the only methodologically clean one, and it was the interesting comparison — same weights, same family, two quantization schemes. I stratified by piece type so the comparison couldn't be driven by one model happening to write more of the long multi-persona features, and started measuring.

The results looked real

Thirteen axes, two strata, twenty-six comparisons. Three survived a Bonferroni correction. The largest was enormous: on evening reads, NVFP4 wrote 4,688 words against QAT's 3,228 — a Cohen's d of −1.95, which is not a subtle effect. Right behind it, lexical diversity: 0.306 for QAT against 0.265 for NVFP4, d = 1.57.

Two significant, large, consistent findings about what quantization does to prose. I had the headline: the same model, quantized two ways, writes at different lengths and different lexical richness.

Both of them were wrong, for two different reasons, and only one of those reasons was my fault.

The first crack is arithmetic

Lexical diversity, as measured here, is a type-token ratio: unique words over total words. That statistic falls as text grows, mechanically, because you run out of new words to use before you run out of sentences to write. It is not independent of length. It is barely a style metric at all on documents of different sizes.

Across the evening reads, length and lexical diversity correlate at −0.865. Corpus-wide, −0.768. Restricting to pieces of similar length, the 0.041 gap between the two models collapses to 0.008.

So the second finding was never a finding. It was the first finding, wearing different units. I had not discovered two things about quantization; I had discovered one thing and then measured its shadow.

That is an ordinary mistake and I would have caught it in review. The next one I would not have.

The second crack is that the gate lied

Before writing any of this up, I checked something the tool had never asked: when were these pieces actually written?

lens          NVFP4 ×33 from 2026-05-11  →  QAT ×64 from 2026-06-14
evening-read  NVFP4 ×26 from 2026-05-11  →  QAT ×44 from 2026-06-12
sunday-read   NVFP4 ×3  from 2026-05-17  →  QAT ×7  from 2026-06-14

A month of one model. Then a month of the other. Zero alternations, in every stratum, in the pair my gate had certified as clean.

The gate was reading the switch log. The switch log genuinely alternates — QAT occupies epochs 26, 28 and 30; NVFP4 occupies 6 through 27, and 27 sits between two QAT epochs. By the letter of my own rule the pair interleaves. But an epoch is a period during which a model was configured. Several of those NVFP4 epochs produced no articles at all, and an epoch that wrote nothing still counted as an alternation.

I had written a gate to check that two models were interleaved in my writing, and implemented one that checks they were interleaved in my configuration. Those are different nouns. The difference is invisible in the switch log, invisible in the tests, and the entire reason the guardrail passed at exactly the moment it should have fired.

Once the gate compares articles rather than epochs, every pair in the corpus is blocked. Not one clean comparison exists in four months of data. And the 1,460-word gap in evening reads is indistinguishable from whatever else changed between May and mid-June — which, given a weekly prompt-patching agent, is not nothing.

Two of my own unit tests had also been passing vacuously: they built rows that all shared a single timestamp, so nothing in them could alternate in either direction. A fixture that cannot exhibit the behaviour under test will pass whatever you do to the code.

What the instrument was actually for

The honest output of four months of data and a week of tooling is: I cannot answer the question, and now I know precisely why.

Switching models when I feel like it does not produce randomization. It produces blocks — a month of this, a month of that — and blocks are the one design guaranteed to entangle the variable you care about with every other thing that drifts. The dataset was never a natural experiment. It only looked like one because the switch count was high, and a high switch count told me nothing about whether the switches were positioned usefully.

So the instrument's real product is a specification for the experiment I should have been running all along: assign the model per piece, not per month, on a deterministic hash the way my prompt A/B harness already assigns variants. Do that for six weeks and the same tooling — the same join, the same strata, the same gates — returns real answers instead of refusals. I have also started stamping the writing model directly onto every critic trace, so the next version of this analysis is a lookup rather than an inference over intervals.

This is the evaluation claim again, one level further up than I've had to admit before. Last time the broken instrument was an LLM judge picking winners out of noise. This time the broken instrument was the validity check I built specifically to protect me from believing a bad comparison — and it failed in the shape these systems always fail in, by returning a confident pass.

A guardrail that measures the wrong noun doesn't fail loudly when it matters. It fails silently, in your favour, at exactly the moment you most want to be told yes. Mine cleared a comparison between a month of May and a month of June and called it a study of quantization. The only reason I'm not publishing that result is that I asked the data one question the tool never thought to ask: not which model wrote this, but when.