My LLM judge was flipping a coin a third of the time

I run an autonomous newsroom on a DGX Spark. It drafts stories, and somewhere in the middle of that pipeline sits a language model whose only job is to look at two drafts and say which one is better.

I trusted that judge for months. It produced verdicts with reasons attached. The reasons were plausible. I shipped changes based on them.

Then I ran the control I should have run on day one: I gave it two drafts generated from an identical configuration — same prompt, same model, same settings, same story, differing only by sampling noise — and asked it to pick a winner.

It picked one. Confidently, with reasons. About a third of the time.

What that actually means

A judge choosing between two genuinely identical-in-expectation inputs should abstain, or split evenly with no confidence. Mine did neither. It manufactured a preference and justified it, on a third of trials, out of nothing.

That number is a noise floor. It is the amount of signal the instrument invents on its own. And it retroactively poisoned everything I had measured with it. A change that "won" 55% of judged comparisons had not beaten anything — it had cleared a bar my instrument was already clearing by accident, on noise, a third of the time.

I did not have a set of wrong results. I had a set of unreadable ones. That distinction mattered, because wrong results can be corrected and unreadable ones have to be re-run.

The fix was not a better prompt

My first instinct was that the judge needed better instructions. Sharper rubric, explicit tie-option, a demand for abstention when the drafts were too close to call.

That helped a little and missed the point entirely. The problem was never that the judge lacked guidance. The problem was that I had no idea how much of its output was noise, because I had never measured it.

What actually worked:

Measure the floor before trusting the verdict. Run the judge on pairs you know are equivalent. Whatever win rate it produces on those is your error bar. It is not a bug to be fixed so much as a property to be known.

Run replicates. A single verdict on a stochastic system is an anecdote. I now run the same comparison many times and look at the distribution, not the outcome. This is the single largest change to how I work, and it is only affordable because the models run locally — when inference is not billed per token, thirty replicates costs patience instead of money.

Refuse to ship inside the error bar. If a change wins by less than the instrument's own noise, the honest reading is "no measurable effect," not "small improvement." I have killed several changes I liked on this rule.

The same mistake wearing different clothes

Once I started looking for this pattern, I found it repeatedly.

I had a metric that reported a 46% collision rate pooled across all days, which sounded alarming. The per-day median was 0%. Both numbers came from the same data. Pooling across days with wildly different sample sizes let the busiest days dominate the total, and the pooled figure described a day that never existed.

I had a fix that read as a 12% improvement measured per pair and a 70% improvement measured per bet. Neither was wrong. They answered different questions, and I had not decided which question I was asking before I ran the numbers. If your denominator varies per subject, that variation is a dimension of the experiment, not a footnote in the write-up.

And I had a constrained-decoding setup where switching on a GBNF grammar took a task from 121.9 seconds to 1.4 seconds on identical input. That looked like a spectacular optimisation until I understood the mechanism: the grammar forces the first token, so the model never enters its reasoning phase at all. Both outputs passed schema validation. Only one of them had thought about the question. A schema guard constrains shape, never content — the JSON stayed green while the answer quietly got worse.

Why I think this generalises

None of this is really about newsrooms, and none of it is exotic. The pattern is:

The measurement felt authoritative, so I never measured the measurement.

LLM-as-judge is now standard practice. It sits inside eval harnesses, RAG pipelines, agent routing, content moderation, and the leaderboards people cite at each other. In most of the setups I have seen — including my own, for months — nobody has established what the judge does when handed two things that are actually the same.

It costs one afternoon to find out. Generate pairs from an identical configuration. Run the judge across them. Read the win rate. That number is the smallest effect you are entitled to claim, and everything you have shipped below it deserves another look.

Mine was 33%. I would genuinely like to know what yours is.