All writingAI evaluation · 6 min read

An LLM judge is a measuring instrument and it needs calibrating

Model-graded evaluation scales the way human review never will, which is exactly why nobody checks whether the grader is reading true.

evalsClaudeRAGAI evaluationGovernanceHealthcareFintechSaaS

A team we sat with had one number they trusted more than any other in their AI programme: the pass rate of their support-drafting assistant, graded by a second model against a rubric they had written themselves. It ran nightly over the full case volume. It went into the board pack. Nobody had read the underlying cases in months.

When we pulled a sample and read them by hand, the headline pass rate held up, which sounds like good news. It was not, because the disagreements were not randomly scattered. The judge was passing almost everything that was fluent, confident and well-structured, and it was failing a small set of terse answers that happened to be correct. Its errors in one direction cancelled its errors in the other. The headline number was right and the instrument underneath it was broken.

That is the thing worth internalising about model-graded evaluation. It is a measuring instrument. Instruments drift, have systematic bias, and read plausibly wrong long before they read obviously wrong. Nobody would run a factory on a pressure gauge they had never compared against a reference. Plenty of teams run an AI roadmap on a judge they have never compared against a human.

Why the judge got adopted without a calibration step

Human review does not scale, and everyone knows it. Reading model outputs carefully eats a specialist's day, and the specialist you want is the one with the least free time. So a model-graded harness is not a shortcut anyone should apologise for. It is the only way most teams get evaluation coverage at all, and running one is better than running none. The problem is the shape of the adoption. Human review is slow and visibly effortful, so its cost forces a conversation about what is worth checking. A judge costs a rounding error next to a specialist's afternoon and returns while you wait, so no conversation happens. The harness gets built during a sprint, produces a number, and the number starts appearing in decks. There is never a natural moment where someone asks whether the instrument reads true, because nothing about running it feels expensive enough to warrant the question.

Human review forces a conversation about what is worth checking. A judge is cheap enough that the conversation never happens.
The pattern, stated plainly

The failure modes you should expect to find

These are well documented in the research literature and easy to reproduce on your own harness in an afternoon. They are not exotic. If your judge has never been checked, assume some of them are live.

Position effects In pairwise comparison, the option presented first is not treated identically to the option presented second. Swap the order on the same pair and some of your verdicts flip. Any pairwise harness that does not run both orders and check for consistency is measuring position as well as quality.
Verbosity and fluency effects Longer, more confidently structured answers tend to score better, even when the shorter answer is more correct. This is the one that bites hardest in support, sales and clinical-summary use cases, because the failure it hides is precisely the failure that matters: a confident wrong answer.
Self-preference A judge is not a neutral observer of text produced by a model family that includes itself. If the generator and the judge share lineage, the comparison you are running is not clean, and you should at minimum know that and say so.
Rubric drift into style Rubrics written in a hurry ask for things like clear, helpful and professional. Those are style properties, and a language model is extremely good at grading style. Correctness, groundedness and refusal-appropriateness are harder to specify and so they quietly get less weight than the rubric author intended.
Silent model upgrades The judge model is a dependency. When it is upgraded underneath you, whether by a version bump you made or a default that moved, your entire time series changes units. A jump in scores that looks like a product improvement can be nothing more than a new grader with slightly different taste.

The working practice, which is unglamorous

There is no clever fix here. The practice that works is the one everybody can describe and almost nobody maintains: hold a human-labelled subset, measure the judge against it, and re-measure whenever anything upstream changes.

01
Label a golden set by handOne or two hundred cases, labelled by someone who actually owns the outcome, not by whoever is free. Oversample the hard and ambiguous cases deliberately. A golden set drawn uniformly from production traffic will be mostly easy cases and will tell you very little.
02
Write the rubric against real disagreementsHave two people label the same set of cases independently before you write the rubric. Wherever the two humans disagree, you have found either an ambiguous case or an underspecified criterion. Resolve those in the rubric text. A rubric that two humans cannot apply consistently will not be applied consistently by a model either.
03
Measure agreement, not just the scoreRun the judge over the golden set and look at the confusion matrix. Where does it pass things a human failed, and where does it fail things a human passed. Those two cells have completely different consequences and should be tracked separately.
04
Fix the instrument, then re-measureMost agreement problems are rubric problems, not model problems. Add the failing patterns as explicit criteria with examples. Randomise pairwise order. Ask for a short justification before the verdict rather than after it, so the reasoning constrains the grade rather than rationalising it.
05
Re-check on every changeNew judge model, new rubric version, new prompt template, new product surface. Each is a recalibration trigger. Version the rubric alongside the code and record which rubric version and judge version produced every stored score, or your historical numbers are not comparable to each other.

The loop is deliberately boring. It is the maintenance, not the design, that people drop.

The last step is where this usually falls apart. Teams do a careful calibration once, at build time, and treat it as a certificate rather than a maintenance schedule. Months and several model versions later, the certificate refers to an instrument that no longer exists. Re-checking agreement quarterly, or on every judge change, costs a sliver of one person's time next to the harness it protects. It is the cheapest insurance in the whole AI stack, and it is nearly always the first thing cut.

A two-by-two grid of judge verdict against human label. The two agreeing cells are quiet. The cell where the judge passed what a human failed is marked missed defects, fluent and confident and wrong, the expensive cell. The cell where the judge failed what a human passed is marked false alarms, terse answers that were correct, the noisy cell. Alongside it, the four changes that force a re-measurement: a new judge model, a new rubric version, a new prompt template, a new product surface. — The single agreement figure hides the only two cells anyone should care about, and the two cells cost you very different things.

Where a judge is the wrong tool entirely

Candour, including about our own approach: there are cases where the honest answer is that model-graded evaluation should not be the primary check, and building one anyway is worse than building nothing, because it manufactures confidence. If the correct answer is deterministic and checkable, check it. A generated SQL query can be run against a fixture database and its result compared to an expected result. That is a test, it is exact, and asking a model whether the SQL looks right instead is a downgrade dressed as sophistication. The same holds for extraction against a known schema, arithmetic, and anything with a canonical form. If the consequence of an error is clinical or financial harm to an identifiable person, a judge is a triage layer, not an approval gate. Use it to route cases to human reviewers and to catch regressions between releases. Do not use it as the thing that decides an output is safe to send, and be honest in writing about which of those two roles it is playing, because the difference tends to blur under delivery pressure. And if you have no human labels at all, you do not have an evaluation. You have a second model's opinion, which may be a fine opinion, but it is unvalidated and should be reported as such rather than as a metric.

Three rows pairing a case with the judge's role in it. When the answer is checkable, run it against a fixture and compare the result, rather than asking for an opinion on whether it looks right. When an error reaches a named person, use the judge to triage and to catch regressions, not as the gate that says an output is safe. When there are no human labels, report the result as a second model's opinion rather than as a metric. — In all three the judge can still be useful. What changes is which decision it is permitted to close.

What to do this week

If you are running a model-graded harness today and cannot immediately say when its agreement with human labels was last measured, that is the finding. You do not need a project to address it. Pull a small batch of cases from last week's runs, weighted towards the ones the judge scored near its decision boundary, and have the person who owns the outcome label them blind. Compare. Look at the two off-diagonal cells. One of three things happens. The judge agrees, and you have earned the right to keep quoting the number, with a date attached. It disagrees in a systematic direction, which is the useful outcome, because systematic disagreement is almost always a rubric fix rather than a rebuild. Or it disagrees in no discernible pattern, which means the task is genuinely ambiguous and the honest next move is to narrow what you are asking the judge to decide. All three are better than the position most teams are in, which is holding a number with no idea whether it points at anything.