When two reviewers grade the same answer differently, the spec is the problem
Grader disagreement is not noise to be averaged away. It is the clearest evidence you have that nobody ever wrote down what a good answer looks like.
A team finishes the first review round on a retrieval-backed assistant. Fifty questions, two reviewers, a one-to-five scale for helpfulness. The sheet comes back and the two columns barely relate to each other. One reviewer's threes are the other's fives. Somebody totals the columns, notices the two means are within a tenth of each other, and reports the feature as scoring around three and a half.
Matching means are not reassurance. They are two people applying two different standards to the same work and happening to land in the same place once you sum it. Ask each reviewer separately which answer on that sheet was the worst one, and they will point at different rows. There is no shared idea of bad here, so there is no shared idea of good either, and the number in the report is describing something that does not exist.
The disagreement is the most useful thing the round produced. It is not reviewer noise, it is not a training problem, and it should not be averaged away. It is direct evidence that the specification for the feature was never written, and that whoever graded on a given afternoon was quietly writing it for you.
Nobody wrote down what good means
The system prompt says be helpful and accurate. The reviewers were told to grade it out of five. Everything that actually decides whether the product is fit to ship lives between those two sentences, and it was left implicit.
So each reviewer answers a set of questions privately, without knowing they are answering them. Is a correct answer that omits a caveat worse than a hedged answer that includes it. Is a citation to the right document but the wrong paragraph a hit or a miss. If the user asked something the system should refuse, is a polite refusal a five or a one. If the answer is right but three times longer than it needs to be, does that cost a point. Two conscientious people will resolve those differently, and both will be defensible.
Two reviewers who disagree have run an experiment on your specification and found it underdetermined.
The pattern, stated plainly
Measure agreement before you measure quality
The first review round is not a measurement of the system. Treat it as a measurement of the rubric. Nothing the reviewers say about model quality is worth acting on until they are producing the same verdict on the same item, because until then you cannot tell an improvement in the model from a change in who was rostered that week.
- 01Write the guideline firstEven a bad one. A page of rules someone can argue with beats a scale with no definition attached to it.
- 02Double-grade a small blind sampleThirty to fifty items, both reviewers, neither able to see the other's labels or the model's confidence.
- 03Meet on the disagreements onlyIgnore the items they agreed on. The agreements teach you nothing about the rubric. Twenty minutes on ten split items is the whole value of the round.
- 04Amend the guideline in the roomEvery resolution becomes a written rule with a worked example attached, not a shared understanding that lives in three people's heads.
- 05Re-grade a fresh sampleNever the same items. You are testing whether the new wording travels, not whether people remember last week's ruling.
Disagreements come in four kinds and only three are yours to fix
- Definition gaps The rubric never anticipated this case. A question with no answer in the corpus, a partially correct answer, an answer that is right for a different user than the one asking. Fixable by writing a rule.
- Evidence gaps The two reviewers were not looking at the same thing. One opened the retrieved passages and checked the claim against them, the other graded the prose on its own. Fixable by fixing the review interface and the order of checks, not by fixing the people.
- Scale drift They rank the items identically but put the boundary between three and four in different places. This is the easiest kind to fix and the most common reason a five-point scale underperforms a two- or three-point one.
- Genuine judgement calls A clinician and a compliance lead disagree about whether a hedge is appropriate, and both are right within their own frame. This is not a rubric defect. It is a business decision, it needs an owner, and the ruling needs to be recorded with the reasoning.
The fourth kind is the one worth protecting. If you drive it out by fiat you have not resolved anything, you have just hidden a live disagreement inside a number that a steering committee will later treat as settled.
What a guideline that actually works looks like
- Decisions, not descriptions Write mark this as a fail if the answer states a figure that does not appear in the retrieved passages. Do not write answers should be well grounded.
- Worked examples at the boundary Real outputs from your own system, not invented ones, sitting either side of each threshold. Boundary cases teach the rule. Obvious cases do not.
- A fixed order of checks Grounding first, then completeness, then tone. Left free, reviewers weight the dimensions differently and you get scale drift back.
- An explicit cannot judge option Otherwise reviewers guess, and guesses look identical to judgements in the data.
- A version number and a date The guideline is the thing that makes this quarter's score comparable to last quarter's. Without a version, they are not comparable and nobody will notice.
Shorter scales usually beat longer ones. Two or three levels with sharp definitions produce far more agreement than five levels with soft ones, and for questions like which of these two answers would you rather send to a customer, a forced pairwise comparison sidesteps the calibration problem almost entirely.
Where this advice is wrong
A calibration programme can become the work. Teams spend six weeks perfecting a rubric for a feature that a fortnight of user contact would have killed. If you do not yet know whether anyone wants the thing, measuring it precisely is the expensive way to find out.
The subtler failure is optimising the wrong number. It is always possible to push agreement up by narrowing the rubric until it only asks questions with mechanical answers, such as whether a citation is present and whether the format is right. Agreement climbs, and the evaluation quietly stops measuring usefulness. High agreement is a necessary condition for a trustworthy score, not a sufficient one. Two reviewers can agree perfectly and both be wrong about what the user needed.
Automate only after the humans agree
A model grader is a reviewer that scales, and the guideline becomes most of its prompt. That is exactly why the sequencing matters. Point Claude, or any capable model, at an underdetermined rubric and it will produce confident, consistent, arbitrary numbers, which is worse than inconsistent human ones because the consistency reads as rigour. Once a written guideline exists, treat the model grader as a third reviewer and measure its agreement with the humans on a held-out set you keep aside. Re-run that check whenever you change the model, the prompt or the guideline.
Keep the labels somewhere queryable, one row per item, reviewer and guideline version. Postgres is plenty. The point is not the storage, it is that six months later somebody will ask whether the assistant got better or the standard got looser, and only the version column can answer that. This is where our pre-deployment assurance work starts, and it is the part clients most often expect to skip.
The first real artefact of an evaluation programme is not a score. It is a document two people can pick up cold, apply to the same output, and reach the same conclusion with. Everything downstream, the dashboards, the regression suites, the governance reporting after launch, inherits its meaning from that document or has none of its own.