Your evals pass and users still say the thing is wrong

A team ships an assistant that answers customer questions about orders. Before launch they build an eval set: a modest set of questions written by the product owner and two support leads, each with an agreed correct answer. The suite passes almost everything. Two weeks after launch, support tickets start arriving that say the assistant is wrong, and nobody can reproduce the failures against the eval set, because the eval set does not contain anything like what people are actually typing.

This is the most common shape of AI evaluation failure we see, and it is not a modelling problem. It is a sampling problem. Those questions encode what three people in a room imagined users would ask. Real users ask about a refund on an order placed under a different email address, or paste in a screenshot of a delivery notice, or ask a question in Hindi with English product names embedded in it. None of that was in the room.

The fix is unglamorous. You need a route from live traffic back into the frozen set, run by a person who knows the domain, at a cadence that survives contact with a normal working week. Most of the difficulty is organisational rather than technical, which is why it is usually skipped.

The eval set has a shelf life and nobody puts a date on it

An offline eval set is a snapshot of your assumptions at one moment. It decays for two independent reasons. First, users change: they discover what the system is good at and push into the edges of it, and the distribution of questions moves. Second, the world changes: you add a product line, a policy changes, a regulation comes into force, and answers that were correct in March are wrong in September without a single line of code changing.

The second one is worse than it sounds because your suite will still pass. The expected answers were written when they were true. A green suite in that state is not evidence of quality, it is evidence that you are grading against a stale answer key. If you have an eval set that has not changed since launch and a product that has, your score is measuring last quarter's problem.

A green suite that has not changed since launch is not evidence of quality. It is evidence that you are grading against a stale answer key.

The pattern, stated plainly

Sampling: what to look at when you cannot look at everything

At any real volume, reviewing every interaction is out of the question, and reviewing a uniform random sample is close to useless. Uniform random sampling spends nearly all of a reviewer's attention on the easy majority. If the large majority of your traffic is handled well, a random sample buys you a handful of interesting cases and a lot of confirmation that ordinary questions get ordinary answers.

Stratify instead. Keep a small uniform stream so you retain an unbiased read on the overall rate, then spend most of the budget on strata where failures concentrate.

• Explicit negative signal Thumbs down, a retry within the same session, an escalation to a human, a session abandoned mid-answer. These are the cheapest failure detectors you will ever build and most teams do not log them.
• Model-side uncertainty Refusals, hedged answers, responses where retrieval returned nothing or returned documents with low similarity. A RAG system answering confidently from an empty context is a specific and very findable bug.
• Distributional outliers Inputs unlike anything in the eval set. Embed incoming queries, cluster them, and flag clusters with no near neighbour in your frozen set. New clusters are new user behaviour, and they arrive before the complaints do.
• High-consequence paths Anything touching money, medical guidance, eligibility or a legal statement gets sampled at a much higher rate than its traffic share, regardless of how well it appears to be doing.
• Uniform baseline A small unweighted slice, so you can still answer 'what fraction of all traffic is wrong' without the stratification poisoning the estimate.

The reviewer is the constraint, not the pipeline

The pipeline is the easy part. A queue table in Postgres, a sampler that writes rows into it, a thin interface over the top. The hard part is that the only person who can say whether an answer about a refund policy is correct is someone who knows the refund policy, and that person already has a full-time job.

Everything about the review tool should therefore be designed around one number: seconds per item. A review interface that takes minutes per item goes unused, however carefully it was built. An ugly one that takes seconds gets used daily.

• One item per screen The question, the answer, the retrieved sources, and nothing else. No dashboard, no filters, no aggregate charts on the same page.
• A default verdict of three options Correct, wrong, unclear. Not a one-to-five scale. Scales invite the middle and produce labels that cannot be turned into a test.
• Structure the reason, not the prose A short fixed list of failure modes: wrong fact, right fact wrongly applied, missing context, retrieval found nothing, tone, policy breach. Free text as an optional extra. Prose reasons are unaggregatable and reviewers stop writing them within a few weeks.
• Keyboard only Verdict on one key, reason on another, next item automatically. Every mouse trip is seconds you will not get back across a long queue.
• A visible finish line A queue sized to empty in one sitting gets cleared daily. An open-ended queue gets abandoned quickly.

Promotion is a judgement, not an automatic step

A confirmed failure is not yet an eval case. To promote it you have to write down the expected answer, and that is real work: someone must decide what the system should have said, in enough detail to grade against. This is the step teams try to automate and should not. If you cannot state the right answer, you have not understood the failure, and a test case built on a vague expectation will produce arguments later about whether it passed.

Promote the class, not the instance. A user who asked about a refund on an order placed under a different email has exposed an identity-resolution gap. Add several cases across that class, not the single verbatim question, otherwise you are teaching to one string and the next phrasing of the same gap sails through.

Your score is supposed to get worse

The most damaging incentive in production evaluation is the one that treats the eval score as a performance metric. If a team is judged on the number going up, the rational move is to stop adding hard cases, and the suite quietly ossifies into a set of questions the system is known to answer. Everyone is happy and the number is fiction.

Report two things instead. Pass rate on the eval set as it stood at the last release, which is comparable over time and tells you whether a change broke something. And separately, the number of new failure classes found this cycle, which measures whether you are still looking. A month with zero new classes on a live product is not a triumph, it is usually a signal that nobody cleared the queue.

Where the engineering effort actually goes

Almost none of it goes into the grading. It goes into logging: capturing the full interaction including retrieved context, tool calls, model version and prompt version, in a form you can query months later. A failure you cannot reconstruct is a complaint, not a finding. Teams that instrument this at build time can answer 'what changed' in an afternoon. Teams that did not end up guessing, and the guess is usually the model, and the model is usually not what changed.

The rest goes into wiring the frozen set into the release path, so that no prompt edit, retrieval change or model upgrade ships without running it, and so a regression is visible before a user finds it. That is ordinary orchestration work and it is the part that turns a folder of test cases into something that protects you.

None of this makes an assistant good. It makes the gap between what you believe and what users experience small enough to act on, which is the only version of quality that survives contact with production.