All writingAI evaluation · 6 min read

An agent has to be graded on the steps it took, not only the answer

A correct answer can hide a wrong tool call, a guessed parameter and a lucky input. Grading the steps costs more, and it is the part that predicts next month.

ClaudeevalsorchestrationRAGPostgresAI evaluationApplications and automationSaaSFintechProfessional services

An agent is asked how much was refunded last month. It returns a number. Somebody checks the number against the ledger, it matches, the row in the eval sheet goes green. Then you open the transcript. The agent first called the orders tool rather than the refunds tool, got back a payload it could not use, guessed a date format, had the query rejected, guessed again, and finally ran a filter wide enough to have swept in cancellations as well as refunds. The number was right because that particular month happened to contain no cancellations.

That agent is not working. It is passing. The difference matters because next month will contain a cancellation, the same sequence of steps will run, and the answer will be wrong in a way nobody notices, because the only check anyone built was on the output and the output still looks like a number in the right shape.

Grading the trajectory means keeping every tool call the agent made, with the arguments as they were actually sent and the result as it actually came back, and then scoring whether each of those calls was warranted given what the agent knew at that moment. It is considerably more work than scoring the final response. It is also the only version of the exercise that says anything about behaviour on inputs you have not seen.

An output score measures your test set as much as your agent

With tool access, the space of paths from question to answer is large, and a good number of those paths land on the correct answer for the wrong reason. Wide queries that happen to contain nothing extra. A cached value that was still fresh. A retry that succeeded because a rate limit cleared, not because the argument improved. Output-only scoring cannot separate those from a clean run, so what you end up measuring is how forgiving your test inputs were.

The second problem shows up when the answer is wrong. An output score tells you that something failed and nothing about where. Was the retrieval bad, or was the retrieval fine and the model ignored what came back? Did the agent call the right tool with a malformed filter, or the wrong tool competently? Those have completely different fixes, and one of them is not a prompt change. Without the steps, you are left rerunning the case by hand and reading tea leaves, which is the slowest debugging loop available to anyone.

A pass you cannot explain is a pass you cannot rely on.
The pattern, stated plainly

Record first, score later

Instrumentation comes before any rubric. If the trace is incomplete, no amount of clever scoring will recover what happened, and you will discover the gap on the day something goes badly wrong in production. Log the trajectory in full from the start, even before you know what you want to ask of it.

The call as sent The arguments after templating and coercion, not the intention the model described in its reasoning. The gap between the two is where a large share of failures live.
The result as received The payload the tool actually returned, and whether it was truncated before reaching the model. A model reasoning over the first two kilobytes of a result is working from a different input than one that saw all of it.
Errors and retries With the error text, kept in order. An agent that succeeded on the fourth attempt is a different agent from one that succeeded on the first, and the output score for both is identical.
Step index and parent run So a trajectory can be reassembled in order, including calls the orchestrator issued in parallel.
Model and prompt versions A trajectory that cannot be attributed to a specific prompt revision cannot be compared with last week's, which removes most of the reason for collecting it.

Two agent runs side by side, both ending in the same green tick marked with an identical final answer. The left run is three clean calls. The right run has a wrong tool call, a rejected argument, a retry, and a query drawn wider than the question required, with those three steps marked in a warning colour. — Identical outputs, different agents. Only one of them will still be correct when the data changes.

What a step-level rubric actually asks

A useful rubric is short and it is applied per step, not per run. Five questions cover most of it, and they are worth writing down explicitly because reviewers otherwise drift towards judging the run as a whole, which is just output scoring again in slower clothing.

01
Was this step necessaryGiven the state at that point, did the task require another call at all, or had the agent already gathered enough to answer? Unnecessary calls cost money and, where tools write, cause damage.
02
Was the right tool chosenAmong the tools available and described, was this the appropriate one? Persistent wrong choices are usually a tool description problem rather than a model problem.
03
Were the arguments correct and completeFilters, date boundaries, tenant or account scoping, pagination. This is where a guessed parameter that happens to work quietly enters the trace.
04
Was the result used faithfullyDid the claim the agent went on to make follow from what the tool returned? For retrieval steps, this is the question of whether the cited chunk actually supports the sentence, not whether it is topically related.
05
Was stopping correctAgents fail at the end as well as the middle, either by answering while a needed fact is still missing, or by continuing to call tools long after they had what they needed.

The rubric is applied to each step in the trajectory, then rolled up. A run with one unwarranted step is not a clean run.

Not all of that needs a language model to judge it. Push as much as possible into deterministic assertions over the recorded trace, because they are cheap, they do not drift, and they can run on every request rather than on a sample. Assert that no query ran without a tenant filter. Assert that the same tool was never called twice with identical arguments. Assert that any figure appearing in the final answer also appears in some tool result. A trace table in Postgres with the arguments and results in a jsonb column carries this a long way before anything specialised is warranted. Reserve model-graded scoring for the genuinely judgement-shaped questions, and have a person read a sample of trajectories every week regardless, because the failure modes you have not thought to assert on are visible to a human in about ninety seconds.

Three rows comparing grading layers. Deterministic assertions over the recorded trace run on every request, are cheap and do not drift, and catch a query that ran with no tenant filter, the same tool called twice with the same arguments, and a figure in the answer with no tool result behind it. Model-graded scoring runs on the judgement calls, costs more as steps rise, and catches whether the cited chunk supports the sentence rather than merely sitting on the same topic. A person reading trajectories takes about ninety seconds each on a weekly sample and catches the failure modes nobody thought to assert on. — The weekly read is not a formality. It is where the next assertion comes from, which is why the cheapest layer keeps growing.

Where this is the wrong thing to build

If the agent has two tools and no branching, step grading is overhead and the output score is nearly the whole story. The case for trajectory grading strengthens with the width of the search space, with tools that write rather than read, and with the cost of a confident wrong answer reaching somebody who will act on it. A read-only assistant that drafts a summary for a human to check does not need this. An agent that issues credits does.

There is a sharper caveat. Step grading scores adherence to a path that somebody decided was correct, which means an error in that judgement gets enforced at scale and with great consistency. Where several routes are genuinely valid, a rubric that expects one of them will penalise good runs and train the team to fix the agent towards a preference rather than towards correctness. Score constraints rather than an exact reference path, and when the author of the rubric is also the author of the agent, get somebody else to write it. Marking your own homework produces very good marks.

The honest cost

Full trajectories are large, and traces contain whatever your customers' data contains, which makes retention, redaction and access control part of the build rather than an afterthought. Judge calls add cost proportional to steps rather than runs. Building the rubric takes real argument among people who each thought the correct behaviour was obvious. None of that is hidden work, and all of it is smaller than the cost of finding out from a customer that the number has been wrong since February.

The first week this is switched on is usually unflattering. Agents that had been reporting healthy pass rates turn out to be reaching correct answers through paths nobody would sign off on if they saw them written out. That week is the point of the exercise. Everything after it is maintenance.