The model changed underneath you and nothing in your release notes says so
Behaviour drifts without a deploy, and the first person to notice is a customer. What it actually takes to make provider-side change detectable.
The incident has a recognisable shape. A feature that has behaved the same way for months starts producing something slightly different. Summaries get longer. A classifier sends more records to the catch-all bucket. An extraction step that always returned a date now sometimes returns a sentence containing a date. Nobody deployed anything. The prompt file has not been edited since winter. The retrieval index is the same index. Git blame is clean all the way down.
The first instinct is to look for a change in the input, because that is where changes usually come from. Somebody adds a new document type, or a client starts sending a different file format, and the pipeline meets something it has not met before. That is worth checking, and it is often the answer. When it is not the answer, the search gets uncomfortable, because the remaining explanation is that the part of the system you do not own moved while you were not looking.
Providers change things. They have to. Models are retrained and reissued, serving stacks are optimised, safety layers are tuned, defaults are revised, older snapshots are retired on a published schedule. Almost none of that arrives as a line in your own release notes, because it did not happen in your repository. It happened upstream, and the only trace of it in your system is a behavioural difference that somebody has to notice.
Pinning helps, and it is weaker than it sounds
The standard advice is to pin the model version, and it is good advice. Calling a dated snapshot rather than a floating alias means the weights behind your calls stay put, and that removes the single largest source of silent change. Do it. But treat it as one control rather than the whole answer, because a pin covers less ground than teams assume.
Pins expire. Snapshots are retired on announced timelines, and when yours goes, you are migrating whether or not it suits your quarter. Pins also cover the model and not the system around it: moderation and safety layers, request routing, serving-side optimisations, rate-limit behaviour under load, and the handling of very long inputs can all be adjusted independently of the weights. And a surprising share of drift is not upstream at all. An SDK minor version changes a default. A framework updates the template it wraps around your messages. Somebody bumps a library that quietly normalises whitespace differently, and your carefully tuned few-shot block is no longer character-identical to the one you tested.
A version pin is a promise about weights. It is not a promise about behaviour.
The distinction that matters in production
The real failure is that you could not tell
It is tempting to frame this as a stability problem and to go looking for a configuration that stops the world from moving. That configuration does not exist, and chasing it produces brittle systems that fall over the day a migration is forced on them. The useful frame is narrower. Change is going to happen. The failure is not that it happened, it is that a customer found out before you did, and that when they told you, nobody in the room could say what had changed or when.
That reframing is helpful because detectability is a much easier engineering target than immutability. You do not need to prevent the upstream world from moving. You need a signal that fires within hours, and a record that lets somebody attribute the shift to a specific date and a specific change. Both are achievable with unglamorous machinery you already know how to build.
A small suite, running on a schedule, against the live path
The mechanism is a frozen set of examples, scored the same way every time, run automatically against the same code path production uses. Not a benchmark, not a leaderboard, not a research artefact. A regression suite for a component whose implementation you do not own.
- 01Freeze a set from your own trafficA couple of hundred real inputs, sampled to cover the boring middle and the awkward edges, labelled once by somebody who knows the domain. Real traffic matters more than volume. Synthetic cases test the model, production cases test your system.
- 02Keep it out of prompt iterationThe moment the suite is used to tune prompts, it stops measuring drift and starts measuring how well you fitted the suite. Hold a slice back and never look at it during development.
- 03Score deterministically where the task allowsExact match, schema validation, numeric tolerance, set overlap. A judge model is itself a model that can change underneath you, so use one only where the task genuinely has no checkable answer, and pin and monitor the judge as carefully as the system it grades.
- 04Run it on a schedule, not on mergeMerge-triggered runs only fire when you change something, which is exactly the case this is not. Put it on an orchestrator with a daily or hourly cadence, alongside the rest of your pipelines, so a run happens on the days nobody touches the code.
- 05Alert on shape, not only on a headline scoreAverage accuracy moves slowly and hides a lot. Output length distribution, refusal rate, schema-failure rate, latency and token counts, and per-category shares will move first and move visibly.
None of this needs new infrastructure. If you already run scheduled jobs in something like Dagster, the suite is another asset with a sensor on it. Results go into Postgres next to everything else, one row per run per case, and the query that answers the question is a comparison between today's distribution and last week's. The engineering is genuinely ordinary. The discipline of keeping the labelled set honest is the hard part.
The record is half the value
When behaviour moves, the first question anybody asks is what else moved. That question is answerable only if you were already writing things down, so record the context alongside every run and, ideally, alongside every production call you sample.
- The resolved model identifier Not the alias you asked for. What the provider says actually served the request, taken from the response rather than your own configuration.
- Sampling parameters as sent Including the ones you did not set, resolved to their effective values, because defaults are exactly the thing that changes.
- A hash of the fully assembled prompt After templating, after retrieval, after any middleware. Two systems can share a prompt file and send different bytes.
- Library and SDK versions The client, the framework, anything that sits between your string and the wire.
- Provider notices, in the same place Deprecation announcements and changelog entries pasted into the same log as your deploys, so one timeline holds both. This is a five-minute manual habit that repays itself the first time it is needed.
With that record, an unexplained shift becomes a lookup. Behaviour changed on the fourteenth, the suite caught it on the fourteenth, the only thing that changed on the fourteenth was the resolved snapshot, and here is the diff in refusal rate by category. You can then make an engineering decision instead of an argument. Without the record, the same investigation is three people re-reading a prompt they wrote in January and disagreeing about whether it always did that.
When this is more machinery than the problem deserves
Not every AI feature needs a scheduled suite, and pretending otherwise is how governance work gets a reputation for being paperwork. If a human reads and edits every output before it goes anywhere, the human is the monitor, and they will notice a change faster than a daily job will. If the feature is genuinely exploratory and nobody has committed to what correct looks like, building the harness first produces a number that nobody trusts and everybody argues with. Write down what correct means, in the domain's own language, before you build anything to measure it.
The case for the machinery is strongest where the output feeds something automated, where wrongness is quiet rather than loud, and where a regulator, an auditor or a customer might one day ask you to account for a decision made eight months ago. That covers most of what gets built in lending, in claims, in clinical operations, and in any product where the AI step writes to a system of record rather than to a screen a person is watching.
The asymmetry is what settles it. The cost of the suite is a week of somebody's attention and then a small, boring line item forever. The cost of finding out from a customer is a week of panic, an apology, an unknown number of decisions that have to be reviewed by hand, and a conversation about controls that you will have to have anyway, only now with an incident attached to it.