A supplier changed a column and your pipeline found out in production

The Monday morning report showed a plant running at zero output for the whole of Friday. It had not been. What had happened was that the ERP vendor shipped a minor release over the weekend, and in that release a field called PROD_QTY became PRODUCTION_QTY. The extract job carried on happily, because the extract job was selecting everything the source offered and writing it wherever it landed. The column simply arrived under a new name, nothing was mapped to it downstream, and every model that summed the old name summed nothing at all.

Nobody was negligent here. The vendor did not think of it as a breaking change, because from where they sat it was a rename in their own product. The team ingesting it had no way to know it was coming, because they were not on any list that gets told. The pipeline did not fail, because nothing about it was written to fail. The first person to notice was a plant manager who knew his own numbers well enough to distrust the dashboard.

This is the ordinary shape of schema drift when the producer sits outside your organisation. A supplier's CSV export, an ERP you licence rather than build, a partner's SFTP drop, a third-party API with a versioning policy that exists mostly in spirit. You depend on a structure you do not own, cannot review before it changes, and often cannot get a warning about. Most of the published advice on this problem assumes away exactly the thing that makes it hard.

Data contracts need a producer who can be made to care

The standard answer is a data contract: a machine-readable declaration of the schema, the types, the nullability, the freshness expectation, checked in CI, breaking the producer's build if they violate it. Where that works, it works very well. It moves the cost of a change onto the team making the change, which is where it belongs, and it turns a silent semantic break into a loud red pipeline in someone else's repository.

The load-bearing part of that sentence is "someone else's repository". A contract is only a contract if the other party is bound by it. Inside one company, that binding is usually organisational: the platform team can insist, the CTO can adjudicate, and a producing team that ignores a contract will eventually be made to stop ignoring it. None of that is available to you when the producer is a vendor for whom you are one account among many, or a supplier who sends you a spreadsheet because that is what their system does, or a logistics partner whose IT contact is a shared mailbox.

A contract with no enforcement mechanism and no named owner is not a contract, it is a wish written in YAML.

The pattern, stated plainly

We have seen contract documents accumulate in repositories for exactly this reason. Someone reads about the practice, writes the schema out carefully, and puts the file next to the pipeline. It is accurate on the day it is written. By the time anyone opens it again the source has moved on, and because nothing ever checked the file against reality, nobody knows it is stale. It has become documentation of the past, wearing the costume of a control.

What you actually control is the boundary

When you cannot govern the producer, govern the point of arrival. The useful mental model is that data crosses a border into your system, and borders have checks. The design goal is not to prevent drift, which you cannot do, but to guarantee that drift is discovered at the border rather than in a board pack.

That means the landing zone should be deliberately dumb and deliberately faithful. Land what arrived, as it arrived, with the ingestion timestamp and the source file or response identifier attached. Do not correct it, do not coerce it, do not drop unexpected columns on the way in. A tool like Airbyte will happily do schema evolution for you, and it is genuinely convenient, but understand what you have chosen: automatic evolution means the shape of your warehouse tables now changes without a human decision. That is fine for the raw layer. It is not fine anywhere a business definition lives.

Fail before the model layer, not after it

The most expensive mistake is not missing the drift. It is detecting the drift after the model layer has already run and published. A dbt project that tests its outputs will tell you that a metric looks wrong, which is true and useful, but by then the models have materialised and anything reading them downstream has read a wrong number. Testing on the way in and testing on the way out are different jobs, and only one of them is preventative.

So put the structural assertions on the typed boundary table and make them blocking. Required columns present. Primary key unique and not null. Foreign keys resolving against the dimensions you join to. Enumerated fields containing only values you have seen before, which is the one cheap check that catches a slice of semantic drift, because a new status code shows up as an unrecognised member of a set. Then let the models run only if that gate passes. Stale-but-consistent data usually costs a business less than fresh-and-wrong data, and that trade is worth stating out loud to the people who will be waiting for the refresh.

• Contract-as-artifact Generate the expected schema from the pipeline code rather than maintaining it by hand. A definition that is a by-product of the thing it describes cannot drift away from it.
• Row-level quarantine Records that fail parsing go to a rejects table with the reason attached, not to the floor. The rejects table is the first place anyone looks during an incident, and it wants to be queryable in Postgres or wherever the rest of your operational state lives.
• Lineage as a severity function You need to answer "who consumes this column" in seconds to grade an alert correctly. Without that, every drift event is either an emergency or ignored, and teams converge on ignoring.
• A vendor register For each external source: who sends it, which human at their end, what their release notification looks like if any, and when it last changed. Unglamorous, and it is the artefact that turns an alert into a phone call.

Where this advice is wrong

If you have one source, it changes rarely, and one analyst owns the whole path from ingest to report, building all of this is a poor use of their time. The analyst will notice. The honest version of the recommendation is that boundary machinery earns its cost somewhere around the point where no single person can hold the full path in their head, or where a wrong number reaches someone who will act on it before anyone can catch it.

And where you do have real influence over the producer, use it, because it is strictly better. A supplier who agrees to a stable export format and a change notice has removed the problem rather than instrumented it. That influence is often commercial rather than technical: it lives in the contract renewal, not in the pipeline. It is worth asking the procurement team whether anyone has ever tried, because in our experience the answer is frequently that nobody thought to ask.

What is never right is the middle position, where a schema document exists, no process compares it to what arrived, no alert has an owner, and the organisation believes it has data contracts. That belief is worse than having nothing, because it stops people from looking. The plant that reported zero output had a schema document. It described the field as PROD_QTY, and it still does.

A reasonable first week

Pick the external source that would cause the most damage if it went quietly wrong, and instrument only that one. Record its observed schema on every run. Add the parse step and the rejects table. Write down the human at the vendor. Then wait for the first alert and see whether the person it reached could actually do anything about it, because that is the part everyone gets wrong on the first attempt. Broaden to the second source once the first one has caught something real.