Skip to content
woodfrog
Data Foundations
Data productsUse case, no client result yet

Stop pipelines breaking every time a source changes.

Someone upstream renames a column, adds a status value or upgrades an API. Nobody tells the data team. The load fails overnight, or worse, it runs and fills your reports with wrong figures.

A source change, caught at the gateSchematic
3.3×Target return in 12 months, on your own numbers
Week 1Fixed-fee audit
6 weeksTo the first live result
3Senior people on the work
TypeUse case: what we build, not a client story
ForTeams whose reports, models or customer-facing data products depend on systems they do not control
EngagementWeek 1 audit, then build in about six weeks, fixed scope, fixed fee
Runs onYour existing pipelines and warehouse, or Antvia in your own cloud
Before and after

From firefighting to changes caught at the gate.

Before
  • Source changes arrive without warning
  • Loads fail at night, or pass with wrong data
  • Breaks found by a user looking at a report
  • Each fix is a one-off patch by whoever is on call
After
  • A written contract for every source the pipeline reads
  • Checks on arrival that hold a bad load before it lands
  • The owner of the source told first, with the exact field
  • Reports keep the last good data while the fix is made
The problem

Your pipeline depends on systems you do not control.

Every pipeline reads from somewhere else: an ERP, a CRM, a vendor feed, another team's database. Those systems change for their own reasons. Without an agreement on what each source sends, the first person to learn about a change is whoever opens the report the next morning.

Rows of server cabinets with cables
Every source is someone else's system
What was broken
  • No written agreement on the fields, types and values each source sends.
  • Nothing checks data on arrival, so a change flows straight into the warehouse.
  • Silent breaks: a renamed field loads as empty, a new value falls out of a filter.
  • Nobody can say which reports a given source field feeds, so every fix is a guess.
  • Pipeline code changes go live without tests.
You might be here if
  • Your data team starts most days fixing last night's loads.
  • A report went blank and it took hours to find out why.
  • You learn about source changes from users, not from the source team.
  • People stopped trusting a dashboard after one bad week.

Why it mattered. Each break costs engineering time and a little more trust. Trust lost is slow to win back, and people return to their own spreadsheets.

The approach

Agree what each source sends, then check it at the door.

We start with your last few breaks, not with a tool. For each one we trace what changed upstream, how far it travelled and who found it. Then we write a contract for every source the pipeline reads: the fields, the types, the allowed values and how fresh it must be. The source team signs it off. A check on arrival enforces it, every run.

An engineer reading pipeline code on two monitors
Fixed once, at the source, not every morning
Sources you do not control
ERP and CRM
Vendor and partner feeds
Other teams' databases
woodfrog method
Trace recent breaksWhat changed, who found it
Write a contract per sourceFields, types, values, freshness
Gate every runHold, quarantine or pass
What changes
Changes caught on arrival
Source owner told first
Last good data kept
Pipelines that hold
Deliberately not doneFreezing the source systems. They should change. The pipeline should notice.
Source teams sign off contracts. Claude helps draft them from real data.schematic
The solution

Contracts at the source, gates in the pipeline.

What we build
  • A contract for each source: fields, types, allowed values, keys and how fresh the data must be, kept in your repository and signed off by the source team.
  • Schema checks on arrival that spot a renamed, dropped or retyped field before anything loads.
  • Quality gates between raw, cleaned and reporting layers: bad rows go to quarantine, a bad load is held, and reports keep the last good data.
  • Alerts that go to the owner of the source with the exact field and change, not to a shared inbox.
  • Field-level lineage, so you can see which tables and reports a source field feeds before you change it.
  • Tests that run on every change to pipeline code, so your own changes do not break things either.
It fits the pipelines and warehouse you already run, or Antvia, our lakehouse with bronze, silver and gold layers in your own cloud. Contracts and checks live in your repositories as code. If your team cannot change them without us, we have not finished the job.
How it would runWe start with the sources that broke most often, so the first gates land where they save the most time. Operate is open-ended and drawn at one week only to fit.
The impact

What we would measure.

Where a source change gets caughtThe later it is found, the more it costs to fix.
Schematic, no values
Before: found by a person looking at a blank or wrong report. After: found by a check, before anything loads.
What we would measureNo client values yet
Pipeline failures caused by source changesPer month, before and after
Breaks found by a check, not by a userShare of all breaks
Time from a source change to a fixFrom discovery to a passing load
Sources with a signed contractOut of all sources the pipeline reads
Return on the engagementTarget: at least 3.3× within 12 months, measured on your own numbers
Let source changes stop at the gate, not in your reports.A short call, or record a message. A person replies within one business day.
Contact usCertified Services Partner, Claude Partner Network