Jobs that finish. Migrations that reconcile. Storage that stops paying for cold data.
This is the platform discipline underneath everything else, not analytics preparation. We tune pipelines and backend jobs for runtime and cost together, take systems across field by field with reconciliation that proves what landed matches what left, lay out storage for how the data is actually read, and leave lineage your auditors can follow. Reporting is one consumer of a good foundation, never the reason to build one.
- Companies delivered for
- 20+
- Median to first production-grade artefact
- 6 weeks
- Median recovered or saved, verified by customer Finance
- $240k
- Week 1 audit
- Fixed fee, two calls, one-pager kept either way
- Pune, delivering across
- APAC, India and US
Where this work sits, and what runs on top of it
Every layer below is somebody's operational problem before it is anybody's reporting problem, and the last box is the one most vendors start with.
- 01
Source systems and their exceptions
The CRM, the ERP, the point of sale, the spreadsheet a regional manager has quietly maintained for years. Each one allows things its own documentation says it does not. We find those rules before anything moves.
- 02
Ingestion and backend jobs
Batch and streaming jobs written to finish inside the window they are given and to stop costing more every month. Runtime and spend are the same design decision, so they get profiled together.
- 03
Storage and access design
Partitioning, file sizing and hot, warm, cold tiering with lifecycle rules, chosen for how the data is read rather than for the order it happened to arrive in.
- 04
Contracts, quality and lineage
Schema contracts at the boundaries, quality checks that fail a run rather than warn in a log nobody reads, ownership per dataset, and lineage that answers where a figure came from.
- 05
Whatever runs on top
Operational apps, the month end close, machine learning features, agents, and yes reporting. Reporting is one consumer of this foundation and usually not the one paying for it.
What we have actually delivered
The lineage discipline is the part that travels, and it is why a figure survives contact with an auditor.
Where the number ends up
The point of the pipeline discipline is that a figure survives the room it is presented in.
The symptoms that start most of these conversations
None of these arrive described as a data problem; they arrive as an operations problem with a deadline attached.
The nightly window has run out
The batch used to be done before the working day started. Now it is not, and the first part of every morning goes on asking whether yesterday actually loaded.
The migration is months past its date
The connectors were the easy part. What stalled it were the fields nobody could explain and the records the old system quietly allowed against its own rules.
The bill grows faster than the data
Storage is one line and compute is another, and both usually trace back to the same mistake. Everything hot, everything scanned, nothing tiered, nothing compacted.
Nobody can say where the number came from
Two teams quote different figures for the same measure, and the disagreement gets settled by seniority rather than by evidence.
The tool does not fit the shape of the data
The off-the-shelf modelling layer has an opinion your data does not match, so the same long query gets rewritten by hand every week and nobody dares change it.
Each symptom belongs to a layer
The complaint arrives from operations, and the fix always sits one layer down in the platform.
- The nightly window has run outIngestion and backend jobs
- The migration is months past its dateSource systems and their exceptions
- The bill grows faster than the dataStorage and access design
- Nobody can say where the number came fromContracts, quality and lineage
- The tool does not fit the shape of the dataQuery and domain layers
What a cutover has to prove before we call it done
A migration is finished not when the data lands but when somebody in Finance or Operations agrees the new system is the one they will use tomorrow.
Counts reconcile by entity and by period
Every object, every period, source against target, with the differences listed individually rather than summarised into a percentage.
Money ties back to the system of record
Balances, invoices and credits reconcile to the finance system itself, not to a convenient intermediate extract.
Dropped fields are named in advance
Anything not carried across is written down with the reason and the person who agreed to it, before cutover rather than after somebody misses it.
Exceptions get a decided destination
The records the old system permitted against its own rules are modelled, not filtered out. These are the ones that stall a cutover after everything else is ready.
The old system stays readable for a while
A planned period with both systems queryable, so a disputed record can be checked instead of argued about.
Reruns are safe
Loads are written to be repeatable, so a failed step is run again rather than patched by hand at two in the morning.
The mapping is the project, not the connectors
Tooling moves rows; what takes the time is agreeing what each field means on the other side, and naming the decision for the ones with no honest home.
- account_ownerOwner (user ID)
- billing_countryBilling country (ISO)
- close_dateClose date (UTC)
- stage_custom_7Pipeline stage
- notes_freetextActivity note
- discount_flag_legacyNot carried across, documented
Two decisions we make on your workloads, not on preference
Both get decided by profiling your own systems first, because the answer differs by shop and a general rule of thumb is worth very little here.
Laid out the way the data arrived
Most platforms are laid out the way the data arrived rather than the way it is read. Queries scan far more than they need, and data nobody has touched in a long time sits on the most expensive tier available.
Unglamorous and measurable
Partition on the columns people actually filter by, compact small files so the engine stops paying per file, and set lifecycle rules that move data from hot to warm to cold automatically.
The packaged tool is normally right
Buying the packaged modelling tool is usually right, and we will say so. It stops being right when your data has a shape the tool has no opinion about.
When the tool has no opinion
Then the same logic ends up pasted into several places and the team cannot safely change any of them. We build a small documented query layer over the data you have, versioned and handed to your team to own.
Why six weeks is possible: we do not start from an empty repository
Two things shorten this work: scaffolding we have already run in production, and Claude doing the reading and drafting that used to eat the first fortnight.
- Already built
Scaffolding we bring with us
Ingestion patterns, quality check harnesses, reconciliation runners and lineage capture that have already run in production elsewhere. Proven in other engagements, then fitted to your systems rather than written again from scratch.
- AI first pass
Reading what nobody documented
Claude reads legacy schemas, stored procedures and the exception handling nobody wrote down, then drafts the candidate field mapping and flags what it could not explain. A person confirms every decision, and the flags are where we start asking questions.
- AI first pass
Generating the checks, not just the code
Reconciliation and quality tests are drafted from the mapping itself, so the thing that proves the migration is built alongside the migration rather than after it, when nobody has budget left.
- Human call
The decisions stay with the engineer
Which exception is a rule and which is a bug, what gets left behind, what a threshold should be. AI does not make those calls here, and we will not pretend otherwise on a call.
- What you keep
Yours to run without us
Documented, versioned and handed over, including the prompts and harnesses. We are not renting you a black box you have to call us to change.
APAC grocery: the lineage came before the agent
APAC retail grocery, over $500M revenue, six weeks: the agent that raised the margin flag belongs to our AI work, and what belongs here is the lineage and the checks that made the flag worth trusting.
Where day 14 falls in a six-week build
The first margin flag on the APAC grocery engagement landed at day 14 of the six weeks to the first production-grade artefact.
Questions platform and operations teams ask us
Is AI writing our system, then?
No. AI does first passes and we review every one. The decisions that make a system trustworthy, the mapping calls, the thresholds, the rules a regulator would ask about, are made by the people whose names are on the work.
So what does the AI actually take away?
The transcription: reading a thousand columns, drafting the obvious, generating the harness that checks it. That is where the weeks go, and it is the part nobody enjoys or does consistently well by hand.
We already have engineers. Why bring in a pod?
Because your engineers are usually the only people who know the exceptions, and they are already fully committed. We take the mapping, the job rewrites and the reconciliation, working alongside them rather than around them.
How quickly does anything actually work?
A pod of three runs the Ship phase from Week 2, and something works by Week 3, which is early enough for your team to disagree with us while disagreeing is still cheap.
Our data is a mess. Should we clean it up first?
Nobody has ever finished that clean-up before starting. The Week 1 audit exists for this state: two calls, a fixed fee, and a one-pager naming what is broken, what it costs you and the order to fix it in. You keep it either way.
Will this be another platform we cannot maintain?
The test we hold ourselves to is whether your team can change it without us in the room. If we have built something only we can operate, we have failed at the part that matters.
What happens after the build?
Quarterly reviews from Week 7 keep it honest, jobs, contracts and lineage are documented and handed over, and on-call governance is a service you can stop at any point.
Can you migrate without freezing the business?
Cutovers run with the old system readable while the new one takes writes, and reconciliation attached at each step. Freeze periods are short and planned. The long, painful freezes almost always come from having no reconciliation, so nobody will commit to the switch.
How do we know what landed matches what left?
You get counts by entity and period, money tied back to the system of record, and a named list of everything deliberately dropped. On the APAC grocery engagement Finance signed within 30 days, because they could follow the lineage themselves.
Is this really just preparation for dashboards?
No. The jobs that reconcile your ledgers, the systems your operations team types into all day, and any agent you run later sit on this foundation. If a dashboard is the only reason, we will tell you to buy off the shelf instead.
The rest of the practice
These are genuinely different jobs with different ways of failing. Most engagements start in one of them.
- Data integration and governanceOne set of records the finance team and the operations team both accept.
- Data platforms and modernisationOff the platform you outgrew, without a twelve-month freeze on new reporting.
- Apache SupersetSuperset built and run by people who commit to the project, including embedding and Kubernetes.
- Command centresThe one screen an operations floor runs the day from, not another dashboard.
- AI audit and roadmapOne week, and you know which two or three AI initiatives are worth building.
- AI evaluationA graded eval set, because an AI system cannot simply pass or fail a test suite.
- AI governanceGovernance you can defend in a board meeting and audit on demand.
- AI agentsOne reasoning agent on Claude, on your live data, with evidence and a kill switch.
- Data agentsAgents that watch the data, flag what moved, and explain what is driving it.
- AI use case, guaranteedOne production agent in eight weeks, or you do not pay for the build.
- Applications and automationThe system your team works in all day, built or replaced in slices.
- Application modernisationThe system nobody wants to touch, replaced a slice at a time rather than rewritten.
- System integrationSystems that stop disagreeing about the same customer, order and item.
- Process digitisationThe process that still runs on paper, WhatsApp and one shared spreadsheet.
Start with the Week 1 audit
Two calls, a fixed fee, and a one-pager naming what is broken in your platform, what it is costing you and the order to fix it in. You keep it either way. NDA-friendly, fixed scope. Write to hello@woodfrog.tech.