All writingAntvia · 6 min read

Bronze, silver and gold are a naming convention, not a design

Three schemas called bronze, silver and gold are not an architecture. A layer earns its place by holding a guarantee you can test, and some estates need two.

Apache IcebergdbtDatabricksSnowflakeorchestrationData engineeringManufacturingRetailFintech

Open the catalogue on most warehouses built in the last few years and you will find three schemas called bronze, silver and gold. Ask four people on the team what has to be true of a table before it is allowed into silver, and you will get four answers. One says it is where the joins happen. One says it is where the data is clean. One says it is where dbt models live, as opposed to ingestion. The fourth says silver is whatever feeds gold, which is correct and tells you nothing.

The diagram travelled further than the reasoning behind it. It is a good diagram. It came from teams with a specific problem, which was that raw ingested files and business-facing tables were being written into the same place, and nobody could tell whether a number had been touched on the way through. Splitting them apart fixed that. But the fix was a separation of guarantees, and what got copied was a separation of names.

A layer is only real if you can finish this sentence: everything in here is true of every table inside it, and is not yet true of anything upstream. That sentence is the design. If you cannot finish it for silver, you do not have a silver layer. You have a schema called silver, and a second copy of your data.

The guarantee is the boundary

Take bronze first, because it is the one people get closest to right by accident. If bronze means what the source sent us, byte for byte, with arrival time attached, never edited, then it has a guarantee you can check and a job only it can do. You can rebuild every downstream table without going back to the source system, which matters on the day a modelling bug is found and the API you loaded from only retains ninety days. You can also say what the source claimed on a Tuesday in March when finance disputes a figure. That is worth the storage on its own, and it is nothing to do with the word bronze.

Silver is where the copying hurts most. Written properly, its guarantee is something like: one row per business entity at a stated grain, keys conformed across source systems, duplicates resolved by a rule that is written down. Almost all of the difficult work in a data platform is inside that sentence. Deciding that customer 4471 in the billing system and CUST-4471 in the CRM are the same customer is not a transformation step. It is a decision somebody has to own, defend and revise. A layer whose guarantee is "cleaner" has quietly skipped the part that was hard.

Gold is the easiest to state and the easiest to over-build. A gold table exists because a person, a dashboard or a model asks one specific question repeatedly and answering it from scratch is expensive. That is the whole test. If nobody asks the question, the table is not gold, it is clutter that still has to be backfilled every time the grain changes upstream.

A layer you can only describe by naming the tools inside it is not a layer.

The pattern, stated plainly

Some estates need four, plenty need two

Once the guarantees are written rather than assumed, the number of layers stops being three by default. Regulated work is the clearest case for four. In reconciliation and payments there are records that fail validation and must not be silently dropped, because somebody has to adjudicate them and there is a reporting obligation attached. That is a distinct guarantee: arrived, failed a named rule, retained with the rule it failed, awaiting a human decision. It cannot live in bronze, because bronze applies no judgement. It cannot live in silver, because silver promises conformed and resolved. So it earns its own place, with its own tests and its own owner.

Process manufacturing gets there by a different route. Sensor history at one reading per second per tag is the record you keep for traceability and incident investigation. Almost every question anyone asks is at one minute or one shift. A downsampled layer with a stated aggregation, including how gaps and out-of-range readings are treated, is a real guarantee that sits between raw and conformed. Skipping it means every analyst quietly invents their own averaging, and two dashboards disagree about the same furnace.

Three estates drawn as vertical stacks. A reconciliation and payments estate has four layers, with quarantine sitting between bronze and silver. A process manufacturing estate also has four, with a downsampled layer in the same position. A single-source estate has two, raw and modelled, with a dashed empty box where a middle layer would be.
Each of these is the right shape for its estate. None of them is three because three was the diagram.

The two-layer case is more common than anyone admits. One source system, twenty tables, one team, no second system to conform against. Adding silver here means writing a copy of each table with renamed columns and a cast, then maintaining an extra lineage hop that promises nothing. The honest architecture is raw and modelled. It will be criticised in a design review for not looking like the diagram, and it will be easier to change for the next three years.

Table format matters here too, and it changes the arithmetic. If you are on Iceberg or an equivalent with snapshot history and time travel, part of what bronze was doing, namely proving what the data looked like before the last run, is provided by the format. Retention on those snapshots is a configuration decision, not an eternal fact, so check it before you rely on it. But it does mean the replay and audit argument for a separate raw layer is weaker than it was five years ago, and worth re-deciding rather than inheriting.

Two stacked diagrams compared. The upper one shows three identical boxes labelled bronze, silver and gold with an arrow between each and no other text. The lower one shows the same boxes with a written guarantee under each, one crossed out and removed, and a fourth box added to the side labelled quarantine with its own guarantee.
The layers are the same shape either way. The difference is whether anything is written underneath them.

Write the guarantee before you write the model

  1. 01
    State each layer in one sentenceSay what is true of every table in it. If two people in the room write different sentences for the same layer, you have found the actual problem and it is not technical.
  2. 02
    Turn each sentence into testsUniqueness on the stated grain, not-null on business keys, referential checks across conformed entities, freshness against the source. Tests at the boundary, not scattered through the models.
  3. 03
    Delete any layer whose sentence you could not writeMerging two layers is a one-off cost paid now. Carrying a layer that promises nothing is a cost paid at every schema change, forever.
  4. 04
    Add a layer only when a guarantee has nowhere to liveQuarantine, downsampling and slowly changing history are the three that usually qualify. Convenience is not a guarantee.
The order matters. Guarantees first, then models, then the folder names.

How to tell your layers are only names

  • The row counts match Same grain, same count, and the diff between two layers is renamed columns and a date cast. That is a view wearing a costume.
  • Models reach past a layer A gold model reading straight from bronze because silver did not have what it needed. This is the most honest signal you will get, and it means the middle layer failed its own promise.
  • Nothing breaks when bad data arrives If a duplicate can pass through the deduplication layer without a test failing, the deduplication is a comment, not a property.
  • New tables need a meeting If somebody has to ask which layer a table belongs in, the guarantees are not written down. When they are, the table sorts itself in about ten seconds.
  • Backfills go back to the source If a rebuild means re-pulling from the API rather than reprocessing what you already hold, your raw layer is storage rather than a guarantee.

Where we get this wrong too

We have built four-layer estates that should have been two, and the reason is not ignorance. A layer is cheap to add and awkward to remove, an extra hop looks like diligence when you are presenting a design, and "we will collapse it later" is a sentence nobody has ever acted on. The correction we now use is small and slightly rude: make somebody say the guarantee out loud in the review. If the best available answer is that things are tidier there, the layer goes. Antvia ships with the three-layer default because defaults save arguments on day one, and the layer names in it are still just folders until tests are written against them.

Keep the vocabulary. Bronze, silver and gold are useful in the way that any shared shorthand is useful, and there is no gain in renaming them to something clever and local. Just do not let the shorthand do the design. Before the next platform build, or the next migration off a warehouse that has grown three layers of ceremony, write the sentences. You will usually find that one layer is doing two jobs, one is doing none, and the argument you have been avoiding is about entity resolution rather than about tooling.

Not sure your layers are earning their place

If you have a warehouse with three schemas and a growing suspicion that two of them do the same thing, we can go through it with you: what each layer actually guarantees, which tests would prove it, and which hops can be collapsed without losing the audit trail. It is a short piece of work and it usually pays for itself in build time alone.