Rebuilding ingestion people trust

Situation

Analytics pipelines decay quietly. Nothing breaks loudly, freshness slips, sampling creeps in, a schema change upstream silently drops a field, and one day you notice the numbers people are making decisions on have not been right for a while. The rebuild question is rarely “is this broken.” It is “can we still justify trusting it.”

Different sources decay in different ways, and the rebuild is shaped by which kind you are dealing with.

Vendor platforms you do not control decay fastest, because the export changes on the vendor’s schedule rather than yours. The shift from session-based to event-based web analytics is the clearest recent example: the schema changes shape, the metric definitions underneath change meaning, and pipelines built against the old assumptions keep running and keep producing numbers. They just stop producing the numbers anyone thinks they are producing.

Operational databases decay differently. The schema is under someone’s control, just not yours: an application team ships a migration, a column changes meaning, and nothing in the pipeline notices because the type did not change. Nightly batch extracts also stop being adequate long before anyone says so; the business starts making intraday decisions on figures that are eighteen hours old and nobody flags it because the dashboard never said how fresh it was.

Third-party APIs decay through rate limits, silent pagination changes, and backfill semantics that differ from what the documentation claims. The failure is usually partial, some records arrive, some do not, which is considerably worse than an outage because it is invisible.

Most real rebuilds involve several of these at once, and the interesting constraint is that they do not agree with each other. The same event is timestamped at capture in one system and at processing in another; the same customer has different identifiers with different lifetimes; one source is immutable and another is updated in place with no change history.

The specific decay pattern worth naming: a pipeline that fails loudly gets fixed. A pipeline that silently drops three percent of events accumulates a growing, invisible error, and the longer it runs the more historical analysis is quietly built on top of it. By the time someone notices, the question is not just “what do we fix” but “how far back do we not trust.”

Constraint

The constraints that typically apply:

• The export model of the source platform. You get what the vendor exports, in the shape the vendor exports it, with whatever sampling and retention limits the tier imposes. Working around those limits is most of the engineering.
• Existing reports cannot break mid-quarter. Anything feeding a target, forecast or board pack has to keep producing comparable numbers throughout.
• Budget for running two pipelines at once. Parallel run is not free, and the cost is visible on a bill in a way that “reduced trust in the numbers” is not.
• Team capacity. The rebuild competes with the ongoing operation of the thing being rebuilt.

The third is the one that gets negotiated down, and it should not be. It is the cheapest part of the project relative to what it protects.

Decision

Two decisions matter: how the data gets in, and how you prove it is right. The second is the one people skip.

Choosing the integration approach

The single most consequential architectural choice is where transformation happens, and the answer that has held up across every rebuild I have done is: land raw, transform in the warehouse, define business logic in exactly one layer.

Transforming during ingestion looks efficient and is a trap. It destroys the ability to reprocess history when a definition changes, and definitions always change. Once the raw form is discarded, a definitional correction means re-extracting from a source that may no longer hold the history, and sometimes that is simply not possible. Storage is cheap; irreversibility is not.

On tooling, the honest position is that this decision is driven by source constraints rather than by preference, and it changes per source:

• Managed connectors are correct for well-supported vendor sources. What you are buying is not the extraction, which is a few days of work. It is somebody else maintaining it when the vendor changes the API without notice. The failure mode is a long tail of sources nobody supports well, where you pay platform cost and still write the connector.
• Change data capture is correct for operational databases where freshness matters and load on the source does not. It is genuinely harder to operate than it looks: schema evolution, replication lag during backfill, and out-of-order delivery all have to be handled explicitly rather than discovered.
• Batch extraction remains correct more often than its reputation suggests. If the business genuinely makes decisions daily, a well-instrumented nightly batch is cheaper to run, far easier to reason about, and much easier to reprocess. Streaming a source nobody reads intraday is cost with no corresponding benefit.
• Writing it yourself is correct when the source is strange enough that no connector fits, and the mistake is choosing it by default because building is more interesting than evaluating.

The question worth asking about any of these is not “which is best” but “what happens when this breaks at 3am, and who fixes it.” A team of three should be buying far more of this than a team of thirty.

Reconciling sources that disagree

Where multiple sources describe the same entity, the reconciliation rules belong in the design rather than being discovered during the parallel run. In practice that means deciding explicitly, and writing down: which source is authoritative for each field, which timestamp is canonical and what the others are called, how identity is resolved when identifiers differ, and what happens to a record that arrives late enough to change a figure somebody has already reported.

That last one is the one that gets skipped, and it is the one that produces the “the number changed since yesterday” conversation that erodes trust faster than being wrong once.

Proving it

The decision most worth explaining is the validation approach.

Rather than cutting over and fixing discrepancies as they surfaced in production, both pipelines ran side by side for a defined window with a daily variance report. Nobody was asked to switch until each remaining difference was explained, not merely small. “Within two percent” is not an explanation. “Two percent because the old pipeline double-counted a session type” is.

That distinction is the whole method. An unexplained two percent variance is not a small problem; it is an unknown problem that happens to be small today. The mechanism producing it could produce twenty percent next month under different traffic conditions. Explaining it converts an unknown into a known, and knowns can be accepted deliberately.

What the daily variance report should contain:

• Row counts and key metric totals, both pipelines, side by side, per day
• Absolute and percentage difference per metric
• A written explanation field per open variance, with an owner
• A visible count of unexplained variances, trending toward zero

That last line is the one that does the work. It gives stakeholders a single number that means “how close are we,” and it removes the data team from the position of repeatedly asserting readiness. The report asserts it instead.

The tradeoff I accepted

Parallel run costs real money and real attention for its whole duration. You pay twice for ingestion and you spend analyst time chasing variances that sometimes turn out to be nothing. A four to six week window is typical for a pipeline of any complexity, and it needs to span at least one full monthly cycle, month-end processing surfaces variance that ordinary weekdays never will.

I took that cost because the alternative failure mode is worse and effectively irreversible: cut over, surface a wrong number in front of an executive, and the new pipeline carries a credibility deficit for a year regardless of how correct it subsequently becomes. Trust is far cheaper to preserve than to rebuild.

The honest cost beyond the invoice: parallel run creates a period where two numbers exist for the same thing, and people will find both. Someone will quote the new pipeline in a meeting before it is blessed, someone else will quote the old one, and the resulting confusion lands on the data team. This is manageable with clear labelling and a single stated source of record for the duration, but it is real friction and it is worth warning stakeholders about at the start rather than explaining it in the moment.

The second cost is subtler: a long parallel window tempts you to keep extending it. There is always one more variance. At some point the remaining differences are genuinely immaterial and the right move is to write down why, accept them, and cut over. A parallel run with no exit criterion is not caution. It is indecision with a budget line.

Outcome

What good looks like at cutover:

• Every remaining variance has a written explanation and a named owner
• The unexplained count has been zero for long enough to include a month-end
• Freshness improves and becomes measurable rather than assumed
• The old pipeline is switched off on a scheduled date, not left running “just in case”, which is how you end up maintaining two pipelines permanently

The strongest signal is behavioural: people stop asking whether the number is right and start arguing about what it means. That shift is the entire point of the exercise.

What I’d do differently

Define the exit criterion before starting the parallel run. Writing down “cutover when unexplained variance is zero across a full monthly cycle” at the start is trivial. Deciding it halfway through, with a stakeholder asking why it is taking so long, is a negotiation you will lose.

Publish the variance report to stakeholders from day one, not when it looks good. The instinct is to fix the embarrassing gaps privately first. That instinct costs you the credibility the report is meant to build. A report that starts ugly and visibly improves is far more convincing than one that appears fully green, and it makes the eventual cutover a formality rather than a request for trust.

Instrument the new pipeline for the next rebuild. Every pipeline gets replaced eventually. Building it so the next team can run a parallel comparison against it (stable keys, reprocessable history, no transformation during ingestion) costs very little at the time and is enormously valuable later. The pipeline I would design today is one that assumes it will be audited by its successor.