Earning trust in the numbers

Situation

Everything else a data team builds assumes something that is rarely true: that when the platform produces a number, people believe it.

Trust is the actual product of a data function. Not dashboards, not pipelines, not models: the willingness of somebody making a decision to use your number instead of their own. And it behaves in a way that is deeply counterintuitive to engineers, because it is not proportional to correctness.

The trust problem I was hired into was not a wrong number anyone had found. It was a disagreement everyone knew about. Two departments each reported total revenue, the two had never matched, and every ratio built on them disagreed in turn. Executives had made their peace with it: the numbers would not match, but as long as they were close, that was fine. The analysts lived in their departments, were hired by them, and wrote their own definitions, with no code review, no governance and no owner for the data. Nobody had to discover the problem. Everybody had learned to work around it.

My first message was short. Yes, this is a problem, and a known issue is not something to live with and move on from. Get the owners of the business data in a room, agree on the definitions, and put the business logic inside the governed model, so that no analyst can come up with their own version of a metric and everyone speaks the same language. Building a data function from zero is what that turned into.

Tolerated inconsistency is the slow way to lose trust.

The fast way is a step function. People trust the platform, then there is one confidently wrong number in a high-visibility setting, then they check everything by hand, for a long time. The drop is abrupt and triggered by a single event, and later correctness alone does little to speed the recovery. What speeds it is how the incident is handled, which the last part of the Decision below covers.

The asymmetry is the thing to internalize. A pipeline that fails loudly costs an afternoon: you fix it, you backfill, and nobody outside the team notices. A pipeline that silently produces a plausible wrong number costs a decision, and then costs the credibility of every number the platform has ever produced. The second failure is cheaper to prevent and vastly more expensive to survive, and engineering instinct optimizes for the first, because the first is the one that pages you.

What makes this hard is that the failure is usually invisible from inside the team. Nobody reports a number that looks reasonable. It gets used, a decision gets made, and the discovery comes weeks later during a reconciliation. By then the damage is not one wrong figure but a general suspicion that has no specific target and therefore cannot be specifically answered.

Constraint

• You cannot test everything, and pretending otherwise produces alert fatigue. A platform of any size has thousands of assertions worth making. Making all of them produces a channel nobody reads, which is functionally the same as making none of them.
• The people who need to trust the numbers are not the people who can evaluate the tests. An executive cannot assess your coverage. They assess whether the number was right the last time it mattered.
• Quality work is invisible when it succeeds. There is no recognition for the incident that did not happen, which keeps this chronically under-resourced relative to its importance.
• Upstream systems change without telling you. Most of the quality failures I have dealt with started outside the data platform: an application migration, a vendor schema change, a well-intentioned fix to a source system.

The first constraint shapes the design. Quality is not a coverage problem to be maximized. It is a prioritization problem, and treating it as coverage is how teams end up with ten thousand tests and no trust.

Decision

Prioritize by blast radius, not by coverage. The question is not “what could be wrong” but “what would be catastrophic if it were wrong, and who would find out, and how.”

Start with the numbers your most skeptical stakeholder uses

Identify the handful of metrics that appear in the decisions that matter, such as the board pack, the weekly leadership review and the figures finance defends externally, and make those bulletproof before touching anything else. Include operational decisions with real consequences, even when no executive ever sees them. Not the most technically interesting pipelines. The most politically expensive ones.

That is an unnatural priority for an engineering team, because those pipelines are often boring and the interesting reliability problems are elsewhere. It is also the only ordering that protects the step.

Distinguish the four failure modes and test for each

Most teams test for correctness and stop. The failures that actually damage trust are usually the other three:

• Freshness. The number is correct and eighteen hours stale, and the dashboard does not say so. The most common trust failure in practice, and the easiest to instrument with a freshness target.
• Completeness. Some records arrived and some did not. Partial success is far more dangerous than total failure, because nothing looks broken. Row-count changes against historical ranges catch most of it, and source-to-target checks catch what a stable total hides.
• Correctness. The classic case: a join that duplicates, a filter that drops, a definition that drifted from what the business means. Uniqueness, not-null, relationship and accepted-value tests catch the structural defects. Known business examples and reconciliation against an authoritative source test the meaning.
• Consistency. Two systems disagree about the same fact, and both are internally valid. This is the one that surfaces in meetings rather than in monitoring. Owned definitions prevent most of it, and reconciliation checks between authoritative sources catch the rest.

Make freshness and lineage visible to consumers

The single most effective thing available, and it is not a test: put the as-of time on the dashboard, and make lineage traceable without asking anyone.

A stale number that announces its staleness is not a trust failure. It is information. A stale number that looks current is a trust failure. In the same way, when somebody questions a figure, the ability to trace it to source in minutes turns a credibility incident into a technical conversation. Without lineage, every question becomes a referendum on the team.

Instrument the sources you do not control

Since so many failures start upstream, monitoring belongs at the boundary: schema change detection on ingestion, volume anomaly detection against expected ranges, and explicit contract checks on the handful of fields that matter most. Finding out from your own monitoring that an application team changed a column is a normal Tuesday. Finding out from an executive is an incident.

Run incidents like incidents

This is where trust is won or lost, and most data teams handle it badly, by treating a wrong number as an engineering bug rather than a communications event.

What works:

1. Announce it before it is found. Proactive disclosure of a wrong number costs a fraction of what discovery costs. It is genuinely difficult, and it is the whole thing.
2. State the blast radius plainly. What was wrong, since when, and which decisions may have used it. Vagueness here reads as concealment.
3. Say what changes so it does not recur, and then do it. A test, a source contract or a repaired review step, with an owner. A postmortem that changes nothing is theater, and people notice.
4. Do not over-apologize. Executives deal in operational failure all the time. A clear, unemotional account of what happened and what changed lands far better than contrition, which reads as a team that is rattled.

A team that finds its own failures and reports them promptly accumulates trust even while having incidents. A team that is repeatedly caught loses trust even while having fewer. Disclosure does not make an unreliable service trustworthy, though: repeated incidents still need prevention. And once a wrong number has reached a decision, repairing the data does not finish the response. Following the correction into that decision is its own work.

The tradeoff I accepted

I chose deep coverage on a narrow set of critical numbers over broad coverage across the platform, and the long tail genuinely suffered.

The consequence is real and I would not soften it: less-critical datasets ran with thin testing, and some of them were wrong for a while before anyone noticed. Analysts working in those areas occasionally found problems the platform should have caught. That is a cost paid by real people doing real work, and it is a direct result of my prioritization.

I accepted it because the alternative, uniform moderate coverage everywhere, is worse in the specific way that matters. Uniform coverage gives the board metric the same protection as an exploratory dataset, which makes the failure that destroys organizational trust exactly as likely as the failure nobody notices. Trust does not average out across the platform. It is set by the worst high-visibility failure. Cheap baseline checks can still run everywhere. The deep validation goes where the consequences are.

Other costs:

• Alert tuning is permanent, unglamorous work. Thresholds drift, seasonality breaks naive anomaly detection, and an alerting channel degrades into noise within a quarter without active maintenance. Nobody wants this job, and it has to be somebody’s.
• Some of the discipline reads as bureaucracy. Requiring tests on a model someone wants to ship today is a real tax on speed, and the people paying it never experience the incident it prevented.
• Proactive disclosure is culturally expensive at first. The first few times a team announces its own error, it feels like volunteering for criticism. It only pays off once there are enough instances for people to recognize the pattern, and that takes longer than it feels like it should.

Outcome

What I can point to is an operating foundation: named data ownership, shared metric definitions, lineage, a catalog and observability, alongside review and deployment standards. That foundation carried both the reporting and, later, natural-language self-service.

The signal I judge it by is not test count or incident count. It is who finds the problem first. A platform where monitoring finds most failures before consumers do is trusted almost regardless of how many failures there are. A platform where an executive finds them in a meeting is not trusted regardless of how few.

What good looks like:

• Failures are found by monitoring, and the team communicates them before anyone asks.
• Questions about a number are answered by tracing lineage in minutes, not by launching an investigation.
• Stale data is visibly labeled as stale, so nobody is misled by a number that was merely old.
• The conversation shifts from “is this right” to “what does this mean.”

Uber’s data platform team wrote in 2021 that its consolidated quality platform caught around nine in ten incidents across the two thousand critical datasets it covered, by combining checks with incident handling and by putting quality information into the tools people consumed data in. What I take from it is that the response workflow mattered as much as the tests.

What I’d do differently

Instrument freshness on day one, before building any transformation. It is the cheapest signal available, it catches the most common trust failure, and I have consistently built it later than I should have, because it feels like monitoring rather than progress.

Define the critical metric set with stakeholders rather than for them. I inferred which numbers mattered most from usage. Asking directly, “which figures would embarrass you if they were wrong,” produces a different and better list, and the act of asking does its own work on the relationship.

Write the incident communication template before the incident. Composing a clear disclosure while an incident is live, under pressure, produces worse communication than a template drafted calmly. It takes twenty minutes on a quiet afternoon, and it is worth far more than that.

Resource the alert maintenance explicitly. I treated tuning as something that would happen alongside other work. It does not. It decays, the channel fills with noise, people stop reading it, and the monitoring becomes decorative several months before anyone notices.