Giving the models away

Situation

This is usually filed as a tooling decision, and it is not one. It is an org-design decision about where authority sits, and the tooling is what makes the chosen arrangement survivable. Teams that approach it the other way around, adopting the tool and expecting the operating model to follow, get the tool and keep the bottleneck.

The bottleneck itself is common and self-inflicted: every model change routes through a central data team that is also responsible for the platform. Both jobs suffer. Analysts wait days for changes they could make themselves, and the platform work gets deferred indefinitely, because request queues are visible and technical debt is not.

The shape of the queue matters more than its length. A backlog of large, genuinely complex requests is a staffing problem. What actually accumulates is different: a long tail of small changes, each taking twenty minutes of work and sitting for a week. Add a column, adjust a filter, rename a field to match what the business now calls it. The ratio is the tell. When most of the queue is work that takes less time to do than to triage, the queue is not a capacity problem. It is a permissions problem wearing a capacity problem’s clothing.

The cost compounds in two directions. Analysts, entirely rationally, stop asking. They build the logic in their BI tool instead, or in a spreadsheet, or in a CTE they paste into every query. The warehouse stops being the place the definitions live, and the central team’s ownership becomes nominal: it owns a shrinking fraction of the logic that actually drives decisions, and it owns it carefully.

Meanwhile the platform work never happens. Nobody escalates a deferred migration the way they escalate a blocked dashboard.

Constraint

• SQL and version-control fluency varies across analysts. Some are effectively engineers. Some are fluent in SQL and have never opened a pull request. Designing for the average serves nobody.
• Production models must stay safe from well-meaning mistakes. The risk is not malice. It is a subtly wrong join in a model that feeds a board report.
• No headcount for dedicated reviewers. The review burden has to be absorbed by automation or by the contributors themselves, because there is no budget for a gatekeeping function.

The fluency spread is the hard one. It means the safety layer cannot rely on contributors knowing what they are doing. It has to catch the mistakes of someone competent in SQL and new to engineering practice.

Decision

Give the ownership away, and build the guardrails that make it survivable. The governance is the product here, not the tooling.

Tests as the entry price. Every model a domain team owns carries tests: not-null and uniqueness on keys, accepted values on anything categorical, relationship tests on foreign keys. It is unglamorous, and it is the single most effective thing in the system. A contributor who cannot yet write the model correctly can usually still state what must be true about its output, and the test catches the gap between the two. Tests need business examples too: a model can satisfy every key constraint and still calculate the wrong measure.

CI on every change, before merge. Build the modified models and everything downstream of them, run the tests, and block the merge on failure. The state:modified+ pattern, rebuilding only what changed and its descendants, keeps this fast enough that people do not learn to route around it. In my experience, a CI check that takes forty minutes is bypassed within a month.

A review model that moves with capability. During onboarding, engineers pair with analysts and review changes for engineering risk, while domain peers review the business meaning. Once a domain can own its models and support them, routine review can sit there. The central team keeps review of shared upstream models and of changes with broad consequences. Downstream owners get a say on breaking contracts. They do not need to approve every compatible change.

Naming and ownership conventions, enforced mechanically. Every model has an owner recorded in code. Layer prefixes make the dependency direction visible in the name. A linter enforces it. Conventions that rely on people remembering are conventions that decay.

Training that continues past launch. The migration is the easy part. What makes federation stick is ongoing support: office hours, worked examples, someone to ask. Teams that hand over ownership with a one-off workshop get federation on paper and a central team quietly fixing things on Fridays.

Zhamak Dehghani’s data mesh writing, first published on Martin Fowler’s site in 2019, argues for domain ownership for a related reason: data work cannot scale if every change runs through one central team. I agree with the direction. What I think the argument underweights is how much of the outcome depends on the test layer being in place first.

This is worth doing when the central queue is the constraint and the domains can own both their changes and the consequences. If a domain has one overloaded analyst and no backup, handing them production ownership makes things more fragile, not less. Support that team first.

The tradeoff I accepted

The part people implementing this tend to skip: model quality becomes less uniform. A central team produces consistent work slowly. A federated one produces more work, faster, with more variance in how it is built.

You will get models that are correct but inefficient. Duplicated logic across domains that should have been a shared upstream model and was not, because the team that needed it did not know it existed. Documentation that is excellent in one domain and absent in another. None of this is hypothetical. It is the predictable consequence of trading control for throughput, and pretending otherwise is how people get talked into federation and then resent it.

I judged the throughput worth the variance, on the condition that tests, review and clear ownership were in place first. Some of the variance still reached production. Do this without that layer and you have not federated ownership. You have distributed the ability to break things.

What it actually cost:

• Duplication got worse before it got better. Domains solved the same problem independently. Some of it needed consolidating later, and that consolidation was work nobody asked for and nobody was measured on.
• The central team’s job changed, and not everyone wanted the new one. Owning the platform, the standards and the shared layer is a different role from owning every model. It reaches further and is less immediately visible. The engineers felt closest to the transformations and believed they were theirs to handle, and some took the handover as a demotion. That deserved a straight conversation rather than a reassurance, and the straight version was about pace: with business definitions, and then the AI work, changing constantly, every small change waiting on engineering could not keep up, and the engineers’ time was worth more on the platform.
• Incidents shifted in character. Fewer bottleneck problems, more “a domain changed something upstream of another domain and nobody noticed.” Contract tests on shared models address this, and they are worth building before you need them rather than after the first incident.

Outcome

What I observed was broader participation in production model changes, and engineering effort shifting from writing each change to reviewing it. Analysts came to own the models tied to the business logic they understood, supported by tests, CI, and approval for changes that affected downstream consumers. Engineering review stayed real work. Moving more of it into a capable domain is the next step, not something the handover did by itself.

The metric that matters is not throughput. It is the number of people who can safely make a change. Raising that changes what the organization is capable of, and it lasts. Throughput is a consequence; capability is the asset.

The second-order effects:

• The engineers got time to build the platform and make it ready for AI, and the analysts stopped waiting on engineering capacity to push a small change to a model.
• Definitions moved back into the warehouse from BI tools and spreadsheets, because contributing became faster than working around.
• Test coverage became something domain teams cared about, because it was the thing standing between them and a broken dashboard with their name on it.

I did not keep a clean before-and-after measure. For a new rollout I would track the time to a supported change, the review effort, the defects that escape, and whether someone other than the first contributor can maintain the domain. More contributors is only useful if the support stays viable.

What I’d do differently

Build the test and CI layer before announcing the change, not alongside it. Gaps in that layer made the early contributions harder to support. Running them in parallel means the first wave of contributors gets the version without guardrails, and their early bad experience sets the tone for everyone watching.

Start with one domain, not a phased rollout across all of them. A single team taken all the way to genuine independence teaches you what the guardrails are missing, at a scale where fixing it is cheap. A broad phased rollout surfaces the same gaps everywhere at once.

Define the shared-layer boundary explicitly, on day one. The ambiguity about which models are “everyone’s” and which belong to a domain caused more friction than any technical decision. Write the boundary down, name an owner for everything on the central side of it, and revisit it deliberately rather than letting it drift.

Say the tradeoff out loud to leadership, in advance. “Quality will become less uniform, and we are accepting that deliberately” is a sentence that is easy to say beforehand and very hard to say for the first time while explaining an incident.