What the platform costs

Situation

There is a question that separates data leaders treated as engineering managers from those treated as owners of a business function: what does your platform cost, and what is it buying?

A surprising number cannot answer it. Not because they are careless, but because cloud data platforms make the question hard. Consumption pricing turns an architectural decision into an operating expense that arrives a month later, aggregated, with no attribution to the team, model or dashboard that caused it. The bill says the warehouse cost a figure. It does not say that a scheduled refresh nobody reads accounts for a meaningful share of it.

I will not promise a percentage of recoverable spend before looking at the workload. It depends on usage, contracts, architecture and how much optimization has already happened. The causes, though, recur everywhere I have looked: compute left running well past active query time, auto-suspend set too long or disabled, redundant queries bypassing the result cache, retention nobody chose, and background features quietly consuming capacity.

Two things have made this sharper. Pricing models differ in kind, not just degree: per-second billing suits bursty workloads, reserved capacity suits steady ones, and pay-per-query pricing suits occasional ad hoc analysis, so the same workload can cost very differently depending on a decision made years earlier. And AI-assisted development has increased the volume of SQL being proposed. The risk is not the assistance itself but query patterns that scan far more data than necessary, written quickly by people who cannot see what a scan costs.

The organizational failure is predictable. Finance notices the line item growing and asks for it to be reduced. The data team responds with across-the-board cuts, because it has no attribution and therefore no basis for a targeted answer. Something important gets slower, someone complains, the cut is partly reversed, and nothing is learned.

Constraint

• Attribution is not available by default. Knowing which team, model or dashboard caused which cost takes deliberate instrumentation that no platform gives you for free.
• The people creating the cost cannot see it. An analyst writing a query has no signal about whether it scans a little or a lot, and no reason to care without one.
• Cost reduction competes directly with everything else. Optimization produces no new capability, so it is permanently easy to defer until it becomes urgent, at which point it is done badly.
• Finance wants a forecast; consumption pricing resists forecasting. Spend follows usage, usage follows business activity, and a data leader is asked to predict a number they only partly control.

The second constraint is the root cause. Every other problem here is downstream of the fact that cost is invisible at the point it is created.

Decision

Attribution before optimization, and visibility before governance. Cutting spend without knowing where it comes from produces arbitrary cuts. The durable work is making cost legible to the people who generate it.

Make cost attributable

Tag compute by team, domain and workload type, and separate the categories that behave differently: scheduled transformation, ad hoc analysis, BI serving, ingestion, and whatever machine learning or experimentation exists. They have different cost profiles and different owners, and adding them together produces a number nobody can act on.

The output is the ability to say “this domain’s transformations cost this much, and here is what changed since last quarter.” That sentence is the entire foundation. Without it, every cost conversation is a negotiation about vibes.

Put the cost in front of the people creating it

Once attribution exists, show it to the teams as visibility, not as a chargeback. In my experience most waste is not carelessness. It is an absence of signal. An analyst who discovers that a dashboard refreshing every fifteen minutes is read twice a week will fix it themselves, usually the same day, without any policy being required.

Chargeback is worth resisting for as long as possible. It turns an engineering conversation into a budget negotiation, and it encourages teams to hide workloads rather than improve them. Where you do end up with it, it only works for costs a team actually controls, and that boundary has to be agreed with finance first.

Spotify’s engineering team described the same instinct in 2020, in its account of Cost Insights: show each team how its cloud spend is trending against the business’s growth, and let it weigh optimization against its goals. Only the teams whose costs were growing faster than the business were nudged to look.

Fix the recurring structural causes

These are worth checking before anyone proposes a platform move:

• Idle compute. Auto-suspend configured generously or disabled entirely. Almost free to fix and almost always present.
• Refresh frequency exceeding decision frequency. Hourly pipelines feeding weekly decisions. The right question is never “how fresh can this be” but “how fresh does the decision need this to be.”
• Full rebuilds where incremental would serve. Often a leftover from when the table was small.
• Retention and time-travel settings nobody chose. Defaults left in place on large tables, paying to keep history that has no consumer.
• Orphaned work. Dashboards, models and scheduled jobs that nothing reads and nobody owns. The archaeology is tedious and the recovery is usually significant.

Count the operating responsibility that moves with a purchase

I have replaced vendor-provided capability with a system the team owned. Ending the subscription was the visible outcome. Taking responsibility for the replacement meant staffing and operating it, not just building it. The saving and the new obligation belong in the same decision, and removing a vendor invoice is evidence of a changed cost structure, not by itself of a lower total cost.

Choose the pricing model deliberately, and revisit it

The pricing model is an architectural decision that gets made once and then inherited indefinitely. Workload shape changes. A platform that was bursty and exploratory in year one is often steady and scheduled by year three, and the pricing model that suited the first shape can be substantially wrong for the second. That deserves an annual review, not a migration project undertaken in a panic.

On switching platforms to save money: it is occasionally correct, and more often a way of paying a large migration cost to avoid a smaller optimization effort. Establish that the mismatch between workload and pricing is structural before treating migration as the remedy.

Give finance a forecast they can use

Rather than a single predicted figure, give finance a model: baseline committed spend, variable spend driven by a named business metric, and a stated range with the conditions that would push it to either end. That is a conversation finance recognizes and can plan around, and it turns an unpredictable line item into a governed one.

The tradeoff I accepted

I accepted a materially higher bill than the technical minimum, deliberately, and I would defend that choice.

The platform could always be cheaper. Every optimization above has a further increment available: tighter suspend timings, more aggressive incremental loads, lower retention, fewer materialized layers. Pursuing all of it produces a cheaper platform that is slower to develop against, harder to debug, and more fragile when something unusual happens.

The specific trade I made repeatedly: keeping raw data and history that was expensive to store and rarely queried. It is straightforwardly inefficient. It is also what makes a definitional correction reprocessable rather than impossible, and I have needed that often enough to consider the storage cost excellent value. Storage is cheaper than irreversibility.

I also left several things unoptimized on purpose:

• Development environments. Making them costly to run is a false economy that slows every change the team makes.
• Low-spend workloads, however inefficient. Optimization effort should follow spend, not offend engineering sensibilities. A grossly inefficient query that costs very little is not a priority, and treating it as one is how teams spend a week to save a rounding error.
• Redundancy on critical paths. Some duplication exists to make failure recoverable. Removing it looks like efficiency until the day it does not.

The costs of this position:

• It needs defending, repeatedly. “We could spend less and I have chosen not to” is a sentence that has to be made again to each new finance partner, and it is only credible if you can show exactly what the extra spend buys.
• The line between deliberate and neglected is genuinely blurry. Some of what I called an accepted tradeoff was, honestly, work I had not gotten to. Being rigorous about which is which takes a discipline I did not always apply.
• Attribution work is dull and never finishes. Tagging decays as new workloads appear, and without periodic effort the attribution quietly becomes fiction while continuing to produce confident reports.

Outcome

The vendor replacement retired a subscription and the external data exchange that came with it, and gave the team direct control of the capability, along with the work of running it. I would judge its net financial result only after counting that work.

For the platform as a whole, the goal is not a lower bill. It is a bill that is explicable and owned, where each material component has a team attached, a reason, and someone who would notice if it changed.

What good looks like:

• Spend is attributable by team and workload type, and the attribution is current rather than aspirational.
• Cost per unit of business activity is flat or falling even as total spend grows. Growth in the total is not a problem if the platform is doing more. Growth in the unit cost is.
• Teams correct their own spend when shown it, without a policy being required.
• Finance receives a range with stated drivers rather than a number that turns out to be wrong, which changes the relationship from oversight to planning.
• Cost comes up in architectural discussions as an ordinary consideration, rather than arriving quarterly as a complaint.

What I’d do differently

Instrument attribution before it is needed. Every time I have done this reactively, in response to a finance question, the first several weeks went on building visibility that should already have existed, while the pressure to produce savings was already on. Attribution built calmly is cheap. Attribution built under scrutiny is expensive and worse.

Publish the cost of common operations to the team. People cannot weigh what they cannot see. A short internal reference on what a full table scan costs relative to a partitioned read changes behavior more effectively than any review process, and it takes an afternoon to produce.

Separate the deliberate inefficiencies from the unexamined ones, in writing. I carried a general sense of which costs I had accepted on purpose. Writing down what each one was, why I accepted it and what would change the decision would have made the annual conversation with finance much easier, and it would have exposed which ones were decisions and which were just untouched. The Tradeoffs exercise is a small version of that record.

Review the pricing model on a schedule rather than on a prompt. Workload shape drifts continuously, and the pricing decision is reviewed almost never. An annual hour spent checking whether the model still fits the workload is one of the highest-return recurring commitments available.