What the platform costs

Situation

There is a question that separates data leaders who are treated as engineering managers from those treated as owners of a business function: what does your platform cost, and what is it buying?

A surprising number cannot answer it. Not because they are careless, but because cloud data platforms are designed to make the question hard. Consumption pricing converts an architectural decision into an operating expense that arrives a month later, aggregated, with no attribution to the team, model or dashboard that caused it. The bill says the warehouse cost a figure. It does not say that a scheduled refresh nobody reads accounts for a meaningful share of it.

The pattern is consistent across platforms and well documented: for teams spending materially on a cloud warehouse, something in the range of 30–50% of spend is typically recoverable through right-sizing, scheduling and pricing-model changes. The specific causes recur, compute left running well past active query time, auto-suspend set too long or disabled, redundant queries bypassing result caching, retention settings far exceeding what anyone needs, and background features quietly consuming capacity.

Two things have made this sharper recently. First, the pricing models genuinely differ in kind rather than degree, per-second billing suits bursty workloads, reserved capacity suits steady state, slot-based pricing suits ad hoc analytics, so the same workload can cost very differently depending on a decision made years earlier. Second, AI-assisted development has increased the volume of SQL being shipped, and the cost risk is not the assistance itself but ungoverned query patterns that scan far more data than necessary, written quickly by people who cannot see what a scan costs.

The organizational failure mode is predictable. Finance notices the line item growing, asks for it to be reduced, and the data team responds with across-the- board cuts because it has no attribution and therefore no basis for a targeted answer. Something important gets slower, someone complains, the cut is partially reversed, and nothing is learned.

Constraint

• Attribution is not available by default. Knowing which team, model or dashboard caused which cost requires deliberate instrumentation that no platform gives you for free.
• The people creating the cost cannot see it. An analyst writing a query has no signal about whether it scans a little or a lot, and no reason to care without one.
• Cost reduction competes directly with everything else. Optimization work produces no new capability, and is therefore permanently easy to defer until it becomes urgent, at which point it is done badly.
• Finance wants a forecast; consumption pricing resists forecasting. Spend follows usage, usage follows business activity, and a data leader is being asked to predict a number they only partly control.

The second constraint is the root cause. Every other problem here is downstream of the fact that cost is invisible at the point it is created.

Decision

Attribution before optimization, and visibility before governance. Reducing spend without knowing where it comes from produces arbitrary cuts; the durable work is making cost legible to the people who generate it.

Make cost attributable

Tag compute by team, domain and workload type, and separate the categories that behave differently: scheduled transformation, ad hoc analysis, BI serving, ingestion, and whatever ML or experimentation work exists. These have genuinely different cost profiles and different owners, and aggregating them produces a number nobody can act on.

The output is the ability to say “this domain’s transformations cost this much, and here is what changed since last quarter.” That sentence is the entire foundation. Without it, every cost conversation is a negotiation about vibes.

Put the cost in front of the people creating it

Once attribution exists, show it to the teams, not as a chargeback mechanism, which creates perverse incentives and political overhead, but as visibility. In my experience most waste is not carelessness; it is a genuine absence of signal. An analyst who discovers that a dashboard refreshing every fifteen minutes is read twice a week will fix it themselves, usually the same day, without any policy being required.

Chargeback is worth resisting for as long as possible. It converts an engineering conversation into a budget negotiation, and it incentivizes teams to hide workloads rather than improve them.

Fix the recurring structural causes

A small number of patterns account for most recoverable spend, and they are the same everywhere:

• Idle compute. Auto-suspend configured generously or disabled entirely. Almost free to fix and almost always present.
• Refresh frequency exceeding decision frequency. Hourly pipelines feeding weekly decisions. The right question is never “how fresh can this be” but “how fresh does the decision need this to be.”
• Full rebuilds where incremental would serve. Often a historical artifact from when the table was small.
• Retention and time-travel settings nobody chose. Defaults left in place on large tables, paying to keep history that has no consumer.
• Orphaned work. Dashboards, models and scheduled jobs that nothing reads and nobody owns. The archaeology is tedious and the recovery is usually significant.

Choose the pricing model deliberately, and revisit it

The pricing model is an architectural decision that gets made once and then inherited indefinitely. Workload shape changes. A platform that was bursty and exploratory in year one is often steady-state and scheduled by year three, and the pricing model that suited the first shape can be substantially wrong for the second. This is worth an explicit annual review rather than a migration project undertaken in a panic.

On switching platforms to save money: it is occasionally correct and more often a way of paying a large migration cost to avoid a smaller optimization effort. Published cases of teams halving their bill by moving platforms are real, and they generally involve a genuine mismatch between workload and pricing model rather than one vendor simply being cheaper. Establish that the mismatch is structural before treating migration as the remedy.

Give finance a forecast they can use

Rather than a single predicted figure, what works is a model: baseline committed spend, variable spend driven by a named business metric, and a stated range with the conditions that would push it to either end. That is a conversation finance recognizes and can plan around, and it converts an unpredictable line item into a governed one.

The tradeoff I accepted

I accepted a materially higher bill than the technical minimum, deliberately, and I would defend that choice.

The platform could always be cheaper. Every optimization listed above has a further increment available: tighter suspend timings, more aggressive incrementalization, lower retention, fewer materialized layers. Pursuing all of it produces a cheaper platform that is slower to develop against, harder to debug, and more fragile when something unusual happens.

The specific trade I made repeatedly: keeping raw data and history that was expensive to store and rarely queried. It is straightforwardly inefficient. It is also what makes a definitional correction reprocessable rather than impossible, and I have needed that often enough to consider the storage cost excellent value. Storage is cheap; irreversibility is not.

I also deliberately left several things unoptimized:

• Development environments. Making them costly to run is a false economy that slows every change the team makes.
• Low-spend workloads, regardless of how inefficient. Optimization effort should follow spend, not offend engineering sensibilities. A grossly inefficient query that costs very little is not a priority, and treating it as one is how teams spend a week to save a rounding error.
• Redundancy on critical paths. Some duplication exists to make failure recoverable. Removing it looks like efficiency until the day it does not.

The honest costs of this position:

• It requires defending, repeatedly. “We could spend less and I have chosen not to” is a sentence that needs to be re-made to each new finance partner, and it is only credible if you can show exactly what the extra spend buys.
• The line between deliberate and neglected is genuinely blurry. Some of what I called an accepted tradeoff was, honestly, work I had not got to. Being rigorous about which is which requires a discipline I did not always apply.
• Attribution work is dull and never finishes. Tagging decays as new workloads appear, and without periodic effort the attribution quietly becomes fiction while continuing to produce confident reports.

Outcome

The goal is not a lower bill. It is a bill that is explicable and owned, where each material component has a team attached, a reason, and someone who would notice if it changed.

What good looks like:

• Spend is attributable by team and workload type, and the attribution is current rather than aspirational.
• Cost per unit of business activity is flat or declining even as absolute spend grows. Absolute growth is not a problem if the platform is doing more; unit cost growth is.
• Teams self-correct when shown their consumption, without a policy being required.
• Finance receives a range with stated drivers rather than a number that turns out to be wrong, which changes the relationship from oversight to planning.
• Cost appears in architectural discussions as an ordinary consideration rather than arriving quarterly as a complaint.

What I’d do differently

Instrument attribution before it is needed. Every time I have done this reactively, in response to a finance question, the first several weeks were spent building visibility that should already have existed, while the pressure to produce savings was already on. Attribution built calmly is cheap; attribution built under scrutiny is expensive and worse.

Publish the cost of common operations to the team. People cannot weigh something they cannot see. A short internal reference on what a full table scan costs relative to a partitioned read changes behaviour more effectively than any review process, and it takes an afternoon to produce.

Separate the deliberate inefficiencies from the unexamined ones, in writing. I carried a general sense of which costs I had accepted on purpose. Writing them down, what, why, and what would change the decision, would have made the annual conversation with finance considerably easier and would have honestly exposed which ones were decisions and which were just untouched.

Review the pricing model on a schedule rather than on a prompt. Workload shape drifts continuously and the pricing decision is reviewed almost never. An annual hour spent checking whether the model still fits the workload is one of the highest-return recurring commitments available.