What a faster refresh actually buys

A request to make a report refresh every five minutes sounds like a small improvement. The current version runs hourly. The source can be read more often. Change the schedule, check the load, and the chart will move sooner.

Before approving that change, I want to know what becomes possible in the minutes it returns. Someone might release an order before a dispatch cutoff, stop a faulty process before more work accumulates, or arrive at a meeting with evidence that changes the plan. Each can justify spending more. A faster timestamp on the same unattended report is harder to defend.

In What the platform costs, I name refresh frequency as one of the recurring costs worth examining. It needs a more careful argument than matching the schedule to the meeting calendar. A weekly decision can require very recent information. A frequently opened report can contain numbers nobody is able to act on until tomorrow.

The requirement I want is the latest point at which useful information can arrive, with enough time left to do something about it. The architecture and the operating cost follow from that.

Find the deadline the refresh is supposed to protect

Take two illustrative uses of the same order data. The timings below are assumptions for the example, not targets taken from a deployment.

Use	Decision it supports	What the delivery must provide
Weekly capacity review	Allocate capacity for the coming week	Checked history through the previous business day, available before preparation starts at 8 a.m.
Dispatch exception queue	Reprioritize orders before the current dispatch closes	Relevant status changes visible within five minutes during operating hours, leaving time to intervene

For the planning report, a useful improvement might be completing validation before preparation starts. Refreshing it throughout the meeting could instead leave participants looking at different totals. A dated snapshot may serve that decision better, with a clear route for material changes that arrive after it was prepared.

The dispatch queue has a different constraint. An overnight answer would describe an opportunity already lost. Its owner needs a current view while the work can still be rearranged, and someone must be available and authorized to rearrange it. If changing an order takes fifteen minutes and the cutoff is twenty minutes away, a view already ten minutes behind can be too late.

Work backward from the action. Allow time for someone to notice the change, check what it means, decide, and carry out the response. What remains is the time available for capture, processing, and delivery, including some room for ordinary variation. A system acting automatically has a different response time, but it still needs a defined action and a limit on what it may change.

Ask about exceptions before slowing the planning pipeline. The weekly report may also feed a daily alert or an export whose owner never opens the dashboard. Low page views do not establish that a report has no use. The decision about frequency has to include those consumers.

Fresher data earns its cost when it leaves someone more time to change what happens next.

A recent refresh can still show old events

An updated timestamp can describe several different things: when the source recorded an event, when the platform received it, when a model ran, or when a report refreshed. The difference matters to the person trying to release an order before a cutoff.

A model that runs every five minutes against a source delivered once an hour mostly produces new copies of old information. A fast ingestion path can still feed a report whose cache updates slowly. A recent row in a table does not establish that every location or partition has caught up.

Trace the delay through the whole path to the consumer. Measure source arrival, processing, and presentation, and distinguish event time from the time each system handled the record. Where completeness matters, use evidence of source progress or reconciliation as well as timestamps. The newest event alone cannot prove that earlier ones are all present.

Even a platform setting explicitly named for freshness needs interpretation. Snowflake’s dynamic-table documentation describes target lag as a best-effort staleness target, measured from the base tables at the root of the pipeline. It can be exceeded under load. That setting does not measure the time an event spent outside those tables or the delay before a person receives the result. My requirement would be measured at the point of use, with the upstream delays included.

Check the busy periods and the slow cases. An average can look excellent while the report misses the only cutoff that matters. For the planning report, record whether checked data was ready by the preparation deadline. For the dispatch queue, examine how often relevant changes reached the operator too late, and how long those delays lasted.

Choose what may change after publication

An earlier answer can contain less information. Some records arrive late, some are corrected, and some sources describe the same event at different points in its life. Refreshing faster does not make those differences go away.

This is a design choice the Dataflow Model paper by Tyler Akidau and colleagues makes explicit. For continuously arriving, out-of-order data, the authors treat later arrivals and revisions as normal conditions and describe choices among correctness, latency, and cost. My application of that idea here is to agree what a consumer may do with a provisional result before making it arrive sooner.

The dispatch queue can show the current known order state and accept later updates, provided the action tolerates that uncertainty and the operator can see when the source is behind. The weekly review may need a reconciled period and a stable version that everyone uses. A number suitable for directing attention during a shift may be unsuitable for judging the shift’s final performance.

Name the provisional status, the expected revision process, and the evidence needed before treating the result as settled. If a late arrival changes a figure already used, make the revision visible. The ingestion case study covers why those reconciliation rules belong in the design.

Sometimes waiting is the correct cost to accept. If an action is difficult to reverse and a missing source could change the answer materially, an earlier partial result may have less value than a later checked one. In another use, waiting would itself cause the harm. The consumer and the platform owner need to agree which risk they are accepting.

Price the promise through a failed refresh

The estimate needs to include what it takes to keep the earlier answer useful when a source slows down or a refresh fails.

Start with the workload. More frequent runs can repeat scans, incur startup overhead, keep capacity active, or add contention to a shared source. The effect depends on how much work each run performs. Processing only changes can be both faster and cheaper than repeatedly rebuilding a large result. There is no universal multiplier that turns twelve times as many runs into twelve times the cost.

Include the engineering and operating work: handling duplicates and late arrivals, replaying missed changes, testing revised logic against history, and understanding what consumers saw during a failure. Some of that machinery may already exist. Count the additional responsibility the proposal creates, rather than charging it for every capability on the platform.

Then price the response expectation. A daily report with several hours of recovery room and a dispatch queue needed through the shift impose different demands on support. The tighter target may require more spare capacity, quicker escalation, or a staffed response during operating hours. A reliable fallback can change that calculation: an operator able to consult the source application has a different dependency from one whose only view is the queue.

The cost comparison should cover the normal run, recovery from a missed run, and the useful work displaced by maintaining the faster path. Set those against the consequence of waiting. If earlier delivery could prevent missed dispatches, estimate that benefit from observed opportunities and a stated range of assumptions. A faster test run by itself is not evidence that any dispatch was saved.

Reducing frequency also needs honest accounting. It may lower metered usage without changing a committed bill. It may free shared capacity that is immediately used elsewhere. Those can be useful outcomes, but they are different from cash savings. Explain what the proposal actually recovers.

Give the urgent consumer a smaller dependency

One urgent use can pull an entire reporting chain onto a tighter schedule. The dispatch view needs current order status, but the model it reads also waits for historical enrichment and several sources used only in planning. Speeding up everything upstream makes the small operational need expensive.

I would first check whether the existing operational application can supply the decision directly. If it already holds the necessary state, a new route through the analytical platform needs a reason: perhaps the action depends on information from several systems that the application cannot combine.

Where a separate path is justified, keep its dependencies to what the action requires. The planning history can retain its own publication schedule. Both paths still need agreed identifiers and meanings for the states they share. A smaller dependency does not justify inventing a second definition of an open order.

That split has a cost. There are now separate delivery promises to observe, and enough overlap to require reconciliation. Reuse common logic where practical, test the shared meanings, and explain any intentional differences in period or completeness. If those obligations outweigh the additional refresh cost, keeping one path at the tighter target may be the better choice.

This is the architectural decision worth recording: which consumer requires the speed, which dependencies it carries, and why the boundary is worth maintaining. A diagram that labels the whole platform real time cannot answer those questions.

Write down the delay you are accepting

Choosing a slower schedule gives up something. A new event waits longer to be seen. An unusual request may need a manual refresh. A use nobody disclosed may turn out to depend on the old timing. Those are reasons to investigate and agree the change, rather than quietly reduce frequency after a budget review.

For each material consumer, record the required data and period, the deadline or tolerated delay at the point of use, the operating hours, and what happens when delivery misses that promise. Include who can accept provisional data, what fallback is available, and who decides whether the requirement changes. This can fit in the existing service documentation.

Before committing to the new schedule, compare it against representative history, including busy periods, late sources, and known exceptions. Where appropriate, run it alongside the current path for a bounded period and review the decisions it would have served. Historical replay can show when information would have arrived; it cannot prove what a person would have done with it. Review that part with the consumer.

Agree the condition for revisiting the choice. A new operating shift, an earlier dispatch cutoff, or a newly automated action can make yesterday’s freshness target inadequate. A slower schedule is defensible while its assumptions hold.

A cheaper refresh schedule comes with a delay someone has agreed to carry.

For the two uses at the start, I would pay for a checked planning snapshot before preparation and a timely dispatch view during the hours when someone can intervene. I would want evidence that both promises hold under the conditions that matter. The resulting schedules may be different. Their justification is the same: what becomes possible before the opportunity to act closes, and what it costs to keep that opportunity open.