Skip to content
Anil Thapa
Trust & governance

Changing the metric without erasing the decision

A better definition can make the same history tell a different story. Who approves the change, when it applies, how to compare periods, and what to preserve about the decisions made under the old one.

6 min read

A metric can become less useful without becoming wrong.

Say an activated workspace used to mean one that created a project in its first fourteen days. The product now depends on collaboration, so the proposed definition requires completing a shared project in that window. It asks a more demanding question, and it may be better at showing where onboarding needs work.

It also produces a lower activation rate for the same workspaces. Swap the new calculation in behind an unchanged chart, and the next review opens with a conversation about declining performance that the chart manufactured. No pipeline failed. The code does exactly what its author intended. The failure is letting a change in meaning arrive looking like a change in the business.

In One number, two histories I wrote about reconciling definitions across platforms. A definition change inside one platform is the same problem across time: people need a useful measure now, a defensible way to compare it with the past, and a record of what earlier decisions were based on. Those are three separate choices.

Name the decision the new definition will serve

Before changing the calculation, write down what the current definition fails to distinguish and which decision would improve if it did. If setup is still worth tracking, keep the old measure under a name that says what it is for, rather than replacing one broad label with another and leaving the same argument for the next product change.

When I unified the revenue definitions across departments, that became a rule. Every definition was classed as one of two kinds. Corporate standard meant approved, and the same in every report. Corporate custom meant governed, but specific to the one department that relied on it. The corporate reports used only the standard set, and each department kept custom reports for its own tracking, under its own names. Departments kept working the way they always had, and nobody could mistake a department’s number for the company’s.

The person who maintains the model can say what is feasible and what will break. The person accountable for the metric’s use approves its meaning. Consult the consumers, then name who settles a disagreement, because ownership that exists only in the catalog will not resolve an argument about which team now appears to be missing its target.

Name the kind of change, too. If the agreed definition already excluded test workspaces and the query accidentally included them, that is an error, and the decisions made from it need their own follow-up. If the definition is being replaced because the business needs a different measure, say that. The UK Office for National Statistics draws the same line in its revisions and corrections policies: a revision brings in better methods or data that were not available at first publication, a correction fixes a mistake, and both are flagged to readers. An internal metric change deserves the same label.

Put the effective date beside the approval

A deployment date leaves the important questions open. Does the new definition apply to workspaces created from that day, to the next reporting period, or to every cohort the platform can reconstruct? Those produce different series from identical code. So record when the change was approved, which periods or cohorts it covers, and when consumers first see it. A team might approve the definition before the quarter, publish a comparison while consumers prepare, and switch the official scorecard when the new quarter begins.

Alongside the dates, keep the rules themselves and a few examples that sit on either side of the boundary. A workspace that created a project but never completed one teaches more in a review than a sentence saying the new measure reflects engagement better. Those examples belong in validation as well, because a schema check passes happily while the meaning of a column changes completely.

Compare two periods under one definition

Before switching the scorecard, calculate both definitions over the same inputs for an overlap period, holding the population and the window fixed so the difference can be pinned on the rule being changed. With made-up numbers:

Cohort Created a project Completed a shared project
Earlier cohort 60% 40%
Later cohort 64% 44%

Under either definition, the later cohort is four points higher. Join the earlier cohort’s old result to the later cohort’s new one and you get a sixteen-point fall. Every value is calculated correctly. Their combination tells a misleading story.

A line chart can turn a change in definition into a story about performance.

Look inside the overlap, too: which workspaces drop out, and whether the effect differs by plan or acquisition route. One conversion factor applied to every old result rarely has a sound basis. If the events exist reliably in older data, publish a restated series and say how far back it goes. If shared-project completion was only recently instrumented, keep the old series under its old definition and show the break. A shorter comparable history beats a continuous line built on an assumption.

Keep the view the earlier decision used

A restated series shows how the past looks under today’s definition. A report preserved from the time shows what people saw when they decided. Both are useful, and they need labels that keep them apart.

If the earlier activation rate supported a decision to cut onboarding work, a later review should be able to find the measure used, its definition, the period, the filters and the recommendation it informed. Keeping that record does not prove the earlier choice was sound. It makes the judgment possible, where overwriting it leaves the discussion to memory. Keep the current route obvious, though: someone asking about today’s activation should land on today’s definition, not in a picker full of obsolete calculations.

Recomputing last quarter can improve the comparison. It cannot change what was available when last quarter's decision was made.

Move the targets and consumers with the calculation

A target set against the old definition does not become meaningful under the new one just because the label still matches. Keeping the same activation target after requiring a completed shared project quietly raises the bar. Replacing it with an easier number is a decision too, and it needs an owner. Neither should happen as a side effect of a model release.

Then follow the metric into everything that reads it. An alert threshold, a recurring deck, an exported analysis and an assistant’s saved description can all keep the old meaning after the warehouse starts returning the new value. Lineage finds the technical dependencies, and the owners of the decisions find the rest. When I unified those definitions, the team evaluated and inventoried every dashboard and brought each one under governance. An inventory like that finds the consumers a lineage graph misses.

Put the change note where people read the metric: the reason, the effective date, the effect on comparison, and where to find the old basis. An announcement in a technical channel is not evidence that the scorecard’s readers understand why the number moved.

Give the old calculation an end to routine use

Running both versions for a while helps explain the difference and move consumers. Running them indefinitely because nobody can decide which is current recreates the disagreement governance was meant to settle. Agree on the migration period and the conditions for retiring the old version, and give any remaining use an explicit reason and an owner.

Doing this carefully costs something. Someone has to build the comparison, investigate the differences, revisit targets and help people read the change, and a cleaner model or a requested report may wait. Keeping the old definition forever costs too: it keeps rewarding behavior that no longer represents the outcome you want.

Done well, the next review talks about whether onboarding produces useful collaboration, instead of spending its time working out why the activation line suddenly fell.