What the assistant was allowed to know

Situation

Questions arrived in the team chat tool, addressed to whoever on the data team had answered last time, and they waited in a queue measured in days. By the time a language model could write a competent query, several vendors were offering to make that queue disappear, and a demo takes an afternoon.

The demo works because the schema is small and the question is one the builder knows how to answer. My warehouse was neither. It carried a decade of history, tables that had been superseded and never dropped, and several defensible definitions of revenue depending on refunds and what partners were owed. A person querying that warehouse pauses at the ambiguity, some of the time. A model does not pause. It picks one and speaks fluently.

The platform had spent the year before getting its house in order: named owners for data, shared metric definitions in code, lineage, a catalog, and observability on the pipelines. All of that had been built for people. This is what happened when the thing reading it stopped being a person.

So the decision was never whether to build an assistant. It was what the assistant would be allowed to know, who would be allowed to ask it, and how I would find out it was wrong before an executive did.

Constraint

• A channel is an audience. An answer posted in a shared channel reaches everyone in it, including people who could not query the source themselves. Access had to hold at delivery, not only at retrieval.
• No verified answers existed. Nobody had a set of questions with known-correct results to test against, so the first evaluation set had to be built before anything could be measured.
• No headcount. The team of ten-plus was already running the platform. The first build had to fit in the margins of my own week, and the running of it inside analyst time that was supposed to be freed, not consumed.
• One plausible wrong number would end it. Trust in a data system falls faster than it recovers, and the first audience included people whose numbers appear in leadership reviews.
• Definitions were governed in the center and thin at the edges. The metrics the company was run on had owners and code. The long tail did not, and the assistant would be asked about the long tail on day two.

The first and the last of these shaped the design more than the model did. Everything else was engineering.

Decision

Make the platform, not the prompt, responsible for what the assistant knows. A prompt is a guideline, and a guideline is followed by whoever read it. A definition that lives in the platform and reaches the agent as a tool is followed by default.

I learned that in the first test. Someone asked for yesterday’s total revenue, and the assistant answered, confidently, with a number about ten times too high. The data layer had several columns with revenue in the name, and neither the rules I had written for the assistant nor the schema said which one meant what. It got a laugh in the channel, because ten times too high is easy to spot. It was also the most useful failure of the whole build: nothing about meaning could be left to the agent to interpret, and everything below came from taking that seriously.

Metrics as tools, context as documents

From then on, the assistant stopped reading the raw schema and guessing. It got three things. A metric lookup that returned a governed metric’s formula, grain, exclusions and freshness, from the same definitions the dashboards used. A table lookup that returned what a table meant, its keys, how it joined, and its known caveats. And a read-only query path with a row cap and a timeout, for the questions that still needed SQL after the first two had narrowed them down.

Around those sat a knowledge base of a few hundred pages, retrieved by meaning rather than by keyword: metric definitions in plain language, table notes, and the caveats that had lived in analysts’ heads, such as which region reports a week late and which measure finance takes to the board. From the second week it also held the questions already answered, so a question asked twice had a verified answer to match.

The hard part was not retrieval. It was writing down what analysts had been quietly working around for years, and finding out in the process how much of the platform’s apparent consistency had been people.

Launch narrower than feels reasonable

The first version, the one in that test, was meant to answer questions about twelve metrics in two decision areas and decline the rest by name. It ran in one channel whose members all held the same data access, so the audience problem was solved by the guest list before it was solved by the system. About twenty people were invited, someone from every department and a few executives, and the invitation was explicit: break it. Ask it what you already know. Ask it the same thing two ways. Ask it something it should refuse. Say when it is right for the wrong reason.

The questions it declined were not wasted. They became the users’ own list of what to connect next.

The narrow phase ran six weeks. Widening was tied to evidence rather than to a date: the evaluation results below, and a failure log with nothing open that could mislead a decision.

The feedback loop is the product

Every answer was logged with the question, the query it ran, the sources it cited and the reaction it got in the thread. An analyst who knew the domain reviewed every flagged answer each week, and a sample of the unflagged ones. Failures sorted into a short list: dates and periods, definitions with more than one reading, a wrong or superseded table, an ambiguous question answered instead of clarified, and an out-of-scope question answered instead of declined.

Each failure was answered twice. Once to the person, in the thread. Once to the knowledge base, as a new page, a corrected definition or a scope note, so the same failure could not come back in the same form. Those corrections fed the evaluation set, which existed before the first test: about sixty questions with answers validated and verified by hand, the set the launch was tested against. It grew to about two hundred and fifty within two quarters, a third of them from real failures, and it was re-run after every change to the definitions, the retrieval or the model.

Stress tests before widening

The verified set tested whether supported answers were right. The stress tests asked a different question: whether the assistant could be made to do the wrong thing on purpose. Before the second channel opened, two sessions tried to break it: requests for data outside the asker’s access, questions designed to produce contradictory answers, instructions pasted into a question to see whether the assistant would follow them, and requests for a citation to see whether it would invent one.

Three controls came out of those sessions. Access was enforced in the query path and checked against the channel at delivery, never in the prompt. Citations were derived from the query that actually ran, not generated. And there was an explicit path for “not here”: a sensitive question in a public channel got a pointer to a private one, not an answer.

Graduate the glue

The first version was built on a workflow-automation tool, because that was the fastest way to change it every week, and weekly change was the whole method. It was also glue: no tests, a thin runbook, and one person who understood it end to end.

Once the definitions and the context had settled, they moved out of the workflow and into tools on an MCP server, so that any agent could call them, including the ones people were already bringing to their desks. The assistant in the chat window became one client of that layer rather than the layer itself, which is where the work that mattered had been all along. What that layer makes possible once business users start commissioning their own reports is a separate piece.

The tradeoff I accepted

I traded coverage for trust, and paid for it in goodwill and in curation.

The narrow launch disappointed people who had seen the demo. About a third of the questions in the first month fell outside the twelve metrics, and the assistant said so, politely, to people who had expected magic. Holding that line took explaining, more than once, that the narrowness was the reason the answers could be believed.

The data team’s work moved rather than shrank. Less answering, more curating: writing the pages, reviewing the failures, settling the definitions the assistant had exposed. That is a standing half-day a week of a domain analyst’s time, and it appears on no roadmap as a deliverable. It is also the most effective governance forcing function I have encountered. Every inconsistency a human analyst had quietly worked around was now surfaced in a public channel, in front of the person who asked. Nothing about the mess was new. The audience was.

Building on glue first was the right call, and it had a bill. The first installment came on day one. The knowledge base, the rules and the connection to the model were all running on my own machine, and I opened the test to all twenty people at once. When they started asking, the machine queued everything, and every answer took two or three minutes, in front of a few executives. A rookie mistake: even a beta belongs on a proper server. The larger installment came later. By the time the layer graduated into proper tools, the workflow had become load-bearing in a way its runbook did not deserve, and the graduation came later than it should have.

The model bill was the smallest cost of all. An answer costs cents. The people who make the answers trustworthy cost what people cost.

Outcome

In the quarter after widening, the routine request queue fell by roughly forty percent against the quarter before, and a supported question got an answer in under a minute instead of waiting a day or more. Analyst time moved to work that still needed investigation and judgment, and to the curation above, so part of the drop is work that moved rather than work that disappeared.

On the evaluation set, the share of supported questions answered correctly rose from roughly seventy percent at launch to above ninety after two quarters of the feedback loop. Read that rise with some care: a third of the set came from failures that were then fixed, so part of the improvement is the set learning the assistant’s old mistakes. The weekly review of unflagged answers was the fairer check, and next time I would hold a share of questions out of the set entirely. What stayed wrong was mostly ambiguity the definitions had not yet settled, which is a governance backlog with an owner rather than a model problem.

Restricted questions never got through in any evaluation run after the stress tests, because access was enforced in the query path rather than left to the model. That was the result I cared about most, and the one the stress tests had been for. Out-of-scope questions were the weaker side, because declining them depended on the model following its scope rules. A few got answered instead of declined, and each one became a scope note in the knowledge base.

What I cannot tell you is whether any decision got better because the number arrived faster. I did not track which decisions cited an answer, and next time I would.

For a new rollout I would watch the same things, in this order: whether restricted questions are declined, whether supported questions match verified answers, how many open failures could mislead a decision, and how much analyst time the curation is taking. Message counts measure curiosity, which is free.

What I’d do differently

Put even the beta on a proper server, and do not open it to everyone at once. Twenty people asking at the same moment is a load test, whether or not you planned one.

Settle period and time-zone rules in the definitions first. Date handling was the largest failure category in the first month, and every instance was the same failure wearing a different question. It belonged in the metric definitions, not in the knowledge base as a series of corrections.

Set the graduation criterion in advance. “When the definitions stop changing weekly, the layer moves into tools” is easy to say before the workflow becomes load-bearing and hard to schedule afterward.

Publish the failure log to the users, from the first week. The instinct is to fix the embarrassing answers privately. A log that starts ugly and visibly shortens builds more trust than a clean launch, for the same reason a variance report does.