An analyst in the chat window

Data questions do not arrive through the dashboard. They arrive in the chat tool, usually as a message to whoever on the data team answered last time, and the answer arrives hours later, when that person has finished what they were doing. The dashboard exists. The question was not on it. Most questions are not.

So the obvious move, once a language model can write a competent query, is to put an assistant where the questions already are. Ask in plain language, in the channel, and get an answer back with the number and its source. No dashboard, no SQL, no queue.

I have done this, pairing natural-language access with retrieval of documented business context and a review-and-feedback loop on the responses. The context work mattered because a schema alone could not explain which definition or source a question required.

The technical guardrails that make it safe against a real warehouse, a semantic layer, restricted surface area, forced citation of sources, are a separate subject and I have written about them elsewhere. This post is about the rollout: the decisions about size, people and expectations that determined whether anyone was still using it a month later.

Start smaller than feels reasonable

The first temptation is to connect the assistant to everything. The warehouse is right there, the model can read a schema, and a narrow launch feels like under-selling the capability.

Resist it. The right first version is a minimal architecture connected to a small, curated set of governed sources, and it should be narrow on purpose rather than narrow because time ran out. Two reasons.

The first is trust economics. Every wrong answer costs more than every right answer earns, and trust in a data system does not recover at the speed it is lost. A small surface means fewer ways to be wrong, and fewer ways to be wrong means the early experience is mostly right, which is the only foundation that expansion can be built on.

The second is that a narrow surface tells you what people actually want. When the assistant can only answer questions about a handful of core metrics, the questions it cannot answer become a prioritised list of what to connect next, written by the users themselves. Connect everything on day one and that signal never appears.

Broaden as trust is earned, not as capacity allows. Those are different schedules, and the second one is always faster.

Recruit people to break it

The second decision is who gets it first, and the answer is not the people most likely to be impressed. It is the people most likely to break it.

The framing I used with early users was direct: try it, break it, confuse it, surprise it. Ask it the question you already know the answer to and see if it agrees. Ask it something ambiguous and see what it assumes. Ask it the same thing two ways. Say when it is wrong, and say when it is right for the wrong reason.

Nearly every meaningful defect in the first weeks came from this group rather than from testing: the edge case in a date filter, the metric with two plausible definitions, the question phrased in a way the assistant misread. Testing finds the failures you anticipated. Users find the ones you did not, and they find them faster when they have been told explicitly that finding them is the job.

This also sets the relationship correctly. A tool launched as finished invites disappointment at the first error. A tool launched as something its users are shaping invites feedback, and feedback is the raw material of the weekly iteration that makes it better. Stability, accuracy and context all improve on that cadence, and they improve because someone reported the thing that was wrong.

What it does to the questions

Here is the part I did not fully anticipate. When the interface stops being the constraint, the quality of the question becomes the constraint, immediately and visibly.

A broad question such as “why is conversion down?” needs clarification. A hypothesis about a new checkout field gives the investigation somewhere to start, but it can also steer the assistant toward confirming the suggestion. Ask what would disprove it and compare alternatives such as traffic mix or tracking changes. Specificity helps investigation; it does not establish a cause.

The habits that made someone good at asking an analyst turn out to be exactly the habits that make the assistant useful, and I have set them out separately. But there is a sequence that reliably gets a good analysis out of a chat assistant, and it is worth teaching directly, because it is the sequence a good analyst would follow anyway:

1. Start with the metric over a period. Monthly for the year so far, say.
2. Ask which periods moved most, up and down.
3. Take the worst one and break it down by two or three dimensions: product line, customer segment, acquisition channel.
4. Ask what other attributes might explain the change.
5. Check the movement against known events: campaigns, seasonality, releases, anything external.
6. Finish with the three most important findings and what you would do about each.

An assistant led through that sequence produces something close to an analyst’s write-up. The same assistant asked only the first question produces a chart. The asking matters, alongside the sources, definitions, and limits of the assistant. Training needs to cover both investigation and the point at which the user should hand the question to an analyst.

The failure modes are human

The technical failure modes, the wrong column and the plausible number, are covered in the other post. The human ones are different and just as costly.

Accepting the first number. The assistant is fast and fluent, and fast, fluent answers feel finished. The habit to instil is one more question: which definition did you use, and which table did that come from? Make the assistant state both by default, and make asking a normal part of the exchange rather than a sign of distrust.

Treating it as an oracle. The framing I have found most useful is that the assistant is a very fast junior analyst who joined this morning. Enormously useful, and nobody puts a first draft from that person in front of the board unreviewed.

Expecting magic on day one. A narrow launch disappoints people who arrived expecting the demo. Someone has to hold the line on scope in the face of that, and explain, more than once, that the narrowness is the reason the answers can be trusted.

What to measure

Not messages sent. Message counts measure curiosity, which is free.

Measure whether a usable answer replaces a ticket, including the time spent checking and correcting it. Compare recurring questions with a baseline, and track abandonment and escalation alongside completion. Message volume alone cannot tell you whether the assistant reduced work or simply moved it.

Challenges are a useful feedback channel, but their absence is ambiguous. Pair user reports with sampled review so that silence is not mistaken for accuracy. Use domain review for consequential decisions and lighter sampling for approved low-risk uses, consistent with the evaluation approach in the companion piece.

What changed in practice

The routine analyst request queue fell substantially after governed self-service became available. Supported questions could be answered much sooner, and analyst capacity shifted toward work that still needed investigation and judgment. That is the operational outcome I would defend: fewer routine requests waiting on an analyst, rather than a claim that every kind of analysis became faster.

Early users also exposed date-filter edge cases, ambiguous definitions, and misunderstood questions. Reviewing responses and feeding corrections back into the system became continuing work. The reduction in requests should therefore be read alongside the effort spent curating context, evaluating answers, and handling exceptions.

What it costs

The data team’s work moves. Less answering, more curating. Definitions have to be settled, because the assistant will use whichever one it finds, and every inconsistency in the warehouse that a human analyst quietly worked around is now surfaced in a public channel, in front of the person who asked. That is uncomfortable, and it is also the most effective governance forcing function I have encountered. The mess was always there. Now it has an audience.

The narrow start costs goodwill with people who expected more, and expansion is slower than anyone wants. And the assistant makes the habits problem impossible to ignore, which means the rollout of a tool turns into a programme of teaching people how to ask, whether or not that was the plan.

The beginning, not the destination

An assistant in the chat window is not the end state. It is the delivery mechanism for something older: the ability of the people who know the business to get answers from the data without waiting for someone else. The tool is the newest way to do that and probably not the last.

What it can change is who gets to ask within the supported scope. That includes business users in the place they already work, provided the answer is visible only to people permitted to receive it. That was always the promise of self-service. It took the interface disappearing to find out that the promise was never really about the interface.