An analyst in the chat window

Data questions do not arrive through the dashboard. They arrive in the chat tool, usually as a message to whoever on the data team answered last time, and the answer arrives hours or days later, when that person has finished what they were doing. The dashboard exists. The question was not on it. Most questions are not.

So the obvious move, once a language model can write a competent query, is to put an assistant where the questions already are. Ask in plain language, in the channel, and get an answer back with the number and its source. No dashboard, no SQL, no queue.

I have done this, pairing natural-language access with retrieval of documented business context and a review-and-feedback loop on the responses. The context work mattered because a schema alone could not explain which definition or source a question required.

The technical guardrails that make it safe against a real warehouse, a semantic layer, restricted surface area, forced citation of sources, are a separate subject and I have written about them elsewhere. This post is about the rollout: the decisions about size, people and expectations that determined whether anyone was still using it a month later.

Start smaller than feels reasonable

The first temptation is to connect the assistant to everything. The warehouse is right there, the model can read a schema, and a narrow launch feels like under-selling the capability.

Resist it. The right first version is a minimal architecture connected to a small, curated set of governed sources, and it should be narrow on purpose rather than narrow because time ran out. Two reasons.

The first is trust economics. Every wrong answer costs more than every right answer earns, and trust in a data system does not recover at the speed it is lost. A small surface means fewer ways to be wrong, and fewer ways to be wrong means the early experience is mostly right, which is the only foundation that expansion can be built on.

The second is that a narrow surface tells you what people actually want. When the assistant can only answer questions about a handful of core metrics, the questions it cannot answer become a prioritized list of what to connect next, written by the users themselves. Connect everything on day one and that signal never appears.

Broaden as trust is earned, not as capacity allows. Those are different schedules, and the second one is always faster.

Recruit people to break it

The second decision is who gets it first, and the answer is not the people most likely to be impressed. It is the people most likely to break it.

The framing I used with early users was direct: try it, break it, confuse it, surprise it. Ask it the question you already know the answer to and see if it agrees. Ask it something ambiguous and see what it assumes. Ask it the same thing two ways. Say when it is wrong, and say when it is right for the wrong reason.

Nearly every meaningful defect in the first weeks came from this group rather than from testing: the edge case in a date filter, the metric with two plausible definitions, the question phrased in a way the assistant misread. Testing finds the failures you anticipated. Users find the ones you did not, and they find them faster when they have been told explicitly that finding them is the job.

This also sets the relationship correctly. A tool launched as finished invites disappointment at the first error. A tool launched as something its users are shaping invites feedback, and feedback is the raw material of the weekly iteration that makes it better. Stability, accuracy and context all improve on that cadence, and they improve because someone reported the thing that was wrong.

What it does to the questions

Here is the part I did not fully anticipate. When the interface stops being the constraint, the quality of the question becomes the constraint, immediately and visibly.

A dashboard filters questions by making bad ones hard to ask. An assistant answers anything. “Why is conversion down?” produces a fluent paragraph. “Conversion is down because of the new checkout field; what is the drop-off on that step?” produces a specific number from a specific table. Both look like answers. Only one is. The difference is entirely in the asking, which means the training that matters is not training on the tool. It is training on the method. A good hypothesis can also steer the assistant toward confirming it, so the method includes asking what would disprove it.

The habits that made someone good at asking an analyst turn out to be exactly the habits that make the assistant useful, and I have set them out separately. But there is a sequence that reliably gets a good analysis out of a chat assistant, and it is worth teaching directly, because it is the sequence a good analyst would follow anyway:

1. Start with the metric over a period. Monthly for the year so far, say.
2. Ask which periods moved most, up and down.
3. Take the worst one and break it down by two or three dimensions: product line, customer segment, acquisition channel.
4. Ask what other attributes might explain the change.
5. Check the movement against known events: campaigns, seasonality, releases, anything external.
6. Finish with the three most important findings and what you would do about each.

An assistant led through that sequence produces something close to an analyst’s write-up. The same assistant asked only the first question produces a chart. The asking matters, alongside the sources, definitions, and limits of the assistant. Training needs to cover both investigation and the point at which the user should hand the question to an analyst.

The failure modes are human

The technical failure modes, the wrong column and the plausible number, are covered in the other post. The human ones are different and just as costly.

Accepting the first number. The assistant is fast and fluent, and fast, fluent answers feel finished. The habit to instill is one more question: which definition did you use, and which table did that come from? Make the assistant state both by default, and make asking a normal part of the exchange rather than a sign of distrust.

Treating it as an oracle. A framing that helps users: the assistant is a very fast junior analyst who joined this morning. Enormously useful, and nobody puts a first draft from that person in front of the board unreviewed.

Expecting magic on day one. A narrow launch disappoints people who arrived expecting the demo. Someone has to hold the line on scope in the face of that, and explain, more than once, that the narrowness is the reason the answers can be trusted.

What to measure

Not messages sent. Measure the questions that used to be tickets and are now answered in the channel, including the time spent checking and correcting the answers. Measure the time from question to usable answer, before and after. And measure how often an answer gets challenged, because a healthy number of challenges means people are checking. A number that falls to zero means either the assistant became perfect or people stopped looking. It is never the first one. Pair the challenges with a sampled review of your own, so that silence is never mistaken for accuracy.

What changed in practice

The routine request queue fell by roughly forty percent in the quarter after the assistant widened, and supported questions were answered in under a minute instead of a day. That is the outcome I would defend: fewer routine requests waiting on an analyst, not a claim that every kind of analysis got faster. The early users also exposed date-filter edge cases, ambiguous definitions and misread questions, and feeding the corrections back became permanent work, so the drop in requests has to be read next to the curation it created.

Uber’s engineering team reported its own bounded rollout, of its QueryGPT assistant, in 2024: a few hundred daily users in a limited release to operations and support teams, most of them saying generated queries cut their authoring time. That is a good model for reporting a bounded rollout: who used it, for what, and what they said it saved.

What it costs

The data team’s work moves rather than shrinks. Less answering, more curating. Definitions have to be settled, because the assistant will use whichever one it finds, and every inconsistency a human analyst used to work around quietly now shows up in a public channel, where the person who asked can see it. The case study has what that curation cost in analyst time.

The narrow start costs goodwill with people who expected more, and expansion is slower than anyone wants. And the assistant makes the habits problem impossible to ignore, which means the rollout of a tool turns into a program of teaching people how to ask, whether or not that was the plan.

The beginning, not the destination

An assistant in the chat window is not the end state. It is the delivery mechanism for something older: the ability of the people who know the business to get answers from the data without waiting for someone else. The tool is the newest way to do that and probably not the last.

What it changes, if it is rolled out with restraint, is who gets to ask. Not just analysts. Every team, at the moment the question occurs to them, in the place they were already talking, with the answer visible only to people allowed to see it. That was always the promise of self-service. It took the interface disappearing to find out that the promise was never really about the interface.