What breaks when agents meet a real warehouse

Every text-to-SQL demo you have seen runs against a schema somebody designed that morning. Six tables, unambiguous names, foreign keys that mean exactly what they say. The agent writes a correct query, the room nods, and someone asks why the data team still exists.

Mature warehouses do not look like that. A schema that has been in production for a decade typically has several columns, across several schemas, that a reasonable person would call “customer,” and they do not agree on who counts as one. That is not a defect somebody failed to clean up. It is the residue of a business changing its mind about itself several times, which is what businesses do. The schema is a fossil record of old decisions, and the agent has no idea which layer it is standing on.

What the benchmarks say, and what they don’t

When the Spider 2.0 benchmark was introduced in 2024, its authors reported that their best agent, built on o1-preview, solved about 17 percent of its enterprise tasks, against 91 percent on the original Spider benchmark, and about 10 percent when built on GPT-4o. Spider 2.0 changes more than the schema, with longer contexts, several SQL dialects and multi-step workflows, but the gap made the point: a clean academic task and a real one are different problems.

Agent systems built for the task have since pushed the top scores well past those first results, and the leaderboard moves often enough that any figure I put here would date. So the 2024 gap was not a permanent ceiling.

I read the leaderboard as a bound on what is possible, not a forecast of what an assistant will do against your definitions. Evaluate the workflow you intend to deploy. A public benchmark is context for that decision, not its acceptance test.

The costly error is an answer that looks usable

Here is the part that surprised me. SQL generation was supposed to be the hard problem, and it largely stopped being one. Point a frontier model at a well-described schema with a clear question and it writes competent SQL: window functions, nested CTEs, the lot. The problem is not that the agent writes bad SQL. It is that the agent writes plausible SQL against a schema it has misread, and the result looks exactly like a correct answer.

Say someone asks for revenue by region last quarter. The warehouse holds gross revenue, revenue net of refunds, and revenue net of refunds and partner commission. All three are valid, and the names tell them apart only if you already know the business well enough to decode names chosen years ago. The agent picks one. The number is in the right range, because revenue figures are all roughly the same size, and it is wrong by whatever goes to partners. Nobody catches it in the meeting. It gets caught three weeks later, when someone reconciles against finance, and by then two decisions have been made on top of it.

My own first test produced the funny version. Someone asked for yesterday’s total revenue and got a confident figure roughly ten times the real one, since several columns had revenue in the name and nothing said which one meant what. The channel laughed, because ten times is easy to spot. The one to worry about is only off by what goes to partners.

The plausible kind showed up too. In the deployment I wrote up in What the assistant was allowed to know, the largest failure category in the first month was date handling: the same question about last month answered against a calendar month, a rolling thirty days and the fiscal period, each time fluently and each time with a query that ran. It was one failure repeated across different questions, and it belonged in the metric definitions rather than in a growing list of corrections.

A wrong query that errors costs nothing: you fix it and move on. A wrong query that returns costs a decision, and then the credibility of the system that produced it.

What tends to hold

A semantic layer the agent has to go through. This is the single most valuable guardrail, and it is not an AI intervention at all. If “revenue” resolves to one governed definition rather than three ambiguous columns, the agent cannot pick the wrong one, because there is only one to pick. It still cannot resolve an intention the user never stated, so the agent also has to be able to ask which decision the number is for, or decline. Agents did not create the case for a semantic layer. They made it urgent, and they made it fundable.

Deliberately restricted surface area. Point the agent at a curated set of well-named, well-documented marts rather than the whole warehouse. It feels like under-using the technology, and it is exactly right. Every table you expose is a table the agent can misread. Uber’s engineering team found the same thing in 2024 with its QueryGPT assistant: accuracy fell as more schemas and examples were added, and narrowing intent and table selection was part of the fix.

Citations derived from what actually ran. Require every answer to state the tables, filters and period it used, taken from the executed query rather than generated alongside it. The obvious benefit is auditability. The subtler one is that it changes the reviewer’s job from “does this number feel right,” which nobody can do reliably, to “is this the right table and the right filter,” which a domain expert can answer in seconds. It does not prove the joins are right. It makes the wrong ones findable.

Access enforced in the system, including at delivery. The assistant inherits the user’s permitted access, and an answer posted in a shared channel is checked against everyone who can read it, not just the person who asked. A prompt is not an access-control mechanism.

Review by consequence. Internal exploration can run on sampled audits. Anything reaching a customer, an executive or a consequential decision is reviewed by someone who knows the domain, at least while trust is being established. Skipping that to demonstrate speed is how you spend trust you have not earned yet.

Evaluation against questions you already know the answer to. Build a fixed set of verified questions and re-run it on every change to the definitions, the prompts, the retrieval or the model. It is the least exciting item on the list, and the one that catches the regressions nobody would otherwise notice until a user did. It also has to cover the answers the assistant should not give:

Question type	What a useful evaluation checks
A known business metric	Value, grain, filters, period and definition agree with a verified answer
An ambiguous request	The assistant asks for the missing decision context
An unsupported question	It names the limit and offers a useful handoff
A restricted request	Access holds through both retrieval and delivery
A familiar question phrased differently	The meaning stays consistent across variations
A stale or partial source	The limitation reaches the user and changes whether an answer is given

What doesn’t

Documentation as a substitute for structure. The theory: write thorough column descriptions, feed them to the agent, and it will disambiguate correctly. In practice, documentation of a messy schema describes the mess accurately. Three revenue columns with three excellent descriptions still require the agent to infer which one the question meant, and that inference is exactly what it is bad at. Documentation helps a well-structured schema. It does not rescue a badly structured one.

Confidence scores as a review trigger. Asking the model how sure it is and routing low-confidence answers to a person is an appealing design. The trouble is that a model’s stated confidence is a poor guide to whether it is right. The answers that most need review are the ones where the model has confidently misread the schema, which is exactly when it reports high confidence. Filtering on that signal routes away the errors you most need to catch.

Broad access, constrained by the prompt. The instruction “only use the analytics schema” works until a question is easier to answer another way. If the agent should not touch a table, revoke its access. Do not ask it not to.

Assuming users will verify. They will, for about two weeks. Then the system will be right often enough that checking feels like theater, and they will stop. Design for the world where nobody checks, because that is the world you will be operating in by month three.

Where I’ve landed, for now

The framing I have settled on: an agent is a very fast analyst with no institutional memory and no instinct for when a number smells wrong. That instinct is most of what senior people on a data team are actually paid for. It is not the SQL. It is looking at a result and thinking that revenue figure is too round before anyone else has finished reading the slide. Automating the query without automating the smell test does not remove the analyst. It moves the scarce skill from writing to verifying.

Which leads to a prediction I will put my name to, and you are welcome to disagree with it. The organizations that get real value from agents against their warehouse will be the ones that already invested in semantic layers, test coverage and clean marts, and the gap between them and everyone else will widen rather than close.

The common assumption is the opposite: that agents are a leveler, letting teams with messy data skip the unglamorous modeling work and query their way out of it. I think that is backwards. Agents multiply the quality of your semantic foundation, and a multiplier applied to a poor foundation produces confident nonsense at scale.

If that is right, the first response to agentic analytics is not a pilot. It is finishing the data modeling you have been deferring, because that work is now the constraint on something the business actively wants. I would start a pilot when one useful area has settled definitions, permitted access, an owner for failures and a baseline for evaluation, and I would hold the expansion whenever errors are hard to detect or review costs more than the workflow saves.

The uncomfortable corollary: “our data is too messy for this” is not a reason to wait. It is the finding. It was always true, it was always costing you, and until now it was merely invisible.