When AI Agents Skip the Semantic Layer, Metrics Diverge

Two AI agents can answer “what was revenue last quarter?” from the same warehouse and hand two executives two different numbers — both derived from valid, syntactically correct SQL. That isn’t a hallucination in the sense most teams worry about. It’s a governance gap: nobody told either agent what “revenue” means, so each one guessed, and both guesses looked confident enough to end up in a board deck.

The core finding

As AI agents get direct, ad hoc query access to production data warehouses, they increasingly bypass the semantic or metrics layer — the governed definitions of things like revenue, active users, or churn that BI teams spent years building — and instead write their own SQL straight against raw tables. Pointed at a schema instead of a certified metric catalog, an agent re-derives joins, grain, and business logic on every single prompt, which means the same question can return a different, silently wrong answer depending on which agent asked, which session it asked in, or which model generated the query that day (Cube, 2026).

What’s happening

For most of the last decade, the semantic layer — whether that was a dbt Semantic Layer, LookML, or a warehouse-native metric store — was a nice-to-have that mainly served human analysts building dashboards. It defined, once, what “revenue” meant: gross or net, inclusive of tax or not, recognized at order time or ship time, and which join path connected orders to customers without silently fanning out and double-counting rows. Analysts building reports against that layer got consistent numbers because the hard modeling work was done centrally and reused everywhere.

AI agents broke that assumption almost by accident. The fastest way to wire an LLM to a warehouse is to hand it schema information — table and column names, maybe a few example queries — and let it write SQL from scratch. It demos beautifully. The agent writes clean, working SQL against orders, customers, and subscriptions, and it looks like the semantic layer was never necessary — the same optimistic starting point behind most ungrounded text-to-SQL agents, which fail for related but distinct reasons. Then it meets production, where a table named orders doesn’t say whether “revenue” excludes refunds, and three tables all plausibly look like “the customer table” (Cube, 2026).

dbt Labs re-ran its own 2023 benchmark comparing text-to-SQL against the dbt Semantic Layer using 2026-generation models, expecting the accuracy gap to have closed as models got better at writing SQL. It didn’t close for the failures that matter most. Text-to-SQL accuracy nearly doubled since 2023, but for queries that fall inside a well-modeled semantic layer’s scope, the semantic layer still hit 98–100% accuracy against 84–90% for raw text-to-SQL on the same 2026 models — and the failure mode of the semantic layer, when a query falls outside its scope, is an honest error message. Text-to-SQL’s failure mode is a confident, wrong number that looks exactly like a right one (dbt Labs, 2026).

That distinction — a query that fails loudly versus a query that fails silently and gets read out in a meeting — is the entire governance problem in one sentence. A pipeline that was built to catch schema errors, null violations, or freshness failures has no mechanism to catch “the agent computed net revenue when the dashboard next to it shows gross revenue,” because both queries executed successfully and returned real, structurally valid data. This is a category of data-quality failure most pipeline monitoring was never designed to detect.

Who this affects

Data engineering and analytics engineering leads own the decision of whether agents query a governed layer or raw tables, and they’re the ones who have to explain, after the fact, why two dashboards disagree. If a semantic or metrics layer already exists for human-facing BI, the failure is usually that nobody extended agent access through it — the agent got warehouse credentials and a schema dump instead of a connection to the same layer analysts already use, the same gap that shows up when nobody keeps the human sign-off on what an agent is allowed to change.

CTOs and heads of data are accountable for the operational and reputational cost when this surfaces in front of a customer, a board, or an auditor rather than in an internal Slack thread. A metric-inconsistency incident is much harder to communicate than an outage: there’s no error, no downtime, no alert — just two numbers that don’t match and a room full of people trying to figure out which one, if either, is real.

Finance, RevOps, and any team whose KPIs feed board reporting or compliance filings are the ones most exposed, because they’re the most likely consumers of an agent-generated number that quietly used the wrong join or the wrong definition of an already-ambiguous term like “active user” or “churned account.” A single wrong figure in a board deck or a regulatory filing carries a cost well beyond the engineering time it takes to fix the query.

Anyone building customer-facing or embedded AI analytics — an agent that answers questions for external users rather than internal ones — inherits a second problem on top of metric drift: without governance compiled into the query itself, a correct query and a query that leaks another customer’s data look identical to the model generating it (Cube, 2026).

When this becomes a real problem

This isn’t a future risk — it’s already the default failure mode for any team that gave an agent direct warehouse access in the last year without also giving it a governed metrics interface. dbt Labs’ 2026 benchmark data shows the underlying cause is not going away as models improve: even the newest generation of models, given raw schema access, still had to guess at business logic that a semantic layer would have made explicit, and “guessing well most of the time” is exactly the property that makes this failure mode dangerous — it’s inconsistent, not consistently wrong, which is what makes it hard to catch in testing.

The near-term trajectory (the next 6-18 months) is that agent-generated analytics moves from “an analyst double-checks anything an agent produced” to “the agent’s output goes straight into a report, a Slack summary, or a customer-facing chat,” because that’s the entire point of deploying these agents in the first place. Every step that removes a human from between the agent’s answer and its consumption is a step that removes the informal error-correction layer that caught inconsistent metrics for the last decade. The Model Context Protocol (MCP) is accelerating this specific pattern in 2026: it gives agents a standard way to discover and query a warehouse directly, which makes the “just point it at the schema” path the easy default unless a team deliberately routes that MCP connection through a governed semantic layer instead of raw tables (Cube, 2026).

How this actually manifests

The mechanics are almost always the same three failures, layered on top of each other.

Metric-definition drift. The agent picks a plausible but unstated definition — gross versus net revenue, trailing 30 days versus calendar month, active as “logged in” versus active as “took a billable action” — and that choice changes silently between sessions or between which agent answered. Nothing in the query is syntactically wrong; the ambiguity was never resolved by anyone, human or machine, so the model resolved it on the fly, invisibly, every time.

Join and grain errors. Real warehouses are full of fan-out joins, slowly changing dimensions, and multiple tables that all look like “the customer table” from the outside. An agent that gets the grain wrong doesn’t error — it silently double-counts or under-counts, and the resulting number looks exactly as plausible as the correct one. A human analyst who has internalized years of tribal knowledge about which join path is safe catches this instinctively; an agent working from column names alone has no equivalent instinct.

No access control in a raw SELECT. When permissions live downstream of the query, or nowhere at all, a query that returns exactly the rows a user is entitled to and a query that leaks another tenant’s or department’s data are indistinguishable to the model generating them. Post-hoc filtering or SQL-scanning approaches are brittle in practice, because SQL has too many ways to reach the same data — subqueries, CTEs, views, joins — for a linter to reliably anticipate every path, which is the same reason identity-based row-level security has to sit below the query rather than after it (Cube, 2026).

The pattern compounds because none of these three failures throws an error. A pipeline built to catch nulls, schema drift, or freshness violations has nothing that watches for “this number disagrees with the certified one,” because as far as the database is concerned, both queries succeeded.

Actions to take now

1. Audit which agents have raw warehouse or database credentials today, versus which are querying through a governed layer. This is a same-week task: pull the connection list for every agent, MCP server, or AI tool with query access, and classify each one as “governed” or “raw.” Most teams have never inventoried this and are surprised by how many fall into the raw category.

2. Identify your highest-stakes metrics first and model those, not everything. You don’t need a fully modeled semantic layer before you can act — dbt Labs’ benchmark found that adding as few as three well-chosen models closed most of the accuracy gap for previously unanswerable questions. Start with the metrics that go into board decks, investor updates, or compliance filings, since those carry the highest cost when they’re wrong.

3. Route agent access through the same semantic or metrics layer your BI tools already use, rather than building a second, agent-specific path to the warehouse. If you already have a dbt Semantic Layer, Cube, LookML, or a warehouse-native metric layer serving dashboards, extending that same governed interface to agents — increasingly via MCP — is far less work than building parallel governance for agent queries from scratch.

4. Push access control into the query-compilation step, not a post-query filter. Row-level and role-based rules should be part of how the SQL gets generated in the first place, so an agent structurally cannot construct a query that returns forbidden rows, rather than relying on a downstream check that a clever prompt or an unusual join might route around.

5. Require every agent-generated metric to return its lineage alongside the number — the metric definition used, the filters applied, the time range, and the underlying query — so a person (or another agent) can verify the answer instead of taking it on faith. An unlabeled number in a Slack message is a liability; a number with its definition attached is auditable.

6. Run known-answer evals before trusting agent-generated metrics in anything customer-facing or board-facing. Test representative business questions against numbers your data team has already verified, including deliberately ambiguous phrasing, denied-access cases, and questions that fall outside the modeled layer’s scope — the goal is confirming the agent fails loudly on those, not silently.

7. For text-to-SQL you can’t yet route through a semantic layer, restrict it to internal, single-analyst, sandboxed use rather than customer-facing or automated-report use cases, and treat any number it produces as a draft that needs verification before it leaves that sandbox.

Key takeaways

• AI agents given raw warehouse access re-derive metric definitions, joins, and grain on every prompt, so the same question can return different — and silently wrong — numbers depending on which agent or session answered it.
• This failure mode produces valid, successfully executing SQL with no error thrown, which means standard pipeline monitoring for nulls, schema drift, or freshness doesn’t catch it.
• 2026 benchmark data from dbt Labs shows semantic-layer-grounded queries hitting 98-100% accuracy versus 84-90% for raw text-to-SQL on the same models, and the semantic layer’s failures are loud errors rather than plausible wrong answers.
• The fix is routing agent access through the same governed semantic layer your BI tools already use, with access control compiled into the query itself rather than filtered afterward.
• The risk compounds fastest wherever a human stops double-checking agent output before it reaches a board deck, a customer, or a compliance filing.

Metric drift from ungoverned agent access is a pipeline governance problem with a known fix, not a reason to slow down AI adoption. Get in touch if your data platform needs a governed path for AI agents to query it safely.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.