This is the fifth piece in a series on database-side controls for AI agents with direct SQL execute access. The first refused badly-shaped statements. The second queued schema changes for human review. The third scoped access by identity. The fourth audited whether that identity still deserved to be trusted. None of them ask the question this one does: even if a query is well-formed, authorized, correctly scoped, and issued by a trustworthy identity — is it expensive?

The incident that motivates it

A three-person agency ate a $14,000 AWS bill in a single day after attackers extracted static access keys and burned Claude invocations on Bedrock. Different layer than what this post covers — infrastructure billing, not database compute — but the same underlying failure shape as everything else in this series: nothing was checking cost before it was incurred, and at agent speed, that gap turns a small mistake into a large one before a human notices.

The same failure exists one layer down, at the query level, every time an agent runs something expensive before anyone checks what it will cost. A join missing its condition, a WHERE clause that defeats every available index — a human writing SQL by hand rarely produces these by accident more than once. An agent generating queries at machine speed produces them constantly, and by the time the query is running, the cost is already being paid.

Key takeaways

• A query can pass every check built earlier in this series — correct shape, authorized, correctly scoped, trustworthy identity — and still be the single most expensive thing that ran that day
• SQL Server can estimate a query’s cost before running it, via SET SHOWPLAN_XML — refuse anything over a threshold using that estimate alone, and the refusal itself costs milliseconds regardless of how expensive the query would have been
• Intuition about what’s expensive and what a cost-based optimizer actually finds expensive can point in different directions — my own assumption about which demo query would be the worst offender was wrong, and the real data is more useful than the guess would have been
• The right cost threshold isn’t a number to copy from someone else’s project — SQL Server’s cost units are relative to the optimizer’s internal accounting, not wall-clock time or dollars, and have to be calibrated against real data volume

What was actually built

The core of it: get the estimated plan without running the query, read its cost, decide.

def _estimate_cost(self, sql: str) -> float:
    with self.engine.connect() as conn:
        conn.execute(text("SET SHOWPLAN_XML ON"))
        try:
            result = conn.execute(text(sql))
            plan_xml = result.scalar()
        finally:
            conn.execute(text("SET SHOWPLAN_XML OFF"))
    return _parse_estimated_cost(plan_xml)

def run(self, sql: str, agent_id: str, intent: str) -> GovernedQueryResult:
    estimated_cost = self._estimate_cost(sql)

    if estimated_cost > self.max_estimated_cost:
        self._audit(agent_id, intent, sql, estimated_cost, None, None, "BLOCKED_COST")
        raise CostGovernorBlocked(
            f"Refused: estimated cost {estimated_cost:.2f} exceeds "
            f"{self.max_estimated_cost:.2f}. The query was never executed."
        )

    # only reached if the estimate was within budget
    start = time.perf_counter()
    with self.engine.connect() as conn:
        result = conn.execute(text(sql))
        rows = result.fetchall()
    elapsed_ms = (time.perf_counter() - start) * 1000
    ...

SET SHOWPLAN_XML ON is the mechanism worth calling out specifically: subsequent statements on that connection return their estimated execution plan instead of executing. A blocked query here never touches a single row — a meaningfully different guarantee than the guardrail project earlier in this series, which runs a write inside a transaction and rolls back after measuring the actual row count. That approach already pays the execution cost for anything it later decides to refuse. This one never spends it.

Seeing the actual numbers, including the one I got wrong

Four scenarios against a 100,000-row orders table: a cheap indexed lookup, a query whose WHERE clause defeats the only relevant index, a cross join missing its join condition entirely, and a legitimate full-table aggregate.

=== SCENARIO C: a join missing its condition -- a cartesian product ===
[analytics-agent] intent: Join accounts to their orders for a summary
          sql:    SELECT a.owner_name, o.amount FROM dbo.demo_accounts a CROSS JOIN dbo.demo_orders o
          -> BLOCKED: Refused: agent 'analytics-agent' submitted a query with estimated
             cost 2.26, exceeding the 1.50 threshold. The query was never executed --
             only its estimated plan was.
          (checked and refused in 11.4ms -- the query itself never ran)

That query, run in an earlier uncalibrated pass before I fixed the threshold, actually took 1,584 milliseconds and returned 400,000 rows. The governor refused it in 11.4 milliseconds. That comparison is most of the pitch by itself.

Here’s the part I got wrong going in: I built this demo expecting the non-sargable WHERE YEAR(order_date) = 2026 query — the one deliberately designed to defeat the only index on that column — to be the standout expensive scenario. It wasn’t. At 100,000 rows, a full table scan on narrow columns costs almost exactly the same, by SQL Server’s own accounting, as an index seek returning a similar row count (0.52 vs. 0.51 estimated cost). The actual outlier was the missing join condition, by a wide margin. My first threshold guess of 5.0 was wrong for a related reason — every single scenario passed under it, including the 400,000-row cartesian join, because SQL Server’s cost units are much smaller in absolute terms at this data volume than I’d assumed. I recalibrated to 1.5 based on the real observed numbers, not a second guess.

Worth stating plainly: don’t assume which query pattern is “the expensive one” without measuring it against real data. That’s the actual argument for building a cost governor around the optimizer’s own estimate instead of a simpler rule like “flag any WHERE clause without a matching index” — the cost-based optimizer’s model of what’s expensive is a better judge of that than intuition is, mine included.

What this doesn’t solve

SQL Server’s estimated cost is relative, not literal — it’s not wall-clock time or dollars, and the right threshold is specific to your schema, data volume, and hardware. Copying the 1.5 used in this demo directly into a production deployment without calibrating against real traffic would be a mistake of exactly the kind I made on the first attempt here. This also only measures magnitude, not intent — it can’t distinguish a legitimately expensive month-end report from a malicious one, and it doesn’t estimate cost for DDL statements at all (those are still the sibling guardrail and approval-queue projects’ job).

Across all five projects now: the firewall refuses what should never run. The approval queue routes schema changes to a human. Row-level security scopes what an agent can reach by identity. The credential auditor checks whether that identity still deserves trust. This refuses what would cost too much, regardless of whether every other check already passed. An AI agent with real database access needs checks at all five layers, because a query can clear four of them and still be the one that matters.

The full project — the SHOWPLAN_XML-based estimator, the calibration numbers, and all four scenarios runnable via Docker Compose against a real SQL Server 2025 instance — is open source: github.com/mrivanlima/ai-agent-query-cost-governor. If your agents can already run arbitrary queries and nothing is checking their cost before they run, that’s worth measuring before the bill does it for you. Get in touch if you want help calibrating this against your own environment.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

Hallucinated joins aren’t a sign the LLM is dumb. They’re what happens every time an agent gets asked to write SQL against a schema it has never actually seen — no catalog, no DDL, no foreign key map, just a table name typed into the prompt and a guess. Single-shot text-to-SQL fails on real enterprise schemas for the same reason a new hire fails when you hand them a business question and no data dictionary: not because they can’t reason, but because they’re reasoning over a schema that exists only in your head.

Why one-shot prompting can’t survive a real schema

The demo version of text-to-SQL looks convincing because demo schemas are small. Five tables, obvious column names, one sensible join path. Production schemas aren’t like that. They’re forty tables deep, with customer_id meaning three different things across three different systems, nullable foreign keys, and join paths that only make sense if you know which table got deprecated in 2019 but never dropped.

Ask a model to write SQL against that with nothing but a natural-language description of the schema, and it will do exactly what it’s built to do: produce the most statistically plausible query, whether or not that query matches the actual constraints in the catalog. It invents a customers.region column because regions feel like something a customer table would have. It joins on order_id instead of order_uuid because that’s the more common convention it’s seen in training data. None of that is a reasoning failure. It’s a grounding failure — the model was never given the one artifact that would have prevented it: the actual schema, queried at generation time, not summarized in a prompt.

Why “better prompting” doesn’t fix it either

The usual fix is to stuff more schema description into the prompt — table names, a few sample rows, a paragraph explaining the business logic. That helps, marginally, and it doesn’t scale. Every enterprise schema changes: columns get renamed, tables get partitioned, a nullable column becomes required after a migration. A static schema description in a prompt goes stale the first time someone runs ALTER TABLE, and nobody updates the prompt when that happens. You end up maintaining a second, shadow copy of your schema in English, by hand, forever — which is precisely the kind of manual synchronization problem database engineers have spent decades building tools to eliminate.

The actual fix isn’t better wording. It’s giving the agent the same thing you’d give a new engineer on day one: read access to the catalog itself, plus a way to find out when it’s wrong.

The database-first fix: execute, catch, diagnose, rewrite

This is a multi-step loop, not a single inference call, and every step maps to something a database engineer already does instinctively when a query fails:

1. The agent drafts a candidate query based on the user’s question and whatever schema context it has.
2. It executes that query against a sandboxed, read-only connection — never production, never with write privileges.
3. If execution fails, it catches the actual database error: column "region" does not exist, a constraint violation, a type mismatch.
4. It reads information_schema (or the engine’s equivalent catalog views) to resolve the real column names, types, and foreign keys for the tables in question.
5. It rewrites the query against what the schema actually says, and retries — with a hard cap on retry count so a genuinely bad question doesn’t loop forever.

MAX_ATTEMPTS = 4

def run_text_to_sql(question: str, session_id: str, agent_id: str) -> QueryResult:
    attempt = 0
    context = {}  # accumulates real schema facts as errors surface

    while attempt < MAX_ATTEMPTS:
        sql = draft_sql(question, context)
        try:
            result = execute_readonly(sql)
            persist_ledger_state(session_id, agent_id, {
                "final_sql": sql, "attempts": attempt + 1, "status": "success"
            })
            return result
        except DatabaseError as err:
            attempt += 1
            missing = resolve_from_information_schema(err, sql)
            context.update(missing)  # real column names, types, FK targets
            persist_ledger_state(session_id, agent_id, {
                "last_error": str(err), "attempts": attempt, "status": "retrying"
            })

    raise UnresolvableQueryError(question, context)

That persist_ledger_state call isn’t decorative. This retry loop has state that matters across attempts — how many tries have been spent, what schema facts have already been resolved, what the last error was — and that state belongs in the same place I described in the first post in this series: the Agent State Ledger. A retry loop that keeps its progress in a local variable loses everything on a restart mid-loop and starts back at attempt one with no memory of what it already ruled out. A retry loop that writes its progress to durable state resumes exactly where it left off. Same failure mode as agent memory generally, just at a smaller scale.

Recent research backs the shape of this loop, not just the intuition behind it. MAC-SQL’s Refiner agent executes a candidate query, observes the actual error or an empty result set, and rewrites accordingly, rather than trying to reason its way to a correct query in a single pass (arXiv:2312.11242). CHESS goes further for large, enterprise-scale schemas, pairing a Schema Selector that prunes an oversized catalog down to the relevant sub-schema with a Unit Tester that validates candidate queries before they’re trusted (arXiv:2405.16755). Across this line of work, using the database’s own execution feedback — a real error message, not a model second-guessing itself — is consistently what makes the correction loop actually converge instead of drifting into a different wrong answer.

The analogy

A self-correcting SQL agent reading its own execution traceback and retrying is functionally what a query optimizer does when a plan fails: it doesn’t sit and philosophize about why the plan might be suboptimal, it captures what actually happened, diagnoses the specific cause, and recompiles a new plan against ground truth. Nobody would trust an optimizer that picked a plan once and refused to ever look at execution statistics again. We shouldn’t trust a text-to-SQL agent that works the same way.

Practical guidance

• Never let the agent execute against production or with write privileges during query drafting. A sandboxed, read-only connection is non-negotiable — this loop will generate plenty of invalid SQL on the way to valid SQL.
• Query information_schema (or your engine’s catalog views) live, at generation time. Don’t hand the model a static schema summary and hope it stays current.
• Cap retries explicitly. A question the agent can’t resolve in four attempts against the real catalog usually means the question itself is ambiguous, not that attempt five will succeed.
• Persist retry state — attempt count, resolved schema facts, last error — to durable storage, not a local variable, so a mid-loop restart doesn’t erase progress.
• Log every failed attempt and its real database error. That log is what tells you whether your schema is genuinely hard to query or whether your agent’s prompt context is stale.

Key takeaways

• Hallucinated SQL isn’t an LLM defect — it’s what happens when an agent generates queries against a schema it’s never actually seen at the DDL/catalog level.
• Static schema descriptions in prompts go stale the moment the real schema changes; querying information_schema live doesn’t.
• The fix is a bounded loop: draft, execute against a sandboxed connection, catch the real error, resolve against the catalog, rewrite, retry.
• This loop needs durable state across attempts — the same Agent State Ledger pattern from post one, not a variable that disappears on restart.
• This is post two of three in Database-First Agent Architecture. Post three covers the governance boundary this loop depends on: exactly what an autonomous agent is allowed to execute once it’s confident in a query, building on the access-control ground covered in AI Agent Guardrails for Databases.

If your agents are guessing at your schema instead of reading it, that’s a fixable architecture problem, not a prompting problem. Get in touch or see how we approach it on our services page.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

Most database and warehouse billing was built around a traffic shape that no longer describes how your systems get used: a moderate number of longer-running queries from a relatively small number of human sessions. AI agents query differently — many more requests, each shorter, arriving in unpredictable bursts — and when that pattern meets a billing model with per-query minimums or coarse-grained compute increments, the minimum itself becomes a cost multiplier nobody budgeted for. A recent review of 127 enterprise agentic AI implementations found 73% went over budget, some by more than 2.4x, and the database and data-platform layer is a disproportionate and under-tracked share of why.

What’s actually happening

Two things are colliding at once. The first is agent adoption outpacing the billing models built to meter it. McKinsey’s 2026 research on agentic AI system performance found that a large share of agentic cost — 60% in their sample — comes from response refinement and retry loops, not raw model inference, and that cost driver is frequently invisible to teams tracking aggregate spend instead of per-agent, per-query attribution. The second is that the database layer specifically has its own version of this problem: billing minimums.

Cloud data warehouses and managed databases commonly bill in fixed increments — a minimum charge per query, a minimum compute-second block, a minimum container spin-up cost — because that pricing model was designed around infrequent, longer-running human or batch workloads where the minimum rarely binds. Agent workloads invert that assumption. An agent orchestrating a multi-step task might issue dozens of short lookups per user request, each one individually trivial but each one tripping the same billing floor a much larger human-driven query would trip once. Analysis of serverless and warehouse pricing models in 2026 makes the mechanism explicit: billing minimums act as a cost multiplier that’s easy to underestimate unless total cost of ownership is calculated against actual query-duration distributions rather than average load — and agent traffic is defined by exactly the kind of spiky, bursty distribution that average-load math misses.

Layer on top of this a second, related finding: Gartner’s 2026 Hype Cycle for Agentic AI names “FinOps for agentic AI” as an emerging category in its own right — a signal that the market itself has recognized existing cost-governance tooling doesn’t cleanly cover agent-driven infrastructure spend yet. The database layer is squarely inside that gap, because most FinOps tooling built for cloud cost management was designed to allocate spend to teams and services, not to the specific agent, task, or orchestration step that triggered a given burst of queries.

Who this affects

Data platform and DBA teams own the choice of billing model, instance sizing, and query-batching strategy — the levers that actually determine whether agent traffic hits a billing minimum repeatedly or gets consolidated into fewer, larger operations. They’re also usually the first to notice the symptom (a cost anomaly on the database line item) without necessarily knowing which agent or workflow caused it.

FinOps and cloud cost teams own cost attribution and budget forecasting, and this is precisely the blind spot the FinOps Foundation’s State of FinOps 2026 report describes: AI pricing models based on tokens, inference requests, and now agent-triggered database calls don’t map cleanly onto cost-allocation frameworks built for traditional infrastructure, and 98% of FinOps practitioners are now actively working this problem compared to under a third two years ago.

Engineering leadership and CTOs own the budget line that absorbs the overrun, and they’re the ones who need to ask a question most teams currently can’t answer: what does it cost, per agent workflow, to hit the database — not what does the database cost in aggregate this month.

Product and platform teams building agent-facing features are the ones whose design choices — how chatty an agent’s tool-calling pattern is, whether it retries liberally, whether it batches lookups — directly set the query-volume dial that determines whether billing minimums matter or don’t.

When this becomes real

This isn’t a future risk — it’s already showing up in the numbers. The 73%-over-budget figure and the 2.4x overrun cases reflect implementations running today, not a hypothetical. What’s still developing is the tooling and discipline to fix it.

Already happening: any organization running agents against a metered database or warehouse today is already paying whatever premium its billing model’s minimums impose on spiky traffic — most just haven’t isolated that line item from the rest of their AI spend yet.

Near-term, 2026-2027: expect two things to mature roughly in parallel — cloud and database vendors introducing pricing tiers or batching features explicitly aimed at agent traffic patterns (some, like consumption-based serverless databases with sub-second billing granularity, already exist and are a partial answer), and FinOps tooling catching up on agent-level cost attribution, per Gartner’s naming of it as an emerging Hype Cycle category this year. Neither is fully mature yet, which means the gap between what agents cost and what teams can currently measure will likely widen before it narrows.

Longer horizon, 2028 and beyond: as agent-to-agent and multi-agent orchestration patterns become more standard (rather than single-agent-to-database), query fan-out will increase further, and billing models that haven’t adapted by then will impose a compounding, not linear, cost penalty on organizations that haven’t restructured how their databases meter agent traffic.

How this actually plays out in a database environment

The mechanism is straightforward once you isolate it, which is exactly why it’s easy to miss when you’re looking at an aggregate bill.

Take a warehouse or managed database with, say, a 10-second minimum billable compute block per query and a cold-start cost for spinning up compute if the system has been idle. A human analyst running a handful of substantial queries an hour rarely triggers that minimum in a way that matters — their queries often run longer than the minimum anyway, and the minimum is a rounding error relative to the query’s actual cost. An agent orchestrating a customer-support workflow might issue a lookup to check order status, another to check inventory, another to check a return policy table, another to log the interaction — four or more short queries, each taking a fraction of a second of actual compute, each billed at the 10-second minimum. Multiply by however many workflow steps an agent takes per user interaction, by however many interactions per hour, and the minimum — not the actual compute consumed — becomes the dominant cost driver.

This compounds with retrieval-augmented generation specifically: vector database costs scale with retrieval volume, and over-retrieval — an agent pulling more context than a task actually needs, often because prompt engineering erred toward “retrieve broadly to be safe” — generates additional similarity-search queries on top of the workflow’s core database calls, each one subject to the same minimum-billing dynamic.

The failure mode compounds further because of orchestration depth. A single user-facing request today can trigger an orchestrator, multiple retrieval calls, several tool invocations, and possibly sub-agent delegation, each layer capable of hitting the database independently and each one invisible to a cost dashboard that only shows total database spend for the month. That’s the attribution problem McKinsey’s research names directly: the cost driver is buried in the agent graph, not visible at the aggregate billing layer, which means the team that could actually fix it — by batching the four lookups into one, or caching the return-policy table lookup that never changes — often doesn’t know it needs to.

Actions to take now

Start with visibility, because you can’t fix a cost driver you can’t see, and work toward architectural changes as the pattern becomes clear.

1. Pull query-duration and query-count distributions for your database and warehouse workloads, not just monthly spend totals. Look specifically for a high volume of very short queries clustered near your billing model’s minimum threshold — that’s the signature of agent-driven billing-minimum multiplication, and most cost dashboards don’t surface it by default.

2. Tag or attribute database calls to the agent, workflow, or orchestration step that triggered them, even if it’s a rough first pass using request headers or a correlation ID threaded through your agent framework. You need per-workflow cost, not just per-service cost, to find where the multiplier is actually happening.

3. Audit for redundant or over-broad retrieval in any RAG-backed agent workflow — cases where an agent queries a vector store or lookup table more broadly, or more often, than the task strictly requires. Tightening retrieval scope is usually the cheapest fix available and doesn’t require any infrastructure change.

4. Batch what can be batched. Many multi-step agent workflows issue several small, independent lookups that could be combined into one query or one round-trip. This is standard query optimization, but it matters more now because the cost of not doing it scales with agent call volume in a way it never did with human traffic.

5. Evaluate your database and warehouse billing model against your actual (not average) traffic shape. If your provider bills in coarse minimums and your workload is agent-driven and spiky, compare it against consumption-based or sub-second-granularity alternatives — the cost delta at agent-scale query volume can be substantial, and several providers now offer pricing tiers built explicitly for this pattern.

6. Build agent-level cost attribution into your FinOps practice now, not after the first budget overrun. This is the specific gap the FinOps Foundation and Gartner are both flagging as unresolved in 2026 — getting ahead of it means your organization isn’t discovering the multiplier effect for the first time in a quarterly cost review.

Key takeaways

• A 2026 review of 127 enterprise agentic AI implementations found 73% went over budget, some by more than 2.4x, and database-layer costs are a disproportionately under-tracked contributor.
• Billing models with per-query minimums or coarse compute increments were built for infrequent, longer-running human traffic — agent workloads are the opposite shape: frequent, short, and bursty — and the minimum itself becomes a cost multiplier.
• McKinsey research attributes 60% of agentic cost to response refinement and retry behavior that’s typically invisible without agent-level, not aggregate, cost attribution.
• Gartner’s 2026 Hype Cycle for Agentic AI names “FinOps for agentic AI” as an emerging category, signaling that existing cost-governance tooling doesn’t yet cleanly cover this gap.
• The cheapest fixes — retrieval scope audits and query batching — require no infrastructure change and are available today; billing-model changes and full agent-level attribution take longer but close the gap for good.

If your database costs are climbing faster than your agent workload growth explains, that mismatch is worth investigating before it shows up as next quarter’s budget overrun. Get in touch if you want help finding where it’s hiding.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

Use pgvector if you’re already running Postgres and your workload stays under roughly 10 to 50 million vectors with moderate query volume – it’s cheaper, it’s one less system to operate, and 2026 benchmarks show it’s no longer the performance compromise it used to be. Reach for a purpose-built vector database like Qdrant, Milvus, or Pinecone once you’re past that range, need sub-10ms p95 latency at high query-per-second rates, or require advanced filtering and multi-tenant isolation that a general-purpose relational engine wasn’t built for. The rest of this post is the reasoning behind that line, and where it actually sits for your workload.

Why this decision keeps coming up

Every team adding retrieval-augmented generation, semantic search, or a recommendation feature to an existing product hits the same fork: bolt vector search onto the Postgres database that’s already running production traffic, or stand up a dedicated vector database next to it. Two years ago the answer leaned heavily toward “dedicated” – pgvector’s HNSW implementation was new, slow to build, and fell over past a few million rows. That’s no longer the full picture, and treating it as settled either way costs teams real money and real latency.

What changed: pgvector closed the performance gap

The biggest shift is pgvectorscale, Timescale’s extension that adds StreamingDiskANN indexing and statistical binary quantization on top of pgvector. In Timescale’s own benchmark, Postgres with pgvectorscale hit 471 queries per second at 99% recall against 50 million vectors, versus 41.47 QPS for Qdrant on the same hardware – an 11.4x throughput advantage, with 28x lower p95 latency than Pinecone’s storage-optimized index at matching recall (Tiger Data, “pgvector vs. Qdrant”). That’s a vendor-run benchmark – Timescale built pgvectorscale and published the numbers – so treat the magnitude with appropriate skepticism, but the direction is consistent with what other teams have independently reported: the raw performance gap that used to make “just use a real vector database” the safe default answer has mostly closed for mid-scale workloads.

Binary quantization is doing a lot of that work. AWS’s own guidance for Aurora PostgreSQL shows the same pattern: quantizing vectors to binary representations cuts memory footprint dramatically and keeps HNSW’s RAM ceiling from becoming the limiting factor as dataset size grows (AWS Database Blog). Without quantization, HNSW keeps the full graph in memory, and that’s the mechanism behind most “pgvector fell over” stories from 2024.

Where pgvector still runs out of road

None of that erases pgvector’s real ceiling. Each vector(1536) value takes roughly 6KB on disk before row overhead and index cost, so 100 million rows of raw vector storage alone exceeds 600GB before you’ve built an index against it (ParadeDB, “pgvector Limitations”). pgvector also only supports HNSW and IVFFlat as index types – IVFFlat builds fast but degrades as the dataset grows, and HNSW gives good recall but has long build times and high memory demand during construction. Single-table PostgreSQL limits (32TB per table, TOAST object size ceilings) mean billion-vector deployments need partitioning strategies that most teams aren’t set up to maintain. And critically, none of pgvector’s scaling techniques change the fact that it’s still bound by Postgres’s general-purpose query planner and storage engine – it was never rebuilt from the ground up around approximate nearest-neighbor search the way Qdrant’s Rust engine or Milvus’s distributed architecture were.

If you’re already tuning HNSW and IVFFlat indexes on a relational workload, you’ve likely felt this tension firsthand: the index type that gives you the best recall is also the one most likely to blow past your maintenance_work_mem budget during a rebuild.

Key takeaways

• Under ~10M vectors, on Postgres already: pgvector wins on cost and operational simplicity – no new system, no new failure mode to monitor.
• 10M-50M vectors, with pgvectorscale/quantization tuning: pgvector is now genuinely competitive on throughput, not just “good enough.”
• Past 50M-100M vectors, or high sustained QPS: purpose-built databases pull ahead, and self-hosting one becomes cost-justified once a managed option like Pinecone would otherwise run well past $5,000/month at that scale.
• Advanced filtering, hybrid search, or strict multi-tenant isolation: purpose-built engines like Qdrant were designed around these patterns; pgvector supports them but with more manual query engineering.
• The index type matters as much as the database: IVFFlat vs. HNSW vs. DiskANN changes build time, memory ceiling, and recall independently of which database holds them.

What actually drives the cost line

The economics flip at a predictable point. A team already paying for Postgres gets pgvector close to free at low scale – no new infrastructure bill, no new operational surface. But HNSW’s memory requirements scale with vector count, and once a workload needs a large, RAM-heavy instance just to hold the index, the “database you already run” line item quietly turns into a dedicated, expensive machine anyway. At that point the comparison isn’t “free vs. paid,” it’s “one expensive Postgres instance vs. one purpose-built system” – and the purpose-built system usually wins on throughput per dollar once you’re actually paying real infrastructure cost either way.

This is also where embedding hygiene starts to matter more, not less. A system carrying tens of millions of vectors across a partitioned, quantized index is much harder to audit for drift when embedding models change underneath it than a small, single-table pgvector setup – scale amplifies the blast radius of a silent re-embedding mistake.

The actual decision framework

Ask four questions, in this order: How many vectors, today and in 18 months? What’s the sustained query volume, not the peak demo number? Does the workload need hybrid search, complex metadata filtering, or hard multi-tenant isolation? And is there already a team that owns Postgres operations, or would a vector database be the first new piece of infrastructure this team has run? Most teams answering honestly land on pgvector for anything under 10-20 million vectors with a Postgres-literate team already in place, and shift the calculus toward a purpose-built system only when the vector count, query volume, or filtering complexity genuinely outgrows what a well-tuned Postgres extension can carry.

If you’re mid-migration and unsure which side of that line your workload actually sits on, get in touch – this is exactly the kind of tradeoff that’s cheap to model before committing infrastructure spend to it.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

This is the fourth piece in a series on database-side controls for AI agents with direct SQL execute access. The first post refused badly-shaped statements outright. The second queued schema changes for human review instead of refusing them outright. The third scoped what an agent could see and touch based on its declared identity — and closed with an honest caveat I want to pick up here: Row-Level Security enforces scope given an honestly declared identity. It doesn’t authenticate that identity, and it doesn’t check whether the identity’s grant is still appropriate. This post is what closes that gap.

The incident underneath the incident

The Mexican government breach that motivated the row-level-security post — nine agencies, 195 million taxpayer records, 220 million civil records, December 2025 through February 2026 — had a root cause underneath its root cause. The excessive, shared permissions that made the breach possible didn’t appear the day it happened. They existed for a long time before that, unaudited, because nobody was checking whether a grant made sense against how it was actually being used.

That’s not a hypothetical pattern. A 2026 least-privilege research report analyzing over 3 billion permissions found that on average only about 4% had been used in the trailing 90 days, while nearly one in three could modify or delete sensitive data. And per IBM’s Cost of a Data Breach Report 2025, 97% of AI-related breaches involved organizations lacking proper AI access controls — not lacking access controls generically, lacking ones scoped and audited for AI and agent identities specifically. Nobody had to break in cleverly the first time either. The permissions were just never checked again after they were granted.

Key takeaways

• Row-level security and statement-shape guardrails both assume the calling identity is trustworthy — none of the controls earlier in this series verify that assumption or check whether a grant is still appropriate
• Treat agent credentials the way DBAs already treat human access reviews: register what was granted, log what’s actually used, and periodically audit the gap between the two
• Three checks catch three distinct failure modes: agents sharing one underlying credential (a single compromise takes out every agent using it), credentials nobody has rotated, and broad grants an agent’s actual activity never exercises
• This is the first project in the series that isn’t a runtime gate — it’s a detective control, a periodic review, not a preventive one blocking anything in real time

What was actually built

A registry of declared agent identities, an activity log of what they actually do, and three audit checks run against the gap between them:

def audit_shared_credentials(self) -> list:
    """Flags any credential_login used by more than one declared agent identity."""
    with self.engine.connect() as conn:
        rows = conn.execute(text(
            "SELECT credential_login, agent_id FROM dbo.agent_registry "
            "ORDER BY credential_login, agent_id"
        )).fetchall()

    by_login = {}
    for r in rows:
        by_login.setdefault(r.credential_login, []).append(r.agent_id)

    return [
        SharedCredentialFinding(credential_login=login, agent_ids=agents)
        for login, agents in by_login.items() if len(agents) > 1
    ]

The scope-creep check is the one I’d point to as the most directly useful in practice: for every agent granted the broadest scope, it compares that grant against the regions its activity log actually shows it touching, and reports what percentage of the grant has simply never been exercised — the same 4%-utilization finding from the research above, computed against this project’s own data instead of cited as an external statistic.

Seeing it flag real patterns

Five registered agents — three deliberately sharing one credential, one with a credential nobody’s rotated in 270 days, two granted the broadest scope — after three weeks of simulated activity:

--- Shared-credential findings ---
  RISK: credential 'svc_shared_agent_login' is shared by 3 declared agents:
        billing-agent, migration-agent-west, support-bot-east
        A single compromised or leaked credential here compromises all 3
        agent identities at once.

--- Stale-credential findings (>180 days since rotation) ---
  RISK: agent 'ops-admin-agent' credential 'svc_ops_admin_login' last
        rotated 270 days ago.

--- Scope-creep findings (granted 'ALL', actual usage narrower) ---
  RISK: agent 'ops-admin-agent' declared scope=ALL, but activity history
        only ever touches: US-EAST, US-WEST
        33% of the granted region scope has never actually been used.
  RISK: agent 'reporting-agent' declared scope=ALL, but activity history
        only ever touches: US-EAST
        67% of the granted region scope has never actually been used.

Worth being precise about the shared-credential check’s mechanics, because a technically sharp reader will ask: it audits the registry’s declared credential label, not the raw SQL login SQL Server itself observes via SUSER_SNAME(). In this demo — and in most real deployments routing agents through one shared MCP gateway connection — every agent’s observed SQL login is identical by construction, which is itself the exact anti-pattern being flagged. A deployment provisioning real, distinct SQL logins per agent would get the same signal directly from SUSER_SNAME() instead of the registry.

The honest, slightly anticlimactic lesson

Every other project in this series had a real bug story: a clock-drift issue in the point-in-time restore, a future-dated post silently dropped, a wrong assumption about how a block predicate fails. This one worked correctly the first time I ran it against SQL Server. That’s worth naming rather than manufacturing drama for consistency — the reason is structural. This project never touches SQL Server’s transactional or predicate machinery; it’s plain table reads and threshold comparisons in Python. That category of code is inherently easier to get right than anything involving transaction boundaries or predicate timing, which is exactly what the earlier three projects kept tripping on.

That’s part of the actual pitch for building this one, not an afterthought: detective controls are cheaper to get right than preventive ones, and this is the control that tells you whether the other three are even pointed at the right identities in the first place. If you’re deciding where to start hardening AI agent database access and can only build one thing first, an audit of what’s already been granted is a reasonable place to begin — before the guardrail, the queue, or the row-level policy, because all three of those inherit whatever the credential layer already got wrong.

What this doesn’t solve

This finds the gap; it doesn’t close it. Narrowing an over-broad grant or rotating a stale credential is still a human decision this project deliberately doesn’t automate, the same way the schema-approval project surfaces DDL requests rather than resolving them unilaterally. And the shared-credential check specifically depends on the registry being maintained honestly — if nobody updates it when a new agent starts reusing an old credential, the audit won’t catch what it was never told about.

Across all four projects now: the firewall refuses what should never run. The approval queue routes what might be legitimate to a human. Row-level security scopes what an agent can reach by identity. This audits whether that identity — and its grant — still deserves to be trusted at all. An AI agent with real database access needs checks at every one of those layers, because none of them alone covers what the others miss.

The full project — the registry, activity log, and all three audit checks runnable via Docker Compose against a real SQL Server 2025 instance — is open source: github.com/mrivanlima/ai-agent-credential-auditor. If you can’t currently answer “which of our agents’ grants have actually been used in the last 90 days,” that’s worth finding out before an audit happens under worse circumstances. Get in touch if you want help running this against your own environment.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

If you’re a DBA watching the AI hiring wave and wondering where you fit, here’s the direct answer: you’re not being replaced, you’re being asked to run a bigger system than the one you already run. Multi-agent AI systems are, underneath the marketing, state management, transaction persistence, and retrieval-index problems — the exact domain you’ve spent your career in. Python is the wiring. The database is the machine.

The misconception this post is correcting

Most agent frameworks are built by people who’ve never designed a schema under concurrent load. That’s not a jab — it’s a description of the hiring pipeline. Companies staffing “AI engineering” teams default to full-stack and Python developers, because Python is the language every agent framework — LangChain, LangGraph, CrewAI, AutoGen — ships its interface in. So the assumption becomes: agentic AI is a software engineering discipline, and databases are a backend detail someone else handles.

That assumption breaks the moment an agent system leaves the demo. A demo runs one agent, one user, one session, no concurrent load, no failure recovery, no audit trail. Production runs many agents, many sessions, retries, partial failures, and multiple writers touching shared state at the same time. Every one of those requirements has a name, and the name is “database problem.” Transaction persistence, concurrent access management, retrieval consistency, access control across tenants — none of it is Python’s job. Python calls the functions. It doesn’t own the guarantees.

Why the app-layer view falls apart under real load

Watch what actually happens when a multi-agent system scales past a demo. An agent needs to remember what it did three turns ago — that’s session state, and state that doesn’t survive a restart is a data durability problem, not a prompting problem. Two agents need to read and write a shared plan without stepping on each other — that’s concurrency control, the same problem two bank tellers hitting the same account balance have solved for fifty years with row locks and isolation levels. An agent needs to route a query to the right knowledge source out of several — that’s a routing and indexing problem, the same shape as a query planner choosing an index.

Vector databases make this concrete. Raw FAISS — the library most tutorials reach for first — has no API endpoints, no concurrent access management, no multi-tenancy, and no access control built in. It’s an index, not a database. The moment more than one agent needs to query it safely at the same time, or different agents need scoped access to different slices of the corpus, someone has to build the database layer FAISS doesn’t have: connection handling, isolation, permissions (StarTeck Manchester, “Vector Databases in 2026”). Teams that reach for ChromaDB, Qdrant, or Weaviate instead of raw FAISS in production are making exactly the decision a DBA would make: don’t build a database from scratch when correctness matters. Agentic RAG systems increasingly route queries between multiple retrieval strategies — semantic search, keyword match, hybrid — based on confidence scores, with agents passing work to each other the way a query optimizer chooses between an index scan and a table scan based on cost (Towards Data Science, “Multi-Agent SQL Assistant”).

You Already Know This

That’s the thesis of this series, and it’s worth naming outright: the skills agentic AI needs aren’t new. They’re the ones you already have, wearing unfamiliar names.

• Session memory that survives a restart is durable state — the same problem a database solves every time a server reboots mid-transaction.
• Two agents editing shared plan data is concurrency control — locking, isolation levels, optimistic version checks. Nothing here that MVCC didn’t already solve.
• Routing a query to the right retrieval source is query planning — picking the cheapest correct path to an answer based on cost and selectivity.
• Deciding which agent can read or write which data is access control — grants, roles, row-level security, the stuff you enforce daily and most application developers have never had to think about.

Each of the next two posts in this series works through one of these translations in detail: Pydantic’s structured validation as relational schema constraints in disguise, and LangGraph’s checkpointing as a write-ahead log wearing a new label. This post is the thesis. The next two are the proof. For a concrete look at what happens when agent state isn’t given a durable home in the first place, see Why Your AI Agents Keep Crashing — the same failure mode, viewed from the state-durability side rather than the retrieval side.

The analogy

An agent framework without a database architect designing its state layer is a query optimizer with no statistics — it will run, it will even return answers, and it has no idea whether the path it picked is fast, safe, or correct. It’s making decisions blind. A DBA sitting on that architecture isn’t a nice-to-have; they’re the missing statistics table.

Practical guidance

If you’re a DBA looking at the agentic AI wave and wondering where to plant a flag:

• Don’t learn Python to become a “real” AI engineer. Learn enough Python to read what an agent framework is doing to your data, then fix the data layer underneath it — that’s the leverage move.
• Ask, for any agent system you’re brought in on, three questions: where does state live when a process dies, what happens when two agents write at once, and who’s allowed to read what. If nobody can answer cleanly, that’s your opening.
• Treat vector databases as databases, not as a separate AI-only category. Concurrency, indexing strategy, and access control apply the same way they do to any other index.
• Push back on the framing that Python developers own “the AI layer” and DBAs own “the storage layer.” In a system that has to survive production, those are the same layer.

Key takeaways

• Multi-agent AI systems are state management, transaction persistence, and retrieval-index problems wearing new terminology — not a distinct engineering discipline.
• Python is the orchestration glue connecting agents to databases. It is not the architecture itself.
• Raw vector index libraries like FAISS lack the database fundamentals — concurrency control, multi-tenancy, access control — that production multi-agent systems require.
• Agentic RAG routing between retrieval strategies is functionally query planning: choosing the cheapest correct path based on cost.
• This is post one of three in “You Already Know This” — Pydantic-as-schema-constraints and LangGraph-checkpointer-as-WAL are next.

If your team is staffing an “AI engineering” effort and hasn’t put a database architect on it, that’s the gap. Get in touch or see how we approach it on our services page.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

A June 2026 executive order pulled the federal government’s post-quantum cryptography deadline forward by four to five years — key establishment must be migrated by December 31, 2030, not 2035. That timeline compression matters to every database team, not just federal contractors, because the underlying threat it responds to doesn’t check who your customer is. If your database holds data that needs to stay confidential past 2030, an adversary can copy your encrypted backups today and simply wait.

What’s actually happening

On June 22, 2026, the White House signed an executive order that rewrites the government’s post-quantum cryptography (PQC) timeline. The prior target, set by 2022’s National Security Memorandum 10, gave agencies until 2035 to migrate high-value and high-impact systems. The new order splits that into two hard dates: key establishment — the cryptography that protects data in transit and at rest, including most database-level encryption — must move to quantum-resistant algorithms by December 31, 2030, and digital signatures by December 31, 2031. The Federal Acquisition Regulatory Council has 180 days to write a rule extending the December 2030 deadline to “covered contractors,” which in practice means most software and infrastructure vendors selling into government.

The order didn’t invent new cryptographic requirements — it accelerated the clock on standards that already existed. NIST finalized its first three PQC standards (FIPS 203, 204, and 205) in August 2024, and NIST Internal Report 8547 already targets 2030 for deprecating RSA and ECC — the two algorithm families underpinning the vast majority of production database encryption today, from TLS connections to transparent data encryption (TDE) key wrapping to column-level encryption. CNSA 2.0 already requires national security systems to begin adopting quantum-resistant cryptography starting in 2027. The executive order is a forcing function, not a new invention — but forcing functions are exactly what change budget priorities.

The reason this can’t wait for 2030 to become urgent is a specific attack pattern with an unglamorous name: harvest now, decrypt later (HNDL). An adversary doesn’t need to break your encryption today. They only need to copy the encrypted bytes — a stolen backup, an intercepted replication stream, an exfiltrated export — and hold them until a cryptographically relevant quantum computer exists to break RSA or ECC retroactively. Three papers published between May 2025 and March 2026 reduced the estimated quantum resource requirement to break RSA-2048 from roughly 20 million qubits to under one million, with some architectures suggesting a path toward 100,000 qubits. That doesn’t mean a quantum computer capable of the attack exists yet — it doesn’t — but it means the theoretical resource bar keeps dropping faster than most risk models assumed when they were written.

Who this affects

This is squarely a database and security leadership problem, not a cryptography-researcher problem. Three roles carry direct responsibility:

Database administrators and platform engineers own the actual encryption configuration — TDE settings, key management service integration, column-level and always-encrypted configurations, backup encryption, replication encryption. They’re the ones who will need to know, concretely, which algorithm every one of those settings currently uses, because “SQL Server handles that” or “it’s whatever the cloud provider defaults to” won’t survive a real audit.

CISOs and security architects own the crypto-agility strategy and the migration roadmap, and they’re the ones who’ll be asked by a board or a regulator whether the organization has a plan — a question that requires an inventory to answer honestly.

Compliance and legal teams own the exposure calculation for data with a multi-decade confidentiality requirement: health records, financial account data, government contract data, trade secrets, anything under a data-retention mandate that extends past 2030. If that data is sitting in a database encrypted with RSA-2048 or ECC today, its confidentiality window may already be shorter than its retention requirement.

Notably, none of this is federal-only. The FAR contractor rule extends the 2030 deadline contractually to any vendor selling into government, and the EU’s own PQC roadmap pushes member states and regulated industries to begin migration activity by the end of 2026 — well ahead of the U.S. federal timeline. If your organization sells software, holds government contracts, operates in a regulated industry, or simply stores data with a retention window past 2030, this is your timeline too, whether or not an executive order names you directly.

When this becomes real

Break the timeline into three honest phases, because overstating urgency is as unhelpful as understating it.

Already happening: HNDL collection is not a future risk — assume any sufficiently motivated adversary (nation-state actors are the most credible threat model here) is already harvesting encrypted traffic and backups today, betting on future decryption capability. This phase requires no new quantum hardware to be a live risk; it only requires that your data’s confidentiality window outlasts the arrival of a cryptographically relevant quantum computer, whenever that turns out to be.

Near-term, 2026-2028: This is the planning and pilot phase, and it’s where most organizations sit right now. NIST’s own guidance frames 2026 as the year to move from planning to pilot-at-scale. Realistically, expect vendor and platform support for PQC key establishment (database engines, cloud KMS providers, HSM vendors) to mature substantially in this window — some of it already has, some is still catching up — which is exactly why an inventory now matters more than a migration now.

2029-2031 and beyond: This is when the deadlines bite. December 2030 for key establishment, December 2031 for digital signatures, under the new federal order — with the FAR contractor rule following close behind. Estimates for when a cryptographically relevant quantum computer could actually exist remain genuinely uncertain, ranging from the early 2030s to considerably later depending on the source — but the compliance deadlines don’t wait for that uncertainty to resolve, and neither should your migration plan.

How this actually plays out in a database environment

The mechanics matter more than the deadline, because “post-quantum cryptography” is not a single switch a DBA can flip.

Database encryption touches multiple independent layers, each with its own migration path: TLS/network encryption between application and database, transparent data encryption for data at rest, column-level or “always encrypted” schemes for specific sensitive fields, backup and snapshot encryption, replication-stream encryption, and the key management layer wrapping all of the above — often a cloud KMS or on-prem HSM using RSA or ECC key exchange under the hood. A migration plan has to account for all of them separately, because they don’t upgrade together, and some (backups, especially long-retention ones) are easy to forget entirely.

The most common failure mode is not resistance to PQC — it’s the discovery step. Cryptographic discovery is consistently the most time-intensive part of a PQC program, because most organizations have never built or maintained a complete inventory of where cryptography lives in their environment — which databases, which key vaults, which third-party integrations, which legacy systems nobody has touched in years but that still hold live, encrypted, long-retention data. A database that was provisioned five years ago with default TDE settings and has quietly accumulated sensitive customer records since is exactly the kind of system that gets missed, because nobody currently owns the question “what encrypts this.”

The second failure mode is architectural rigidity: systems where the cryptographic algorithm is hard-wired into the application or database configuration rather than abstracted behind a swappable interface. That’s the absence of crypto-agility — the ability to rotate algorithms without re-architecting the system around them — and it’s what turns a PQC migration from a configuration change into a multi-quarter engineering project. Systems built without crypto-agility in mind face exactly that kind of forklift upgrade when the deadline arrives, regardless of how much runway existed beforehand.

Legacy systems compound both problems. Older database engines, on-prem HSMs nearing end-of-support, and vendor-managed encryption with no visibility into the underlying algorithm are the hardest and slowest parts of any migration — and they’re disproportionately common in exactly the industries (healthcare, financial services, government) with the longest data-retention requirements and the most HNDL exposure.

Actions to take now

Start cheap and get more specific as the deadline gets closer. None of this requires waiting for PQC database features to be fully mature — the early steps are about knowing what you have.

1. Inventory every place cryptography touches your database environment. TLS certificates, TDE key wrapping, column-level encryption, backup encryption, replication encryption, and the KMS or HSM underneath each one. This is the single highest-leverage step and the one most organizations skip. Treat it as a living document, not a one-time audit — new deployments need to be captured continuously.

2. Flag data with a confidentiality window that extends past 2030. Health records, financial account data, long-retention government or legal data, trade secrets. This is your HNDL priority list — it needs migration attention before anything else, regardless of where it sits in a broader roadmap.

3. Ask your cloud provider and database vendor what their PQC roadmap actually is. Specifically: when does their KMS support PQC key establishment algorithms, and what’s the migration path for existing encrypted data. Get this in writing, not a sales conversation — you need dates, not intentions.

4. Separate your cryptographic policy from your application code. If algorithm choice is hard-wired into schema, connection strings, or application logic instead of abstracted behind a key-management interface, that’s the crypto-agility gap that turns this into a rebuild instead of a rotation. Fixing this now, before a hard deadline forces it, is dramatically cheaper.

5. Pilot PQC key establishment on one non-critical system in 2026 or early 2027. Don’t wait for a mandate to test how your stack actually behaves with hybrid classical/PQC key exchange — vendor support is uneven right now, and you want to find the gaps on a system that doesn’t matter before you find them on one that does.

6. Build the 2028-2030 migration budget and staffing plan this year, even if execution doesn’t start until next year. Discovery and pilots are the cheap phase; production migration across every database, backup set, and legacy integration is not, and a plan written under deadline pressure is worse than one written with runway.

Key takeaways

• A June 2026 executive order moved the federal PQC deadline for key establishment from 2035 to December 2030, with digital signatures following in December 2031.
• Harvest-now-decrypt-later means encrypted data with a long confidentiality window is at risk today, even though no quantum computer can currently break RSA or ECC.
• Recent research has lowered the estimated quantum-hardware bar for breaking RSA-2048, increasing urgency without changing the fact that a cryptographically relevant quantum computer doesn’t yet exist.
• Cryptographic discovery — knowing exactly where encryption lives across TDE, column-level encryption, backups, and replication — is consistently the hardest and most time-consuming part of a PQC program, and most database teams haven’t done it.
• Crypto-agility, not any single algorithm swap, is what determines whether this migration is a configuration change or a multi-quarter rebuild.

If your database environment doesn’t have a current cryptographic inventory, that’s the actual starting line for this deadline — not the migration itself. Get in touch if you need help building one.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

Rebuild an index when logical fragmentation is high and page density is low – not on logical fragmentation alone. avg_fragmentation_in_percent, the number most maintenance scripts check by default, measures whether pages are out of physical order, which barely matters on SSD/NVMe storage. avg_page_space_used_in_percent, the number most scripts never look at because it costs more to compute, measures how full each page actually is – and that’s the one that costs you in buffer pool memory and I/O regardless of storage type. This post reproduces realistic fragmentation from status-column churn on a 1.5-million-row table, shows how to read both signals correctly, applies an online rebuild with storage-appropriate thresholds, and closes with a scheduled way to catch the next index quietly headed the same way.

Key takeaways

• avg_fragmentation_in_percent measures logical fragmentation (pages out of order) – a weak signal on modern SSD/NVMe storage, where random and sequential I/O cost about the same
• avg_page_space_used_in_percent measures page density – how full each page is – and drives buffer pool and I/O cost on every storage type; it requires SAMPLED or DETAILED mode, not the LIMITED default most scripts use
• Fragmentation on a real table comes from UPDATEs on indexed columns causing page splits, not from inserts – an ever-increasing key like an identity column barely fragments on its own
• Thresholds should account for storage: REORGANIZE at 10-30% / REBUILD above 30% on spinning disks; on SSD/NVMe, many practitioners push those to 30% / 60% before it’s worth the resource cost
• AI is a good fit for scanning every index on an instance daily and ranking candidates by both fragmentation signals – but which threshold, fill factor, and maintenance window fit a given index stays a human call

The problem

An Orders table has a nonclustered index on OrderStatus, supporting the dashboard queries that filter orders by status. The table isn’t unusually large – about 1.5 million rows – and nothing about the schema is wrong. But a weekly maintenance job flags the index at 68% fragmented, a report gets forwarded, and someone schedules an emergency rebuild during business hours “before it gets worse.”

That reaction is common, and it’s usually not wrong to rebuild – but it’s frequently wrong about why. The 68% figure comes from avg_fragmentation_in_percent, which measures logical fragmentation: whether the physical order of pages on disk matches their logical order in the index. On a spinning disk, out-of-order pages mean the read head jumps around instead of sweeping in one direction, and that’s genuinely expensive. On the SSD or NVMe storage running most production SQL Server today – on-prem or cloud-managed – there’s no read head. Random and sequential I/O cost is close enough that logical fragmentation alone is a weak predictor of anything.

The number worth checking instead is avg_page_space_used_in_percent – page density, how full each 8KB page actually is. A table with plenty of empty space per page needs more pages to hold the same data, which means more memory to cache it in the buffer pool, more I/O to read it from disk, and larger backups. That cost is real on every storage type, and it’s the one most default maintenance scripts never surface, because computing it requires running sys.dm_db_index_physical_stats in SAMPLED or DETAILED mode instead of the faster LIMITED mode that ships as the default in Ola Hallengren’s popular maintenance solution and most third-party tools (Ola Hallengren, SQL Server Index and Statistics Maintenance).

Here’s what actually caused the 68%: nothing to do with the identity-column primary key, which only ever grows at the end and barely fragments regardless of maintenance. The cause is the OrderStatus index itself, sitting on a column that gets UPDATEd constantly as every order moves Pending -> Shipped -> Delivered. Each of those UPDATEs changes an indexed value, which means the row may no longer belong where it currently sits in the index – and when a page doesn’t have room for the row in its new sorted position, SQL Server splits the page in two. Repeat that across a million status transitions and you get exactly the pattern in this table: high logical fragmentation and low page density, both driven by the same underlying cause, but only one of them showing up in the default report.

The expert fix

Pull both DMV columns for the table, not just the one:

SELECT
    OBJECT_NAME(ips.object_id)          AS TableName,
    i.name                              AS IndexName,
    ips.avg_fragmentation_in_percent    AS LogicalFragPct,
    ips.avg_page_space_used_in_percent  AS PageDensityPct,
    ips.page_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Orders'), NULL, NULL, 'SAMPLED') ips
JOIN sys.indexes i ON i.object_id = ips.object_id AND i.index_id = ips.index_id
WHERE ips.index_level = 0
ORDER BY ips.avg_fragmentation_in_percent DESC;

On the reproduction table used for this post, IX_Orders_OrderStatus comes back around 68% logical fragmentation and roughly 60% page density – both signals agree something is worth fixing. That agreement matters: a high fragmentation number with page density still near 90% is a much weaker case for a rebuild than the same fragmentation number paired with page density down in the 50s or 60s.

Which action to take, and at what threshold, should account for storage. Microsoft’s long-standing documented guidance is REORGANIZE between 5% and 30% logical fragmentation, REBUILD above 30% – written with spinning disks in mind. On SSD/NVMe-backed instances, which describes most cloud-managed SQL Server today, that 5% floor triggers on noise, and practitioners have converged on pushing both thresholds higher: REORGANIZE around 10-30%, REBUILD above 30%, or even REORGANIZE at 30% and REBUILD at 60% on the fastest storage, always weighed against page density rather than fragmentation percentage alone (Erik Darling, Why SQL Server Index Fragmentation Isn’t a Problem on Modern Storage Hardware).

With the diagnosis confirmed, rebuild online with a deliberate fill factor:

ALTER INDEX IX_Orders_OrderStatus ON dbo.Orders
REBUILD WITH (ONLINE = ON, FILLFACTOR = 90, MAXDOP = 0);

ONLINE = ON (Enterprise Edition or Azure SQL Database) keeps readers and writers unblocked during the rebuild – necessary on a live table this size. FILLFACTOR = 90 leaves 10% of each page empty on purpose, so the next round of status-value UPDATEs has somewhere to land without immediately splitting the page again – fixing today’s fragmentation without guaranteeing tomorrow’s. A fill factor that’s too aggressive (60-70%) wastes memory and disk on empty space that never gets used; one that’s too tight (100%) guarantees the very next write on that page causes a split. The right number depends on how often that specific index takes updates – there’s no single correct value across every table.

Re-running the fragmentation query afterward should show logical fragmentation near 0 and page density near 90%, matching the fill factor. SET STATISTICS IO, TIME ON around a representative query against the table should show fewer logical reads than before the rebuild – fewer, fuller pages to scan for the same result. The full seed script and before/after evidence are in the companion example on GitHub.

For lighter cases in the REORGANIZE band, ALTER INDEX ... REORGANIZE is always online regardless of edition and works incrementally in place – the tradeoff is that it doesn’t update statistics, so pair it with an explicit UPDATE STATISTICS if the table’s had significant write volume.

The AI-automation angle

Reading the right DMV columns for one table by hand is manageable. Doing it across every index on an instance, on a schedule, and surfacing only the ones actually worth someone’s attention is exactly the kind of scanning-and-ranking work worth automating – while leaving the decision of what to run, and when, with a person.

A scheduled, read-only query can check both fragmentation signals for every index above a minimum size, apply storage-aware thresholds, and log candidates to a review table instead of acting on them directly:

DECLARE @StorageType VARCHAR(10) = 'SSD';
DECLARE @ReorgThreshold DECIMAL(5,2) = CASE WHEN @StorageType = 'SSD' THEN 30.0 ELSE 5.0 END;
DECLARE @RebuildThreshold DECIMAL(5,2) = CASE WHEN @StorageType = 'SSD' THEN 60.0 ELSE 30.0 END;
DECLARE @MinPageCount INT = 1000;
DECLARE @MinPageDensity DECIMAL(5,2) = 75.0;

INSERT INTO dbo.IndexFragmentationLog
    (DatabaseName, SchemaName, TableName, IndexName, LogicalFragPct, PageDensityPct, PageCount, RecommendedAction)
SELECT
    DB_NAME(), s.name, t.name, i.name,
    ips.avg_fragmentation_in_percent, ips.avg_page_space_used_in_percent, ips.page_count,
    CASE
        WHEN ips.avg_fragmentation_in_percent >= @RebuildThreshold
             OR ips.avg_page_space_used_in_percent < @MinPageDensity THEN 'REBUILD'
        WHEN ips.avg_fragmentation_in_percent >= @ReorgThreshold THEN 'REORGANIZE'
        ELSE 'MONITOR'
    END
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, 'SAMPLED') ips
JOIN sys.indexes i ON i.object_id = ips.object_id AND i.index_id = ips.index_id
JOIN sys.tables t ON t.object_id = ips.object_id
JOIN sys.schemas s ON s.schema_id = t.schema_id
WHERE ips.index_level = 0
  AND ips.page_count >= @MinPageCount
  AND (ips.avg_fragmentation_in_percent >= @ReorgThreshold OR ips.avg_page_space_used_in_percent < @MinPageDensity);

Run daily against each database that matters, this produces a ranked queue: which indexes cross the threshold on either signal, ordered by severity. A human reviews the queue and decides, per index, whether the resource cost of a rebuild is worth it right now, what fill factor fits that index’s actual write pattern, and whether ONLINE = ON is even available on that edition. The watcher never issues ALTER INDEX itself – its job stops at producing a report nobody had to generate by hand. That’s the right boundary for this kind of automation: scanning every index on an instance and applying a consistent, storage-aware rule is tedious, mechanical work AI handles well; deciding the actual maintenance window and fill factor for a specific business-critical table is a judgment call that stays with whoever owns that workload. The full watcher query and a PowerShell scheduling wrapper are in the companion example.

If your maintenance job is still flagging every index at the default 5%/30% split without checking page density, or a rebuild schedule was set up years ago and never revisited for the storage it’s actually running on, see how a modernization engagement addresses this or get in touch and we can walk through what your indexes actually need.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

This is the third piece in a series on database-side controls for AI agents with direct SQL execute access. The first post covered a firewall that refuses badly-shaped statements outright — unscoped writes, schema changes. The second covered a human-approval queue for the schema changes that firewall refuses, so a legitimate change doesn’t just get re-run by hand later. Both of those check the shape of an agent’s SQL. This one checks something neither of them can: given a statement that’s perfectly well-formed and perfectly scoped-looking, does the agent issuing it actually have any business touching those rows at all?

The incident that motivates this one

Between December 2025 and February 2026, a single attacker used Claude Code and GPT-4.1 to breach nine Mexican government agencies — the federal tax authority, Mexico City’s civil registry, the electoral institute — exposing 195 million taxpayer records and 220 million civil records, plus more than 150GB of additional data.

The root cause wasn’t a novel exploit technique. It was excessive, shared permissions never enforced at the data layer itself. Whatever had execute access could see and touch far more than its actual job required, because nothing below the application layer was checking. A 2026 least-privilege research report analyzing over 3 billion permissions found that on average only about 4% had been used in the trailing 90 days — and nearly one in three could modify or delete sensitive data. Nobody had to break in cleverly. The door was already open wider than anyone was using it.

Key takeaways

• A statement-shape guardrail can’t catch this failure mode: WHERE region = 'US-EAST' is a perfectly scoped clause, and still exactly the wrong thing for an agent with no business in US-EAST to run
• SQL Server’s native Row-Level Security enforces what an agent can see and touch based on its declared identity, independent of how its SQL is written — a SELECT with zero WHERE clause, or a deliberately broad WHERE 1=1, still only returns rows inside that agent’s scope
• RLS has two distinct enforcement mechanisms that fail differently, and conflating them will make your own testing look broken when it isn’t: a filter predicate silently hides out-of-scope rows before they can even be matched, while a block predicate throws an explicit error when a write’s result would violate the predicate on a row that was visible to begin with
• This complements, not replaces, statement-shape guardrails — an agent correctly scoped to its own data can still submit an unscoped DELETE within that scope, which is a different problem this project doesn’t solve

What was actually built

The core of it is a SQL Server security policy backed by a schema-bound predicate function:

CREATE OR ALTER FUNCTION dbo.fn_agent_region_predicate(@region NVARCHAR(20))
RETURNS TABLE
WITH SCHEMABINDING
AS
RETURN SELECT 1 AS fn_result
WHERE @region = CAST(SESSION_CONTEXT(N'agent_region') AS NVARCHAR(20))
   OR CAST(SESSION_CONTEXT(N'agent_region') AS NVARCHAR(20)) = 'ALL';

CREATE SECURITY POLICY dbo.AgentRegionPolicy
    ADD FILTER PREDICATE dbo.fn_agent_region_predicate(region) ON dbo.demo_accounts,
    ADD BLOCK PREDICATE dbo.fn_agent_region_predicate(region) ON dbo.demo_accounts AFTER INSERT,
    ADD BLOCK PREDICATE dbo.fn_agent_region_predicate(region) ON dbo.demo_accounts AFTER UPDATE,
    ADD BLOCK PREDICATE dbo.fn_agent_region_predicate(region) ON dbo.demo_accounts BEFORE UPDATE,
    ADD BLOCK PREDICATE dbo.fn_agent_region_predicate(region) ON dbo.demo_accounts BEFORE DELETE
    WITH (STATE = ON);

Agent identity is carried via SESSION_CONTEXT, set once per connection before any of that agent’s SQL runs — mirroring how a real MCP database server or agent gateway typically authenticates a caller against a shared connection, rather than provisioning a distinct SQL login per agent (the same predicate pattern works against SUSER_SNAME() instead, if your environment does provision real per-agent logins). A thin Python wrapper, AgentIdentityGuard, establishes that identity and writes to an audit table — but the actual enforcement is entirely SQL Server’s, not application code the agent could reason its way around.

Seeing it hold up, including the part that surprised me

Against a real SQL Server 2025 instance, with an agent scoped to US-WEST trying two different ways to reach outside its region:

=== SCENARIO C: a well-scoped-looking write reaching OUTSIDE the agent's identity ===
    (This has a WHERE clause -- a shape-based guardrail would approve it.
     This agent is scoped to US-WEST; the target rows are US-EAST.)
  -> APPROVED, rows_affected=0 -- the filter predicate hid the US-EAST rows
     before the WHERE clause could ever match them. No error, no rows
     touched: the agent's own SQL was syntactically fine, it just had
     nothing it was allowed to see.

=== SCENARIO D: the same agent tries to relabel a row IT CAN see out of its own scope ===
    (Exfiltration by relabeling: not reaching for another region's data,
     but trying to move its own row into a region it isn't scoped to.)
  -> BLOCKED: Refused: agent 'migration-agent-west' (scope=US-WEST)
     attempted a UPDATE that Row-Level Security's block predicate rejected
     -- the target row(s) are outside this agent's identity scope. The
     write never reached the table.

My first draft of this demo only had scenario C, and I assumed it would throw the same error as an out-of-scope INSERT. It didn’t — it silently affected zero rows. It took a moment to click: the filter predicate hides out-of-scope rows from ever being matched by a WHERE clause, so the UPDATE never reaches the point where a block predicate would need to fire — there’s nothing left to violate it. The block predicate only fires on a result violation: a row that was visible ending up with a post-operation value that breaks the rule, which is a genuinely different scenario. I had to write scenario D — an agent trying to relabel its own row into a scope it doesn’t hold — to actually exercise that path.

Worth stating plainly rather than glossing over: RLS’s “fail silently” behavior on reads and matches, and its “fail loudly” behavior on result-violating writes, are two different mechanisms doing two different jobs. A demo (or a test suite) that expects one where the other actually applies will look broken even though the security property held the entire time.

What this doesn’t solve

Row-Level Security enforces scope given an honestly declared identity — it doesn’t itself authenticate that identity. If the gateway setting SESSION_CONTEXT can be tricked into declaring the wrong agent or the wrong scope, this control doesn’t catch that; it assumes the layer establishing identity is trustworthy, same as any authentication system has to assume somewhere. And it doesn’t replace the first two posts in this series — an agent correctly scoped to exactly the data it should touch can still submit an unscoped DELETE or a DROP TABLE within that scope. That’s still the firewall’s job, not this one’s.

Between the three projects now: the firewall refuses what should never run regardless of who’s asking. The approval queue routes what might be legitimate to a human instead of deciding alone. Row-level security scopes what an agent can reach in the first place, before either of the other two questions is even relevant. Different layer, same underlying problem — an AI agent with real database access needs checks it can’t talk its way around, and no single layer covers all of them.

The full project — the predicate function, the security policy, and all six scenarios runnable via Docker Compose against a real SQL Server 2025 instance — is open source: github.com/mrivanlima/ai-agent-row-level-security. If your agents have database access scoped by convention rather than enforced by the engine, that’s worth closing before it’s the reason nine agencies’ worth of records end up on someone else’s list. Get in touch if you want help wiring this into your own environment.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.

Your multi-agent system didn’t crash because LLMs are bad; it crashed because you treated memory like a temporary JSON file instead of a relational state machine. Every “agent forgot everything after a redeploy” incident traces back to the same root cause: the team building it never asked where state lives when the process dies. That’s not a prompting problem. It’s a state management problem, and it has a well-understood, decades-old answer.

The failure mode nobody wants to name

Here’s the pattern. A team ships an agent that works beautifully in the demo. It holds context across a long conversation, calls tools, updates a plan, recovers from a bad tool call. Then it goes to production, autoscaling kicks in, a pod restarts, and every bit of that context evaporates. The user has to start over. Nobody can explain why, because nobody ever wrote down where “memory” actually lived. The answer, almost every time, is a Python dictionary sitting in process RAM.

I’ve watched this exact failure take down a multi-agent workflow more than once — always the same root cause. A dict, a class attribute, an in-memory cache library with no persistence layer behind it. It works right up until the process that’s holding it doesn’t exist anymore.

This isn’t a fringe case. Reliability reviews of production agent deployments through 2025 and into 2026 consistently flag memory-related failures as the single most common category of reliability incident in agent systems — agents that forget instructions mid-task, silently lose prior context, or degrade across long sessions, because memory was never given an intentional persistence layer in the first place (Atlan). The LLM isn’t degrading. The scaffolding around it was never built to survive a restart.

Why “add retries” and “use a bigger context window” don’t fix it

The instinctive fixes are application-layer patches, and they don’t touch the actual defect.

Retries assume the failure is transient — that if you just call the model again, you’ll get a good result. But if the state you needed was never durable, retrying doesn’t recover it. You’re retrying against an empty memory, and you’ll get a coherent-sounding answer built on no history at all, which is worse than an obvious crash.

A bigger context window doesn’t solve persistence either — it solves how much state you can hold within a single running process, for as long as that process happens to stay alive. It does nothing for the moment that process restarts, gets rescheduled onto a different node, or gets killed by an autoscaler mid-request. Context window size and state durability are unrelated problems that keep getting treated as the same one.

And neither fix addresses the failure mode that’s actually more dangerous than losing memory outright: two concurrent requests corrupting shared in-memory state at the same time. Picture two workers both reading the same in-process cache entry for a session, both mutating it based on stale reads, and both writing back — the second write silently clobbers the first with no record that a conflict ever happened. No error. No log line. Just quietly wrong state that the agent will confidently act on next turn. A dict has no isolation levels, no locking, no concurrency control of any kind. It’s not a data store. It’s a variable that happens to hold data until something restarts it or two threads fight over it.

The database-first fix: the Agent State Ledger

This is where database design stops being a “nice to have” and becomes the actual missing requirement for durable agent execution. Production-grade agent workflows need explicit, externalized state, backed by a system that was built from the ground up to survive process death and handle concurrent writers correctly. That system already exists. It’s called a relational database.

Here’s the pattern I use — call it the Agent State Ledger. It’s a real Postgres schema, not a metaphor:

CREATE TABLE agent_state_ledger (
    session_id      UUID NOT NULL,
    agent_id        TEXT NOT NULL,
    state           JSONB NOT NULL,
    version         BIGINT NOT NULL DEFAULT 1,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (session_id, agent_id)
);

-- Every write checks the version it read, not just the primary key.
UPDATE agent_state_ledger
SET state = $1,
    version = version + 1,
    updated_at = now()
WHERE session_id = $2
  AND agent_id = $3
  AND version = $4;   -- fails silently-safe if another writer got here first

That version column is doing the real work. PostgreSQL has no built-in optimistic locking primitive, but the pattern is simple and well-established: every table that can be written concurrently carries a version column, and every update’s WHERE clause checks that the version hasn’t moved since the writer last read it (Reintech). If the UPDATE affects zero rows, you know — deterministically, not by guessing — that a concurrent writer got there first, and your agent can reread and retry instead of silently overwriting good state with stale state.

That’s the whole fix for the corruption case. And because the table lives outside the process, a pod restart, a redeploy, or an autoscale event no longer means the agent forgot who it was talking to. The session survives because the session was never actually stored in the session.

This is post one of three on a discipline I’m calling Database-First Agent Architecture, built on three pillars: Durable State, Verified Execution, and Bounded Autonomy. This post is about the first pillar. The next two cover getting agents to stop hallucinating SQL against schemas they’ve never seen, and putting hard governance boundaries around what an autonomous agent is allowed to execute.

The analogy, if you need it

Agent memory without a database is a web server holding session state in RAM with no replication. The industry stopped doing that for web applications twenty years ago — not because it was theoretically wrong, but because it kept taking down production. Sticky sessions, lost carts, users logged out mid-checkout when a server rebooted. We solved it by moving session state into Redis, or a database, or anything external to the process. Agent frameworks are relitigating that exact mistake right now, just with a chat history instead of a shopping cart.

Practical guidance

If you’re running agents in production, or about to:

• Audit every place your agent stores “memory” and ask, specifically, what survives a process restart. If the honest answer is “nothing,” that’s your first fix, not a backlog item.
• Give every stateful table a version column and enforce optimistic concurrency on every write path, not just the ones you’ve already seen fail.
• Separate short-lived working context (safe to lose) from durable session state (not safe to lose) at the schema level, not just in code comments.
• Log every rejected write (version mismatch) instead of silently retrying — that log is your evidence for whether concurrent corruption is actually happening in your system.
• Treat schema design for agent state as a first-class part of your architecture review, not something the application team bolts on after the demo works.

Key takeaways

• Agents that lose context on restart aren’t suffering an LLM problem — they’re suffering a state persistence problem with a known, boring, database-shaped fix.
• Retries and larger context windows patch symptoms; neither creates durability or concurrency safety.
• The Agent State Ledger pattern — session ID, agent ID, JSONB state, and a version column enforcing optimistic concurrency — gives agents state that survives process death and resists silent corruption from concurrent writers.
• This is the same lesson the web already learned with session state twenty years ago, now playing out again in agent frameworks.
• This is pillar one — Durable State — of Database-First Agent Architecture, with Verified Execution and Bounded Autonomy still to come.

If your agent architecture doesn’t have an answer for what happens on restart, it doesn’t have an architecture — it has a demo.

If your team is building agents on state that can’t survive a redeploy, that’s a schema problem before it’s a prompting problem, and it’s fixable. Get in touch or see how we approach it on our services page.

Ivan Lima is a data engineer specializing in database modernization for AI systems. Get in touch if your database needs to be ready for what’s next.