The compute bill of being wrong
If you build or operate a text-to-SQL system, you already know the generated query can be wrong. Here's the part that doesn't get measured: the wrong query usually isn't just wrong — it's expensive. The classic silent error in generated SQL, join fan-out, multiplies rows before aggregating them. Your warehouse happily bills you for every multiplied row.
So we measured it. We took real databases from the BIRD benchmark, wrote the fan-out version and the correct version of the same aggregate, and ran both on the same engine. Three cases:
| database (real BIRD data) | query shape | result inflation | rows through the aggregate | wall time |
|---|---|---|---|---|
| financial | SUM(loan.amount) joined to trans | 271.9× too high | 682 → 191,556 (281×) | 0.1 ms → 568 ms (~8,000×) |
| formula_1 | SUM(results.points) joined to lapTimes | 37.3× too high | 23,657 → 419,985 (18×) | 1.4 ms → 30 ms (22×) |
| codebase_community | SUM(posts.ViewCount) joined to votes | 3.1× too high | cheaper and wrong — see below | |
The first case deserves a second look. The correct query sums 682 loan rows in a tenth of a millisecond. The fan-out version pushes 191,556 rows through the same aggregate — every loan repeated once per transaction on its account — takes roughly 8,000× longer, and returns a number that is 272× too large. On a laptop that's half a second. On a cloud warehouse, compute is billed by work: a query that does 280× the row-work does it on your invoice, every dashboard refresh, every scheduled run, until someone notices. And because the query is syntactically perfect and returns plausible-looking output, nobody notices.
The third case is the honest counterexample — and a second failure mode.
The inner join to votes drops posts that have no
votes while multiplying posts that have many. The query runs
faster than the correct one and still returns a total 3.1× too
high. So no, you can't catch this class by watching query cost — sometimes
being wrong is cheap. You catch it by checking the join's declared
cardinality before execution.
The three bills
Put together with what we've measured previously, a text-to-SQL system without a verification layer runs up three separate bills:
- The warehouse bill — the table above. Fan-out queries do 20× to 8,000× the row-work of their correct versions on the same engine.
- The guardrail bill — the industry's default check is LLM-as-judge. When we A/B-tested it on live agent drafts, the judge cost ~$0.03 per check (at small-model prices — a frontier-model judge costs several times more), took seconds per verdict, and its only rejections in 44 checks were false alarms on correct queries. At 10,000 agent queries a day, that's roughly $300/day (~$108k/yr) of judge spend that changed zero outcomes in our experiment. Even Snowflake's Cortex Analyst team reported that judging SQL with LLMs "can be as challenging as generating the correct SQL itself."
- The retry bill — when a check does flag something, a vague critique ("re-examine your joins") burns generation tokens across multiple repair rounds. A deterministic rejection carries a machine-actionable fix ("pre-aggregate loan to account_id before joining trans"); in our benchmark, applying the hint verbatim produced a passing query 10 out of 10 times in one round.
What a deterministic gate costs instead
The comparison number for all three bills: a constraint check against declared facts runs in 0.7 ms (median, measured on live agent drafts) at $0 per check, gives the same verdict every time, and in our head-to-head never blocked a correct query. It catches exactly the class above — fan-out, wrong join keys, unsafe aggregation — by lookup, before execution, so the 191,556-row version never runs at all.
The gate doesn't just prevent wrong numbers. It prevents paying for wrong numbers — three times.
"But we already have a semantic layer / our model is better than Haiku"
Both fair. Two honest boundaries and one unlock:
- Semantic layers prevent this for queries routed through them. MetricFlow, Cube, and friends generate correct SQL by construction — inside their walls. The SQL your agent writes directly, the ad-hoc notebook query, the CI diff: none of it routes through the layer. The gate covers everything outside the walls — and it reads the layer's own declarations to do it.
- Better models commit this class less often — not never. The BIRD benchmark's own expert-written gold answers contain fan-out errors (we proved one wrong by 8× by executing it). If experts writing an answer key commit the bug, your generator sometimes will too. Tail events × query volume = weekly events.
- The unlock: your users' declarations already exist. sqlsure builds its rulebook from dbt tests, MetricFlow semantic models, WrenAI MDL, OSI models, plain PK/FK — or, with no semantic layer at all, straight from the live database catalog. If you ship a text-to-SQL product, the verification layer is one function call around your generator, and it speaks your users' existing metadata.
Reproduce it
Everything above is checkable: the fan-out measurements run on BIRD's public databases with the queries shown; the gate-vs-judge experiment ships as a script in the repo with per-question receipts; the benchmark and its limits are in METRICS.md. Numbers we can't defend, we don't publish — measured on a laptop's SQLite, it's the relative factors that transfer to your warehouse, and your join depths are deeper than a benchmark's.
pip install sqlsureBuilding an NL2SQL product and want the gate inside it — or want to argue with our numbers? Open an issue. Both conversations are welcome.