We told a benchmark its schema was wrong. It shipped the fix.
In July we published an audit
of the answer keys behind BIRD and Spider — the benchmarks nearly every
text-to-SQL model is graded on. Most of that post was about wrong gold
answers. One finding was quieter and, it turns out, more consequential: in
BIRD's published schema for the european_football_2 database,
two foreign keys that exist in the shipped SQLite file were missing from
the annotations. We filed it
upstream.
The maintainers replied: "We will review this issue and fix the missing foreign key annotations in the next patch." Last month they shipped the 2026 Mini-Dev release — corrected questions, evidence and gold SQL, a new machine-readable schema artifact, and an acknowledgements section that credits the community reporters who filed issues, this author among them.
Verifying the fix, mechanically
The interesting part isn't that they fixed it. It's that the fix is checkable — by the same mechanism that found the problem. We diffed foreign-key declarations three ways across all 11 dev databases: the old annotations, the new ones, and the databases themselves.
| source | foreign keys declared |
|---|---|
old dev_tables.json (2024) | 102 |
new baseline_inference/schemas (2026) | 104 |
the shipped .sqlite files | 104 |
The new annotations match the actual databases exactly, on every one
of the 11. The two added edges are precisely the ones we reported —
match.country_id → country.id and match.league_id →
league.id. Nothing else moved.
You can run this yourself in about five seconds. The checker is stdlib-only, read-only, and takes either published annotation format:
python verify_fk_annotations.py \
--db-dir dev_databases --annotations dev_tables.json
european_football_2 annotated: 29 in db: 31 missing: 2
match.country_id -> country.id
match.league_id -> league.id
Point it at the 2026 schemas and it prints "every foreign key declared by the databases is annotated." (source; we've offered it upstream for the remaining splits.)
Why two missing edges mattered
Something we only appreciated after the fix landed. BIRD's baseline prompt template tells the model, in so many words, that the schema it is being given includes "primary keys, foreign keys, and any relevant relationships or constraints." So those two absent edges weren't just a metadata nit in a side file — they were part of what every baseline-evaluated model was told about the world. A model asked to join matches to leagues was working from a schema that said no such relationship exists.
And the error propagates outward with a multiplier. Published schema annotations are consumed by prompt builders, by schema-linking modules, by retrieval over table metadata, by every paper that reports a number on this benchmark, and by tools (ours included) that treat declared keys as ground truth. One wrong row in a metadata file is read thousands of times.
What this is evidence for
Benchmark metadata is code, and deserves CI
The defect wasn't carelessness; it came from an extraction edge case.
SQLite lets you write REFERENCES parent without naming a
column, meaning "the parent's primary key." A reader that only looks for
explicit column lists silently drops those. It's exactly the kind of bug
humans don't catch by reading — and a three-line check catches forever.
Benchmarks ship their databases and their description of those
databases; nothing but a script can keep the two honest.
Machine-checkable claims get fixed
We've made arguments to maintainers before. This one wasn't an argument. It was: here is a file, here is the database, here is the one-command diff between them. Nobody had to be persuaded, and nobody had to take our word. That's the real difference between a deterministic check and an opinion — not accuracy, actionability. A finding anyone can reproduce in five seconds gets shipped; a finding that requires trusting the finder gets discussed.
The cost asymmetry is the whole argument
Finding this took one pass of a checker over a corpus — no human review of 2,568 queries, no LLM judging each one, no tokens. Verifying the fix took one command. The class of problem is enormous and the cost of checking it is approximately zero. That ratio is why we think constraint checking belongs in the pipeline permanently rather than as an occasional audit.
Who should care about this
Different readers, different stakes:
- Benchmark maintainers. If you publish databases alongside a schema description, you have the same exposure — BIRD's extraction bug is not BIRD-specific. The verifier is free, stdlib-only, and belongs in CI next to your other checks. (Spider, LiveSQLBench, and the newer agentic SQL benchmarks all ship metadata the same way.)
- Anyone publishing a leaderboard number. Your score is a function of the schema the model was given. When that schema changes — as it just did — comparisons across releases are no longer apples to apples. Report which annotation release you evaluated against, the way you'd report a model version.
- Researchers building on these datasets. If a paper's schema-linking or join-path reasoning was evaluated on the pre-fix annotations, two edges of the graph were missing from the input. Probably minor; worth knowing it's a variable at all.
- Data and analytics teams. This is the part that generalizes hardest. Your warehouse has the same split-brain: the database declares one thing, the documentation and semantic layer declare another, and they drift silently. The consequence in a benchmark is a slightly wrong score. The consequence in a company is a dashboard that double-counts revenue because a join nobody declared turned out to be one-to-many. Same check, higher stakes.
- Teams shipping AI agents over SQL. Agents inherit their metadata's mistakes with no ability to notice. An agent told "there is no relationship between matches and leagues" will either refuse a reasonable question or invent a join. Checking the metadata against the database is upstream of every prompt-engineering fix you might reach for.
- People who maintain anything open-source and get bug reports. The thing that made this easy to act on was a reproduction, not a report. Worth copying in both directions.
What's still open
Honestly stated: this fix landed in Mini-Dev. The maintainers were
explicit that the update focused there and that other splits come later,
and the full dev and train distributions are
unchanged. Our scan of the 9,428 train-split gold queries counts 788
joins whose key pair matches no declared foreign key, across 21
databases, 8 of which declare no foreign keys at all. A good share of those
are legitimate shared-key joins rather than missing annotations — which is
exactly what a FK-from-database pass would separate out in one run. We've
offered the script; it's their call and their timeline.
A benchmark is two artifacts that have to agree: the data, and the story the data tells about itself. Only one of them is usually tested.
sqlsure is open source (Apache-2.0) — a deterministic semantic gate for SQL, with a rerunnable measurement behind every claim.
pip install sqlsure