We told a benchmark its schema was wrong. It shipped the fix.

November 2026 · a closed loop, and what it is evidence for

In July we published an audit of the answer keys behind BIRD and Spider — the benchmarks nearly every text-to-SQL model is graded on. Most of that post was about wrong gold answers. One finding was quieter and, it turns out, more consequential: in BIRD's published schema for the european_football_2 database, two foreign keys that exist in the shipped SQLite file were missing from the annotations. We filed it upstream.

The maintainers replied: "We will review this issue and fix the missing foreign key annotations in the next patch." Last month they shipped the 2026 Mini-Dev release — corrected questions, evidence and gold SQL, a new machine-readable schema artifact, and an acknowledgements section that credits the community reporters who filed issues, this author among them.

Verifying the fix, mechanically

The interesting part isn't that they fixed it. It's that the fix is checkable — by the same mechanism that found the problem. We diffed foreign-key declarations three ways across all 11 dev databases: the old annotations, the new ones, and the databases themselves.

sourceforeign keys declared
old dev_tables.json (2024)102
new baseline_inference/schemas (2026)104
the shipped .sqlite files104

The new annotations match the actual databases exactly, on every one of the 11. The two added edges are precisely the ones we reported — match.country_id → country.id and match.league_id → league.id. Nothing else moved.

You can run this yourself in about five seconds. The checker is stdlib-only, read-only, and takes either published annotation format:

python verify_fk_annotations.py \
  --db-dir dev_databases --annotations dev_tables.json

european_football_2    annotated: 29   in db: 31   missing: 2
  match.country_id -> country.id
  match.league_id  -> league.id

Point it at the 2026 schemas and it prints "every foreign key declared by the databases is annotated." (source; we've offered it upstream for the remaining splits.)

Why two missing edges mattered

Something we only appreciated after the fix landed. BIRD's baseline prompt template tells the model, in so many words, that the schema it is being given includes "primary keys, foreign keys, and any relevant relationships or constraints." So those two absent edges weren't just a metadata nit in a side file — they were part of what every baseline-evaluated model was told about the world. A model asked to join matches to leagues was working from a schema that said no such relationship exists.

And the error propagates outward with a multiplier. Published schema annotations are consumed by prompt builders, by schema-linking modules, by retrieval over table metadata, by every paper that reports a number on this benchmark, and by tools (ours included) that treat declared keys as ground truth. One wrong row in a metadata file is read thousands of times.

What this is evidence for

Benchmark metadata is code, and deserves CI

The defect wasn't carelessness; it came from an extraction edge case. SQLite lets you write REFERENCES parent without naming a column, meaning "the parent's primary key." A reader that only looks for explicit column lists silently drops those. It's exactly the kind of bug humans don't catch by reading — and a three-line check catches forever. Benchmarks ship their databases and their description of those databases; nothing but a script can keep the two honest.

Machine-checkable claims get fixed

We've made arguments to maintainers before. This one wasn't an argument. It was: here is a file, here is the database, here is the one-command diff between them. Nobody had to be persuaded, and nobody had to take our word. That's the real difference between a deterministic check and an opinion — not accuracy, actionability. A finding anyone can reproduce in five seconds gets shipped; a finding that requires trusting the finder gets discussed.

The cost asymmetry is the whole argument

Finding this took one pass of a checker over a corpus — no human review of 2,568 queries, no LLM judging each one, no tokens. Verifying the fix took one command. The class of problem is enormous and the cost of checking it is approximately zero. That ratio is why we think constraint checking belongs in the pipeline permanently rather than as an occasional audit.

Who should care about this

Different readers, different stakes:

What's still open

Honestly stated: this fix landed in Mini-Dev. The maintainers were explicit that the update focused there and that other splits come later, and the full dev and train distributions are unchanged. Our scan of the 9,428 train-split gold queries counts 788 joins whose key pair matches no declared foreign key, across 21 databases, 8 of which declare no foreign keys at all. A good share of those are legitimate shared-key joins rather than missing annotations — which is exactly what a FK-from-database pass would separate out in one run. We've offered the script; it's their call and their timeline.

A benchmark is two artifacts that have to agree: the data, and the story the data tells about itself. Only one of them is usually tested.

sqlsure is open source (Apache-2.0) — a deterministic semantic gate for SQL, with a rerunnable measurement behind every claim.

pip install sqlsure