This article exposes critical flaws in Text-to-SQL benchmarks like BIRD and Spider. An audit of gold queries reveals that several contain incorrect joins, causing mathematically wrong results to be established as ground truth. Since standard execution accuracy measures performance by comparing outputs against these faulty reference answers, models are often penalized for being correct and rewarded for mimicking human errors. To address this, the author proposes a constraint-aware evaluation method that validates SQL logic against declared data semantics rather than relying on potentially incorrect gold results.
- Discrepancies between benchmark gold queries and database schema facts
- The inherent risks of using execution accuracy as the primary metric
- How annotation errors impact model rankings and enterprise deployments
- Introduction of constraint-aware evaluation to ensure semantic validity
- Discrepancies between benchmark gold queries and database schema facts
- The inherent risks of using execution accuracy as the primary metric
- How annotation errors impact model rankings and enterprise deployments
- Introduction of constraint-aware evaluation to ensure semantic validity
