Pith. sign in

REVIEW 4 cited by

Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.12243 v4 pith:5FP5RBOU submitted 2024-02-19 cs.CL

Niklas Wretblad , Fredrik Gordh Riseby , Rahul Biswas , Amin Ahmadi , Oskar Holmström This is my paper · ORCID
classification cs.CL
keywords noisegoldqueriesbenchmarkbird-benchmodelsquestionstext-to-sql
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge. However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors. This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models. While BIRD-Bench was created to model dirty and noisy database values, it was not created to contain noise and errors in the questions and gold queries. We found that noise in questions and gold queries are prevalent in the dataset, with varying amounts across domains, and with an uneven distribution between noise types. The presence of incorrect gold SQL queries, which then generate incorrect gold answers, has a significant impact on the benchmark's reliability. Surprisingly, when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods. We conclude that informative noise labels and reliable benchmarks are crucial to developing new Text-to-SQL methods that can handle varying types of noise. All datasets, annotations, and code are available at https://github.com/niklaswretblad/the-effects-of-noise-in-text-to-SQL.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Establishing Best Practices for Building Rigorous Agentic Benchmarks

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Agentic benchmarks frequently mis-grade agents, and the new ABC checklist helps identify and correct such errors in ten popular benchmarks.

  2. CoddLLM: Empowering Large Language Models for Data Analytics

    cs.DB 2025-02 conditional novelty 6.0 of 10

    CoddLLM, a Mistral-NeMo-12B model post-trained on synthetic analytics instruction data, outperforms GPT-4o on the paper's table-selection average and improves text-to-SQL over its base model, with the largest gains on...

  3. Confidence Estimation for Error Detection in Text-to-SQL Systems

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Entropy-based selective classifiers can detect errors in text-to-SQL systems, and T5 models are better calibrated than GPT-4 and Llama 3 under distribution shift.

  4. ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Multitask supervised fine-tuning plus collaborative prompting on schema linking, noise correction, and continuation writing lifts open-source LLM Text2SQL accuracy on SPIDER and BIRD.

Pith tools