Pith. sign in

REVIEW 3 cited by

AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.19073 v2 pith:KMUFMQX2 submitted 2024-06-27 cs.CL

Irina Saparina , Mirella Lapata This is my paper · ORCID
classification cs.CL
keywords ambiguityambiguousambrosiabenchmarkevenquestionsdatabaseparsers
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

    cs.SE 2025-01 conditional novelty 6.0 of 10

    A 55-criteria guideline and audit of 274 code benchmarks finds that most benchmarks skip data quality checks, prompting calls for more rigorous, reproducible benchmark construction.

  2. Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

    cs.CL 2024-12 conditional novelty 6.0 of 10

    MMSQL is a multi-turn text-to-SQL benchmark with four question types, and a multi-agent framework with a Question Detector improves LLM performance on it.

  3. Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A systematic review organizing LLM-based text-to-SQL methods into pre-processing, in-context learning, fine-tuning, and post-processing paradigms, with a catalog of datasets, metrics, challenges, and future directions.

Pith tools