Pith. sign in

REVIEW 2 cited by

CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.05378 v1 pith:ZGR7QREH submitted 2019-09-11 cs.CL cs.AI

CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

Tao Yu , Rui Zhang , He Yang Er , Suyi Li , Eric Xue , Bo Pang , Xi Victoria Lin , Yi Chern Tan
show 16 more authors
Tianze Shi Zihan Li Youxuan Jiang Michihiro Yasunaga Sungrok Shim Tao Chen Alexander Fabbri Zifan Li Luyao Chen Yuwen Zhang Shreya Dixit Vincent Zhang Caiming Xiong Richard Socher Walter S Lasecki Dragomir Radev
This is my paper
classification cs.CL cs.AI
keywords cosqldialogueuserquestionsbaselineschallengescross-domaindatabases
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SANE Schema-aware Natural-language Evaluation of Biological Data

    cs.CL 2026-06 unverdicted novelty 6.0

    SANE is a new schema-aware benchmark paradigm for text-to-SQL evaluation that demonstrates few-shot LLMs with structured prompting can generate accurate queries on constrained biological data schemas without fine-tuning.

  2. Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems

    cs.AI 2026-04 unverdicted novelty 5.0

    STEF is a schema-agnostic evaluation framework that scores SQL generation accuracy from natural language inputs using semantic feature alignment and a composite metric.