REVIEW 2 cited by
Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Neural models have achieved significant results on the text-to-SQL task, in which most current work assumes all the input questions are legal and generates a SQL query for any input. However, in the real scenario, users can input any text that may not be able to be answered by a SQL query. In this work, we propose TriageSQL, the first cross-domain text-to-SQL question intention classification benchmark that requires models to distinguish four types of unanswerable questions from answerable questions. The baseline RoBERTa model achieves a 60% F1 score on the test set, demonstrating the need for further improvement on this task. Our dataset is available at https://github.com/chatc/TriageSQL.
Forward citations
Cited by 2 Pith papers
-
ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction
A new benchmark shows that text-to-SQL models detect problematic questions but fail to pinpoint the exact problem type and to resolve the question after a useful clarification.
-
Confidence Estimation for Text-to-SQL in Large Language Models
Consistency-based methods are the most reliable confidence signal for text-to-SQL in black-box LLMs, and executing queries against a database adds a useful correctness signal.
Discussion (0). Sign in to comment.