Pith. sign in

REVIEW 4 major objections 4 minor 32 references

Ambiguous user queries expose two text-to-SQL bottlenecks

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:40 UTC pith:5IAKSQDA

load-bearing objection Useful benchmark for ambiguous/unanswerable text-to-SQL, but its headline numbers rest on lightly validated LLM-generated gold labels—treat the quantitative claims as provisional. the 4 major comments →

arxiv 2607.23340 v1 pith:5IAKSQDA submitted 2026-07-25 cs.DB

ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction

Giovanni Sullutrone , Luca Sala , Sania Aftar , Georgia Koutrika , Sonia Bergamaschi This is my paper
classification cs.DB
keywords text-to-SQLambiguityunanswerable questionsbenchmarkLLM agentsinteractive evaluationtaxonomymulti-turn dialogue
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that benchmark scores on curated text-to-SQL tasks overstate how well systems handle real users, because real questions are often ambiguous or unanswerable and require back-and-forth dialogue. To test this, the authors build ABISS, a benchmark that drops an agent into multi-turn conversations with a simulated user, and generate over 7,000 validated questions across a taxonomy of eight categories of problematic questions. Testing eight open-source models on two datasets, they find two persistent bottlenecks: models usually recognize that a question is problematic but fail to identify which subcategory it belongs to, and even after receiving a relevant clarification they frequently fail to produce the correct final SQL. Supplying the correct category label improves both execution and feedback substantially, yet ambiguous-question execution stays low even with oracle labels. The paper concludes that progress on real-world text-to-SQL depends less on single-turn accuracy and more on precise problem diagnosis and on turning clarifications into queries.

Core claim

Current text-to-SQL agents fail on realistic questions at two distinct points: classifying the problem, and converting a clarification into SQL. In tests with eight open-source models on two datasets, recognition of whether a question is answerable, ambiguous, or unanswerable is consistently higher than exact subcategory classification (best: 70.5% vs 59.1%). Supplying the true subcategory raises feedback accuracy from 67.5% to 87.6% and ambiguous execution from 17.6% to 32.3% on ABISS-BIRD, with similar jumps on ABISS-Spider—yet ambiguous execution stays near 32–40% even with oracle categories. Strong models stop asking after a relevant answer, while execution remains low, so the final conv

What carries the argument

ABISS (Ambiguity Benchmark using Interaction-Simulated Sessions) is the central instrument: a simulation environment in which a text-to-SQL agent and a style-aware simulated user exchange multiple turns, with a council of open-source LLMs generating user responses that are classified as Relevant, Technical, or Irrelevant. A companion taxonomy groups problematic questions into eight categories (e.g., Lexical Vagueness, Missing Schema Elements, Missing User Knowledge, Conflicting Knowledge) and eleven problematic subcategories. The evaluation protocol isolates diagnostic accuracy from interactive ability through four category-usage modes—ground-truth, predicted, taxonomy-only, taxonomy-free—so

Load-bearing premise

The load-bearing assumption is that the LLM-council-generated ground-truth labels—especially that 'unanswerable' questions are truly unsolvable—are accurate enough to measure real performance; the authors note Stage 7 cannot eliminate false unanswerable labels, and human validation on 52 questions reached only 75–78% correctness, so noisy gold answers could distort the reported numbers.

What would settle it

Run a large human-annotation study over the full ABISS datasets. If human experts solve a substantial fraction of questions labeled 'unanswerable' (or reject a substantial fraction of gold SQL/feedback labels), the measured feedback and execution numbers would change; the classification-bottleneck conclusion would be weakened if corrected labels move predicted-mode scores close to oracle-mode scores.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Subcategory classification, not coarse recognition, is the primary bottleneck; systems that can pinpoint whether a question is vague, missing schema elements, or missing user knowledge should see large improvements in feedback and execution.
  • Providing the correct category label is worth more than providing taxonomy definitions, since taxonomy-only and taxonomy-free modes perform similarly; explicit diagnosis is what matters.
  • Even with oracle categories, ambiguous-question execution remains around a third, so research should target converting clarifications into SQL under ambiguity.
  • Code-specialized models do not automatically solve these problems; the Qwen2.5 32B code and general variants perform close, implying the difficulty is in language-level diagnosis and dialogue.
  • Strong frontier models show high answer-incorporation rates but low ambiguous execution, meaning their remaining errors are in the SQL synthesis step, not in deciding when to stop the conversation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the bottleneck pattern holds beyond open-source models, evaluation of commercial text-to-SQL services should include interaction-based tests rather than single-turn accuracy; otherwise deployment failures are hidden.
  • The same taxonomy and interaction protocol could transfer to other code-generation tasks (e.g., text-to-Python or data-analysis assistants), where vagueness, missing user context, and contradictory knowledge also block correctness.
  • The human-validation sample (52 questions) shows SQL correctness near 75% and feedback accuracy near 78%, suggesting absolute benchmark numbers may shift with better gold labels; the relative pattern—classification and clarification-conditioned generation as bottlenecks—should be re-tested with human-verified labels at scale.
  • A future benchmark could measure whether giving users the correct category label in the reply, rather than hiding it, changes the outcome—if users can disambiguate after being told the problem type, part of the current failure shifts from model to interface design.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ABISS, a benchmark and simulation environment for evaluating text-to-SQL systems on ambiguous and unanswerable natural language questions. It contributes (1) a taxonomy with 8 top-level categories and 13 subcategories, including Missing User Knowledge and Conflicting Knowledge; (2) a multi-agent generation pipeline in which three open-source LLMs generate candidate questions and a council of the same models applies ten validation stages; and (3) a multi-turn interactive evaluation protocol with a style-aware simulated user. Eight open-source models are evaluated on two datasets, ABISS-BIRD and ABISS-Spider, under four category-usage modes. The reported central findings are that models recognize problematic questions much better than they classify the exact subcategory; that giving the ground-truth category greatly improves feedback accuracy and ambiguous-question execution; and that even with the ground-truth category, ambiguous-question execution remains low while answer-incorporation rates are high, implying a bottleneck in the final clarification-conditioned SQL generation step.

Significance. If the empirical findings are robust, the paper addresses an important gap: standard single-turn text-to-SQL benchmarks do not test system behavior on ambiguous or unanswerable queries, and existing interactive benchmarks bypass the hardest parts of diagnosis and dialogue control. The taxonomy contribution is useful, the generation pipeline is reusable, and the code and data release is a real asset. The claim that progress should focus on subcategory diagnosis and on converting clarifications into SQL, rather than on single-turn accuracy, is consequential and falsifiable. However, the current evidence base is too thin to establish the quantitative conclusions as model-level facts: the gold labels are calibrated on only 52 human-annotated questions, the unanswerability label is explicitly an operational approximation, and the same model families appear on both sides of the evaluation (dataset construction, user simulation, grading, and system agents).

major comments (4)
  1. [Table 2; §5.3] The only human calibration of the benchmark labels is Table 2: 52 questions (four per subcategory), reporting Category Correctness 92.3%, SQL Correctness 75.0%, Feedback Accuracy 78.1%, and Disambiguation Correctness 87.5%, with no inter-annotator agreement. These are the gold labels against which EX and FB are computed in Tables 4 and 5. A 25% gold-SQL error rate is high enough to change the central comparison in Table 5 between ambiguous EX under ground-truth category (32.3% BIRD, 39.8% Spider) and predicted category (17.6%, 22.6%), and to affect the claim in §6.3 that clarification-conditioned SQL generation is the bottleneck. A model that correctly incorporates a relevant clarification but produces a valid SQL that differs from the gold can be scored as incorrect. Similarly, 78.1% feedback accuracy means a correct explanation that differs from the gold hidden-knowledge wording can be
  2. [§4.2 Stage 7; §7.1; Table 7] Unanswerability is defined operationally in Stage 7 as 'no council model finds a valid SQL,' and §7.1 concedes that false unanswerable labels cannot be eliminated entirely. Since FB (§5.3) is computed only over questions labeled unanswerable, the FB numbers in Table 5 and Figure 2 are conditional on that approximation. Stage 7 rejects 39.6% of candidates entering it (Table 7), so the procedure is doing real work, but the surviving unanswerable set may still contain solvable questions. Please quantify the false-unsolvable rate on a human-validated random sample and show how the headline FB and classification findings change when those questions are re-labeled.
  3. [§4.3; §5.2; §5.3; §6.1] The same three model families (GPT-OSS-120B, Qwen3.5-122B, Nemotron-3-120B) generate the datasets, constitute the user-simulation council, judge feedback accuracy, and are themselves among the eight evaluated system agents. This creates a same-model agreement channel that is not neutral across the model roster. When the evaluated model is also a council member, the wording and grading of relevance/feedback may align with that model's text more often than with a foreign model's. The AIR comparisons in Table 6 and the FB leaderboard in Table 4 could therefore favor council members. Please add a control where the user and grading councils exclude the system under evaluation, or otherwise demonstrate that the interaction-level conclusions are stable across council composition.
  4. [Tables 2 and 3] The validation samples are small and under-reported for a benchmark paper. Table 2 has only four questions per subcategory, and Table 3 is 150 conversations/221 user turns. For the user simulator, relevancy-label correctness is 87.8% and style realism 70.6%. The text reports 'combined results across both annotators' without per-annotator scores or agreement, so the reader cannot assess label reliability on borderline cases. Please report inter-annotator agreement (e.g., Cohen's kappa), per-annotator scores, and expand the audit to cover all subcategories and both datasets.
minor comments (4)
  1. [Abstract; §3.2.8] The paper uses '8 categories', 'eleven problematic subcategories', and '13 subcategories' at different points. Make the category/subcategory hierarchy explicit at first use (8 top-level categories, 13 subcategories including two answerable) to avoid confusion.
  2. [Table 4] The caption says results are for Predicted and Ground-Truth (in parentheses) modes, but Recognition and Classification have no parenthetical values. State that Rec./Cls. are measured in the classification phase and are mode-independent, or show them for both modes.
  3. [§6.1; §6.2] No confidence intervals or multiple-seed variation are reported for model comparisons. Claims such as 'the same frontier group remains strongest' or small differences between the two 32B variants use differences of 1–3 points that may be within run-to-run noise; report variance or at least discuss its magnitude.
  4. [§5.2] The claim that 'ABISS never makes restrictions on the response type' is slightly misleading: for unanswerable questions the user simulator is hardwired to return an Irrelevant refusal. This is a reasonable design choice, but the asymmetry should be stated where the protocol is introduced.

Circularity Check

0 steps flagged

No circular derivation; ABISS is an empirical benchmark whose central claims are measurements, not conclusions forced by definition or by a self-citation chain.

full rationale

ABISS is an empirical benchmark, not a mathematical derivation. The taxonomy (Section 3) is stipulated rather than derived; the multi-stage generation pipeline (Section 4) uses LLM councils and a small human audit (Table 2, 52 questions) to construct labels; and the eight evaluated models are not fitted to any subset of the benchmark, so no fitted parameter is renamed as a prediction. The controlled comparison in Section 6.3 (ground-truth category vs. predicted category vs. taxonomy-only vs. taxonomy-free) is an experimental ablation, not a definitional identity: giving the model the ground-truth subcategory can improve execution and feedback, but the model still has to produce the correct SQL or feedback, and ambiguous execution remains low even under oracle labels. The paper's own Section 7.1 limitations are substantive validity threats rather than circularity: Stage 7 approximates unanswerability operationally and 'cannot eliminate them entirely'; council composition was fixed; and the same three models that generated the data and act as the user-simulator council are also among the evaluated system agents (Section 6.1). These create a self-evaluation contamination that limits how far the quantitative claims can be trusted, but they do not make any claimed result equivalent to its own inputs by construction. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation; the benchmark conclusions are empirical findings with acknowledged label-quality caveats.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The benchmark's quantitative outputs depend on several hand-set generation hyperparameters and dialogue limits, none fitted to the target conclusions. The central claims rest on the assumption that LLM-council judgments and the unanswerability approximation produce trustworthy ground-truth labels; the paper only partially validates this assumption with a small human evaluation.

free parameters (4)
  • generation temperature = 0.7
    Used for question generation; not fitted, but affects linguistic diversity and which candidates survive validation.
  • frequency penalty = 0.2
    Used for question generation; affects repetition and style distribution.
  • clarification turn limit = 3
    Conversations limited to 3 clarification turns plus one final turn; directly shapes AIR and termination metrics.
  • candidate samples per combination = 3
    Three candidate NLQs per (model, subcategory, style, difficulty, database); affects final dataset composition and diversity.
axioms (4)
  • domain assumption LLM majority-vote judgments reliably approximate human judgments of question category, SQL correctness, and feedback quality.
    The entire validation pipeline and metric computation (EX/FB) rely on LLM councils rather than formal verification or full human annotation; Sections 4.2 (Stages 3–9) and 5.3.
  • domain assumption The deterministic mapping f(q,D,K) (Definition 1) and the set S of non-equivalent valid SQL queries are well-defined; f returns all correct queries.
    Section 3.1 formalization; assumes each NLQ has a well-defined gold answer set under the relaxed-equivalence criterion.
  • ad hoc to paper The unanswerability approximation: if a council of LLMs cannot generate a valid SQL, the question is unsolvable.
    Stage 7 (Section 4.2) and acknowledged in Section 7.1 that this is not a formal proof and cannot eliminate false unanswerable labels entirely.
  • domain assumption The three selected open-source models are sufficiently diverse to form a neutral council for generation and validation.
    Section 4.3 asserts diversity is essential; the paper does not empirically test council size/composition effects (acknowledged in Section 7.1).
invented entities (3)
  • Style-aware simulated user agent no independent evidence
    purpose: Generates user responses to system clarifications in multi-turn dialogues, conditioned on hidden knowledge and linguistic style.
    Section 5.2; no external verification beyond the manual audit on 221 turns, so its behavior is an in-benchmark construct.
  • Hidden Knowledge field no independent evidence
    purpose: Carries disambiguation information for ambiguous NLQs and actionable feedback for unanswerable NLQs in the dataset.
    Section 4.1; labels are LLM-generated and only partially human-validated, so they are not externally grounded.
  • ABISS taxonomy categories (e.g., Missing User Knowledge, Conflicting Knowledge) no independent evidence
    purpose: Ground-truth labeling scheme for the benchmark; new categories claimed to be absent from prior taxonomies.
    The taxonomy is introduced by this paper and used both for data generation and as the evaluation standard, with no external validation beyond a small human study.

pith-pipeline@v1.3.0-alltime-deepseek · 22105 in / 14041 out tokens · 141391 ms · 2026-07-31T23:40:59.547717+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) demonstrate high performance on curated Text-to-SQL benchmarks; nevertheless, real-world users frequently pose ambiguous or unanswerable questions that current systems handle poorly. Three interconnected gaps hinder progress: incomplete taxonomies, realistic benchmark generation for real-world settings, and static user interaction. We address all of the above issues through three contributions: (1) a unified taxonomy of 8 categories covering ambiguous and unanswerable questions; (2) a multi-agent generation pipeline with a two-stage process (NLQ generation followed by SQL grounding) and an explicit Category Conformance validation stage, producing questions from arbitrary databases validated by a council of local open-source models; and (3) ABISS (Ambiguity Benchmark using Interaction-Simulated Sessions), a dynamic simulation environment where Text-to-SQL agents interact with style-aware simulated users across multi-turn dialogues. Experiments with eight open-source models on ABISS-BIRD and ABISS-Spider reveal two fundamental bottlenecks. The first is subcategory classification: models detect that a question is problematic yet struggle to pinpoint the specific subcategory. The second is clarification-conditioned SQL generation: even after receiving useful user information, models often still fail in the final resolution step. Providing the ground truth category yields large gains in both execution and feedback across both datasets, yet ambiguous-question execution remains low even under oracle category labels. We release our code for data generation and benchmark on GitHub (https://github.com/giosullutrone/ABISS-Evaluating-Text-to-SQL-Systems-Through-Agent-Interaction).

Figures

Figures reproduced from arXiv: 2607.23340 by Georgia Koutrika, Giovanni Sullutrone, Luca Sala, Sania Aftar, Sonia Bergamaschi.

Figure 1
Figure 1. Figure 1: Category distribution of generated questions for [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-category performance under the Ground Truth (GT) and Predicted modes, averaged across the evaluated models. Left: Execution Accuracy (EX) for answerable and ambiguous categories. Right: Feedback Accuracy (FB) for unanswerable categories [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [1]

    Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and Improving Text-to-SQL Generation under Ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 7053–7074. doi:10.186...

    doi:10.18653/v1/2023.emnlp- 2023

  2. [2]

    Shuaichen Chang and Eric Fosler-Lussier. 2023. How to Prompt LLMs for Text- to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings.arXiv preprint arXiv:2305.11853(2023)

    Pith/arXiv arXiv 2023

  3. [3]

    Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng, and Zhiguo Wang. 2025. PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the As...

    2025

  4. [4]

    Venkatesh Emani, Vivek Pandit, Victor Shnayder, Wenjing Wang, and Carlo Curino

    Avrilia Floratou, Fotis Psallidas, Fuheng Zhao, Shaleen Deep, Gunther Hagleither, Wangda Tan, Joyce Cahoon, Rana Alotaibi, Jordan Henkel, Abhik Singla, Alex Van Grootel, Brandon Chow, Kai Deng, Katherine Lin, Marcos Campos, K. Venkatesh Emani, Vivek Pandit, Victor Shnayder, Wenjing Wang, and Carlo Curino. 2024. NL2SQL is a solved problem... Not!. InConfer...

    2024

  5. [5]

    Jonathan Fürst, Catherine Kosten, Farhad Nooralahzadeh, Yi Zhang, and Kurt Stockinger. 2025. Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries. InProceedings 28th International Conference on Extending Database Technology, EDBT 2025, Barcelona, Spain, March 25-28, 2025, Alkis Simitsis, Bettina Kemme, Anna Queralt, Oscar...

    doi:10.48786/edbt.2025.13 2025

  6. [6]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation.Proc. VLDB Endow.17, 5 (Jan. 2024), 1132–1145. doi:10. 14778/3641204.3641221

    arXiv 2024

  7. [7]

    Izzeddin Gür, Semih Yavuz, Yu Su, and Xifeng Yan. 2018. Dialsql: Dialogue based structured query generation. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1339–1349. Giovanni Sullutrone, Luca Sala, Sania Aftar, Georgia Koutrika, and Sonia Bergamaschi

    2018

  8. [8]

    Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, and Reynold Cheng. 2025. BIRD-INTERACT: Re-imagining Text-to-SQL Ev...

    arXiv 2025

  9. [9]

    Fei Li and Hosagrahar V Jagadish. 2014. Constructing an interactive natural language interface for relational databases.Proceedings of the VLDB Endowment 8, 1 (2014), 73–84

    2014

  10. [10]

    Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025. OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale.Proc. VLDB Endow.18, 11 (July 2025), 4695–4709. doi:10.14778/3749646.3749723

    arXiv 2025

  11. [11]

    Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: decoupling schema linking and skeleton parsing for text-to-SQL. InProceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial In...

    doi:10.1609/aaai.v37i11.26535 2023

  12. [12]

    Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. CodeS: Towards Building Open-source Language Models for Text-to-SQL.Proc. ACM Manag. Data2, 3, Article 127 (may 2024), 28 pages. doi:10.1145/3654930

    doi:10.1145/3654930 2024

  13. [13]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, and et al. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing S...

    2023

  14. [14]

    What Do You Mean by That?

    Yuntao Li, Bei Chen, Qian Liu, Yan Gao, Jian-Guang Lou, Yan Zhang, and Dong- mei Zhang. 2020. “What Do You Mean by That?” A Parser-Independent Interac- tive Approach for Enhancing Text-to-SQL. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Associa...

    work page doi:10.18653/v1/2020.emnlp-main.561 2020

  15. [15]

    NVIDIA. 2025. NVIDIA Nemotron 3: Efficient and Open Intelligence. https: //arxiv.org/abs/2512.20856 White Paper

    Pith/arXiv arXiv 2025

  16. [16]

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Apple- baum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives,...

    Pith/arXiv arXiv 2025

  17. [17]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2197, 31 pages

    2024

  18. [18]

    Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: decomposed in- context learning of text-to-SQL with self-correction. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 1577, 10 pages

    2023

  19. [19]

    Mohammadreza Pourreza and Davood Rafiei. 2024. Din-sql: Decomposed in- context learning of text-to-sql with self-correction.Advances in Neural Informa- tion Processing Systems36 (2024)

    2024

  20. [20]

    Luyu Qiu, Jianing Li, Chi Su, and Lei Chen. 2025. Interactive Text-to-SQL via Expected Information Gain for Disambiguation.ArXivabs/2507.06467 (2025). https://api.semanticscholar.org/CorpusID:280083151

    Pith/arXiv arXiv 2025

  21. [21]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  22. [22]

    Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5

    2026

  23. [23]

    Irina Saparina and Mirella Lapata. 2024. AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 90600–90628. https://proceedings.neurips.cc/ paper_files/pap...

    2024

  24. [24]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieil- lard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas B...

    Pith/arXiv arXiv 2025

  25. [25]

    Mistral Team. 2025. Mistral Small 3.1 | Mistral AI. https://mistral.ai/news/mistral- small-3-1

    2025

  26. [26]

    Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023. Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 5701–5714. doi:10.18653/v...

    doi:10.18653/v1/2023.findings-acl.352 2023

  27. [27]

    Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2025. MAC- SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. InProceedings of the 31st International Conference on Computational Linguistics (COLING 2025). 540–557

    2025

  28. [28]

    Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified formulation and a text-to-sql case study. In2019 Conference on Empirical Methods in Natural Language Processing (EMNLP’19). ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction

    2019

  29. [29]

    Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sun- grok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019. CoSQL: A Conversational Text-t...

    2019

  30. [30]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Proces...

    2018

  31. [31]

    Yusen Zhang, Xiangyu Dong, Shuaichen Chang, Tao Yu, Peng Shi, and Rui Zhang

  32. [2020]

    Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL. arXiv:2010.12634 [cs.CL] https: //arxiv.org/abs/2010.12634 Giovanni Sullutrone, Luca Sala, Sania Aftar, Georgia Koutrika, and Sonia Bergamaschi A Relaxed SQL Equivalence Criteria The equivalence check used in Stages 2 and 5 of the validation pipeline a...

    Pith/arXiv arXiv 2010