Pith. sign in

REVIEW 5 major objections 7 minor 48 references

Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces MMSQL, a multi-turn text-to-SQL benchmark with four question types, and a multi-agent framework that improves LLM performance on it.

desk verdict MMSQL is a genuinely useful benchmark combining multi-turn dialogue with four question types, but the claimed multi-agent gains rest on 149 test rounds with no significance testing, so treat the enhancement result as suggestive, not established. read the letter →

arxiv 2412.17867 v4 pith:ZGYJLCFZ submitted 2024-12-21 cs.CL cs.AI

Ziming Guo , Chao Ma , Yinggang Sun , Tiancheng Zhao , Guangyao Wang , Hai Huang This is my paper · ORCID
classification cs.CLcs.AI
keywords text-to-SQLmulti-turndialoguequestionclassificationmulti-agentframeworklargelanguagemodelsbenchmarkambiguousquestionsunanswerable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that text-to-SQL systems are evaluated too narrowly: most benchmarks score only single-turn, unambiguous queries, while real users ask ambiguous, unanswerable, and off-topic questions across multiple turns. To close this gap it presents MMSQL, a test suite of 6,493 training and 149 test dialogue rounds in which every question is labeled answerable, ambiguous, unanswerable, or improper. Using MMSQL, it measures eight open- and closed-source LLMs and finds that even GPT-4 Turbo performs poorly on ambiguous and unanswerable turns. It then proposes a multi-agent framework with a Question Detector that routes questions to the right answering strategy, and reports that this framework raises TDEX scores by 3 to 8 points on GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B. A sympathetic reader would take away that explicit intent detection is a workable way to make conversational database interfaces more reliable.

What carries the argument

The carrying mechanism is the question-type taxonomy plus the TDEX metric that rewards both correct classification (for unanswerable and improper turns) and correct execution (for answerable and ambiguous turns). The proposed method's engine is the Question Detector agent, which takes the schema subset and the dialogue history, labels the current turn, and routes it: answerable questions go to the Question Decomposer for chain-of-thought sub-query generation; ambiguous questions are rewritten into multiple possible answerable questions, each with candidate SQL, and the ambiguity is explained to the user; unanswerable questions receive an explanation of why; improper questions get a conversational reply. Two auxiliary agents, the Schema Selector and the SQL Refiner, reduce schema noise and repair faulty SQL through execution feedback.

What would settle it

Build an independently human-annotated multi-turn text-to-SQL test set covering the same four question types without any QDA-SQL-generated turns, then rerun GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B with and without the multi-agent framework; if the TDEX gains reported in Table VI shrink to near zero or reverse, the framework's measured benefit is an artifact of the synthetic data generator rather than of conversational understanding.

Watch

Extended reading notes

Core claim

MMSQL is presented as the first test suite to combine four question types (answerable, ambiguous, unanswerable, improper) with multi-turn dialogue, built from refined CoSQL samples and QDA-SQL-generated samples for a total of 6,493 training and 149 test rounds. On this suite, the paper evaluates eight LLMs in a zero-shot setting and reports that GPT-4 Turbo leads with a TDEX of 67.0, that Llama3-70B and Llama3-8B rival GPT-3.5 Turbo, and that all models perform markedly worse on ambiguous questions (for GPT-4 Turbo, 25.9 precision, 70.0 recall) and unanswerable questions than on answerable ones. The paper's core claim is that a four-agent framework — Question Detector, Question Decomposer, Schema Selector, SQL Refiner — improves each baseline it is applied to, with Llama3-70B gaining 7.9 TDEX points (62.8 to 70.7) and Gemini-1.5 Flash gaining 3.0 response-quality points (4.0 to 7.0), and that ablation shows the Question Detector contributes the largest share.

Load-bearing premise

The test suite's 149 rounds, assembled from manually refined CoSQL samples and LLM-generated QDA-SQL samples, adequately represent the distribution of real-world conversational database queries, and the LLM-judge-based quality scores align with human judgments, so that measured gains on MMSQL transfer to live use.

Editorial extensions

If this is right

  • MMSQL provides a reusable evaluation protocol: any text-to-SQL system can now be scored on TDEX, which forces it to say 'I don't know' or 'please clarify' when SQL is not the answer, instead of hallucinating a query.
  • Introducing a Question Detector before SQL generation turns out to be worth 3 to 8 TDEX points on top-grade LLMs, suggesting that explicit intent routing is a higher-leverage intervention than further SQL-tuning.
  • Ambiguous questions should be answered with clarification plus provisional SQL: the paper shows execution accuracy on ambiguous turns rises sharply when models first ask for clarification (e.g., GPT-3.5 Turbo from 34.5% to 49.0%).
  • Open-source Llama3 models are close enough to GPT-4 Turbo on this task (TDEX 62.8 to 64.0 vs 67.0) that reliable conversational text-to-SQL does not require closed APIs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 149-round test set likely underdetermines per-type reliability; a larger human-annotated set could show which question types drive the reported gains and whether unanswerable-turn handling is genuinely improved.
  • The multi-agent design can be viewed as dialogue-state tracking for database interfaces: the Question Detector labels each turn's intent, which suggests the framework could generalize to text-to-API or text-to-workflow tasks where multiple intents compete.
  • Because the natural-language responses are scored by GPT-4o-mini, part of the RQS gains may reflect the judge's preference for polite clarification over terse SQL; a user-study measuring task completion time would test whether the framework helps people get answers faster.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces MMSQL, a multi-turn, multi-type text-to-SQL test suite in which each dialogue turn is labeled as answerable, unanswerable, ambiguous, or improper. It evaluates eight LLMs (closed- and open-source) on this suite using standard SQL metrics plus a new composite metric, TDEX, and an LLM-assisted RQS. It then proposes an LLM-based multi-agent framework with a Schema Selector, Question Detector, Question Decomposer, and SQL Refiner, and reports that the framework improves TDEX, EX, RQS, and F1 for GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B on the MMSQL test set.

Significance. The paper addresses a real and under-studied problem: conversational text-to-SQL queries that are ambiguous, unanswerable, or off-topic. MMSQL is a useful step toward evaluating this broader capability, and the proposed multi-agent framework is a plausible design for handling mixed question types. The authors provide a public dataset and code, use a clearly defined composite metric (TDEX), and include an ablation study. The main contributions are significant conditional on the evaluation being trustworthy; however, the empirical support is weakened by the small 149-round test set, the in-house provenance of part of the test data, missing significance testing, and the absence of comparisons with existing multi-agent systems. If these issues are addressed, the benchmark and framework could be a valuable addition to the text-to-SQL reliability literature.

major comments (5)
  1. [Section V-B, Table VI] The central claim of 'significant performance improvement' is not supported by any significance testing. The test set has only 149 rounds, so the reported +3.0 TDEX gain for GPT-4 Turbo corresponds to about 4-5 questions, and the standard error of a proportion near 0.7 is approximately 3.8 percentage points. With no confidence intervals, paired tests, or corrections for multiple comparisons across models and metrics, the GPT-4 and Gemini gains are indistinguishable from sampling noise. Please report paired significance tests (e.g., McNemar or bootstrap) and confidence intervals for all headline numbers in Table VI, and for the per-type precision/recall values in Table V where the per-type subsets are even smaller.
  2. [Section III-B, Table VI] The evaluation is conducted entirely on an in-house test set of 149 rounds, part of which was generated by QDA-SQL (Ref. [15]), a pipeline from overlapping authorship. Because the proposed multi-agent framework is evaluated only on this same benchmark, the measured improvements may reflect artifacts of the QDA-SQL generation process rather than generalizable gains. To alleviate this, the authors should report results on existing multi-turn benchmarks such as CoSQL or SParC, or on an independent human-annotated held-out sample, and should clarify the exact fraction of test rounds derived from CoSQL versus generated by QDA-SQL.
  3. [Section II-B, Section V-B] The paper does not compare the proposed framework against existing multi-agent text-to-SQL systems such as MAC-SQL, CHASE-SQL, MAG-SQL, or DIN-SQL, even though the related work describes them as leading approaches. Without such a comparison, it is unclear whether the proposed Question Detector and decomposition design offer a real advance over the state of the art or only over the selected zero-shot baselines. Please add at least one strong multi-agent baseline evaluated on the same test set and protocol.
  4. [Table V and Section III-E.b] There is an internal inconsistency between the table and the text. The text states that GPT-4 Turbo's unanswerable precision/recall are 56.9/38.4 and ambiguous precision/recall are 25.9/70.0, but the corresponding row in Table V lists 25.9/70.0 before 56.9/38.4 under the Una. and Amb. columns. This discrepancy affects the paper's key finding that ambiguous questions are the most difficult, and it must be corrected and re-analyzed, since the per-type conclusion may change depending on which assignment is correct.
  5. [Table VII, Section V-C] The ablation study's summary claim that 'removal of any component results in a decrease in performance across these metrics' is contradicted by the w/o refiner row, where F1 Score (70.7) and Average RQS (7.05) are identical to the full multi-agent model. The authors should either correct the claim, quantify the differences with significance testing, or explain why the refiner does not affect these metrics in this ablation.
minor comments (7)
  1. [Section III-E.a] The sentence 'highlighting the robust potential of closed-source models' appears to be a typo; the context discusses Llama3-70B, an open-source model.
  2. [Table V] The table header for precision/recall columns is difficult to parse. Please reformat the table so that each question type has clearly labeled Precision and Recall columns.
  3. [Section II-A] The word 'refsources' in 'require considerable refsources and time for training' should be 'resources'.
  4. [Section III-B] The dataset description uses 'rounds' and 'dialogues' interchangeably; the relationship between '6,493 training rounds' and the '# Dialogues 6,493' in Table I should be clarified.
  5. [Table II] The reported p-value of 3.4e-23 for a Pearson correlation of 0.71 on 100 samples appears implausible; please report the correlation methodology, the sample size per question type, and the exact test used.
  6. [Section IV-A] The Schema Selector activation threshold is not specified, and no sensitivity analysis is provided for this threshold or for the QDA-SQL retention threshold; please state the values used and, if feasible, show that the main results are stable over a range of thresholds.
  7. [Section III-C, Equation (1)] For ambiguous questions, TDEX currently rewards only execution correctness and does not directly reward the model for flagging ambiguity or providing clarification; please clarify whether this is intentional and discuss how the RQS metric covers that aspect.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the benchmark and the claimed enhancement are evaluated on the authors' own QDA-SQL-derived test set, but the framework result is not a definitional identity.

  1. other [Section III-B (Construction); Appendix B (Design and Effectiveness Study of QDA-SQL)]
    "The MMSQL data comes from two sources: refined samples from CoSQL and new samples generated by QDA-SQL (Questions Enhanced Dialogue Augmentation for Multi-turn Text-to-SQL) [15]. QDA-SQL uses Chain of Thought (CoT) [25] to guide new sample generation through step-by-step reasoning. ... In this study, we use QDA-SQL to expand the original datasets by inputting samples from SParC and CoSQL, while adhering to MMSQL's classification and formatting standards."

    QDA-SQL is prior work by overlapping authors ([15]: Sun, Guo, Zhao, et al.), and it was explicitly designed to generate data that conforms to MMSQL's own four-type taxonomy. The benchmark's claim to 'simulate real-world scenarios' is therefore supported by an in-house generation pipeline whose labels are produced to match the benchmark's definitions, rather than by an independent external ground truth. All reported improvements of the proposed multi-agent framework are measured on this same 149-round test set, so the central evidence for 'effectively enhances' rests on a self-generated, self-cited evaluation loop.

full rationale

The paper does not contain an equation-level circularity: TDEX, EM, EX, and RQS are defined independently of the multi-agent framework, and the framework's outputs are not built into the metric definitions. The strongest circularity concern is provenance: MMSQL's test rounds are partly generated by QDA-SQL, the same group's prior system, which was prompted to follow MMSQL's own classification and formatting standards. Since the framework is evaluated only on this self-constructed benchmark, the headline gains in Table VI are not independently corroborated on an external test set. The CoSQL component and manual curation provide some external anchor, so the circularity is not total, but the self-citation is load-bearing for the claim that MMSQL represents realistic multi-turn scenarios and that the framework generalizes beyond that in-house distribution. Statistical issues (N=149, no significance tests, a +3.0 TDEX gain for GPT-4 Turbo being within sampling noise) are real but belong to correctness risk rather than circularity; they do not by themselves raise the circularity score. Overall, the central enhancement claim has empirical content but is entangled with the authors' own benchmark-generation pipeline, justifying a moderate score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small, self-constructed test set and an LLM judge. The framework itself adds no fitted parameters but relies on unstated thresholds and benchmark construction choices, so the ledger stays short but the independence of the evidence is limited.

free parameters (2)
  • Schema Selector activation threshold
    Section IV-A states the selector is used only when schema size exceeds a specific threshold, but the threshold value is never reported; it is an unstated design choice that could affect results on different database sizes.
  • QDA-SQL retention score threshold = score > 9
    Appendix B describes that generated Q&A pairs are kept only when Gemini Pro's quality score exceeds 9; this arbitrary cutoff shapes the benchmark distribution.
assumptions (4)
  • domain assumption The four question types (answerable, unanswerable, ambiguous, improper) exhaustively cover real-world conversational queries to a text-to-SQL system.
    Section III-A defines exactly these four types and the benchmark labels every turn accordingly; if real queries fall outside these types, the coverage and realism claims weaken.
  • domain assumption GPT-4o-mini's LLM-as-judge scores are a valid proxy for human judgment of response quality.
    Section III-C validates on 100 samples with three human experts and reports correlations, but the full test set is scored by the LLM judge only.
  • domain assumption The curated 149-round test set is representative of real deployment distributions.
    Section III-B states the test set reflects a more realistic distribution citing [7], but no quantitative evidence about the target distribution is given.
  • domain assumption QDA-SQL-generated questions are natural enough for training and evaluation.
    Appendix B reports an internal pairwise comparison where 62% of enhanced samples are rated superior, but this rating is itself LLM-based.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types." pith.science (2026). https://pith.science/paper/ZGYJLCFZ

@misc{pith2026241217867,
  author       = {Pith},
  title        = {Pith review of: Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZGYJLCFZ}},
  note         = {Machine review of arXiv:2412.17867}
}
read the original abstract

Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries. Our dataset and code are publicly available at https://mcxiaoxiao.github.io/MMSQL.

Figures

Figures reproduced from arXiv: 2412.17867 by the authors.

Figure 1
Figure 1. A four-turn dialogue example that includes four different types of Q&A pairs. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of question types in the curated MMSQL dataset, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance analysis of model outputs on different Q&A scenarios [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The overview of our multi-agent framework comprises four components: (i) the Schema Selector, which narrows down the database schema to focus [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Pairwise comparison of original and QDA-SQL enhanced dataset [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages

  1. [15]

    Chase: A large-scale and pragmatic chinese dataset for cross-database context-dependent text-to-sql,

    J. Guo, Z. Si, Y . Wang, Q. Liu, M. Fan, J.-G. Lou, Z. Yang, and T. Liu, “Chase: A large-scale and pragmatic chinese dataset for cross-database context-dependent text-to-sql,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long...

    work page 2021

  2. [1]

    Interactive-t2s: Multi-turn interactions for text-to-sql with large language models,

    G. Xiong, J. Bao, H. Jiang, Y . Song, and W. Zhao, “Interactive-t2s: Multi-turn interactions for text-to-sql with large language models,”

    work page

  3. [2]

    Chatbi: Towards natural language to complex business intelligence sql,

    J. Lian, X. Liu, Y . Shao, Y . Dong, M. Wang, Z. Wei, T. Wan, M. Dong, and H. Yan, “Chatbi: Towards natural language to complex business intelligence sql,” 2024. [Online]. Available: https: //arxiv.org/abs/2405.00527

    arXiv 2024

  4. [3]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,

    J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo et al., “Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

  5. [4]

    Conda: state-based data augmentation for context-dependent text-to-sql,

    D. Wang, L. Dou, W. Che, J. Wang, J. Liu, L. Li, J. Shang, L. Tao, J. Zhang, C. Fu et al. , “Conda: state-based data augmentation for context-dependent text-to-sql,” International Journal of Machine Learn- ing and Cybernetics , pp. 1–12, 2024

    work page 2024

  6. [5]

    Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql,

    M. Pourreza, H. Li, R. Sun, Y . Chung, S. Talaei, G. T. Kakkar, Y . Gan, A. Saberi, F. Ozcan, and S. O. Arik, “Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql,”

  7. [6]

    Mac-sql: A multi-agent collaborative framework for text-to-sql,

    B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, L. Chai, Z. Yan, Q.-W. Zhang, D. Yin, X. Sun, and Z. Li, “Mac-sql: A multi-agent collaborative framework for text-to-sql,” 2024. [Online]. Available: https://arxiv.org/abs/2312.11242

    arXiv 2024

  8. [7]

    Available: https://arxiv.org/abs/2410.01943

    [Online]. Available: https://arxiv.org/abs/2410.01943

    arXiv

Show all 48 references
  1. [8]

    A survey of hallucination in large foundation models,

    V . Rawte, A. Sheth, and A. Das, “A survey of hallucination in large foundation models,” 2023. [Online]. Available: https: //arxiv.org/abs/2309.05922

    2023 arXiv

  2. [9]

    Know what i don’t know: Handling ambiguous and unknown questions for text-to-sql,

    B. Wang, Y . Gao, Z. Li, and J.-G. Lou, “Know what i don’t know: Handling ambiguous and unknown questions for text-to-sql,” in Findings of the Association for Computational Linguistics: ACL 2023 , 2023, pp. 5701–5714

    2023

  3. [10]

    Trustsql: A reliability benchmark for text-to-sql models with diverse unanswerable questions,

    G. Lee, W. Chay, S. Cho, and E. Choi, “Trustsql: A reliability benchmark for text-to-sql models with diverse unanswerable questions,” arXiv preprint arXiv:2403.15879, 2024

    2024 arXiv

  4. [11]

    Ambrosia: A benchmark for parsing ambiguous questions into database queries,

    I. Saparina and M. Lapata, “Ambrosia: A benchmark for parsing ambiguous questions into database queries,” 2024. [Online]. Available: https://arxiv.org/abs/2406.19073

    2024 arXiv

  5. [12]

    Sparc: Cross-domain semantic parsing in context,

    T. Yu, R. Zhang, M. Yasunaga, Y . C. Tan, X. V . Lin, S. Li, H. Er, I. Li, B. Pang, T. Chen et al. , “Sparc: Cross-domain semantic parsing in context,” arXiv preprint arXiv:1906.02285 , 2019. [Online]. Available: https://yale-lily.github.io/sparc

    1906 arXiv

  6. [13]

    Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries,

    X. He, M. Zhou, X. Xu, X. Ma, R. Ding, L. Du, Y . Gao, R. Jia, X. Chen, S. Han, Z. Yuan, and D. Zhang, “Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries,” 2023. [Online]. Available: https://arxiv.org/abs/2312.13671

    2023 arXiv

  7. [14]

    Did you ask a good question? a cross-domain question intention classification benchmark for text-to-sql,

    Y . Zhang, X. Dong, S. Chang, T. Yu, P. Shi, and R. Zhang, “Did you ask a good question? a cross-domain question intention classification benchmark for text-to-sql,” arXiv preprint arXiv:2010.12634 , 2020

    2010 arXiv

  8. [16]

    Seq2sql: Generating structured queries from natural language using reinforcement learning,

    V . Zhong, C. Xiong, and R. Socher, “Seq2sql: Generating structured queries from natural language using reinforcement learning,” 2017. [Online]. Available: https://arxiv.org/abs/1709.00103

    2017 arXiv

  9. [17]

    Qda-sql: Questions enhanced dialogue augmentation for multi-turn text- to-sql,

    Y . Sun, Z. Guo, H. Yu, C. Liu, X. Li, B. Wang, X. Yu, and T. Zhao, “Qda-sql: Questions enhanced dialogue augmentation for multi-turn text- to-sql,” arXiv preprint arXiv:2406.10593 , 2024

    2024 arXiv

  10. [18]

    CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases,

    T. Yu, R. Zhang, H. Er, S. Li, E. Xue, B. Pang, X. V . Lin, Y . C. Tan, T. Shi, Z. Li, Y . Jiang, M. Yasunaga, S. Shim, T. Chen, A. Fabbri, Z. Li, L. Chen, Y . Zhang, S. Dixit, V . Zhang, C. Xiong, R. Socher, W. Lasecki, and D. Radev, “CoSQL: A conversational text-to-SQL chall...

    2019

  11. [19]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,

    T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman et al. , “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” arXiv preprint arXiv:1809.08887 , 2018. [Online]. Available: https:/...

    2018 arXiv

  12. [20]

    Picard: Parsing incremen- tally for constrained auto-regressive decoding from language models,

    T. Scholak, N. Schucher, and D. Bahdanau, “Picard: Parsing incremen- tally for constrained auto-regressive decoding from language models,” arXiv preprint arXiv:2109.05093 , 2021

    2021 arXiv

  13. [21]

    A comprehensive exploration on wikisql with table-aware word contextualization,

    W. Hwang, J. Yim, S. Park, and M. Seo, “A comprehensive exploration on wikisql with table-aware word contextualization,” 2019. [Online]. Available: https://arxiv.org/abs/1902.01069

    2019 arXiv

  14. [22]

    Text-to-sql with large language models: Exploring the promise and pitfalls,

    L. Sala, G. Sullutrone, and S. Bergamaschi, “Text-to-sql with large language models: Exploring the promise and pitfalls,” in Proceedings of the 32nd Symposium of Advanced Database Systems, Villasimius, Italy, June 23rd to 26th, 2024 , ser. CEUR Workshop Proceedings, M. Atzori,...

    2024

  15. [23]

    RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers,

    B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, “RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetre...

    2020

  16. [24]

    Din-sql: Decomposed in-context learning of text-to-sql with self-correction,

    M. Pourreza and D. Rafiei, “Din-sql: Decomposed in-context learning of text-to-sql with self-correction,” 2023. [Online]. Available: https://arxiv.org/abs/2304.11015

    2023 arXiv

  17. [25]

    Optimization modeling and verification from problem specifications using a multi-agent multi-stage llm framework,

    M. Mostajabdaveh, T. T. Yu, R. Ramamonjison, G. Carenini, Z. Zhou, and Y . Zhang, “Optimization modeling and verification from problem specifications using a multi-agent multi-stage llm framework,” INFOR: Information Systems and Operational Research , pp. 1–19, 2024

    2024

  18. [26]

    Evaluating text-to-sql model failures on real-world data,

    M. Ganti, L. Orr, and S. Wu, “Evaluating text-to-sql model failures on real-world data,” in 2024 IEEE 40th International Conference on Data Engineering (ICDE), 2024, pp. 1–1

    2024

  19. [27]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  20. [28]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2023. [Online]. Available: https://arxiv.org/abs/1910.10683

    2023 arXiv

  21. [29]

    Benchmarking and improving text-to-SQL generation under ambiguity,

    A. Bhaskar, T. Tomar, A. Sathe, and S. Sarawagi, “Benchmarking and improving text-to-SQL generation under ambiguity,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Co...

    2023

  22. [30]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Lin, W. X. Zhao, Z. Wei, and J.-R. Wen, “A survey on large language model based autonomous agents,” 2023

    2023

  23. [31]

    Cognitive mirage: A review of hallucinations in large language models. corr abs/2309.06794 (2023),

    H. Ye, T. Liu, A. Zhang, W. Hua, and W. Jia, “Cognitive mirage: A review of hallucinations in large language models. corr abs/2309.06794 (2023),” 2023

    2023 arXiv

  24. [32]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “Metagpt: Meta programming for a multi-agent collaborative framework,” 2024. [Online]. Available: https://arxiv.org/abs/2308.00352

    2024 arXiv

  25. [33]

    AutoGPT,

    Significant Gravitas, “AutoGPT,” https://github.com/Significant-Gravita s/AutoGPT, 2023, mIT License

    2023

  26. [34]

    Instruction-following evaluation for large language models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,”

  27. [35]

    Mag-sql: Multi-agent generative approach with soft schema linking and iterative sub-sql refinement for text-to-sql,

    W. Xie, G. Wu, and B. Zhou, “Mag-sql: Multi-agent generative approach with soft schema linking and iterative sub-sql refinement for text-to-sql,” 2024. [Online]. Available: https://arxiv.org/abs/2408.07930

    2024 arXiv

  28. [36]

    Mt-eval: A multi-turn capabili- ties evaluation benchmark for large language models,

    W.-C. Kwan, X. Zeng, Y . Jiang, Y . Wang, L. Li, L. Shang, X. Jiang, Q. Liu, and K.-F. Wong, “Mt-eval: A multi-turn capabili- ties evaluation benchmark for large language models,” arXiv preprint arXiv:2401.16745, 2024

    2024 arXiv

  29. [37]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  30. [38]

    G-eval: NLG evaluation using gpt-4 with better human alignment,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: NLG evaluation using gpt-4 with better human alignment,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for ...

    2023

  31. [39]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun, “Chatdev: Communicative agents for software development,” 2024. [Online]. Available: https://arxiv.org/abs/2307.07924

    2024 arXiv

  32. [40]

    Llm-generated natural language meets scaling laws: New explorations and data augmentation methods,

    Z. Wang, G. Xu, and M. Ren, “Llm-generated natural language meets scaling laws: New explorations and data augmentation methods,” 2024. [Online]. Available: https://arxiv.org/abs/2407.00322

    2024 arXiv

  33. [41]

    Large language models are not fair evaluators,

    P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y . Cao, Q. Liu, T. Liu, and Z. Sui, “Large language models are not fair evaluators,”

  34. [42]

    Available: https://arxiv.org/abs/2305.17926

    [Online]. Available: https://arxiv.org/abs/2305.17926

    arXiv

  35. [43]

    Wizardlm: Empowering large language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang, “Wizardlm: Empowering large language models to follow complex instructions,” 2023. [Online]. Available: https: //arxiv.org/abs/2304.12244

    2023 arXiv

  36. [44]

    Grade score: Quantifying llm performance in option selection,

    D. Iourovitski, “Grade score: Quantifying llm performance in option selection,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12043 APPENDIX A IMPLEMENTATION DETAILS To ensure reproducibility, we applied greedy decoding strategies during both inference and evaluation. ...

    2024 arXiv

  37. [45]

    Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities,

    W. Lu, R. K. Luu, and M. J. Buehler, “Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities,” 2024. [Online]. Available: https://arxiv.org/abs/2409.03444

    2024 arXiv

  38. [46]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globe...

    2023

  39. [2023]

    Available: https://arxiv.org/abs/2311.07911

    [Online]. Available: https://arxiv.org/abs/2311.07911

    arXiv

  40. [2024]

    Available: https://arxiv.org/abs/2408.11062

    [Online]. Available: https://arxiv.org/abs/2408.11062

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.