{"id":"9b0b27ce-cb35-4178-8265-456e64ea1edb","arxiv_id":"2412.17867","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MMSQL is a multi-turn text-to-SQL benchmark with four question types, and a multi-agent framework with a Question Detector improves LLM performance on it.","lead":"This paper introduces MMSQL, a multi-turn text-to-SQL benchmark that labels each user question as answerable, unanswerable, ambiguous, or improper, and evaluates eight LLMs on it. It also proposes a four-agent LLM framework that detects question types and rewrites ambiguous questions, reporting improved scores on MMSQL.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed gain from the multi-agent framework rests on a 149-round test set with no significance tests; a +3.0 TDEX point gain for GPT-4 Turbo is within sampling noise.","rationale":"The reader's weakest assumption flagged test-set representativeness; I flag statistical power on the 149-round set. The two are related: even if the test set were perfectly representative, the N=149 evaluation cannot support the 'significant improvement' claim without error bars or a paired significance test. The dataset itself may still be a useful resource, and the multi-agent framework may genuinely help; but the empirical support in Table VI is not yet sufficient. This concern sharpens the condition already implied by the reader's CONDITIONAL verdict: the authors should provide significance testing or a larger, independently constructed test set before the enhancement claim is accepted. I therefore do not change the verdict, but I emphasize that the reported point gains should not be read as reliable effect sizes until such testing is done.","tokens_in":18015,"tokens_out":6718,"duration_ms":59820,"concrete_test":"Run a paired McNemar test on per-round TDEX correctness for each model in Table VI (with vs. without the multi-agent framework), and compute 95% bootstrap confidence intervals over the 149 test rounds. If the GPT-4 Turbo +3.0 point gain has a confidence interval overlapping zero or a McNemar p-value above 0.05, the claimed significant improvement is not supported by the current data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-B reports the central enhancement claim in Table VI: TDEX gains of +3.0 (GPT-4 Turbo), +3.6 (Gemini-1.5 Flash), and +7.9 (Llama3-70B) over baselines, and the Discussion calls this a 'significant performance improvement.' All numbers come from a single evaluation on the 149 test rounds described in Section III-B. With N=149, each 1.0 percentage point is about 1.5 questions, so the GPT-4 gain is roughly 4-5 questions. The standard error of a proportion near 0.7 is about 3.8 points, and the standard error of the difference between two correlated proportions is larger; the +3.0 gain is therefore not distinguishable from chance. The Llama3 gain of about 12 questions is more encouraging, but no confidence interval, paired test, or correction for multiple comparisons (3 models x 4 metrics) is provided. Per-type precision and recall in Table V are based on even smaller subsets; for example, GPT-4's ambiguous recall of 70% with roughly 14 true positives has a 95% CI spanning tens of percentage points. The paper's evidence for 'effectively enhances' is thus load-bearing on unquantified small-sample noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MMSQL, a multi-turn, multi-type text-to-SQL test suite in which each dialogue turn is labeled as answerable, unanswerable, ambiguous, or improper. It evaluates eight LLMs (closed- and open-source) on this suite using standard SQL metrics plus a new composite metric, TDEX, and an LLM-assisted RQS. It then proposes an LLM-based multi-agent framework with a Schema Selector, Question Detector, Question Decomposer, and SQL Refiner, and reports that the framework improves TDEX, EX, RQS, and F1 for GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B on the MMSQL test set.","tokens_in":18282,"tokens_out":4302,"duration_ms":37701,"significance":"The paper addresses a real and under-studied problem: conversational text-to-SQL queries that are ambiguous, unanswerable, or off-topic. MMSQL is a useful step toward evaluating this broader capability, and the proposed multi-agent framework is a plausible design for handling mixed question types. The authors provide a public dataset and code, use a clearly defined composite metric (TDEX), and include an ablation study. The main contributions are significant conditional on the evaluation being trustworthy; however, the empirical support is weakened by the small 149-round test set, the in-house provenance of part of the test data, missing significance testing, and the absence of comparisons with existing multi-agent systems. If these issues are addressed, the benchmark and framework could be a valuable addition to the text-to-SQL reliability literature.","major_comments":[{"comment":"The central claim of 'significant performance improvement' is not supported by any significance testing. The test set has only 149 rounds, so the reported +3.0 TDEX gain for GPT-4 Turbo corresponds to about 4-5 questions, and the standard error of a proportion near 0.7 is approximately 3.8 percentage points. With no confidence intervals, paired tests, or corrections for multiple comparisons across models and metrics, the GPT-4 and Gemini gains are indistinguishable from sampling noise. Please report paired significance tests (e.g., McNemar or bootstrap) and confidence intervals for all headline numbers in Table VI, and for the per-type precision/recall values in Table V where the per-type subsets are even smaller.","section":"Section V-B, Table VI"},{"comment":"The evaluation is conducted entirely on an in-house test set of 149 rounds, part of which was generated by QDA-SQL (Ref. [15]), a pipeline from overlapping authorship. Because the proposed multi-agent framework is evaluated only on this same benchmark, the measured improvements may reflect artifacts of the QDA-SQL generation process rather than generalizable gains. To alleviate this, the authors should report results on existing multi-turn benchmarks such as CoSQL or SParC, or on an independent human-annotated held-out sample, and should clarify the exact fraction of test rounds derived from CoSQL versus generated by QDA-SQL.","section":"Section III-B, Table VI"},{"comment":"The paper does not compare the proposed framework against existing multi-agent text-to-SQL systems such as MAC-SQL, CHASE-SQL, MAG-SQL, or DIN-SQL, even though the related work describes them as leading approaches. Without such a comparison, it is unclear whether the proposed Question Detector and decomposition design offer a real advance over the state of the art or only over the selected zero-shot baselines. Please add at least one strong multi-agent baseline evaluated on the same test set and protocol.","section":"Section II-B, Section V-B"},{"comment":"There is an internal inconsistency between the table and the text. The text states that GPT-4 Turbo's unanswerable precision/recall are 56.9/38.4 and ambiguous precision/recall are 25.9/70.0, but the corresponding row in Table V lists 25.9/70.0 before 56.9/38.4 under the Una. and Amb. columns. This discrepancy affects the paper's key finding that ambiguous questions are the most difficult, and it must be corrected and re-analyzed, since the per-type conclusion may change depending on which assignment is correct.","section":"Table V and Section III-E.b"},{"comment":"The ablation study's summary claim that 'removal of any component results in a decrease in performance across these metrics' is contradicted by the w/o refiner row, where F1 Score (70.7) and Average RQS (7.05) are identical to the full multi-agent model. The authors should either correct the claim, quantify the differences with significance testing, or explain why the refiner does not affect these metrics in this ablation.","section":"Table VII, Section V-C"}],"minor_comments":[{"comment":"The sentence 'highlighting the robust potential of closed-source models' appears to be a typo; the context discusses Llama3-70B, an open-source model.","section":"Section III-E.a"},{"comment":"The table header for precision/recall columns is difficult to parse. Please reformat the table so that each question type has clearly labeled Precision and Recall columns.","section":"Table V"},{"comment":"The word 'refsources' in 'require considerable refsources and time for training' should be 'resources'.","section":"Section II-A"},{"comment":"The dataset description uses 'rounds' and 'dialogues' interchangeably; the relationship between '6,493 training rounds' and the '# Dialogues 6,493' in Table I should be clarified.","section":"Section III-B"},{"comment":"The reported p-value of 3.4e-23 for a Pearson correlation of 0.71 on 100 samples appears implausible; please report the correlation methodology, the sample size per question type, and the exact test used.","section":"Table II"},{"comment":"The Schema Selector activation threshold is not specified, and no sensitivity analysis is provided for this threshold or for the QDA-SQL retention threshold; please state the values used and, if feasible, show that the main results are stable over a range of thresholds.","section":"Section IV-A"},{"comment":"For ambiguous questions, TDEX currently rewards only execution correctness and does not directly reward the model for flagging ambiguity or providing clarification; please clarify whether this is intentional and discuss how the RQS metric covers that aspect.","section":"Section III-C, Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The close relationship between the QDA-SQL data-generation pipeline (Ref. [15]) and the present submission, combined with the exclusive use of the resulting benchmark for the headline evaluation, is a novelty and independence concern that the editor may wish to probe during the revision process. The manuscript would be substantially stronger if the authors evaluated on an external benchmark or an independently annotated sample."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know two things about arXiv:2412.17867. First, the MMSQL benchmark is a real contribution: it combines multi-turn dialogue with four question types (answerable, unanswerable, ambiguous, improper) in one test suite, which none of the cited prior datasets do. That fills a genuine gap for evaluating text-to-SQL systems in conversational settings. Second, the paper's headline claim—that their multi-agent framework 'significantly enhances' baseline models—is not actually supported by the evidence as reported. The central comparison in Table VI uses 149 test rounds. A +3.0 TDEX gain for GPT-4 Turbo is roughly 4–5 questions, and the standard error on a proportion near 0.7 with N=149 is about 3.8 points. The +3.0 and +3.6 gains are within sampling noise. The Llama3-70B gain of +7.9 is more encouraging, but there are no confidence intervals, paired tests, or corrections for multiple comparisons anywhere in the paper. The stress-test note is on target.\n\nWhat the paper does well: the benchmark construction is transparent. They describe refining CoSQL samples and generating new ones with QDA-SQL, they report a manual review of generated data quality, and they provide per-type precision/recall for eight models. The dataset and code are public. The TDEX metric is clearly defined and reasonable for this task. The RQS metric is LLM-as-judge, but they calibrate it against human ratings on 100 samples and report correlations. That is more than many benchmark papers do.\n\nThe soft spots, in proportion: the small test set is the load-bearing weakness, and it undermines the framework claim more than the benchmark claim. The benchmark itself can still be useful for future evaluations, though 149 rounds is small for drawing fine distinctions. A second issue is a moderate circularity burden: the test set is partly generated by the authors' own QDA-SQL pipeline, and the framework is evaluated only on that test set, with no comparison against existing multi-agent systems like MAC-SQL or CHASE-SQL. A third, minor issue: the paper says the improvements are 'significant' but provides no statistical support. That wording should be toned down.\n\nWho is this for? Researchers working on conversational text-to-SQL reliability, especially those who want a benchmark that includes unanswerable, ambiguous, and improper queries in a multi-turn setting. The benchmark is worth having even if you are skeptical of the multi-agent results.\n\nMy recommendation: send it to peer review. The benchmark contribution deserves referee time, and the framework claim can be fixed with either a larger test set, significance tests, or a softer conclusion. I would not desk-reject it. If I were the reviewer, I would push for the statistical rigor before accepting the enhancement claim.","headline":"MMSQL is a genuinely useful benchmark combining multi-turn dialogue with four question types, but the claimed multi-agent gains rest on 149 test rounds with no significance testing, so treat the enhancement result as suggestive, not established.","tokens_in":840,"tokens_out":1874,"would_cite":true,"duration_ms":25150,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MMSQL, a multi-turn text-to-SQL benchmark with four question types, and a multi-agent framework that improves LLM performance on it.","keywords":["text-to-SQL","multi-turn dialogue","question classification","multi-agent framework","large language models","benchmark","ambiguous questions","unanswerable questions"],"falsifier":"Build an independently human-annotated multi-turn text-to-SQL test set covering the same four question types without any QDA-SQL-generated turns, then rerun GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B with and without the multi-agent framework; if the TDEX gains reported in Table VI shrink to near zero or reverse, the framework's measured benefit is an artifact of the synthetic data generator rather than of conversational understanding.","tokens_in":17845,"feed_emoji":"💬","tokens_out":7783,"duration_ms":63307,"temperature":0.7,"pith_summary":"The paper argues that text-to-SQL systems are evaluated too narrowly: most benchmarks score only single-turn, unambiguous queries, while real users ask ambiguous, unanswerable, and off-topic questions across multiple turns. To close this gap it presents MMSQL, a test suite of 6,493 training and 149 test dialogue rounds in which every question is labeled answerable, ambiguous, unanswerable, or improper. Using MMSQL, it measures eight open- and closed-source LLMs and finds that even GPT-4 Turbo performs poorly on ambiguous and unanswerable turns. It then proposes a multi-agent framework with a Question Detector that routes questions to the right answering strategy, and reports that this framework raises TDEX scores by 3 to 8 points on GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B. A sympathetic reader would take away that explicit intent detection is a workable way to make conversational database interfaces more reliable.","feed_headline":"Framework lifts chat-to-SQL accuracy 3 to 8 points","feed_subtitle":"New MMSQL benchmark grades models on ambiguous, unanswerable, answerable, and improper questions","key_machinery":"The carrying mechanism is the question-type taxonomy plus the TDEX metric that rewards both correct classification (for unanswerable and improper turns) and correct execution (for answerable and ambiguous turns). The proposed method's engine is the Question Detector agent, which takes the schema subset and the dialogue history, labels the current turn, and routes it: answerable questions go to the Question Decomposer for chain-of-thought sub-query generation; ambiguous questions are rewritten into multiple possible answerable questions, each with candidate SQL, and the ambiguity is explained to the user; unanswerable questions receive an explanation of why; improper questions get a conversational reply. Two auxiliary agents, the Schema Selector and the SQL Refiner, reduce schema noise and repair faulty SQL through execution feedback.","core_discovery":"MMSQL is presented as the first test suite to combine four question types (answerable, ambiguous, unanswerable, improper) with multi-turn dialogue, built from refined CoSQL samples and QDA-SQL-generated samples for a total of 6,493 training and 149 test rounds. On this suite, the paper evaluates eight LLMs in a zero-shot setting and reports that GPT-4 Turbo leads with a TDEX of 67.0, that Llama3-70B and Llama3-8B rival GPT-3.5 Turbo, and that all models perform markedly worse on ambiguous questions (for GPT-4 Turbo, 25.9 precision, 70.0 recall) and unanswerable questions than on answerable ones. The paper's core claim is that a four-agent framework — Question Detector, Question Decomposer, Schema Selector, SQL Refiner — improves each baseline it is applied to, with Llama3-70B gaining 7.9 TDEX points (62.8 to 70.7) and Gemini-1.5 Flash gaining 3.0 response-quality points (4.0 to 7.0), and that ablation shows the Question Detector contributes the largest share.","pith_inferences":["The 149-round test set likely underdetermines per-type reliability; a larger human-annotated set could show which question types drive the reported gains and whether unanswerable-turn handling is genuinely improved.","The multi-agent design can be viewed as dialogue-state tracking for database interfaces: the Question Detector labels each turn's intent, which suggests the framework could generalize to text-to-API or text-to-workflow tasks where multiple intents compete.","Because the natural-language responses are scored by GPT-4o-mini, part of the RQS gains may reflect the judge's preference for polite clarification over terse SQL; a user-study measuring task completion time would test whether the framework helps people get answers faster."],"forward_implications":["MMSQL provides a reusable evaluation protocol: any text-to-SQL system can now be scored on TDEX, which forces it to say 'I don't know' or 'please clarify' when SQL is not the answer, instead of hallucinating a query.","Introducing a Question Detector before SQL generation turns out to be worth 3 to 8 TDEX points on top-grade LLMs, suggesting that explicit intent routing is a higher-leverage intervention than further SQL-tuning.","Ambiguous questions should be answered with clarification plus provisional SQL: the paper shows execution accuracy on ambiguous turns rises sharply when models first ask for clarification (e.g., GPT-3.5 Turbo from 34.5% to 49.0%).","Open-source Llama3 models are close enough to GPT-4 Turbo on this task (TDEX 62.8 to 64.0 vs 67.0) that reliable conversational text-to-SQL does not require closed APIs."],"supporting_citations":[{"why":"CoSQL supplies the conversational text-to-SQL task and the human-annotated samples from which part of the MMSQL test set is refined.","marker":"[18]"},{"why":"QDA-SQL generates the additional multi-turn samples with controlled question types that form the rest of the MMSQL data; its dialogue augmentation method is used in dataset construction.","marker":"[15]"},{"why":"MAC-SQL is the prior multi-agent collaborative text-to-SQL framework that the paper explicitly refines by adding a Question Detector agent.","marker":"[6]"},{"why":"The 'Know what I don't know' work motivates handling ambiguous and unknown questions and supplies the real-world question-type distribution that shapes the MMSQL test split.","marker":"[7]"},{"why":"TrustSQL is the reliability benchmark for unanswerable questions whose abstention-based strategy the paper contrasts with its own clarification-and-rewrite strategy.","marker":"[10]"},{"why":"AMBROSIA is the benchmark for parsing ambiguous questions into multiple candidate SQL queries, an idea the paper extends to multi-turn clarification within its framework.","marker":"[9]"},{"why":"BIRD provides the evaluation metrics (EM, EX) and the large-scale benchmark standard that MMSQL extends with multi-turn, multi-type question handling.","marker":"[3]"}],"fun_headline_variants":["New benchmark MMSQL tests chat-to-SQL on ambiguous queries","Four-agent framework boosts multi-turn text-to-SQL by up to 7.9 points","MMSQL: first suite to grade LLMs on ambiguous and unanswerable queries","Multi-agent system lifts Llama3-70B to SQL score 70.7 from 62.8","Benchmark MMSQL reveals LLMs struggle with ambiguous questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test suite's 149 rounds, assembled from manually refined CoSQL samples and LLM-generated QDA-SQL samples, adequately represent the distribution of real-world conversational database queries, and the LLM-judge-based quality scores align with human judgments, so that measured gains on MMSQL transfer to live use.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark MMSQL tests chat-to-SQL on ambiguous queries","Four-agent framework boosts multi-turn text-to-SQL by up to 7.9 points","MMSQL: first suite to grade LLMs on ambiguous and unanswerable queries","Multi-agent system lifts Llama3-70B to SQL score 70.7 from 62.8","Benchmark MMSQL reveals LLMs struggle with ambiguous questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3079,"prompt_tokens":985,"completion_tokens":2094,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1997}},"tokens_in":601,"tokens_out":2094,"duration_ms":12568,"temperature":1.0,"reasoning_tokens":1997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:27:29.957400+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an independently human-annotated multi-turn text-to-SQL test set covering the same four question types without any QDA-SQL-generated turns, then rerun GPT-4 Turbo, Gemini-1.5 Flash, and Llama3-70B with and without the multi-agent framework; if the TDEX gains reported in Table VI shrink to near zero or reverse, the framework's measured benefit is an artifact of the synthetic data generator rather than of conversational understanding.","supporting_citations":[{"cited_title":"CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases,","cited_arxiv_id":null,"evidence_quote":"CoSQL supplies the conversational text-to-SQL task and the human-annotated samples from which part of the MMSQL test set is refined."},{"cited_title":"Chase: A large-scale and pragmatic chinese dataset for cross-database context-dependent text-to-sql,","cited_arxiv_id":null,"evidence_quote":"QDA-SQL generates the additional multi-turn samples with controlled question types that form the rest of the MMSQL data; its dialogue augmentation method is used in dataset construction."},{"cited_title":"Know what i don’t know: Handling ambiguous and unknown questions for text-to-sql,","cited_arxiv_id":null,"evidence_quote":"AMBROSIA is the benchmark for parsing ambiguous questions into multiple candidate SQL queries, an idea the paper extends to multi-turn clarification within its framework."},{"cited_title":"Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,","cited_arxiv_id":null,"evidence_quote":"BIRD provides the evaluation metrics (EM, EX) and the large-scale benchmark standard that MMSQL extends with multi-turn, multi-type question handling."}],"review_version":1}