{"id":"62c682c8-9821-4d37-a1fe-5d86a9379c54","arxiv_id":"2412.17068","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent LLM module that checks, reflects on, and rewrites flawed questions yields small average execution-accuracy gains (1.6% Spider, 2.0% BIRD) on NL2SQL dev sets.","lead":"REWRITER is a plug-in module that rewrites unclear or wrong natural language queries before a system translates them into SQL, aiming to improve database question answering. On the Spider and BIRD benchmarks, it lifts execution accuracy by about 1.6 and 2.0 points on average, though gains vary and some exact-match scores fall.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main results may be transductive: if Reflector experience weights are updated across dev-set queries and then used to evaluate those same queries, the reported gains do not establish a plug-and-play improvement on fresh inputs.","rationale":"The reader's weakest assumption already identifies this exact issue: whether experience weights are updated across dev-set queries during evaluation. I agree it is the load-bearing point. The paper's own text supports the concern: Reflector updates weights based on Checker feedback 'before the next round,' Memory accumulates historical data across workflows, and Table 4's LE ablation shows learning-updated experiences contribute to the best results. If the main tables use this stateful procedure, the reported gains are not a static plug-and-play property; they are adaptation to the dev distribution. The proposed test would settle it by comparing against a frozen-memory condition. I do not see a separate internal inconsistency that would justify rejecting the paper outright: the Checker precision issue is real but the ablations show Checker still helps relative to rewriting everything, and the DTS-SQL EM collapse (77.2 to 47.9) is concerning but the central claim is framed in execution accuracy. The main deficiency is the underspecified, potentially transductive evaluation protocol, which is exactly why the result should remain conditional pending clarification or a frozen-experience rerun.","tokens_in":12793,"tokens_out":6370,"duration_ms":56808,"concrete_test":"Split Spider-dev into two non-overlapping halves. Run REWRITER on half A with the full memory/experience-update pipeline, then freeze the learned experience weights and run on half B; also run half B with per-query memory reset (no updates across queries). Compare EX and EM on half B in both conditions to the sequential-update numbers. If the frozen/reset condition loses most of the 1.6% average gain, the headline result is transductive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that REWRITER, as a black-box-compatible module, 'consistently enhances' downstream NL2SQL models (abstract; Tables 1-2). What must be true for that claim is that the improvement is a property of the rewriting mechanism applied to each query, not of adapting to the evaluation set. The paper does not state whether the Reflector's experience weights are updated across dev-set queries during evaluation. The Reflector section says weights are adjusted 'before the next round of reasoning and rewriting' using Checker feedback, and Memory 'can continuously accumulate historical data from different NL2SQL methods'; Table 4's LE row is described as experience 'summarized and updated in the multi-round rewriting process.' If the dev set is processed sequentially with shared memory, then the main numbers are the result of an online, transductive procedure: Checker feedback on earlier dev queries shapes the experience used for later dev queries, and the same queries are then reported as the benchmark result. This directly threatens the central claim because the largest ablation gains appear in LE (e.g., NatSQL+T5-3B: 73.7 to 75.7 in Table 4), and Table 3 'All' presumably includes that learned memory. With no frozen-experience or per-query-reset condition reported, the average improvements of 1.6%/2.0% cannot be attributed to a fixed plug-and-play rewriter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes REWRITER, a plug-and-play multi-agent module that rewrites flawed natural language queries before they are passed to an existing NL2SQL system. The module consists of a Checker that decides whether a generated SQL query matches the user's NL, a Reflector that analyzes flaws and accumulates rewriting experience, a Rewriter that produces corrected NL, and a shared Memory. The authors report experiments on the Spider and BIRD dev sets with several NL2SQL baselines, claiming average execution-accuracy gains of 1.6% on Spider and 2.0% on BIRD. Ablations show that removing the Checker or the Reflector generally hurts performance, and that learning-updated experience (LE) gives larger gains than hand-crafted or self-initialized experience.","tokens_in":13096,"tokens_out":2195,"duration_ms":21235,"significance":"If the claimed plug-and-play property holds, the paper addresses a genuinely underexplored bottleneck in NL2SQL: the quality of the user's natural language input rather than the SQL generation model itself. The idea of checking generated SQL, reflecting on flaws, and rewriting only flawed inputs is simple, practical, and broadly applicable to black-box NL2SQL systems. The paper also ships a concrete multi-agent architecture with prompts in the appendix, and the self-initialized (IN) ablation provides partial evidence that the rewriting mechanism itself can help without training data. However, the central claim of 'consistent enhancement' depends on the evaluation protocol being a fair test of a fixed plug-and-play module on fresh inputs, and on the reported averages being reliable. The current manuscript leaves those points ambiguous, so the significance cannot yet be fully assessed.","major_comments":[{"comment":"The main results may be transductive. The Reflector section states that experience weights are adjusted 'before the next round of reasoning and rewriting' based on Checker feedback, and the Memory 'can continuously accumulate historical data from different NL2SQL methods.' Table 4's LE row is 'experience summarized and updated in the multi-round rewriting process.' If the dev set is processed sequentially with shared memory, then Checker feedback on earlier dev queries shapes the experience used to rewrite later dev queries, and the same dev queries are used for the final reported numbers. This would mean the headline gains in Tables 1-2 do not measure a fixed plug-and-play rewriter on fresh inputs. The paper must state explicitly whether experience weights are updated during dev-set evaluation, and if so, report a frozen-experience or per-query-reset condition. The IN row in Table 4 is encouraging but does not resolve this, since the 'All' configurations in Table 3 presumably include learned memory.","section":"Reflector and Table 4"},{"comment":"The DTS-SQL + REWRITER row shows EM collapsing from 77.2 to 47.9, a drop of 29.3 points, while EX improves only from 75.2 to 77.5. The paper does not mention this catastrophic exact-match degradation anywhere, and the abstract's claim that REWRITER 'consistently enhances downstream models' is directly contradicted on this row by its own metric definition. Even if the authors regard EX as the primary metric, the EM collapse indicates that rewriting can substantially change the generated SQL's structure, and it undermines the claim in the introduction and ablation discussion that the Checker 'minimizes unnecessary rewriting and potential hallucinations.' The paper needs to acknowledge this case and explain why EX is the appropriate conclusion metric here, or provide a decomposition of what happened on this configuration.","section":"Table 1, DTS-SQL row"},{"comment":"The reported improvements are small (0.3 to 4.3 EX points in most rows) and no error bars, variance, or significance tests are provided for the main NL2SQL results. Figure 5 reports token efficiency averaged over 5 runs, but Tables 1-4 appear to be single runs with LLM-based agents, which are stochastic and multiplicatively amplify variance through the Checker, Reflector, and Rewriter stages. A 1.6% average gain could easily be within run-to-run noise for configurations like C3 + GPT-3.5-turbo (81.9 to 82.4) or DAIL-SQL + GPT-4 (83.1 to 83.6). The authors should report multiple seeds or at least confidence intervals for the headline numbers, especially because the main claim is about consistent improvement.","section":"Tables 1-4 and Figure 5"},{"comment":"The Checker precision (CP) values in Tables 1-2 range from 53.0 to 82.2, meaning that in several configurations a substantial share of the queries flagged as flawed are actually correct (e.g., CP 55.8 for C3 + GPT-3.5-turbo and CP 53.0 for DAIL-SQL + GPT-4 on BIRD). Since only flagged queries are rewritten, low CP means many correct queries are unnecessarily rewritten, which could introduce hallucinations. The paper does not report how many such false-positive rewrites actually hurt, nor does it analyze the interaction between CP and downstream EX. A precision-recall analysis of the Checker, or an error analysis of the rewritten-but-originally-correct cases, is needed to support the claim that the Checker successfully minimizes negative impact.","section":"Checker evaluation, Tables 1-2"}],"minor_comments":[{"comment":"The base EX numbers for the same configurations are inconsistent across tables: NatSQL + T5-Base is 69.7 in Table 1 but 69.4 in Tables 3 and 4; NatSQL + T5-3B is 71.4 in Table 1 but 71.8 in Tables 3 and 4; DTS-SQL is 75.2 in Table 1 but the same value appears in Table 4. The authors should clarify whether these are the same runs or different evaluation conditions, and if they differ, why.","section":"Tables 1, 3, 4"},{"comment":"Several appendix prompt blocks end with '### OUPUT:' instead of '### OUTPUT:', and one block has a stray 'DONT CONVERT IT INTO QUERY.' without a closing quotation mark. These typos should be fixed for reproducibility.","section":"Appendix prompts"},{"comment":"The description of Checker Precision (CP) is circular: CP is defined as 'the precision of the bad samples that NL does not match the SQL,' but it is not stated what the ground-truth label for a 'bad sample' is, nor how the reference standard was obtained for the dev set. The paper should specify the labeling procedure.","section":"Settings"},{"comment":"Figure 4 reports multi-round rewriting effects on Spider-dev, but the number of rounds and the stopping criterion are not defined in the text. The reader cannot tell whether 'multi-round' means two rounds, three rounds, or until convergence, and this affects both the accuracy and token-efficiency conclusions.","section":"Figure 4"},{"comment":"The Discussion states that 'running and testing of the REWRITER framework and above NL2SQL method resulted in a significant consumption of tokens, estimated to be around 45 million tokens.' This sentence is incomplete about what the 45M tokens correspond to (all experiments? one configuration?) and reads more like a cost footnote than a scientific result. It should be moved or clarified.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is timely and the architecture is clearly described, but the evaluation protocol is not yet sufficiently controlled to support the 'plug-and-play consistent enhancement' claim. The most serious concern is the possibility of transductive learning through the Reflector's memory during dev-set evaluation; this is fixable by reporting a frozen-experience condition and clearly separating any online adaptation from the reported benchmark results. The DTS-SQL EM collapse is also a red flag that the authors need to address honestly. Given that these issues concern the load-bearing evidence rather than the method's feasibility, I would recommend major revision rather than rejection. The paper would also benefit from adding variance estimates and a more careful treatment of Checker false positives."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: sensible idea, shaky evaluation. REWRITER—Checker, Reflector, Rewriter with a shared memory—is a plausible way to clean up flawed NL before it hits an NL2SQL model. What's new is applying the check-reflect-rewrite pattern to the input side rather than post-processing SQL, and the ablation shows that even self-initialized experiences (no learning on the dev set) give gains (Table 4: DTS-SQL 75.2→76.4, NatSQL+T5-3B 71.8→73.7). That's honest evidence the mechanism can help.\n\nThe soft spots are real. Table 1 hides an EM collapse for DTS-SQL (77.2→47.9) while the paper only touts EX. That's a red flag. The headline gains are 1-3 points with no error bars or repeated runs; with that effect size, variance matters. More seriously, the paper never states whether Reflector experience weights are updated across dev-set queries during evaluation. The method section says weights adjust 'before the next round' using Checker feedback, and Table 4's LE row is experience updated in multi-round rewriting. If the dev set is processed sequentially with shared memory, the main numbers are transductive. The stress-test is right to push on this. It doesn't kill the paper, but it means the central 'plug-and-play consistently enhances' claim isn't established for fresh inputs.\n\nAlso, Checker precision is 55-82% CP, so a decent share of flagged queries are actually fine and get rewritten anyway; the paper should report the cost of unnecessary rewrites. The discussion admits token costs around 45 million and says more methods couldn't be tested—fair, but a limitation. Related work is adequate, no citation inflation. No code or data released.\n\nWho gets value: NL2SQL pipeline builders who want a drop-in input-side fix. It deserves a serious referee, but only with the protocol clarified and the negative EM result addressed. I'd send it to review, and I'd tell the authors to release code and report per-run variance.\n\nBest","headline":"Sensible NL-rewriting idea for NL2SQL, but the main evaluation may be transductive and an EM collapse is hidden; worth review with protocol fixes.","tokens_in":13627,"tokens_out":2832,"would_cite":false,"duration_ms":23430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"REWRITER proposes a plug-and-play module that automatically rewrites flawed natural-language questions—using schema, column values, and foreign keys—and reports consistent execution-accuracy gains across text-to-SQL systems.","keywords":["NL2SQL","text-to-SQL","query rewriting","self-reflection","multi-agent LLM","execution accuracy","Spider benchmark","BIRD benchmark"],"falsifier":"Run REWRITER on a fresh set of questions with the Reflector's experience weights frozen at their initialization, then run it again with weights updated during evaluation, and compare execution accuracy. If the frozen-weight run does not reproduce the reported 1.6% and 2.0% average gains, the improvement is partly an artifact of updating on the test questions rather than a general property of the rewriting.","tokens_in":12597,"feed_emoji":"🗃️","tokens_out":7989,"duration_ms":62652,"temperature":0.7,"pith_summary":"The paper tries to establish that the user's natural-language question itself is a fixable bottleneck in text-to-SQL, and that rewriting the question—rather than retraining or swapping the SQL generator—can reliably improve results. REWRITER is a plug-and-play module placed in front of any NL2SQL model, which it treats as a black box; a Checker decides whether the generated SQL matches the user's intent, and only when it does not do a Reflector and Rewriter edit the question using database schema, column values, and foreign-key relationships. The claimed payoff is consistent average execution-accuracy gains of 1.6 points on Spider and 2.0 points on BIRD across baselines ranging from rule-based parsers to GPT-4. A sympathetic reading of the experiments is that selective, database-informed rewriting is a working complement to better SQL models, not a replacement for them.","feed_headline":"Rewriting flawed questions lifts text-to-SQL accuracy by 1–2%","feed_subtitle":"A check-reflect-rewrite loop adds database knowledge to user questions, lifting SQL accuracy on Spider and BIRD.","key_machinery":"The load-bearing mechanism is the weighted self-reflection loop in the Reflector. The Reflector starts by initializing a memory of generic flaw types and rewriting actions, each with a weight; before rewriting a question it loads the highest-weighted experiences as rules, and after the downstream SQL is produced the Checker's binary verdict ('SQL matches NL' or not) updates those weights. This makes REWRITER a data-free, in-context learner whose rewriting guidance adapts to the distribution of flaws it encounters. The second mechanism is the Checker's two-stage gate, which uses executable SQL and an LLM's judgment to restrict rewriting to genuinely flawed questions, minimizing the new hallucinations that rewriting correct questions would introduce.","core_discovery":"REWRITER's central claim is that a check-reflect-rewrite loop turns vague, incomplete, or misremembered user questions into clearer ones that existing NL2SQL models translate more accurately. The Checker gates the pipeline: it first discards queries whose SQL fails to execute, then asks an LLM to judge whether the execution result matches the natural-language intent, labeling each sample 'SQL matches NL' or 'SQL does not match NL.' The Reflector extracts the key entities and conditions in the flagged question, compares them against the database, and emits a (keyword, flaw, action) reflection, while the Rewriter applies that reflection to complete missing information, correct wrong entities, resolve ambiguity, or normalize non-standard phrasing. The experiments report consistent execution-accuracy improvements on Spider and BIRD—averaging 1.6% and 2.0% respectively—across bases including C3, DAIL-SQL, DTS-SQL, NatSQL+T5, RESDSQL, and direct GPT-3.5/GPT-4 prompting, and the ablations attribute the gain to the Checker's selective rewriting and the Reflector's accumulated experience.","pith_inferences":["Editorial inference: if the Checker's precision (55–82% on the reported bad samples) is the main ceiling, then a more reliable or calibrated verifier—or a confidence threshold that rewrites only high-certainty positives—could roughly double the reported accuracy gains.","Editorial inference: the experience-weight update looks like an online reward-weighted learner, so a natural stress test is whether the weights learned on Spider transfer to BIRD or to an unseen database domain; the paper does not report cross-benchmark transfer.","Editorial inference: because weights may be updated on the very dev queries being scored, part of the gain could be transductive; a fixed-weight or train/dev-split evaluation would quantify how much of the improvement generalizes."],"forward_implications":["Any NL2SQL model can gain accuracy by being wrapped with REWRITER, without changing its parameters; the paper shows gains on rule-based, small open-source, and large closed-source generators.","Rewriting only the queries the Checker flags is what preserves accuracy; the ablation in which every question is rewritten typically hurts, so selectivity is the safety mechanism.","The Reflector's self-learned experience beats hand-crafted experience on most tested models, implying the loop improves as it sees more flawed questions.","Multi-round rewriting can squeeze out additional gains on some systems, with token cost growing slowly because later rounds revisit only the still-flawed subset.","Because REWRITER outputs natural language rather than SQL, it should compose with future NL2SQL pipelines, including agent-based ones, without internal changes."],"supporting_citations":[{"why":"Supplies the Spider benchmark, the primary evaluation set for the reported execution-accuracy gains.","marker":"(Yu et al. 2018)"},{"why":"Supplies the BIRD benchmark, the second evaluation set where REWRITER reports a 2.0% average gain.","marker":"(Li et al. 2023b)"},{"why":"Provides the DAIL-SQL baseline used on both Spider and BIRD, one of the main systems REWRITER is plugged into.","marker":"(Gao et al. 2023)"},{"why":"Provides the GPT-3.5/GPT-4 baselines and the backbone models used for REWRITER's agents.","marker":"(OpenAI 2023)"},{"why":"Provides the C3 baseline, a zero-shot ChatGPT text-to-SQL method that REWRITER enhances.","marker":"(Dong et al. 2023)"},{"why":"Provides the DTS-SQL baseline, an open-source small-LLM text-to-SQL method in the Spider experiments.","marker":"(Pourreza and Rafiei 2024)"},{"why":"Provides the NatSQL+T5 baselines, semantic-boundary-based parsers in the Spider experiments.","marker":"(Rai et al. 2023)"},{"why":"Provides the RESDSQL baseline, a schema-linking and skeleton-parsing text-to-SQL system in the Spider experiments.","marker":"(Li et al. 2023a)"}],"fun_headline_variants":["Rewriting flawed queries lifts SQL accuracy by 1–2%","Plug-and-play rewriter adds DB knowledge, boosts NL2SQL","Check-rewrite loop fixes vague questions, gains ~2%","LLM rewriter reduces NL2SQL errors by up to 2%","Clearing ambiguous text lifts text-to-SQL accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire improvement rests on the Checker's binary verdict being a reliable reward signal, yet in the reported runs that verdict is correct on only 55–82% of the samples it flags, so many rewritten questions were probably never flawed.","fun_headline_variants_meta":{"raw":{"variants":["Rewriting flawed queries lifts SQL accuracy by 1–2%","Plug-and-play rewriter adds DB knowledge, boosts NL2SQL","Check-rewrite loop fixes vague questions, gains ~2%","LLM rewriter reduces NL2SQL errors by up to 2%","Clearing ambiguous text lifts text-to-SQL accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1367,"prompt_tokens":1025,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":252}},"tokens_in":641,"tokens_out":342,"duration_ms":3792,"temperature":1.0,"reasoning_tokens":252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:49:09.524296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run REWRITER on a fresh set of questions with the Reflector's experience weights frozen at their initialization, then run it again with weights updated during evaluation, and compare execution accuracy. If the frozen-weight run does not reproduce the reported 1.6% and 2.0% average gains, the improvement is partly an artifact of updating on the test questions rather than a general property of the rewriting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NatSQL+T5 baselines, semantic-boundary-based parsers in the Spider experiments."}],"review_version":1}