{"id":"2b054c73-26cd-4938-97bf-8920e9863d9c","arxiv_id":"2608.11889","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A non-fine-tuning text-to-SQL pipeline with deep schema disambiguation notes, database-agnostic correction rules mined from training data, and dependency-tree-guided SQL generation reports state-of-the-art execution accuracy on BIRD-Dev and Spider-Test.","lead":"DexterSQL is a prompting-based system that improves text-to-SQL by analyzing column data distributions, mining reusable correction rules from training databases, and generating SQL through a dependency-tree intermediate representation. It reports execution accuracy of 67.6% on BIRD-Dev with an open-weight LLM and 71.6-72.2% with closed-weight models, above prior non-fine-tuning systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The closed-weight 'at least 0.9%' claim compares DexterSQL against published APEX-SQL numbers, while only DeepEye-SQL was rerun locally; the paper's own footnote shows local reruns can differ, so the GPT-4o margin is not established.","rationale":"The reader's weakest_assumption focuses on transfer of mined correction rules. That is a real risk, and the paper's only direct evidence is BIRD-Train to BIRD-Dev and Spider-Train to Spider-Test; however, the open-weight headline (Table 5) and the GPT-5.2 comparison (Table 7) are controlled local comparisons, so the rule-transfer concern primarily affects a component's generality rather than the headline margin. I found a more immediately load-bearing gap in the GPT-4o closed-weight result. Table 6 is explicitly mixed: most non-DexterSQL rows are published numbers, and the footnote to Table 7 states that running the released APEX-SQL code in the authors' setting lowers its accuracy by one point on GPT-5.2. That is an internal admission that published and local numbers are not interchangeable in this paper's harness. Since the abstract's 'at least 0.9%' margin is exactly the difference between DexterSQL and one published APEX-SQL number, the claim's truth depends on an uncontrolled comparison. The proposed test, rerunning the closest baselines and DexterSQL under the same GPT-4o harness, would settle it. I credit the paper for running all baselines locally for the GPT-OSS-120B experiments and for local APEX-SQL and DeepEye-SQL runs on GPT-5.2; those parts are better controlled. No ad hominem intended; the issue is experimental comparability, not authorial conduct.","tokens_in":24036,"tokens_out":9989,"duration_ms":101927,"concrete_test":"Re-run APEX-SQL, RSL-SQL, OpenSearch-SQL, DeepEye-SQL, and DexterSQL on BIRD-Dev with the same GPT-4o API snapshot, same benchmark evidence, same few-shot retrieval, same execution environment, and the same selection threshold, then report per-system execution accuracy. If DexterSQL's margin over the strongest baseline is less than 0.9 percentage points, or if APEX-SQL's local EX reaches or exceeds 71.6%, the abstract's closed-weight 'at least 0.9%' claim is not supported by the reported evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's closed-weight claim ('improvement of at least 0.9%', 71.6% vs APEX-SQL's 70.7% on GPT-4o) is not a controlled comparison. In §4.2.2 and Table 6, only DeepEye-SQL and DexterSQL were executed locally on GPT-4o; RSL-SQL, OpenSearch-SQL, and APEX-SQL rows are taken from their published papers. The footnote to Table 7 reveals the fragility of this comparison: although the APEX-SQL paper reports 70.7% with GPT-4o, running the released APEX-SQL code with GPT-5.2 in the authors' setting yields 69.7%, one full point below the published number. If a similar downward shift occurs for APEX-SQL on GPT-4o in the authors' harness, the 0.9-point margin can disappear or reverse; if the published number came from a different API snapshot, prompt set, evidence handling, or selection strategy, the margin is not a like-for-like comparison. No code or prompt templates are released, so the reader cannot diagnose the gap. The GPT-5.2 comparison in Table 7 is better controlled because APEX-SQL and DeepEye-SQL were run locally, but the GPT-4o 'at least 0.9%' subclaim rests on mixed-source evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DexterSQL, a non-fine-tuning (prompting-only) text-to-SQL system combining three components: a deep schema explorator that mines data-level statistics and LLM-generated notes for ambiguous column pairs, a database-agnostic rule creator that mines recurring SQL generation failures from training databases into correction rules, and a multi-path SQL generator that uses a dependency-tree-based intermediate representation alongside few-shot and divide-and-conquer generation. The system is evaluated on BIRD-Dev and Spider-Test with an open-weight model (GPT-OSS-120B) and closed-weight models (GPT-4o, GPT-5.2), reporting execution accuracy of 67.6% on BIRD-Dev with GPT-OSS-120B (claimed at least 2.7% over the strongest baseline) and 71.6%/72.2% with GPT-4o/GPT-5.2 (claimed at least 0.9% over prior work). Ablations attribute gains to all three components, with full-pipeline accuracy dropping to 63.3% when they are all removed.","tokens_in":24326,"tokens_out":7005,"duration_ms":70005,"significance":"If the headline numbers are taken at face value, DexterSQL would be a credible state-of-the-art non-fine-tuning system, particularly for open-weight deployment, and the open-weight comparison in Table 5 appears the strongest part of the evidence because the authors state that baselines were run locally. The rule-mining design is not circular on label-leakage grounds: rules are mined from a disjoint training database and applied to target queries, which is a meaningful methodological distinction. The dependency-tree-based intermediate representation is a reasonable and clearly motivated addition to multi-path generation. However, the closed-weight claim rests on mixed-source comparisons, and the headline 67.6% is obtained by selecting a confidence threshold on the test set itself, which inflates the reported accuracy and weakens the ablation; these issues must be addressed before the central claims can be accepted.","major_comments":[{"comment":"The paper states that \"We executed all baseline approaches, with available implementations, ourselves locally using GPT-OSS-120B,\" but the phrase \"with available implementations\" leaves it unclear whether all ten rows in Table 5 are locally produced or whether some rows are taken from published papers. Since the central open-weight claim of \"at least 2.7%\" improvement over DeepEye-SQL depends on a like-for-like local comparison, please specify for each baseline whether it was run locally or cited from its paper, and report the version/configuration used for any local rerun.","section":"§4.2.1, Table 5"},{"comment":"The GPT-4o comparison is not fully controlled. In Table 6, only DeepEye-SQL and DexterSQL are reported from local runs on GPT-4o; the RSL-SQL, OpenSearch-SQL, and APEX-SQL rows are taken from their published papers. Footnote 7 explicitly shows that running the released APEX-SQL code locally with GPT-5.2 yields 69.7%, a full point below the published GPT-4o number of 70.7%, which demonstrates that published numbers are not necessarily reproduced in the authors' harness. Consequently, the abstract's claim of \"at least 0.9%\" improvement on closed-weight models (71.6% vs 70.7%) is not established. Please rerun the close baselines locally on GPT-4o, or reword the claim to explicitly state that the GPT-4o margin is against a published number rather than a controlled local comparison.","section":"§4.2.2, Table 6, footnote 7"},{"comment":"The confidence threshold is selected by maximizing execution accuracy on the BIRD-Dev test set: Figure 4 shows EX rising from 66.7% at threshold 0 to 67.6% at threshold 0.6 and then falling to 67.1% at threshold 1, and the text concludes that \"the highest EX is achieved at a threshold of 0.6.\" This is test-set hyperparameter tuning: the reported 67.6% is the best accuracy over the threshold grid, and the 2.7% improvement over DeepEye-SQL in Table 1 is therefore partly attributable to this selection. Because the baselines did not have the same opportunity to tune this free parameter on the test set, the comparison is unfair. Please select the threshold on a validation split, use a fixed default threshold justified a priori, or report the full sensitivity curve with a clear statement that the peak is test-set-selected.","section":"§4.2.5, Figure 4"},{"comment":"The ablation results report single-point accuracy values with no variance, number of runs, temperature, or seed information. The difference between the full pipeline (67.6%) and the pipeline without dependency-tree-based generation (67.2%) is only 0.4 percentage points, which is within the range of run-to-run variability expected from stochastic LLM sampling. Additionally, because the ablation is evaluated at the test-set-tuned threshold of 0.6, the component contributions are confounded with threshold selection. Please report multiple runs with standard deviations (or a paired bootstrap), specify the decoding temperature and seeds, and ideally show the ablation across a range of thresholds.","section":"Table 11, Exp. 6"},{"comment":"The paper's \"database-agnostic\" rule transfer claim is a central novelty, yet its support is limited. Rules are mined from training databases and applied to target databases, but the only evidence that they transfer is the overall BIRD-Dev and Spider-Test accuracy, with no analysis of the mined rules themselves: the paper does not report how many rules were synthesized, how many error groups were formed, how often each rule fired, whether rule application ever decreased accuracy, or whether the rules are stable across training samples. The reader cannot tell whether the 2.1-point gain from Rule Creator (65.5 to 67.6 in Table 11) is a robust transferable effect or an artifact of a few high-frequency rules fitting the two test benchmarks. Please provide a rule audit (counts, examples, firing statistics, and cross-benchmark transfer analysis) or otherwise substantiate the transfer claim.","section":"§3.1.4, Exp. 6"}],"minor_comments":[{"comment":"The sentence \"DexterSQL's shows a high improvement\" is ungrammatical; it should read \"DexterSQL shows a high improvement\".","section":"Abstract"},{"comment":"The text begins with \"TPrevious step produces useful but low-level evidence\"; \"TPrevious\" should be \"The previous\".","section":"§3.1.3, Step 4"},{"comment":"The comparison in Table 8 mixes different underlying LLMs (Gemini-1.5-Pro, DeepSeek-V3.1, GPT-4o); the caption and findings should explicitly state that this is a cross-model comparison and not a controlled evaluation, so that readers do not infer a like-for-like advantage over AutoLink or CHESS.","section":"Table 8"},{"comment":"The paper does not state whether code, prompt templates, or the mined rule set will be released, and it does not specify the decoding temperature or number of runs for the LLM calls; given the prompting-heavy nature of the system, this information is essential for reproducibility.","section":"§4.1, Reproducibility"},{"comment":"The figure caption says \"confidence shortcut thresholds\"; the word \"shortcut\" appears to be unintended and should be removed or defined.","section":"Figure 4"},{"comment":"The OpenSearch-SQL row has dashes for the Simple/Moderate/Challenging breakdown; please state whether these breakdowns are unavailable from the original paper or were omitted for brevity.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's central open-weight result is likely sound if Table 5 is indeed fully local, but the abstract overstates the closed-weight evidence: footnote 7 contains a number that directly undercuts the GPT-4o 'at least 0.9%' claim, and the authors should either provide local GPT-4o reruns or rephrase the claim. The most serious methodological issue is the test-set threshold selection in §4.2.5, which inflates the headline 67.6% and complicates interpretation of all ablated numbers. I would encourage the authors to treat the confidence threshold as a fixed default or validate it on a hold-out set, and to provide a rule audit for the Rule Creator, since rule transferability is their main claimed contribution beyond existing prompting pipelines. The paper is publishable after these revisions, but not in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine engineering contribution to non-fine-tuning text-to-SQL, and the open-weight result is the part to trust. The three components — deep schema exploration, database-agnostic rule mining, and dependency-tree-based generation — are all clearly motivated and sensibly integrated. The dependency-tree intermediate representation is the most novel piece; it deterministically preserves question elements better than free-form decomposition. The rule-mining idea is also clever, and I appreciate that the authors explicitly separate database-specific from database-agnostic failures.\n\nThe strongest evidence is Table 5: they say they ran all baselines locally on GPT-OSS-120B, and DexterSQL beats DeepEye-SQL by 2.7 points on BIRD-Dev and 2.5 on Spider. That's a real, controlled result. The ablation (Table 11) is consistent with the claims, though the component contributions are uneven — removing rule-based correction or schema exploration costs ~2 points, but removing the dependency-tree path only costs 0.4, which is within noise given no variance is reported.\n\nThe soft spots are mostly about the closed-weight narrative. The abstract's 'at least 0.9%' on GPT-4o compares DexterSQL (run locally) against APEX-SQL's published 70.7. Only DeepEye-SQL was rerun locally on GPT-4o. The footnote shows how fragile published numbers are: the released APEX-SQL code gives 69.7 on GPT-5.2, a full point below its reported GPT-4o number — and those aren't even the same model. So the 0.9-point margin is not established in a common harness. The GPT-5.2 comparison (Table 7) is properly controlled and shows a 2.5-point gain, so the central thesis survives, but the abstract overstates the closed-weight case.\n\nNo code or prompts are released. For a systems paper whose entire contribution is the pipeline, that's a real reproducibility problem.\n\nOne more moderate concern: the confidence threshold in Exp 5 is tuned on BIRD-Dev itself (best at 0.6). That's selection on the evaluation set. Minor, but it should be acknowledged.\n\nWho should read it: people building practical text-to-SQL systems, especially those who care about local/private deployment with open-weight models. It deserves a serious referee. The experimental section needs a common harness for at least the top 2-3 baselines, error bars, and an artifact release; with those, it would be a solid systems paper.","headline":"The open-weight result looks real and the dependency-tree idea is worth taking seriously, but the closed-weight headline number is built on mixed-source baselines and the paper ships no artifacts.","tokens_in":24839,"tokens_out":3700,"would_cite":true,"duration_ms":34214,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DexterSQL, a prompting-only text-to-SQL pipeline that never changes the LLM's weights, reaches 67.6% execution accuracy on BIRD-Dev with an open-weight model and 71.6–72.2% with closed-weight models, beating prior non-fine-tuning systems.","keywords":["text-to-SQL","non-fine-tuning","prompting","schema linking","deep schema exploration","dependency tree","SQL correction rules","execution accuracy"],"falsifier":"Run the full pipeline on a held-out set of databases not drawn from BIRD or Spider, with and without the rule-based correction stage, and count how often applying a mined rule changes a wrong SQL into a right one. If rule-triggered rewrites help on BIRD-Dev but fail on unseen databases, or fire mostly on queries that were already correct, then the claim that database-agnostic correction rules generalize would be refuted.","tokens_in":23843,"feed_emoji":"🗄️","tokens_out":9368,"duration_ms":83171,"temperature":0.7,"pith_summary":"DexterSQL is a text-to-SQL system that does not fine-tune or change the underlying LLM; it improves generation entirely through prompting and offline-computed database context. The paper reports execution accuracy of 67.6% on the BIRD-Dev benchmark with an open-weight model, 84.4% on Spider-Test, and 71.6–72.2% with closed-weight models, exceeding prior non-fine-tuning systems in each setting. The practical significance is that open-weight models can be deployed locally on sensitive data, so accuracy gains without weight updates strengthen private, cheaper deployments. The paper attributes the gains to three components: data-distribution notes that disambiguate confusable columns, mined correction rules for recurring SQL mistakes, and a dependency-tree-guided multi-path generator. If these results hold beyond the two benchmarks, the main takeaway is that database-content understanding and reusable failure patterns, not model retraining, drive much of text-to-SQL performance.","feed_headline":"Prompt-only SQL generation hits 67.6% on BIRD-Dev","feed_subtitle":"Deep column profiling, mined correction rules, and dependency-tree paths let prompting-only systems beat previous methods.","key_machinery":"The machinery is a pipeline of three offline-built artifacts reused at inference. First, the deep schema explorator: for column pairs that look confusable by name or profile, it computes value-set overlap, join coverage, fan-out, and agreement, then has an LLM summarize the evidence into short disambiguation notes stating which column to use when. Second, the rule creator: it samples training questions, keeps generated SQL whose execution differs from gold, filters out explanations judged to be tied to a particular database, clusters the remaining explanations, and synthesizes correction rules of the form 'gist, bad-pattern, correct-pattern, fix'. Third, multi-path SQL generation: one path parses the question into a dependency tree and maps the tree to an SQL skeleton so question mentions and literals are preserved, while the other two paths are few-shot in-context learning and divide-and-conquer. A confidence-aware selector then clusters corrected candidates by execution result and invokes LLM adjudication only when agreement is low.","core_discovery":"The paper's central claim is that DexterSQL, which leaves LLM weights untouched, achieves execution accuracy of 67.6% on BIRD-Dev with the open-weight GPT-OSS-120B model (84.4% on Spider-Test), and 71.6% with GPT-4o and 72.2% with GPT-5.2, outperforming prior prompt-based systems by margins of at least 2.7 and 0.9 percentage points, respectively. The central discovery is that three offline-computed artifacts—column-distribution disambiguation notes, mined correction rules, and a dependency-tree-derived SQL skeleton—carry most of the improvement: removing the schema-exploration or rule-based correction drops accuracy to about 65.4–65.5%, and removing all three drops it to 63.3%. This is a claim about a prompting-only pipeline, not about a fine-tuned model.","pith_inferences":["Beyond the paper, if the mined rules are truly database-agnostic, one could build shared, model-specific rule libraries from many training databases and test whether they transfer to enterprise or scientific schemas outside BIRD and Spider, and whether they decay as base LLMs improve.","Beyond the paper, the coverage, fan-out, and agreement statistics computed for ambiguous column pairs double as schema-quality diagnostics, potentially flagging denormalized attributes or misleading column names in a data catalog.","Beyond the paper, the dependency-tree-to-SQL-skeleton idea generalizes to other structured-output tasks, such as generating SPARQL, API calls, or dataframe code, whenever a question carries several conditions that all have to survive decomposition."],"forward_implications":["A prompting-only system can outperform previous non-fine-tuning systems on both BIRD-Dev and Spider-Test using open-weight and closed-weight LLMs, so accuracy gains do not require retraining or weight updates.","The three generation paths are complementary: the dependency-tree path is the strongest alone at 66.3% EX, and combining all three raises EX to 67.6% and UB-EX to 74.8%.","Rule-based correction is responsible for a substantial share of the gain: disabling it lowers accuracy from 67.6% to 65.5%, while disabling deep schema exploration lowers it to 65.4%.","Deep schema exploration improves schema linking, reaching 97.09% recall and 72.26% precision on BIRD-Dev, so data-distribution analysis recovers ambiguous columns that schema linking alone would miss."],"supporting_citations":[{"why":"BIRD benchmark; supplies the BIRD-Dev test set and BIRD-Train training set on which all accuracy numbers are measured.","marker":"[18]"},{"why":"Spider benchmark; supplies the Spider-Test set on which DexterSQL reports 84.4% execution accuracy.","marker":"[38]"},{"why":"Closest baseline on BIRD-Dev that DexterSQL outperforms; also the prior source of the divide-and-conquer generation path.","marker":"[13]"},{"why":"Strongest GPT-4o baseline that DexterSQL claims to exceed on BIRD-Dev.","marker":"[1]"},{"why":"Supplies the few-shot in-context-learning generation path and serves as a baseline.","marker":"[8]"},{"why":"Motivates multi-path reasoning and candidate selection used in DexterSQL's generator.","marker":"[22]"},{"why":"Provides the vector-index library used to build the profile and value indices during preprocessing.","marker":"[6]"},{"why":"Provides the SQL parser used for deterministic syntax checks in the correction phase.","marker":"[20]"}],"fun_headline_variants":["Schema deep-dive lifts prompt-only SQL to 67.6%","Rule mining and schema profiling boost Text-to-SQL without fine-tuning","DexterSQL: Offline artifacts push prompt-based SQL past prior best","Prompt-only SQL: Deep schema and rules beat fine-tuning-free rivals","DexterSQL hits 67.6% on BIRD-Dev with no parameter updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the correction rules, learned from failures on training databases and filtered to remove explanations tied to any particular database, still apply to unseen target databases, so the measured gain from rule-based correction is not just tuned to the two benchmarks in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Schema deep-dive lifts prompt-only SQL to 67.6%","Rule mining and schema profiling boost Text-to-SQL without fine-tuning","DexterSQL: Offline artifacts push prompt-based SQL past prior best","Prompt-only SQL: Deep schema and rules beat fine-tuning-free rivals","DexterSQL hits 67.6% on BIRD-Dev with no parameter updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1918,"prompt_tokens":1112,"completion_tokens":806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":706}},"tokens_in":728,"tokens_out":806,"duration_ms":7356,"temperature":1.0,"reasoning_tokens":706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:29.429737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a held-out set of databases not drawn from BIRD or Spider, with and without the rule-based correction stage, and count how often applying a mined rule changes a wrong SQL into a right one. If rule-triggered rewrites help on BIRD-Dev but fail on unseen databases, or fire mostly on queries that were already correct, then the claim that database-agnostic correction rules generalize would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BIRD benchmark; supplies the BIRD-Dev test set and BIRD-Train training set on which all accuracy numbers are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Closest baseline on BIRD-Dev that DexterSQL outperforms; also the prior source of the divide-and-conquer generation path."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SQL parser used for deterministic syntax checks in the correction phase."}],"review_version":1}