{"id":"1ffc5086-f6b3-4b75-bbfe-72bdc299aa19","arxiv_id":"2507.22478","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A post-training pipeline combining SFT, GRPO reinforcement learning, and corrective self-consistency lifts 0.5B-1.5B models to 56.9%-67.1% execution accuracy on BIRD dev.","lead":"SLM-SQL shows that small language models with 0.5B to 1.5B parameters can reach 56.9% to 67.1% execution accuracy on the BIRD text-to-SQL benchmark after a combination of supervised fine-tuning, reinforcement learning, and corrective self-consistency. The result matters because it suggests practical, low-cost SQL generation may not require large proprietary models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6's 1.5B 'SFT+RL' entry (50.59) contradicts Table 4's w/o-CSC row (61.95); with code/data withheld, the headline 67.08 and the claimed 31.4-point average are not yet independently established.","rationale":"The reader's weakest assumption—that merge revision requires at least one of the two highest-vote draft SQLs to be correct—is a real modeling limitation: Appendix F.2 explicitly tells the model 'one of the draft SQLs is correct' and forbids constructing a new SQL, so the merge module is a candidate-selection mechanism, not a true corrector. If both drafts are wrong, it can only choose the less wrong query. This explains the ceiling on the 5-point CSC gain but does not by itself falsify the reported BIRD dev numbers; it is a limitation of the method's mechanism rather than a direct threat to the central empirical claim. The most load-bearing threat to the central claim is instead the reliability of the paper's own numerical reporting. Table 6's 1.5B SFT+RL entry (50.59) is identical to the 0.5B entry and lower than the SFT-only row, while Table 4's w/o-CSC row (61.95) is the same condition. Appendix B says three runs were averaged, but no variance is reported, so there is no way to decide which value is real. This is an internal inconsistency, not a disagreement with consensus, and it directly affects whether the 67.08 headline is trustworthy. The Spider generalization overclaim is also real—SynSQL-2.5M includes Spider training data, so the Spider results cannot support cross-dataset generalization—but it is secondary to the BIRD dev claim. A single concrete check, re-running the 1.5B SFT+RL condition and the full pipeline on BIRD dev from released artifacts, would settle whether Table 6's value is a typo or whether the ablation support for the headline is invalid. Until that check is performed, CONDITIONAL remains the appropriate verdict.","tokens_in":15151,"tokens_out":15162,"duration_ms":166026,"concrete_test":"Release the artifacts and independently rerun the 1.5B SFT+RL condition on BIRD dev (no CSC) for three seeds, following Appendix B's protocol with 64 generation samples and the same prompt and split. If the mean is approximately 61.95, Table 6's 50.59 is a typo and the central claim retains its internal support; if the mean is approximately 50.59, Table 4's ablation and the 5-point CSC gain are wrong, and the headline's supporting analysis collapses. Also rerun the full SLM-SQL-1.5B condition to check whether 67.08 reproduces within run-to-run noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the paper's own evidence for the headline is internally inconsistent. Table 6 reports SLM-SQL-1.5B 'SFT + RL' on the BIRD dev set as 50.59 EX (All), identical to the 0.5B row and 3.58 points below the SFT-only row (54.17). Table 4's 'w/o Corrective Self-Consistency' row, which is exactly the SFT+RL condition, reports 61.95 EX for the same model and split—an 11.36-point discrepancy. Appendix B states the authors 'report the average performance over three independent runs for each experimental setting,' but no variance or standard deviation is given, so both values cannot be true averages of the same condition. Because the central claim is a set of empirical numbers (56.87/67.08, average +31.4), and because the code, datasets, and models are withheld until after review, a demonstrably erroneous table elsewhere in the same experimental pipeline leaves the headline numbers unsupported: they may be correct, but the paper as written does not yet establish them.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SLM-SQL, a post-training recipe for small language models (0.5B-1.5B) on Text-to-SQL. It constructs two derived datasets from SynSQL-2.5M (SynSQL-Think-916K for SQL generation and SynSQL-Merge-Think-310K for merge revision), applies SFT and GRPO-based RL to both the generation and merge-revision models, and performs inference with the corrective self-consistency (CSC) procedure from CSC-SQL. On the BIRD development set it reports 56.87% EX for a 0.5B model and 67.08% EX for a 1.5B model, with an average gain of 31.4 points over five base models, and it also reports large gains when these models are evaluated on Spider. The central empirical claims are not yet fully supported because of internal numerical inconsistencies, training-data overlap in the Spider generalization experiment, and missing uncertainty quantification.","tokens_in":15458,"tokens_out":7133,"duration_ms":73834,"significance":"If the headline results are correct, they would be notable: sub-2B models outperforming several 7B-32B open-source systems and even GPT-4-based methods on the BIRD development set. The paper's main contribution is an assembly of existing components—CSC-SQL, SQL-R1, and Think2SQL—plus two newly derived datasets and a systematic evaluation across five small models. The planned release of datasets, models, and code is valuable. However, the manuscript currently withholds code and models until after review, reports no error bars despite claiming three runs per setting, and contains an internal contradiction between Table 6 and Table 4. The empirical claims therefore cannot be independently verified as written.","major_comments":[{"comment":"Table 6 and Table 4 report mutually incompatible numbers for the same condition. In Table 4, the \"w/o Corrective Self-Consistency\" row for SLM-SQL-1.5B is 61.95 EX on the BIRD dev set; this condition is exactly \"SFT + RL\". In Table 6, the \"SFT + RL\" row for the same model reports All = 50.59. Table 6 also lists All = 50.59 for SLM-SQL-0.5B SFT + RL, identical to the 1.5B value, while Table 4 gives 50.96 for the 0.5B w/o-CSC condition. Appendix B states that each setting is averaged over three runs, but no variance or per-run values are reported, so both cannot be valid averages. Because the headline BIRD numbers (56.87 and 67.08) come from the same experimental pipeline, this inconsistency must be resolved before the empirical claims can be accepted.","section":"Appendix D.2 / Section 3.3"},{"comment":"The claimed cross-dataset generalization to Spider is not supported because the SFT training data overlap with the Spider evaluation data. Section 2 states that SynSQL-2.5M \"also includes the Spider and BIRD training sets\", and SynSQL-Think-916K is derived from SynSQL-2.5M. The SQL generation model is SFT-trained on SynSQL-Think-916K, so it has seen Spider training questions during SFT. Section 3.2 then describes \"directly tested models trained on the BIRD dataset\" on Spider and interprets the gains as generalization. The paper should either exclude all Spider-derived samples from SFT, or explicitly report an evaluation on a truly unseen split, before drawing the generalization conclusion.","section":"Section 2 / Section 3.2"},{"comment":"The merge revision step relies on the prompt-level assumption that at least one of the two highest-vote draft SQLs is correct, and the prompt explicitly forbids constructing a new SQL. If both drafts are wrong, the merge model can only select the less incorrect query. The paper does not measure how often this precondition holds (e.g., the coverage of the top-two vote groups as a function of k), although Appendix D.1 reports pass@64 = 0.8264 for SLM-SQL-1.5B, which implies that for roughly 17% of questions no sampled SQL is correct. The roughly 5-point CSC gain in Table 4 therefore depends on an unverified assumption; a coverage analysis is needed to interpret the ablation.","section":"Appendix F.2 / Section 3.3"},{"comment":"The paper states \"We report the average performance over three independent runs for each experimental setting,\" but no standard deviation, confidence interval, or per-run numbers appear in any table or figure. Since several comparisons are decided by small margins (e.g., 1.5B SFT+RL vs w/o RL in Table 4 differ by 0.96 points), the absence of variance information leaves the reported differences unquantified. At minimum, the authors should add standard deviations to the main tables or provide per-run results in an appendix.","section":"Appendix B"}],"minor_comments":[{"comment":"The x-axis labels appear as \"22 23 24 25 26\" rather than 2^2, 2^3, ... or 4, 8, ...; please fix the typography for readability.","section":"Figure 3"},{"comment":"The table uses \"SynsQL-Think-916k\" and \"SynsQL-Merge-Think-310k\" while the text uses \"SynSQL\"; please unify the capitalization.","section":"Table 1"},{"comment":"Several reference-list entries are malformed; for example, the DeepSeek-Coder entry begins with a concatenated author string (\"Dejian Yang Zhenda Xie Kai Dong Wentao Zhang...\") rather than a proper citation format.","section":"References"},{"comment":"The abstract says \"We will release our dataset, model, and code to github\", while Section 6 says release will happen \"after the review process\"; please specify the intended timeline and repository status.","section":"Abstract / Section 6"},{"comment":"SLM-SQL rows list Test EX as \"-\"; if BIRD test results are not reported, the paper should state that the comparison is development-set-only, since many baseline rows include test numbers.","section":"Table 2"},{"comment":"The caption and appendix should clarify which merge revision model is used for each base model; the statement that one 0.5B merge model is used for several architectures is easy to miss and is important for reproducibility.","section":"Table 3 / Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The first author has a closely related prior paper (CSC-SQL), and the current method is largely an incremental combination of CSC-SQL with SFT-before-RL as in SQL-R1 and Think2SQL. The identical All values for two different models in Table 6 look like a copy-paste error, but combined with the absence of error bars and the withheld code/data, the inconsistency is severe enough that I would want corrected tables and ideally code or per-run results before acceptance. The Spider generalization claim is also confounded by training-data overlap and needs either re-analysis or reframing. I see no evidence of deliberate misconduct, but the manuscript in its current form does not yet establish the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper's headline numbers — 56.87 and 67.08 BIRD dev EX for 0.5B and 1.5B models — are plausible and, if they survive scrutiny, genuinely useful for edge Text-to-SQL. But the paper as submitted does not yet establish them. The evidence is internally inconsistent: Table 6 lists SLM-SQL-1.5B \"SFT + RL\" at 50.59 EX (All), the same value as the 0.5B row, while Table 4's \"w/o Corrective Self-Consistency\" row — the same SFT+RL condition — reports 61.95. The paper says every setting is averaged over three runs, but no variance is given, so both numbers cannot be correct averages. That is not a minor typo; it sits in the same evaluation pipeline as the headline.\n\nWhat is genuinely new: the two derived datasets (SynSQL-Think-916K, SynSQL-Merge-Think-310K), and the demonstration that the SFT-then-GRPO recipe, which has worked at 7B+, transfers to 0.5-1.5B models when combined with corrective self-consistency. If the BIRD dev numbers are real, that is a legitimately useful result for edge deployment. The ablation logic is sound — every component removal hurts, and the SFT effect is large.\n\nThe soft spots, in order of severity. First, the Table 6/Table 4 contradiction above. Second, the Spider generalization claim: SynSQL-2.5M includes Spider training data, so Table 3 is not a clean cross-dataset test; the paper should say this plainly. Third, the merge-revision assumption: Appendix F.2 tells the model one draft SQL is correct, so when both top-vote drafts are wrong the model can only choose the less-wrong query. The authors don't hide this, and it's a known property of self-consistency, but it means the 5-point CSC gain depends on the sampling budget almost always producing a correct candidate. Fourth, no BIRD test-set numbers and no error bars, despite claiming three runs. Code, data, and models are withheld until after review, so nothing is independently reproducible yet.\n\nBottom line: this deserves a serious referee, not a desk reject. The core claim is important enough and the methodology is coherent enough to warrant the time. The referee should demand a corrected Table 6, variance estimates, an explicit statement about Spider overlap, and release of artifacts before acceptance. I would not cite it until those are out, but I'd be glad to see the revised version.","headline":"Plausible and useful if true, but the paper's own tables contradict each other on the main result, so the headline BIRD numbers are not yet established.","tokens_in":15996,"tokens_out":2499,"would_cite":false,"duration_ms":26668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small language models post-trained on synthetic reasoning data can match or surpass 7B–32B models on the BIRD text-to-SQL benchmark.","keywords":["Text-to-SQL","small language models","supervised fine-tuning","reinforcement learning","GRPO","corrective self-consistency","BIRD benchmark","Spider dataset"],"falsifier":"On BIRD development, for each question check whether a correct SQL appears among the 64 sampled candidates and whether it appears in the top two execution-vote groups; if the top-two groups are correct on only a small fraction of the questions the final answer gets right, the merge stage cannot be making the claimed repairs. Alternatively, run the merge revision model without the instruction that one draft SQL is correct and see whether the roughly five-point execution-accuracy gain disappears.","tokens_in":14913,"feed_emoji":"🗄️","tokens_out":7516,"duration_ms":73716,"temperature":0.7,"pith_summary":"This paper tries to establish that small language models with 0.5B to 1.5B parameters can be turned into competitive Text-to-SQL systems through post-training, not by scaling up. The recipe it proposes is supervised fine-tuning on 916K synthetic chain-of-thought samples, reinforcement learning with execution-based rewards, and a corrective self-consistency stage at inference. On the BIRD development set, the five tested models gain on average 31.4 points in execution accuracy, with the 0.5B model reaching 56.87% and the 1.5B model 67.08%, both numbers above several 7B–32B baselines. The same BIRD-trained models transfer to Spider with test-set gains of 24.98 points on average, which the paper offers as evidence of generalizability. A sympathetic reader would care because the claim, if correct, means edge-sized models can do database-querying work that has been reserved for models tens of times larger.","feed_headline":"1.5B model hits 67% on BIRD, beating 7B-32B rivals","feed_subtitle":"SFT on synthetic reasoning data, RL, and corrective self-consistency lift five small models by 31 points on average.","key_machinery":"The load-bearing mechanism is a two-dataset post-training pipeline followed by corrective self-consistency at inference. From the open-source SynSQL-2.5M dataset, the authors filter and re-tag 916K samples into SynSQL-Think-916K, where each question is paired with a <think>...</think> chain of thought and a <answer>...</answer> SQL query; a separate 310K merge-revision dataset is synthesized by generating eight candidate SQLs with a 7B teacher model, grouping them by execution result, and keeping the two highest-vote groups as drafts. The SQL generation model is SFT on the first dataset, then RL-trained with GRPO on BIRD; the merge revision model learns to compare two draft SQLs and their execution results and pick the correct one. At inference, the generator samples 64 SQLs, votes by execution result, and only when votes disagree does the revision model choose between the two leading drafts; the corrective gain is that this second stage repairs errors that self-consistency alone would keep.","core_discovery":"The central claim is that a sub-2B model trained to generate SQL via an explicit reasoning chain can outperform much larger models on the same benchmark, provided the training data is reorganized into two synthetic sets and inference is run with corrective self-consistency. On the BIRD development set, SLM-SQL-1.5B achieves 67.08% execution accuracy and SLM-SQL-0.5B achieves 56.87%, after training on SynSQL-Think-916K and SynSQL-Merge-Think-310K with SFT followed by GRPO-based reinforcement learning. The same models, trained only on BIRD, transfer to Spider with 79.06% test execution accuracy for the 1.5B model and 73.5% for the 0.5B model, which the paper reads as evidence that the method generalizes. Ablations attribute the gains to all three components: synthetic SFT (worth up to 21.9 points on the 0.5B model), RL (about 4–5 points), and the merge-revision step (about 5 points).","pith_inferences":["One testable extension is to remove the prompt instruction that one of the draft SQLs is correct and allow the merge model to write a fresh SQL; if the roughly five-point gain persists, the revision step learns genuine correction rather than selection between two candidates.","Because the method's pass@64 is 0.8264 while final execution accuracy is 67.08%, the gap suggests the revision stage is the current ceiling; improving candidate diversity or allowing the model to combine fragments could push small models further.","The same SFT-on-synthetic, RL-with-execution-rewards, corrective-consistency recipe is a natural fit for other code-generation tasks such as Python generation, where execution-based rewards exist, so the contribution may generalize beyond SQL.","If the method is as data-efficient as reported, 0.5B models could plausibly serve on-device database interfaces, where 56.87% execution accuracy at low cost may be a usable trade-off; the paper does not measure end-to-end latency."],"forward_implications":["A 1.5B model reaching 67.08% on BIRD development surpasses several 7B, 15B, and 32B open-source baselines, so parameter count alone is not the binding constraint for text-to-SQL once post-training is right.","The same BIRD-trained models transfer to Spider without additional RL (79.06% test execution accuracy for 1.5B), suggesting the recipe induces general SQL skill rather than benchmark-specific memorization.","Inference cost is small in absolute terms: at $0.26 per hour GPU rental, 64 samples cost $0.00046 per question on BIRD development, so the performance is attainable on commodity hardware.","Increasing test-time samples monotonically improves the SLM-SQL pipeline (pass@64 = 0.8264), meaning the bottleneck is partly sampling budget, not raw model capability.","Removing SFT, RL, or corrective self-consistency each drops accuracy by roughly 4–22 points, so the paper claims all three components are necessary for the reported results."],"supporting_citations":[{"why":"Supplies the SynSQL-2.5M synthetic dataset with chain-of-thought annotations that are filtered into SynSQL-Think-916K, and the OmniSQL baselines compared in Table 2.","marker":"(Li et al., 2025b)"},{"why":"Defines the corrective self-consistency two-stage generation and merge-revision method that SLM-SQL adapts, and supplies the 3B baseline it outperforms.","marker":"(Sheng and Xu, 2025)"},{"why":"Provides the SQL-R1 recipe of SFT on synthetic data followed by RL that the authors adopt for the SQL generation model.","marker":"(Ma et al., 2025)"},{"why":"Provides Think2SQL, cited alongside SQL-R1 as the precedent for SFT-before-RL post-training on synthetic text-to-SQL data.","marker":"(Papicchio et al., 2025)"},{"why":"Supplies GRPO, the reinforcement-learning algorithm used for post-training both the SQL generation and merge revision models.","marker":"(Shao et al., 2024)"},{"why":"Defines the BIRD benchmark and its development set, where the headline execution-accuracy numbers are measured.","marker":"(Li et al., 2024b)"},{"why":"Defines the Spider dataset used to test cross-dataset generalization of the BIRD-trained models.","marker":"(Yu et al., 2019)"},{"why":"Defines the Qwen2.5-Coder base models, including the 0.5B and 1.5B variants and the 7B teacher used to synthesize merge-revision candidates.","marker":"(Hui et al., 2024)"}],"fun_headline_variants":["Small SQL models gain 31 pts on BIRD with reasoning data","Synthetic reasoning lifts tiny SQL models to 67% on BIRD","0.5B and 1.5B SQL models outdo bigger rivals","SLM-SQL: sub-2B models beat larger ones on BIRD","0.5B hits 57%, 1.5B hits 67% on BIRD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The merge-revision step assumes that at least one of the two highest-vote draft SQLs is correct—the training prompt literally tells the model this—so the roughly five-point corrective gain and the headline 67.08% hold only when the sampling budget almost always puts a correct query in the top two vote groups.","fun_headline_variants_meta":{"raw":{"variants":["Small SQL models gain 31 pts on BIRD with reasoning data","Synthetic reasoning lifts tiny SQL models to 67% on BIRD","0.5B and 1.5B SQL models outdo bigger rivals","SLM-SQL: sub-2B models beat larger ones on BIRD","0.5B hits 57%, 1.5B hits 67% on BIRD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001653,"raw_usage":{"total_tokens":6612,"prompt_tokens":1038,"completion_tokens":5574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":5469}},"tokens_in":654,"tokens_out":5574,"duration_ms":41086,"temperature":1.0,"reasoning_tokens":5469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:37:11.932443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On BIRD development, for each question check whether a correct SQL appears among the 64 sampled candidates and whether it appears in the top two execution-vote groups; if the top-two groups are correct on only a small fraction of the questions the final answer gets right, the merge stage cannot be making the claimed repairs. Alternatively, run the merge revision model without the instruction that one draft SQL is correct and see whether the roughly five-point execution-accuracy gain disappears.","supporting_citations":[],"review_version":1}