{"id":"740f5b02-10e3-42da-b545-23e10b9cc810","arxiv_id":"2502.06759","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A pipeline that generates database-validated chain-of-thought SQL rationales from a 70B teacher, and fine-tunes an 8B student on the longest rationales, improves BIRD dev execution accuracy to 67.41%.","lead":"This paper trains small language models to write SQL queries in step-by-step chunks, using a large teacher model to generate and validate the intermediate steps against a database. Fine-tuning on the longest validated rationales raised BIRD dev execution accuracy from 66.17% to 67.41%, with the largest gains on complex queries and the added benefit of a visible reasoning trace.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Execution-match validation in §2.2 only checks the final SQL; if intermediate steps are unsound, the interpretability claim and the source of the accuracy gain are unverified.","rationale":"The reader's weakest assumption identifies execution equivalence as a proxy for reasoning correctness. My stress-test agrees: the validation in Section 2.2 is the single most load-bearing assumption because the entire method is built on the quality of these synthetic rationales. If the intermediate steps are not sound, the paper's claim of 'interpretable, user-friendly output' collapses, and the accuracy improvement could be a side effect of the final SQL in the trace rather than of genuine step-by-step reasoning. The reader's verdict of CONDITIONAL already accounts for this concern, so my read does not change the verdict. The proposed audit is a direct, feasible check: it would settle whether the labeled CoTs actually contain sound reasoning. If the audit passes, the concern is resolved; if it fails, the paper must either revise the validation to check intermediate steps or temper its interpretability claim.","tokens_in":8593,"tokens_out":4580,"duration_ms":43469,"concrete_test":"Have two independent annotators, blind to the paper's results, audit a random sample of 200 validated CoTs (100 from the dynamic few-shot stage and 100 from the rationalization stage). For each CoT, annotators check every intermediate SQL statement and explanation for: (a) syntactic validity of each intermediate SQL, (b) whether the explanation accurately describes the SQL, and (c) whether the step is a plausible partial solution that the next step extends (e.g., no table or column introduced without justification, no condition that contradicts the question). Compute inter-annotator agreement. If more than 10% of audited CoTs contain at least one unsound intermediate step, the interpretability claim is unsupported and the accuracy gain cannot be attributed to sound reasoning without further analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuning on validated SQL-building rationales improves accuracy and provides interpretable step-by-step reasoning. The validation procedure (Section 2.2) labels a CoT as positive only when the final SQL in the trace executes to the exact same result set as the gold SQL. This never checks the correctness of the intermediate SQL statements or the explanatory text. The paper's own manual assessment covers only the 86 remaining instances after the rationalization stage (Section 3.1); no systematic audit of the thousands of validated CoTs is reported. Consequently, a large fraction of the training rationales may contain flawed intermediate steps (e.g., an intermediate query that joins the wrong tables, an explanation that misstates a SQL keyword, or a step that is logically unnecessary) while still passing the final-result check. Since the student model is trained to autoregressively produce these rationales, it can learn to emit plausible but unsound chains. That would directly undermine the 'interpretable, user-friendly output' part of the claim. It also leaves open the possibility that the observed execution-accuracy gain, if real, is driven by the final SQL embedded in the trace rather than by sound intermediate reasoning. The paper provides no evidence that the intermediate steps are valid, so the central claim depends on an untested assumption about the quality of the very data the method produces.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for generating chain-of-thought (CoT) rationales for text-to-SQL by combining a small set of manually annotated seeds, a dynamic few-shot distillation procedure from a 70B teacher model, execution-based validation of the final SQL, and a fine-tuned rationalization model that reaches full training coverage. The authors fine-tune Llama 3.1 8B on the BIRD dataset with and without these rationales and report that the longest CoT variant at full coverage improves execution accuracy over a gold-SQL baseline, especially on challenging queries (67.41 vs. 66.17 overall; +4.86 on the challenging subset).","tokens_in":8820,"tokens_out":5422,"duration_ms":46353,"significance":"If the result holds, the framework would offer a practical way to produce step-by-step SQL-building rationales at scale for training smaller text-to-SQL models, improving accuracy on complex queries and providing interpretable outputs. The paper is transparent about coverage numbers and reports the full pattern of results at both coverage levels, which is a strength. However, the current evidence is not yet fully convincing because the evaluation relies on single runs without error bars, the best-performing variant is chosen after inspecting dev-set results, and the validation of rationales checks only the final SQL, not the soundness of intermediate steps.","major_comments":[{"comment":"The central comparison rests on a single fine-tuning run per condition, with no error bars, confidence intervals, or significance tests. The reported overall improvement is +1.24 percentage points (67.41 vs. 66.17) on the dev set, and the +4.86 improvement on challenging queries is on an unreported subset size. Because the short and long variants are evaluated on the same dev set and the best-performing variant is highlighted, the headline result may reflect selection over multiple configurations. Multiple seeds or a pre-registered selection rule are needed to support the claim that long CoTs enhance accuracy.","section":"Section 3.2, Table 2"},{"comment":"The validation component labels a generated CoT as positive only when the final SQL in the trace executes to the same result set as the gold SQL. This does not verify the correctness of the intermediate SQL statements or the textual explanations. The manual assessment described in Section 3.1 covers only the 86 instances remaining after the rationalization stage, not the thousands of validated CoTs used for training. Consequently, the interpretability claim is unverified, and the observed accuracy gain could come from the final SQL embedded in the trace rather than from sound intermediate reasoning. A systematic audit of intermediate steps, or an ablation in which intermediate steps are corrupted to measure their effect, would be needed to support the interpretation.","section":"Section 2.2, Section 3.1"},{"comment":"The conclusion states that 'long CoTs enhance the accuracy of text-to-SQL tasks,' but Table 2 shows this only in the full-coverage (99.02%) Long CoT condition. At 73.86% coverage, both CoT variants are below the gold-SQL baseline overall (63.82 and 61.99 vs. 63.95), although they improve on moderate and challenging queries. As written, the claim overgeneralizes; it should be restricted to the full-coverage, Long-variant setting, or the paper should analyze the conditions under which rationales help.","section":"Conclusion vs. Table 2"}],"minor_comments":[{"comment":"The cross-references to 'Table 3.1' and 'Table 3.2' in the text should be corrected to 'Table 1' and 'Table 2'.","section":"Section 3.1, Section 3.2"},{"comment":"The text says 'Using 73.68% of the training set,' but Table 2 reports 73.86% coverage for the dynamic few-shot stage; please reconcile.","section":"Section 3.2"},{"comment":"Decimal places are inconsistent (e.g., '70.7' vs. '70.70', '46.9' vs. '46.90'), and the column header 'Challenge' is used alongside 'challenging' in the text; please unify terminology.","section":"Table 2"},{"comment":"The phrase 'an increase of +4.86%' should be stated as '+4.86 percentage points' to avoid ambiguity.","section":"Section 3.2"},{"comment":"The statement that two result sets 'match exactly' should specify whether row order and duplicate rows are considered, since this affects the interpretation of SQL equivalence.","section":"Section 2.2"},{"comment":"The paper does not report fine-tuning hyperparameters (learning rate, epochs, batch size, sequence length), which would be needed for reproduction.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style contribution with a useful pipeline and transparent reporting, but the evaluation is not yet at journal strength. The two load-bearing issues are the lack of statistical grounding for small accuracy differences and the unverified soundness of intermediate rationales. If the authors add multiple runs or significance tests and provide at least a targeted audit of intermediate steps, the paper could become publishable. The subset size for the challenging queries should also be disclosed, as the headline gain may rest on a small number of examples."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one real thing: it shows that fine-tuning an 8B text-to-SQL model on execution-validated chain-of-thought rationales can nudge BIRD dev accuracy from 66.17 to 67.41, with the gain concentrated on hard queries. The pipeline is a clean combination of known pieces—dynamic few-shot with SQL-keyword retrieval, STaR-style rationalization, execution checking—and it is honest enough to report that the gain only appears after full coverage and with the longest CoT variant.\n\nWhat is new: the SQL-keyword similarity retrieval for few-shot selection and the iterative coverage-growth distillation. These work in the narrow sense that validation coverage goes from 53% with two manual seeds to 74% after dynamic few-shot, then to 99% with the rationalization model. That is a concrete, reproducible-sounding number.\n\nThe main soft spot is the validation criterion. Matching the final result set guarantees the final SQL is correct, but says nothing about intermediate steps. Since the paper's secondary claim is interpretability, that is a real gap. The manual check of only 86 instances is too small to license the claim. Second, the comparison is single-run and the Long-vs-Short choice is made after looking at the dev set. At 74% coverage the CoT models are actually worse overall, so the positive result is conditional on the rationalization stage and on dev-set selection. That makes the headline 'long CoTs enhance accuracy' too strong. Third, the gain is about one point; with no error bars, it could be noise, though it is consistent across the three difficulty strata.\n\nTo its credit, the paper reports the negative intermediate result and does not hide the six training-set comparison. The appendix example makes the rationale format concrete.\n\nWho is this for? People working on small-model text-to-SQL or rationale distillation. It is not a breakthrough; it is a decent data-generation recipe with a plausible-but-thin evaluation. I would send it to a serious referee. It deserves revision rather than rejection: multi-seed runs, held-out selection, and a systematic audit of intermediate steps would materially improve it. The accuracy claim is worth engaging; the interpretability claim needs more evidence.","headline":"A competent pipeline for execution-validated CoT rationales in text-to-SQL, with a small but plausible BIRD gain; the interpretability claim is softer than the accuracy claim because only final SQL is validated.","tokens_in":9400,"tokens_out":2264,"would_cite":false,"duration_ms":20948,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a small text-to-SQL model on validated, step-by-step SQL-building rationales improves execution accuracy on the BIRD benchmark, with the longest rationale variant gaining 4.86 points on the most challenging queries.","keywords":["text-to-SQL","chain-of-thought","rationalization","knowledge distillation","dynamic few-shot","BIRD benchmark","execution accuracy","SQL generation"],"falsifier":"Take the same 8,807 training pairs and the same longest validated rationales, replace each intermediate SQL statement with a different executable statement that still leads to the same final result set, and fine-tune the same Llama 3.1 8B model; if BIRD dev accuracy does not drop, the reported gain comes from output format and length rather than from the correctness of the intermediate reasoning.","tokens_in":8367,"feed_emoji":"🧠","tokens_out":9924,"duration_ms":77637,"temperature":0.7,"pith_summary":"This paper claims that fine-tuning a small text-to-SQL model on step-by-step 'SQL-building rationales'—chains of increasingly complex executable SQL statements with explanations—makes the model more accurate on hard queries and more transparent in its reasoning. The authors build a pipeline that starts from two manually annotated examples, uses a 70B teacher model with dynamic few-shot selection to propose rationales for the BIRD training set, keeps only those whose final SQL executes to the exact result set of the gold SQL, and then trains an 8B rationalization model to cover the remaining training pairs. On the BIRD dev set, an 8B Llama model fine-tuned on the longest validated rationales reaches 67.41% execution accuracy, versus 66.17% for the same model trained on gold SQL alone, with a 4.86-point gain on the most challenging queries. The paper takes this as evidence that long, validated reasoning chains transfer to small models and can be produced at scale with minimal manual annotation.","feed_headline":"Chain-of-thought SQL rationales lift text-to-SQL accuracy","feed_subtitle":"Fine-tuning an 8B model on validated reasoning steps gains +4.86 points on the hardest BIRD queries.","key_machinery":"The load-bearing mechanism is the validated-SQL-CoT loop: a teacher LLM receives a schema-filtered prompt, the question, the gold SQL, and the top-n most similar validated rationales, and produces a Markdown-formatted plan followed by intermediate executable SQL statements; a DBMS validates the trace by requiring exact result-set equality between the final SQL and the gold SQL. Validated traces accumulate in a repository and are used as few-shot examples in the next iteration, selected by cosine similarity over SQL-keyword vectors. A rationalization model, fine-tuned on these traces with (question, gold SQL) as input, then generates rationalizations for every remaining training pair. The comparison between short and long validated rationales is what isolates length of reasoning as the factor that raises execution accuracy on hard queries.","core_discovery":"The central discovery is that for text-to-SQL, the target used in fine-tuning can be upgraded from the final SQL answer to a validated, step-by-step construction of that answer, and that among the validated constructions the longer ones are the better training signal. A rationale is considered valid only if the final SQL it contains, when executed against the database, returns exactly the same result set as the gold SQL. The generation loop iterates dynamic few-shot prompting with a 70B teacher, where the few-shot examples are the most structurally similar previously validated rationales, ranked by cosine similarity of SQL-keyword frequency vectors. Once coverage stops growing, an 8B rationalization model is fine-tuned on the validated rationales and applied to all remaining training pairs, with the gold SQL given as input, so that the training set is fully covered by rationalizations. The controlled comparison on BIRD shows that the longest-Co-trained model outperforms the gold-only baseline across all difficulty levels and gains the most on challenging queries, while also emitting human-readable explanations of each SQL-building step.","pith_inferences":["A stricter test of the paper's interpretation would separate format from reasoning: if the same long traces with their intermediate steps shuffled still produce the same dev accuracy, the gain is a property of output length and structure, not of the validity of the reasoning.","The SQL-keyword cosine-similarity selector is a deliberately simple retrieval; replacing it with semantic or learned similarity could change the coverage curve, so the reported gains are partly a statement about this particular few-shot selection.","The method should transfer to other tasks with executable intermediate artifacts—for example, data-analysis code or spreadsheet formulas—where result-set equivalence is cheap to check and the final output can validate the trace.","Because the validation criterion certifies only the final result, the generated rationales are optimized for execution accuracy rather than for psychological plausibility; user-facing explanations may therefore be correct in outcome but not in the stated rationale."],"forward_implications":["If the claim holds, small text-to-SQL models can be improved by changing only the training target—from SQL answers to validated step-by-step rationales—without changing architecture or scaling.","The rationalization model, once trained, can annotate new text-to-SQL datasets across different domains without repeating the manual-annotation and dynamic-few-shot loop, as the paper explicitly proposes.","Longer validated rationales become a practical knob: fine-tuning on the longest traces gives the best accuracy on simple, moderate, and challenging queries alike, while preserving interpretability.","The validation step doubles as a data-cleaning tool, since rationalization failures often expose inconsistencies or errors in the gold SQL itself."],"supporting_citations":[{"why":"Supplies the BIRD benchmark, its train/dev splits, and the difficulty-level categorization used in all accuracy comparisons.","marker":"Li et al., 2023"},{"why":"Provides Llama 3.1, the 70B teacher and 8B student models used for generation, rationalization, and fine-tuning.","marker":"Dubey et al., 2024"},{"why":"Provides the STaR rationale-versus-rationalization distinction and the strategy of bootstrapping reasoning by fine-tuning on validated traces.","marker":"Zelikman et al., 2022"},{"why":"Supplies the extractive schema linker that filters database schemas in every prompt.","marker":"Glass et al., 2025"},{"why":"Defines chain-of-thought prompting, the reasoning format that the generated SQL rationales instantiate.","marker":"Wei et al., 2022"}],"fun_headline_variants":["Rationalizing SQL step-by-step boosts hard-query accuracy","Validated step-by-step SQL rationales: better training for text-to-SQL","Longer rationales make SQL fine-tuning smarter on complex queries","Explainable step-by-step SQL rationales lift accuracy on BIRD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that when a generated chain-of-thought's final SQL executes to the exact result set of the gold SQL, the intermediate steps are correct enough to serve as a training target.","fun_headline_variants_meta":{"raw":{"variants":["Rationalizing SQL step-by-step boosts hard-query accuracy","Validated step-by-step SQL rationales: better training for text-to-SQL","Longer rationales make SQL fine-tuning smarter on complex queries","Explainable step-by-step SQL rationales lift accuracy on BIRD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3036,"prompt_tokens":887,"completion_tokens":2149,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2074}},"tokens_in":503,"tokens_out":2149,"duration_ms":14871,"temperature":1.0,"reasoning_tokens":2074,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:24:49.062478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 8,807 training pairs and the same longest validated rationales, replace each intermediate SQL statement with a different executable statement that still leads to the same final result set, and fine-tune the same Llama 3.1 8B model; if BIRD dev accuracy does not drop, the reported gain comes from output format and length rather than from the correctness of the intermediate reasoning.","supporting_citations":[],"review_version":1}