{"id":"046d74cd-1094-419a-8898-0efd8169e2fa","arxiv_id":"2505.12768","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An execution-aware reinforcement learning framework that interleaves intermediate SQL execution into the reasoning path improves text-to-SQL accuracy on Spider and BIRD at the 7B scale.","lead":"ReEx-SQL trains a 7B language model to write SQL queries that check their work by running test queries against the database during generation, then uses that feedback to fix mistakes before outputting the final query. It reports 88.8% accuracy on Spider and 64.9% on BIRD, about 2.7 and 2.6 points above a standard reasoning baseline, while cutting inference time with tree-structured search.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dev-set reward-weight tuning likely confounds the claimed 2.6–2.7% gain: the baseline's reward weights are unreported and untuned, while ReEx-SQL's headline result uses weights selected on BIRD Dev.","rationale":"The reader's stated weakest assumption is the correctness of GRPO when execution-feedback tokens are masked from the loss. That concern is real but largely addressed by the paper's explicit masking statement: the likelihood ratio and KL penalty are computed only over model-generated tokens, which is the standard way to handle environment tokens in multi-turn policy optimization. A more concrete threat to the central claim is the fairness of the Table 3 comparison. Table 5 reveals that the headline results depend on a reward-weight configuration selected on BIRD Dev, while the baseline's weights are undisclosed and apparently not tuned. ReEx-SQL with the Base weights is essentially indistinguishable from the baseline, so the claimed gain may be an artifact of hyperparameter search. The Table 3 anomaly (Execution-Aware Reasoning No Post-Training dropping to 37.0 on Spider Test) further complicates the interpretation, as the untrained model's poor prompt compliance means GRPO is partly teaching format following rather than execution-aware reasoning. The proposed concrete test directly addresses the most load-bearing issue without requiring code release or theoretical re-derivation.","tokens_in":28133,"tokens_out":8785,"duration_ms":87437,"concrete_test":"Re-run the Standard Reasoning GRPO baseline using the same reward-weight tuning protocol applied to ReEx-SQL: search over the Table 5 configurations (with the exploration reward omitted for the baseline) on BIRD Dev, select the best weights, and report the resulting EX/TS on all benchmarks. Also report the exact reward weights used for the Table 3 baseline. If the tuned baseline reaches or exceeds ReEx-SQL's 63.4 BIRD Dev / 88.4 Spider Dev EX, the claimed 2.6/2.7-point advantage is not attributable to execution-aware reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 3, where ReEx-SQL GRPO outperforms the Standard Reasoning GRPO baseline by 2.6% on BIRD Dev and 2.7% on Spider Dev. Section 4.4, Table 5 shows that these headline numbers use the 'MaxTune' weight vector (w = 2.0, 1.0, 3.0, 1.0, 2.0), explicitly labeled 'Best' after selection on BIRD Dev. The paper says the baseline uses 'identical training configurations' except for the prompt and the exploration reward, yet the baseline's exact reward weights are never reported, and no equivalent dev-set tuning procedure is described. This matters because ReEx-SQL with the 'Base' weights in Table 5 (w = 1.5, 0.8, 1.5, 0.8, 0.5) achieves 60.9 EX on BIRD Dev, essentially tied with the baseline's 60.8. The entire reported advantage appears to come from raising the format, execution, and exploration weights, not from execution-aware reasoning itself. If the standard baseline were tuned by the same protocol, the 2.6/2.7-point gap could shrink or disappear, directly undermining the central assertion that interleaving execution feedback into decoding is what improves accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReEx-SQL, a text-to-SQL framework that interleaves intermediate SQL execution feedback into the model’s decoding-time reasoning. The method uses structured prompts with markup tags (<think>, <intermediate_sql>, <result>, <final_sql>), a stepwise rollout mechanism with a SQL executor, and GRPO training with a composite reward consisting of format, exact-match, execution, entity-match, and exploration terms. It also proposes a tree-structured decoding strategy with execution-based self-consistency. Experiments on Qwen2.5-Coder-7B-Instruct report 88.8% execution accuracy on Spider Dev and 64.9% on BIRD Dev, outperforming a standard-reasoning GRPO baseline by 2.7 and 2.6 points respectively, and reducing syntax errors from 4.5% to 1.9% on BIRD Dev.","tokens_in":28468,"tokens_out":7186,"duration_ms":77192,"significance":"If the claimed results hold, the paper makes a useful practical contribution: it demonstrates that a small open model can be trained to use live execution feedback during decoding, that a controlled GRPO comparison shows gains over standard reasoning, and that tree-structured decoding can cut inference time substantially. The controlled Table 3 ablation, the reward-component ablations, and the error analysis are valuable. However, the central comparison is weakened by reward-weight selection on the reported dev sets, the baseline is not tuned by the same protocol, no variance estimates are provided, and the theoretical justification for masking external tokens is incomplete. The potential of the idea is clear, but the current evidence does not yet establish that the reported gains come from execution-aware reasoning itself.","major_comments":[{"comment":"Table 3’s headline comparison (Execution-Aware GRPO vs. Standard Reasoning GRPO) uses the MaxTune weight vector, which Table 5 labels \"Best\" after selection on BIRD Dev. The standard-reasoning baseline’s reward weights are never reported, and no equivalent tuning procedure is described for it. This is load-bearing: under the Base configuration (Table 5), ReEx-SQL achieves 60.9 EX on BIRD Dev, nearly identical to the baseline’s 60.8 EX in Table 3. Thus the claimed 2.6/2.7-point advantage may be attributable to reward-weight tuning rather than to execution-aware reasoning. Please either select weights on a disjoint validation split, report the baseline under all weight configurations, or otherwise demonstrate that the baseline is compared at its own best setting.","section":"§4.3, §4.4 (Tables 3 and 5)"},{"comment":"The GRPO objective in Eq. (1) writes the importance ratio πθ(yi|x;E)/πθold(yi|x;E) over full rollouts yi, but the loss is computed only over model-generated tokens, with external execution-feedback tokens masked. The paper asserts without proof that this masking \"ensures correct GRPO optimization.\" Without a formal likelihood decomposition that separates model-generated tokens from deterministic environment-generated tokens, the masked ratio is not shown to be an unbiased estimator of the intended policy gradient. Please provide that derivation and state under what conditions the masked update is valid; otherwise the trained policy is not precisely the one described.","section":"§3.1, Eq. (1)"},{"comment":"Execution-Aware Reasoning with No Post-Training collapses on Spider Test (EX 37.0, TS 57.4) while staying far closer to other paradigms on BIRD Dev (48.6) and on other Spider variants (57.4–80.0). The paper does not explain this roughly 40-point drop. Because this collapse inflates the reported GRPO gain for execution-aware reasoning (14.8 points vs. 11.3 for the baseline), the paper should quantify format non-compliance or other failure causes for this condition, or remove/adjust this comparison.","section":"§4.3, Table 3"},{"comment":"All main results are single-run point estimates without standard deviations, seeds, or significance tests. The central advantage over the baseline is 2.6–2.7 EX points, which could easily be within run-to-run noise on a dev set of roughly 1,500 examples. No code or data are provided, so variance cannot be assessed and the results are not reproducible. Please report multiple seeds or confidence intervals, and release the code and the trained model to support the central claim.","section":"§4.2, §4.3, Tables 1 and 3"}],"minor_comments":[{"comment":"The exploration reward uses a sigmoid σ(n) over the interaction count, but σ is never defined. Please define it explicitly, including its range and how it scales with n.","section":"§3.3, Eq. (4)"},{"comment":"The KL penalty is written as β DKL(πθ ∥ πθref) without an outer expectation over the input distribution x; aligning the notation with the standard GRPO objective would remove ambiguity.","section":"§3.1, Eq. (1)"},{"comment":"The \"execution-based self-consistency\" mechanism is not specified in detail. Please state how candidate SQLs are grouped and selected (for example, by exact result-set equality) and how the decoding tree is expanded or pruned.","section":"§3.4"},{"comment":"The text uses \"significantly\" in several places (\"significantly narrowing the gap\", \"significantly improves\") without statistical tests or variance estimates; please qualify these statements.","section":"§4.2"},{"comment":"The notation πθold(·|x;E) and the interleaving operator ⊗ are introduced before being fully defined; a short formal definition at first use would improve readability.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv preprint with no code or data release. Given that the central 2.6–2.7-point gain may depend on reward-weight selection on the reported dev set, I would ask the editor to require the authors to either provide a validation-split weight selection or report the baseline tuned by the same protocol. I would also encourage the journal to require code and seed-variance reporting before considering the main claim established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nReEx-SQL is a reasonable attempt to bring execution feedback into the decoding loop for text-to-SQL, trained with GRPO and a composite reward that includes an exploration bonus. The headline results are decent for a 7B model (88.8% Spider Dev, 64.9% BIRD Dev), and the tree-structured decoding efficiency gain (51.9% time reduction on BIRD Dev) is genuinely useful. The controlled paradigm ablation in Table 3 is the right kind of experiment, and the error analysis showing syntax errors drop from 4.5% to 1.9% supports the story.\n\nThe soft spots are in the comparison's foundations. The reward weight vector is selected on BIRD Dev (Table 5, MaxTune is labeled 'Best') and then reported on the same BIRD Dev. That is mild selection on the evaluation set. More importantly, the baseline's reward weights are never explicitly reported. The paper says the baseline uses 'identical training configurations,' except for the prompt and the exploration reward, which suggests the baseline also uses MaxTune's weights for the shared components. If that is true, ReEx-SQL with the low 'Base' weights (60.9) essentially matching the baseline (60.8) would actually be evidence for the execution-aware paradigm, not against it. But because the baseline's weights are not stated, the reader cannot tell whether the reported 2.6/2.7-point gain is due to the reasoning paradigm or to asymmetric weight tuning. That is a clarity failure, not necessarily a fatal one. The stress-test's stronger claim that the entire advantage comes from weight tuning is not directly supported by the text; it rests on an assumption about baseline weights that the paper does not confirm.\n\nThere are two smaller issues worth naming. The 37.0% EX on Spider Test for untrained execution-aware prompting (Table 3) is a dramatic collapse relative to standard reasoning (78.3%) and is never explained. It suggests the prompt is highly conditioned on GRPO and may be brittle. Also, the GRPO objective in Eq. (1) is written over full rollouts including environment tokens; the paper asserts masking the tokens 'ensures correct GRPO optimization' without proof. That is a gap in theory, though practice may be fine.\n\nNo code or data is released, and there are no variance estimates. These omissions hurt reproducibility but are common in this area.\n\nThis paper is worth a serious referee. It combines known components in a new way with a plausible mechanism and a strong efficiency result. The referee should push for full disclosure of baseline reward weights, a weight-tuning protocol for the baseline, and an explanation of the Spider Test anomaly. I would send it to review, with the expectation of heavy revision.","headline":"A promising but partially underspecified text-to-SQL framework; the headline gain is clouded by unreported baseline reward weights, though the efficiency result is solid.","tokens_in":28963,"tokens_out":8076,"would_cite":false,"duration_ms":75360,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReEx-SQL claims that execution-aware reasoning—interleaving exploratory SQL with execution feedback during decoding—trains a 7B text-to-SQL model to beat standard chain-of-thought by 2.7% on Spider and 2.6% on BIRD.","keywords":["text-to-sql","execution feedback","reinforcement learning","GRPO","reasoning","SQL generation","tree-structured decoding","execution accuracy"],"falsifier":"Run the trained ReEx-SQL-7B on the BIRD development set with the <result> feedback replaced by a constant dummy string (for example, 'ok') while keeping the prompt format unchanged; if execution accuracy does not drop materially below the reported 64.9%, the gain is not attributable to the execution-aware mechanism.","tokens_in":27997,"feed_emoji":"🗄️","tokens_out":5251,"duration_ms":50255,"temperature":0.7,"pith_summary":"The paper argues that a text-to-SQL model can be trained to interleave its own reasoning with live database queries, writing exploratory SQL, reading execution results or error logs, and revising its plan before emitting a final query. This execution-aware reasoning, trained with GRPO and a five-part reward, lets a 7B model reach 88.8% execution accuracy on Spider and 64.9% on BIRD, beating a standard reasoning baseline by 2.7% and 2.6%. If true, it shows execution feedback is not just a post-hoc verification signal but a usable training signal for small open-weight models. The paper also claims that tree-structured decoding over interaction steps cuts inference time by 51.9% on BIRD Dev while improving accuracy.","feed_headline":"SQL model that queries mid-reasoning beats CoT by 2.7%","feed_subtitle":"Execution-aware reinforcement learning lets a 7B model catch its own SQL errors before the final answer.","key_machinery":"The central object is the execution-aware reasoning path: a structured rollout in which the model emits <think> reasoning, an <intermediate_sql> exploratory query, receives <result> execution feedback, and only then produces <final_sql>. Training extends GRPO to these interleaved rollouts, masking the externally generated feedback tokens from the loss, and scores rollouts with a composite reward: format, exact match, execution (validity plus result equality), entity match (schema overlap with the gold SQL), and an exploration reward that penalizes duplicate intermediate queries and rewards more interaction steps when the final SQL is wrong. Decoding uses a tree where each node is an interaction step, with execution-based self-consistency selecting the final SQL.","core_discovery":"ReEx-SQL claims that interleaving SQL execution into the decoding process—rather than using it only to correct or select an already-generated query—is what drives the accuracy gains. On the BIRD development set it reports 64.9% execution accuracy and 73.1% VES, and on Spider development 88.8% EX and 83.7% TS, with a 7B Qwen2.5-Coder model. Compared with the same model trained under a standard chain-of-thought prompt with GRPO, it gains 2.6% on BIRD and 2.7% on Spider, and reduces syntax errors from 4.5% to 1.9%. The gains are largest on the hardest questions, and the method also improves robustness variants such as Spider-Syn, Spider-Realistic, and Spider-DK.","pith_inferences":["An implicit consequence the authors do not develop: the approach turns the database into a learned policy's environment, so the exploration reward is a stand-in for curiosity; decoupling the policy from the executor may enable training in settings where the real database is unavailable.","A testable extension beyond the paper: apply the execution-aware rollout to multi-hop numerical reasoning over tables, where intermediate SQL serves the same verification role but intermediate answers are numeric.","The masking claim in Eq. (1) is asserted rather than proven; a careful reader may want to check whether the KL penalty and importance weighting over the full interleaved sequence change the effective objective.","If the mask is valid, the same training recipe could apply to any environment with external feedback, such as Python executors or API calls, without modifying GRPO."],"forward_implications":["If the central claim holds, execution feedback can serve as a dense training signal for small models, narrowing the gap with large proprietary systems on SQL tasks.","Tree-structured, feedback-guided decoding offers both accuracy and latency gains, making interactive decoding practical in settings where inference cost matters.","The error profile shifts: syntax errors drop from 4.5% to 1.9%, suggesting that in-loop execution catches syntactic issues that post-hoc correction would overwrite.","The same recipe—interleaved tool feedback plus composite reward under GRPO—could transfer to other structured generation tasks such as code repair or data-wrangling."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that the paper extends with execution-aware rollouts.","marker":"[31]"},{"why":"Prior work on execution feedback for text-to-SQL reasoning that ReEx-SQL contrasts with its in-loop approach.","marker":"[17]"},{"why":"A post-hoc multi-path selection baseline that represents the verification-as-separate-stage approach the paper argues against.","marker":"[26]"},{"why":"Provides the database prompts and schema filtering used in training and inference, and serves as an SFT baseline.","marker":"[36]"},{"why":"Defines the BIRD benchmark and its evaluation protocol, the basis for the main BIRD results.","marker":"[43]"},{"why":"Defines the Spider benchmark and its evaluation protocol, the basis for the main Spider results.","marker":"[44]"},{"why":"Introduces the test-suite accuracy metric used for TS scores on Spider and its variants.","marker":"[48]"},{"why":"Introduces chain-of-thought prompting, which the standard reasoning baseline is built on.","marker":"[1]"}],"fun_headline_variants":["ReEx-SQL: interleaving SQL execution into reasoning lifts accuracy by 2.7%","Query during reasoning: ReEx-SQL beats chain-of-thought on SQL","7B SQL model that executes queries mid-reasoning outperforms CoT","Execution feedback woven into decoding lifts SQL accuracy and speed","ReEx-SQL: reason with database feedback, not just after the fact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"It assumes that masking the externally generated execution tokens from the loss still yields a valid GRPO update for the model's own tokens, even though those tokens are part of the rollout sequences used in the objective.","fun_headline_variants_meta":{"raw":{"variants":["ReEx-SQL: interleaving SQL execution into reasoning lifts accuracy by 2.7%","Query during reasoning: ReEx-SQL beats chain-of-thought on SQL","7B SQL model that executes queries mid-reasoning outperforms CoT","Execution feedback woven into decoding lifts SQL accuracy and speed","ReEx-SQL: reason with database feedback, not just after the fact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1567,"prompt_tokens":1018,"completion_tokens":549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":634,"tokens_out":549,"duration_ms":6075,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:26:30.750763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained ReEx-SQL-7B on the BIRD development set with the <result> feedback replaced by a constant dummy string (for example, 'ok') while keeping the prompt format unchanged; if execution accuracy does not drop materially below the reported 64.9%, the gain is not attributable to the execution-aware mechanism.","supporting_citations":[{"cited_title":"Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-sqls","cited_arxiv_id":null,"evidence_quote":"Defines the BIRD benchmark and its evaluation protocol, the basis for the main BIRD results."},{"cited_title":"Chi, Quoc V","cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting, which the standard reasoning baseline is built on."}],"review_version":1}