{"id":"dc26cabc-24f4-4af9-8091-eeca641a1f52","arxiv_id":"2608.02876","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"BAP-SQL improves tight-budget text-to-SQL accuracy by estimating query cost, rewriting expensive queries, and enforcing hard limits with a runtime shield.","lead":"This paper introduces BAP-SQL, a planning system for AI agents that write database queries by estimating how expensive each query will be before running it and sometimes rewriting it to stay within a budget. On the BIRD benchmark it improved the share of correctly answered questions at tight budgets while using fewer tokens than a standard supervised approach.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-level benefit is shown for the estimate+rewrite package, but no retrained control isolates accurate estimates; if a retrained shuffled-estimate policy retains the gain, the central planning claim is not about estimate accuracy.","rationale":"Read in good faith, BAP-SQL is a well-executed empirical study: the budget ladder is frozen before dev evaluation, matched SFT and interface-matched SQL-RL control for training protocol, the locked retraining suite is independent of the main frontier, three seeds and per-seed ranges address training randomness, and the paper honestly reports the regime boundaries (L reversal, no database-work reduction, FINER token increase). The reader's weakest assumption about estimator accuracy is on the right track, but the sharper issue is internal rather than about transfer: the training-level factorial does not separate accurate estimates from the presence of the estimate+rewrite interface. The paper's own limitation paragraph explicitly names the missing retrained shuffled-estimate placebo. I do not regard this as a reason to reject: the package-level effect is real and replicated, and a fixed-policy noise test shows the trained policy is not indifferent to estimate content. But because the abstract and conclusion present pre-execution estimation as the mechanism, the missing retrained control leaves a gap in the central causal claim. The manuscript should either run that control or explicitly narrow the claim to 'the estimate+rewrite package under a calibrated estimator.' The existing editorial issues (empty 'Section .' reference, no artifact link) are minor; the scientific condition is the retrained estimator-robustness check. Thus the verdict remains CONDITIONAL, with partial agreement with the reader's framing.","tokens_in":12769,"tokens_out":15088,"duration_ms":139084,"concrete_test":"Retrain the locked-suite full BAP-SQL condition with the same SFT init, prompt pool, online filter, optimizer, and 120-update budget, but replace the policy-visible p50/p95 payloads with within-episode shuffled (or 30%-noise-corrupted) values while the runtime shield retains true estimates; run three seeds. Compare the full-minus-no-planning budgeted-success difference at XS/S to the reported 1.9/2.0 points. If the retrained shuffled-estimate difference stays within the [0.7,3.1]/[0.8,3.2] intervals, accurate estimates are not load-bearing; if it drops below those lower bounds or reverses, the central planning claim should be made conditional on estimator calibration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central training-level evidence (Table 4 left, locked retraining suite) is a 2x2 factorial whose 'planning' factor removes both policy-visible p50/p95 estimates and the Rewrite action, so it establishes the value of the combined estimate+rewrite package, not of accurate estimates. The paper's own limitation section states: 'The retrained no-planning control establishes a training-level contribution for the combined policy-visible estimate-and-rewrite package. It does not isolate estimator calibration or rewriting separately; a retrained shuffled-estimate placebo would address part of this gap.' The 30% estimator-noise injection and the shuffled-estimate row (Table 4 right) are fixed-policy: they measure how a policy trained on accurate estimates reacts at inference time, not whether retraining on degraded estimates preserves the gain. Since the abstract's mechanism is 'estimates query risk, rewrites SQL when useful,' the claimed advantage is conditional on estimator accuracy, but that condition is untested at the training level. If a retrained policy with shuffled estimates still shows the 1.9/2.0-point gain, the novelty reduces to action availability and reward shaping; if it collapses, the paper should state that the benefit is contingent on estimator calibration. The 98.8/97.0/96.0% p95 coverage reports calibration of the tail, not the accuracy of the p50 signals that drive rewrite decisions, so it does not close this gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes BAP-SQL, an agentic text-to-SQL loop with a pre-execution planning stage. Before executing SQL, an estimator returns p50/p95 predictions for rows, result tokens, and database work from EXPLAIN QUERY PLAN, a zero-row probe, and catalog statistics with calibration factors fitted on disjoint databases; a policy trained by RL across four joint budget levels can decide to inspect, estimate, rewrite, execute, manage evidence, answer, or abstain, while an independent runtime shield enforces hard limits. On a BIRD-derived dev set with Qwen3.5-4B, BAP-SQL reports Budgeted Success gains of 3.4/3.6 percentage points over matched SFT at XS/S with 4.5/5.0% fewer total tokens, and similar directional gains over interface-matched SQL-RL and adapted BACM-RL. The paper also reports transfer to FINER-SQL-4B and a 7B coder, a locked retraining factorial, fixed-policy diagnostics, and estimator calibration. The benefit attenuates as budgets loosen, reverses at the loosest budget L, and does not reduce database work.","tokens_in":12893,"tokens_out":10294,"duration_ms":87760,"significance":"The empirical package is careful and internally consistent. Strengths include paired database bootstrap intervals with per-seed stability, a locked 2x2 retraining suite, correctness-gated reward shaping, calibration factors fitted on disjoint databases, and explicit disclosure of limitations. The headline tight-budget gains are supported as an observed property of the trained system. The main weakness is that the training-level evidence for the advocated mechanism is at the package level: the no-planning control removes both policy-visible estimates and the Rewrite action, so the contribution of accurate cost estimates to learning is not isolated. Because the abstract attributes the gain to an estimate-then-rewrite loop, this gap is load-bearing for the paper's mechanistic claim rather than for the raw system-level result.","major_comments":[{"comment":"The locked retraining suite's headline planning effect compares Full BAP-SQL RL (34.5/37.2) with No-planning RL (32.6/35.2), but the no-planning condition removes both policy-visible p50/p95 estimates and the Rewrite action. The +1.9/+2.0 pp effect therefore establishes the combined estimate-and-rewrite package under the shared reward, not the contribution of estimator accuracy. The fixed-policy shuffled-estimate row (31.9/34.4) shows inference-time sensitivity but cannot tell us whether a policy retrained on shuffled estimates would retain the gain; the paper's own Limitations section says exactly this. Because the abstract and contribution statements present the mechanism as 'estimates query risk, rewrites SQL when useful,' the estimate component is load-bearing. Please add a retrained shuffled-estimate (or equivalent no-estimate) condition to the locked protocol, or explicitly demote the mechanistic claim to package-level in the abstract and conclusion. The reward x planning interaction does not resolve the confound because both factors are defined at the package level.","section":"Ablations; Table 4 (left panel)"},{"comment":"The evidence for estimator quality is p95 coverage (98.8% rows, 97.0% result tokens, 96.0% database work) and a fixed-policy 30% noise injection costing 1.8 pp. Coverage of the tail does not measure the accuracy of the p50 signals that most plausibly drive the policy's rewrite decisions, and the noise-injection result is measured with the policy frozen. Please report calibration metrics for the p50 estimates (e.g., median absolute relative error or calibration slope by database and query shape) and state which quantile the policy actually consumes. Without this, the central 'estimate query risk' mechanism remains supported only by an inference-time sensitivity analysis, not by training-level evidence.","section":"Estimator behavior / Pre-execution query planning"}],"minor_comments":[{"comment":"The sentence 'configuration details appear in Section .' has an empty cross-reference; please fill in the section number or remove the dangling reference.","section":"RL objective and configuration"},{"comment":"The term 'Database-t' is used as an analysis name but never defined; please define it in the Metrics and inference paragraph.","section":"Experimental Setup, Metrics and inference"},{"comment":"Evidence Support is defined as a metric vocabulary item but is not reported in any results table; either report it in the main results or remove the definition to avoid dead notation.","section":"Problem Formulation"},{"comment":"The important caveat that bootstrap intervals condition on the three evaluated adapters appears in the caption and in Limitations, but it should also appear in the RQ1/RQ2 results text so that the headline intervals are not over-read as covering training randomness.","section":"Results, RQ1 and Table 3 caption"},{"comment":"The 7B coder experiment is reported in two sentences without model or training details; a short appendix entry describing the model, data split, and RL configuration would aid reproducibility.","section":"Results, Boundary conditions"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is a careful empirical package with honest limitations and strong reproducibility practices. The main substantive gap is the absence of a retrained placebo that isolates estimator information from the rewrite action; this is exactly what the authors identify in their Limitations section. If the authors can run the locked-suite shuffled-estimate condition, or convincingly reframe the abstract's mechanism claims to package-level, I would support acceptance. As written, the abstract's causal language is somewhat ahead of the training-level evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid, honest empirical paper about a real problem—agentic text-to-SQL agents blowing their context or DB budget before they get useful evidence. The genuinely new idea is treating the SQL query itself as a budget-control action: estimate the query's cost before execution, let the policy decide to rewrite it, and delegate hard enforcement to a separate runtime shield. That's meaningfully different from post-hoc compression (ContextBudget) or efficiency-shaping rewards (OTC-PO) because it changes what observation arrives, not just how it is displayed.\n\nThe experiments are well built. The primary XS/S gains over matched SFT are 3.4/3.6 percentage points with 4.5/5.0% fewer tokens, supported by 10k paired database bootstraps, three seeds, a locked retraining suite, and a 2x2 factorial that separates policy-visible planning from reward shaping. The training-level planning effect is 1.9/2.0 points with paired intervals [0.7,3.1]/[0.8,3.2]. They also test transfer to FINER-SQL and a 7B model, and they honestly report boundary conditions: the benefit reverses at the loosest budget, does not reduce database work, and costs extra tokens on FINER. The estimator's p95 coverage numbers are reported, and the shield works as intended.\n\nThe soft spots, in proportion. The stress-test about estimator accuracy is real: the locked suite compares the full estimate+rewrite package against a no-planning control, so it does not isolate whether accurate estimates are what drive the gain. The paper says exactly that in its limitations—\"does not isolate estimator calibration or rewriting separately; a retrained shuffled-estimate placebo would address part of this gap.\" So it is a missed experiment, not a hidden flaw. At inference time, 30% estimator noise costs 1.8 points, which shows the benefit depends on estimate reliability; the causal training-level dependency is just not nailed. A retrained shuffled-estimate control would have made the mechanism claim cleaner. Second, the bootstrap intervals condition on the three adapters rather than marginalizing all training randomness—again acknowledged. Third, two presentation issues: an empty \"Section .\" reference and no working artifact link in the text. These are minor but should be fixed.\n\nWho this is for: anyone building budget-constrained SQL agents or studying tool-use efficiency. It is a subfield contribution, not a field reorganization, but it is a well-measured one.\n\nRecommendation: send it to peer review. The central package-level result is supported, the authors are candid about limits, and the missing estimator-placebo is a reasonable revision request, not grounds for rejection. Ask them to run the retrained shuffled-estimate control if feasible, fix the dangling reference, and add the artifact link.","headline":"A well-built empirical paper on pre-execution cost-aware SQL rewriting; the package-level gain is real, the estimator-accuracy isolation gap is openly acknowledged, and it deserves peer review.","tokens_in":13585,"tokens_out":3020,"would_cite":true,"duration_ms":24382,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewriting SQL before execution lifts tight-budget agent success","keywords":["budget-aware agents","text-to-SQL","pre-execution query planning","reinforcement learning","runtime shielding","observation formation","query cost estimation","context budget"],"falsifier":"Run the full pipeline on held-out databases whose cardinality statistics differ sharply from the calibration databases, for example after multiplying base table sizes by ten or changing join selectivity, and check whether the tight-budget gains persist relative to matched supervised fine-tuning. A complementary check is a retrained shuffled-estimate placebo in which the policy sees permuted p50/p95 values while the shield still uses true estimates: if that retrained policy matches full BAP-SQL, the effect would not require trustworthy estimates.","tokens_in":12409,"feed_emoji":"🗄️","tokens_out":6432,"duration_ms":49256,"temperature":0.7,"pith_summary":"The paper argues that in agentic text-to-SQL the SQL query itself is a budget-control decision: it determines how many rows, tokens, and database operations the next observation will consume. BAP-SQL inserts a planning stage before execution in which the policy estimates a candidate query's cost, may rewrite it into a cheaper aggregate, and is stopped by an independent shield if the action would break a hard limit. Across a general 4B generator, a specialized SQL generator, and a 7B generator, the trained policy improves budgeted success on tight budgets, gaining 3.4 and 3.6 percentage points over matched supervised fine-tuning while using 4.5 and 5.0 percent fewer total tokens on the primary setting. The gain narrows as budgets loosen, reverses at the loosest setting, and does not reduce database work; the intended significance is that observation formation, not just post-hoc compression, is a controllable stage for tool-using agents.","feed_headline":"Estimate-then-rewrite SQL lifts tight-budget success","feed_subtitle":"A pre-execution planning loop beats matched SFT by 3.4–3.6 points while using about 5% fewer tokens.","key_machinery":"The load-bearing mechanism is a closed pre-execution loop built from three parts. First, an estimator combines EXPLAIN QUERY PLAN, a zero-row LIMIT 0 probe, and per-database catalog statistics to return calibrated p50 and p95 predictions for rows, result tokens, and database work. Second, the policy sees those estimates plus the remaining budget and may issue a Rewrite action that changes the query before execution; rewrites need not preserve the candidate's semantics, because success is judged against the user question and visible evidence. Third, an independent runtime shield configures VM-step, row, byte, and result-token caps before execution, so estimates expose risk while the shield guarantees feasibility. This loop is trained with a correctness-gated efficiency reward that gives zero reward to wrong or infeasible trajectories and ranks only the successful ones by evidence quality and tool-use count.","core_discovery":"BAP-SQL's central claim is that exposing pre-execution cost estimates to a reinforcement-learned policy and letting it rewrite SQL before execution makes tool-using agents more likely to answer correctly under tight context, query, result-token, and database-work budgets. The paper shows this through a trained policy that observes remaining budget and p50/p95 estimates of rows, result tokens, and database work; chooses among inspect, estimate, rewrite, execute, manage-evidence, answer, and abstain actions; and is governed by a runtime shield that makes hard budget breaches unreachable. The primary result is a 3.4/3.6 percentage-point budgeted-success gain over matched SFT at tight budgets with 4.5/5.0 percent fewer total tokens. The paper attributes the gain to policy-visible planning and budget-sensitive rescue: in a locked retraining suite, removing the estimate-and-rewrite package costs 1.9/2.0 points, and a cost-matched placebo with shuffled estimates confirms that the policy relies on the estimate content. The advantage is regime-specific: it concentrates on budget-sensitive failures, shrinks with stronger generators and larger budgets, and reverses at the loosest budget, where a correctness-only SQL-RL baseline is stronger.","pith_inferences":["If estimator calibration transfers across database distributions, the same plan-before-execution principle could generalize to other tool-using agents whose actions determine the size and content of the next observation, such as search and API-call agents.","The measured 30 percent estimator-noise penalty suggests a testable extension: training the policy under noisy or perturbed estimates could make it robust to calibration drift, which the current training does not explicitly address.","The reversal at the loosest budget hints that a budget-conditional gating policy, one that rewrites aggressively at tight budgets and executes broad queries when there is headroom, could extend the gains beyond the current fixed regime."],"forward_implications":["Under tight observation budgets, a pre-execution estimate-and-rewrite loop is a reliable way to improve answer quality without increasing total tokens on general small and SQL-specialized generators.","Removing the policy-visible estimates and the rewrite action costs roughly 2 percentage points of budgeted success, so the interface itself, not merely the reward, carries the effect.","Correctness-gated efficiency shaping and pre-execution planning are complementary: the reward helps more when the policy can plan, with a planning-by-reward interaction of about 1.8/1.4 points.","The saved tokens come from shorter trajectories and fewer turns, not from admitting fewer result tokens; database work does not decrease and may rise slightly.","The benefit is largest at tight budgets and disappears or reverses at the loosest budget, so the method is a targeted complement to stronger task policies rather than a general accuracy booster."],"supporting_citations":[{"why":"Supplies the evaluation benchmark, its database-level task split, and the official execution-accuracy anchor the paper reports alongside budgeted success.","marker":"Li et al. [2023]"},{"why":"Provides the correctness-gated multiplicative tool-efficiency reward form that BAP-SQL adapts to rank only successful trajectories.","marker":"Wang et al. [2025]"},{"why":"ContextBudget is the external post-hoc context-compression comparator that BAP-SQL is compared against on quality and token use.","marker":"Wu et al. [2026]"},{"why":"LoRA low-rank adapters are the training mechanism for all SFT and RL conditions.","marker":"Hu et al. [2021]"},{"why":"The specialized SQL checkpoint used to test transfer of the planning gains to a stronger, SQL-specialized backbone.","marker":"Hoang et al. [2026]"},{"why":"Online difficulty filtering selects the groups whose outcomes provide the reinforcement-learning signal.","marker":"Bae et al. [2026]"},{"why":"Group-relative policy optimization is the RL optimizer used to train the adapters.","marker":"Shao et al. [2024]"}],"fun_headline_variants":["BAP-SQL: Plan rewrites before execution for tight budgets","Estimate risk, rewrite SQL: budget-aware agentic planning","Budget-aware SQL agent: 3.4pt gain, 5% fewer tokens","Rewrite prior to run: SQL agents survive tight budgets","BAP-SQL shields hard limits: pre-execution rewrites win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes the pre-execution estimator's p50/p95 forecasts are accurate enough to guide rewriting; if those estimates do not transfer to a new database's data distribution, the planning advantage could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["BAP-SQL: Plan rewrites before execution for tight budgets","Estimate risk, rewrite SQL: budget-aware agentic planning","Budget-aware SQL agent: 3.4pt gain, 5% fewer tokens","Rewrite prior to run: SQL agents survive tight budgets","BAP-SQL shields hard limits: pre-execution rewrites win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":2042,"prompt_tokens":985,"completion_tokens":1057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":965}},"tokens_in":601,"tokens_out":1057,"duration_ms":9751,"temperature":1.0,"reasoning_tokens":965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:56:20.986540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on held-out databases whose cardinality statistics differ sharply from the calibration databases, for example after multiplying base table sizes by ten or changing join selectivity, and check whether the tight-budget gains persist relative to matched supervised fine-tuning. A complementary check is a retrained shuffled-estimate placebo in which the policy sees permuted p50/p95 values while the shield still uses true estimates: if that retrained policy matches full BAP-SQL, the effect would not require trustworthy estimates.","supporting_citations":[{"cited_title":"Boosting Small Language Models for Text-to-","cited_arxiv_id":null,"evidence_quote":"The specialized SQL checkpoint used to test transfer of the planning gains to a stronger, SQL-specialized backbone."}],"review_version":2}