{"id":"1e6bbff4-4810-4960-8cde-466568f264dc","arxiv_id":"2506.01273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An interactive reasoning agent that explores databases with live queries raises text-to-SQL execution accuracy from 44.8% to 56.5% and achieves 81.8% best-of-N coverage on BIRD.","lead":"This paper presents RAISE, an AI agent that writes SQL queries by exploring a database interactively, running test queries, and reflecting on results before answering. On a sample of the BIRD benchmark, this approach raised execution accuracy from 44.8% to 56.5% and reached 81.8% best-of-N candidate coverage without fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 compares RAISE's oracle Best-of-N on a 10% subset to CHASE-SQL's end-to-end accuracy on full dev; the 'rivaling 82.79%' claim is not apples-to-apples.","rationale":"The reader correctly identifies the underspecified 10% sample as a serious threat to the headline comparison. I agree, but the more decisive problem is that Table 1 does not compare the same quantity even before sampling is considered: RAISE's 81.8% is an oracle Best-of-N upper bound, whereas CHASE-SQL - Full's 82.79% is the end-to-end accuracy of a pipeline that includes a learned selector. Those metrics are ordered: any selector's accuracy is at most the oracle coverage of its candidate pool, so CHASE-SQL's true candidate-pool coverage is unknown and plausibly higher. The paper's own Table 1 footnote signals the asymmetry by drawing all other CHASE-SQL rows from published scaling graphs but not CHASE-SQL - Full. The within-sample controlled comparison (44.8% to 56.5%) is a reasonable experimental design and is not the primary weakness. However, the strongest central claim in the abstract and conclusion is the rivalry with 82.79%, and that claim rests on an apples-to-oranges comparison. The conditional verdict remains appropriate because a full-dev, same-metric evaluation could resolve the concern; no change in verdict is needed, but the paper should be revised to report identical metrics on identical evaluation sets.","tokens_in":7749,"tokens_out":10974,"duration_ms":124327,"concrete_test":"Retrieve the CHASE-SQL paper's metric definition for 82.79% and its best-of-N candidate coverage on the full BIRD dev set; then run RAISE's full pipeline on the full dev set and report both Best-of-N and selected accuracy. If CHASE-SQL's true candidate-pool coverage exceeds RAISE's full-dev Best-of-N, or if RAISE's selected accuracy is below 82.79, the Table 1 'rivaling' claim should be withdrawn or heavily qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §5.2, RAISE's headline number is defined as Best-of-N: a question counts as solved if any candidate in the pool matches the gold query under BIRD's execution check. That is an oracle upper bound, with no selection model applied. The comparator 'CHASE-SQL - Full' in Table 1 is 82.79%, which the CHASE-SQL paper reports as the end-to-end accuracy of its complete pipeline, including a trained candidate-selection model, over the full BIRD dev set. The Table 1 footnote even excludes CHASE-SQL - Full from the 'scaling graphs' used for the other CHASE-SQL rows. Therefore the claim that RAISE 'rivals the 82.79% achieved by the top-ranked published solution' compares two different quantities: an unselected upper bound on a 10% subset and a deployed system's accuracy on the full set. Since a selector's accuracy is a lower bound on the underlying pool's Best-of-N coverage, CHASE-SQL's true candidate-pool coverage is unknown and likely exceeds 82.79%. The conclusion in Section 6 that the agent 'can achieve a best-of-N accuracy of 81.8%... rivaling top-performing generation components' is thus not supported by the evidence as presented. Separately, RAISE's numbers are on an unspecified 10% sample with no error bars or seed, so all cross-pipeline comparisons inherit an additional sampling hazard.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAISE, a unified agentic text-to-SQL framework in which a reasoning LLM (DeepSeek-R1-Distill-Llama-70B) interacts with a database through tools such as reading table names/columns/documentation and executing arbitrary queries. The authors report a controlled comparison on a 10% stratified sample of the BIRD dev set, finding that adding dynamic database exploration raises Execution Accuracy from 44.8% to 56.5% compared with a static agent, and that accuracy improves with the depth of exploration up to about 15 executed commands. The full RAISE pipeline, which adds diverse candidate generation with o3-mini and Claude 3.7 Sonnet plus column-order post-processing, reaches a Best-of-N Execution Accuracy of 81.8% with 8 rounds of candidate generation; the paper compares this figure with the 82.79% reported for CHASE-SQL and concludes that RAISE rivals the top-ranked published solution.","tokens_in":8097,"tokens_out":4340,"duration_ms":45327,"significance":"The controlled comparison between a static agent and an interaction agent is a genuine strength: it isolates the effect of dynamic database exploration from query refinement and provides evidence for a new test-time-compute strategy in text-to-SQL. The finding that exploration depth correlates with accuracy, with a plateau, is concrete and falsifiable, and the authors share prompts and sample data. The main significance is conditional: if the 10% sample is representative and the effects replicate, the unification of schema linking and query generation in one agent is a valuable simplification. However, the headline comparison with CHASE-SQL is currently not apples-to-apples because it compares an oracle Best-of-N bound on a subset with an end-to-end accuracy on the full dev set, and the sampling uncertainty is not quantified.","major_comments":[{"comment":"The claim that RAISE's 81.8% 'rivals' CHASE-SQL's 82.79% compares two different quantities. RAISE's 81.8% is a Best-of-N oracle upper bound: a question counts as solved if any candidate matches under BIRD's execution check, with no selection model applied. CHASE-SQL - Full's 82.79% is the end-to-end accuracy of the complete pipeline, including its trained candidate-selection model, on the full BIRD dev set. Because any selector's accuracy is a lower bound on the underlying pool's Best-of-N coverage, CHASE-SQL's true candidate-pool coverage is unknown and could be substantially higher than 82.79%. The table footnote's decision to exclude CHASE-SQL - Full from the 'scaling graphs' used for the other rows confirms that the reporting protocols differ. The paper should either compare against CHASE-SQL's published candidate-pool Best-of-N values (if available) or explicitly state that the comparison is between RAISE's oracle upper bound on a sample and CHASE-SQL's deployed end-to-end accuracy, and should soften the abstract and conclusion accordingly.","section":"Section 5.2, Table 1, Abstract, Section 6"},{"comment":"All numerical results are measured on a single, unspecified 10% stratified sample of the BIRD dev set, with no description of the stratification variables, the sampling procedure, the random seed, or the number of runs. Consequently, the 44.8%-to-56.5% improvement and the 81.8% headline figure carry unknown sampling variability, and the cross-pipeline comparison in Table 1 inherits the risk that the sample is not representative of the full dev set. The authors should report full details of the sample construction, provide bootstrap confidence intervals or repeated-run variability, and ideally verify at least the main comparison on the full dev set.","section":"Section 4, Section 5.1, Table 1"},{"comment":"The abstract and Section 5.2 state that RAISE 'achieves a Best-of-N accuracy of 81.8% on the BIRD dataset.' Since the measurement was made on a 10% sample rather than the full BIRD dev set, this wording is an overstatement. The claim should be qualified as 'on a 10% stratified sample of the BIRD dev set,' and the authors should discuss evidence that the sample is representative, or rerun on the full set. This qualification is load-bearing because the headline comparison to CHASE-SQL depends on the sample's representativeness.","section":"Abstract and Section 5.2"}],"minor_comments":[{"comment":"References [9] and [10] appear to describe the same CHASE-SQL paper, and [10] is malformed ('Pourreza, M., al.: Chase-sql...'). Please merge or correct them.","section":"References [9] and [10]"},{"comment":"There is a typo in 'start-of-the-art' in the introduction; it should be 'state-of-the-art'.","section":"Section 1"},{"comment":"The Dropbox link is written as 'thisDropbox folder' in the text; please ensure the full URL is visible in the published version.","section":"Footnote 1"},{"comment":"The sentence 'The accuracy improves significantly from 3 to 15 commands' uses 'significantly' without a statistical test; consider rephrasing to 'markedly' or provide an error-bar or significance analysis.","section":"Section 5.1 and Figure 4"},{"comment":"The RAISE row reports Best-of-N accuracy but does not state the number of candidates N or the size of the sample used; please include these values so the comparison is interpretable.","section":"Table 1"},{"comment":"The post-processing step is described only for o3-mini; it is unclear whether Claude 3.7 Sonnet's outputs also undergo column-order post-processing. Please clarify the full pipeline.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The core controlled experiment is interesting and likely publishable after revision, but the headline 'rivaling 82.79%' claim is not supported in its current form because it conflates an oracle upper bound on a subset with an end-to-end metric on the full set. I do not see this as a fatal flaw: the paper can be repaired by reframing the comparison, adding sampling details and uncertainty estimates, and qualifying all claims about the full BIRD dataset. I would encourage the editor to request a revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The part of this paper worth taking seriously is the controlled experiment in Section 5.1. Isolating dynamic database exploration from refinement is a clean design, and the jump from 44.8% to 56.5% execution accuracy on a 10% sample of BIRD dev is internally plausible. That is a genuinely useful finding: it suggests that letting an LLM run arbitrary queries against the database buys more than iterative repair of the final SQL. The framing of scaling test-time compute as exploration depth is a nice reframing, and the authors are honest that Best-of-N is an upper bound, not a deployed accuracy.\n\nThe soft spots are real and load-bearing for the headline. First, the 10% stratified sample has no seed, no stratification details, and no error bars. One unrepeated run on an unspecified subset cannot support cross-pipeline comparisons. Second, the stress-test note is correct: Table 1 compares RAISE's Best-of-N pool coverage on that subset to CHASE-SQL's end-to-end accuracy on the full dev set. Those are different quantities. A selector's accuracy is a lower bound on pool coverage, so CHASE-SQL's pool coverage is unknown and likely higher than 82.79%. Calling RAISE's 81.8% \"rivaling\" that number is not supported. Third, the prompt and post-processing engineering in Section 3.3 is explicitly adapted to BIRD's evaluation preferences (column order, join patterns), so some of the gain may be benchmark-specific.\n\nNone of this kills the central controlled finding. Static 44.8% vs interactive 56.5% stands, and the scaling plot in Figure 4 looks consistent. But the paper as written overreaches in the abstract, Section 5.2, and the conclusion. The authors need to at least provide full-dev results (or clear error bars and a detailed sampling recipe), and they need to compare Best-of-N to Best-of-N with CHASE-SQL's pool coverage, or clearly label their number as an unselected upper bound.\n\nThis deserves peer review, not a desk rejection, because the controlled experiment is worth refereeing and the approach has legs. I would ask the authors to fix the comparison and the sample description before publication. I would not cite the 81.8% claim in my own work yet, but I would cite the exploration-vs-refinement result if it held up on the full dev set.","headline":"A solid controlled result about exploration helping text-to-SQL, but the headline comparison to CHASE-SQL is apples-to-oranges and the sample is underspecified.","tokens_in":8571,"tokens_out":2099,"would_cite":false,"duration_ms":23982,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single LLM agent that explores a database interactively before writing SQL can match the candidate coverage of complex multi-stage pipelines, reaching 81.8% Best-of-N on a BIRD sample without fine-tuning.","keywords":["Text-to-SQL","NL2SQL","Reasoning agents","Test-time compute","Database exploration","Execution accuracy","BIRD benchmark","LLM agent"],"falsifier":"Run the identical pipeline on the full BIRD development set, or across many 10% stratified seeds, and compare Best-of-N Execution Accuracy with 81.8%; if the sample result is an outlier, the headline comparison fails. The exploration effect itself is falsified if an interactive agent with query execution does not beat the static agent at a matched query-refinement budget.","tokens_in":7560,"feed_emoji":"🗄️","tokens_out":9728,"duration_ms":90739,"temperature":0.7,"pith_summary":"This paper tries to show that translating natural-language questions into SQL queries can be done by a single LLM-driven agent instead of a stack of specialized modules for schema linking, query generation, and refinement. The agent reads table names, columns, and documentation, and runs its own exploratory SQL queries, using the results to revise its understanding before writing a final query. On a stratified 10% sample of the BIRD development set, giving the agent this dynamic exploration raises Execution Accuracy from 44.8% to 56.5% with a reasoning LLM, no fine-tuning. When two additional LLMs read the same exploration transcript and produce diverse candidate queries, at least one candidate is correct 81.8% of the time in Best-of-N evaluation, close to the 82.79% reported for the top published pipeline. If this holds, scaling test-time compute by exploring the data more deeply is a practical substitute for much of the hand-built scaffolding.","feed_headline":"One agent that explores data lifts SQL accuracy from 44.8% to 56.5%","feed_subtitle":"The full pipeline reaches 81.8% Best-of-N on a BIRD sample, near the top published 82.79%, without fine-tuning.","key_machinery":"The load-bearing mechanism is a tool-augmented reasoning loop. The model writes its plan as free text and emits a special [EXECUTE] tag; the harness parses the tag, runs one of four tools (read table names, read table columns, read column documentation, run query), appends the result to the context, and lets generation resume. This loop lets the agent test hypotheses about the data—checking missing records, value formats, or documentation mismatches—before committing to SQL. Control tokens steer behavior: the trajectory begins with a forced statement to explore rather than assume, a 1400-token cap without tool calls pushes the model back to the database, and a 10,000-token cap forces a final answer. The paper's scaling result is that Execution Accuracy rises with the number of exploration commands included in the final prompt and plateaus after about 15 commands.","core_discovery":"The paper's central claim is that database exploration itself is a form of test-time compute worth scaling: a reasoning model equipped with four tools—read table names, read table columns, read column documentation, and run query—can do the work of schema linking, query generation, and refinement in one closed loop, and the more exploratory commands it executes, the better its final query. Evidence is reported on a stratified 10% sample of the BIRD development set: a static agent that can inspect the schema but not run queries reaches 42.9% Execution Accuracy, 44.8% with query refinement, while the interactive agent reaches 56.5%. The full pipeline, which adds two other models that independently write SQL from the same exploration transcript and a column-selection post-processing step, reaches 81.8% Best-of-N after 8 rounds of generation, compared with 82.79% for the strongest published candidate-generation pipeline, all without fine-tuning.","pith_inferences":["Editorial inference: if exploration depth is the causal lever, the same agent should improve even with smaller or faster models when given a larger exploration budget; a test would compare different reasoning models at matched command counts.","Editorial inference: the unseeded 10% sample makes the 81.8%-versus-82.79% comparison fragile; an obvious check is running the identical pipeline on the full dev set and on multiple random seeds to bound sample variance.","Editorial inference: exploration transcripts are a reusable byproduct; they could be mined as few-shot demonstrations for training a cheaper agent to explore more effectively."],"forward_implications":["A single reasoning agent with four database tools can absorb the roles of schema linking, generation, and refinement, so future text-to-SQL systems can drop the modular pipelines.","Accuracy improves with the number of executed exploration commands up to the observed plateau, meaning teams can trade inference budget for better data understanding instead of engineering more stages.","Candidate diversity is better obtained by pooling different LLMs over one shared exploration than by raising temperature alone.","Best-of-N coverage near 82% without fine-tuning means the remaining bottleneck is the selection of a single final query, not candidate generation."],"supporting_citations":[{"why":"Supplies the BIRD benchmark and development set on which all reported accuracies are measured.","marker":"[18]"},{"why":"Defines CHASE-SQL, whose 82.79% full-pipeline Best-of-N is the main comparison point.","marker":"[9]"},{"why":"Provides the CHESS baseline (71.0%) and represents the multi-stage schema-linking approach the agent is compared with.","marker":"[8]"},{"why":"Supplies the DeepSeek-R1-Distill-Llama-70B reasoning model that plays the role of the interaction agent.","marker":"[23]"},{"why":"Argues that scaling test-time compute can beat scaling model parameters, the premise reframed as exploration depth here.","marker":"[15]"},{"why":"Grounds the paper's use of reasoning traces and self-correction in a state-of-the-art reasoning LLM.","marker":"[14]"},{"why":"Is the Reasoning-SQL pipeline whose SFT and GRPO Best-of-N numbers (76.5%, 73.5%) appear in the comparison table.","marker":"[21]"}],"fun_headline_variants":["Single agent explores DB, lifts SQL accuracy from 44.8% to 56.5%","Exploration as test-time compute: SQL agent hits 56.5% on BIRD","One loop for schema, query, refine: 56.5% Execution Accuracy","Agentic SQL explorer nears top published without fine-tuning","Depth over pipelines: interactive SQL agent reaches 56.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that the unseeded 10% stratified sample of the BIRD development set has the same difficulty as the full set, so the 81.8% Best-of-N measured on the sample is directly comparable to the 82.79% reported for CHASE-SQL on the full set.","fun_headline_variants_meta":{"raw":{"variants":["Single agent explores DB, lifts SQL accuracy from 44.8% to 56.5%","Exploration as test-time compute: SQL agent hits 56.5% on BIRD","One loop for schema, query, refine: 56.5% Execution Accuracy","Agentic SQL explorer nears top published without fine-tuning","Depth over pipelines: interactive SQL agent reaches 56.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1742,"prompt_tokens":1016,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":632,"tokens_out":726,"duration_ms":7940,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:45:32.881938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline on the full BIRD development set, or across many 10% stratified seeds, and compare Best-of-N Execution Accuracy with 81.8%; if the sample result is an outlier, the headline comparison fails. The exploration effect itself is falsified if an interactive agent with query execution does not beat the static agent at a matched query-refinement budget.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems36(2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the BIRD benchmark and development set on which all reported accuracies are measured."},{"cited_title":"arXiv preprint (2024)","cited_arxiv_id":null,"evidence_quote":"Defines CHASE-SQL, whose 82.79% full-pipeline Best-of-N is the main comparison point."}],"review_version":1}