{"id":"9e7e128e-8d72-4082-af38-8f3707b69bc8","arxiv_id":"2607.09251","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":7,"one_line_summary":"A correctness-gated, full-denominator SQL rewrite benchmark finds that seven representative rewrite methods all deliver negative end-to-end optimization quality on 180 cases.","lead":"SQL-RewriteBench is a 180-case benchmark that scores SQL rewrite tools only after they accept, execute, and preserve results, not just when a rewrite looks faster. It shows that seven academic and LLM rewrite systems all score negative on the full set, mostly by failing early, breaking results, or making queries slower.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's enrichment caveat; the negative CGOQ@N claim is internally well-supported by the full-denominator ledger.","rationale":"The paper's strongest claim is a negative full-denominator result under an explicit deployability contract, not a claim of production prevalence or cross-engine universality. The enrichment step is the clearest external-validity soft spot, and the reader correctly flags it; the manuscript already documents constraints, atom frequencies (Table 2), pool suitability scoring, and reference certification that keep the claim internally coherent. PostgreSQL-only executable track and workstation-tied runtimes are acknowledged limitations (Section 8) that justify CONDITIONAL rather than unconditional acceptance, but they do not falsify the measured CGOQ@N < 0 on the released artifact. Sensitivity of CGOQ parameters leaves all methods negative; formal verifiers cannot serve as the gate yet are audited. Therefore no stronger load-bearing attack is warranted, and the reader's CONDITIONAL / HIGH-confidence verdict stands without adjustment.","tokens_in":19401,"tokens_out":556,"duration_ms":7082,"concrete_test":"Re-run the full Evaluation Harness on the frozen 180 packages after replacing each Benchmark Input Query with its provenance/base_query.sql (when present) while keeping the same Checker Contract and Reference Rewrite; if any method's CGOQ@N becomes positive or the terminal-status distribution collapses to near-zero N/A/Unsafe, enrichment was load-bearing for the negative claim; otherwise the claim holds on the unenriched Base Queries as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (DBA-guided opportunity enrichment may bias toward anti-patterns that current Calcite/LLM front ends cannot handle) is real but already scoped by the paper: enrichment is constrained not to alter join predicates, selection, grouping, multiplicities, NULL behavior, or observable ordering solely for runtime (Section 4.1); every pair is certified executable and result-consistent with positive reference CGOQ (Table 10); formal-tool coverage gaps and N/A diagnoses are reported explicitly (Sections 7.4, 7.7–7.8). The central claim is not that the 180 cases are a prevalence sample of production SQL, but that under a correctness-gated, full-denominator protocol on opportunity-enriched yet semantics-preserving instances, every evaluated method scores CGOQ@N < 0. That claim is directly evidenced by Tables 11–15, Figure 1, and the 81-config sensitivity grid (Table 8). No additional internal inconsistency or hidden assumption undermines the reported negative result.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"SQL-RewriteBench proposes a deployment-oriented benchmark for statement-level SQL rewriting. Unlike DBMS workloads, equivalence checkers, or dialect-translation suites, it evaluates the full SQL-in/SQL-out path under correctness gating and full-denominator accounting. The metric suite separates Source Acceptance, Generation, Execution Coverage, Result Consistency, UnsafeRewrite Rate, and speedup; it also defines SCS (a deterministic AST structural score) and CGOQ (optimization credit only after a case-specific Checker Contract). The artifact ships 180 Executable Case Packages in EQUIV, PERF, ROBUST, and DIALECT pools, each with input SQL, reference rewrite, schema, evidence, and opportunity documentation. Across eight configurations of seven academic and LLM-based methods, every full-benchmark CGOQ@N is negative: methods fail before rewrite, fail result checks, or emit correct but slower/no-better SQL. The authors conclude that deployable rewrite needs broader front ends, result validation, and benefit-aware decisions.","tokens_in":19795,"tokens_out":1370,"duration_ms":25607,"significance":"If the reported protocol and results hold, the paper fills a genuine evaluation gap: statement-level rewrite systems have not been scored as end-to-end deployable components with an explicit failure denominator. Strengths include a released artifact (case packages, harness, adapters), reference-pair certification (Table 10: all 180 pairs result-consistent; PERF GM 4.12×), terminal-outcome decomposition (Figure 1), N/A diagnosis for Calcite-backed engines (Table 13), formal-tool coverage audit (Section 7.4), and an 81-configuration CGOQ sensitivity grid (Table 8) under which all methods remain negative. The full-denominator framing and the distinction between safe inactivity and verified optimization value are useful methodological contributions for the community, independent of any single baseline ranking.","major_comments":[{"comment":"Section 4.1 (opportunity enrichment) and the abstract/conclusion claim that deployable rewrite “requires broader input handling…”. Enrichment is constrained not to alter join predicates, selection, grouping, multiplicities, NULL behavior, or observable ordering solely for runtime, and Table 10 certifies positive reference CGOQ. Still, the corpus is intentionally opportunity-enriched (Table 2: redundant derived tables 51.1%, nested blocks 36.7%, etc.). The negative CGOQ@N result is internally well-supported on this corpus, but the leap from “fails on these 180 enriched instances” to a general deployment requirement needs a clearer external-validity bound: either a small non-enriched control slice, or an explicit statement that the claim is capability-diagnostic rather than prevalence-based (Section 8 already gestures at this; the abstract should match that scope).","section":"Section 4.1; Abstract; Section 8"},{"comment":"Section 5.4 and Table 11: CGOQ@N assigns zero to No-Rewrite / N/A / unsafe / non-executable outcomes, so a near-inactive system (LLM-R2-plan: Generation Rate 3.9%, CGOQ@N −0.55) ranks above methods that attempt rewrites and often regress (e.g., GPT-5.5 −29.92). The paper correctly insists that CGOQ@N be read with Generation Rate and terminal statuses (Figure 1), and states that a full No-Rewrite baseline is an intentional status-quo reference. For a leaderboard-facing artifact, the primary ranking rule should be stated more operationally (e.g., multi-key order or a constrained CGOQ that does not reward pure inactivity over verified mild gains) so that “least negative by doing almost nothing” cannot be misread as best deployable method.","section":"Section 5.4; Table 11; Figure 1"}],"minor_comments":[{"comment":"Abstract and Introduction say “seven representative… methods” while experiments report eight configurations (three Direct-LLM models plus five systems). Align the count wording throughout.","section":"Abstract; Section 7.1"},{"comment":"Table 5 lists GM Speedup Ratio over timed result-consistent rewrites; Table 11 reports values well below 1× for most methods. A one-sentence reminder in the table caption that GM is conditional on |Tm| (and that low |Tm| can make the ratio fragile) would help readers.","section":"Table 5; Table 11"},{"comment":"Section 5.2: SCS–runtime Spearman ρ = 0.389 overall but near zero in ROBUST (0.044). The text already treats SCS as stratification, not a cost model; a short caution in the SCS definition paragraph would prevent over-interpretation of simplification credit in ROBUST.","section":"Section 5.2; Table 7"},{"comment":"Figure 2 cell labels and color scale are dense; ensure the camera-ready version has readable numeric labels and a colorbar that distinguishes positive PERF cells from large negative EQUIV/ROBUST cells.","section":"Figure 2"},{"comment":"Related Work could briefly situate middleware rewriters (QueryBooster, SlabCity) against the SQL-in/SQL-out contract so readers see why those systems’ own evaluations are not substitutes for this denominator.","section":"Section 2"},{"comment":"Minor consistency: “UnsafeRewrite Rate” vs “Unsafe Rewrite” spacing; “No-Rewrite Decision” hyphenation; and “postgres:16” / PostgreSQL 16.14 environment details could be moved to a short reproducibility appendix pointer in the main text.","section":"Throughout; Section 7.2"}],"recommendation":"minor_revision","confidential_remarks":"The central negative-CGOQ claim is directly evidenced and the artifact appears unusually complete for a benchmark paper. I would not block on enrichment if the authors tighten abstract/conclusion scope language and clarify leaderboard multi-metric ranking. Fit for an Experiment/Analysis & Benchmark track is good; novelty is methodological and empirical rather than a new rewrite algorithm, which is appropriate for the stated contribution type."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: this is not another TPC-style runtime table. It is a deployable SQL-in/SQL-out evaluation contract—acceptance, generation, execution, result consistency, unsafe rewrites, and only then optimization credit—and under that contract every full-benchmark CGOQ@N is negative.\n\nWhat is actually new is the package, not a rewrite algorithm. Full-denominator stage rates, a case-specific Checker Contract, SCS as a frozen structural index, and CGOQ that withholds optimization credit until the gate passes. The 180 Executable Case Packages (EQUIV/PERF/ROBUST/DIALECT) with withheld references, evidence, and a public harness make the claim inspectable. Reference pairs are certified result-consistent with real recoverable value (PERF GM ~4.12×). The multi-method ledger (Direct LLMs, LearnedRewrite, LLM-R2 modes, R-Bot, QUITE) is diagnostic: front-end/N/A failures, unsafe generation, and correct-but-slower rewrites are separated instead of collapsed into “success rate.” Sensitivity on CGOQ parameters and SCS weights does not flip the sign. Citations cover the right neighborhoods (workloads, equivalence tools, rewrite systems) without pretending formal equivalence is the operational gate.\n\nSoft spots, in proportion: DBA-guided opportunity enrichment is the load-bearing construction choice. The paper constrains it (no semantic surgery just to create slowdowns) and certifies pairs, but the corpus is still opportunity-enriched, not a prevalence sample of production SQL. Executable track is PostgreSQL-only; runtimes are workstation/Docker-tied. Those limit how far you should treat CGOQ@N as a universal ranking, not whether the negative full-denominator result holds on this artifact. Formal tools (SQLSolver/VeriEQL) are correctly demoted to supplementary evidence after coverage gaps.\n\nThis is for people building rewrite middleware, Calcite-style rule engines, or LLM rewrite agents who still report speedups only on survivors. It deserves a serious referee. I would bring it to reading group and cite the metric contract when evaluating rewrite systems. Send it to peer review; ask for clearer external-validity language on enrichment and dialect scope, not a rewrite of the core claim.","headline":"A real evaluation contract for SQL rewrite systems: full-denominator ledger + correctness-gated CGOQ, and every evaluated method scores negative.","tokens_in":20427,"tokens_out":563,"would_cite":true,"duration_ms":13488,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"No evaluated statement-level SQL rewrite method improves queries once every failure is counted and only correct rewrites can score.","keywords":["SQL rewriting","query optimization","correctness gating","benchmark","CGOQ","result consistency","full-denominator evaluation","statement-level rewrite"],"falsifier":"Find a method that posts positive CGOQ@N on the frozen 180-case release while keeping high result consistency and low unsafe-rewrite rates under the published harness, or show that removing the opportunity enrichment (evaluating only base queries with the same scoring rules) flips the full-benchmark scores positive for representative systems.","tokens_in":20311,"feed_emoji":"🗄️","tokens_out":899,"duration_ms":22659,"temperature":0.7,"pith_summary":"Statement-level SQL rewriting promises faster or cleaner queries without changing the database engine. Prior benchmarks usually measure only successful cases, plan quality, or formal fragments, so they can hide parse failures, no-rewrite choices, crashes, and result-changing outputs. SQL-RewriteBench instead treats rewrite as a full SQL-in/SQL-out deployment path: accept the input, optionally emit one replacement statement, run it, check a case-specific result contract, then score runtime and structure. On 180 packaged instances spanning equivalence boundaries, performance anti-patterns, structural hardness, and dialect-sensitive forms, every representative academic and language-model configuration scores negative on the full-denominator quality index. Systems fail before rewriting, emit unsafe SQL, or return correct rewrites that are slower or no better than the input. The practical point is that deployable rewrite needs broader input handling, result validation, and benefit-aware decisions, not only higher generation rates on selected wins.","feed_headline":"Every tested SQL rewrite method scores negative","feed_subtitle":"A 180-case ledger that counts failures shows rewrites often break, slow, or never run","key_machinery":"Correctness-Gated Optimization Quality (CGOQ): a continuous score that combines measured runtime improvement with bounded structural simplification (via SCS) only after the rewritten query executes and satisfies the case-specific Checker Contract; all other outcomes contribute zero under full-denominator accounting (CGOQ@N).","core_discovery":"Across seven representative rewrite families evaluated as eight configurations on 180 executable instances, every full-benchmark CGOQ is negative. Existing methods often never produce a candidate, fail the case-specific Checker Contract, or return result-consistent rewrites that regress runtime or add no optimization value. The paper concludes that deployable statement-level SQL rewrite therefore requires broader front-end handling, result validation, and selective, benefit-aware rewrite policy.","pith_inferences":["The same full-denominator, correctness-gated ledger would likely expose similar overclaiming in neighboring systems tasks such as automatic indexing or plan hinting.","Because cases are opportunity-enriched, absolute CGOQ levels may look less negative on unenriched production SQL even if relative method rankings stay similar.","Closing the loop with live result checks and measured runtime feedback may matter more than larger demonstration pools or longer prompts.","Expanding beyond PostgreSQL to MySQL and Spark will probably first increase front-end and dialect failures before it improves performance scores."],"forward_implications":["Average speedup reported only over successful rewrites systematically overstates deployable value.","Production rewrite pipelines need an explicit result-consistency gate before a replacement statement is accepted.","A safe no-rewrite decision is better than shipping a correct-but-slower rewrite when verified value is negative.","Rule engines need broader SQL front ends and cleaner executor interfaces before larger search or better rule ranking can help.","Method design should optimize for full-denominator, correctness-gated scores rather than generation coverage alone."],"fun_headline_variants":["All seven SQL rewrite methods post negative CGOQ scores","No tested SQL rewriter clears the full-denominator bar","180-case ledger: rewrites fail, regress, or never run","Correctness-gated bench finds every method CGOQ-negative","Existing rewriters miss deployable statement-level gains"],"cache_read_input_tokens":12416,"weakest_assumption_plain":"The load-bearing premise is that human-guided enrichment of real workload queries creates fair industrial rewrite targets without systematically manufacturing anti-patterns that current tools cannot handle while still preserving true result behavior.","fun_headline_variants_meta":{"raw":{"variants":["All seven SQL rewrite methods post negative CGOQ scores","No tested SQL rewriter clears the full-denominator bar","180-case ledger: rewrites fail, regress, or never run","Correctness-gated bench finds every method CGOQ-negative","Existing rewriters miss deployable statement-level gains"]},"model":"grok-4.5","effort":"low","cost_usd":0.00311,"raw_usage":{"total_tokens":1153,"prompt_tokens":856,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":31100000,"prompt_tokens_details":{"text_tokens":856,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":230,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":856,"tokens_out":67,"duration_ms":4052,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T04:23:06.641383+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Find a method that posts positive CGOQ@N on the frozen 180-case release while keeping high result consistency and low unsafe-rewrite rates under the published harness, or show that removing the opportunity enrichment (evaluating only base queries with the same scoring rules) flips the full-benchmark scores positive for representative systems.","supporting_citations":[],"review_version":1}