REVIEW 2 major objections 6 minor 33 references
SQL-RewriteBench: A Correctness-Gated, Full-Denominator Benchmark for Statement-Level SQL Rewriting [Experiment,Analysis & Benchmark]
T0 review · 2 major / 6 minor · reviewed 2026-07-13 · grok-4.5
Pith's one-line read No evaluated statement-level SQL rewrite method improves queries once every failure is counted and only correct rewrites can score.
desk verdict A real evaluation contract for SQL rewrite systems: full-denominator ledger + correctness-gated CGOQ, and every evaluated method scores negative. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Correctness-Gated Optimization Quality (CGOQ): a continuous score that combines measured runtime improvement with bounded structural simplification (via SCS) only after the rewritten query executes and satisfies the case-specific Checker Contract; all other outcomes contribute zero under full-denominator accounting (CGOQ@N).
What would settle it
Find a method that posts positive CGOQ@N on the frozen 180-case release while keeping high result consistency and low unsafe-rewrite rates under the published harness, or show that removing the opportunity enrichment (evaluating only base queries with the same scoring rules) flips the full-benchmark scores positive for representative systems.
Extended reading notes
Core claim
Across seven representative rewrite families evaluated as eight configurations on 180 executable instances, every full-benchmark CGOQ is negative. Existing methods often never produce a candidate, fail the case-specific Checker Contract, or return result-consistent rewrites that regress runtime or add no optimization value. The paper concludes that deployable statement-level SQL rewrite therefore requires broader front-end handling, result validation, and selective, benefit-aware rewrite policy.
Load-bearing premise
The load-bearing premise is that human-guided enrichment of real workload queries creates fair industrial rewrite targets without systematically manufacturing anti-patterns that current tools cannot handle while still preserving true result behavior.
Editorial extensions
If this is right
- Average speedup reported only over successful rewrites systematically overstates deployable value.
- Production rewrite pipelines need an explicit result-consistency gate before a replacement statement is accepted.
- A safe no-rewrite decision is better than shipping a correct-but-slower rewrite when verified value is negative.
- Rule engines need broader SQL front ends and cleaner executor interfaces before larger search or better rule ranking can help.
- Method design should optimize for full-denominator, correctness-gated scores rather than generation coverage alone.
Reading between the lines
- The same full-denominator, correctness-gated ledger would likely expose similar overclaiming in neighboring systems tasks such as automatic indexing or plan hinting.
- Because cases are opportunity-enriched, absolute CGOQ levels may look less negative on unenriched production SQL even if relative method rankings stay similar.
- Closing the loop with live result checks and measured runtime feedback may matter more than larger demonstration pools or longer prompts.
- Expanding beyond PostgreSQL to MySQL and Spark will probably first increase front-end and dialect failures before it improves performance scores.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. SQL-RewriteBench proposes a deployment-oriented benchmark for statement-level SQL rewriting. Unlike DBMS workloads, equivalence checkers, or dialect-translation suites, it evaluates the full SQL-in/SQL-out path under correctness gating and full-denominator accounting. The metric suite separates Source Acceptance, Generation, Execution Coverage, Result Consistency, UnsafeRewrite Rate, and speedup; it also defines SCS (a deterministic AST structural score) and CGOQ (optimization credit only after a case-specific Checker Contract). The artifact ships 180 Executable Case Packages in EQUIV, PERF, ROBUST, and DIALECT pools, each with input SQL, reference rewrite, schema, evidence, and opportunity documentation. Across eight configurations of seven academic and LLM-based methods, every full-benchmark CGOQ@N is negative: methods fail before rewrite, fail result checks, or emit correct but slower/no-better SQL. The authors conclude that deployable rewrite needs broader front ends, result validation, and benefit-aware decisions.
Significance. If the reported protocol and results hold, the paper fills a genuine evaluation gap: statement-level rewrite systems have not been scored as end-to-end deployable components with an explicit failure denominator. Strengths include a released artifact (case packages, harness, adapters), reference-pair certification (Table 10: all 180 pairs result-consistent; PERF GM 4.12×), terminal-outcome decomposition (Figure 1), N/A diagnosis for Calcite-backed engines (Table 13), formal-tool coverage audit (Section 7.4), and an 81-configuration CGOQ sensitivity grid (Table 8) under which all methods remain negative. The full-denominator framing and the distinction between safe inactivity and verified optimization value are useful methodological contributions for the community, independent of any single baseline ranking.
major comments (2)
- [Section 4.1; Abstract; Section 8] Section 4.1 (opportunity enrichment) and the abstract/conclusion claim that deployable rewrite “requires broader input handling…”. Enrichment is constrained not to alter join predicates, selection, grouping, multiplicities, NULL behavior, or observable ordering solely for runtime, and Table 10 certifies positive reference CGOQ. Still, the corpus is intentionally opportunity-enriched (Table 2: redundant derived tables 51.1%, nested blocks 36.7%, etc.). The negative CGOQ@N result is internally well-supported on this corpus, but the leap from “fails on these 180 enriched instances” to a general deployment requirement needs a clearer external-validity bound: either a small non-enriched control slice, or an explicit statement that the claim is capability-diagnostic rather than prevalence-based (Section 8 already gestures at this; the abstract should match that scope).
- [Section 5.4; Table 11; Figure 1] Section 5.4 and Table 11: CGOQ@N assigns zero to No-Rewrite / N/A / unsafe / non-executable outcomes, so a near-inactive system (LLM-R2-plan: Generation Rate 3.9%, CGOQ@N −0.55) ranks above methods that attempt rewrites and often regress (e.g., GPT-5.5 −29.92). The paper correctly insists that CGOQ@N be read with Generation Rate and terminal statuses (Figure 1), and states that a full No-Rewrite baseline is an intentional status-quo reference. For a leaderboard-facing artifact, the primary ranking rule should be stated more operationally (e.g., multi-key order or a constrained CGOQ that does not reward pure inactivity over verified mild gains) so that “least negative by doing almost nothing” cannot be misread as best deployable method.
minor comments (6)
- [Abstract; Section 7.1] Abstract and Introduction say “seven representative… methods” while experiments report eight configurations (three Direct-LLM models plus five systems). Align the count wording throughout.
- [Table 5; Table 11] Table 5 lists GM Speedup Ratio over timed result-consistent rewrites; Table 11 reports values well below 1× for most methods. A one-sentence reminder in the table caption that GM is conditional on |Tm| (and that low |Tm| can make the ratio fragile) would help readers.
- [Section 5.2; Table 7] Section 5.2: SCS–runtime Spearman ρ = 0.389 overall but near zero in ROBUST (0.044). The text already treats SCS as stratification, not a cost model; a short caution in the SCS definition paragraph would prevent over-interpretation of simplification credit in ROBUST.
- [Figure 2] Figure 2 cell labels and color scale are dense; ensure the camera-ready version has readable numeric labels and a colorbar that distinguishes positive PERF cells from large negative EQUIV/ROBUST cells.
- [Section 2] Related Work could briefly situate middleware rewriters (QueryBooster, SlabCity) against the SQL-in/SQL-out contract so readers see why those systems’ own evaluations are not substitutes for this denominator.
- [Throughout; Section 7.2] Minor consistency: “UnsafeRewrite Rate” vs “Unsafe Rewrite” spacing; “No-Rewrite Decision” hyphenation; and “postgres:16” / PostgreSQL 16.14 environment details could be moved to a short reproducibility appendix pointer in the main text.
Circularity Check
No significant circularity: negative CGOQ@N is measured under a fixed, reference-independent protocol whose scoring parameters are not fitted to force method failure.
full rationale
SQL-RewriteBench is an evaluation/benchmark paper, not a first-principles derivation. The load-bearing claim—that every evaluated method has negative full-denominator CGOQ@N—is an empirical ledger outcome (Tables 11–15, Figure 1), not a quantity algebraically forced by the metric definitions. Methods are scored only after the case-specific Checker Contract; they never see Reference Rewrites; and ranking is explicitly reference-independent (Section 5.1). SCS caps τf are frozen Q0.95 normalizers on the input corpus (Eq. 1), not parameters fitted to method outcomes; equal-weight SCS and an 81-config CGOQ sensitivity grid leave method order and the negative sign of all method CGOQ@N unchanged while Reference Rewrites stay strongly positive (Tables 7–8, 10). Pool Suitability Scores are construction evidence only and do not enter ranking (Section 4.2). There is no self-definitional loop (X defined as Y then “predicted”), no fitted-input-called-prediction, no load-bearing self-citation uniqueness theorem, and no renaming of a known result as a forced derivation. Opportunity enrichment is a corpus-validity concern, not circular scoring: the claim is scoped to measured behavior on the released instances under the stated protocol. Honest non-finding: the evaluation chain is self-contained against its own fixed contracts and measured runtimes.
Assumptions & free parameters
free parameters (7)
- CGOQ runtime-neutral band δ
- CGOQ simplification weight λ
- CGOQ runtime scale τ (via reference speedup)
- SCS simplification thresholds (θC, cC)
- SCS feature caps τf
- Pool admission thresholds
- PERF reference speedup floor
assumptions (5)
- domain assumption Case-specific Checker Contract (execution Result Consistency under bag/multiset semantics, declared dialect, packaged schema/data) is a sufficient operational Correctness Gate for optimization credit.
- domain assumption Opportunity enrichment preserves observable semantics while exposing realistic industrial rewrite objectives.
- ad hoc to paper Equal weighting of six SCS feature groups is an acceptable structural policy for stratification and simplification credit.
- domain assumption Median of five post-warmup runs on the stated PostgreSQL/Docker setup is adequate for speedup and CGOQ comparisons in this study.
- ad hoc to paper Top-1 final rewrite only; no credit for unselected candidates or oracle selection after outcomes are known.
invented entities (4)
-
Static SQL Complexity Score (SCS)
independent evidence
-
Correctness-Gated Optimization Quality (CGOQ)
independent evidence
-
Checker Contract / Executable Case Package
independent evidence
-
Pool Suitability Score
independent evidence
Cite this review
Pith. "Pith review of SQL-RewriteBench: A Correctness-Gated, Full-Denominator Benchmark for Statement-Level SQL Rewriting [Experiment,Analysis & Benchmark]." pith.science (2026). https://pith.science/paper/PEZ2CHTL
@misc{pith2026260709251,
author = {Pith},
title = {Pith review of: SQL-RewriteBench: A Correctness-Gated, Full-Denominator Benchmark for Statement-Level SQL Rewriting [Experiment,Analysis & Benchmark]},
year = {2026},
howpublished = {\url{https://pith.science/paper/PEZ2CHTL}},
note = {Machine review of arXiv:2607.09251}
}
read the original abstract
Statement-level SQL rewriting can improve query performance and maintainability without changing the DBMS kernel, but existing benchmarks do not evaluate rewrite methods as deployable systems. They typically focus on DBMS performance, rule regression, query equivalence, or dialect translation, while missing the full path from accepting an input query to producing an executable, result-consistent, and operationally useful rewrite. We present SQL-RewriteBench, a benchmark for statement-level SQL rewriting that applies correctness gating and full-denominator accounting. Its metric suite explicitly separates Source Acceptance, Generation Rate, Execution Coverage, Result Consistency, UnsafeRewrite Rate, and speedup distribution. It also defines SCS, a deterministic index of static SQL structure, and CGOQ, a correctness-gated optimization-quality score that gives optimization credit only after the case-specific Checker Contract is satisfied. CGOQ combines runtime improvement with structural simplification through a continuous scoring function, making it suitable for deployment-oriented rewrite assessment. As an artifact, SQL-RewriteBench provides 180 executable Benchmark Instances organized into EQUIV, PERF, ROBUST, and DIALECT pools, each packaged with SQL, schema metadata, provenance, evidence, and rewrite-opportunity documentation. Across seven representative academic and LLM-based methods, every full-benchmark CGOQ is negative. Existing methods often fail before rewriting, fail result checks, or return correct rewrites that are slower or no better than the input. These results show that deployable SQL rewrite requires broader input handling, result validation, and benefit-aware rewrite decisions.
Figures
Reference graph
Works this paper leans on
-
[1]
Qiushi Bai, Sadeem Alsudais, and Chen Li. 2023. QueryBooster: Improving SQL Performance Using Middleware Services for Human-Centered Query Rewriting.Proceedings of the VLDB Endowment16, 11 (2023), 2911–2924
2023
-
[2]
Mior, and Daniel Lemire
Edmon Begoli, Jesús Camacho-Rodríguez, Julian Hyde, Michael J. Mior, and Daniel Lemire
-
[3]
InProceedings of the 2018 International Conference on Management of Data
Apache Calcite: A Foundational Framework for Optimized Query Processing Over Het- erogeneous Data Sources. InProceedings of the 2018 International Conference on Management of Data. 221–230. 20
2018
-
[4]
Shumo Chu, Brendan Murphy, Jared Roesch, Alvin Cheung, and Dan Suciu. 2018. Axiomatic Foundations and Algorithms for Deciding Semantic Equivalences of SQL Queries.Proceedings of the VLDB Endowment11, 11 (2018), 1482–1495. https://doi.org/10.14778/3236187. 3236200
doi:10.14778/3236187 2018
-
[5]
Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017. Cosette: An Automated Prover for SQL. InProceedings of the 8th Biennial Conference on Innovative Data Systems Research
2017
-
[6]
Shumo Chu, Konstantin Weitz, Alvin Cheung, and Dan Suciu. 2017. HoTTSQL: Proving Query Rewrites with Univalent SQL Semantics. InProceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages. 510–524. https://doi.org/10.1145/ 3009837.3009838
arXiv 2017
-
[7]
Djellel Eddine Difallah, Andrew Pavlo, Carlo Curino, and Philippe Cudré-Mauroux. 2013. OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases. InProceedings of the VLDB Endowment, Vol. 7. 277–288
2013
-
[8]
Bailu Ding, Surajit Chaudhuri, Johannes Gehrke, and Vivek Narasayya. 2021. DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems.Proceedings of the VLDB Endowment14, 13 (2021), 3376–3388. https://doi.org/10.14778/3484224.3484234
Show all 33 references
-
[9]
Haoran Ding, Zhaoguo Wang, Yicun Yang, Dexin Zhang, Zhenglin Xu, Haibo Chen, Ruzica Piskac, and Jinyang Li. 2023. Proving Query Equivalence Using Linear Integer Arithmetic. Proceedings of the ACM on Management of Data1, 4, Article 227 (2023). https://doi.org/ 10.1145/3626768
2023 doi
-
[10]
Rui Dong, Jie Liu, Yuxuan Zhu, Cong Yan, Barzan Mozafari, and Xinyu Wang. 2023. SlabCity: Whole-Query Optimization Using Program Synthesis.Proceedings of the VLDB Endowment 16, 11 (2023), 3151–3164
2023
-
[11]
Goetz Graefe. 1995. The Cascades Framework for Query Optimization.IEEE Data Engineering Bulletin18, 3 (1995), 19–29
1995
-
[12]
Goetz Graefe and William J. McKenna. 1993. The Volcano Optimizer Generator: Extensibility and Efficient Search. InProceedings of the 9th International Conference on Data Engineering. 209–218
1993
-
[13]
Yang He, Pinhan Zhao, Xinyu Wang, and Yuepeng Wang. 2024. VeriEQL: Bounded Equivalence Verification for Complex SQL Queries with Integrity Constraints. arXiv:2403.03193
2024 arXiv
-
[14]
Zhe Jiang, Zhaoguo Wang, Haoning Lan, Chuzhe Tang, Haoran Ding, Lefeng Wang, Songyun Zou, Zhuoran Wei, Yongcun Liu, Xiang Yu, Yang Ren, Guoliang Li, and Haibo Chen. 2025. GRewriter: Practical Query Rewriting with Automatic Rule Set Expansion in GaussDB. Proceedings of the VLDB...
2025
-
[15]
Viktor Leis, Andrey Gubichev, Atul Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann. 2015. How Good Are Query Optimizers, Really?Proceedings of the VLDB Endowment9, 3 (2015), 204–215
2015
-
[16]
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2023. Can LLM Already Serve as a Database Interface? A Big 21 Bench for Large-Scale Database Grounded Text-to-SQLs. InAdvances in Neural Information Processin...
2023
-
[17]
Zhaodonghui Li, Haitao Yuan, Huiming Wang, Gao Cong, and Lidong Bing. 2024. LLM-R2: A Large Language Model Enhanced Rule-Based Rewrite System for Boosting Query Efficiency. Proceedings of the VLDB Endowment18, 1 (2024), 53–65. https://doi.org/10.14778/ 3696435.3696440
2024
-
[18]
Jie Liu and Barzan Mozafari. 2024. Query Rewriting via Large Language Models. arXiv:2403.09060
2024
-
[19]
Limin Ma, Ken Pu, and Ying Zhu. 2024. Evaluating LLMs for Text-to-SQL Generation With Complex SQL Workload. arXiv:2407.19517
2024 arXiv
-
[20]
Hellerstein, and Waqar Hasan
Hamid Pirahesh, Joseph M. Hellerstein, and Waqar Hasan. 1992. Extensible/Rule Based Query Rewrite Optimization in Starburst. InProceedings of the 1992 ACM SIGMOD International Conference on Management of Data. 39–48
1992
-
[21]
Selinger, Morton M
Patricia G. Selinger, Morton M. Astrahan, Donald D. Chamberlin, Raymond A. Lorie, and Thomas G. Price. 1979. Access Path Selection in a Relational Database Management System. InProceedings of the 1979 ACM SIGMOD International Conference on Management of Data. 23–34
1979
-
[22]
Yuyang Song, Hanxu Yan, Jiale Lao, Yibo Wang, Yufei Li, Yuanchun Zhou, Jianguo Wang, and Mingjie Tang. 2025. QUITE: A Query Rewrite System Beyond Rules with LLM Agents. arXiv:2506.07675
2025
-
[23]
SQLStorm Authors. 2025. SQLStorm: A Complex SQL Workload for Database-System Evaluation. Project artifact, https://github.com/SQL-Storm/SQLStorm
2025
-
[24]
Zhaoyan Sun, Xuanhe Zhou, Guoliang Li, Xiang Yu, Jianhua Feng, and Yong Zhang. 2025. R-Bot: An LLM-Based Query Rewrite System.Proceedings of the VLDB Endowment18, 12 (2025), 5031–5044.https://doi.org/10.14778/3750601.3750625
2025 doi
-
[25]
Transaction Processing Performance Council. 2024. TPC-DS Benchmark Specification. https://www.tpc.org/tpcds/
2024
-
[26]
Transaction Processing Performance Council. 2024. TPC-H Benchmark Specification. https://www.tpc.org/tpch/
2024
-
[27]
Shuxian Wang, Sicheng Pan, and Alvin Cheung. 2024. QED: A Powerful Query Equivalence Decider for SQL.Proceedings of the VLDB Endowment17, 11 (2024), 3602–3614. https: //doi.org/10.14778/3681954.3682024
2024 doi
-
[28]
Zhaoguo Wang, Zhou Zhou, Yicun Yang, Haoran Ding, Gansen Hu, Ding Ding, Chuzhe Tang, Haibo Chen, and Jinyang Li. 2022. WeTune: Automatic Discovery and Verification of Query Rewrite Rules. InProceedings of the 2022 International Conference on Management of Data. 94–107.https://...
2022 doi
-
[29]
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and 22 Text-to-SQL Task. I...
2018
-
[30]
Navathe, William Harris, and Jinpeng Wu
Qi Zhou, Joy Arulraj, Shamkant B. Navathe, William Harris, and Jinpeng Wu. 2022. SPES: A Symbolic Approach to Proving Query Equivalence Under Bag Semantics. InProceedings of the 38th IEEE International Conference on Data Engineering. 2735–2748. https://doi.org/10. 1109/ICDE537...
2022
-
[31]
Navathe, William Harris, and Dong Xu
Qi Zhou, Joy Arulraj, Shamkant B. Navathe, William Harris, and Dong Xu. 2019. Automated Verification of Query Equivalence Using Satisfiability Modulo Theories.Proceedings of the VLDB Endowment12, 11 (2019), 1276–1288
2019
-
[32]
Wei Zhou, Guoliang Li, Haoyu Wang, Yuxing Han, Xufei Wu, Fan Wu, and Xuanhe Zhou. 2025. PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation. NeurIPS 2025 project website,https://code4db.github.io/parrot-bench/
2025
-
[33]
Xuanhe Zhou, Guoliang Li, Chengliang Chai, and Jianhua Feng. 2022. A Learned Query Rewrite System Using Monte Carlo Tree Search.Proceedings of the VLDB Endowment15, 1 (2022), 46–58.https://doi.org/10.14778/3485450.3485456 23
2022 doi
Reviewed July 13, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.