REVIEW 3 major objections 5 minor 1 cited by
Bootstrapping Learned Cost Models with Synthetic SQL Queries
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that an LLM-driven synthetic SQL generator can train a learned cost model more accurately using 45% fewer queries than a mechanical generator, and can cut cross-engine routing time by 10%.
desk verdict A plausible LLM-based synthetic query generation pipeline, but the central 45% claim rests on one unmatched, unrepeated comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the coverage-steered synthetic query generation loop. It starts by cutting the schema into 187 connected subschemas joined by foreign keys, so prompts are small and semantically coherent. A few-shot prompt shows the LLM CREATE statements and mechanically built example queries, optionally biased toward group by or order by clauses. Validators filter for syntax and relevance; then a SQL parser computes how often each table, column, operator, join, and sub-select appears, and the pipeline biases the next round toward uncovered areas. The other half of the machinery is the multi-predictor-head GNN cost model from the authors' earlier work, which turns the generated
What would settle it
Run the same 1,000-query test with several independently generated synthetic and mechanical training sets at matched sizes, and compare the distributions of median Q-error and total routing time; if the synthetic advantage does not persist across seeds, or vanishes when sizes are equalized, the 45%-fewer-queries claim is not supported.
Extended reading notes
Core claim
The paper reports that the distribution of queries used to train a learned cost model matters more than the sheer number, and that an LLM-based generator with coverage steering can produce a better distribution from scratch. The pipeline decomposes the schema into 187 foreign-key-connected subschemas, prompts an instruction-tuned model with CREATE statements and mechanically built seed examples, filters the output for syntactic correctness and semantic relevance, and parses accepted queries to count table, column, and operator coverage; any gaps trigger another generation round. The resulting 2,200-query training set yields a median Q-error of 1.18 versus 1.20 for the 4,000-query mechanical
Load-bearing premise
The result rests on a single comparison run with different training-set sizes: if the 1.20-to-1.18 Q-error gap and the 165-to-150-minute routing gap are within run-to-run noise, the claimed advantage of synthetic queries disappears.
Editorial extensions
If this is right
- If the result holds, a new database deployment can train a learned cost model immediately from its schema, eliminating the months of suboptimal bootstrapping operation that historical-query approaches require.
- Training-data volume can shrink by 45%, meaning fewer queries have to be actually executed to get ground-truth labels, the expensive step in building these models.
- Coverage steering means the generator can be biased toward under-represented SQL constructs, so training sets can be deliberately balanced rather than reflecting whatever queries happened to be logged.
- The same diverse, validated query sets can serve other schema-dependent tasks such as stress testing and vulnerability testing, which also need realistic workloads.
Reading between the lines
- The paper does not run a matched-size control: training on 2,200 mechanically generated queries would separate the effect of query diversity from the effect of dataset size, which the current comparison conflates.
- The larger mean Q-error improvement (1.41 to 1.34) compared with the median suggests the benefit is concentrated in slow or otherwise hard-to-predict queries; reporting error separately per runtime bucket would test whether the even runtime distribution is the causal driver.
- A natural extension the authors only gesture at is to close the loop using the learned model itself: let its prediction errors or uncertainty choose which queries to generate next, turning coverage metrics into an acquisition function.
- The same pipeline could generalize to other cost-sensitive database tasks, such as learning join-order choices or physical plan selection, where diverse schema-bound queries are equally scarce.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This extended abstract proposes using LLM-based synthetic data generation (SDG) to create diverse SQL training queries for learned cost models (LCMs). The authors describe a pipeline based on the DiGiT framework: schema preprocessing, subschema creation, few-shot prompting of a Granite instruct model, validation/filtering, and Apache Calcite-based coverage analysis. They evaluate the LCM from [27] on the TPC-H schema, comparing ErrGNN+SDG trained on 2,200 SDG-generated queries with ErrGNN+Mech trained on 4,000 mechanically generated queries. They report a slight Q-error improvement (qmedian 1.20 to 1.18, qmean 1.41 to 1.34, qp95 2.38 to 2.37) and a 165- to 150-minute routing improvement on 1,000 test queries, concluding that SDG enables training with 45% fewer queries.
Significance. If validated, the result is practically significant: LLM-based SDG could reduce the bootstrapping cost of learned cost models by replacing large volumes of mechanical or historical training queries with fewer, more diverse synthetic ones. The paper has concrete strengths: it ships an open-source artifact, describes a real pipeline, and uses Calcite-based coverage analysis to characterize query diversity. The comparison is also not circular: the same LCM architecture and training procedure are used for both arms, so only the training-data source and size differ. However, the central quantitative claim rests on a small, single-run experiment in which training size and generation method are confounded. The significance is therefore conditional on the evaluation being strengthened.
major comments (3)
- [§5, Table 1] The headline claim of '45% fewer queries' is not established because the two arms differ in both generation method and training set size: ErrGNN+SDG uses 2,200 queries while ErrGNN+Mech uses 4,000. The reported Q-error and routing differences could be due to the size difference alone. To support the abstract's claim, the authors should train on matched sizes (e.g., 2,200 and 4,000 for both generators) or provide a learning-curve ablation over training sizes so the reader can see how much of the improvement comes from SDG rather than fewer or more samples.
- [§5, Table 1 and §A.3, Table 2] All results are from a single run with no error bars, repeated seeds, or significance tests. The differences are small (qmedian 1.20 vs. 1.18; qp95 2.38 vs. 2.37) and the routing result (165 vs. 150 minutes) is a single observation on one 1,000-query test set. These differences are within plausible run-to-run or optimizer-nondeterminism noise. The qmean drop is larger (1.41 to 1.34), but without per-query distributions or confidence intervals it is unclear whether it reflects broad improvement or a few long-tail queries. I recommend reporting repeated runs (at least 5--10 seeds), confidence intervals, and/or per-query Q-error distributions.
- [§5, 'Bootstrapping Learned Cost Models'] The composition of the 1,000-query test set is not described. If the test queries were generated by the same SDG pipeline, the comparison may favor the SDG-trained model; if they were generated mechanically, it may favor the mechanical baseline. The authors should state how the test set was produced, how it relates to the training distributions, and how representative it is of the intended workload. This is load-bearing for interpreting both the accuracy and the routing results.
minor comments (5)
- [References] A few reference typos should be fixed: [19] lists 'Lerner Lerner', [24] lists 'Pedreira Pedreira' and 'Binnig Binnig' appears in [4]; [25] has '!rst' instead of 'first'.
- [§5] The qmedian formula is defined in-line, but qmean and qp95 are used without definitions. Add one sentence defining them so the tables are self-contained.
- [§5] There is a duplicated phrase: 'the LCM trained when trained with the 2200 queries' should read 'the LCM trained with the 2200 queries'.
- [§A.3, Table 2] Table 2 reports only qmean per engine/provisioning. For consistency with Table 1, reporting qmedian and qp95 as well would make the per-engine picture clearer.
- [§4, Figure 1] Figure 1 is referenced but appears only as a pipeline diagram without detailed caption text; a short caption explaining the artifacts at each stage would help readers.
Circularity Check
No material circularity: the comparison is empirical and the LCM is shared between arms; self-citations are not load-bearing.
full rationale
The paper's central claim (LLM-based SDG trains an LCM to equal or better accuracy with 45% fewer queries) is an experimental result, not a derivation. The LCM from [27] is used identically for both ErrGNN+Mech and ErrGNN+SDG; no parameter is fitted to the target metric and no equation defines the outcome in terms of the input. The 2,200-vs-4,000 comparison is confounded by training-set size and lacks repeated trials, but that is an experimental-design/correctness risk, not a circularity: the result could have gone either way and is evaluated against an external TPC-H test set. Self-citations to DiGiT [8] and the authors' own LCM [27] supply components, but the evaluation harness is shared and the comparison is externally grounded, so the citations do not carry the argument by themselves. No fitted-input-called-prediction, uniqueness-import, ansatz-smuggling, or renaming pattern is present.
Assumptions & free parameters
free parameters (2)
- SDG training set size =
2,200 queries
- Mechanical training set size =
4,000 queries
assumptions (4)
- domain assumption The LCM from [27] is a reliable and unbiased evaluation harness for comparing training-data quality.
- domain assumption A single run on a 1,000-query mechanically generated test set is sufficient to assess predictive accuracy and routing.
- ad hoc to paper Calcite-derived coverage counts (tables, columns, clauses, functions) are a valid proxy for the diversity needed to train an LCM.
- domain assumption TPC-H and the granite-3.3-8b-instruct model are representative for evaluating synthetic SQL generation for cost models.
Cite this review
Pith. "Pith review of Bootstrapping Learned Cost Models with Synthetic SQL Queries." pith.science (2026). https://pith.science/paper/XT3OEG2F
@misc{pith2026250819807,
author = {Pith},
title = {Pith review of: Bootstrapping Learned Cost Models with Synthetic SQL Queries},
year = {2026},
howpublished = {\url{https://pith.science/paper/XT3OEG2F}},
note = {Machine review of arXiv:2508.19807}
}
read the original abstract
Having access to realistic workloads for a given database instance is extremely important to enable stress and vulnerability testing, as well as to optimize for cost and performance. Recent advances in learned cost models have shown that when enough diverse SQL queries are available, one can effectively and efficiently predict the cost of running a given query against a specific database engine. In this paper, we describe our experience in exploiting modern synthetic data generation techniques, inspired by the generative AI and LLM community, to create high-quality datasets enabling the effective training of such learned cost models. Initial results show that we can improve a learned cost model's predictive accuracy by training it with 45% fewer queries than when using competitive generation approaches.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Rethinking Query Optimization for Multi-Agent Systems [Vision]
Agentic data pipelines are built by hand today; this paper sets a research agenda for automatically optimizing their structure, model choices, and execution engines jointly as a new query-optimization problem.
Reference graph
Works this paper leans on
-
[27]
András Strausz, Niels Pardon, and Ioana Giurgiu. 2025. A Learned Cost Model- based Cross-engine Optimizer for SQL Workloads. arXiv:2506.02802 [cs.DB] https://arxiv.org/abs/2506.02802
arXiv 2025
-
[1]
[n. d.]. Transaction Processing and Performance Council.https://tpc.org/ ([n. d.])
-
[2]
Peter Akioyamen, Zixuan Yi, and Ryan Marcus. 2024. The Unreasonable Effec- tiveness of LLMs for Query Optimization. arXiv. https://doi.org/10.48550/arXiv. 2411.02862
-
[3]
Christoph Anneser, Nesime Tatbul, David Cohen, Zhenggang Xu, Prithviraj Pandian, Nikolay Laptev, and Ryan Marcus. 2023. AutoSteer: Learned Query Optimization for Any SQL Database. Proceedings of the VLDB Endowment 16, 12. https://doi.org/10.14778/3611540.3611544
arXiv 2023
-
[4]
Lawrence Benson, Carsten Binnig, Jan-Micha Bodensohn, Federico Lorenzi, Jigao Luo, Danica Porobic, Tilmann Rabl, Anupam Sanghi, Russell Sears, Pınar Tözün Tözün, and Tobias Ziegler. 2024. Surprise Benchmarking: The Why, What, and How. In DBTest’24: Proceedings of the Tenth International Workshop on Testing Database Systems. 1–8
work page 2024
-
[5]
Robin Chan, Katsiaryna Myrilenka, Thomas Gschwind, Christoph Miksovic- Czasch, Paolo Scotton, Enrico Toniato, and Abdel Labbi. 2024. Adapting LLMs for Structured Natural Language API Integration. In Proc. of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track . 991–1000
work page 2024
-
[6]
Transaction Processing Performance Council. [n. d.]. TPC-H Benchmark. See https://www.tpc.org/tpch/ (accessed 6 May 2025)
work page 2025
-
[7]
Aaron Grattafiori et.al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783
arXiv 2024
Show all 36 references
-
[8]
fms-dgt [n. d.]. Scalable Synthetic Data Generation (SDG). https://github.com/ foundation-model-stack/fms-dgt (accessed 9 June 2025)
2025
-
[9]
Documentation for IBM watsonx as a Service. [n. d.]. Generating synthetic unstructured data (beta). https://dataplatform.cloud.ibm.com/docs/content/wsj/ synthetic/sdg-unstructured-overview.html?context=wx&audience=wdp/ (ac- cessed 9 June 2025)
2025
-
[10]
Apache Software Foundation. [n. d.]. Apache Calcite. See https://calcite.apache. org/ (accessed 2 May 2025)
2025
-
[11]
Bogdan Ghit, Nicolas Poggi, Josh Rosen, Reynold Xin, and Peter Boncz. 2020. SparkFuzz: searching correctness regressions in modern query engines. In Pro- ceedings of the Workshop on Testing Database Systems (Portland, Oregon) (DBTest ’20). Association for Computing Machinery, ...
2020
-
[12]
Benjamin Hilprecht and Carsten Binnig. 2022. Zero-shot cost models for out- of-the-box learned cost prediction. Proceedings of the VLDB Endowment 15, 11. https://doi.org/10.14778/3551793.3551799
2022
-
[13]
Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL. arXiv:2406.08426 [cs.CL] https://arxiv.org/abs/2406. 08426
2025
-
[14]
Paulina Toro Isaza, Yu Deng, Michael Nidd, Amar Prakash Azad, and Laura Shwartz. 2022. Improving Model Performance Using Metric-Guided Data Selec- tion Framework. In 2022 IEEE International Conference on Big Data (Big Data) . IEEE, 4750–4757
2022
-
[15]
Zu-Ming Jiang, Jia-Ju Bai, and Zhendong Su. 2023. DynSQL: Stateful Fuzzing for Database Management Systems with Complex and Valid SQL Query Generation. In 32nd USENIX Security Symposium (USENIX Security 23) . USENIX Association, Anaheim, CA, 4949–4965. https://www.usenix.org/c...
2023
-
[16]
Amin Kamali, Verena Kantere, Calisto Zuzarte, and Vincent Corvinelli. [n. d.]. Roq: Robust Query Optimization Based on a Risk-aware Learned Cost Model
-
[17]
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. InThe Twelfth Internation...
2024
-
[18]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang, and Ion Stoica. 2023. vLLM: Easy, Fast, and cheap LLM Serving with PagedAttention. See https://vllm.ai/ (accessed 2 May 2025)
2023
-
[19]
Alberto Lerner Lerner, Matthias Jasny, Theo Jepsen, Carsten Binnig Binnig, and Philippe Cudré-Mauroux. 2022. DBMS Annihilator: A High-Performance Database Workload Generator in Action. In Proc. VLDB Endow. 15, 12 . 3682—- 3685
2022
-
[20]
Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runchi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2024. A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?. In https://doi.org/10.48550/arXiv.2408.05109
2024 doi
-
[21]
Karime Maamri, Fadhil Abu baker, Daniel Jaroslawicz, and Amine Mhedhbi
-
[22]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Al- izadeh, and Tim Kraska. 2021. Bao: Making Learned Query Optimization Practical. In Proceedings of the 2021 International Conference on Management of Data . ACM, Virtual Event China. https://doi.org/10.1145/3...
2021
-
[23]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019. Neo: a learned query optimizer. Proceedings of the VLDB Endowment 12, 11. https://doi.org/10. 14778/3342263.3342644
2019
-
[24]
Pedro Pedreira Pedreira, Orri Erling, Konstantinos Karanasos, Scott Schneider, Wes McKinney, Satya R Valluri, Mohamed Zait, and Jacques Nadeau. 2023. The Composable Data Management System Manifesto. In PVLDB, 16(10). 2679–2685
2023
-
[25]
Meikel Poess, Tilmann Rabl, and Hans-Arno Jacobsen. 2017. Analysis of TPC-DS: the !rst standard benchmark for SQL-based big data systems. In Proceedings of the 2017 Symposium on Cloud Computing . 573–585
2017
-
[26]
Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2024. A Survey on Employing Large Language Models for Text-to-SQL Tasks. arXiv:2407.15186 [cs.CL] https://arxiv.org/abs/2407.15186
2024 arXiv
-
[28]
synthetic-data-kit [n. d.]. Synthetic Data Kit. https://github.com/meta-llama/ synthetic-data-kit (accessed 9 June 2025)
2025
-
[29]
IBM Granite Team. [n. d.]. Granite-3.3-8B-Instruct. (accessed 9 June 2025) https://huggingface.co/ibm-granite/granite-3.3-8b-instruct
2025
-
[30]
Adrian Vogelsgesang, Michael Haubenschild, Jan Finis, Alfons Kemper, Viktor Leis, Tobias Muehlbauer, Thomas Neumann, and Manuel Then. 2018. Get Real: How Benchmarks Fail to Represent the Real World. In DBTest’18: Proceedings of the Workshop on Testing Database Systems. 1–6
2018
-
[31]
Smith, Daniel Khashabi, and Hannaneh Hajishirzi
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560 [cs.CL] https://arxiv. org/abs/2212.10560
2023 arXiv
-
[32]
Ziniu Wu, Ryan Marcus, Zhengchun Liu, Parimarjan Negi, Vikram Nathan, Pascal Pfeil, Gaurav Saxena, Mohammad Rahman, Balakrishnan Narayanaswamy, and Tim Kraska. 2024. Stage: Query execution time prediction in amazon redshift. In Companion of the 2024 International Conference on...
2024
-
[33]
Geoffrey X Yu, Ziniu Wu, Ferdi Kossmann, Tianyu Li, Markos Markakis, Amadou Ngom, Samuel Madden, and Tim Kraska. 2024. Blueprinting the Cloud: Unifying and Automatically Optimizing Cloud Data Infrastructures with BRAD–Extended Version. arXiv preprint arXiv:2407.15363 (2024)
2024 arXiv
-
[34]
Rong Zhu, Wei Chen, Bolin Ding, Xingguang Chen, Andreas Pfadler, Ziniu Wu, and Jingren Zhou. 2023. Lero: A Learning-to-Rank Query Optimizer. Proceedings of the VLDB Endowment 16, 6. https://doi.org/10.14778/3583140.3583160
2023
-
[35]
Xiaohu Zhu, Qian Li, Lizhen Cui, and Yongkang Liu. 2024. Large Language Model Enhanced Text-to-SQL Generation: A Survey. In https://arxiv.org/pdf/2410.06011. 5 A APPENDIX A.1 Coverage of Structural Complexity Categories Figure 5 shows the distribution of order by, group by, an...
2024 arXiv
-
[2024]
In NeurIPS 2024 Third Table Representation Learning Workshop
The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models. In NeurIPS 2024 Third Table Representation Learning Workshop
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.