Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Bootstrapping Learned Cost Models with Synthetic SQL Queries

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read The paper claims that an LLM-driven synthetic SQL generator can train a learned cost model more accurately using 45% fewer queries than a mechanical generator, and can cut cross-engine routing time by 10%.

desk verdict A plausible LLM-based synthetic query generation pipeline, but the central 45% claim rests on one unmatched, unrepeated comparison. read the letter →

arxiv 2508.19807 v1 pith:XT3OEG2F submitted 2025-08-27 cs.DB cs.AI

classification cs.DBcs.AI
keywords learnedcostmodelssyntheticdatagenerationSQLquerylargelanguageroutingworkloaddiversityTC-Hbenchmarkbootstrapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Learned cost models predict how long a query will take on a given database engine, but they are only as good as the workload they train on. Real query histories take months to collect, and mechanical generators produce large but narrow sets of queries. This paper tries to solve that cold-start problem with a synthetic-data pipeline that uses a large language model to generate diverse, schema-specific SQL queries, validated and steered by coverage metrics. On a standard decision-support benchmark schema, training the authors' learned cost model on 2,200 such queries beat training it on 4,000 mechanically generated queries, improving median prediction error and cutting the time to route 1,000 test queries from 165 to 150 minutes. If the result holds, database deployments could bootstrap efficient cross-engine query routing from the schema alone, without waiting for historical workload data.

What carries the argument

The load-bearing mechanism is the coverage-steered synthetic query generation loop. It starts by cutting the schema into 187 connected subschemas joined by foreign keys, so prompts are small and semantically coherent. A few-shot prompt shows the LLM CREATE statements and mechanically built example queries, optionally biased toward group by or order by clauses. Validators filter for syntax and relevance; then a SQL parser computes how often each table, column, operator, join, and sub-select appears, and the pipeline biases the next round toward uncovered areas. The other half of the machinery is the multi-predictor-head GNN cost model from the authors' earlier work, which turns the generated

What would settle it

Run the same 1,000-query test with several independently generated synthetic and mechanical training sets at matched sizes, and compare the distributions of median Q-error and total routing time; if the synthetic advantage does not persist across seeds, or vanishes when sizes are equalized, the 45%-fewer-queries claim is not supported.

Watch

Extended reading notes

Core claim

The paper reports that the distribution of queries used to train a learned cost model matters more than the sheer number, and that an LLM-based generator with coverage steering can produce a better distribution from scratch. The pipeline decomposes the schema into 187 foreign-key-connected subschemas, prompts an instruction-tuned model with CREATE statements and mechanically built seed examples, filters the output for syntactic correctness and semantic relevance, and parses accepted queries to count table, column, and operator coverage; any gaps trigger another generation round. The resulting 2,200-query training set yields a median Q-error of 1.18 versus 1.20 for the 4,000-query mechanical

Load-bearing premise

The result rests on a single comparison run with different training-set sizes: if the 1.20-to-1.18 Q-error gap and the 165-to-150-minute routing gap are within run-to-run noise, the claimed advantage of synthetic queries disappears.

Editorial extensions

If this is right

  • If the result holds, a new database deployment can train a learned cost model immediately from its schema, eliminating the months of suboptimal bootstrapping operation that historical-query approaches require.
  • Training-data volume can shrink by 45%, meaning fewer queries have to be actually executed to get ground-truth labels, the expensive step in building these models.
  • Coverage steering means the generator can be biased toward under-represented SQL constructs, so training sets can be deliberately balanced rather than reflecting whatever queries happened to be logged.
  • The same diverse, validated query sets can serve other schema-dependent tasks such as stress testing and vulnerability testing, which also need realistic workloads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run a matched-size control: training on 2,200 mechanically generated queries would separate the effect of query diversity from the effect of dataset size, which the current comparison conflates.
  • The larger mean Q-error improvement (1.41 to 1.34) compared with the median suggests the benefit is concentrated in slow or otherwise hard-to-predict queries; reporting error separately per runtime bucket would test whether the even runtime distribution is the causal driver.
  • A natural extension the authors only gesture at is to close the loop using the learned model itself: let its prediction errors or uncertainty choose which queries to generate next, turning coverage metrics into an acquisition function.
  • The same pipeline could generalize to other cost-sensitive database tasks, such as learning join-order choices or physical plan selection, where diverse schema-bound queries are equally scarce.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This extended abstract proposes using LLM-based synthetic data generation (SDG) to create diverse SQL training queries for learned cost models (LCMs). The authors describe a pipeline based on the DiGiT framework: schema preprocessing, subschema creation, few-shot prompting of a Granite instruct model, validation/filtering, and Apache Calcite-based coverage analysis. They evaluate the LCM from [27] on the TPC-H schema, comparing ErrGNN+SDG trained on 2,200 SDG-generated queries with ErrGNN+Mech trained on 4,000 mechanically generated queries. They report a slight Q-error improvement (qmedian 1.20 to 1.18, qmean 1.41 to 1.34, qp95 2.38 to 2.37) and a 165- to 150-minute routing improvement on 1,000 test queries, concluding that SDG enables training with 45% fewer queries.

Significance. If validated, the result is practically significant: LLM-based SDG could reduce the bootstrapping cost of learned cost models by replacing large volumes of mechanical or historical training queries with fewer, more diverse synthetic ones. The paper has concrete strengths: it ships an open-source artifact, describes a real pipeline, and uses Calcite-based coverage analysis to characterize query diversity. The comparison is also not circular: the same LCM architecture and training procedure are used for both arms, so only the training-data source and size differ. However, the central quantitative claim rests on a small, single-run experiment in which training size and generation method are confounded. The significance is therefore conditional on the evaluation being strengthened.

major comments (3)
  1. [§5, Table 1] The headline claim of '45% fewer queries' is not established because the two arms differ in both generation method and training set size: ErrGNN+SDG uses 2,200 queries while ErrGNN+Mech uses 4,000. The reported Q-error and routing differences could be due to the size difference alone. To support the abstract's claim, the authors should train on matched sizes (e.g., 2,200 and 4,000 for both generators) or provide a learning-curve ablation over training sizes so the reader can see how much of the improvement comes from SDG rather than fewer or more samples.
  2. [§5, Table 1 and §A.3, Table 2] All results are from a single run with no error bars, repeated seeds, or significance tests. The differences are small (qmedian 1.20 vs. 1.18; qp95 2.38 vs. 2.37) and the routing result (165 vs. 150 minutes) is a single observation on one 1,000-query test set. These differences are within plausible run-to-run or optimizer-nondeterminism noise. The qmean drop is larger (1.41 to 1.34), but without per-query distributions or confidence intervals it is unclear whether it reflects broad improvement or a few long-tail queries. I recommend reporting repeated runs (at least 5--10 seeds), confidence intervals, and/or per-query Q-error distributions.
  3. [§5, 'Bootstrapping Learned Cost Models'] The composition of the 1,000-query test set is not described. If the test queries were generated by the same SDG pipeline, the comparison may favor the SDG-trained model; if they were generated mechanically, it may favor the mechanical baseline. The authors should state how the test set was produced, how it relates to the training distributions, and how representative it is of the intended workload. This is load-bearing for interpreting both the accuracy and the routing results.
minor comments (5)
  1. [References] A few reference typos should be fixed: [19] lists 'Lerner Lerner', [24] lists 'Pedreira Pedreira' and 'Binnig Binnig' appears in [4]; [25] has '!rst' instead of 'first'.
  2. [§5] The qmedian formula is defined in-line, but qmean and qp95 are used without definitions. Add one sentence defining them so the tables are self-contained.
  3. [§5] There is a duplicated phrase: 'the LCM trained when trained with the 2200 queries' should read 'the LCM trained with the 2200 queries'.
  4. [§A.3, Table 2] Table 2 reports only qmean per engine/provisioning. For consistency with Table 1, reporting qmedian and qp95 as well would make the per-engine picture clearer.
  5. [§4, Figure 1] Figure 1 is referenced but appears only as a pipeline diagram without detailed caption text; a short caption explaining the artifacts at each stage would help readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the comparison is empirical and the LCM is shared between arms; self-citations are not load-bearing.

full rationale

The paper's central claim (LLM-based SDG trains an LCM to equal or better accuracy with 45% fewer queries) is an experimental result, not a derivation. The LCM from [27] is used identically for both ErrGNN+Mech and ErrGNN+SDG; no parameter is fitted to the target metric and no equation defines the outcome in terms of the input. The 2,200-vs-4,000 comparison is confounded by training-set size and lacks repeated trials, but that is an experimental-design/correctness risk, not a circularity: the result could have gone either way and is evaluated against an external TPC-H test set. Self-citations to DiGiT [8] and the authors' own LCM [27] supply components, but the evaluation harness is shared and the comparison is externally grounded, so the citations do not carry the argument by themselves. No fitted-input-called-prediction, uniqueness-import, ansatz-smuggling, or renaming pattern is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities; it combines existing components (DiGiT SDG pipeline, Apache Calcite, granite LLM, the LCM from [27]) and adapts them to LCM training data generation. The 'subschema groupings' and 'coverage gaps' are methodological constructs within the pipeline, not independent postulated entities requiring external evidence.

free parameters (2)
  • SDG training set size = 2,200 queries
    The central '45% fewer queries' claim depends on comparing 2,200 SDG queries to 4,000 mechanical queries. No matched-size baseline is provided, so the choice of 2,200 is not controlled.
  • Mechanical training set size = 4,000 queries
    Baseline from [12]; size differs from the SDG condition, confounding training data quantity with generation method.
assumptions (4)
  • domain assumption The LCM from [27] is a reliable and unbiased evaluation harness for comparing training-data quality.
    Section 5 uses the LCM from [27] for all measurements without independent validation of the model's accuracy per engine.
  • domain assumption A single run on a 1,000-query mechanically generated test set is sufficient to assess predictive accuracy and routing.
    Section 5 reports one Q-error table and one routing result (165 vs 150 min) with no error bars or repeated trials.
  • ad hoc to paper Calcite-derived coverage counts (tables, columns, clauses, functions) are a valid proxy for the diversity needed to train an LCM.
    Section 4 'Coverage' uses these counts to steer generation; no experiment links coverage metrics to LCM prediction accuracy.
  • domain assumption TPC-H and the granite-3.3-8b-instruct model are representative for evaluating synthetic SQL generation for cost models.
    All experiments use a single benchmark schema and one LLM; generalization to other schemas and models is asserted, not shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bootstrapping Learned Cost Models with Synthetic SQL Queries." pith.science (2026). https://pith.science/paper/XT3OEG2F

@misc{pith2026250819807,
  author       = {Pith},
  title        = {Pith review of: Bootstrapping Learned Cost Models with Synthetic SQL Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XT3OEG2F}},
  note         = {Machine review of arXiv:2508.19807}
}
read the original abstract

Having access to realistic workloads for a given database instance is extremely important to enable stress and vulnerability testing, as well as to optimize for cost and performance. Recent advances in learned cost models have shown that when enough diverse SQL queries are available, one can effectively and efficiently predict the cost of running a given query against a specific database engine. In this paper, we describe our experience in exploiting modern synthetic data generation techniques, inspired by the generative AI and LLM community, to create high-quality datasets enabling the effective training of such learned cost models. Initial results show that we can improve a learned cost model's predictive accuracy by training it with 45% fewer queries than when using competitive generation approaches.

Figures

Figures reproduced from arXiv: 2508.19807 by the authors.

Figure 1
Figure 1. The pipeline consists of the following steps: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 1
Figure 1. SDG pipeline with input and output (SQL queries) artifacts. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Generation prompt outline [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Query structure by generation method. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png]
Figure 4
Figure 4. Figure 4: Runtime characteristics of generated queries. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Query Distribution by Generation and Bias [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Runtime Characteristics of Generated Queries [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Query Optimization for Multi-Agent Systems [Vision]

    cs.DB 2025-12 conditional novelty 6.0 of 10

    Agentic data pipelines are built by hand today; this paper sets a research agenda for automatically optimizing their structure, model choices, and execution engines jointly as a new query-optimization problem.

Reference graph

Works this paper leans on

36 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [27]

    András Strausz, Niels Pardon, and Ioana Giurgiu. 2025. A Learned Cost Model- based Cross-engine Optimizer for SQL Workloads. arXiv:2506.02802 [cs.DB] https://arxiv.org/abs/2506.02802

  2. [1]

    [n. d.]. Transaction Processing and Performance Council.https://tpc.org/ ([n. d.])

  3. [2]

    Peter Akioyamen, Zixuan Yi, and Ryan Marcus. 2024. The Unreasonable Effec- tiveness of LLMs for Query Optimization. arXiv. https://doi.org/10.48550/arXiv. 2411.02862

  4. [3]

    Christoph Anneser, Nesime Tatbul, David Cohen, Zhenggang Xu, Prithviraj Pandian, Nikolay Laptev, and Ryan Marcus. 2023. AutoSteer: Learned Query Optimization for Any SQL Database. Proceedings of the VLDB Endowment 16, 12. https://doi.org/10.14778/3611540.3611544

  5. [4]

    Lawrence Benson, Carsten Binnig, Jan-Micha Bodensohn, Federico Lorenzi, Jigao Luo, Danica Porobic, Tilmann Rabl, Anupam Sanghi, Russell Sears, Pınar Tözün Tözün, and Tobias Ziegler. 2024. Surprise Benchmarking: The Why, What, and How. In DBTest’24: Proceedings of the Tenth International Workshop on Testing Database Systems. 1–8

  6. [5]

    Robin Chan, Katsiaryna Myrilenka, Thomas Gschwind, Christoph Miksovic- Czasch, Paolo Scotton, Enrico Toniato, and Abdel Labbi. 2024. Adapting LLMs for Structured Natural Language API Integration. In Proc. of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track . 991–1000

  7. [6]

    Transaction Processing Performance Council. [n. d.]. TPC-H Benchmark. See https://www.tpc.org/tpch/ (accessed 6 May 2025)

  8. [7]

    Aaron Grattafiori et.al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

Show all 36 references
  1. [8]

    fms-dgt [n. d.]. Scalable Synthetic Data Generation (SDG). https://github.com/ foundation-model-stack/fms-dgt (accessed 9 June 2025)

  2. [9]

    Documentation for IBM watsonx as a Service. [n. d.]. Generating synthetic unstructured data (beta). https://dataplatform.cloud.ibm.com/docs/content/wsj/ synthetic/sdg-unstructured-overview.html?context=wx&audience=wdp/ (ac- cessed 9 June 2025)

  3. [10]

    Apache Software Foundation. [n. d.]. Apache Calcite. See https://calcite.apache. org/ (accessed 2 May 2025)

  4. [11]

    Bogdan Ghit, Nicolas Poggi, Josh Rosen, Reynold Xin, and Peter Boncz. 2020. SparkFuzz: searching correctness regressions in modern query engines. In Pro- ceedings of the Workshop on Testing Database Systems (Portland, Oregon) (DBTest ’20). Association for Computing Machinery, ...

  5. [12]

    Benjamin Hilprecht and Carsten Binnig. 2022. Zero-shot cost models for out- of-the-box learned cost prediction. Proceedings of the VLDB Endowment 15, 11. https://doi.org/10.14778/3551793.3551799

  6. [13]

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL. arXiv:2406.08426 [cs.CL] https://arxiv.org/abs/2406. 08426

  7. [14]

    Paulina Toro Isaza, Yu Deng, Michael Nidd, Amar Prakash Azad, and Laura Shwartz. 2022. Improving Model Performance Using Metric-Guided Data Selec- tion Framework. In 2022 IEEE International Conference on Big Data (Big Data) . IEEE, 4750–4757

  8. [15]

    Zu-Ming Jiang, Jia-Ju Bai, and Zhendong Su. 2023. DynSQL: Stateful Fuzzing for Database Management Systems with Complex and Valid SQL Query Generation. In 32nd USENIX Security Symposium (USENIX Security 23) . USENIX Association, Anaheim, CA, 4949–4965. https://www.usenix.org/c...

  9. [16]

    Amin Kamali, Verena Kantere, Calisto Zuzarte, and Vincent Corvinelli. [n. d.]. Roq: Robust Query Optimization Based on a Risk-aware Learned Cost Model

  10. [17]

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. InThe Twelfth Internation...

  11. [18]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang, and Ion Stoica. 2023. vLLM: Easy, Fast, and cheap LLM Serving with PagedAttention. See https://vllm.ai/ (accessed 2 May 2025)

  12. [19]

    Alberto Lerner Lerner, Matthias Jasny, Theo Jepsen, Carsten Binnig Binnig, and Philippe Cudré-Mauroux. 2022. DBMS Annihilator: A High-Performance Database Workload Generator in Action. In Proc. VLDB Endow. 15, 12 . 3682—- 3685

  13. [20]

    Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runchi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2024. A Survey of NL2SQL with Large Language Models: Where are we, and where are we going?. In https://doi.org/10.48550/arXiv.2408.05109

  14. [21]

    Karime Maamri, Fadhil Abu baker, Daniel Jaroslawicz, and Amine Mhedhbi

  15. [22]

    Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Al- izadeh, and Tim Kraska. 2021. Bao: Making Learned Query Optimization Practical. In Proceedings of the 2021 International Conference on Management of Data . ACM, Virtual Event China. https://doi.org/10.1145/3...

  16. [23]

    Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019. Neo: a learned query optimizer. Proceedings of the VLDB Endowment 12, 11. https://doi.org/10. 14778/3342263.3342644

  17. [24]

    Pedro Pedreira Pedreira, Orri Erling, Konstantinos Karanasos, Scott Schneider, Wes McKinney, Satya R Valluri, Mohamed Zait, and Jacques Nadeau. 2023. The Composable Data Management System Manifesto. In PVLDB, 16(10). 2679–2685

  18. [25]

    Meikel Poess, Tilmann Rabl, and Hans-Arno Jacobsen. 2017. Analysis of TPC-DS: the !rst standard benchmark for SQL-based big data systems. In Proceedings of the 2017 Symposium on Cloud Computing . 573–585

  19. [26]

    Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2024. A Survey on Employing Large Language Models for Text-to-SQL Tasks. arXiv:2407.15186 [cs.CL] https://arxiv.org/abs/2407.15186

  20. [28]

    synthetic-data-kit [n. d.]. Synthetic Data Kit. https://github.com/meta-llama/ synthetic-data-kit (accessed 9 June 2025)

  21. [29]

    IBM Granite Team. [n. d.]. Granite-3.3-8B-Instruct. (accessed 9 June 2025) https://huggingface.co/ibm-granite/granite-3.3-8b-instruct

  22. [30]

    Adrian Vogelsgesang, Michael Haubenschild, Jan Finis, Alfons Kemper, Viktor Leis, Tobias Muehlbauer, Thomas Neumann, and Manuel Then. 2018. Get Real: How Benchmarks Fail to Represent the Real World. In DBTest’18: Proceedings of the Workshop on Testing Database Systems. 1–6

  23. [31]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560 [cs.CL] https://arxiv. org/abs/2212.10560

  24. [32]

    Ziniu Wu, Ryan Marcus, Zhengchun Liu, Parimarjan Negi, Vikram Nathan, Pascal Pfeil, Gaurav Saxena, Mohammad Rahman, Balakrishnan Narayanaswamy, and Tim Kraska. 2024. Stage: Query execution time prediction in amazon redshift. In Companion of the 2024 International Conference on...

  25. [33]

    Geoffrey X Yu, Ziniu Wu, Ferdi Kossmann, Tianyu Li, Markos Markakis, Amadou Ngom, Samuel Madden, and Tim Kraska. 2024. Blueprinting the Cloud: Unifying and Automatically Optimizing Cloud Data Infrastructures with BRAD–Extended Version. arXiv preprint arXiv:2407.15363 (2024)

  26. [34]

    Rong Zhu, Wei Chen, Bolin Ding, Xingguang Chen, Andreas Pfadler, Ziniu Wu, and Jingren Zhou. 2023. Lero: A Learning-to-Rank Query Optimizer. Proceedings of the VLDB Endowment 16, 6. https://doi.org/10.14778/3583140.3583160

  27. [35]

    Xiaohu Zhu, Qian Li, Lizhen Cui, and Yongkang Liu. 2024. Large Language Model Enhanced Text-to-SQL Generation: A Survey. In https://arxiv.org/pdf/2410.06011. 5 A APPENDIX A.1 Coverage of Structural Complexity Categories Figure 5 shows the distribution of order by, group by, an...

  28. [2024]

    In NeurIPS 2024 Third Table Representation Learning Workshop

    The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models. In NeurIPS 2024 Third Table Representation Learning Workshop

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.