REVIEW 3 major objections 6 minor 2 cited by
A Learned Cost Model-based Cross-engine Optimizer for SQL Workloads
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read One learned cost model with per-engine predictor heads can automatically route SQL queries to the engine with the lowest predicted execution time, cutting workload runtime by up to 25.2% zero-shot and 30.4% few-shot versus random routing.
desk verdict A competent multi-task GNN for cross-engine routing with honest evaluation, but the headline claims outrun the narrow synthetic workload it was tested on. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a shared bottom-up Graph Neural Network (GNN) encoder that converts a deserialized Substrait plan, expanded into a heterogeneous graph of Relation, Operation, Literal, Field, and Table nodes, into a single query embedding via mean pooling over all relation nodes. Each engine or provisioning gets a small MLP predictor head on top of that embedding, and all heads are trained jointly with a Q-error-based loss. This decouples the generic query representation from engine-specific cost behavior, which is what allows new engines to be added by fine-tuning one head while keeping the embedding fixed.
What would settle it
Collect a set of real production queries with five or more joins and runtimes beyond one minute, feed them to the trained model, and compare per-engine prediction errors and end-to-end routing against static and random routing; if the model's routing no longer beats static or Q-error degrades sharply, the synthetic-workload assumption fails.
Extended reading notes
Core claim
The paper's central claim is that cost prediction across heterogeneous execution engines is best treated as one multi-task learning problem over a shared query representation, not as a collection of engine-specific models. Feeding the predictor an optimized logical plan decorated with cardinality hints rather than an unoptimized plan improves average Q-error by up to 12.6% on IMDB, and the resulting per-engine estimates support query-to-engine routing that reduces total workload runtime by up to 25.2% zero-shot and 30.4% few-shot over random routing, while a fine-tuned model also beats static routing by up to 12.7% on IMDB. The claim includes database-agnostic behavior across five datasets and cheap extensibility: a new engine provisioning is added by training only a new predictor head on 250 queries.
Load-bearing premise
The load-bearing premise is that short synthetic queries with at most three joins and a one-minute runtime cap represent real lakehouse workloads; the paper acknowledges its generator gives weak control over query complexity and semantic plausibility.
Editorial extensions
If this is right
- A single shared embedding replaces per-engine cost models: all predictor heads are trained jointly, so adding or updating an engine no longer requires retraining the full model.
- Routing by the model reduces total workload runtime by up to 25.2% zero-shot and 30.4% few-shot relative to random routing, and on IMDB the few-shot model beats static routing by 12.7%.
- Using an optimized logical plan with cardinality hints as model input improves prediction accuracy, reducing average Q-error by up to 12.6% on IMDB over unoptimized plans.
- A new engine provisioning can be added by fine-tuning a single predictor head on about 250 queries, keeping data collection and training cost low.
- The model predicts execution time on unseen queries with median Q-error 1.21, meaning half of predictions deviate by at most 21% from measured times.
Reading between the lines
- The authors do not claim this, but the multi-head design suggests an entirely new engine family could be added with little data if its runtime correlates with the shared embedding; only a new provisioning of a seen engine is tested.
- The authors do not test this, but because the embedding mean-pools relation nodes, the same representation could support node-level cost estimates for distributed query execution, which they name as future work.
- The measured gains are bounded by the workload: the paper notes queries are under one minute, so longer and heavier workloads would plausibly amplify the routing advantage, though this remains untested.
- A natural follow-up is to use the model's prediction error on past queries to generate harder synthetic queries, which the authors themselves propose as a next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a learned cost model (LCM) for cross-engine query routing in lakehouse systems. The system translates SQL into optimized Substrait plans with cardinality hints, encodes the plans as heterogeneous graphs, and uses a bottom-up GNN with mean pooling over relation nodes to build a shared query embedding. Multiple predictor heads then estimate execution time for different engine provisionings, and the model can be extended to a new engine by fine-tuning only the new head on 250 queries. The evaluation uses five datasets and two engines (PrestoDB and SparkSQL with several provisionings), reporting median Q-error of 1.21 on held-out queries, zero-shot and few-shot workload-runtime reductions up to 25.2% and 30.4% versus random routing, and up to a 12.6% Q-error improvement when using optimized plans versus unoptimized plans.
Significance. The multi-head, shared-embedding design is a practical and sensible extension of prior zero-shot cost models: it avoids training separate GNNs per engine and makes new-engine addition cheap. The held-out and cross-dataset evaluations are genuine out-of-sample tests, not circular fits, and the artifact link is a strength. However, the central generalization claim is only supported within a narrow synthetic regime: queries have at most three joins and a one-minute runtime cap, and the authors themselves concede that the generator produces semantically degenerate queries. The headline routing gains are measured against random routing, a weak baseline, while the comparison against static routing is reported for only one dataset and without variance estimates. The significance of the contribution therefore depends on whether the workload-representativeness and baseline-comparison concerns can be addressed.
major comments (3)
- [Section 4.1.2 and Section 5] The training and evaluation workloads are restricted to synthetic queries with at most three joins and a one-minute runtime cap, and Section 5 concedes that the generator "only provides weak control over the nature of the generated queries," producing near-empty joins or full-table joins. Because the GNN message-passing depth equals plan depth and the predictor heads are trained on a truncated runtime distribution, the paper provides no evidence that the routing or Q-error advantages transfer to real lakehouse workloads with longer, more complex, or semantically plausible queries. This is load-bearing for the abstract's "diverse SQL queries" claim. I ask for an evaluation on at least one real workload (e.g., JOB) or a systematic study with more joins and longer runtimes, or, failing that, a clearly scoped claim that the results apply only to short synthetic queries.
- [Section 4.3, Figure 3] The reported routing savings are computed primarily against random routing, which is a weak baseline; the only stated advantage over static routing is 12.7% on IMDB in the few-shot setting, with "similar or lower" reported elsewhere. Figure 3 also has no error bars or fold-level variation, even though the zero-shot results come from cross-validation. Please report per-dataset zero-shot and few-shot totals against both baselines, with variance across folds, so that the robustness of the routing advantage over static routing can be assessed.
- [Section 4.2.3, Table 5] The claimed 12.6% average Q-error improvement from using optimized plans is computed on a single dataset (IMDB) and a single metric (qmean). Other datasets show much smaller or inconsistent gains (e.g., Stack Overflow in the same table), so the abstract's wording "decreases the average Q-error by even 12.6%" is misleading without a per-dataset reporting of the optimized-versus-unoptimized difference. Please report this improvement consistently for all datasets and metrics, or qualify the claim accordingly.
minor comments (6)
- [Abstract and Section 4.3] The abstract states a 25.2% zero-shot improvement over random routing, while Section 4.3 reports "up to 25.3%"; please reconcile the numbers.
- [Figure 3] The runtime axis begins at 90 minutes, which visually exaggerates the differences between strategies; a zero baseline or an explicit axis break would make the plot easier to interpret fairly.
- [Section 4.2.2] The "unseen queries" scenario evaluates held-out queries from the same databases used in training, which is different from the zero-shot scenario of Section 4.2.3; renaming it "same-database held-out queries" would avoid confusion.
- [Section 4.3] The routing experiment should state whether the reported totals include plan-creation and LCM inference overhead, and how routing ties are broken.
- [Section 3.2] The loss for negative predictions is described only as "an arbitrarily large penalty"; please specify the implemented value or functional form for reproducibility.
- [Table 2] The column header "#Rel" is cryptic; the caption defines it as the number of foreign-key constraints used for query generation, so the header or caption should be expanded for clarity.
Circularity Check
No significant circularity: the cost model is a supervised regressor trained on measured execution times and evaluated on held-out queries; the Q-error and routing claims do not reduce to fitted inputs or self-citations.
full rationale
The paper's derivation chain is a standard supervised ML pipeline: a GNN encodes query plans, per-engine MLP heads predict measured execution times, and the model is trained with a Q-error-based loss (Section 3.2). Predictions are evaluated on held-out data: Section 4.1.3 states '1000 queries are held out from each dataset' for unseen queries, and the zero-shot scenario uses cross-validation where '4 datasets serve as the training set and the fifth is used for evaluation.' The routing experiments compare the learned ranking against random and static baselines on these workloads, so the reported 25.2% and 30.4% improvements are genuine measurements on queries not used to fit the model. No fitted parameter is renamed as a prediction: the only fitted quantities are network weights trained on true execution times, and the reported Q-errors are computed on held-out queries. The few-shot scenario fine-tunes predictor heads on 250 queries drawn from the test dataset (Section 4.1.3), which is a legitimate domain-adaptation setup; the paper does not state that the exact fine-tuning queries are used for the reported evaluation metrics, and the reported median Q-errors around 1.2-1.5 are not consistent with re-reporting training fits. The paper contains no self-citations: it adopts the Bottom-up GNN from Hilprecht and Binnig [13] and compares against Kipf et al. [16] and BRAD [33], all external prior work. The Section 5 limitation that the random query generator 'only provides weak control over the nature of the generated queries' concerns workload representativeness and external validity, not circularity: the measured improvements remain genuine on the tested distribution. No step in the claimed derivation is equivalent to its own inputs by construction.
Assumptions & free parameters
free parameters (4)
- Fitted GNN and predictor weights =
4.7M total parameters (GNN); 1.6M (set-based)
- Shared embedding dimension d' =
256
- Few-shot fine-tuning query count =
250 (about 5% of dataset)
- Learning rate =
0.001
assumptions (4)
- domain assumption Generated queries are limited to at most 3 joins and a 1-minute runtime cap
- domain assumption Calcite with predicate pushdown, relation merging, and greedy join ordering is an adequate optimizer, and Selinger selectivity hints provide useful features
- domain assumption Execution time is the only cost relevant for routing; storage, data movement, provisioning, and load are ignored
- domain assumption Presto caching is disabled so measured times are independent
Cite this review
Pith. "Pith review of A Learned Cost Model-based Cross-engine Optimizer for SQL Workloads." pith.science (2026). https://pith.science/paper/D65JUSRB
@misc{pith2026250602802,
author = {Pith},
title = {Pith review of: A Learned Cost Model-based Cross-engine Optimizer for SQL Workloads},
year = {2026},
howpublished = {\url{https://pith.science/paper/D65JUSRB}},
note = {Machine review of arXiv:2506.02802}
}
read the original abstract
Lakehouse systems enable the same data to be queried with multiple execution engines. However, selecting the engine best suited to run a SQL query still requires a priori knowledge of the query computational requirements and an engine capability, a complex and manual task that only becomes more difficult with the emergence of new engines and workloads. In this paper, we address this limitation by proposing a cross-engine optimizer that can automate engine selection for diverse SQL queries through a learned cost model. Optimized with hints, a query plan is used for query cost prediction and routing. Cost prediction is formulated as a multi-task learning problem, and multiple predictor heads, corresponding to different engines and provisionings, are used in the model architecture. This eliminates the need to train engine-specific models and allows the flexible addition of new engines at a minimal fine-tuning cost. Results on various databases and engines show that using a query optimized logical plan for cost estimation decreases the average Q-error by even 12.6% over using unoptimized plans as input. Moreover, the proposed cross-engine optimizer reduces the total workload runtime by up to 25.2% in a zero-shot setting and 30.4% in a few-shot setting when compared to random routing.
Figures
Forward citations
Cited by 2 Pith papers
-
Rethinking Query Optimization for Multi-Agent Systems [Vision]
Agentic data pipelines are built by hand today; this paper sets a research agenda for automatically optimizing their structure, model choices, and execution engines jointly as a new query-optimization problem.
-
Bootstrapping Learned Cost Models with Synthetic SQL Queries
LLM-based synthetic SQL generation can train a learned cost model with fewer, more diverse queries than mechanical generation, though the measured accuracy gains are small and the comparison is not matched by training size.
Reference graph
Works this paper leans on
-
[1]
2025. IBM watsonx.data. https://www.ibm.com/docs/en/watsonx/watsonxdata/ 2.1.x?topic=overview
work page 2025
-
[2]
Divy Agrawal, Sanjay Chawla, Bertty Contreras-Rojas, Ahmed Elmagarmid, Yasser Idris, Zoi Kaoudi, Sebastian Kruse, Ji Lucas, Essam Mansour, Mourad Ouzzani, Paolo Papotti, Jorge-Arnulfo Quiané-Ruiz, Nan Tang, Saravanan Thiru- muruganathan, and Anis Troudi. 2018. RHEEM: enabling cross-platform data processing: may the big data be with you! Proceedings of the...
arXiv 2018
-
[3]
Peter Akioyamen, Zixuan Yi, and Ryan Marcus. 2024. The Unreasonable Effective- ness of LLMs for Query Optimization. https://doi.org/10.48550/arXiv.2411.02862 arXiv:2411.02862 [cs]
-
[4]
Rana Alotaibi, Damian Bursztyn, Alin Deutsch, Ioana Manolescu, and Stamatis Zampetakis. 2019. Towards Scalable Hybrid Stores: Constraint-Based Rewriting to the Rescue. In 2019 International Conference on Management of Data . ACM, 1660–1677. https://doi.org/10.1145/3299869.3319895
arXiv 2019
-
[5]
Christoph Anneser, Nesime Tatbul, David Cohen, Zhenggang Xu, Prithviraj Pandian, Nikolay Laptev, and Ryan Marcus. 2023. AutoSteer: Learned Query Optimization for Any SQL Database. Proceedings of the VLDB Endowment 16, 12 (Aug. 2023), 3515–3527. https://doi.org/10.14778/3611540.3611544
arXiv 2023
-
[6]
Edmon Begoli, Jesús Camacho Rodríguez, Julian Hyde, Michael J. Mior, and Daniel Lemire. 2018. Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources. In Proceedings of the 2018 International Conference on Management of Data . 221–230. https://doi.org/10. 1145/3183713.3190662 arXiv:1802.10233 [cs]
arXiv 2018
-
[7]
Yuri Breitbart, Hector Garcia-Molina, and Avi Silberschatz. 1992. Overview of Multidatabase Transaction Management. In VLDB Journal, vol. 1, no. 2 . ACM, 181–239
work page 1992
-
[8]
George-Octavian Bărbulescu, Taiyi Wang, Zak Singh, and Eiko Yoneki. 2024. Learned Graph Rewriting with Equality Saturation: A New Paradigm in Relational Query Rewrite and Beyond. http://arxiv.org/abs/2407.12794 arXiv:2407.12794 [cs]
work page Pith review arXiv 2024
Show all 40 references
-
[9]
Rich Caruana. [n.d.]. Multitask Learning. ([n. d.])
-
[10]
Vijay Gadepally, Peinan Chen, Jennie Duggan, Aaron Elmore Elmore, Brandon Haynesk, Jeremy Kepner Kepner, Samuel Madden Madden, Tim Mattson Matt- son, and Michael Stonebraker. 2015. The BigDAWG Polystore System. InACM SIGMOD Record, vol. 44, no. 2 . ACM, 11–16
2015
-
[11]
Wiener, Armando Fox, Michael Jordan, and David Patterson
Archana Ganapathi, Harumi Kuno, Umeshwar Dayal, Janet L. Wiener, Armando Fox, Michael Jordan, and David Patterson. 2009. Predicting Multiple Metrics for Queries: Better Decisions Enabled by Machine Learning. In 2009 IEEE 25th International Conference on Data Engineering . 592–...
2009
-
[12]
Haralampos Gavriilidis, Lennart Behme, Sokratis Papadopoulos, Stefano Bortoli, Jorge-Arnulfo Quiané-Ruiz, and Volker Markl. [n.d.]. Towards a Modular Data Management System Framework. ([n. d.])
-
[13]
Benjamin Hilprecht and Carsten Binnig. 2022. Zero-shot cost models for out-of- the-box learned cost prediction. Proceedings of the VLDB Endowment 15, 11 (July 2022), 2361–2374. https://doi.org/10.14778/3551793.3551799
2022
-
[14]
Vanja Josifovski, Peter Schwarz Schwarz, Laura Haas Haas, and Eileen Lin. 2002. Garlic: A New Flavor of Federated Query Processing for DB2. In 2002 ACM SIGMOD International Conference on Management of Data . ACM, 524–532
2002
-
[15]
Amin Kamali, Verena Kantere, Calisto Zuzarte, and Vincent Corvinelli. [n.d.]. Roq: Robust Query Optimization Based on a Risk-aware Learned Cost Model. ([n. d.])
- [16]
-
[17]
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann. 2015. How good are query optimizers, really? Proceedings of the VLDB Endowment 9, 3 (Nov. 2015), 204–215. https://doi.org/10.14778/ 2850583.2850594
2015
- [18]
- [19]
-
[20]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Al- izadeh, and Tim Kraska. 2021. Bao: Making Learned Query Optimization Practical. In Proceedings of the 2021 International Conference on Management of Data . ACM, Virtual Event China, 1275–1288. https://doi.or...
2021
-
[21]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019. Neo: a learned query optimizer. Proceedings of the VLDB Endowment 12, 11 (July 2019), 1705–1718. https://doi.org/10.14778/3342263.3342644
2019
- [22]
-
[23]
Parimarjan Negi, Matteo Interlandi, Ryan Marcus, Mohammad Alizadeh, Tim Kraska, Marc Friedman, and Alekh Jindal. 2021. Steering Query Optimizers: A Practical Take on Big Data Workloads. InProceedings of the 2021 International Conference on Management of Data. ACM, Virtual Even...
2021
-
[24]
Pedro Pedreira, Orri Erling, Konstantinos Karanasos, Scott Schneider, Wes McKin- ney, Satya R Valluri, Mohamed Zait, and Jacques Nadeau. 2023. The Composable Data Management System Manifesto. Proceedings of the VLDB Endowment 16, 10 (June 2023), 2679–2685. https://doi.org/10.1...
2023
-
[25]
Maksim Podkorytov and Michael Gubanov. 2019. Hybrid.Poly: A Consolidated In- teractive Analytical Polystore System. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) . IEEE, 1996–1999
2019
-
[26]
Calton Pu. 1998. Superdatabases for Composition of Heterogeneous Databases. In Fourth International Conference on Data Engineering (ICDE) . IEEE, 548–555
1998
-
[27]
P Griffiths Selinger, M M Astrahan, D D Chamberlin, R A Lorie, and T G Price. [n.d.]. Access Path Selection in a Relational Database Management System. ([n. d.])
-
[28]
substrait-io. 2021. Substrait: Cross-Language Serialization for Relational Algebra. https://github.com/substrait-io/substrait original-date: 2021-08-31T21:40:13Z
2021
-
[29]
Ji Sun and Guoliang Li. 2019. An end-to-end learning-based cost estimator. Proceedings of the VLDB Endowment 13, 3 (Nov. 2019), 307–319. https://doi.org/ 10.14778/3368289.3368296
2019
-
[30]
Marco Vogt, Alexander Stiemer, and Heiko Schuld. 2018. Polypheny-DB: Towards a Distributed and Self-Adaptive Polystore. In 2018 IEEE International Conference on Big Data (Big Data) . IEEE, 3364–3373
2018
-
[31]
Ziniu Wu, Ryan Marcus, Zhengchun Liu, Parimarjan Negi, Vikram Nathan, Pascal Pfeil, Gaurav Saxena, Mohammad Rahman, Balakrishnan Narayanaswamy, and Tim Kraska. 2024. Stage: Query Execution Time Prediction in Amazon Redshift. http://arxiv.org/abs/2403.02286 arXiv:2403.02286 [cs]
2024 arXiv
-
[32]
Zongheng Yang, Wei-Lin Chiang, Sifei Luan, Gautam Mittal, Michael Luo, and Ion Stoica. 2022. Balsa: Learning a Query Optimizer Without Expert Demonstrations. In Proceedings of the 2022 International Conference on Management of Data . ACM, Philadelphia PA USA, 931–944. https://...
2022
-
[33]
Yu, Ziniu Wu, Ferdi Kossmann, Tianyu Li, Markos Markakis, Amadou Ngom, Samuel Madden, and Tim Kraska
Geoffrey X. Yu, Ziniu Wu, Ferdi Kossmann, Tianyu Li, Markos Markakis, Amadou Ngom, Samuel Madden, and Tim Kraska. 2024. Blueprinting the Cloud: Unifying and Automatically Optimizing Cloud Data Infrastructures with BRAD – Extended Version. Proceedings of the VLDB Endowment 17, ...
2024
-
[34]
Xiang Yu, Guoliang Li, Chengliang Chai, and Nan Tang. 2020. Reinforcement Learning with Tree-LSTM for Join Order Selection. In2020 IEEE 36th International Conference on Data Engineering (ICDE). IEEE, Dallas, TX, USA, 1297–1308. https: //doi.org/10.1109/ICDE48307.2020.00116
2020
-
[35]
Jianqiu Zhang, Kaisong Huang, Tianzheng Wang, and King Lv. 2022. Skeena: Effi- cient and Consistent Cross-Engine Transactions. In2022 International Conference on Management of Data . ACM, 34–48
2022
-
[36]
Yue Zhao, Gao Cong, Jiachen Shi, and Chunyan Miao. 2022. QueryFormer: a tree transformer model for query plan representation. Proceedings of the VLDB Endowment 15, 8 (April 2022), 1658–1670. https://doi.org/10.14778/3529337. 3529349
2022 doi
-
[37]
Xiuwen Zheng, Subhasis Dasgupta, Arun Kumar Kumar, and Amarnath Gupta
-
[38]
Xuanhe Zhou, Guoliang Li, Chengliang Chai, and Jianhua Feng. 2021. A learned query rewrite system using Monte Carlo tree search. Proceedings of the VLDB Endowment 15, 1 (Sept. 2021), 46–58. https://doi.org/10.14778/3485450.3485456
2021
-
[39]
Rong Zhu, Wei Chen, Bolin Ding, Xingguang Chen, Andreas Pfadler, Ziniu Wu, and Jingren Zhou. 2023. Lero: A Learning-to-Rank Query Optimizer. Proceedings of the VLDB Endowment 16, 6 (Feb. 2023), 1466–1479. https://doi.org/10.14778/ 3583140.3583160 7 6 APPENDIX 6.1 Set-based mod...
2023
-
[2022]
In https://arxiv.org/pdf/2112.00833
AWESOME: Empowering Scalable Data Science on Social Media Data with an Optimized Tri-Store Data System. In https://arxiv.org/pdf/2112.00833
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.