REVIEW 5 major objections 5 minor 2 cited by
HERO: Hint-Based Efficient and Reliable Query Optimizer
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A hint-based query optimizer that uses no neural network reaches 98.4% of the best possible JOB speedup with zero timeouts and zero degradations, and runs more than five times faster at inference than learned NN-based optimizers.
desk verdict HERO is a solid systems paper with a real new knob (dop) and an honest dynamic-workload evaluation, but the headline non-degradation guarantee only holds in the static lookup setting, not for unseen queries. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ensemble of context-aware models organized as a graph storage. Vertices are observed execution plans; edges record the hint that transforms one plan into another along with the resulting latency boost. For a new query, HERO generates the default plan, finds the closest model by the distance metric (logical equivalence plus statistics-vector similarity), then follows graph edges to locate promising hints without repeated planning, and only executes the chosen hint combination if its distance to known plans stays below the reliability threshold. The second piece is the parameterized local search procedure: its parameters control which operator families (joins, scans) and which degrees of parallelism are adjusted jointly, how many iterations are run, and whether the Index-Nested-Loop join is handled specially, turning exponential search into a budget-controlled exploration that outperforms greedy search.
What would settle it
Construct two queries with identical default logical plans and statistically similar plan vectors, then apply the hint HERO recommends for the first to the second; if the second query slows down dramatically while the first speeds up, the similarity assumption is broken and the non-degradation guarantee fails for that pair.
Extended reading notes
Core claim
The paper's central claim is that query latency under a hint depends on the hint only through the execution plan it produces, and that two queries sharing the same default logical plan and similar plan-statistics vectors will respond to the same hint in the same way. On that basis, HERO replaces the neural network with an ensemble of context-aware models: each model is attached to one default plan, stores the custom plans, hints, and observed speedups for queries with that default, and the inference schema reuses previously stored plans from a graph instead of calling the planner repeatedly. A distance metric—infinitely large when logical structures differ, otherwise the relative difference in the plans' statistics vectors—picks the most relevant model and also refuses prediction when the candidate plan is too far from anything seen before, which is what formally provides the non-degradation guarantee. The paper further claims that extending the hint space with degree-of-parallelism control is necessary to unlock the full potential of operator hints, and that a semantics-informed parameterized local search, unlike greedy search, finds near-optimal hint sets within a given exploration budget.
Load-bearing premise
The load-bearing premise is that queries sharing the same default logical plan and similar plan-statistics vectors will respond to the same hints in the same way, so that the distance metric and its reliability threshold truly guarantee non-degradation.
Editorial extensions
If this is right
- If the reliability guarantee holds, deployment becomes safer: when no close match is found, HERO falls back to the default plan, so an unfamiliar query cannot be slowed down by a bad hint.
- Because inference reuses stored plans instead of planning under many hint combinations, per-query inference cost drops by more than five times compared to learned NN-based hint optimizers, making hinting practical on large search spaces.
- Adding degree-of-parallelism control roughly doubles the potential speedup of hint-based optimization on JOB, from about 2x to 3x, and HERO attains that 3x gain.
- NN-based hint optimizers can inherit part of HERO's safety: using its pruned local search as their inference strategy reduces their inference overhead and degradation rate, although they remain less reliable than HERO on dynamic workloads.
- The parameterization of local search gives users an explicit dial to trade exploration time against achieved speedup, a flexibility that fixed greedy or exhaustive search does not offer.
Reading between the lines
- The plan-collision principle—identical logical plans can hide opposite responses to the same hint—suggests that any learned optimizer that conditions only on a plan, without query context, faces an inherent accuracy ceiling; clustering by default-plan plus statistics is one way out, and clustering by richer context could improve dynamic-workload behavior.
- The abstention mechanism could serve as a general safety layer for learned query optimizers: attach the same distance threshold to any hint recommender so that it says 'no hint' when uncertain, independent of whether the recommender is neural or not.
- Because the experimental measurements were collected on a single database system with a fixed default degree of parallelism, the 3x dop gain and the reliability threshold should be re-tested on another relational engine before the results are taken as engine-agnostic.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents HERO, a hint-based query optimizer that replaces the neural-network predictor used in systems such as Bao and AutoSteer with an ensemble of context-aware models organized as a graph, adds a parameterized local-search exploration procedure, and extends the hint space with degree-of-parallelism (dop) control. The central claims are that HERO ensures non-degradation of query latency, trains and infers faster than NN-based alternatives, and reaches near-optimal boosts on static benchmarks: 98.4% of the optimal boost on JOB with 0% timeouts and 0% degradations. The evaluation uses exhaustive plan-latency datasets collected on openGauss for JOB, SQ, and TPC-H, and compares HERO with reimplemented NN-based variants under static, online, and dynamic workload conditions.
Significance. The paper has real strengths: it releases an open dataset and benchmark platform, gives a detailed empirical account of why NN regressors fail on plan-collision-heavy data, and shows that a semantics-informed parameterized local search can find near-optimal hint sets faster than exhaustive search. A working non-neural, interpretable hint optimizer with low degradation risk and dop control would be a useful production-oriented contribution. That said, the current significance is conditional: the headline reliability guarantee is not supported by the dynamic experiments, and the static comparison is close to a lookup benchmark because the training data contains the answer for every test query. These issues materially change what the paper establishes.
major comments (5)
- [7.2, Table 8] The static-workload result is a lookup rather than a generalization test. According to Section 7.2, the offline scenario assumes 'all data was observable at training and the exploration budget was unlimited,' and Section 1 describes a dataset containing execution plans and latencies for every query and every hint combination. Under these conditions HERO can reconstruct the best stored hint for each JOB query at inference, so the 98.4% E2E Boost (% of Opt) and 0.0% degradations in Table 8 are expected consequences of exhaustive enumeration, not evidence that HERO predicts reliably for unseen queries. Because the abstract and introduction cite the static results as the basis for the reliability claim, this distinction is load-bearing and must be stated clearly.
- [3.3, 4.2, Table 9] The non-degradation guarantee is not established and is contradicted by the dynamic workload results. The constraint in Section 3.3 requires t_i^M + t_i(theta) <= t_i(default) for every query in the support, yet Table 9 shows HERO Local with 4.2% timeouts and 4.2% degradations on the Structure Split. The mechanism intended to enforce this, the distance threshold of Section 4.2, is described only verbally; there is no theorem or calibration experiment showing that the plan-statistics distance bounds the latency difference between a stored plan and a new query's plan. Table 3 is a concrete counterexample at the plan level: queries 6b and 6d share the same default plan, and the same hint theta_ops=126 produces a x1.62 speedup for 6b and a timeout for 6d. Unless the distance metric provably separates such pairs, the threshold cannot deliver the advertised guarantee. The paper's own suggested mitigation, disabling underperforming context models in Section 7.3, is post-hoc and does not satisfy the formal constraint.
- [1, Tables 8 and 9] The central latency measurements are reported as single point estimates without repeated runs or confidence intervals. The dataset construction in Section 1 records one latency per query-hint combination on a pre-warmed, exclusively used server, and Tables 8 and 9 contain only single numbers for E2E boost, timeouts, degradations, and inference time. Query latencies on a real DBMS are noisy, so differences such as 64.6% versus 56.2% E2E boost, or inference times of 26.7s versus 58.5s, cannot be assessed for significance. This is especially important because the 'more than five times faster' inference claim and the 3x dop improvement are based on these point estimates.
- [7.1, Tables 5-7] The local-search parameterization is evaluated with parameters selected on the same benchmark used to report the result. The paper states that it tested 'tens of thousands of configurations' and then presents the best parameter sets per benchmark in Tables 5-7; these are then used for the comparison in Table 8. Without a separate tuning/validation split or a sensitivity analysis, the comparison between HERO's parameterized local search and the fixed greedy baseline may overstate the advantage, since the greedy baseline does not receive equivalent per-benchmark tuning. The note that the parameters are not universally optimal in Section 7.1 does not address this selection bias.
- [5.2, 7] The comparison to existing NN-based systems is performed with the authors' own reimplementations rather than the original systems. Section 7 says 'we implemented a NN architecture that aligns with those in Bao and AutoSteer' and emulates QO-Advisor by pruning a local search. This is a reasonable controlled setup, but the paper's general statements that Bao suffers from 'long training and inference times' and 'unreliable predictions' in Section 2 are then conclusions about this emulation. The inference-time advantage 'more than five times faster' should be reported as relative to the authors' reimplementation, not to the production systems, and the implementation details should be documented to allow a fair comparison.
minor comments (5)
- [3.3] The objective and constraint equations contain typesetting errors and missing delimiters, so the formal problem statement is hard to parse; please rewrite them as numbered, self-contained equations.
- [4.2] The 'specialized distance metric' is described verbally but the statistics vector and the norm used for the relative difference are never defined; please give the exact formula.
- [4.3] The 'super-fast inference' mode based on query templates is not evaluated separately; please report its effectiveness and any accuracy cost relative to the standard inference path.
- [1, Abstract] The '3x latency improvement' claim should state in the abstract that it is relative to the default openGauss execution time on JOB and is achieved by adding dop control on top of operator hints; Figure 2 makes this clearer than the abstract does.
- [References and notation] There are minor typographical issues: references [1] and [3] contain 'Acessed' instead of 'Accessed', the expression 'theta 0b0011000 = theta_24' is not formatted as a bitmask, and the em-dash in Section 4.3 is followed by a space before a period.
Circularity Check
Headline static-workload result is a lookup, not a prediction: HERO's 98.4%-of-optimal and 0.0%-degradation on JOB are obtained by retrieving exhaustively precomputed hint latencies for the same queries, making the central quality/reliability claim partially circular; the dynamic results give only partial independent support.
-
fitted input called prediction
[Method of Experimental Evaluation (p.1) and §7.2, Table 8]
"for every query from these benchmarks and for all possible hint combinations, we saved execution plans and their latencies obtained on openGauss DB [1]. This allowed us to replace actual query execution with a simple table lookup. ... The results of comparing HERO and NN-based solutions in conditions, where all data was observable at training and the exploration budget was unlimited, are presented in Table 8."
HERO's inference is explicitly storage-based: §4.2 stores 'observed performance gains for queries with the same p0', and §4.3 says 'we store directly obtained prior plans and use them to avoid unnecessary planning calls'. In the §7.2 offline scenario every JOB query and every hint combination has already been executed and saved, so for each test query HERO retrieves the best explored hint rather than predicting one for an unseen query. The reported 98.4% of optimal boost and 0.0% degradations therefore equal the best precomputed value in the table with inference overhead subtracted; they are a property of the exhaustive dataset, not of the contextual-transfer machinery that the paper claims provides reliability.
full rationale
The central derivation chain for HERO's headline reliability claim has one genuinely circular link. The static-workload result, which produces the striking 98.4%-of-optimal / 0.0%-degradation numbers, is measured under conditions where 'all data was observable at training and the exploration budget was unlimited', and the paper's own methodology says all queries and all hint combinations were pre-executed and stored so that 'actual query execution' could be replaced by 'a simple table lookup'. Since HERO's ensemble is literally a graph of stored plans and observed gains, the static evaluation reduces to retrieving the stored optimum for the same queries; calling this 'reliable hint recommendation' makes the quality metric definitionally equal to its input. This is not the whole paper, however: the dynamic-workload experiments in Table 9 provide an independent, genuinely predictive setting, and there HERO is safer than the NN baselines (e.g., 0% degradations on three of four splits), which prevents a score of 8 or 10. The paper also contains a non-circular but serious evidentiary gap: §3.3 promises non-degradation for all queries, §4.2 claims the distance threshold ensures this, yet Table 3 shows two queries with the same default plan where the same hint gives x1.62 speedup for 6b and a timeout for 6d, and Table 9 shows HERO itself degrading on the Structure Split; the paper explicitly concedes HERO 'may occasionally arrive at hint sets that slow down queries'. Thus the reliability guarantee is unsupported, and the static evidence for it is circular, even though the underlying retrieval architecture and dynamic comparison have independent content. No load-bearing self-citation or imported uniqueness theorem is present.
Assumptions & free parameters
free parameters (3)
- Local search configuration per benchmark =
JOB: Beta=10.0, Join/Scan/dop/Joinxdop/INL enabled, Iter=infinity; TPCH: Beta=1.0, Join/Scan/dop/Joinxdop enabled…
- Reliability distance threshold =
not specified
- Candidate dop values =
not specified beyond default of 64 threads
assumptions (4)
- domain assumption Query latency depends on hints only via their influence on the plan (O2, Section 3.2).
- domain assumption Queries sharing the same default plan and similar plan statistics respond similarly to the same hint (plan collision similarity, Section 4.1).
- domain assumption Search space assumptions A1-A4: joins matter more than scans; join and dop joint search is efficient; few iterations suffice; INL on large relations is the common mistake (Section 4.5).
- domain assumption openGauss implements hints by heavily overestimating costs of alternative plans, so costs are excluded from plan encoding (Section 5.1).
Cite this review
Pith. "Pith review of HERO: Hint-Based Efficient and Reliable Query Optimizer." pith.science (2026). https://pith.science/paper/MABOV7LN
@misc{pith2026241202372,
author = {Pith},
title = {Pith review of: HERO: Hint-Based Efficient and Reliable Query Optimizer},
year = {2026},
howpublished = {\url{https://pith.science/paper/MABOV7LN}},
note = {Machine review of arXiv:2412.02372}
}
read the original abstract
We propose a novel model for learned query optimization which provides query hints leading to better execution plans. The model addresses the three key challenges in learned hint-based query optimization: reliable hint recommendation (ensuring non-degradation of query latency), efficient hint exploration, and fast inference. We provide an in-depth analysis of existing NN-based approaches to hint-based optimization and experimentally confirm the named challenges for them. Our alternative solution consists of a new inference schema based on an ensemble of context-aware models and a graph storage for reliable hint suggestion and fast inference, and a budget-controlled training procedure with a local search algorithm that solves the issue of exponential search space exploration. In experiments on standard benchmarks, our model demonstrates optimization capability close to the best achievable with coarse-grained hints. Controlling the degree of parallelism (query dop) in addition to operator-related hints enables our model to achieve 3x latency improvement on JOB benchmark which sets a new standard for optimization. Our model is interpretable and easy to debug, which is particularly important for deployment in production.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 2 Pith papers
-
Learned Offline Query Planning via Bayesian Optimization
BayesQO combines variational autoencoders and Bayesian optimization with censored timeouts to discover faster join-order plans offline for repetitive analytic workloads.
-
Training-Free Query Optimization via LLM-Based Plan Similarity
LLM-PM, a training-free system that transfers optimizer hints from nearest-neighbour LLM plan embeddings, reports a 21.1% total runtime reduction on JOB-CEB over openGauss.
Reference graph
Works this paper leans on
- [1]
-
[2]
[n.d.]. pg_hint_plan Documentation. https://pg-hint-plan.readthedocs.io/en/ latest/. Accessed: 2024-10-10
work page 2024
-
[3]
[n.d.]. PostgreSQL. https://www.postgresql.org/about/. Acessed: 2024-10-10
work page 2024
-
[4]
Christoph Anneser, Nesime Tatbul, David Cohen, Zhenggang Xu, Prithviraj Pandian, Nikolay Laptev, and Ryan Marcus. 2023. Autosteer: Learned query 12 optimization for any sql database. Proceedings of the VLDB Endowment 16, 12 (2023), 3515–3527
work page 2023
-
[5]
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Van- dergheynst. 2017. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine 34, 4 (2017), 18–42
work page 2017
-
[6]
Surajit Chaudhuri. 1998. An overview of query optimization in relational systems. In Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems . 34–43
1998
-
[7]
Surajit Chaudhuri, Umeshwar Dayal, and Vivek Narasayya. 2011. An overview of business intelligence technology. Commun. ACM 54, 8 (2011), 88–98
work page 2011
-
[8]
Transaction Processing Council. [n.d.]. TPC-H. https://www.tpc.org/tpch/. Accessed: 2024-10-10
work page 2024
Show all 27 references
-
[9]
Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. 2018. Learned cardinalities: Estimating correlated joins with deep learning. arXiv preprint arXiv:1809.00677 (2018)
2018 arXiv
-
[10]
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann. 2015. How good are query optimizers, really? Proceedings of the VLDB Endowment 9, 3 (2015), 204–215
2015
-
[11]
Guy Lohman. 2014. Is query optimization a “solved” problem. In Proc. Workshop on Database Query Optimization , Vol. 13. Oregon Graduate Center Comp. Sci. Tech. Rep, 10
2014
-
[12]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Al- izadeh, and Tim Kraska. 2020. Bao: Learning to steer query optimizers. arXiv preprint arXiv:2004.03814 (2020)
2020 arXiv
-
[13]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Al- izadeh, and Tim Kraska. 2021. Bao: Making learned query optimization practical. In Proceedings of the 2021 International Conference on Management of Data . 1275– 1288
2021
-
[14]
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019. Neo: A learned query optimizer. arXiv preprint arXiv:1904.03711 (2019)
2019 arXiv
-
[15]
Ryan Marcus and Olga Papaemmanouil. 2018. Deep reinforcement learning for join order enumeration. In Proceedings of the First International Workshop on Exploiting Artificial Intelligence Techniques for Data Management . 1–4
2018
-
[16]
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural net- works over tree structures for programming language processing. In Proceedings of the AAAI conference on artificial intelligence
2016
-
[17]
Parimarjan Negi, Matteo Interlandi, Ryan Marcus, Mohammad Alizadeh, Tim Kraska, Marc Friedman, and Alekh Jindal. 2021. Steering query optimizers: A practical take on big data workloads. In Proceedings of the 2021 International Conference on Management of Data . 2557–2569
2021
-
[18]
Parimarjan Negi, Ryan Marcus, Andreas Kipf, Hongzi Mao, Nesime Tatbul, Tim Kraska, and Mohammad Alizadeh. 2021. Flow-loss: Learning cardinality estimates that matter. arXiv preprint arXiv:2101.04964 (2021)
2021 arXiv
-
[19]
Parimarjan Negi, Ryan Marcus, Hongzi Mao, Nesime Tatbul, Tim Kraska, and Mohammad Alizadeh. 2020. Cost-guided cardinality estimation: Focus where it matters. In 2020 IEEE 36th International Conference on Data Engineering Workshops (ICDEW). IEEE, 154–157
2020
-
[20]
Jennifer Ortiz, Magdalena Balazinska, Johannes Gehrke, and S Sathiya Keerthi
-
[21]
Tarique Siddiqui, Alekh Jindal, Shi Qiao, Hiren Patel, and Wangchao Le. 2020. Cost models for big data query processing: Learning, retrofitting, and our findings. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 99–113
2020
-
[22]
Ji Sun, Jintao Zhang, Zhaoyan Sun, Guoliang Li, and Nan Tang. 2021. Learned cardinality estimation: A design space exploration and a comparative evaluation. Proceedings of the VLDB Endowment 15, 1 (2021), 85–97
2021
-
[23]
Lucas Woltmann, Jerome Thiessat, Claudio Hartmann, Dirk Habich, and Wolf- gang Lehner. 2023. Fastgres: Making learned query optimizer hinting effective. Proceedings of the VLDB Endowment 16, 11 (2023), 3310–3322
2023
-
[24]
Zongheng Yang, Wei-Lin Chiang, Sifei Luan, Gautam Mittal, Michael Luo, and Ion Stoica. 2022. Balsa: Learning a query optimizer without expert demonstrations. In Proceedings of the 2022 International Conference on Management of Data. 931–944
2022
-
[25]
Wangda Zhang, Matteo Interlandi, Paul Mineiro, Shi Qiao, Nasim Ghazanfari, Karlen Lie, Marc Friedman, Rafah Hosn, Hiren Patel, and Alekh Jindal. 2022. Deploying a steered query optimizer in production at microsoft. In Proceedings of the 2022 International Conference on Managem...
2022
-
[26]
Yue Zhao, Zhaodonghui Li, and Gao Cong. 2023. A Comparative Study and Com- ponent Analysis of Query Plan Representation Techniques in ML4DB Studies. Proceedings of the VLDB Endowment 17, 4 (2023), 823–835. 13
2023
-
[2019]
arXiv preprint arXiv:1905.06425 (2019)
An empirical analysis of deep learning for cardinality estimation. arXiv preprint arXiv:1905.06425 (2019)
2019 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.