{"id":"16048824-d334-4da9-95be-8d2c7e0aae75","arxiv_id":"2412.02372","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HERO combines an ensemble of context-aware models stored as a graph with a parameterized local search to recommend query hints and parallelism settings, reporting up to 3x speedup on JOB at lower inference cost than NN-based hint optimizers.","lead":"HERO is a learned query optimizer that recommends execution hints, including a parallelism setting, by reusing plans and latencies from similar past queries instead of training a neural network. In static benchmarks it reports near-optimal speedups with no degradations, though the headline result relies on seeing the test queries during exploration.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reliability claim is unsupported: static 0% degradation is a lookup over training queries, and HERO itself degrades on dynamic splits (Table 9), so the §4.2 threshold does not provide the advertised non-degradation guarantee.","rationale":"The paper's central value proposition is reliability: a hint recommender that can be deployed without degradation risk. That property is asserted in the abstract and formalized as a constraint in §3.3. The only experiments showing 0.0% degradation are the offline static ones, which, by §7.2's own description, use 'all data observable at training and unlimited exploration budget'—for HERO this means the optimal hint for each test query is already in the graph store, so the static result does not test prediction. The dynamic experiments (Table 9) are the real test of generalization, and HERO degrades or times out on every split. The §4.2 reliability threshold is the only mechanism that could prevent this, but the paper provides no proof that plan-statistics similarity bounds latency ratios; Table 3 shows the same default plan and the same hint can produce a speedup for one JOB query and a timeout for another. Thus the headline claim is either trivially true in the memorized static setting or empirically false in the predictive setting. The concrete test using the released dataset would determine whether the threshold separates such counterexample pairs; until then, conditional acceptance with revised claims is appropriate. This aligns with the reader's weakest-assumption analysis, so no verdict change is needed.","tokens_in":16675,"tokens_out":8296,"duration_ms":89898,"concrete_test":"Using the released dataset (github.com/zinchse/hero), enumerate all pairs of JOB/SQ queries with identical default logical plan whose plan-statistics distance is below the §4.2 reliability threshold. For each pair, apply the hint that is optimal for the first query to the second and compare the second query's latency against its default. If any such pair degrades or times out, the threshold does not guarantee non-degradation; if none do, the reliability claim can be validated empirically on the full dataset. Report the 6b/6d pair explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"HERO's headline reliability claim ('ensuring non-degradation of query latency', abstract and §3.3 constraint) is not established for unseen queries. The static 0.0% degradation result in Table 8 is measured in the offline scenario where, per §7.2, 'all data was observable at training and the exploration budget was unlimited': for every JOB query HERO can retrieve the stored optimal hint, making the setting closer to lookup than prediction. The predictive setting is Table 9, where HERO Local itself shows 4.2% timeouts and 4.2% degradations under the Structure Split, and nonzero degradation in other splits. The mechanism intended to prevent this, the distance threshold of §4.2, is heuristic: no theorem bounds the difference between a stored plan's latency and a new query's latency from similarity of default-plan statistics. The paper's own Table 3 is a counterexample at the plan level: 6b and 6d share the same default plan, and the same hint θops=126 yields a x1.62 speedup for 6b and a timeout for 6d. Unless the statistics distance separates these two, the claimed threshold cannot guarantee non-degradation; the paper does not show this separation. Thus the central reliability claim is either a tautology in the static setting or unsupported/false in the dynamic setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents HERO, a hint-based query optimizer that replaces the neural-network predictor used in systems such as Bao and AutoSteer with an ensemble of context-aware models organized as a graph, adds a parameterized local-search exploration procedure, and extends the hint space with degree-of-parallelism (dop) control. The central claims are that HERO ensures non-degradation of query latency, trains and infers faster than NN-based alternatives, and reaches near-optimal boosts on static benchmarks: 98.4% of the optimal boost on JOB with 0% timeouts and 0% degradations. The evaluation uses exhaustive plan-latency datasets collected on openGauss for JOB, SQ, and TPC-H, and compares HERO with reimplemented NN-based variants under static, online, and dynamic workload conditions.","tokens_in":16933,"tokens_out":10210,"duration_ms":118759,"significance":"The paper has real strengths: it releases an open dataset and benchmark platform, gives a detailed empirical account of why NN regressors fail on plan-collision-heavy data, and shows that a semantics-informed parameterized local search can find near-optimal hint sets faster than exhaustive search. A working non-neural, interpretable hint optimizer with low degradation risk and dop control would be a useful production-oriented contribution. That said, the current significance is conditional: the headline reliability guarantee is not supported by the dynamic experiments, and the static comparison is close to a lookup benchmark because the training data contains the answer for every test query. These issues materially change what the paper establishes.","major_comments":[{"comment":"The static-workload result is a lookup rather than a generalization test. According to Section 7.2, the offline scenario assumes 'all data was observable at training and the exploration budget was unlimited,' and Section 1 describes a dataset containing execution plans and latencies for every query and every hint combination. Under these conditions HERO can reconstruct the best stored hint for each JOB query at inference, so the 98.4% E2E Boost (% of Opt) and 0.0% degradations in Table 8 are expected consequences of exhaustive enumeration, not evidence that HERO predicts reliably for unseen queries. Because the abstract and introduction cite the static results as the basis for the reliability claim, this distinction is load-bearing and must be stated clearly.","section":"7.2, Table 8"},{"comment":"The non-degradation guarantee is not established and is contradicted by the dynamic workload results. The constraint in Section 3.3 requires t_i^M + t_i(theta) <= t_i(default) for every query in the support, yet Table 9 shows HERO Local with 4.2% timeouts and 4.2% degradations on the Structure Split. The mechanism intended to enforce this, the distance threshold of Section 4.2, is described only verbally; there is no theorem or calibration experiment showing that the plan-statistics distance bounds the latency difference between a stored plan and a new query's plan. Table 3 is a concrete counterexample at the plan level: queries 6b and 6d share the same default plan, and the same hint theta_ops=126 produces a x1.62 speedup for 6b and a timeout for 6d. Unless the distance metric provably separates such pairs, the threshold cannot deliver the advertised guarantee. The paper's own suggested mitigation, disabling underperforming context models in Section 7.3, is post-hoc and does not satisfy the formal constraint.","section":"3.3, 4.2, Table 9"},{"comment":"The central latency measurements are reported as single point estimates without repeated runs or confidence intervals. The dataset construction in Section 1 records one latency per query-hint combination on a pre-warmed, exclusively used server, and Tables 8 and 9 contain only single numbers for E2E boost, timeouts, degradations, and inference time. Query latencies on a real DBMS are noisy, so differences such as 64.6% versus 56.2% E2E boost, or inference times of 26.7s versus 58.5s, cannot be assessed for significance. This is especially important because the 'more than five times faster' inference claim and the 3x dop improvement are based on these point estimates.","section":"1, Tables 8 and 9"},{"comment":"The local-search parameterization is evaluated with parameters selected on the same benchmark used to report the result. The paper states that it tested 'tens of thousands of configurations' and then presents the best parameter sets per benchmark in Tables 5-7; these are then used for the comparison in Table 8. Without a separate tuning/validation split or a sensitivity analysis, the comparison between HERO's parameterized local search and the fixed greedy baseline may overstate the advantage, since the greedy baseline does not receive equivalent per-benchmark tuning. The note that the parameters are not universally optimal in Section 7.1 does not address this selection bias.","section":"7.1, Tables 5-7"},{"comment":"The comparison to existing NN-based systems is performed with the authors' own reimplementations rather than the original systems. Section 7 says 'we implemented a NN architecture that aligns with those in Bao and AutoSteer' and emulates QO-Advisor by pruning a local search. This is a reasonable controlled setup, but the paper's general statements that Bao suffers from 'long training and inference times' and 'unreliable predictions' in Section 2 are then conclusions about this emulation. The inference-time advantage 'more than five times faster' should be reported as relative to the authors' reimplementation, not to the production systems, and the implementation details should be documented to allow a fair comparison.","section":"5.2, 7"}],"minor_comments":[{"comment":"The objective and constraint equations contain typesetting errors and missing delimiters, so the formal problem statement is hard to parse; please rewrite them as numbered, self-contained equations.","section":"3.3"},{"comment":"The 'specialized distance metric' is described verbally but the statistics vector and the norm used for the relative difference are never defined; please give the exact formula.","section":"4.2"},{"comment":"The 'super-fast inference' mode based on query templates is not evaluated separately; please report its effectiveness and any accuracy cost relative to the standard inference path.","section":"4.3"},{"comment":"The '3x latency improvement' claim should state in the abstract that it is relative to the default openGauss execution time on JOB and is achieved by adding dop control on top of operator hints; Figure 2 makes this clearer than the abstract does.","section":"1, Abstract"},{"comment":"There are minor typographical issues: references [1] and [3] contain 'Acessed' instead of 'Accessed', the expression 'theta 0b0011000 = theta_24' is not formatted as a bitmask, and the em-dash in Section 4.3 is followed by a space before a period.","section":"References and notation"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection because the local-search contribution, the open dataset, and the NN failure analysis are salvageable and useful. However, the reliability/non-degradation claim as written is not defensible, and the static experiment cannot be presented as evidence of generalization. The authors should substantially revise the abstract and Section 3.3, or change the central claim to 'lower degradation risk on static workloads' with the dynamic results reported as a safety comparison rather than a guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Gist: HERO is worth a serious look, not because the static 98.4%-of-optimal number is a prediction—it's a lookup over precomputed latencies—but because the dop dimension, the graph-storage ensemble, and the semantics-informed local search are genuine additions to hint-based optimization, and the paper is unusually candid about where they fail.\n\nWhat's new: extending the hint space to degree-of-parallelism control, showing it unlocks additional speedups and interacts with join hints; an ensemble of per-default-plan context models with a plan-statistics distance metric; and a parameterized local search that trades exploration budget against quality, with concrete benchmark-specific guidance. The analysis of NN reliability in §6 is also useful, and the released datasets and code raise the reproducibility bar.\n\nSoft spots, in order. First, the reliability claim is oversold. The abstract and §3.3 state non-degradation as a hard constraint, but the static 0% degradation is measured after unlimited exploration where every JOB query's optimal hint is already in the store. The predictive setting is Table 9, where HERO Local itself shows 4.2% timeouts and 4.2% degradations on the structure split, and essentially zero gain on the speed splits. The distance threshold in §4.2 is a heuristic; Table 3 shows two queries with the same default plan and the same hint yielding a 1.62x speedup versus a timeout, and the paper never demonstrates that the statistics distance separates such cases. So the formal guarantee does not extend to unseen queries, and the paper should say so explicitly. Second, the headline latency numbers have no error bars; Table 8 is single-run. Third, the NN baselines are emulated rather than the original systems, which is defensible but should be signaled earlier in the paper.\n\nBottom line: the core engineering is sound, the dop insight is real, the dynamic evaluation is more honest than most, and the paper self-corrects in §7.3. It deserves a serious referee, with the main request being to align the abstract and problem statement with the dynamic results and report variance.","headline":"HERO is a solid systems paper with a real new knob (dop) and an honest dynamic-workload evaluation, but the headline non-degradation guarantee only holds in the static lookup setting, not for unseen queries.","tokens_in":17460,"tokens_out":2973,"would_cite":true,"duration_ms":32330,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hint-based query optimizer that uses no neural network reaches 98.4% of the best possible JOB speedup with zero timeouts and zero degradations, and runs more than five times faster at inference than learned NN-based optimizers.","keywords":["query optimization","hint-based optimization","learned optimizers","execution plan reuse","plan collisions","degree of parallelism","reliability","local search"],"falsifier":"Construct two queries with identical default logical plans and statistically similar plan vectors, then apply the hint HERO recommends for the first to the second; if the second query slows down dramatically while the first speeds up, the similarity assumption is broken and the non-degradation guarantee fails for that pair.","tokens_in":16468,"feed_emoji":"⚡","tokens_out":8875,"duration_ms":86397,"temperature":0.7,"pith_summary":"This paper tries to establish that reliable hint-based query optimization does not require a neural network, and that the usual tension between acceleration and safety can be broken. HERO recommends hints by reusing stored execution plans from queries that share the same default logical plan and similar plan statistics, and it refuses to recommend a hint when the current query is too far from anything previously seen. On the JOB benchmark with static workloads and unlimited training data, HERO achieves 98.4% of the best achievable speedup with zero timeouts and zero degradations, while the neural-network-based baselines degrade or time out even under those ideal conditions. Adding control over the degree of parallelism to the hint space raises the achievable speedup on JOB from roughly 2x to 3x, and HERO realizes that gain. The same components—the context-aware ensemble, the distance metric with a reliability threshold, and a parameterized local search—can also be grafted onto neural-network optimizers to make them safer.","feed_headline":"Neural-free hint optimizer hits 98.4% of optimal query boost","feed_subtitle":"Matches learned neural optimizers on the JOB benchmark with zero timeouts and zero degradations at 5x faster inference.","key_machinery":"The central object is the ensemble of context-aware models organized as a graph storage. Vertices are observed execution plans; edges record the hint that transforms one plan into another along with the resulting latency boost. For a new query, HERO generates the default plan, finds the closest model by the distance metric (logical equivalence plus statistics-vector similarity), then follows graph edges to locate promising hints without repeated planning, and only executes the chosen hint combination if its distance to known plans stays below the reliability threshold. The second piece is the parameterized local search procedure: its parameters control which operator families (joins, scans) and which degrees of parallelism are adjusted jointly, how many iterations are run, and whether the Index-Nested-Loop join is handled specially, turning exponential search into a budget-controlled exploration that outperforms greedy search.","core_discovery":"The paper's central claim is that query latency under a hint depends on the hint only through the execution plan it produces, and that two queries sharing the same default logical plan and similar plan-statistics vectors will respond to the same hint in the same way. On that basis, HERO replaces the neural network with an ensemble of context-aware models: each model is attached to one default plan, stores the custom plans, hints, and observed speedups for queries with that default, and the inference schema reuses previously stored plans from a graph instead of calling the planner repeatedly. A distance metric—infinitely large when logical structures differ, otherwise the relative difference in the plans' statistics vectors—picks the most relevant model and also refuses prediction when the candidate plan is too far from anything seen before, which is what formally provides the non-degradation guarantee. The paper further claims that extending the hint space with degree-of-parallelism control is necessary to unlock the full potential of operator hints, and that a semantics-informed parameterized local search, unlike greedy search, finds near-optimal hint sets within a given exploration budget.","pith_inferences":["The plan-collision principle—identical logical plans can hide opposite responses to the same hint—suggests that any learned optimizer that conditions only on a plan, without query context, faces an inherent accuracy ceiling; clustering by default-plan plus statistics is one way out, and clustering by richer context could improve dynamic-workload behavior.","The abstention mechanism could serve as a general safety layer for learned query optimizers: attach the same distance threshold to any hint recommender so that it says 'no hint' when uncertain, independent of whether the recommender is neural or not.","Because the experimental measurements were collected on a single database system with a fixed default degree of parallelism, the 3x dop gain and the reliability threshold should be re-tested on another relational engine before the results are taken as engine-agnostic."],"forward_implications":["If the reliability guarantee holds, deployment becomes safer: when no close match is found, HERO falls back to the default plan, so an unfamiliar query cannot be slowed down by a bad hint.","Because inference reuses stored plans instead of planning under many hint combinations, per-query inference cost drops by more than five times compared to learned NN-based hint optimizers, making hinting practical on large search spaces.","Adding degree-of-parallelism control roughly doubles the potential speedup of hint-based optimization on JOB, from about 2x to 3x, and HERO attains that 3x gain.","NN-based hint optimizers can inherit part of HERO's safety: using its pruned local search as their inference strategy reduces their inference overhead and degradation rate, although they remain less reliable than HERO on dynamic workloads.","The parameterization of local search gives users an explicit dial to trade exploration time against achieved speedup, a flexibility that fixed greedy or exhaustive search does not offer."],"supporting_citations":[{"why":"The database system on which every execution plan and latency in the experiments was collected; supplies the experimental substrate.","marker":"[1]"},{"why":"The greedy steering approach whose inference strategy HERO's parameterized local search is compared against and shown to outperform.","marker":"[4]"},{"why":"The JOB benchmark, the primary workload for measuring optimization boost, timeouts, and degradations.","marker":"[10]"},{"why":"The first hint-based learned optimizer, which serves as the main NN-based baseline and whose exhaustive planning HERO replaces with graph reuse.","marker":"[12]"},{"why":"The tree-convolutional neural architecture used to build the NN baselines whose reliability limits the paper analyzes.","marker":"[16]"},{"why":"The context-aware ensemble approach whose design HERO adapts while adding plan information and a reliability metric.","marker":"[23]"},{"why":"The production system that mitigates degradation by A/B testing; HERO's abstention mechanism targets the same risk without extra staging.","marker":"[25]"}],"fun_headline_variants":["HERO guarantees no slowdown, 3x faster on JOB","Reliable query hints without neural nets: 3x speedup","Hint optimizer with zero degradation, 5x faster inference","Context-aware query hints: non-degrading, 3x JOB speedup","HERO: efficient, reliable query optimization via hints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that queries sharing the same default logical plan and similar plan-statistics vectors will respond to the same hints in the same way, so that the distance metric and its reliability threshold truly guarantee non-degradation.","fun_headline_variants_meta":{"raw":{"variants":["HERO guarantees no slowdown, 3x faster on JOB","Reliable query hints without neural nets: 3x speedup","Hint optimizer with zero degradation, 5x faster inference","Context-aware query hints: non-degrading, 3x JOB speedup","HERO: efficient, reliable query optimization via hints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1316,"prompt_tokens":946,"completion_tokens":370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":562,"tokens_out":370,"duration_ms":4919,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:31:48.821727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two queries with identical default logical plans and statistically similar plan vectors, then apply the hint HERO recommends for the first to the second; if the second query slows down dramatically while the first speeds up, the similarity assumption is broken and the non-degradation guarantee fails for that pair.","supporting_citations":[{"cited_title":"openGauss","cited_arxiv_id":null,"evidence_quote":"The database system on which every execution plan and latency in the experiments was collected; supplies the experimental substrate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The greedy steering approach whose inference strategy HERO's parameterized local search is compared against and shown to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The tree-convolutional neural architecture used to build the NN baselines whose reliability limits the paper analyzes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The context-aware ensemble approach whose design HERO adapts while adding plan information and a reliability metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The production system that mitigates degradation by A/B testing; HERO's abstention mechanism targets the same risk without extra staging."}],"review_version":1}