{"id":"7bc2c74f-7d60-4b53-b7f4-56db1e384a56","arxiv_id":"2507.03384","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid LLM-and-encoder framework recommends query optimization hints offline and reports better speedup and latency than Bao, Lero, and FASTgres.","lead":"LLM4Hint combines a small language model with a lightweight query-plan encoder to recommend PostgreSQL hints for repeatedly run SQL queries. It shows speedups over existing learned optimizers on JOB, TPC-DS, and Stack workloads, at the cost of offline fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Execution-time labels in Eq. 2 are single measurements with no reported variance; noisy timing could make the four-class labels and all claimed SU/GMRL gains unreliable.","rationale":"The reader identified timing noise as the weakest assumption, and I agree that it is the single most load-bearing concern. The entire evaluation pipeline—ground-truth labels (Eq. 2), comparator training (Eq. 3), inference scoring (Algorithm 2), and the SU/GMRL metrics in Table 2—depends on execution-time measurements t(p_h_q). The paper provides no repeated measurements, no warm-up description, no variance reporting, and no error bars, despite the fact that DBMS execution times vary run-to-run. This is a correctness risk: if labels are noisy, the trained comparator learns wrong relationships and the final hint scores are unstable. It is also an externally verifiable concern: the gap to the best competitor is a 22% SU difference and 8.1% GMRL difference, which is plausible but must withstand remeasurement. The second concern (search-space optimality) is real but secondary: Oh* being defined over 16 hints is standard in this line of work, and the comparison to Bao/Lero/FASTgres is fair as long as competitors use comparable hint sets. The paper does deserve credit for publishing code and for ablating components (Tables 3-4), showing that the LLM contributes on string-heavy predicates and that results are not backbone-specific. Those internal checks do not, however, resolve external noise in raw timing data. The concrete test I propose (median-of-5 with bootstrap) would settle the concern: if the ranking survives, the central claim stands; if not, the paper needs either repeated-run methodology or claims scaled to reflect uncertainty. The verdict CONDITIONAL is appropriate: accept only after the timing methodology is clarified or the test confirms the numbers.","tokens_in":21480,"tokens_out":1822,"duration_ms":19028,"concrete_test":"Re-run the JOB-Q evaluation with each plan's execution time measured as the median of at least 5 runs after a warm-up run, and perturb labels by resampling measured times within their observed per-plan variance (or bootstrap across repeated runs) to see whether LLM4Hint's SU 1.70 / GMRL 0.79 still beats FASTgres's 1.39 / 0.86 and whether the per-hint scores in Algorithm 2 change for any test query. If the ranking reverses or the gap becomes statistically insignificant, the headline claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—consistent SU/GMRL improvement over Bao, Lero, and FASTgres—rests on ground-truth labels computed from single execution-time measurements t(p_h_q) in Algorithm 1 line 5 and Equation 2. The paper reports no repeated runs, warm-up, variance, or error bars anywhere, despite execution times in PostgreSQL on a shared-memory server being notoriously noisy (OS scheduling, cache effects, buffer state, parallel plan variability). The four-class thresholds are tight: r0 >= 1.5 or r1 >= 1000 ms separates 'much faster' from 'faster', and r0 in [2/3, 1) with r1 in [-1000, 0) separates 'poorer' from 'much poorer'. A single slow run can flip a label between adjacent classes, and even between label 1 (faster) and label 2 (poorer), directly changing the training signal for the comparator and the hint scores in Algorithm 2 (lines 9-10). Because SU and GMRL are computed from the same single measurements, the headline numbers in Table 2 have unknown error bars: the claimed 22.0% SU gain over FASTgres on JOB-Q (1.70 vs 1.39) and the 8.1% GMRL reduction could shift if execution times were remeasured. The paper also inherits the assumption that the 16 Bao hint combinations are the complete search space; Oh* is the upper bound within that set only. But the timing-noise issue is the most load-bearing because it threatens the validity of every reported number, not just the generality of the search space.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM4Hint, a learned hint-recommendation system for offline query optimization. It combines a lightweight tree-plan encoder (QueryFormer) with a moderate-sized backbone LLM (GPT-2) that receives a soft prompt from the encoder, an optional SQL-to-natural-language rewrite from GPT-4, and an explicit matching prompt linking table aliases to plan embeddings. A pairwise comparator with four classes ('much faster' through 'much poorer') ranks hints, and the best hint from a set of 16 Bao-style hint combinations is selected. Experiments on JOB, TPC-DS, and Stack compare LLM4Hint against Bao, Lero, and FASTgres using SU and GMRL, reporting consistent improvements; ablations isolate the contributions of the soft prompt, rewritten SQL, and matching text.","tokens_in":21863,"tokens_out":10544,"duration_ms":114826,"significance":"LLM4Hint addresses a timely and practical problem, and if the reported numbers are robust it is an interesting demonstration that LLM text understanding can be combined with a learned plan encoder to improve hint selection over existing learned optimizers. The paper is generally well structured, the system is described in enough detail to be reimplemented, the ablations are informative, and the anonymous code link is a positive reproducibility feature. The main caveats are that the headline results rest on single non-repeated execution-time measurements, the inference pseudocode appears to invert the comparator scores, and the template-level generalization experiments do not include competitor baselines. These issues prevent acceptance in the current form.","major_comments":[{"comment":"The ground-truth labels and all reported metrics depend on execution times t(p_h_q) that are collected as single measurements, with no repeated runs, warm-up, variance, or error bars reported anywhere in the paper. Because Equation 2 uses tight thresholds (ratio 3/2 and difference 1000 ms) to assign the four classes, timing noise can flip a label between adjacent classes, altering the comparator training signal and the hint scores in Algorithm 2. Since SU and GMRL in Table 2 are computed from the same single measurements, the claimed gains (e.g., 22.0% SU over FASTgres on JOB-Q, 1.70 vs 1.39) have unknown confidence intervals. Please repeat executions, report medians or means with variance, and add a sensitivity analysis showing that the headline comparisons are stable under timing noise.","section":"Section 4.1, Algorithm 1 line 5, Equation 2"},{"comment":"There is a direct inconsistency between the pseudocode and the surrounding text. Equation 2 defines y=0 as 'much faster' when r0 = t(p_h0)/t(p_h1) >= 3/2, i.e., h0 is much faster than h1. The text in Section 3.7 says that in this case h0 should receive the higher score (3) and h1 the lower score (1). However, Algorithm 2 line 10 sets Sh0 <- s and Sh1 <- 4 - s, so for s=0 the algorithm gives h0 zero points and h1 four points. Maximizing the cumulative score would therefore select the hint that was judged to be much slower. Please correct the pseudocode to match the stated scoring rule (or reverse the label convention), and verify that the released implementation follows the corrected rule.","section":"Algorithm 2 line 10 vs Section 3.7 text"},{"comment":"The template-level generalization experiments (JOB-T and TPC-DS-T) compare only LLM4Hint variants with and without the matching-text module; no competitor (Bao, Lero, or FASTgres) is evaluated under the same template-level split. The abstract claims that LLM4Hint outperforms state-of-the-art learned optimizers 'in terms of both effectiveness and generalization,' but the generalization part of that claim is not directly tested against competitors. Please add competitor results on the template-level splits, or explicitly restrict the generalization claim to the ablation setting where the evidence actually exists.","section":"Section 4.3, Table 4"}],"minor_comments":[{"comment":"The line 'Apply Mc(eh0_q, eh0_q)' appears to be a typo; it should likely read 'Apply Mc(eh0_q, eh1_q)' to compare the two plan embeddings.","section":"Algorithm 1 line 15"},{"comment":"Rows 3 and 4 of Table 3 both show the same three checkmarks under 'Rewritten SQL, Hint, Soft Prompt,' but the text compares them as original SQL versus rewritten SQL. Please add a column or other notation that distinguishes these two prompt states, otherwise the comparison is not visible in the table.","section":"Table 3"},{"comment":"The hint set is described as 'the first 16 hint combinations out of a total of 48' in Section 3.3 but as 'the top 16 efficient hint combinations' in Section 4.1. Please clarify how the 16 hints were selected and whether this choice favors Bao's hint space over those of Lero and FASTgres.","section":"Section 3.3 and Section 4.1"},{"comment":"The statement that Oh* 'represents the upper bound of the performance of all learned plan selection methods' is too strong: Oh* is optimal only within the preselected 16-hint search space, not over all possible plan-selection methods. Please qualify the claim.","section":"Section 4.1, Optimal Optimizer definition"},{"comment":"The phrase 'consistent improvements over all competitors in all datasets' is overstated because FASTgres is not evaluated on TPC-DS (marked with a dash), so the comparison is not complete across all datasets for all competitors.","section":"Section 4.2, Table 2"},{"comment":"The hand-set thresholds r0 and r1 are not justified, and the interaction between these thresholds and timing noise is not analyzed. A small sensitivity study around the threshold values would strengthen the paper, particularly given the single-measurement timing issue raised above.","section":"Section 3.6, Equation 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for cs.DB and the anonymous code link is a positive signal for reproducibility. The main risk is the timing-noise issue: if the authors cannot provide robust estimates of variance or repeated-run stability, the empirical claims would need to be substantially softened. The scoring inconsistency in Algorithm 2 should be resolved before the paper is reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read LLM4Hint. Bottom line: it is a genuine, well-scoped contribution to offline hint recommendation, and the specific concern about a missing LLM-only baseline is wrong—Table 3 includes an LLM-only row (SU 1.57 on JOB-Q). The real soft spot is the one the stress-test flags: every ground-truth label in Eq. 2 and every reported SU/GMRL number comes from a single execution-time measurement, with no repeated runs, warm-up, or error bars anywhere.\n\nWhat is new and good: the combination of a QueryFormer plan encoder with a moderate-sized LLM (GPT-2) via soft prompts is a sensible adaptation of the Time-LLM/TableGPT2 recipe to query hint selection. The GPT-4-based SQL rewriting and the explicit table-matching text are practical and, judged by the ablations, effective—especially the absolute matching text for template-level generalization. The paper compares fairly against Bao, Lero, and FASTgres on three workloads, reports a per-query breakdown, and ships anonymous code. The central design story holds up: the LLM helps most on queries with complex string predicates, which matches the stated motivation.\n\nSoft spots, in proportion. The timing-noise issue is load-bearing. With class thresholds as tight as r0 = 2/3 vs 1 and r1 = 1000 ms, a single slow or fast run can flip a label between 'faster' and 'poorer', directly changing the training signal and the final SU/GMRL. That does not make the claimed gains fake, but it means the headline numbers (e.g., 1.70 vs 1.39 on JOB-Q) have unknown error bars. This is fixable in revision: repeat executions, report medians and confidence intervals, and ideally release the measured times. A second, minor issue: the optimal upper bound is defined only over the 16 Bao hint combinations; the paper states this, but it should be emphasized when interpreting how much headroom remains. The template-level generalization experiments are only run for the matching module, not against competitors; that is a limitation, not a fatal one.\n\nOverall: this paper deserves a serious referee, but the referee should ask for variance analysis before the numeric claims are trusted. If the authors can show the results are stable across repeats, this is a solid VLDB/SIGMOD-style system paper.","headline":"A sensible offline hint recommender with a real experimental-rigor problem: the timing labels behind every reported number come from single runs with no variance reported.","tokens_in":22347,"tokens_out":2214,"would_cite":true,"duration_ms":28504,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM4Hint claims that combining a moderate-sized LLM with a lightweight plan encoder, using plan embeddings as a soft prompt, recommends query hints better than existing learned optimizers on recurring offline workloads.","keywords":["query optimization","hint recommendation","large language models","learned optimizer","soft prompt","execution plan","offline optimization","generalization"],"falsifier":"Re-run Algorithm 1's label collection on the same query-hint pairs multiple times and measure execution-time variance; if a substantial fraction of pairs change their four-class label across runs, LLM4Hint's reported SU and GMRL margins would not reproduce.","tokens_in":21320,"feed_emoji":"⚡","tokens_out":4313,"duration_ms":50368,"temperature":0.7,"pith_summary":"LLM4Hint is an offline query-hint recommender that pairs a small language model with a lightweight tree-plan encoder. The paper claims this combination understands both the SQL text and the structured execution plan better than previous learned optimizers, improving both effectiveness and generalization. On JOB-Q it reports a speedup of 1.70 versus 1.39 for the best prior competitor, and on template-level splits it maintains gains on unseen query templates. The practical stakes are that expensive hint selection for repeated enterprise SQL could be automated with one GPU and a one-time training pass.","feed_headline":"LLM4Hint recommends query hints better than learned optimizers","feed_subtitle":"On JOB, TPC-DS and Stack, it hits 1.70x speedup and generalizes across unseen query templates.","key_machinery":"The central mechanism is the combined LLM-lightweight model with a soft prompt: QueryFormer encodes tree-structured execution plans into a sequence of embeddings, a projection layer maps those embeddings into the LLM's vector space, and the LLM processes the rewritten SQL, the hint text, the soft prompt, and an explicit absolute matching text that names each table's embedding. A router-based comparator then makes four-way relative judgments about which of two plans is faster, using a weighted cross-entropy loss that emphasizes the much-faster and much-poorer classes.","core_discovery":"LLM4Hint is a selection-based optimizer for recurring queries: it enumerates execution plans under 16 PostgreSQL hints inherited from Bao, encodes each tree-structured plan with QueryFormer to produce a soft prompt, and feeds that prompt together with SQL text rewritten into natural language by a larger LLM to a moderate-sized backbone LLM such as GPT-2. A pairwise comparator trained with four relative-latency classes then picks the best hint. The paper reports consistent improvements over Bao, Lero, and FASTgres on JOB, TPC-DS, and Stack, with SU up to 1.70 and GMRL down to 0.79 on JOB-Q, and shows that an explicit matching prompt linking table names to plan embeddings is what makes template-level generalization work.","pith_inferences":["Going beyond the paper, the same soft-prompt alignment recipe could be tested on other DBMS hint spaces or on index-advisor tasks, since nothing in the mechanism is specific to PostgreSQL's hint syntax.","The paper does not isolate whether the LLM contributes genuine semantic understanding or simply extra model capacity; an ablation feeding identical plan embeddings through a non-LLM transformer of similar size would clarify this.","Because the ground-truth labels come from single execution-time measurements, a reproducibility check with repeated runs and reported variance would show how much of the claimed margin survives timing noise.","The search space is fixed to the 16 Bao hints, so expanding the hint set or letting the model propose hints dynamically could move performance toward the reported upper bound of 2.11 SU on JOB-Q."],"forward_implications":["If the central claim holds, recurring enterprise SQL workloads can be optimized automatically by offline hint recommendation without per-query LLM inference latency.","Template-level results imply the method can recommend hints for query templates never seen in training, which matters when workloads shift over time.","The SQL rewriting step is invoked once per query, so the main ongoing cost is a single GPU fine-tuning phase rather than repeated access to a large commercial model.","The four-class pairwise comparator gives the model a finer training signal than binary faster-or-slower labels, which could make hint selection more reliable when plan times differ by orders of magnitude."],"supporting_citations":[{"why":"Supplies the 16-hint search space and the Bao baseline that LLM4Hint must beat.","marker":"Marcus et al., [n. d.]"},{"why":"Establishes the pairwise relative-comparison setting and the Lero baseline.","marker":"Zhu et al., 2023"},{"why":"Provides the FASTgres baseline that LLM4Hint compares against.","marker":"Woltmann et al., 2023"},{"why":"Supplies the QueryFormer tree-transformer used as the lightweight plan encoder.","marker":"Zhao et al., 2022"},{"why":"Provides GPT-4 as the larger LLM used for SQL-to-natural-language rewriting.","marker":"Achiam et al., 2023"},{"why":"Supplies GPT-2, the default moderate-sized backbone LLM.","marker":"Radford et al., 2019"},{"why":"Provides the soft-prompt technique that motivates inserting plan embeddings into the LLM context.","marker":"Lester et al., 2021"},{"why":"Demonstrates the combined LLM-lightweight-model alignment approach that LLM4Hint extends.","marker":"Jin et al., 2024"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method tested on the backbone LLM.","marker":"Hu et al., 2021"}],"fun_headline_variants":["LLM4Hint: LLM-based hint recommendation beats learned optimizers","LLM4Hint uses soft prompts and LLMs to recommend query hints","LLM4Hint: generalizes to unseen query templates, up to 1.70x speedup","LLM4Hint: LLM rewriter and prompt alignment improve hint selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single execution-time measurements used to label each training pair are stable enough to be trusted, since noisy timings would corrupt the four-class labels and the learned comparisons.","fun_headline_variants_meta":{"raw":{"variants":["LLM4Hint: LLM-based hint recommendation beats learned optimizers","LLM4Hint uses soft prompts and LLMs to recommend query hints","LLM4Hint: generalizes to unseen query templates, up to 1.70x speedup","LLM4Hint: LLM rewriter and prompt alignment improve hint selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1532,"prompt_tokens":1017,"completion_tokens":515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":633,"tokens_out":515,"duration_ms":6083,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:11:51.215152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Algorithm 1's label collection on the same query-hint pairs multiple times and measure execution-time variance; if a substantial fraction of pairs change their four-class label across runs, LLM4Hint's reported SU and GMRL margins would not reproduce.","supporting_citations":[],"review_version":1}