{"id":"23e40e1b-5adc-47b6-8cb6-86bf6ea24292","arxiv_id":"2606.04866","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper establishes the first distribution-dependent sample complexity bounds showing that informative priors reduce required evaluations in multi-fidelity HPO while uninformative priors recover baseline performance.","lead":"This paper derives the first distribution-dependent sample complexity bounds for multi-fidelity hyperparameter optimization with priors, framed as fixed-budget best-arm identification. A smart generalist might read it to see how prior knowledge can provably cut the computing cost of tuning machine learning models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Modeling priors directly over arm means leaves unspecified how multi-fidelity bias and cost structure enter the distribution-dependent bounds","rationale":"The reader correctly flags the modeling step as the weakest link, but the multi-fidelity setting introduces an extra modeling choice (how lower-fidelity observations update beliefs about the high-fidelity mean) that is not addressed in the abstract and could invalidate the budget-reduction statement even if the single-fidelity case is correct.","tokens_in":1701,"tokens_out":361,"duration_ms":24680,"concrete_test":"Extract the precise statement of the main error bound (presumably in the theoretical analysis section) and instantiate it for a two-fidelity problem with known additive bias b at the low fidelity and cost ratio c_low : c_high = 1 : 10. Compute the minimal total budget B required to keep error probability ≤ δ both with and without an informative prior; if the reduction in B disappears once the bias term is restored, the multi-fidelity claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"For the central claim to hold, the explicit error bounds must show that an informative prior (concentrating mass near the optimal high-fidelity mean) reduces the minimal budget needed to reach a target error probability, even when most pulls occur at cheaper but biased lower fidelities. The abstract states only that priors are placed on arm means “as configuration performance” and that the resulting bounds are distribution-dependent; it supplies no indication that the derivation incorporates a fidelity-dependent observation model, bias term, or cost-weighted allocation. If the bounds treat every evaluation as a direct, unbiased observation of the target mean, the claimed reduction in total compute (rather than raw evaluation count) does not necessarily follow.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper claims to provide the first distribution-dependent sample complexity bounds for multi-fidelity hyperparameter optimization with priors, derived via the fixed-budget best-arm identification framework. By modeling priors directly over arm means (as configuration performance), it derives explicit error bounds quantifying how informative priors (concentrating mass near optimal arms) reduce required evaluations, while uninformative or misleading priors recover baseline performance. Proof-of-concept experiments on synthetic data and LCBench are reported to achieve up to 90% budget reduction while retaining solution quality.","tokens_in":1857,"tokens_out":325,"duration_ms":13206,"significance":"If the bounds correctly handle multi-fidelity structure, the work supplies a principled, quantitative link between prior informativeness and evaluation budget in HPO, offering a foundation for compute-efficient AutoML. The explicit distribution-dependent bounds and experimental validation on a standard benchmark constitute strengths that enable falsifiable predictions.","major_comments":[{"comment":"Abstract and theoretical analysis: the claim of multi-fidelity HPO bounds with reduced sample cost is not supported by the stated modeling. The derivation places priors directly over arm means as configuration performance and produces distribution-dependent error bounds, but supplies no indication that a fidelity-dependent observation model, bias term, or cost-weighted allocation enters the bounds. If every evaluation is treated as a direct, unbiased observation of the target mean, the claimed reduction in total compute (rather than raw evaluation count) does not necessarily follow.","section":"theoretical analysis"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. The feedback highlights an important gap between the modeling assumptions and the multi-fidelity claims, which we address directly below.","responses":[{"response":"We agree that the current theoretical derivation models each observation as an unbiased sample from the target arm mean and does not yet incorporate a fidelity-dependent observation model, bias terms for lower-fidelity evaluations, or explicit cost-weighted allocation. Consequently, the stated bounds quantify reductions in the number of evaluations rather than total computational cost. To resolve this, we will revise the theoretical analysis section to introduce a multi-fidelity observation model (with fidelity-specific bias and variance) and a cost function that enters the fixed-budget allocation rule. The revised bounds will then explicitly relate prior informativeness to reductions in total compute while recovering the original results as a special case when all evaluations are at full fidelity.","revision_made":"yes","referee_comment":"[theoretical analysis] Abstract and theoretical analysis: the claim of multi-fidelity HPO bounds with reduced sample cost is not supported by the stated modeling. The derivation places priors directly over arm means as configuration performance and produces distribution-dependent error bounds, but supplies no indication that a fidelity-dependent observation model, bias term, or cost-weighted allocation enters the bounds. If every evaluation is treated as a direct, unbiased observation of the target mean, the claimed reduction in total compute (rather than raw evaluation count) does not necessarily follow."}],"tokens_in":1310,"tokens_out":317,"duration_ms":11260,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main contribution is a set of explicit, distribution-dependent error bounds for fixed-budget best-arm identification when priors are placed directly on arm means. Informative priors that concentrate mass near the optimum reduce the number of evaluations needed; uninformative or bad priors recover the usual rates. That framing is new for this setting and gives a quantitative handle on something that has mostly been handled heuristically.\n\nThe experiments on synthetic data and LCBench are presented as confirmation and report up to 90% budget reduction while keeping solution quality. That is useful evidence if the runs actually track the derived bounds.\n\nThe soft spot is the multi-fidelity part. The abstract models priors on configuration performance but gives no indication that the bounds include a fidelity-dependent observation model, bias correction, or cost-weighted allocation. If every evaluation is treated as an unbiased draw from the target mean, the claimed reduction in total compute does not automatically follow from a reduction in raw evaluation count. The stress-test note on this point looks like it lands.\n\nThe paper is aimed at researchers who want a principled account of prior-guided HPO rather than another heuristic method. It shows clear engagement with the best-arm identification literature and the formal steps are worth checking, so it deserves a serious referee even if the multi-fidelity modeling needs tightening.","headline":"The paper supplies the first distribution-dependent bounds linking prior quality to sample cost in multi-fidelity HPO via best-arm identification, but the multi-fidelity bias and cost modeling look underspecified.","tokens_in":2344,"tokens_out":346,"would_cite":false,"duration_ms":11970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Informative priors concentrating probability mass on near-optimal arms reduce the evaluations needed in multi-fidelity hyperparameter optimization.","keywords":["hyperparameter optimization","prior-guided HPO","best-arm identification","sample complexity","multi-fidelity optimization","distribution-dependent bounds","AutoML"],"falsifier":"An experiment in which priors concentrating on near-optimal arms produce no reduction in evaluations below the no-prior baseline, or in which observed sample needs violate the derived bounds.","tokens_in":2606,"feed_emoji":"📉","tokens_out":606,"duration_ms":15173,"temperature":0.7,"pith_summary":"The paper establishes that modeling priors directly over arm means as configuration performance allows derivation of distribution-dependent sample complexity bounds in the fixed-budget best-arm identification setting. These bounds show that priors concentrating on near-optimal configurations lower the number of required evaluations, while uninformative or misleading priors recover standard baseline performance. A reader would care because large-scale HPO in AutoML is computationally expensive, and this supplies the first quantitative link between prior informativeness and evaluation budget. Proof-of-concept runs on a synthetic benchmark and LCBench achieve up to 90 percent budget reduction while preserving solution quality.","feed_headline":"Informative priors cut HPO evaluation budget","feed_subtitle":"Distribution-dependent bounds quantify how priors on near-optimal arms lower samples needed, with baseline recovered otherwise.","key_machinery":"Fixed-budget best-arm identification framework with priors modeled over arm means, yielding distribution-dependent sample complexity bounds.","core_discovery":"By modeling priors directly over arm means as configuration performance within the fixed-budget best-arm identification framework, we derive explicit distribution-dependent error bounds that quantify the relationship between prior informativeness and evaluation budget. Our analysis shows that informative priors, which concentrate probability mass on near-optimal arms, yield reductions in the number of required evaluations, whereas baseline performance is recovered with uninformative or misleading priors.","pith_inferences":["The bounds could guide construction or selection of priors that maximize efficiency in new HPO tasks.","The same modeling approach may extend to quantify prior benefits in other black-box optimization problems.","The results support analysis of energy savings when prior information is available in large-scale ML pipelines."],"forward_implications":["Informative priors reduce the number of evaluations required for multi-fidelity HPO.","Uninformative or misleading priors produce no improvement and recover baseline performance.","Up to 90 percent budget reduction is observed on synthetic and LCBench benchmarks while retaining solution quality.","The bounds supply a principled foundation for incorporating priors into compute-efficient AutoML."],"fun_headline_variants":["Priors over arm means reduce HPO samples","Bounds quantify prior impact on HPO budget","Multi-fidelity HPO with distribution-dependent bounds","Informative priors lower HPO evaluation count","Uninformative priors recover HPO baseline"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Priors can be modeled directly over the means of arm performances inside a fixed-budget best-arm identification setup to produce explicit error bounds linking prior quality to sample needs.","fun_headline_variants_meta":{"raw":{"variants":["Priors over arm means reduce HPO samples","Bounds quantify prior impact on HPO budget","Multi-fidelity HPO with distribution-dependent bounds","Informative priors lower HPO evaluation count","Uninformative priors recover HPO baseline"]},"model":"grok-4.3","cost_usd":0.003651,"raw_usage":{"total_tokens":1891,"prompt_tokens":646,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":36512000,"prompt_tokens_details":{"text_tokens":646,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1179,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":646,"tokens_out":66,"duration_ms":7978,"temperature":1.0,"reasoning_tokens":1179,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T07:38:56.969151+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment in which priors concentrating on near-optimal arms produce no reduction in evaluations below the no-prior baseline, or in which observed sample needs violate the derived bounds.","supporting_citations":[],"review_version":1}