{"id":"41aa8ec1-759c-49f1-95f5-712737e4e054","arxiv_id":"1908.06674","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Tuning Bayesian optimization's own hyperparameters substantially improves its anytime performance, and the tuned settings transfer within and sometimes across benchmark families.","lead":"Bayesian optimization is a standard way to tune machine learning models, but this paper asks who tunes the tuner. The authors show that tuning Bayesian optimization's own settings improves its performance and that good settings often carry over to new problems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tuned-BO gains on SVM/ParamNet rest on unvalidated surrogate benchmarks; the paper gives no fidelity check, so its practical conclusion about default tool settings is not yet established.","rationale":"The paper's central practical claim is that tuning BO's own hyperparameters yields meaningful gains over defaults derived from common tools. This claim depends on the benchmarks faithfully representing real HPO problems. The artificial-function family is real, but the SVM and ParamNet families are surrogate benchmarks whose fidelity is asserted rather than demonstrated. Because Eq. (3) uses the surrogate's predicted optimum as ground truth, any mismatch between surrogate and true landscape can change both the optimal BO settings and the measured improvement. The reader's weakest assumption identified the same issue, and the paper's own limitations paragraph partially concedes the gap by noting that the assumptions of known optimum and cheap target functions do not hold in practice. A concrete fidelity check on held-out true evaluations would settle whether the surrogate-based gains transfer. Since the paper is explicitly framed as a first step and the verdict is already CONDITIONAL, this concern does not change the verdict but should be a stated condition for accepting the practical conclusion.","tokens_in":10413,"tokens_out":7001,"duration_ms":82352,"concrete_test":"Validate the surrogate benchmarks before accepting the practical claim: for each of the 10 SVM and 6 ParamNet surrogates, draw held-out configurations not used to fit the surrogate, evaluate them with the true ML pipeline or original benchmark data, and compare surrogate predictions to true values; then rerun Table 2's DEF-versus-tuned comparison with the true objective on a matched budget. If the tuned configurations lose their advantage on true functions, the reported gains are surrogate artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The externally relevant evidence for the central claim comes from two function families that are surrogate benchmarks (Sec. 5.1), and Eq. (3) defines the meta-loss as log-regret to each surrogate's predicted optimum. If the surrogates are smoother or otherwise differ from true HPO/architecture-search landscapes, the optimal BO hyperparameters and the magnitude of the improvement over defaults need not transfer to practice. The paper asserts that these surrogates 'fairly well approximate their optima' but provides no validation of surrogate fidelity, such as prediction error on held-out configurations or correlation of surrogate optima with true optima, and all reported gains for SVM and ParamNet are measured on the same surrogates. Therefore the strongest conclusion in Sec. 6, that defaults derived from common tools leave ample room for improvement in real use, is not supported by the experiments as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically studies whether hyperparameter optimization (HPO) of Bayesian optimization's own hyperparameters is beneficial, and whether optimized settings transfer across functions. The authors define a meta-loss in Eq. (3) as time-averaged log regret to known optima, use SMAC as a meta-optimizer to tune target-BO configurations for three predictive models (GP-ML, GP-MAP, RF), and evaluate on three families: artificial functions, SVM HPO, and ParamNet HPO+NAS. They report that tuned configurations substantially outperform defaults, that leave-one-function-out tuning generalizes within families, that out-of-family transfer is partially effective, and that only a small set of BO hyperparameters matters. The main claim is that BO's own hyperparameters matter and that default configurations derived from common tools leave room for improvement.","tokens_in":10623,"tokens_out":4077,"duration_ms":46595,"significance":"If the empirical conclusions hold, this is a useful contribution to the BO and AutoML communities. The paper addresses a real and often ignored issue, and it provides practical guidance about which BO hyperparameters matter. Strengths include a carefully separated meta-optimization setup, a leave-one-function-out protocol for within-family generalization, multiple seeds for validation, and an ablation analysis. The experimental design also attempts to cover three different predictive model families, which goes beyond many existing studies. However, the practical conclusions rest heavily on surrogate benchmarks whose fidelity to real HPO/NAS landscapes is asserted but not validated, and the reported aggregate tables omit variance information. These issues are fixable within the paper's scope, so the central claim is defensible but needs additional support.","major_comments":[{"comment":"The two application-oriented benchmark families, HPO SVM and HPO+NAS ParamNet, are surrogate benchmarks [Eggensperger et al., 2015], and Eq. (3) evaluates log regret against the surrogates' approximated optima. The manuscript asserts in §5.1 that these surrogates 'fairly well approximate their optima' but provides no fidelity check, such as prediction error on held-out configurations, rank correlation between surrogate predictions and true objective values, or a comparison of surrogate optima with the best known real HPO results. Since all SVM and ParamNet gains in Tables 2 and 3 are measured on these same surrogates, the §6 conclusion that the default hyperparameter configurations derived from well-known tools leave ample room for improvement on real HPO/NAS problems is not established for those families. The conclusion should either be restricted to the surrogate landscapes or accompanied by surrogate-validation evidence.","section":"§5.1 and §6"},{"comment":"Tables 2 and 3 report only point estimates of log regret aggregated over target functions and 20 repetitions, without standard errors, confidence intervals, effect sizes, or per-function breakdowns. The text states that significant differences are highlighted, but the highlighting is not visible in the manuscript text, and the test setup (e.g., number of paired comparisons and whether multiple-testing correction was applied) is not described. This makes it impossible to verify the central quantitative claim that tuned settings significantly outperform defaults and that cross-family transfer is often significantly worse than within-family tuning.","section":"Tables 2 and 3"},{"comment":"The cross-family comparison in Table 3 is confounded by training-set size: the diagonal is obtained with leave-one-function-out tuning within the row family, whereas off-diagonal entries are obtained by tuning on all functions of the column family and evaluating on the row family. A configuration tuned on a full source family has seen more target functions than the diagonal LOFO configuration, so the comparison conflates the amount of training data with the family difference. The claim that out-of-family tuning 'often worked surprisingly well' needs a matched comparison, for example tuning on n-1 functions from the source family and evaluating on a held-out target family, or at least a discussion of how much of the gap is attributable to data quantity.","section":"Table 3"}],"minor_comments":[{"comment":"The caption contains a stray word ('setting We highlight results...'), and the promised highlighting is not visible in the rendered text; please fix both.","section":"Table 2 caption"},{"comment":"The statement that default configurations 'do far too much exploration' on Branin is supported only by visual inspection of sampling plots; it would be more convincing to quantify exploration behavior or to soften the wording.","section":"§5.2"},{"comment":"The claim that tuning on a different family sometimes gives slightly better performance than tuning on the original family, e.g., tuning GP-MAP on SVM and applying it to ParamNet, would be easier to assess if the relevant entries in Table 3 were accompanied by variance or uncertainty information.","section":"§5.4"},{"comment":"Figure 3 would be much easier to read with explicit axis labels, a legend identifying the three predictive models, and a note explaining how the x-axis changes relate to the ablation procedure.","section":"§5.5 and Figure 3"},{"comment":"Please clarify whether the 'best target-BO configuration found overall' was selected from the combined 20 meta-optimizer runs before the 20-seed validation, and whether any selection-bias correction was considered; the current wording leaves this ambiguous.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study from an established group, and I do not see a novelty or scope problem. The main concern is that the practical conclusions rely on surrogate benchmarks whose fidelity is not demonstrated; adding fidelity checks and variance reporting would materially strengthen the paper. The use of the authors' own baseline tools and meta-optimizer is not by itself problematic, but independent or external validation would increase confidence in the transferability claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. It is the first systematic look at whether and how much tuning Bayesian optimization's own hyperparameters helps, across three model families (GP-ML, GP-MAP, RF) and three benchmark families (artificial, SVM, ParamNet). The headline finding is believable: default BO settings are not trustworthy baselines, and a modest amount of meta-optimization improves anytime performance. The leave-one-function-out transfer results and the ablation study identifying which hyperparameters matter are genuinely useful and go beyond earlier component-level tuning work by Snoek et al. and Malkomes and Garnett.\n\nThe paper is also honest about its limitations. It explicitly says the optimum is assumed known and that target functions are cheap, and it does not claim surrogates are perfect replacements for real HPO surfaces. That honesty is real, not perfunctory.\n\nNow the soft spots. The stress-test note is right that the SVM and ParamNet evidence rests on surrogate benchmarks with no fidelity check in the paper. That is a real limitation, but it is not fatal: the artificial-function results point in the same direction, and the surrogates come from an established line of work. Still, the Sec. 6 claim that defaults from common tools 'leave ample room for improvement' is stronger than the evidence supports, because the defaults are only 'inspired by' Spearmint, RoBO, GpyOpt, and SMAC, not the actual defaults, and the real-HPO evidence is all surrogate-based. I would soften that conclusion.\n\nTwo smaller issues. Tables 2 and 3 report means with significance tests but no variance or confidence intervals, which makes the magnitude of the gains hard to judge. And the cross-family table is slightly confounded: the diagonal uses leave-one-function-out within a family, while off-diagonal entries can use all functions from the source family, so you are partly comparing training-set size, not just family difference. Not a fatal flaw, but worth a referee asking for a cleaner comparison.\n\nNo code or data artifacts appear to be provided, which is a miss for a purely empirical paper of this kind.\n\nWho is this for? Anyone working on BO benchmarking, AutoML, or algorithm configuration. It deserves a serious referee and would go in with 'major revision' rather than desk rejection. My own verdict is conditional: the central qualitative result is likely right, but the paper should validate the surrogates, report variance, and calibrate the practical claims about tool defaults. I would bring it to reading group and would cite it if I were writing about BO benchmarking methodology.","headline":"A solid, honest empirical study showing that tuning BO's own hyperparameters matters and transfers, with the main caveat that the practical claims lean on unvalidated surrogate benchmarks and tool-default approximations.","tokens_in":11114,"tokens_out":2105,"would_cite":true,"duration_ms":25911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuning Bayesian optimization itself pays off in performance.","keywords":["Bayesian optimization","hyperparameter optimization","meta-optimization","algorithm configuration","surrogate benchmarks","transferability","ablation study","anytime performance"],"falsifier":"Run the same meta-tuning protocol directly on the original, non-surrogate SVM and small-neural-network benchmarks used to build the surrogates, with the optima estimated by extensive random search, and compare default versus tuned configurations on held-out datasets. If the tuned configurations do not beat the defaults there, or if leave-one-function-out and cross-family gains disappear, the paper’s central conclusion would be falsified.","tokens_in":10259,"feed_emoji":"⚙️","tokens_out":8446,"duration_ms":75539,"temperature":0.7,"pith_summary":"This paper asks whether the internal settings of a Bayesian optimization (BO) routine—the choices practitioners usually leave at their tool’s defaults—should themselves be tuned, and whether tuned settings still help on new problems. Across three predictive models (Gaussian-process maximum-likelihood, Gaussian-process maximum-a-posteriori, and random forest) and three function families (artificial functions, SVM hyperparameter tuning, and small neural-network architecture search), the authors find that tuning BO’s own hyperparameters improves its average log-regret over time. A leave-one-function-out analysis shows that settings tuned on similar problems transfer and often beat the defaults, and tuning on a different problem family sometimes works too, though with more risk. The paper also identifies, via ablation, that only a small set of BO hyperparameters matters, and that the important ones depend on the predictive model and benchmark family. These results suggest that default BO configurations in common tools leave substantial performance on the table.","feed_headline":"Tuning Bayesian optimization itself pays off in performance","feed_subtitle":"Default settings in common tools leave ample room for improvement, and tuned settings transfer across similar problems","key_machinery":"The load-bearing device is the meta-loss $L(\\lambda)$, which turns “tune the tuner” into a concrete optimization problem: minimize average log-regret over a family of functions using a fixed evaluation budget. Around this loss, the paper constructs three design spaces, each fixing one predictive model (GP-ML, GP-MAP, or RF) and ranging over the initial design, acquisition function, response transformation, interleaved random sampling, and model hyperparameters. Surrogate benchmarks stand in for the SVM and neural-network families so that the meta-optimizer can afford many evaluations and so that the true optimum is approximately known. Leave-one-function-out and cross-family evaluations then test whether an optimized $\\lambda$ generalizes, and an ablation procedure isolates which components of $\\lambda$ carry the gain.","core_discovery":"The central claim is that fixing the internal settings of a Bayesian optimizer—rather than accepting defaults—can substantially improve its anytime performance, and that the gains partly survive being moved to new problems. The paper treats the BO procedure itself as the object to be tuned: a meta-optimizer minimizes the time-averaged log-regret $L(\\lambda)$ over a distribution of target functions, where $\\lambda$ is the target BO’s hyperparameter configuration. The evidence comes from three predictive models (GP-ML, GP-MAP, and RF) and three benchmark families (artificial functions, SVM hyperparameter tuning, and small neural-network architecture search). Independent tuning improves every model on every family; leave-one-function-out tuning beats defaults in five of nine cases; and out-of-family tuning often helps but risks deterioration, especially on the family with categorical and conditional hyperparameters. An ablation study shows that a small set of hyperparameters—kernel choice and length-scale priors for the GPs, random-sampling probability and response transformation for RFs—accounts for most of the improvement.","pith_inferences":["A natural reading of the Branin sampling plots is that common defaults are deliberately exploration-heavy; if so, the performance gap found here may be smaller on harder functions with many local optima, where that exploration is actually needed.","The near-diagonal success of cross-family transfer suggests a cheaper benchmarking protocol: maintain a small portfolio of tuned BO configurations, one per model family, and select by matching the target’s dimensionality and variable type, rather than tuning from scratch per benchmark.","Because only a few hyperparameters matter, a focused sensitivity analysis of just those parameters (kernel family, length-scale priors, random-sampling probability) could serve as a low-cost sanity check in any BO application, even without full meta-optimization."],"forward_implications":["Comparisons between BO variants should not rely on default settings: the paper observes different rankings across models before and after tuning, so untuned defaults can mislead conclusions.","Users of BO packages can obtain meaningful any-time gains by tuning the optimizer’s own settings, even when the tuning is done on similar functions rather than the target itself.","Optimized BO configurations transfer partially across problem families, but transfer to problems with categorical and conditional hyperparameters is the risky case and should be validated rather than assumed.","Tuning effort can be concentrated on a small number of BO hyperparameters, with the identity of those hyperparameters depending on the predictive model in use.","Practical application of this meta-tuning requires knowing the target function’s optimum and having cheap evaluations; the paper points to multi-fidelity optimization and mixing surrogate with real evaluations as the route to lift those requirements."],"supporting_citations":[{"why":"Supplies the surrogate-benchmark methodology that lets the paper approximate real SVM and ParamNet optimization problems cheaply with known optima.","marker":"[Eggensperger et al., 2015]"},{"why":"Provides the algorithm-configuration-as-HPO view and surrogate benchmarks that justify using a meta-optimizer on BO’s own hyperparameters.","marker":"[Eggensperger et al., 2018]"},{"why":"Supplies the random-forest BO model and one of the default design spaces that the paper tunes and compares.","marker":"[Hutter et al., 2011]"},{"why":"Supplies the practical GP-MAP BO approach and default settings that serve as a baseline for tuning.","marker":"[Snoek et al., 2012]"},{"why":"Supplies the GP-ML efficient-global-optimization method and the response transformations that are among the BO design choices studied.","marker":"[Jones et al., 1998]"},{"why":"Supplies the ten SVM hyperparameter-optimization benchmark functions used as one function family.","marker":"[Kuhn et al., 2018]"},{"why":"Supplies the ParamNet HPO-plus-architecture-search benchmark family and motivates the multi-fidelity direction for future practical meta-tuning.","marker":"[Falkner et al., 2018]"},{"why":"Supplies the ablation method used to identify which BO hyperparameters actually carry the improvement.","marker":"[Fawcett and Hoos, 2016]"}],"fun_headline_variants":["Tune your Bayesian optimizer: its own hyperparameters matter","Optimizing BO's hyperparameters improves anytime performance","Transferable gains from tuning Bayesian optimization's settings","Meta-tuning beats default BO settings across benchmarks","Bayesian optimization's self-hyperparameters are worth tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume that the computer-model approximations standing in for the SVM and small-neural-network optimization problems faithfully resemble the real hyperparameter-optimization landscapes; if they do not, the measured improvements and transferability results may not carry over to practice.","fun_headline_variants_meta":{"raw":{"variants":["Tune your Bayesian optimizer: its own hyperparameters matter","Optimizing BO's hyperparameters improves anytime performance","Transferable gains from tuning Bayesian optimization's settings","Meta-tuning beats default BO settings across benchmarks","Bayesian optimization's self-hyperparameters are worth tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1384,"prompt_tokens":878,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":494,"tokens_out":506,"duration_ms":5735,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:36:49.100202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same meta-tuning protocol directly on the original, non-surrogate SVM and small-neural-network benchmarks used to build the surrogates, with the optima estimated by extensive random search, and compare default versus tuned configurations on held-out datasets. If the tuned configurations do not beat the defaults there, or if leave-one-function-out and cross-family gains disappear, the paper’s central conclusion would be falsified.","supporting_citations":[{"cited_title":"Eggensperger, F","cited_arxiv_id":null,"evidence_quote":"Supplies the surrogate-benchmark methodology that lets the paper approximate real SVM and ParamNet optimization problems cheaply with known optima."},{"cited_title":"Eggensperger, M","cited_arxiv_id":null,"evidence_quote":"Provides the algorithm-configuration-as-HPO view and surrogate benchmarks that justify using a meta-optimizer on BO’s own hyperparameters."},{"cited_title":"Hutter, H","cited_arxiv_id":null,"evidence_quote":"Supplies the random-forest BO model and one of the default design spaces that the paper tunes and compares."},{"cited_title":"Snoek, H","cited_arxiv_id":null,"evidence_quote":"Supplies the practical GP-MAP BO approach and default settings that serve as a baseline for tuning."},{"cited_title":"Jones, M","cited_arxiv_id":null,"evidence_quote":"Supplies the GP-ML efficient-global-optimization method and the response transformations that are among the BO design choices studied."},{"cited_title":"Falkner, A","cited_arxiv_id":null,"evidence_quote":"Supplies the ParamNet HPO-plus-architecture-search benchmark family and motivates the multi-fidelity direction for future practical meta-tuning."},{"cited_title":"Fawcett and H","cited_arxiv_id":null,"evidence_quote":"Supplies the ablation method used to identify which BO hyperparameters actually carry the improvement."}],"review_version":1}