Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters

Andr\'e Biedenkapp; Frank Hutter; Katharina Eggensperger; Marius Lindauer; Matthias Feurer

REVIEW 3 major objections 5 minor 1 cited by

Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tuning Bayesian optimization itself pays off in performance.

desk verdict A solid, honest empirical study showing that tuning BO's own hyperparameters matters and transfers, with the main caveat that the practical claims lean on unvalidated surrogate benchmarks and tool-default approximations. read the letter →

arxiv 1908.06674 v1 pith:HP5VNUUP submitted 2019-08-19 cs.LG cs.AIstat.ML

Marius Lindauer , Matthias Feurer , Katharina Eggensperger , André Biedenkapp , Frank Hutter This is my paper · ORCID

classification cs.LGcs.AIstat.ML

keywords Bayesianoptimizationhyperparametermeta-optimizationalgorithmconfigurationsurrogatebenchmarkstransferabilityablationstudyanytimeperformance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the internal settings of a Bayesian optimization (BO) routine—the choices practitioners usually leave at their tool’s defaults—should themselves be tuned, and whether tuned settings still help on new problems. Across three predictive models (Gaussian-process maximum-likelihood, Gaussian-process maximum-a-posteriori, and random forest) and three function families (artificial functions, SVM hyperparameter tuning, and small neural-network architecture search), the authors find that tuning BO’s own hyperparameters improves its average log-regret over time. A leave-one-function-out analysis shows that settings tuned on similar problems transfer and often beat the defaults, and tuning on a different problem family sometimes works too, though with more risk. The paper also identifies, via ablation, that only a small set of BO hyperparameters matters, and that the important ones depend on the predictive model and benchmark family. These results suggest that default BO configurations in common tools leave substantial performance on the table.

What carries the argument

The load-bearing device is the meta-loss $L(\lambda)$, which turns “tune the tuner” into a concrete optimization problem: minimize average log-regret over a family of functions using a fixed evaluation budget. Around this loss, the paper constructs three design spaces, each fixing one predictive model (GP-ML, GP-MAP, or RF) and ranging over the initial design, acquisition function, response transformation, interleaved random sampling, and model hyperparameters. Surrogate benchmarks stand in for the SVM and neural-network families so that the meta-optimizer can afford many evaluations and so that the true optimum is approximately known. Leave-one-function-out and cross-family evaluations then test whether an optimized $\lambda$ generalizes, and an ablation procedure isolates which components of $\lambda$ carry the gain.

What would settle it

Run the same meta-tuning protocol directly on the original, non-surrogate SVM and small-neural-network benchmarks used to build the surrogates, with the optima estimated by extensive random search, and compare default versus tuned configurations on held-out datasets. If the tuned configurations do not beat the defaults there, or if leave-one-function-out and cross-family gains disappear, the paper’s central conclusion would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that fixing the internal settings of a Bayesian optimizer—rather than accepting defaults—can substantially improve its anytime performance, and that the gains partly survive being moved to new problems. The paper treats the BO procedure itself as the object to be tuned: a meta-optimizer minimizes the time-averaged log-regret $L(\lambda)$ over a distribution of target functions, where $\lambda$ is the target BO’s hyperparameter configuration. The evidence comes from three predictive models (GP-ML, GP-MAP, and RF) and three benchmark families (artificial functions, SVM hyperparameter tuning, and small neural-network architecture search). Independent tuning improves every model on every family; leave-one-function-out tuning beats defaults in five of nine cases; and out-of-family tuning often helps but risks deterioration, especially on the family with categorical and conditional hyperparameters. An ablation study shows that a small set of hyperparameters—kernel choice and length-scale priors for the GPs, random-sampling probability and response transformation for RFs—accounts for most of the improvement.

Load-bearing premise

The conclusions assume that the computer-model approximations standing in for the SVM and small-neural-network optimization problems faithfully resemble the real hyperparameter-optimization landscapes; if they do not, the measured improvements and transferability results may not carry over to practice.

Editorial extensions

If this is right

Comparisons between BO variants should not rely on default settings: the paper observes different rankings across models before and after tuning, so untuned defaults can mislead conclusions.
Users of BO packages can obtain meaningful any-time gains by tuning the optimizer’s own settings, even when the tuning is done on similar functions rather than the target itself.
Optimized BO configurations transfer partially across problem families, but transfer to problems with categorical and conditional hyperparameters is the risky case and should be validated rather than assumed.
Tuning effort can be concentrated on a small number of BO hyperparameters, with the identity of those hyperparameters depending on the predictive model in use.
Practical application of this meta-tuning requires knowing the target function’s optimum and having cheap evaluations; the paper points to multi-fidelity optimization and mixing surrogate with real evaluations as the route to lift those requirements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural reading of the Branin sampling plots is that common defaults are deliberately exploration-heavy; if so, the performance gap found here may be smaller on harder functions with many local optima, where that exploration is actually needed.
The near-diagonal success of cross-family transfer suggests a cheaper benchmarking protocol: maintain a small portfolio of tuned BO configurations, one per model family, and select by matching the target’s dimensionality and variable type, rather than tuning from scratch per benchmark.
Because only a few hyperparameters matter, a focused sensitivity analysis of just those parameters (kernel family, length-scale priors, random-sampling probability) could serve as a low-cost sanity check in any BO application, even without full meta-optimization.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this one. It is the first systematic look at whether and how much tuning Bayesian optimization's own hyperparameters helps, across three model families (GP-ML, GP-MAP, RF) and three benchmark families (artificial, SVM, ParamNet). The headline finding is believable: default BO settings are not trustworthy baselines, and a modest amount of meta-optimization improves anytime performance. The leave-one-function-out transfer results and the ablation study identifying which hyperparameters matter are genuinely useful and go beyond earlier component-level tuning work by Snoek et al. and Malkomes and Garnett.

The paper is also honest about its limitations. It explicitly says the optimum is assumed known and that target functions are cheap, and it does not claim surrogates are perfect replacements for real HPO surfaces. That honesty is real, not perfunctory.

Now the soft spots. The stress-test note is right that the SVM and ParamNet evidence rests on surrogate benchmarks with no fidelity check in the paper. That is a real limitation, but it is not fatal: the artificial-function results point in the same direction, and the surrogates come from an established line of work. Still, the Sec. 6 claim that defaults from common tools 'leave ample room for improvement' is stronger than the evidence supports, because the defaults are only 'inspired by' Spearmint, RoBO, GpyOpt, and SMAC, not the actual defaults, and the real-HPO evidence is all surrogate-based. I would soften that conclusion.

Two smaller issues. Tables 2 and 3 report means with significance tests but no variance or confidence intervals, which makes the magnitude of the gains hard to judge. And the cross-family table is slightly confounded: the diagonal uses leave-one-function-out within a family, while off-diagonal entries can use all functions from the source family, so you are partly comparing training-set size, not just family difference. Not a fatal flaw, but worth a referee asking for a cleaner comparison.

No code or data artifacts appear to be provided, which is a miss for a purely empirical paper of this kind.

Who is this for? Anyone working on BO benchmarking, AutoML, or algorithm configuration. It deserves a serious referee and would go in with 'major revision' rather than desk rejection. My own verdict is conditional: the central qualitative result is likely right, but the paper should validate the surrogates, report variance, and calibrate the practical claims about tool defaults. I would bring it to reading group and would cite it if I were writing about BO benchmarking methodology.

Referee Report

3 major / 5 minor

Summary. The paper empirically studies whether hyperparameter optimization (HPO) of Bayesian optimization's own hyperparameters is beneficial, and whether optimized settings transfer across functions. The authors define a meta-loss in Eq. (3) as time-averaged log regret to known optima, use SMAC as a meta-optimizer to tune target-BO configurations for three predictive models (GP-ML, GP-MAP, RF), and evaluate on three families: artificial functions, SVM HPO, and ParamNet HPO+NAS. They report that tuned configurations substantially outperform defaults, that leave-one-function-out tuning generalizes within families, that out-of-family transfer is partially effective, and that only a small set of BO hyperparameters matters. The main claim is that BO's own hyperparameters matter and that default configurations derived from common tools leave room for improvement.

Significance. If the empirical conclusions hold, this is a useful contribution to the BO and AutoML communities. The paper addresses a real and often ignored issue, and it provides practical guidance about which BO hyperparameters matter. Strengths include a carefully separated meta-optimization setup, a leave-one-function-out protocol for within-family generalization, multiple seeds for validation, and an ablation analysis. The experimental design also attempts to cover three different predictive model families, which goes beyond many existing studies. However, the practical conclusions rest heavily on surrogate benchmarks whose fidelity to real HPO/NAS landscapes is asserted but not validated, and the reported aggregate tables omit variance information. These issues are fixable within the paper's scope, so the central claim is defensible but needs additional support.

major comments (3)

[§5.1 and §6] The two application-oriented benchmark families, HPO SVM and HPO+NAS ParamNet, are surrogate benchmarks [Eggensperger et al., 2015], and Eq. (3) evaluates log regret against the surrogates' approximated optima. The manuscript asserts in §5.1 that these surrogates 'fairly well approximate their optima' but provides no fidelity check, such as prediction error on held-out configurations, rank correlation between surrogate predictions and true objective values, or a comparison of surrogate optima with the best known real HPO results. Since all SVM and ParamNet gains in Tables 2 and 3 are measured on these same surrogates, the §6 conclusion that the default hyperparameter configurations derived from well-known tools leave ample room for improvement on real HPO/NAS problems is not established for those families. The conclusion should either be restricted to the surrogate landscapes or accompanied by surrogate-validation evidence.
[Tables 2 and 3] Tables 2 and 3 report only point estimates of log regret aggregated over target functions and 20 repetitions, without standard errors, confidence intervals, effect sizes, or per-function breakdowns. The text states that significant differences are highlighted, but the highlighting is not visible in the manuscript text, and the test setup (e.g., number of paired comparisons and whether multiple-testing correction was applied) is not described. This makes it impossible to verify the central quantitative claim that tuned settings significantly outperform defaults and that cross-family transfer is often significantly worse than within-family tuning.
[Table 3] The cross-family comparison in Table 3 is confounded by training-set size: the diagonal is obtained with leave-one-function-out tuning within the row family, whereas off-diagonal entries are obtained by tuning on all functions of the column family and evaluating on the row family. A configuration tuned on a full source family has seen more target functions than the diagonal LOFO configuration, so the comparison conflates the amount of training data with the family difference. The claim that out-of-family tuning 'often worked surprisingly well' needs a matched comparison, for example tuning on n-1 functions from the source family and evaluating on a held-out target family, or at least a discussion of how much of the gap is attributable to data quantity.

minor comments (5)

[Table 2 caption] The caption contains a stray word ('setting We highlight results...'), and the promised highlighting is not visible in the rendered text; please fix both.
[§5.2] The statement that default configurations 'do far too much exploration' on Branin is supported only by visual inspection of sampling plots; it would be more convincing to quantify exploration behavior or to soften the wording.
[§5.4] The claim that tuning on a different family sometimes gives slightly better performance than tuning on the original family, e.g., tuning GP-MAP on SVM and applying it to ParamNet, would be easier to assess if the relevant entries in Table 3 were accompanied by variance or uncertainty information.
[§5.5 and Figure 3] Figure 3 would be much easier to read with explicit axis labels, a legend identifying the three predictive models, and a note explaining how the x-axis changes relate to the ablation procedure.
[§5.1] Please clarify whether the 'best target-BO configuration found overall' was selected from the combined 20 meta-optimizer runs before the 20-seed validation, and whether any selection-bias correction was considered; the current wording leaves this ambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all reported gains are measured against held-out functions and known optima, not reduced to fitted inputs.

full rationale

This is an empirical study rather than a derivation, and its claims do not reduce to their inputs by construction. The meta-loss in Eq. (3) is defined as time-averaged log-regret to the known optimum of each benchmark function, and it is used to score hyperparameter configurations with a separate meta-optimizer; it is not a fitted quantity that is later renamed as a prediction. Generalization claims are evaluated in a leave-one-function-out scheme and in cross-family transfer, so the reported improvements over defaults are measured on functions not used for tuning and are therefore not statistically forced. The default configurations are taken from existing BO tools rather than optimized on the test functions, and the tuned configurations are validated with fresh random seeds. Self-citations to surrogate benchmarks, SMAC, and BOHB are methodological references with external content; they are not used to import a uniqueness theorem or to forbid alternative explanations. The paper also explicitly acknowledges that its assumptions, known optima and cheap evaluation, do not hold in most practical applications, which is a limitation on external validity rather than a circular step. Any concerns about surrogate fidelity or reuse of the authors' own benchmarks belong to correctness and empirical robustness, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an empirical measurement, so the ledger records the domain assumptions that make the measurements meaningful, plus one free-parameter-style entry for the tuned configurations that are fit to the benchmark data. No new theoretical entities are introduced.

free parameters (1)

Optimized target-BO hyperparameter configuration (per family and model)
SMACv3 selects lambda* minimizing Eq. 3 on 6 to 10 functions per family. The paper reports the performance of these configurations but not their full numerical settings, so the fitted values cannot be audited.

assumptions (4)

domain assumption Surrogate benchmarks approximate real HPO problems sufficiently well for measuring BO performance
Section 5.1 states 'The latter two benchmarks are surrogate benchmarks ... and thus we can fairly well approximate their optima.' This is load-bearing for all claims about HPO and HPO plus NAS.
domain assumption Known optima and log-regret averaged over time form a valid meta-loss
Section 4 defines the meta-loss with a known x*; the authors acknowledge in Section 6 that this assumes knowledge of the optimum, which does not hold in most practical applications.
domain assumption The three benchmark families are representative of distinct problem types
Section 5.1 selects artificial functions, SVM HPO, and ParamNet HPO plus NAS. The transferability conclusions depend on these families being meaningful representatives of BO practice.
ad hoc to paper Default configurations inspired by Spearmint, RoBO, GPyOpt, and SMAC are a fair baseline
Section 4.2 constructs defaults 'inspired by' these tools. The improvement claim depends on this baseline choice being reasonable and not artificially weak.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters." pith.science (2026). https://pith.science/paper/HP5VNUUP

@misc{pith2026190806674,
  author       = {Pith},
  title        = {Pith review of: Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HP5VNUUP}},
  note         = {Machine review of arXiv:1908.06674}
}

read the original abstract

Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of optimizing BO's own hyperparameters and the transferability of the found settings using a wide range of benchmarks, including artificial functions, HPO and HPO combined with neural architecture search. In particular, we show (i) that tuning can improve the any-time performance of different BO approaches, that optimized BO settings also perform well (ii) on similar problems and (iii) partially even on problems from other problem families, and (iv) which BO hyperparameters are most important.

Figures

Figures reproduced from arXiv: 1908.06674 by the authors.

**Figure 2.** Sampling behavior of BO with GP-MAP (left) and RF (right) on the function Branin, including the samples of the default configu [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Results of an ablation study for RF (top), GP-ML (middle) and GP-MAP (bottom) to show which hyperparameters had to be tuned [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Preference-Optimal Multi-Metric Weighting for Parallel Coordinate Plots
cs.HC 2025-06 conditional novelty 4.0 of 10

Given a user-selected point on an approximated Pareto front, the method computes linear metric weights from the front's gradient and colors parallel coordinates accordingly.

Reference graph

Works this paper leans on

43 extracted references · 39 canonical work pages · cited by 1 Pith paper

[1]

harmless

[Ahmed et al., 2016] M. Ahmed, B. Shahriari, and M. Schmidt. Do we need “harmless” Bayesian opti- mization and “ﬁrst-order” Bayesian optimization. In BayesOpt’16,

work page 2016
[9]

Duvenaud, J

[Duvenaud et al., 2014] D. Duvenaud, J. Lloyd, R. Grosse, J. Tenenbaum, and Z. Ghahramani. Structure discovery in nonparametric regression through compositional kernel search. In Proc. of ICML’13, pages 1166–1174,

work page 2014
[11]

Eggensperger, M

[Eggensperger et al., 2018] K. Eggensperger, M. Lindauer, H. H. Hoos, F. Hutter, and K. Leyton-Brown. Efﬁcient benchmarking of algorithm conﬁgurators via model-based surrogates. Machine Learning, 107(1):15–41,

work page 2018
[12]

Falkner, A

[Falkner et al., 2018] S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and Efﬁcient Hyperparameter Optimiza- tion at Scale. In Proc. of ICML’18, pages 1437–1446,

work page 2018
[13]

Fawcett and H

[Fawcett and Hoos, 2016] C. Fawcett and H. Hoos. Analysing differences between algorithm conﬁgurations through ablation. Journal of Heuristics , 22(4):431–458,

work page 2016
[14]

Feurer and F

[Feurer and Hutter, 2019] M. Feurer and F. Hutter.Hyperpa- rameter Optimization, pages 3–33. Springer International Publishing,

work page 2019
[16]

[Frazier, 2018] Peter I. Frazier. A tutorial on Bayesian opti- mization. arxiv:1807.02811,

arXiv 2018
[17]

Grosse, R

[Grosse et al., 2012] R. Grosse, R. Salakhutdinov, W. Free- man, and J. Tenenbaum. Exploiting compositionality to explore a large space of model structures. In N. de Freitas and K. Murphy, editors, Proceedings of the Twenty-Eighth Conference on Uncertainty in Artiﬁcial Intelligence, pages 306–315. AUAI Press,

work page 2012

Show all 43 references

[19]

Hern ´andez-Lobato, M

[Hern´andez-Lobato et al., 2014] J. Hern ´andez-Lobato, M. Hoffman, and Z. Ghahramani. Predictive entropy search for efﬁcient global optimization of black-box functions. In Proc. of NeurIPS’14, pages 918–926,

2014
[20]

Hoffman, E

[Hoffman et al., 2011] M. Hoffman, E. Brochu, and N. de Freitas. Portfolio allocation for Bayesian optimiza- tion. In Procs. of UAI, pages 327–336,

2011
[21]

Hutter, H

[Hutter et al., 2011] F. Hutter, H. Hoos, and K. Leyton- Brown. Sequential model-based optimization for general algorithm conﬁguration. In Proc. of LION’11, pages 507– 523,

2011
[22]

[Jin et al., 2018] H. Jin, Q. Song, and X. Hu. Auto- Keras: An efﬁcient neural architecture search system. arxiv:1806.10282,

2018 arXiv
[23]

Jones, M

[Jones et al., 1998] D. Jones, M. Schonlau, and W. Welch. Efﬁcient global optimization of expensive black box func- tions. JGO, 13:455–492,

1998
[25]

[Kushner, 1964] H. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Fluids Engineering , pages 97–106,

1964
[27]

Li and J

[Li and Malik, 2017] K. Li and J. Malik. Learning to opti- mize. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings,

2017
[28]

Lindauer, K

[Lindauer et al., 2017] M. Lindauer, K. Eggensperger, M. Feurer, S. Falkner, A. Biedenkapp, and F. Hut- ter. Smac v3: Algorithm conﬁguration in python. https://github.com/automl/SMAC3,

2017
[29]

L ´evesque, A

[L´evesque et al., 2017] J. L ´evesque, A. Durand, C. Gagn ´e, and R. Sabourin. Bayesian optimization for conditional hyperparameter spaces. In Proc. of IJCNN’17, pages 286–

2017
[30]

Malkomes and R

[Malkomes and Garnett, 2018] G. Malkomes and R. Garnett. Automating Bayesian optimization with Bayesian opti- mization. In Bengio et al. [2018], pages 5988–5997. [Malkomes et al., 2016] G. Malkomes, C. Schaff, and R. Garnett. Bayesian optimization for automated model selection...

2018
[31]

Marco, F

[Marco et al., 2017] A. Marco, F. Berkenkamp, P. Hennig, A. Schoellig, A. Krause, S. Schaal, and S. Trimpe. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with bayesian optimization. In Proc. of ICRA’17, pages 1557–1563,

2017
[32]

Mockus, V

[Mockus et al., 1978] J. Mockus, V . Tiesis, and A. Zilinskas. The application of Bayesian methods for seeking the ex- tremum. Towards Global Optimization, 2(117-129),

1978
[34]

Perrone, R

[Perrone et al., 2018] V . Perrone, R. Jenatton, M. Seeger, and C. Archambeau. Scalable hyperparameter transfer learning. In Bengio et al. [2018], pages 6845–6855. [Picheny et al., 2013] Victor Picheny, Tobias Wagner, and David Ginsbourger. A benchmark of kriging-based inﬁll c...

2018
[35]

Probst, A

[Probst et al., 2019] P. Probst, A. Boulesteix, and B. Bis- chl. Tunability: Importance of hyperparameters of ma- chine learning algorithms. Journal of Machine Learning Research, 20(53):1–32,

2019
[36]

Shahriari, K

[Shahriari et al., 2016] B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175,

2016
[37]

Snoek, H

[Snoek et al., 2012] J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algo- rithms. In Proc. of NeurIPS’12, pages 2960–2968,

2012
[38]

Snoek, O

[Snoek et al., 2015] J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, Prabhat, and R. Adams. Scalable Bayesian optimization using deep neural networks. In Proc. of ICML’15, pages 2171–2180,

2015
[39]

Springenberg, A

[Springenberg et al., 2016] J. Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with robust Bayesian neural networks. In Proc. of NeurIPS’16,

2016
[40]

Srinivas, A

[Srinivas et al., 2010] N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. of ICML’10, pages 1015–1022,

2010
[41]

GPyOpt: A Bayesian optimization framework in Python

[The GPyOpt authors, 2016] The GPyOpt authors. GPyOpt: A Bayesian optimization framework in Python. http: //github.com/ShefﬁeldML/GPyOpt,

2016
[42]

Thornton, F

[Thornton et al., 2013] C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. Auto-WEKA: combined selection and hyperparameter optimization of classiﬁcation algorithms. In Proc. of KDD’13, pages 847–855,

2013
[43]

Wu and P

[Wu and Frazier, 2016] J. Wu and P. Frazier. The parallel knowledge gradient method for batch Bayesian optimiza- tion. In Proc. of NeurIPS’16, pages 3126–3134, 2016

2016
[1964]

K ¨uhn, P

[K¨uhn et al., 2018] D. K ¨uhn, P. Probst, J. Thomas, and B. Bischl. Automatic exploration of machine learning ex- periments on openml. arXiv:1806.10961,

2018 arXiv
[1978]

Morar, J

[Morar et al., 2017] M. Morar, J. Knowles, and S. Sampaio. Initialization of Bayesian optimization viewed as part of a larger algorithm portfolio. In Proceedings of the inter- national workshop in Data Science meets Optimization (at CEC and CPAIOR 2017),

2017
[1998]

Klein, S

[Klein et al., 2017] A. Klein, S. Falkner, N. Mansur, and F. Hutter. Robo: A ﬂexible and robust Bayesian optimiza- tion framework in Python. InProcs. of BayesOpt’17,

2017
[2010]

Brockhoff, B

[Brockhoff et al., 2015] D. Brockhoff, B. Bischl, and T. Wagner. The impact of initial designs on the perfor- mance of matsumoto on the noiseless BBOB-2015 testbed: A preliminary study. In Proc. of GECCO’15, pages 1159– 1166,

2015
[2011]

[Chen et al., 2017] Y . Chen, M. Hoffman, S. Colmenarejo, M. Denil, T. Lillicrap, M. Botvinick, and N. de Freitas. Learning to learn without gradient descent by gradient de- scent. In Proc. of ICML’17, pages 748–756,

2017
[2012]

Hennig and C

[Hennig and Schuler, 2012] P. Hennig and C. Schuler. En- tropy search for information-efﬁcient global optimization. JMLR, 98888(1):1809–1837,

2012
[2013]

Brochu, V

[Brochu et al., 2010] E. Brochu, V . Cora, and N. de Fre- itas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:1012.2599v1 [cs.LG],

2010 arXiv
[2014]

Eggensperger, F

[Eggensperger et al., 2015] K. Eggensperger, F. Hutter, H.H. Hoos, and K. Leyton-Brown. Efﬁcient benchmarking of hyperparameter optimizers via surrogates. In Proc. of AAAI’15, pages 1114–1120,

2015
[2015]

[Bull, 2011] Adam D. Bull. Convergence rates of efﬁcient global optimization algorithms. JMLR, 12:2879–2904,

2011
[2016]

Bengio, H

[Bengio et al., 2018] S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors. Proceedings of the international conference on Advances in Neural Information Processing Systems,

2018
[2017]

[Dang et al., 2017] N. Dang, L. P ´erez C´aceres, P. De Caus- maecker, and T. St ¨utzle. Conﬁguring irace using surro- gate conﬁguration benchmarks. In Peter Bosman, editor, Proceedings of the Genetic and Evolutionary Computation Conference, pages 243–250. ACM,

2017
[2018]

Bergstra, D

[Bergstra et al., 2013] J. Bergstra, D. Yamins, and D. Cos. Hyperopt: A Python library for optimizing the hyperpa- rameters of machine learning algorithms. In Proceedings of the 12th Python in Science Conference (SCIPY) , pages 13–20,

2013
[2019]

Feurer, A

[Feurer et al., 2015] M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter. Efﬁcient and robust automated machine learning. In Proc. of NeurIPS’15, pages 2962–2970,

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

harmless

[Ahmed et al., 2016] M. Ahmed, B. Shahriari, and M. Schmidt. Do we need “harmless” Bayesian opti- mization and “ﬁrst-order” Bayesian optimization. In BayesOpt’16,

work page 2016

[2] [9]

Duvenaud, J

[Duvenaud et al., 2014] D. Duvenaud, J. Lloyd, R. Grosse, J. Tenenbaum, and Z. Ghahramani. Structure discovery in nonparametric regression through compositional kernel search. In Proc. of ICML’13, pages 1166–1174,

work page 2014

[3] [11]

Eggensperger, M

[Eggensperger et al., 2018] K. Eggensperger, M. Lindauer, H. H. Hoos, F. Hutter, and K. Leyton-Brown. Efﬁcient benchmarking of algorithm conﬁgurators via model-based surrogates. Machine Learning, 107(1):15–41,

work page 2018

[4] [12]

Falkner, A

[Falkner et al., 2018] S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and Efﬁcient Hyperparameter Optimiza- tion at Scale. In Proc. of ICML’18, pages 1437–1446,

work page 2018

[5] [13]

Fawcett and H

[Fawcett and Hoos, 2016] C. Fawcett and H. Hoos. Analysing differences between algorithm conﬁgurations through ablation. Journal of Heuristics , 22(4):431–458,

work page 2016

[6] [14]

Feurer and F

[Feurer and Hutter, 2019] M. Feurer and F. Hutter.Hyperpa- rameter Optimization, pages 3–33. Springer International Publishing,

work page 2019

[7] [16]

[Frazier, 2018] Peter I. Frazier. A tutorial on Bayesian opti- mization. arxiv:1807.02811,

arXiv 2018

[8] [17]

Grosse, R

[Grosse et al., 2012] R. Grosse, R. Salakhutdinov, W. Free- man, and J. Tenenbaum. Exploiting compositionality to explore a large space of model structures. In N. de Freitas and K. Murphy, editors, Proceedings of the Twenty-Eighth Conference on Uncertainty in Artiﬁcial Intelligence, pages 306–315. AUAI Press,

work page 2012