Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BOAH claims that multi-fidelity hyperparameter optimization only becomes practically useful when the search is paired with automatic analysis, so it bundles design-space specification, BOHB, and CAVE into one workflow.

desk verdict A useful integration of previously published AutoML components; the software is real and public, but the effectiveness evidence is thin and the budget-correlation premise is never checked. read the letter →

arxiv 1908.06756 v1 pith:37HJVKG6 submitted 2019-08-16 cs.LG cs.AIstat.ML

Marius Lindauer , Katharina Eggensperger , Matthias Feurer , André Biedenkapp , Joshua Marben , Philipp Müller , Frank Hutter This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords multi-fidelityBayesianoptimizationhyperparameterBOHBHyperBandAutoMLimportanceconfigurationspacedesignreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multi-fidelity hyperparameter optimization becomes practical and interpretable when one suite joins three pieces: a flexible design-space specification, the BOHB optimizer that combines Bayesian optimization with HyperBand's budget schedules, and automatic post-hoc analysis of the search. The suite lets users define conditional, categorical, integer, and log-scale hyperparameters, optimize on cheap budgets such as repetitions or epochs, and receive per-budget hyperparameter importance, cross-budget rank correlations, and footprint plots. The analysis is free in the sense of requiring no additional function evaluations. A cartpole reinforcement-learning showcase applies the whole loop to tune PPO, reporting that BOHB beats PPO's defaults.

What carries the argument

The central mechanism is the multi-fidelity loop: HyperBand allocates evaluations across budgets and successively halves unpromising configurations, while the Bayesian optimization component uses kernel density estimates to propose new configurations from the best-performing ones. The key assumption that makes this efficient is budget transferability, meaning cheap evaluations rank configurations similarly to expensive ones, and CAVE's rank-correlation analysis is the diagnostic that checks it. BOAH's integration is the claimed contribution: ConfigSpace, BOHB, and CAVE in one callable workflow, with the analysis stated to need no extra function evaluations.

What would settle it

Run BOHB on a task where cheap and expensive evaluations are known to disagree, for example a noisy reinforcement-learning problem with one episode as the low budget, and compare final performance against random search and single-fidelity Bayesian optimization using equal total compute. If the multi-fidelity incumbent is not competitive, the budget-transfer assumption fails; the cartpole showcase would also be stronger if it reported the budget rank-correlation diagnostic it defines.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in multi-fidelity AutoML is not just the optimizer but the surrounding workflow: without a simple way to describe the design space and no automatic analysis after the run, users neither know where to search nor why the search succeeded. The paper argues that BOHB, which combines kernel-density-estimate Bayesian optimization with HyperBand-style successive halving on budgets such as epochs, repetitions, or cross-validation folds, should be paired with ConfigSpace for conditional and log-scale spaces and with CAVE for analysis, including local parameter importance, fANOVA on each budget, budget rank-correlation checks, and optimizer footprint plots. It presents a single integrated Python workflow and demonstrates it by tuning PPO on cartpole with the number of runs as budget, finding that the optimized settings beat the default PPO configuration and identifying discount factor, batch size, and learning rate as the important hyperparameters.

Load-bearing premise

A configuration's rank on a small budget must predict its rank on the largest budget; if cheap evaluations are noisy, promising configurations can be discarded before an expensive evaluation would reveal them.

Editorial extensions

If this is right

  • Users who adopt BOAH can move from a conditional design-space definition to an analysis report without writing integration code or spending additional function evaluations.
  • Per-budget fANOVA and local parameter importance give a stage-by-stage view of which hyperparameters drive performance, not just a final ranking.
  • The rank-correlation check between budgets turns the hidden assumption of multi-fidelity search into a visible diagnostic that can warn against unsafe budget schedules.
  • Because the analysis is post-hoc and evaluation-free, one BOHB run can be re-examined with different importance measures or visualization options after the campaign has finished.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to make the budget rank-correlation diagnostic an online stopping rule: pause spending on a low budget as soon as its correlation with the highest budget falls below a threshold, and shift remaining compute to higher budgets.
  • For noisy single-run reinforcement learning, comparing BOHB's early eliminations against an equal-compute no-early-stopping baseline would isolate how much of the reported speedup comes from averaging repeated runs rather than from budget-based search.
  • The same three-component architecture may transfer to neural architecture search with epochs as budgets, giving per-epoch importance of architectural choices and early warnings when low-epoch performance misleads the search.
  • Combining footprint plots with budget rank correlations could expose a failure mode the paper mentions only implicitly: the optimizer focusing early on a region that looks strong at low fidelity but weak at high fidelity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BOAH, a Python tool suite that combines three packages: ConfigSpace for defining conditional design spaces, BOHB for multi-fidelity Bayesian optimization and Hyperband-style search, and an extended CAVE for post-hoc analysis of optimization runs, including per-budget hyperparameter importance, budget rank correlation, and optimizer footprint plots. The suite is demonstrated on a cartpole/PPO hyperparameter optimization task with budgets defined as the number of PPO repetitions. The paper claims that BOAH is the first comprehensive tool suite integrating design-space specification, multi-fidelity search, and automated analysis, and that the analysis requires no additional function evaluations.

Significance. If the tool works as described, the engineering contribution is real: it provides an integrated, publicly available workflow from design-space definition through multi-fidelity search to interpretable analysis, with a runnable example notebook that makes the existence claim independently checkable. The extension of CAVE to multi-fidelity data and the fmin-style interface are useful additions. However, the effectiveness claims are only weakly supported: the single case study uses 10 runs, lacks statistical testing, reports no comparison against the alternatives named in Section 2, and does not verify the budget-rank-correlation premise that the paper itself identifies as central in Section 3.3. The significance of the paper therefore depends on strengthening this empirical grounding rather than on the software description alone.

major comments (3)
  1. [Section 4 / Figure 2] The paper never reports the budget rank-correlation diagnostic that Section 3.3 explicitly identifies as important, stating that multi-fidelity optimizers perform best if similar configurations perform best across budgets. Since Section 4 defines budgets as the number of PPO runs and acknowledges that individual runs provide a very noisy performance estimate, the Spearman correlation between the lowest and highest budgets is exactly the quantity that determines whether BOHB's early eliminations and CAVE's per-budget importance analyses are meaningful. Please add this diagnostic, e.g., the correlation between budget 1 and budget 9, along with a scatterplot of budget-wise rankings, using the data already collected for the showcase.
  2. [Section 4 / Figure 2] The effectiveness comparison is limited to BOHB's incumbent cost versus PPO's default cost, so the reported sample-efficiency improvement could in principle be achieved by HyperBand alone, random search, or chance. The paper should compare BOHB against HyperBand without the Bayesian optimization component, random search, and at least one of the other multi-fidelity tools named in Section 2 (RoBO or Dragonfly), and report the distribution across the 10 repeats together with statistical testing. Without this, the claim of 'substantial' improvement and the attribution of the gain to the Bayesian optimization component are underdetermined.
  3. [Section 5] The concluding claim that BOAH is 'the first tool suite which brings all these three important components together and therefore improves the usability of AutoML substantially' is broader than the evidence presented. The paper demonstrates one showcase and does not provide a user study or comparative usability analysis. Either add additional evidence or temper the conclusion to a claim about the integration and availability of the components.
minor comments (5)
  1. [Section 3.2] The 'hyperopt-like optimizer' component is mentioned without a citation or a description of its parameters; please specify whether this is the TPE algorithm and how it is configured.
  2. [Section 3.3 / Figure 2] The figure caption does not mention that uncertainty estimates for fANOVA and LPI are omitted, even though the text says they are omitted for space reasons; the caption should state this explicitly.
  3. [Section 2] The statement that RoBO and Dragonfly are 'problematic' for multi-fidelity settings because of cubic GP scaling is asserted without a reference or experiment; please soften or support this claim.
  4. [Footnote 1] There is a typo: 'also a links' should read 'also links'.
  5. [Section 5] The phrase 'improves the usability of AutoML substantially' exceeds what the empirical section supports; consider a more measured formulation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: BOAH is an integration paper whose claims are supported by executing the actual open-source tool suite, not by a derivation that reduces to its inputs.

full rationale

BOAH is a system/tool paper, not a derivation. It integrates previously published components (BOHB and CAVE) and demonstrates them on a cartpole PPO showcase. There is no fitted parameter that is later renamed as a prediction: the sample-efficiency comparison in Figure 2 is an executed empirical run, and the hyperparameter importance analyses are produced by the tool itself. The Section 3.3 statement that multi-fidelity optimizers perform best when similar configurations rank similarly across budgets is an explicitly acknowledged assumption, and the paper even provides a rank-correlation diagnostic for it, so it is not smuggled in as a result. The omission of that rank-correlation value in the cartpole showcase is a verification gap and a correctness risk, not circularity. The 'first tool suite' claim is an external novelty claim based on the authors' awareness of related work; it is not derived from the paper's own definitions or equations. While self-citations are frequent, the cited components are open-source, are actually run in the demonstration, and do not need to be rederived for the integration claim to be meaningful. Therefore no circular step is identifiable under the stated criteria.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper is a software contribution, so there is no derivation with fitted constants. The only hand-chosen numbers are the case-study budget schedule in Section 4, which affects the demonstration but not the existence of the suite. The load-bearing assumptions are inherited from prior work, much of it by the same authors: BOHB's effectiveness, the validity of fANOVA and LPI for importance analysis, and the budget-correlation premise stated in Section 3.3. No new theoretical entities are introduced.

free parameters (1)
  • case-study budget schedule and iteration budget = 1 to 9 PPO repetitions, 128 iterations, 10 workers
    Hand-chosen in Section 4. The demonstration's outcome (BOHB beating PPO defaults) and the importance analysis depend on these settings, though the suite's existence claim does not.
assumptions (4)
  • domain assumption Configurations that perform well on low budgets also perform well on high budgets (budget rank correlation holds).
    Section 3.3: 'Multi-fidelity optimizers, such as BOHB, perform best if similar configurations perform best across the various budgets.' This premise makes multi-fidelity elimination and per-budget analysis meaningful, and it is not verified in the showcase.
  • domain assumption BOHB is a robust and efficient multi-fidelity optimizer.
    Section 3.2 adopts BOHB and its components from Falkner et al. 2018, published by the same group, and asserts their strengths without benchmarking them in this paper; the effectiveness claim inherits that prior result.
  • domain assumption fANOVA and local parameter importance (LPI) yield valid importance estimates on budgeted, noisy data.
    Section 3.3 applies LPI (Biedenkapp et al. 2018) and fANOVA (Hutter et al. 2014) per budget and adds uncertainty estimates, but their validity on multi-fidelity reinforcement learning data is assumed rather than established here.
  • standard math Background algorithms (TPE-style KDE sampling, successive halving, Hyperband) behave as published.
    Section 3.2 builds on Bergstra et al. 2011, Jamieson and Talwalkar 2016, and Li et al. 2018 as established results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters." pith.science (2026). https://pith.science/paper/37HJVKG6

@misc{pith2026190806756,
  author       = {Pith},
  title        = {Pith review of: BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/37HJVKG6}},
  note         = {Machine review of arXiv:1908.06756}
}
read the original abstract

Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours. To overcome this, we introduce a comprehensive tool suite for effective multi-fidelity Bayesian optimization and the analysis of its runs. The suite, written in Python, provides a simple way to specify complex design spaces, a robust and efficient combination of Bayesian optimization and HyperBand, and a comprehensive analysis of the optimization process and its outcomes.

Figures

Figures reproduced from arXiv: 1908.06756 by the authors.

Figure 1
Figure 1. Workflow of BOAH the algorithm being optimized that is not available from executing any other multi-fidelity hyperparameter optimization package we are aware of. 2. Related Work While there are many available tools for standard BO, e.g., Spearmint (Snoek et al., 2012), SMAC (Hutter et al., 2011), HyperOpt (Bergstra et al., 2011), GPyOpt and BOTorch, the same cannot be said for multi-fidelity BO; we are only aware of… view at source ↗
Figure 2
Figure 2. Left: Performance of incumbent configuration found by BOHB over time. As a baseline we show the final cost of the PPO’s default configuration. Cost refers to the number of epochs PPO needed to solve the cartpole problem. Middle: Hyperparameter importance with fANOVA and LPI as percentages on the largest budget. Because of space limitations, we do not show uncertainty estimates. Right: Estimated local hyper-parameter… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

    cs.LG 2026-01 conditional novelty 6.0 of 10

    CondPED-ANOVA estimates hyperparameter importance in conditional search spaces by keeping only the within-regime part of the variance, so inactive parameters do not inherit the gating parameter's importance.

  2. Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces

    cs.DC 2025-06 conditional novelty 6.0 of 10

    A config-search data model and a linear representative-point transfer method can reuse measurements across similar cloud workloads, cutting sampled configurations by up to 92% when a linear relationship holds.

  3. Rapid Experimentation with Python Considering Optional and Hierarchical Inputs

    cs.MS 2025-01 conditional novelty 6.0 of 10

    raxpy automates space-filling experimental design for Python functions with optional and hierarchical inputs, and its FSS-LHD-VP-MP algorithm ranks best on the paper's internal metrics.

  4. Enhancing Explainability and Reliable Decision-Making in Particle Swarm Optimization through Communication Topologies

    cs.LG 2025-04 conditional novelty 4.0 of 10

    Using SHAP-based explanations from an adapted IOHxplainer, the paper finds that lower cognitive coefficients help PSO and that Ring topology shows the best in-sample R2, but only on 2D benchmarks with no validation.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages · cited by 4 Pith papers

  1. [1]

    Bergstra and Y

    J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. JMLR, 13: 0 281--305, 2012

    work page 2012

  2. [2]

    Bergstra, R

    J. Bergstra, R. Bardenet, Y. Bengio, and B. K \'e gl. Algorithms for hyper-parameter optimization. In Proc. of NeurIPS'11, pages 2546--2554, 2011

    work page 2011

  3. [3]

    Biedenkapp, J

    A. Biedenkapp, J. Marben, M. Lindauer, and F. Hutter. Cave: Configuration assessment, visualization and evaluation. In Proc. of LION'18, 2018

    work page 2018

  4. [4]

    Falkner, A

    S. Falkner, A. Klein, and F. Hutter. BOHB: robust and efficient hyperparameter optimization at scale. In Proc. of ICML, pages 1436--1445, 2018

    work page 2018

  5. [5]

    Feurer and F

    M. Feurer and F. Hutter. Hyperparameter optimization. In AutoML: Methods, Sytems, Challenges, chapter 1, pages 3--38. Springer, 2019

    work page 2019

  6. [6]

    Feurer, A

    M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In Proc. of NeurIPS'15, pages 2962--2970, 2015

    work page 2015

  7. [7]

    Hutter, H

    F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Proc. of LION'11, pages 507--523, 2011

    work page 2011

  8. [8]

    Hutter, H

    F. Hutter, H. Hoos, and K. Leyton-Brown. An efficient approach for assessing hyperparameter importance. In Proc. of ICML'14, pages 754--762, 2014

    work page 2014

Show all 16 references
  1. [9]

    Jamieson and A

    K. Jamieson and A. Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proc. of AISTATS'16, 2016

    2016

  2. [10]

    Kandasamy, K

    K. Kandasamy, K. Vysyaraju, W. Neiswanger, B. Paria, C. Collins, J. Schneider, B. Poczos, and E. Xing. Tuning hyperparameters without grad students: Scalable and robust B ayesian optimisation with dragonfly. arxiv:1903.06694, 2019

    1903 arXiv

  3. [11]

    Klein, S

    A. Klein, S. Falkner, N. Mansur, and F. Hutter. RoBO : A flexible and robust B ayesian optimization framework in P ython. In NeurIPS Workshop: BayesOpt, 2017

    2017

  4. [12]

    L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. JMLR, 18: 0 185:1--185:52, 2018

    2018

  5. [13]

    Lindauer, K

    M. Lindauer, K. Eggensperger, M. Feurer, S. Falkner, A. Biedenkapp, and F. Hutter. SMAC v3: Algorithm configuration in P ython, 2017

    2017

  6. [14]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017

    2017 arXiv

  7. [15]

    Shahriari, K

    B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of B ayesian optimization. Procs. of the IEEE , 104 0 (1): 0 148--175, 2016

    2016

  8. [16]

    Snoek, H

    J. Snoek, H. Larochelle, and R. Adams. Practical B ayesian optimization of machine learning algorithms. In Proc. of NeurIPS'12, pages 2960--2968, 2012

    2012

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.