REVIEW 3 major objections 6 minor 42 references
Modernizing HEBO: a robust Bayesian optimization baseline for practical heteroskedastic and non-stationary problems
T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5
Pith's one-line read A rebuilt HEBO-style Bayesian optimizer with fixed defaults improves average performance and run-to-run stability on scientific sequential tasks while staying competitive on hyperparameter benchmarks.
desk verdict Solid engineering modernization of HEBO with broad sequential benchmarks and code; the performance claim holds, but the hetero/non-stationarity causal story is unisolated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
tidyHEBO: joint training of Yeo-Johnson (or identity) output warping with the Matérn-5/2 Gaussian process, acquisition evaluation on the untransformed objective, and NSGA-III multi-objective search over the acquisition triplet with novelty enforced inside the evolutionary loop.
What would settle it
An ablation suite that disables only the gated joint output warping (or only the untransformed acquisition evaluation) on the same Olympus and reaction grids and checks whether the mean curves and confidence-interval widths collapse back toward HEBO or LogEI would falsify the claim that those modeling choices drive the robustness gains.
Extended reading notes
Core claim
With fixed default hyperparameters, tidyHEBO delivers competitive-to-superior average performance and visibly tighter run-to-run confidence intervals than original HEBO and conventional LogEI on synthetic, Olympus, fully experimental reaction, and needle-in-a-haystack materials tasks, while remaining competitive on the Bayesmark HPO suite that originally motivated HEBO.
Load-bearing premise
The reported gains are attributed mainly to better handling of heteroskedastic noise and non-stationary landscapes, yet the paper supplies no ablation that isolates warping from the many simultaneous changes in framework, kernel, acquisition, and search, and does not directly measure those violations on the scientific tasks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces tidyHEBO, a BoTorch reimplementation and redesign of HEBO for single-objective sequential Bayesian optimization. It retains HEBO’s high-level design (input/output warping, multi-acquisition Pareto search via a genetic algorithm) while changing the kernel (Matérn 5/2), priors, joint training of Yeo–Johnson output warping with the GP, AF evaluation in the original objective space, LogEI, MC acquisition noise, NSGA-III, novelty constraints inside the GA, and full-history Pareto selection (Table 1). With fixed default hyperparameters, tidyHEBO is evaluated on 6D Ackley/Hartmann, selected Olympus emulators, Shields reaction grids, ZoMBI NIAH materials tasks, and the full Bayesmark suite. The central claim is competitive-to-superior average performance and improved robustness versus original HEBO and LogEI on the scientific tasks, while remaining competitive on Bayesmark (leaderboard score 92.64 vs HEBO 93.34).
Significance. If the empirical package holds, tidyHEBO is a useful, reproducible general-purpose BO baseline for chemistry/materials sequential experimentation, where robustness under small budgets matters as much as mean performance. Strengths include multi-run evaluation (typically 30 runs with bootstrap CIs; 20 full Bayesmark repeats), recovery of HEBO’s published Bayesmark score under sequential evaluation, explicit comparison to domain-specialized methods (EDBO, ZoMBI), and public code. The work is incremental engineering rather than a new theoretical principle, but a carefully modernized, open BoTorch baseline that transfers beyond HPO is of practical value to the applied BO community.
major comments (3)
- The title, abstract, and Results and discussion attribute gains primarily to better handling of heteroskedasticity and non-stationarity via warping and related choices. Table 1, however, simultaneously changes framework, kernel, priors, joint vs separate warping training, AF space, LogEI, MC noise, NSGA-III, novelty filtering, and Pareto selection. No ablation isolates warping/joint training (or any single change) from the rest. Without that isolation, the causal story that justifies “modernizing HEBO for practical heteroskedastic and non-stationary problems” is under-supported even if the performance claim is intact. At minimum, ablations on a subset of tasks (e.g., joint vs separate warping; warped vs untransformed AF space; NSGA-II vs III) are needed, or the interpretive claims should be substantially softened to “a modernized HEBO-inspired stack.”
- The paper never reports a direct diagnostic that the selected Olympus, reaction, or NIAH tasks actually exhibit heteroskedasticity or non-stationarity (e.g., residual variance vs predicted mean; local length-scale variation; or comparison of stationary vs warped GP fit quality on the emulator/experimental data). Figure 1 and Supplementary Note 2 are illustrative only. Without such evidence, the link from design choices to the claimed problem structure remains an assumption rather than a measured property of the benchmarks used to support the central claim.
- Olympus tasks were “manually selected … based on the number of data points … and each problem’s ability to distinguish optimizers by performance” (Results and discussion). Manual selection on discriminability risks optimistic bias for the proposed method. The paper should either (i) report the full Olympus suite (or a pre-registered subset) with the same protocol, or (ii) state the selection criteria and selection date more formally and treat the main figures as illustrative rather than as the primary support for a general-purpose scientific baseline.
minor comments (6)
- Table 1 is the most useful summary of differences; consider elevating a short “what changed and why” paragraph earlier so readers do not have to wait until Methods.
- Figure 1 panels are labeled a–d but the caption and body text are slightly hard to map; ensure consistent panel order and that “corrected versions” are clearly identified as warped models.
- NIAH results use median aggregation without CIs “for clarity”; a short note or supplementary plot with variability would help readers judge robustness claims on those tasks.
- Clarify whether the gated warping criterion σ[T(y)]/σ[y] ≥ 1/2 is applied after joint training or only as a post-hoc gate, and whether the model is fully retrained when the gate rejects warping (Methods text is slightly ambiguous).
- Minor typography: “performan ce”, “B ayesmark”, duplicated “b” labels in Figure 1, and inconsistent hyphenation of non-stationarity / heteroskedasticity.
- Bayesmark recovery of HEBO (~93.34 vs published 93.51) is a strong validation of the experimental setup; state batch-size and sequential protocol differences more prominently in the main text.
Circularity Check
No significant circularity: empirical BO methods paper whose performance claims rest on external multi-run benchmarks rather than self-referential derivation.
full rationale
tidyHEBO is presented as an engineering reconstruction of HEBO (input/output warping + multi-AF genetic search) inside BoTorch, with listed implementation changes (Matérn 5/2, joint Yeo-Johnson training, LogEI, NSGA-III, untransformed AF space, etc.; Table 1 and Methods). The load-bearing claims are sequential optimization curves and robustness (mean/median best-so-far with bootstrap CIs) on synthetic functions, Olympus emulators, Shields reaction grids, ZoMBI NIAH emulators, and Bayesmark leaderboard scores. These are evaluated against independent external oracles and baselines (LogEI, original HEBO, Optuna, EDBO, ZoMBI), not quantities defined from the same fitted parameters. The gated warping criterion, λ ~ N(1,0.5^{2}) prior, and joint MLL+Jacobian loss (Supplementary Note 1) are modeling choices whose effect is measured empirically; they do not redefine the reported objective values or force the ranking by construction. No self-citation supplies a uniqueness theorem or ansatz that the central results then merely rename. The paper is therefore self-contained against its benchmarks; any interpretive attribution of gains to heteroskedasticity/non-stationarity is a causal claim outside the circularity patterns.
Assumptions & free parameters
free parameters (5)
- Yeo-Johnson λ prior N(1, 0.5²)
- Output-warping gate threshold σ[T(y)]/σ[y] ≥ 1/2
- MC acquisition samples = 4096
- Matérn 5/2 kernel + BoTorch default scaled priors
- NSGA-III multi-objective search settings for AF Pareto front
assumptions (5)
- domain assumption A Gaussian process with Matérn 5/2 kernel plus input/output warping is an adequate surrogate class for the targeted scientific black-box objectives under small budgets.
- domain assumption Acquisition functions should be optimized in the original (untransformed) objective space when outputs are warped.
- ad hoc to paper Default hyperparameters held fixed across all tasks are a valid test of generality and robustness.
- ad hoc to paper Manually selected Olympus emulators and the chosen reaction/NIAH suites are representative enough to support a general-purpose scientific BO baseline claim.
- standard math Standard BO sequential decision model: maximize f over a bounded design space with noisy observations y = f(x) + ε.
invented entities (1)
-
tidyHEBO
independent evidence
Cite this review
Pith. "Pith review of Modernizing HEBO: a robust Bayesian optimization baseline for practical heteroskedastic and non-stationary problems." pith.science (2026). https://pith.science/paper/PNPHPBMR
@misc{pith2026260710669,
author = {Pith},
title = {Pith review of: Modernizing HEBO: a robust Bayesian optimization baseline for practical heteroskedastic and non-stationary problems},
year = {2026},
howpublished = {\url{https://pith.science/paper/PNPHPBMR}},
note = {Machine review of arXiv:2607.10669}
}
read the original abstract
Bayesian optimization is increasingly used to guide data-efficient experimentation in chemistry, materials science, and related laboratory settings, but its practical performance depends strongly on how well surrogate-model assumptions match the geometry and noise structure of the underlying objective. We introduce tidyHEBO, a robust Bayesian optimization model inspired by heteroskedastic evolutionary Bayesian optimization (HEBO) for single-objective, sequential optimization. tidyHEBO reconstructs the HEBO design philosophy in BoTorch and revises surrogate training, output-warping selection, acquisition function evaluation, and Pareto-front search. We benchmarked tidyHEBO on synthetic functions, Olympus emulators, fully experimental reaction-optimization datasets, needle-in-a-haystack (NIAH) materials problems, and Bayesmark hyperparameter optimization tasks. On these tasks tidyHEBO achieved competitive to superior performance and improvement in robustness across repeated optimization runs. We therefore propose tidyHEBO as a practical tool for sequential experimentations and a strong general-purpose benchmark for future Bayesian optimization research.
Reference graph
Works this paper leans on
-
[1]
Liang Q, Gongora AE, Ren Z, et al. Benchmarking the performance of Bayesian optimization across multiple experimental materials science doma ins. Npj Comput Mater 2021;7:188. https://doi.org/10.1038/s41524-021-00656-9
-
[2]
Bayesian o ptimization with adaptive surrogate models for automated experimental design
Lei B, Kirk TQ, Bhattacharya A, et al. Bayesian o ptimization with adaptive surrogate models for automated experimental design. Npj Comput Mater 2021;7:194. https://doi.org/10.1038/s41524-021-00662-x
-
[3]
High Performance Computing for Drug Discovery and Biomedicine
Heifetz A, editor. High Performance Computing for Drug Discovery and Biomedicine. vol. 2716. New York, NY: Springer US; 2024. https://doi.org/10.1007/978 -1- 0716-3449-3
doi:10.1007/978 2024
-
[5]
Bayesian optimization for chemical products and functional materials
Wang K, Dowling AW. Bayesian optimization for chemical products and functional materials. Current Opinion in Chem ical Engineering 2022;36:100728. https://doi.org/10.1016/j.coche.2021.100728
-
[6]
Bayesian optimization algorithms for accelerator physics
Roussel R, Edelen AL, Boltz T, et al. Bayesian optimization algorithms for accelerator physics. Phys Rev Accel Beams 2024;27:084801. https://doi.org/10.1103/PhysRevAccelBeams.27.084801
-
[7]
Race to the bottom: Bayesian optimisation for chemical problems
Wu Y, Walsh A, Ganose AM. Race to the bottom: Bayesian optimisation for chemical problems. Digital Discovery 2024;3:1086 –100. https://doi.org/10.1039/D3DD00234A
-
[8]
Weighted least -squares in calibration: What difference does it make? Analyst 2007;132:536
Tellinghuisen J. Weighted least -squares in calibration: What difference does it make? Analyst 2007;132:536. https://doi.org/10.1039/b701696d
-
[9]
Jain RB. Comparison of three weighting schemes in weighted regression analysis for use in a chemistry laboratory. Clinica Chimica Acta 2010;411:270 –9. https://doi.org/10.1016/j.cca.2009.11.021
Show all 42 references
-
[10]
Validation of quantitative NMR
Malz F, Jancke H. Validation of quantitative NMR. Journal of Pharmaceutical and Biomedical Analysis 2005;38:813–23. https://doi.org/10.1016/j.jpba.2005.01.043
2005 doi
-
[11]
Gaussian processes for machine learning
Rasmussen CE, Williams CKI. Gaussian processes for machine learning. 3. print. Cambridge, MA: MIT Press; 2006. https://doi.org/10.7551/mitpress/3206.001.0001
2006 doi
-
[12]
HEBO: Pushing The Limits of Sample- Efficient Hyper -parameter Optimisation
Cowen-Rivers AI, Lyu W, Tutunov R, et al. HEBO: Pushing The Limits of Sample- Efficient Hyper -parameter Optimisation. Journal of Artificial Intelligence Research 2022;74:1269–349. https://doi.org/10.1613/jair.1.13643
2022 doi
-
[14]
Warped Gaussian Processes and Derivative-Based Sequential Designs for Functions with Heterogeneous Variations
Marmin S, Ginsbourger D, Baccou J, et al. Warped Gaussian Processes and Derivative-Based Sequential Designs for Functions with Heterogeneous Variations. SIAM/ASA J Uncertainty Quantification 2018; 6:991–1018. https://doi.org/10.1137/17M1129179
2018 doi
-
[15]
An additive Gaussian process regression model for interpretable non -parametric analysis of longitudinal data
Cheng L, Ramchandran S, Vatanen T, et al. An additive Gaussian process regression model for interpretable non -parametric analysis of longitudinal data. Nat Commun 2019;10:1798. https://doi.org/10.1038/s41467-019-09785-8
2019 doi
-
[16]
Continuous flow Buchwald–Hartwig amination of a pharmaceutical intermediate
Yaseneva P, Hodgson P, Zakrzewski J, et al. Continuous flow Buchwald–Hartwig amination of a pharmaceutical intermediate. Reaction Chemistry & Engineering 2016;1:229–
2016
-
[17]
https://doi.org/10.1039/C5RE00048C
-
[18]
Surrogates: Gaussian process modeling, design, and optimization for the applied sciences
Gramacy RB. Surrogates: Gaussian process modeling, design, and optimization for the applied sciences. Chapman and Hall/CRC; 2020
2020
-
[19]
Toward real-world automated antibody design with combinatorial Bayesian optimization
Khan A, Cowen-Rivers AI, Grosnit A, et al. Toward real-world automated antibody design with combinatorial Bayesian optimization. Cell Reports Methods 2023;3 :100374. https://doi.org/10.1016/j.crmeth.2022.100374
2023 doi
-
[25]
A Literature Survey of Benchmark Functions For Global Optimization Problems
Jamil M, Yang X -S. A Literature Survey of Benchmark Functions For Global Optimization Problems. IJMMNO 2013;4:150. https://doi.org/10.1504/IJMMNO.2013.055204
2013 doi
-
[26]
Optuna: A Next -generation Hyperparameter Optimization Framework
Akiba T, Sano S, Yanase T, et al. Optuna: A Next -generation Hyperparameter Optimization Framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, NY, USA: Associa tion for Computing Machinery; 2019, p. 2623–31. http...
2019 doi
-
[27]
Imprecise Bay esian optimization
Rodemann J, Augustin T. Imprecise Bay esian optimization. Knowledge -Based Systems 2024;300:112186. https://doi.org/10.1016/j.knosys.2024.112186
2024 doi
-
[28]
Revisiting Bayesian Optimization in the light of the COCO benchmark
Riche RL, Picheny V . Revisiting Bayesian Optimization in the light of the COCO benchmark. Struct Multidisc Optim 2021;64:3063 –87. https://doi.org/10.1007/s00158-021- 02977-1
2021 doi
-
[29]
Bayesian Optimization for Anything (BOA): An open -source framework for accessible, user -friendly Bayesian optimization
Scyphers ME, Missik JEC, Kujawa H, et al. Bayesian Optimization for Anything (BOA): An open -source framework for accessible, user -friendly Bayesian optimization. Environmental Modelling & Software 2024;182:106191. https://doi.org/10.1016/j.envsoft.2024.106191
2024 doi
-
[30]
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek J, Larochelle H, Adams RP. Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems, vol. 25, Curran Associates, Inc.; 2012. https://doi.org/10.48550/arXiv.1206.2944
-
[31]
Vanilla Bayesian Optimization Performs Great in High Dimensions
Hvarfner C, Hellsten EO, Nardi L. Vanilla Bayesian Optimization Performs Great in High Dimensions. Proceedings of the 41st International Conference on Machine Learning, PMLR; 2024, p. 20793–817. https://doi.org/10.48550/arXiv.2402.02229
-
[32]
huawei -noah/HEBO: v0.3.4 2022
Grosnit A, Lyu W, Cowen -Rivers A, et al. huawei -noah/HEBO: v0.3.4 2022. https://doi.org/10.5281/zenodo.7344859
2022 doi
-
[33]
An Analysis of Transformations
Box GEP, Cox DR. An Analysis of Transformations. Journal of the Royal Statistical Society Series B: Statistical Methodology 1964;26:211 –43. https://doi.org/10.1111/j.2517- 6161.1964.tb00553.x
1964 doi
-
[34]
The Box -Cox Transformation Technique: A Review
Sakia RM. The Box -Cox Transformation Technique: A Review. The Statistician 1992;41:169. https://doi.org/10.2307/2348250
1992 doi
-
[35]
A new family of power transformations to improve normality or symmetry
Yeo I-K. A new family of power transformations to improve normality or symmetry. Biometrika 2000;87:954–9. https://doi.org/10.1093/biomet/87.4.954
2000 doi
-
[36]
Warped Gaussian Processes
Snelson E, Ghahramani Z, Rasmussen C. Warped Gaussian Processes. Advances in Neural Information Processing Systems, vol. 16, MIT Press; 2003, p. 337–44
2003
-
[37]
Unexpected Improvements to Expected Improvement for Bayesian Optimization
Ament S, Daulton S, Eriksson D, et al. Unexpected Improvements to Expected Improvement for Bayesian Optimization. Advances in Neural Information Processing Systems, vol. 36, Curran Associates, Inc.; 2023, p. 20577 –612. https://doi.org/10.48550/arXiv.2310.20708
- [38]
-
[39]
Pymoo: Multi -Objective Optimization in Python
Blank J, Deb K. Pymoo: Multi -Objective Optimization in Python. IEEE Access 2020;8:89497–509. https://doi.org/10.1109/ACCESS.2020.2990567
2020 doi
-
[40]
A fast and elitist multiobjective genetic algorithm: NSGA-II
Deb K, Pratap A, Agarwal S, et al. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Computat 2002;6:182–97. https://doi.org/10.1109/4235.996017
2002 doi
-
[41]
optimization step
Deb K, Jain H. An Evolutionary Many -Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints. IEEE Trans Evol Computat 2014;18:577 –601. https://doi.org/10.1109/TEVC.2013.2281535. Supplementary m...
2014 doi
- [42]
-
[43]
Olympus: a benchmarking framework for noisy optimization and experiment planning
Häse F, Aldeghi M, Hickman RJ, et al. Olympus: a benchmarking framework for noisy optimization and experiment planning. Mach Learn: Sci Technol 2021;2:035021. https://doi.org/10.1088/2632-2153/abedc8
2021 doi
-
[44]
Olympus, enhanced: benchmarking mixed- parameter and multi-objective optimization in chemistry and materials science 2023
Hickman R, Parakh P, Cheng A, et al. Olympus, enhanced: benchmarking mixed- parameter and multi-objective optimization in chemistry and materials science 2023. https://doi.org/10.26434/chemrxiv-2023-74w8d
2023 doi
-
[45]
Bayesian reaction optimization as a tool for chemical synthesis
Shields BJ, Stevens J, Li J, et al. Bayesian reaction optimization as a tool for chemical synthesis. Nature 2021;590:89–96. https://doi.org/10.1038/s41586-021-03213-y
2021 doi
-
[46]
BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
Balandat M, Karrer B, Jiang D, et al. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems, vol. 33, Curran Associates, Inc.; 2020, p. 21524–38. https://doi.org/10.48550/arXiv.1910.06403
-
[47]
Fast Bayesian optimization of Needle-in-a-Haystack problems using zooming memory-based initialization (ZoMBI)
Siemenn AE, Ren Z, Li Q, et al. Fast Bayesian optimization of Needle-in-a-Haystack problems using zooming memory-based initialization (ZoMBI). Npj Comput Mater 2023;9:79. https://doi.org/10.1038/s41524-023-01048-x
2023 doi
-
[48]
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
Turner R, Eriksson D, McCourt M, et al. Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020. Proceedings of the NeurIPS 2020 Competition and Demonstration Track, vol. 133, PMLR; 20...
- [49]
Reviewed July 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.