{"id":"8646af24-5a73-4dcc-93cc-fbeee00742ff","arxiv_id":"2607.10669","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"tidyHEBO modernizes HEBO in BoTorch and shows competitive-to-better, more robust sequential optimization on scientific and HPO benchmarks under default hyperparameters.","lead":"tidyHEBO is a BoTorch rebuild of HEBO that revises warping, surrogate training, and multi-acquisition Pareto search for sequential black-box optimization. It is offered as a more robust baseline for chemistry and materials experiments where noise and landscape shape break standard Gaussian-process assumptions.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"The performance/robustness claim is empirically supported, but the causal attribution to heteroskedasticity/non-stationarity handling remains unisolated and is the softest load-bearing link.","rationale":"The Reader correctly identifies the missing ablation and lack of direct diagnostics as the weakest assumption. The empirical performance/robustness package itself is not empty: repeated sequential runs, recovery of HEBO’s Bayesmark leaderboard score under sequential batch-1 evaluation, and public code make the practical baseline claim usable. The concern that remains load-bearing is precisely the causal attribution that the paper’s title, abstract, and discussion lean on. Because that attribution is not required for the weaker claim “tidyHEBO is a competitive, more robust default package,” the appropriate verdict stays CONDITIONAL rather than REJECT: the work is still a useful engineering contribution and benchmark candidate once the interpretive over-claim is tightened or ablated. No stronger internal inconsistency or numerical red flag appears in the reported curves or methods. Agreement with the Reader is therefore full on both the soft spot and the overall CONDITIONAL stance.","tokens_in":13828,"tokens_out":720,"duration_ms":8427,"concrete_test":"Run a minimal ablation on the four main Olympus tasks (AGNP, HPLC, colors_bob, crossed_barrel) and the two NIAH emulators: (i) tidyHEBO full, (ii) tidyHEBO with identity output warping forced (no Yeo-Johnson), (iii) tidyHEBO with input warping disabled, (iv) original HEBO reimplemented in BoTorch with only the kernel/prior swap. If mean best-value curves and bootstrap CI widths of (ii)/(iii) remain statistically indistinguishable from full tidyHEBO, the heteroskedasticity/non-stationarity attribution does not land and the claim should be reframed as pure engineering modernization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that tidyHEBO, under fixed defaults, delivers competitive-to-superior average performance and tighter robustness (narrower CIs / better medians) than original HEBO and LogEI on synthetic, Olympus, experimental reaction, and NIAH tasks while remaining competitive on Bayesmark. That empirical package is largely intact: multi-run curves, recovery of HEBO’s Bayesmark score (~93.34 vs published 93.51), and code availability support the practical baseline claim. The load-bearing soft spot is the interpretive step that attributes the gains primarily to better handling of heteroskedasticity and non-stationarity via the warping and related design choices (Introduction; Results and discussion; Table 1; Methods). Table 1 simultaneously changes framework (BoTorch), kernel (Matérn 5/2 vs 3/2+Linear), priors, joint vs separate output-warping training, gated Yeo-Johnson only, AF evaluation space (untransformed vs warped), LogEI, MC noise, NSGA-III, novelty constraint, and full-history Pareto selection. No ablation isolates warping/joint training from the other engineering changes, and the paper never reports a direct diagnostic (e.g., residual variance vs mean, or length-scale variation) that the selected Olympus/reaction/NIAH tasks actually exhibit the claimed heteroskedasticity or non-stationarity. Without that isolation, the causal story that justifies “modernizing HEBO for practical heteroskedastic and non-stationary problems” is under-supported even if the performance numbers hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces tidyHEBO, a BoTorch reimplementation and redesign of HEBO for single-objective sequential Bayesian optimization. It retains HEBO’s high-level design (input/output warping, multi-acquisition Pareto search via a genetic algorithm) while changing the kernel (Matérn 5/2), priors, joint training of Yeo–Johnson output warping with the GP, AF evaluation in the original objective space, LogEI, MC acquisition noise, NSGA-III, novelty constraints inside the GA, and full-history Pareto selection (Table 1). With fixed default hyperparameters, tidyHEBO is evaluated on 6D Ackley/Hartmann, selected Olympus emulators, Shields reaction grids, ZoMBI NIAH materials tasks, and the full Bayesmark suite. The central claim is competitive-to-superior average performance and improved robustness versus original HEBO and LogEI on the scientific tasks, while remaining competitive on Bayesmark (leaderboard score 92.64 vs HEBO 93.34).","tokens_in":14256,"tokens_out":1215,"duration_ms":13039,"significance":"If the empirical package holds, tidyHEBO is a useful, reproducible general-purpose BO baseline for chemistry/materials sequential experimentation, where robustness under small budgets matters as much as mean performance. Strengths include multi-run evaluation (typically 30 runs with bootstrap CIs; 20 full Bayesmark repeats), recovery of HEBO’s published Bayesmark score under sequential evaluation, explicit comparison to domain-specialized methods (EDBO, ZoMBI), and public code. The work is incremental engineering rather than a new theoretical principle, but a carefully modernized, open BoTorch baseline that transfers beyond HPO is of practical value to the applied BO community.","major_comments":[{"comment":"The title, abstract, and Results and discussion attribute gains primarily to better handling of heteroskedasticity and non-stationarity via warping and related choices. Table 1, however, simultaneously changes framework, kernel, priors, joint vs separate warping training, AF space, LogEI, MC noise, NSGA-III, novelty filtering, and Pareto selection. No ablation isolates warping/joint training (or any single change) from the rest. Without that isolation, the causal story that justifies “modernizing HEBO for practical heteroskedastic and non-stationary problems” is under-supported even if the performance claim is intact. At minimum, ablations on a subset of tasks (e.g., joint vs separate warping; warped vs untransformed AF space; NSGA-II vs III) are needed, or the interpretive claims should be substantially softened to “a modernized HEBO-inspired stack.”","section":null},{"comment":"The paper never reports a direct diagnostic that the selected Olympus, reaction, or NIAH tasks actually exhibit heteroskedasticity or non-stationarity (e.g., residual variance vs predicted mean; local length-scale variation; or comparison of stationary vs warped GP fit quality on the emulator/experimental data). Figure 1 and Supplementary Note 2 are illustrative only. Without such evidence, the link from design choices to the claimed problem structure remains an assumption rather than a measured property of the benchmarks used to support the central claim.","section":null},{"comment":"Olympus tasks were “manually selected … based on the number of data points … and each problem’s ability to distinguish optimizers by performance” (Results and discussion). Manual selection on discriminability risks optimistic bias for the proposed method. The paper should either (i) report the full Olympus suite (or a pre-registered subset) with the same protocol, or (ii) state the selection criteria and selection date more formally and treat the main figures as illustrative rather than as the primary support for a general-purpose scientific baseline.","section":null}],"minor_comments":[{"comment":"Table 1 is the most useful summary of differences; consider elevating a short “what changed and why” paragraph earlier so readers do not have to wait until Methods.","section":null},{"comment":"Figure 1 panels are labeled a–d but the caption and body text are slightly hard to map; ensure consistent panel order and that “corrected versions” are clearly identified as warped models.","section":null},{"comment":"NIAH results use median aggregation without CIs “for clarity”; a short note or supplementary plot with variability would help readers judge robustness claims on those tasks.","section":null},{"comment":"Clarify whether the gated warping criterion σ[T(y)]/σ[y] ≥ 1/2 is applied after joint training or only as a post-hoc gate, and whether the model is fully retrained when the gate rejects warping (Methods text is slightly ambiguous).","section":null},{"comment":"Minor typography: “performan ce”, “B ayesmark”, duplicated “b” labels in Figure 1, and inconsistent hyphenation of non-stationarity / heteroskedasticity.","section":null},{"comment":"Bayesmark recovery of HEBO (~93.34 vs published 93.51) is a strong validation of the experimental setup; state batch-size and sequential protocol differences more prominently in the main text.","section":null}],"recommendation":"major_revision","confidential_remarks":"The empirical baseline claim is credible and the code release is a real contribution; the main risk is over-claiming a causal story about heteroskedasticity/non-stationarity without ablations or diagnostics. I would accept after major revision if the authors either add targeted ablations or rewrite the framing as an engineering modernization with empirical transfer results. Fit for a methods/applications venue is good; less so for a pure theory venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is that tidyHEBO is a careful BoTorch rebuild of HEBO’s warping-plus-multi-AF idea, with several consistency fixes, and it delivers competitive-to-better average curves plus visibly tighter robustness than original HEBO and plain LogEI on synthetic, Olympus, Shields reaction grids, and NIAH tasks while staying close on Bayesmark (92.64 vs 93.34). That package is real and reproducible; they recover HEBO’s published BBO score under sequential evaluation, ship code, and run 30-trial bootstrap CIs (median on NIAH).\n\nWhat is actually new is not a new principle. Input/output warping, Matérn GPs, LogEI, and NSGA-style Pareto search over a triplet of AFs are established. The contribution is the specific reconstruction: joint training of Yeo-Johnson with the GP (instead of separate univariate fit), AF evaluation in the original objective space, gated warping with a mild λ prior, MC acquisition noise instead of an extra hyperparameter, NSGA-III plus novelty-as-constraint and full-history Pareto selection, plus default hyperparameters held fixed across domains. Table 1 and the Methods section make those deltas clear. The multi-domain sequential evidence under fixed defaults is the paper’s strongest asset; it is more than a pure HPO re-tune.\n\nThe soft spot is exactly the one the stress-test flags, and it is real but not fatal. Gains are attributed mainly to better handling of heteroskedasticity and non-stationarity, yet there is no ablation that isolates warping/joint training from the simultaneous framework, kernel, prior, AF, and search changes, and no residual or length-scale diagnostic showing that the chosen Olympus/reaction/NIAH tasks actually exhibit those pathologies. Manual Olympus task selection is a minor additional caveat. The empirical performance claim still stands; the interpretive title claim is under-supported.\n\nThis is for applied BO people who need a practical sequential baseline for chemistry/materials experiment planning, and for anyone who wants a modern HEBO reference implementation. Math and citations look standard and solid; free parameters are ordinary modeling choices, not circular. I would send it to peer review. Engage with the code and the numbers; treat the causal story as provisional until someone runs the ablations.","headline":"Solid engineering modernization of HEBO with broad sequential benchmarks and code; the performance claim holds, but the hetero/non-stationarity causal story is unisolated.","tokens_in":14871,"tokens_out":560,"would_cite":true,"duration_ms":8194,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A rebuilt HEBO-style Bayesian optimizer with fixed defaults improves average performance and run-to-run stability on scientific sequential tasks while staying competitive on hyperparameter benchmarks.","keywords":["Bayesian optimization","heteroskedasticity","non-stationarity","output warping","Gaussian process","acquisition functions","sequential experimentation","materials optimization"],"falsifier":"An ablation suite that disables only the gated joint output warping (or only the untransformed acquisition evaluation) on the same Olympus and reaction grids and checks whether the mean curves and confidence-interval widths collapse back toward HEBO or LogEI would falsify the claim that those modeling choices drive the robustness gains.","tokens_in":14687,"feed_emoji":"🔬","tokens_out":660,"duration_ms":8189,"temperature":0.7,"pith_summary":"Bayesian optimization is used for expensive black-box experiments in chemistry and materials science, but standard Gaussian-process surrogates assume constant noise and stationary length scales that real objectives often violate. The authors rebuild the HEBO design philosophy inside a modern framework, revise how output warping is trained and gated, evaluate acquisitions in the original objective space, and search a multi-acquisition Pareto front with a more diversity-aware genetic algorithm. Across synthetic functions, chemistry and 3D-printing emulators, fully experimental reaction grids, needle-in-a-haystack materials problems, and the Bayesmark hyperparameter suite, the resulting model (tidyHEBO) matches or beats original HEBO and a conventional LogEI baseline on average best value while producing tighter confidence bands across repeated runs. Because laboratory budgets are small and a method typically gets only one shot, the authors present the more stable general-purpose optimizer as both a practical tool and a stronger baseline for future work.","feed_headline":"Rebuilt HEBO beats baselines on lab tasks with tighter runs","feed_subtitle":"Fixed-default tidyHEBO stays competitive on HPO while improving average and stability on chemistry and materials problems","key_machinery":"tidyHEBO: joint training of Yeo-Johnson (or identity) output warping with the Matérn-5/2 Gaussian process, acquisition evaluation on the untransformed objective, and NSGA-III multi-objective search over the acquisition triplet with novelty enforced inside the evolutionary loop.","core_discovery":"With fixed default hyperparameters, tidyHEBO delivers competitive-to-superior average performance and visibly tighter run-to-run confidence intervals than original HEBO and conventional LogEI on synthetic, Olympus, fully experimental reaction, and needle-in-a-haystack materials tasks, while remaining competitive on the Bayesmark HPO suite that originally motivated HEBO.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["tidyHEBO rebuild tops HEBO and LogEI with tighter lab-run intervals","Fixed-default tidyHEBO tightens variance on chemistry and NIAH tasks","tidyHEBO matches HEBO on HPO while beating it on experimental problems","Robust tidyHEBO improves average and stability over original HEBO","Rebuilt HEBO in BoTorch yields tighter confidence on reaction datasets"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The reported gains are attributed mainly to better handling of heteroskedastic noise and non-stationary landscapes, yet the paper supplies no ablation that isolates warping from the many simultaneous changes in framework, kernel, acquisition, and search, and does not directly measure those violations on the scientific tasks.","fun_headline_variants_meta":{"raw":{"variants":["tidyHEBO rebuild tops HEBO and LogEI with tighter lab-run intervals","Fixed-default tidyHEBO tightens variance on chemistry and NIAH tasks","tidyHEBO matches HEBO on HPO while beating it on experimental problems","Robust tidyHEBO improves average and stability over original HEBO","Rebuilt HEBO in BoTorch yields tighter confidence on reaction datasets"]},"model":"grok-4.5","effort":"low","cost_usd":0.003354,"raw_usage":{"total_tokens":1079,"prompt_tokens":732,"num_sources_used":0,"completion_tokens":100,"cost_in_usd_ticks":33540000,"prompt_tokens_details":{"text_tokens":732,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":247,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":732,"tokens_out":100,"duration_ms":3887,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T10:05:08.572503+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"An ablation suite that disables only the gated joint output warping (or only the untransformed acquisition evaluation) on the same Olympus and reaction grids and checks whether the mean curves and confidence-interval widths collapse back toward HEBO or LogEI would falsify the claim that those modeling choices drive the robustness gains.","supporting_citations":[],"review_version":1}