{"id":"9c50a191-17d6-4df9-b124-ecde1a16d7ff","arxiv_id":"2601.06820","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Bgolearn is an open-source Bayesian-optimization package for materials science that reaches benchmarks in ~70% fewer iterations than random search and suggested a very hard high-entropy alloy and better medium-Mn steel heat treatments.","lead":"This paper presents Bgolearn, an open-source Python toolkit that applies Bayesian optimization to materials discovery, with a graphical interface and materials-focused workflows. It matters because accessible, experiment-efficient optimization could cut the number of costly synthesis-and-test cycles needed to find better alloys, steels, and cellular structures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 40–60% reduction claim is not supported by the benchmarks: grid search and single-objective genetic algorithms are absent.","rationale":"I focused on the quantitative headline claim. The abstract and conclusion assert a 40–60% reduction relative to random search, grid search, and genetic algorithms. The benchmark section (Table 1 and Appendix C) reports only random search, LHS, and NSGA-II for multi-objective. No grid search or single-objective GA data are provided. This is an internal mismatch: the evidence does not cover the baselines named in the claim. The observed reductions in Table 1 vary from ~42% to ~79%, so the specific 40–60% figure is also not clearly derived. This concern is distinct from but complementary to the reader's point about noiseless benchmarks; both point to the need to revise the abstract or supply additional benchmarks. I propose adding the missing baselines as the decisive test. If the reductions vs. grid search and single-objective GA are consistent with 40–60%, the headline claim can stand; otherwise it should be corrected. The reader's weakest assumption (noisy transfer) is valid as an external-validity concern, but the missing baselines are a more immediate internal support issue; hence partial agreement.","tokens_in":24913,"tokens_out":5997,"duration_ms":60174,"concrete_test":"Implement a standard grid search on the same Hartmann-6D and Ackley-5D benchmarks (with the same 20 LHS initial points if applicable and same 3-candidate batch protocol) and a single-objective genetic algorithm; compute iterations to 90% optimality and the percentage reduction for Bgolearn-GP/EI. If the reductions are not in the 40–60% range relative to these baselines, the abstract claim should be revised to match the evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim Bgolearn reduces required experiments by 40–60% 'compared with random search, grid search, and genetic algorithms.' However, Table 1 and Appendix C only benchmark random search, Latin hypercube sampling, and NSGA-II (multi-objective only). No grid search results appear anywhere, and no single-objective genetic algorithm is tested. The paper itself states in 'Comparison with Baselines' that comparisons were made 'with random search and Latin hypercube sampling (LHS), and, for multi-objective problems, with NSGA-II.' Thus the quantitative central claim is not derivable from the presented data. Additionally, the observed reductions in Table 1 range from ~42% to ~79% depending on baseline and surrogate, so the specific '40–60%' figure appears arbitrary rather than an average of the reported numbers. This is a load-bearing mismatch: the claim names baselines that were not evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Bgolearn, an open-source Python framework for single- and multi-objective Bayesian optimization tailored to materials discovery. It describes the modular architecture (data, surrogate, and acquisition layers), multiple acquisition functions and surrogate models, bootstrap-based uncertainty quantification, and a GUI ('BgoFace'). The central quantitative claim is that benchmark experiments show a 40–60% reduction in the number of required experiments relative to random search, grid search, and genetic algorithms. The paper also reports three application case studies: TPMS elastic-modulus optimization, ultra-hard high-entropy alloy discovery, and medium-Mn steel heat-treatment design, the latter two with new experimental validation. Extensive appendix material provides benchmark definitions, data tables, and implementation details.","tokens_in":25187,"tokens_out":5890,"duration_ms":53372,"significance":"If the efficiency claim is properly substantiated, Bgolearn would be a practically valuable contribution to materials informatics: it lowers the programming barrier, offers multi-objective acquisition functions, and is backed by real experimental demonstrations (HEA nanoindentation and medium-Mn steel tensile data) rather than synthetic examples alone. The open-source release, large user base, and detailed appendix tables also support reproducibility. However, the headline benchmark claim is not supported by the reported baseline comparisons, and the multi-objective results do not measure experimental-effort reduction. The paper's significance therefore hinges on fixing or reframing this claim.","major_comments":[{"comment":"The statement that Bgolearn 'reduces the number of required experiments by 40–60% compared with random search, grid search, and genetic algorithms' is not derivable from Table 1. The table contains only Random Search, Latin Hypercube Sampling, NSGA-II, and Bgolearn variants; no grid search or single-objective genetic algorithm appears. The text itself says comparisons were made 'with random search and Latin hypercube sampling (LHS), and, for multi-objective problems, with NSGA-II.' Moreover, the reported ratios are not uniformly 40–60%: Hartmann-6D Bgolearn-GP achieves ~79% reduction vs random search and ~69% vs LHS; Ackley-5D shows ~69% and ~54%. Please add the missing baselines or revise the abstract/conclusion to state exactly what was compared.","section":"Abstract/Conclusion vs. 'Comparison with Baselines' and Table 1"},{"comment":"The multi-objective rows report normalized hypervolume after 50 fixed iterations. Hypervolume at a fixed budget measures solution quality, not the number of experiments needed to reach a target. Thus the paper's central 'experimental effort reduction' claim cannot be extended to multi-objective problems on the basis of these data. Report an effort-to-target metric (e.g., iterations to reach a fixed hypervolume fraction) for all methods, or explicitly restrict the effort-reduction claim to the single-objective benchmarks.","section":"Table 1, multi-objective rows (ZDT1, DTLZ2)"},{"comment":"The TPMS case study improves the best elastic modulus from 8,560 MPa (in 50 initial samples) to 8,945 MPa after four additional evaluations, a ~4.5% gain. Without a control—e.g., random search, LHS, or a genetic algorithm given the same 4-evaluation budget on the same surrogate—this example demonstrates only that Bgolearn found a better point, not that it reduces experimental effort. The HEA and medium-Mn steel examples likewise lack a quantitative baseline comparison, although the experimental validations themselves are valuable.","section":"Real-World Materials Discovery, TPMS subsection"}],"minor_comments":[{"comment":"The sentence 'as evidenced by a series of publications5, 11–15, which are not listed here individually' is contradictory because references 11–15 are listed. Please rephrase.","section":"Introduction, references"},{"comment":"There is a typo: 'combining Gaussian process surrogates with with EI et al. acquisition strategies' should read 'with EI and other acquisition strategies.'","section":"Results, Functional materials"},{"comment":"'exceeding the upper bound of the best performance reported in literates' should be 'literature.' Also, the recommended Al46.47Co9.16Cr23.47Cu7.22Fe8.10Ni5.58 has Al slightly above the maximum Al in the training data (46.2 at.%); please clarify whether this is intended and how extrapolation was assessed.","section":"Results, HEA subsection"},{"comment":"The benchmark description says 'three new candidate points per iteration' for all methods, but the single-objective results are reported as iterations to 90% optimality. It would help to state explicitly whether each iteration means one batch of three evaluations or a single evaluation, and how batch size affects the iteration counts.","section":"Comparison with Baselines / Figure 3"},{"comment":"The user study (15 researchers, 5 without programming experience) is reported without a protocol or statistical detail. If this is kept, please provide the task, metrics, and variability; otherwise, mark it as anecdotal.","section":"BgoFace user study"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of a software ecosystem report than a new algorithmic contribution. The case studies and open-source release are real strengths, but the central quantitative claim (40–60% reduction vs random search, grid search, and genetic algorithms) is not backed by the reported experiments. This is fixable: add the missing baseline columns or rewrite the claim. If the authors choose to add grid search and single-objective GA baselines, they should also clarify whether the multi-objective claim concerns effort or quality. I believe major revision is appropriate; rejection is not warranted given the package's utility and the existence of new experimental validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a software-package paper, not a methods paper, and the strongest part is the packaged integration. The headline claim, though, overstates what the benchmarks actually show. The abstract says Bgolearn cuts experimental effort by 40–60% versus random search, grid search, and genetic algorithms. Table 1 only includes random search, Latin hypercube sampling, and NSGA-II (multi-objective only). No grid search anywhere, no single-objective GA. The observed reductions on the two single-objective benchmarks are roughly 54–79%, depending on baseline, so the specific 40–60% range is not derivable from the data. That is a real mismatch between claim and evidence.\n\nWhat the paper does well: the framework itself is a legitimate contribution. It packages GP, RF, GB surrogates with several acquisition functions, multi-objective support using EHVI/qNEHVI, bootstrap uncertainty for non-GP models, and a GUI. The code is open source and the benchmark scripts are reproducible. The case studies are also genuinely new results: the TPMS configuration with E = 8945 MPa, the Al46.47Co9.16Cr23.47Cu7.22Fe8.10Ni5.58 HEA with nanoindentation hardness around 1029 HV, and the two medium-Mn steel heat treatments that extend the experimental Pareto front. These are concrete, externally verifiable claims.\n\nSoft spots, in proportion. First and most important: the 40–60% experimental-effort reduction is derived from noiseless analytic functions (Hartmann-6D, Ackley-5D, ZDT1, DTLZ2), then transferred to real experiments with noise, discrete variables, and expensive evaluations. That transfer is asserted, not tested. The benchmark result should be presented as algorithm iterations on synthetic functions, not as a guaranteed reduction in experimental effort. Second, the missing baselines matter because grid search and single-objective GA are named in the abstract. Adding them (or changing the claim) is necessary. Third, the HEA case is missing the synthesis route; the hardness data is solid nanoindentation, but without alloy preparation details the result is hard to replicate. Fourth, the TPMS demonstration uses only four recommended evaluations after 50 initial samples—encouraging but thin.\n\nThe citation pattern is fine; the paper cites BoTorch, qNEHVI, and related BO literature appropriately. Self-citations are to prior applications of the same package, which is normal in a software report.\n\nBottom line: this deserves a serious referee as a software/tool paper, but not with the current abstract. The central claim needs to be recalibrated, baselines added, and the HEA synthesis details supplied. A competent referee would likely request exactly these revisions before acceptance.","headline":"Useful open-source BO toolkit, but the headline 40–60% claim is not supported by the reported benchmarks—grid search and single-objective GA were never run.","tokens_in":25753,"tokens_out":1248,"would_cite":true,"duration_ms":15430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents Bgolearn, a Python framework that makes Bayesian optimization practical for materials research, and claims it cuts required experiments by 40–60% while finding new alloys and structures.","keywords":["Bayesian optimization","materials discovery","multi-objective optimization","active learning","high-entropy alloys","medium-Mn steel","triply periodic minimal surfaces","open-source software"],"falsifier":"Run Bgolearn and a random-search baseline on a real materials system—for example, the same medium-Mn steel heat-treatment protocol—starting both from the same 16 initial experiments and proposing three candidates per round, with repeated noisy replicates. If Bgolearn does not reach the same strength–ductility frontier in materially fewer evaluations, or does not beat random search at all, the headline reduction claim is falsified.","tokens_in":24808,"feed_emoji":"🧪","tokens_out":5971,"duration_ms":55891,"temperature":0.7,"pith_summary":"The paper sets out to show that the adoption barrier for Bayesian optimization in materials science is largely an implementation problem, and that a unified open-source package can solve it. Bgolearn bundles data handling, surrogate models, acquisition functions, and uncertainty quantification behind a simple interface, so a materials researcher can run single- or multi-objective optimization with minimal code. The quantitative anchor is a benchmark result: Bgolearn reaches 90-percent optimality on Hartmann-6D in 18±4 iterations and on Ackley-5D in 22±4 iterations, versus 87±15 and 72±18 for random search, which the paper translates into a 40–60% reduction in experimental effort. Its case studies then show the framework recommending an ultra-hard high-entropy alloy (>1000 HV), a TPMS structure with elastic modulus 8,945 MPa, and medium-Mn steel heat treatments that extend the empirical strength–ductility frontier. The reader is being asked to accept that BO's sample-efficiency advantages, long demonstrated in theory, can now be delivered in practice without specialized programming.","feed_headline":"One open-source toolbox cuts materials experiments by 40–60%","feed_subtitle":"Bgolearn wraps Bayesian optimization in a simple API and finds harder alloys, stiffer structures, and stronger steels.","key_machinery":"The carrying mechanism is the Bgolearn package itself: a three-layer architecture—data, surrogate, acquisition—that reduces a Bayesian-optimization loop to a few lines of code while allowing expert customization. Within it, the acquisition functions do the decision-making work: expected improvement is the default single-objective driver, expected hypervolume improvement (EHVI), which picks candidates that most expand the dominated volume in objective space, guides multi-objective searches, and bootstrap-resampled ensembles supply the predictive uncertainty that non-GP surrogates lack. The key design claim is that automating normalization, cross-validation, jitter, and convergence diagnostics","core_discovery":"On its own terms, the paper's central discovery is that a unified, materials-oriented Bayesian-optimization framework can deliver both ease of use and practical discovery gains. It reports that, across standard single- and multi-objective benchmarks, Bgolearn reaches 90-percent optimality in roughly 18–28 iterations where random search needs 72–87, and it achieves higher hypervolume on ZDT1 and DTLZ2 than genetic search. The framework combines five surrogate families, five single-objective and four multi-objective acquisition functions, and bootstrap-based uncertainty quantification so that non-Gaussian-process surrogates can still drive exploration. Applied to three real problems, Bgolearn","pith_inferences":["Editorial inference: the headline 40–60% savings is measured on noiseless analytic functions; the same advantage on real materials experiments, where noise and discrete variables are unavoidable, is plausible but unproven by the data shown.","Editorial inference: each real-world case study validates only a small number of recommended candidates, and there is no random-search control run in the same materials spaces; replication with more candidates and a baseline would sharpen the evidence.","Editorial inference: a direct extension would be a closed-loop campaign that alternates Bgolearn recommendations with noisy replicate experiments and records the number of evaluations needed to reach a target property, testing whether the noiseless benchmark ratio survives.","Editorial inference: because the framework is modular and open source, its acquisition-function defaults could be stress-tested against alternative policies (for example, pure random exploration or batch random sampling) on the same benchmarks to separate framework gains from BO's intrinsic gains."],"forward_implications":["If the 40–60% benchmark figure transfers, a materials team with a fixed experimental budget can explore roughly twice as wide a design space, or cut the time-to-discovery by about half.","Multi-objective optimization is built in, so strength-versus-ductility and similar trade-offs can be managed directly rather than by scalarizing targets.","The bootstrap uncertainty estimate extends BO beyond Gaussian processes, allowing large or mixed-variable datasets—common in composition and processing spaces—to drive the search.","The graphical interface lets researchers with no programming background run BO and auto-generates equivalent code, changing who is able to use active learning in the lab.","The three case studies identify concrete candidate materials and processing schedules that were previously unexplored."],"fun_headline_variants":["Bgolearn: 40–60% fewer experiments to find optimal materials","Open-source toolkit Bgolearn accelerates materials discovery via BO","Unified Bayesian optimization framework Bgolearn slashes materials testing","Bgolearn makes Bayesian optimization practical for materials scientists"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the 40–60% reduction measured on smooth, noiseless benchmark functions carries over to real materials experiments, which are noisy, expensive, and sometimes involve discrete or categorical variables.","fun_headline_variants_meta":{"raw":{"variants":["Bgolearn: 40–60% fewer experiments to find optimal materials","Open-source toolkit Bgolearn accelerates materials discovery via BO","Unified Bayesian optimization framework Bgolearn slashes materials testing","Bgolearn makes Bayesian optimization practical for materials scientists"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1612,"prompt_tokens":771,"completion_tokens":841,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":768}},"tokens_in":515,"tokens_out":841,"duration_ms":7953,"temperature":1.0,"reasoning_tokens":768,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:18:09.668560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Bgolearn and a random-search baseline on a real materials system—for example, the same medium-Mn steel heat-treatment protocol—starting both from the same 16 initial experiments and proposing three candidates per round, with repeated noisy replicates. If Bgolearn does not reach the same strength–ductility frontier in materially fewer evaluations, or does not beat random search at all, the headline reduction claim is falsified.","supporting_citations":[],"review_version":1}