REVIEW 4 major objections 6 minor 31 references
Asynchronous Batch Bayesian Optimization with Pipelining Evaluations for Experimental Resource$\unicode{x2013}$constrained Conditions
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read PipeBO splits experiments into K pipeline stages and reaches the same solution quality in roughly 56% of the steps at K=2, 50% at K=3, and 38% at K=5.
desk verdict Solid, clearly-scoped extension of async BO with a genuine mid-experiment update; the step-count speedups hold under the stated equal-stage-time model, but the wall-clock claim for real labs needs robustness work. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a $K$-stage pipeline of experiments fused with acquisition-function recalculation. Each experiment is split into $K$ processes; process parameters are fixed only when a process starts, leaving later parameters free. At time $t$, the Gaussian-process acquisition function $\alpha_t$ is built from all completed results, and for each experiment still in progress the free later-stage parameters are set to maximize $\alpha_t(x)$ times a product of local penalizers $\varphi(x; x_{\mathrm{LP}})$, which damp the acquisition function near parameter sets already running. The next experimental set is then chosen by batch Bayesian optimization under the same penalizer. This turns asynchronous results into mid-experiment parameter updates, the paper's main departure from earlier asynchronous batch methods.
What would settle it
Run PipeBO on a $K=2$ task where the first process takes one time unit and the second takes two, and compare wall-clock time to reach a fixed simple regret against vanilla Bayesian optimization; if the wall-clock saving is far smaller than the step-count saving of about 56%, the equal-time assumption is carrying the result.
Extended reading notes
Core claim
The central claim is that pipelining evaluations makes Bayesian optimization practical when parallel equipment is scarce. An experiment is divided into $K$ processes with parameter sets $s_1,\dots,s_K$; with $P=1$, $K$ experiments can be in flight at staggered stages, each one process behind the previous. At each step the acquisition function is rebuilt from all results available at that time, and the undetermined process parameters of experiments already running are re-optimized against this updated acquisition function while earlier parameters stay fixed; a local penalizer keeps running experiments from duplicating each other's parameters. In the benchmark, this reaches the same median simple regret that vanilla Bayesian optimization reaches in 100 steps in about 56% of steps at $K=2$, 50% at $K=3$, and 38% at $K=5$. The paper interprets the gap from the ideal $100/K\%$ as the cost of PipeBO seeing results $K$ steps later than vanilla BO.
Load-bearing premise
The load-bearing premise is that every process in an experiment takes exactly one unit of time, and that this duration does not depend on the parameter values being tested.
Editorial extensions
If this is right
- At $K=2$, PipeBO reaches a fixed simple regret in a median of about 56% of the steps vanilla BO needs; at $K=3$ about 50%, at $K=5$ about 38%, across the 24 benchmark functions.
- Laboratories with scarce equipment can run $K$ times as many experiments through pipelining without adding machines, as long as each experiment can be split into equal-length stages.
- The mid-experiment parameter update gives the largest benefit when later processes contain many tunable parameters, so the gain depends on how optimization parameters are distributed across stages.
- Because PipeBO proposes each experimental set from results that are $K$ steps older than vanilla BO's, its speedup is below the ideal $100/K\%$, and the gain relative to the number of stages shrinks as $K$ grows.
Reading between the lines
- My inference: the paper fixes $P=1$, so the natural untested regime is $P>1$, where several pipelined experimental sets run concurrently; nobody has measured whether the speedup compounds or saturates.
- My inference: the equal-time assumption is the real boundary of the result; with uneven process durations the pipeline stalls at the slowest stage, and the step-count speedup will not convert into wall-clock speedup without scheduling.
- My inference: the mid-experiment update rule could transfer to any asynchronous schedule that returns results while an experiment is running, not only fixed $K$-stage pipelines, since the underlying mechanism is the same acquisition-function recalculation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes PipeBO, a Bayesian optimization method for experimental settings in which a single experiment is divided into K sequential processes and only a small number P of experiments can be run in parallel. The method runs experiments in a K-stage pipeline and, whenever a new result becomes available, recalculates a GP-UCB acquisition function and uses it to update the process parameters of experiments that are still running, while a local penalizer discourages proposing parameters close to those in progress. The authors evaluate PipeBO on 24 noiseless BBOB functions with 50 runs, comparing it with vanilla GP-UCB and with a no-update version of the same pipeline. Their headline result is that PipeBO reaches the same simple regret as vanilla BO reaches in 100 steps in about 56% (K=2), 50% (K=3), and 38% (K=5) of the steps, with source code and data publicly available.
Significance. If the result holds, the paper makes a useful practical contribution for experimental sciences in which multi-stage protocols and scarce equipment make neither purely sequential BO nor batch BO attractive. The empirical study is reasonably broad (24 BBOB functions, 50 runs per configuration), and the public code and data support reproducibility. The paper is also transparent about the equal-process-duration idealization in Section 2.1 and about the existence of functions where the speedup is not achieved. However, the central quantitative claim is expressed in wall-clock terms while being measured only in an idealized step-count model, and that model is not stress-tested against variable or parameter-dependent process durations. The practical significance of the 56/50/38% figures is therefore not yet fully established.
major comments (4)
- [Section 2.1 and Discussion] The central speedup claim is stated in wall-clock terms ('processing time'), but the only quantitative measure is the number of steps under the Section 2.1 assumption that every process takes exactly one unit of time and that duration is independent of the process parameters. The Discussion acknowledges that PipeBO 'may not be suitable' when process times vary considerably and suggests scheduling algorithms, but it does not quantify how the 56/50/38% reductions degrade when durations are unequal or stochastic. Since the motivating examples (e.g., the 5-day agar and 5-day flask culture in the Discussion) naturally involve unequal, parameter-dependent durations, the manuscript should include a sensitivity analysis with simulated variable process durations, or explicitly restrict the claim to the equal-duration idealization.
- [Supplementary Table 1 and Algorithm 1] Algorithm 1 is written for general P (the number of parallel experiments), and Section 2.1 defines experimental sets B_n of size P, but every numerical configuration in Supplementary Table 1 has P=1. The behavior of the method for P>1, including the interaction of the local penalizer across multiple in-flight experiments and the scheduling of parameter updates, is therefore never validated. Either add experiments with P>1 or state clearly that the empirical claims are limited to P=1.
- [Table 1 and Section 3.1] The aggregate reductions (56.2%, 49.5%, 37.6%) are point estimates of the median over 50 runs, reported without confidence intervals or significance tests, and the interquartile ranges in Table 1 are often large (e.g., 116, 139, 159). Moreover, the speedup is not universal: at K=2, F1 requires 126 steps and the reference regret is not reached; at K=3, F4 and F12 require 101 and 103 steps; and at K=5, F19 requires 61 steps, more than at K=2 or K=3. The text should report interval estimates and discuss these exceptions explicitly rather than presenting only the aggregate ratios.
- [Figure 4 and Section 3.2] The comparison between PipeBO and the no-update version is summarized by the percentage of steps in which the median simple regret of PipeBO is lower. This statistic depends on the number of plotted steps and does not directly quantify the contribution of parameter updates to the headline wall-clock reduction; no significance tests or effect sizes are given. In addition, for D=(8,1,1) the median is around 50%, so the update mechanism appears not to help in that configuration; the text should state more precisely the conditions under which updates are beneficial.
minor comments (6)
- [Abstract and Table 1] The wording 'reduced the average processing time ... to about 56% ... for 20 out of the 24 functions' is ambiguous and does not follow directly from Table 1, which shows 22, 21, and 23 functions requiring fewer than 100 steps for K=2, 3, and 5, respectively; please clarify the intended statement.
- [Table 1] The header 'Avarage' is a typo; the F5 row is all dashes; and the note says the median was calculated for 23 functions while the table has 24 rows. Please explain how F5 was excluded and why F1 at K=2 cannot reach the reference regret.
- [Figure 4] The figure caption says 'Each series indicates the number of parameters being optimized by each process,' but the series are not explicitly labeled with their D vectors; add a legend or mapping so the reader can identify which series corresponds to each D setting from Supplementary Table 1.
- [Section 2.3 and Table 1 note] The text says 50 runs were performed, but the Table 1 note says the calculations were based on data from 200 searches; clarify how the 200 searches relate to the 50 runs.
- [Equation (2)] The symbols \hat{L} and \hat{M} in the local penalizer are described only in words; provide their precise definitions (as in the cited local-penalization paper) for reproducibility.
- [Section 3.1] The BBOB benchmark is noiseless, whereas the motivating biological and chemical applications involve noisy experimental readouts; a sentence justifying the noiseless setting or a small noisy experiment would strengthen the practical relevance of the claims.
Circularity Check
No circular derivation: PipeBO's speedup is measured empirically against vanilla BO and a no-update ablation on external BBOB benchmarks.
full rationale
The paper's central claim is an empirical comparison: PipeBO reduces the number of optimization steps needed to reach a fixed simple regret relative to vanilla Bayesian optimization, measured on 24 BBOB functions over 50 runs. The step-count model in Section 2.1 explicitly assumes each process takes one unit of time and that all process times are equal; this is an idealization, but it is applied symmetrically to PipeBO and the vanilla baseline, so the comparison is not circular. The parameter-update ablation is compared against PipeBO without updates, which is identified with the external PLAyBOOK-L algorithm [24], not with an input of the paper. The GP-UCB acquisition function uses a fixed kappa=2, not a fitted parameter, and the varied D configurations in Supplementary Table 1 are experimental design choices. The Discussion states the limitation that PipeBO 'may not be suitable in experiments where the time required for each process varies considerably'; this is a limitation on external validity, not a circular step. No load-bearing self-citation, uniqueness theorem, or definitional identification between input and output was found.
Assumptions & free parameters
free parameters (1)
- GP-UCB exploration coefficient kappa =
2
assumptions (5)
- domain assumption The objective function is a black box with no gradient and expensive evaluations.
- domain assumption Each experiment is divided into K sequential processes, each taking one unit of time independent of parameter values.
- domain assumption Parameters for later processes can be set or changed after an experiment has started.
- domain assumption A Gaussian process with GP-UCB (kappa=2) is an adequate surrogate for the benchmark functions.
- domain assumption BBOB functions are representative of real experimental optimization landscapes.
Cite this review
Pith. "Pith review of Asynchronous Batch Bayesian Optimization with Pipelining Evaluations for Experimental Resource$\unicode{x2013}$constrained Conditions." pith.science (2026). https://pith.science/paper/WV76NGPY
@misc{pith2026241204392,
author = {Pith},
title = {Pith review of: Asynchronous Batch Bayesian Optimization with Pipelining Evaluations for Experimental Resource$\unicodex2013$constrained Conditions},
year = {2026},
howpublished = {\url{https://pith.science/paper/WV76NGPY}},
note = {Machine review of arXiv:2412.04392}
}
read the original abstract
Bayesian optimization is efficient even with a small amount of data and is used in engineering and in science, including biology and chemistry. In Bayesian optimization, a parameterized model with an uncertainty is fitted to explain the experimental data, and then the model suggests parameters that would most likely improve the results. Batch Bayesian optimization reduces the processing time of optimization by parallelizing experiments. However, batch Bayesian optimization cannot be applied if the number of parallelized experiments is limited by the cost or scarcity of equipment; in such cases, sequential methods require an unrealistic amount of time. In this study, we developed pipelining Bayesian optimization (PipeBO) to reduce the processing time of optimization even with a limited number of parallel experiments. PipeBO was inspired by the pipelining of central processing unit architecture, which divides computational tasks into multiple processes. PipeBO was designed to achieve experiment parallelization by overlapping various processes of the experiments. PipeBO uses the results of completed experiments to update the parameters of running parallelized experiments. Using the Black-Box Optimization Benchmarking, which consists of 24 benchmark functions, we compared PipeBO with the sequential Bayesian optimization methods. PipeBO reduced the average processing time of optimization to about 56% for the experiments that consisted of two processes or even less for those with more processes for 20 out of the 24 functions. Overall, PipeBO parallelizes Bayesian optimization in the resource-constrained settings so that efficient optimization can be achieved.
Reference graph
Works this paper leans on
-
[1]
CauchyA(1847).Méthodegénéralepourlarésolutiondessystèmesd’équationssimultanées. ComptesRendus Hebd. Séances Acad. Sci., 25 : 536–538
- [2]
-
[3]
A review on genetic algorithm: past, present, and future
Katoch S, Chauhan SS, and Kumar V (2021). A review on genetic algorithm: past, present, and future. Multimed. Tools and Applica., 80 (5) : 8091–8126,doi: https://doi.org/10.1007/s11042-020-10139-6
-
[4]
Recent Advances in Bayesian Optimization.ACM Comput
Wang X, Jin Y, Schmitt S, and Olhofer M (2023). Recent Advances in Bayesian Optimization.ACM Comput. Surv., 55 (13s),doi: https://doi.org/10.1145/3582078
doi:10.1145/3582078 2023
-
[5]
An improved PSO-based charging strategy of electric vehicles in electrical distribution grid.Appl
Yang J, He L, and Fu S (2014). An improved PSO-based charging strategy of electric vehicles in electrical distribution grid.Appl. Energy, 128 : 82–92,doi: https://doi.org/10.1016/j.apenergy.2014.04.047
-
[6]
Multi-objective optimal design of hybrid renewable energy systems usingPSO-simulationbasedapproach
Sharafi M and ELMekkawy TY (2014). Multi-objective optimal design of hybrid renewable energy systems usingPSO-simulationbasedapproach. Renew.Energy,68:67–79, doi: https://doi.org/10.1016/j.renene. 2014.01.011. 9
doi:10.1016/j.renene 2014
-
[7]
Khansary MA and Sani AH (2014). Using genetic algorithm (GA) and particle swarm optimization (PSO) methodsfordeterminationofinteractionparametersinmulticomponentsystemsofliquid–liquidequilibria. Fluid Phase Equilib., 365 : 141–145,doi: https://doi.org/10.1016/j.fluid.2014.01.016
-
[8]
Bezerra MA, Santelli RE, Oliveira EP, Villar LS, and Escaleira LA (2008). Response surface methodology (RSM) as a tool for optimization in analytical chemistry.Talanta, 76 (5) : 965–977,doi: https://doi.org/10. 1016/j.talanta.2008.05.019
work page 2008
Show all 31 references
-
[9]
Application of response surface methodology (RSM) to optimize coagulation-flocculation treatment of leachate using poly-aluminum chloride (PAC) and alum.J
Ghafari S, Aziz HA, Isa MH, and Zinatizadeh AA (2009). Application of response surface methodology (RSM) to optimize coagulation-flocculation treatment of leachate using poly-aluminum chloride (PAC) and alum.J. Hazard. Mater., 163 (2) : 650–656,doi: https://doi.org/10.1016/j.j...
2009 doi
-
[10]
Application of RSM for optimization of glutamic acid productionby Corynebacteriumglutamicum inbathculture
Fahimitabar A, Razavian SMH, and Rezaei SA (2021). Application of RSM for optimization of glutamic acid productionby Corynebacteriumglutamicum inbathculture. Heliyon,7(6):e07359, doi: https://doi.org/10. 1016/j.heliyon.2021.e07359
2021
-
[11]
On the Experimental Attainment of Optimum Conditions.J
Box GEP and Wilson KB (1951). On the Experimental Attainment of Optimum Conditions.J. R. Stat. Soc. Ser. B, 13 (1) : 1–38,doi: https://doi.org/10.1111/j.2517-6161.1951.tb00067.x
1951
-
[12]
Some New Three Level Designs for the Study of Quantitative Variables
Box GEP and Behnken DW (1960). Some New Three Level Designs for the Study of Quantitative Variables. Technometrics, 2 (4) : 455–475,doi: https://doi.org/10.2307/1266454
1960 doi
-
[13]
A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise.J
Kushner HJ (1964). A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise.J. Basic Eng., 86 (1) : 97–106,doi: https://doi.org/10.1115/1.3653121
1964 doi
-
[14]
Bayesian Optimization for Adaptive Experimental Design: A Review.IEEE Access, 8 : 13937–13948,doi: https://doi.org/10.1109/ACCESS
Greenhill S, Rana S, Gupta S, Vellanki P, and Venkatesh S (2020). Bayesian Optimization for Adaptive Experimental Design: A Review.IEEE Access, 8 : 13937–13948,doi: https://doi.org/10.1109/ACCESS. 2020.2966228
2020
-
[15]
A survey of adaptive sampling for global metamodeling in support of simulation-basedcomplexengineeringdesign
Liu H, Ong YS, and Cai J (2018). A survey of adaptive sampling for global metamodeling in support of simulation-basedcomplexengineeringdesign. Struct.Multidiscip.Optim. ,57(1):393–416, doi: https://doi. org/10.1007/s00158-017-1739-8
2018 doi
-
[16]
A Customized Bayesian Algorithm to Optimize Enzyme-Catalyzed Reactions.ACS Sustain
Tachibana R, Zhang K, Zou Z, Burgener S, and Ward TR (2023). A Customized Bayesian Algorithm to Optimize Enzyme-Catalyzed Reactions.ACS Sustain. Chem. Eng., 11 (33) : 12336–12344,doi: https://doi. org/10.1021/acssuschemeng.3c02402
2023 doi
-
[17]
Bayesian cell therapy process optimization.Biotechnol
Claes E, Heck T, Coddens K, Sonnaert M, Schrooten J, and Verwaeren J (2024). Bayesian cell therapy process optimization.Biotechnol. Bioeng., 121 (5) : 1569–1582,doi: https://doi.org/10.1002/bit.28669
2024 doi
-
[18]
Ginsbourger D, Le Riche R, and Carraro L (2010). Kriging Is Well-Suited to Parallelize Optimization.Com- putational Intelligence in Expensive Optimization Problems, 131–162, Springer Berlin Heidelberg,doi: https: //doi.org/10.1007/978-3-642-10701-6_6. 10
2010 doi
-
[19]
Batch Bayesian Optimization via Local Penalization
González J, Dai Z, Hennig P, and Lawrence N (2016). Batch Bayesian Optimization via Local Penalization. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, 51 : 648–657, PMLR
2016
-
[20]
The Parallel Knowledge Gradient Method for Batch Bayesian Optimization
Wu J and Frazier P (2016). The Parallel Knowledge Gradient Method for Batch Bayesian Optimization. Advances in Neural Information Processing Systems, 29 : 3126–3134, Curran Associates, Inc
2016
-
[21]
Sequential statistical optimization of lactose–based medium and process variables for inulinase production fromPenicillium oxalicumBGPUP-4
Singh RS and Chauhan K (2018). Sequential statistical optimization of lactose–based medium and process variables for inulinase production fromPenicillium oxalicumBGPUP-4. 3 Biotech, 8 (38),doi: https://doi. org/10.1007/s13205-017-1060-7
2018 doi
-
[22]
Pipeline Architecture.ACM Comput
Ramamoorthy CV and Li HF (1977). Pipeline Architecture.ACM Comput. Surv., 9 (1) : 61–102,doi: https: //doi.org/10.1145/356683.356687
1977
-
[23]
Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions
Hansen N, Finck S, Ros R, and Auger A (2009). Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. [Research Report] RR-6829, INRIA. inria-00362633v2, 16p
2009
-
[24]
Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation.Proceedings of the 36th International Conference on Machine Learning, 97 : 253–262, PMLR
Alvi AS, Ru B, Calliess JP, Roberts SJ, and Osborne MA (2019). Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation.Proceedings of the 36th International Conference on Machine Learning, 97 : 253–262, PMLR
2019
-
[25]
Methods and Softw., 36 : 114–144,doi: https://doi
HansenN,AugerA,RosR,MersmannO,TušarT,andBrockhoffD(2021).COCO:APlatformforComparing Continuous Optimizers in a Black-Box Setting.Optim. Methods and Softw., 36 : 114–144,doi: https://doi. org/10.1080/10556788.2020.1808977
2021
-
[26]
Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator.ACM Trans
Matsumoto M and Nishimura T (1998). Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator.ACM Trans. Model. Comput. Simul., 8 (1) : 3–30,doi: https://doi.org/ 10.1145/272991.272995
1998
-
[27]
Srinivas N, Krause A, Kakade S, and Seeger M (2010). Gaussian process optimization in the bandit setting: no regret and experimental design.Proceedings of the 27th International Conference on International Conference on Machine Learning, 1015–1022, Omnipress
2010
-
[28]
GPyOpt: A Bayesian Optimization framework in python
The GPyOpt authors (2016). GPyOpt: A Bayesian Optimization framework in python. http : / / github . com / SheffieldML/GPyOpt
2016
-
[29]
Parallelised Bayesian Optimisation via Thompson Sampling.Proceedings of the 21st International Conference on Artificial Intelligence and Statistics, 84 : 133–142, PMLR
Kandasamy K, Krishnamurthy A, Schneider J, and Poczos B (2018). Parallelised Bayesian Optimisation via Thompson Sampling.Proceedings of the 21st International Conference on Artificial Intelligence and Statistics, 84 : 133–142, PMLR
2018
-
[30]
Optimal Scheduling for Laboratory Automation of Life Science Experiments with Time Constraints.SLAS Technol., 26 (6) : 650–659,doi: https: //doi.org/10.1177/24726303211021790
Itoh TD, Horinouchi T, Uchida H, Takahashi K, and Ozaki H (2021). Optimal Scheduling for Laboratory Automation of Life Science Experiments with Time Constraints.SLAS Technol., 26 (6) : 650–659,doi: https: //doi.org/10.1177/24726303211021790. 11
2021 doi
-
[31]
SAGAS: Simulated annealing and greedy algorithm scheduler for laboratory automation.SLAS Technol., 28 (4) : 264–277,doi: https://doi
Arai Y, Takahashi K, Horinouchi T, Takahashi K, and Ozaki H (2023). SAGAS: Simulated annealing and greedy algorithm scheduler for laboratory automation.SLAS Technol., 28 (4) : 264–277,doi: https://doi. org/10.1016/j.slast.2023.03.001. 12 paper figure Facilities for process 1 1...
2023 doi
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.