Privacy-Aware Collaborative and Distributed Bayesian Optimization

Aditya Rane; Akash Deep; Paritosh Ramanan; Sathwik Yamana; Srikanthan Ramesh

REVIEW 2 major objections 5 minor 20 references

Privacy-Aware Collaborative and Distributed Bayesian Optimization

T0 review · 2 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distributed Bayesian optimization can match a centralized meta-learner by sharing only gradients, but those gradients leak client queries and the leak worsens as the search converges.

desk verdict Clean distributed PACOH that matches centralized BO, plus a real BO-specific gradient-leakage finding and a usable DP trade-off. read the letter →

arxiv 2607.11600 v1 pith:SVDJ43JU submitted 2026-07-13 cs.LG stat.ME

Aditya Rane , Sathwik Yamana , Paritosh Ramanan , Srikanthan Ramesh , Akash Deep This is my paper

classification cs.LGstat.ME

keywords CollaborativeOptimizationMeta-LearningDifferentialPrivacyBayesianGradientLeakagePACOHManufacturingProcess

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Independent labs and manufacturing sites often optimize related black-box processes yet cannot pool raw data because of proprietary or regulatory barriers. This paper presents PACD-BO, a collaborative meta-learning scheme that lets each site keep its observations local while still recovering the same shared inductive bias that a centralized PAC-Bayesian meta-learner would compute. The mechanism works because the meta-update decomposes into a sum of per-client score gradients; the coordinator simply aggregates those gradients and runs Stein variational gradient descent. Empirically the method matches centralized performance on several standard benchmarks and beats both isolated Bayesian optimization and consensus-based alternatives. At the same time the paper shows that the shared gradients themselves are invertible: an adapted deep-leakage attack reconstructs a client’s query locations, and reconstruction becomes easier once the optimizer concentrates samples near the optimum. Task-level differential privacy disperses the reconstructions but slows early-stage convergence, so a concrete privacy–utility trade-off is quantified.

What carries the argument

The additive decomposition of the PACOH score: each client computes only its local term Ai,k = ∇ϕk ln Zβ(Si,Pϕk) and ships the K gradient vectors; the coordinator sums them into the exact centralized score and performs the SVGD particle update.

What would settle it

On a suite of tasks whose optima and shapes diverge more sharply than the four benchmarks used here, measure whether PACD-BO’s final normalized regret remains within one standard deviation of centralized PACOH-BO while still beating independent GPs; a clear degradation relative to the independent baseline would falsify the claim that gradient-only collaboration recovers centralized performance under realistic heterogeneity.

Watch

Extended reading notes

Core claim

By exchanging only the per-task score gradients that appear in the PACOH hyper-posterior update, PACD-BO recovers the identical centralized meta-update (up to floating-point round-off) and therefore matches centralized PACOH-BO regret and early AUC on the tested benchmarks, all while never moving raw client observations. The same gradients, however, can be inverted by a multi-start deep-leakage attack whose success increases as Bayesian-optimization queries cluster near the optimum—an “exploitation leakage” phenomenon specific to sequential design. Task-level Gaussian noise after clipping defeats the attack at a measurable cost to early convergence.

Load-bearing premise

The client objectives must share enough latent structure that a single meta-prior is actually helpful; if the tasks are only weakly related, collaboration can fail to improve on isolated Bayesian optimization.

Editorial extensions

If this is right

Sites that cannot legally or commercially share raw experimental data can still obtain essentially centralized meta-learning performance by exchanging only score gradients.
Any coordinator that receives unprotected PACOH-style gradients can reconstruct a client’s query trajectory, and the reconstruction becomes more accurate precisely when the client’s most valuable (near-optimal) points are being queried.
A simple task-level Gaussian mechanism with noise multiplier σ = 8 is already sufficient to scatter reconstructions across the design space, but it measurably delays early-stage regret reduction.
Consensus-based collaborative BO methods that average proposals rather than meta-priors can under-perform even non-collaborative baselines when client optima differ.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same additive-score pattern appears in other PAC-Bayesian and variational meta-learners, so the gradient-only federation idea may transfer beyond Gaussian-process kernels.
Because leakage intensifies with query concentration, adaptive or acquisition-aware noise schedules could protect late-stage privacy with less early-stage utility loss than constant σ.
Manufacturing networks that already run multi-site design-of-experiments campaigns are natural early adopters; the method’s MPI implementation shows the communication pattern is already practical on modest clusters.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a solid, useful paper. The core move is simple and correct: PACOH’s score is already a sum of per-task terms, so they push the SVGD update out to clients, exchange only the local score gradients, and recover the centralized meta-update up to floating-point noise. Table I and Figure 2 show PACD-BO sitting on top of centralized PACOH-BO and clearly ahead of independent GPs and the CBOC consensus baselines on the four benchmarks. That equivalence claim is not hand-wavy; it follows from the algebra they write down.

What is actually new is the privacy half. They adapt DLG to the sequential BO setting and document “exploitation leakage”: as queries concentrate near the optimum, reconstruction gets easier. That is a BO-specific observation, not just a re-run of the usual federated-learning attack. Task-level Gaussian DP (clip + noise) disperses the recovered points; they pick σ=8 from a dispersion plot and show the expected early-regret hit that mostly closes by the end of the budget. The privacy–utility curves are concrete enough to be useful.

Soft spots are real but proportionate. N=5 clients, mostly synthetic/semi-synthetic tasks, free parameters (σ, C, κ, β, λ, K, η) chosen by inspection, and no code or data release. The load-bearing assumption is that the client objectives share enough structure for a single meta-prior to help; if relatedness is weak the collaborative prior can stop helping. None of that breaks the central claims on the experiments they ran.

This is for people doing multi-site experimental design, manufacturing process optimization, or privacy-aware BO. It is not a field-reorganizing result, but it is a clean, honest extension with a genuine privacy finding. I would send it to referees; the math and the tables are good enough to deserve a serious look, and the missing artifacts are fixable. Worth reading and, for anyone working in collaborative BO or gradient leakage under sequential acquisition, worth citing.

Referee Report

2 major / 5 minor

Summary. The paper proposes PACD-BO, a collaborative meta-learning framework that distributes the PACOH hyper-posterior update across clients by exchanging only per-task score gradients A_{i,k} rather than raw observations. Because the PACOH score (Eq. 4) is a linear sum of independent task contributions, the coordinator’s aggregated score G_k recovers the centralized SVGD update (Eq. 7) up to floating-point round-off; clients then run local UCB BO under the shared particle prior. Empirically, PACD-BO matches centralized PACOH-BO normalized regret and early AUC on four benchmarks (sinusoid, Branin, Hartmann, Wing Weight) while outperforming independent GPs and two CBOC consensus baselines (Table I, Fig. 2). The same gradients are shown to be invertible by an adapted multi-start DLG attack whose success worsens as BO queries concentrate near the optimum (“exploitation leakage,” Fig. 3). A task-level Gaussian DP mechanism (clipping + noise) disperses reconstructions at a measurable cost to early-stage convergence (Figs. 4–5).

Significance. If the results hold, the work supplies a clean, algebraically exact route to federated PAC-Bayesian meta-BO that preserves the centralized meta-update without raw-data pooling, together with the first documented gradient-inversion attack surface specific to sequential acquisition-driven BO. The MPI implementation, multi-seed tables, and explicit privacy–utility curves make the contribution reproducible and immediately usable for multi-site manufacturing or materials optimization under proprietary constraints. The relatedness assumption is standard for meta-learning and is already stated; when it holds, the equivalence claim and the empirical match follow directly.

major comments (2)

The relatedness assumption (Section III.A: tasks τ_i ∼ T share latent structure) is load-bearing for the claim that the collaborative prior improves over independent GPs. Table I and Fig. 2 demonstrate the benefit only under the four constructed heterogeneity patterns (parameter shifts of the same base functions). A controlled ablation that systematically weakens relatedness—e.g., by increasing the divergence of client-specific parameters until the meta-prior becomes harmful—would establish the regime of validity and prevent over-generalization to arbitrary multi-site problems.
Section VI selects the operating noise multiplier σ = 8 from reconstruction-dispersion saturation (Fig. 4) but does not report the corresponding (ε, δ) privacy budget under Rényi accounting for the multi-round, multi-particle protocol. Without an explicit privacy-loss calculation (or a statement that only empirical reconstruction resistance is claimed), the “differentially private defense” remains incompletely characterized relative to the DP-SGD literature the authors cite.

minor comments (5)

Figure 1 caption and surrounding text use both script-S and bold-S for local datasets; a single consistent notation would improve readability.
In Section III.C the UCB form is written AUCB(x) = κσ̃(x) − μ̃(x); the conventional sign for minimization is clear from context, but an explicit statement that the objectives are minimized would avoid momentary confusion.
Table I header “Independent.” ends with a period; the other method names do not. Minor polish.
The Wing Weight description (Section IV.A) mentions ten variables of which six are optimized; a short table of the fixed client-specific operating conditions would make the heterogeneity fully transparent.
References [12] and [13] appear as 2025–2026 arXiv/CDC entries; if they remain unpublished at camera-ready, a note on availability would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: algebraic equivalence of distributed SVGD score to centralized PACOH follows from the cited additive decomposition and is verified empirically against an independent centralized baseline.

full rationale

The paper's central performance claim (PACD-BO recovers the centralized meta-update up to floating-point error, matching PACOH-BO regret/AUC while keeping raw data local) rests on the linear sum structure already present in the external PACOH score (Eq. 4: abla ln Q* = abla ln P + (\lambda/(Neta+\lambda)) \sum Ai,k). Distributing the sum via MPI Gather of per-client Ai,k (Eq. 7) is therefore an exact algebraic rewrite, not a self-definitional loop or fitted prediction. Empirical match is shown against a true centralized PACOH-BO baseline (Table I, Fig. 2) on four external benchmarks, plus independent and CBOC controls. The privacy results are attack measurements (adapted DLG on sequential gradients, exploitation-leakage pattern as queries cluster) and a standard Gaussian DP mechanism with measured privacy-utility trade-off (Figs. 3-5); none reduce by construction to their inputs. Citations (PACOH [6], DLG [7], DP-SGD [15]) are to non-overlapping external authors and supply background machinery, not load-bearing uniqueness theorems or ansätze that force the reported numbers. No self-definitional equations, no parameters fitted then re-predicted, and no renaming of known results. The derivation chain is therefore self-contained and non-circular.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The central performance claim rests on the additive PACOH score (imported from prior work) plus the modeling assumption that client tasks share latent structure. The privacy claim rests on an empirical DLG attack success and a hand-chosen noise multiplier. Free parameters that affect both utility and the reported privacy operating point are listed; invented entities are the named framework and the named leakage phenomenon.

free parameters (6)

DP noise multiplier σ = 8
Chosen as operating point σ=8 after inspecting reconstruction-dispersion saturation in Figure 4; directly controls the privacy–utility curves in Figure 5.
gradient clipping norm C
Sets sensitivity Δ=2C for the Gaussian mechanism (Eqs. 12–13); numerical value not reported, yet required for any concrete privacy guarantee.
UCB exploration weight κ
Controls the acquisition function used for every local query; value not specified.
PACOH regularization parameters β, λ
Appear in the hyper-posterior (Eq. 3) and the score (Eq. 4); control the strength of the meta-prior versus local data.
number of SVGD particles K and step size η
Determine the quality of the hyper-posterior approximation that is broadcast each round; not numerically fixed in the text.
DLG multi-start count R = 5
Set to 5 random restarts per attacked iteration; affects reported reconstruction success.

assumptions (5)

domain assumption Client objectives {f_i} are related draws from a common task distribution T that share latent structure under heterogeneous conditions.
Stated in Section III.A; required for the meta-prior to transfer useful inductive bias. Without it the collaborative method reduces to independent BO plus communication overhead.
domain assumption Observations follow y = f(x) + ε with i.i.d. Gaussian noise of known variance σ_n².
Equation (1); standard GP-BO modeling assumption used for the local surrogates and the GMLL gradients.
standard math The PACOH optimal hyper-posterior takes the closed form Q*(P) ∝ P(P) exp(…) and its score decomposes into a hyper-prior term plus a linear sum of per-task gradients A_{i,k}.
Equations (3)–(4), imported from Rothfuss et al.; the entire distributed protocol is built on this additive structure.
domain assumption The coordinator is honest-but-curious: it follows the protocol but may attempt to invert received gradients.
Threat model in Section V.A; defines the adversary against which the DP defense is evaluated.
standard math Clipped per-particle gradients have ℓ₂ sensitivity bounded by Δ=2C, justifying the Gaussian mechanism of Eqs. (12)–(13).
Standard DP-SGD sensitivity argument applied at task/particle level in Section VI.

invented entities (2)

PACD-BO collaborative framework
purpose: Name the MPI-based coordinator–client protocol that distributes the PACOH SVGD update without raw-data exchange.
The protocol is a direct consequence of the additive score; the name packages the system architecture, communication pattern, and local UCB loop.
Exploitation Leakage
purpose: Label the observed phenomenon that gradient inversion becomes easier as BO queries concentrate near the optimum.
Empirical observation from the sequential DLG experiments (Figure 3 top, Section V.B); not an independent physical entity but a named attack-surface property of acquisition-driven BO.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Privacy-Aware Collaborative and Distributed Bayesian Optimization." pith.science (2026). https://pith.science/paper/SVDJ43JU

@misc{pith2026260711600,
  author       = {Pith},
  title        = {Pith review of: Privacy-Aware Collaborative and Distributed Bayesian Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVDJ43JU}},
  note         = {Machine review of arXiv:2607.11600}
}

read the original abstract

We propose a collaborative meta-learning framework for distributed Bayesian optimization matching centralized performance without raw-data exchange. We show gradient sharing leaks client observations, with leakage worsening as the search converges and queries concentrate near the optimum. We evaluate a differentially private defense and characterize its privacy-utility trade-off.

Figures

Figures reproduced from arXiv: 2607.11600 by the authors.

**Figure 2.** Convergence analysis of the proposed PACD-BO [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Spatial gradient leakage and defense. Unprotected gradients (Top) allow reconstruction of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Reconstruction dispersion analysis. Left: Dispersion [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Privacy-utility trade-off. Optimization performance with [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 3 linked inside Pith

[1]

Exploring machine learning for semiconductor process optimization: A systematic review.IEEE Transactions on Artificial Intelligence, 5(12):5969–5989, 2024

Ying-Lin Chen, Sara Sacchi, Bappaditya Dey, Victor Blanco, Sandip Halder, Philippe Leray, and Stefan De Gendt. Exploring machine learning for semiconductor process optimization: A systematic review.IEEE Transactions on Artificial Intelligence, 5(12):5969–5989, 2024

2024
[2]

Machine-learning-assisted materials discovery using failed experiments.Nature, 533(7601):73–76, 2016

Paul Raccuglia, Katherine C Elbert, Philip DF Adler, Casey Falk, Malia B Wenny, Aurelio Mollo, Matthias Zeller, Sorelle A Friedler, Joshua Schrier, and Alexander J Norquist. Machine-learning-assisted materials discovery using failed experiments.Nature, 533(7601):73–76, 2016

2016
[3]

From materials to management: The expanding role of design of experiments in advanced battery technologies.Energy Storage Materials, page 104890, 2026

Diego Pugliese, Roberto Staffieri, and Federico Bella. From materials to management: The expanding role of design of experiments in advanced battery technologies.Energy Storage Materials, page 104890, 2026

2026
[4]

Federated machine learning: Concept and applications.ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications.ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019

2019
[5]

Peter I. Frazier. A tutorial on bayesian optimization.arXiv preprint arXiv:1807.02811, 2018

arXiv 2018
[6]

Scalable pac-bayesian meta-learning via the pac-optimal hyper-posterior: From theory to practice.Journal of Machine Learning Research, 24(386):1–62, 2023

Jonas Rothfuss, Martin Josifoski, Vincent Fortuin, and Andreas Krause. Scalable pac-bayesian meta-learning via the pac-optimal hyper-posterior: From theory to practice.Journal of Machine Learning Research, 24(386):1–62, 2023

2023
[7]

Deep leakage from gradients

Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019

2019
[8]

Taking the human out of the loop: A review of bayesian optimization.Proceedings of the IEEE, 104(1):148–175, 2015

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization.Proceedings of the IEEE, 104(1):148–175, 2015

2015

Show all 20 references

[9]

Meta-learning reliable priors in the function space.Advances in Neural Information Processing Systems, 34:280–293, 2021

Jonas Rothfuss, Dominique Heyn, Andreas Krause, et al. Meta-learning reliable priors in the function space.Advances in Neural Information Processing Systems, 34:280–293, 2021

2021
[10]

Communication-efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. Pmlr, 2017

2017
[11]

Collaborative and distributed bayesian optimization via consensus.IEEE Transactions on Automation Science and Engineering, 22:11343–11355, 2025

Xubo Yue, Yang Liu, Albert S Berahas, Blake N Johnson, and Raed Al Kontar. Collaborative and distributed bayesian optimization via consensus.IEEE Transactions on Automation Science and Engineering, 22:11343–11355, 2025

2025
[12]

Collaborative contextual bayesian optimization.arXiv preprint arXiv:2604.18912, 2026

Chih-Yu Chang, Qiyuan Chen, Tianhan Gao, David Fenning, Chinedum Okwudire, Neil Dasgupta, Wei Lu, and Raed Al Kontar. Collaborative contextual bayesian optimization.arXiv preprint arXiv:2604.18912, 2026

2026 arXiv
[13]

Collaborative bayesian optimization via wasserstein barycenters

Donglin Zhan, Haoting Zhang, Rhonda Righter, Zeyu Zheng, and James Anderson. Collaborative bayesian optimization via wasserstein barycenters. In2025 IEEE 64th Conference on Decision and Control (CDC), pages 6284–6291. IEEE, 2025

2025
[14]

Differential privacy: Gradient leakage attacks in federated learning environments.arXiv preprint arXiv:2510.23931, 2025

Miguel Fernandez-de Retana, Unai Zulaika, Rubén Sánchez-Corcuera, and Aitor Almeida. Differential privacy: Gradient leakage attacks in federated learning environments.arXiv preprint arXiv:2510.23931, 2025

2025
[15]

Deep learning with differential privacy

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016

2016
[16]

Differentially private meta-learning.arXiv preprint arXiv:1909.05830, 2019

Jeffrey Li, Mikhail Khodak, Sebastian Caldas, and Ameet Talwalkar. Differentially private meta-learning.arXiv preprint arXiv:1909.05830, 2019

1909 arXiv
[17]

Gaussian processes in machine learning

Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pages 63–71. Springer, 2003

2003
[18]

mpi4py.futures: Mpi-based asynchronous task execution for python.IEEE Transactions on Parallel and Distributed Systems, 34(2):611–622, 2023

Marcin Rogowski, Samar Aseeri, David Keyes, and Lisandro Dalcin. mpi4py.futures: Mpi-based asynchronous task execution for python.IEEE Transactions on Parallel and Distributed Systems, 34(2):611–622, 2023

2023
[19]

Virtual library of simulation experiments: Test functions and datasets

Sonja Surjanovic and Derek Bingham. Virtual library of simulation experiments: Test functions and datasets. https://www.sfu.ca/~ssurjano/
[20]

Are we forgetting about compositional optimisers in bayesian optimisation?Journal of Machine Learning Research, 22(160):1–78, 2021

Antoine Grosnit, Alexander I Cowen-Rivers, Rasul Tutunov, Ryan-Rhys Griffiths, Jun Wang, and Haitham Bou-Ammar. Are we forgetting about compositional optimisers in bayesian optimisation?Journal of Machine Learning Research, 22(160):1–78, 2021. 6

2021

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.

[1] [1]

Exploring machine learning for semiconductor process optimization: A systematic review.IEEE Transactions on Artificial Intelligence, 5(12):5969–5989, 2024

Ying-Lin Chen, Sara Sacchi, Bappaditya Dey, Victor Blanco, Sandip Halder, Philippe Leray, and Stefan De Gendt. Exploring machine learning for semiconductor process optimization: A systematic review.IEEE Transactions on Artificial Intelligence, 5(12):5969–5989, 2024

2024

[2] [2]

Machine-learning-assisted materials discovery using failed experiments.Nature, 533(7601):73–76, 2016

Paul Raccuglia, Katherine C Elbert, Philip DF Adler, Casey Falk, Malia B Wenny, Aurelio Mollo, Matthias Zeller, Sorelle A Friedler, Joshua Schrier, and Alexander J Norquist. Machine-learning-assisted materials discovery using failed experiments.Nature, 533(7601):73–76, 2016

2016

[3] [3]

From materials to management: The expanding role of design of experiments in advanced battery technologies.Energy Storage Materials, page 104890, 2026

Diego Pugliese, Roberto Staffieri, and Federico Bella. From materials to management: The expanding role of design of experiments in advanced battery technologies.Energy Storage Materials, page 104890, 2026

2026

[4] [4]

Federated machine learning: Concept and applications.ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications.ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019

2019

[5] [5]

Peter I. Frazier. A tutorial on bayesian optimization.arXiv preprint arXiv:1807.02811, 2018

arXiv 2018

[6] [6]

Scalable pac-bayesian meta-learning via the pac-optimal hyper-posterior: From theory to practice.Journal of Machine Learning Research, 24(386):1–62, 2023

Jonas Rothfuss, Martin Josifoski, Vincent Fortuin, and Andreas Krause. Scalable pac-bayesian meta-learning via the pac-optimal hyper-posterior: From theory to practice.Journal of Machine Learning Research, 24(386):1–62, 2023

2023

[7] [7]

Deep leakage from gradients

Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019

2019

[8] [8]

Taking the human out of the loop: A review of bayesian optimization.Proceedings of the IEEE, 104(1):148–175, 2015

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization.Proceedings of the IEEE, 104(1):148–175, 2015

2015