Gearing Gaussian process modeling and sequential design towards stochastic simulators

Abby Stevens (ANL); Arindam Fadikar (ANL); Mickael Binois (Acumes)

REVIEW 2 major objections 7 minor 12 references

Gearing Gaussian process modeling and sequential design towards stochastic simulators

T0 review · 2 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gaussian process surrogates extend to stochastic simulators by modeling input-dependent noise and adapting sequential design criteria to replication.

desk verdict A competent, useful survey of noisy-GP modeling and sequential design; the only real weakness is the thin illustrative example. read the letter →

arxiv 2412.07306 v1 pith:PKNWSBJK submitted 2024-12-10 math.OC stat.ML

Mickael Binois (ACUMES) , Arindam Fadikar (ANL) , Abby Stevens (ANL) This is my paper · ORCID

classification math.OCstat.ML MSC 62-0262K0562M30

keywords GaussianprocesssurrogatesstochasticsimulatorsheteroscedasticnoisereplicationsequentialdesignquantileprocessesBayesianoptimizationkriging

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This chapter argues that Gaussian process surrogates remain practical and analytically tractable for stochastic simulators when the noise is treated as input-dependent rather than constant. It maps the main modeling options, from heteroscedastic GPs with a latent log-variance process to stochastic kriging, quantile GPs, and deep GPs, and shows what each requires in terms of replication and inference. It then shows how sequential design criteria such as IMSPE, SUR, EI, and UCB can be adapted to decide where to sample and how often to replicate. The reason this matters is that stochastic simulators are common in epidemiology, operations research, and hyperparameter tuning, and deciding whether to replicate or explore is a real budget question.

What carries the argument

The central object is the heteroscedastic Gaussian process with predictive equations $m_N(\mathbf{x}) = \mathbf{c}(\mathbf{x})^\top (C_N + \Lambda_N)^{-1} \mathbf{y}$ and $s_N^2(\mathbf{x}) = r(\mathbf{x}) + \sigma^2 ( c(\mathbf{x},\mathbf{x}) - \mathbf{c}(\mathbf{x})^\top (C_N + \Lambda_N)^{-1} \mathbf{c}(\mathbf{x}) )$, where $r(\mathbf{x})$ is the input-dependent noise variance. The key structural identity is that when replication is available, these equations reduce to a form depending only on the $n$ unique designs: $m_n(\mathbf{x}) = \mathbf{c}(\mathbf{x})^\top(C_n + \Lambda_n A_n^{-1})^{-1} \bar{\mathbf{y}}$, with $A_n$ the diagonal replication counts, so computational cost drops from $N$ to $n$. Around this core, the chapter organizes two inference families: latent log-variance GPs to learn $r(\mathbf{x})$ when replicates are absent, and stochastic kriging to estimate $r(\mathbf{x})$ from empirical variances when replicates are present. For sequential design, the central mechanism is the adaptation of criteria such as IMSPE and SUR to account for $r(\mathbf{x})$ and replication.

What would settle it

Run a systematic benchmark on several stochastic simulators with different signal-to-noise ratios, replication costs, and non-Gaussian error structures, and compare quantitative mean-squared error and budget-accuracy curves; the chapter's visual illustrations alone do not settle whether the claimed ranking of methods holds.

Watch

Extended reading notes

Core claim

On its own terms, the chapter claims that the standard GP surrogate, including its closed-form predictive equations, remains the right backbone for stochastic simulators once the constant-noise assumption is relaxed to a variance function $r(\mathbf{x})$ that changes with the input. It reviews three main routes to this relaxation: known noise from Monte Carlo error or tunable fidelity, latent log-variance GPs estimated by MCMC, EM, variational, or maximum-likelihood schemes, and replication-based empirical variance estimates in stochastic kriging. It further claims that non-Gaussian noise can be handled either by heavy-tailed likelihoods and approximate inference, by modeling conditional quantiles directly with a GP, or by deep (warped) GPs that add flexibility without explicit noise control. For sequential design, it argues that criteria built for deterministic surrogates can be adapted to the noisy setting, with replication itself becoming a design choice.

Load-bearing premise

The chapter's qualitative guidance is illustrated with a single one-dimensional SIR simulator; if that example is atypical of stochastic simulators, the suggested ordering of methods may not transfer.

Editorial extensions

If this is right

With replicated observations, heteroscedastic GP inference can be carried out at a cost driven by the number of unique input locations rather than total simulator calls, enabling large replication budgets.
Because IMSPE depends only on predictive variance, sequential design can look ahead and decide between replicating an existing location and exploring a new one without needing observed outputs.
Quantile GPs and latent-variable models extend noisy surrogate modeling beyond Gaussian error, but require approximate inference and more data.
Plugging predictive means or quantile improvements into EI gives workable Bayesian optimization of the mean response under low signal-to-noise ratios.
For level-set estimation, contour SUR criteria can be adapted to noisy simulators and will place replicates on both sides of the crossing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural codification of the chapter's trade-offs is a selection rule: latent-variable heteroscedastic GPs when replicates per site are scarce, stochastic kriging when replicates are abundant, and quantile GPs when tails or quantiles are the target; the chapter itself stops short of stating such a rule.
The replication-versus-exploration framing suggests that look-ahead IMSPE can be benchmarked against optimal computing budget allocation on discrete design sets, since both answer the same question of where to spend the next simulator calls.
The chapter's arguments imply that model choice should depend not only on data availability but on the smoothness of the noise variance function, a quantity that could be estimated from replicated data and used to forecast when heteroscedastic models will pay off.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a survey/tutorial chapter, not a research contribution, and it is a solid one. The equations and attributions check out, and the organization is genuinely useful. The main weakness is the illustrative example—a single 1D SIR simulator, visual comparisons only, no error bars or repeated-seed variability—so the practical recommendations are directionally sensible but not empirically established.

The chapter does a good job consolidating the literature on GP surrogates for stochastic simulators: heteroscedastic GPs, stochastic kriging, quantile/expectile models, deep GPs, and the corresponding sequential design criteria (IMSPE, SUR, EI/UCB variants). I found the discussion of replication versus latent-variable approaches helpful, and the trade-offs in Section 5 are honest. It also points to the original sources carefully, including the authors' own work where appropriate; the citations are not self-serving.

The soft spots are where the stress-test note lands. Section 2.3 fits homoscedastic and heteroscedastic models to one dataset and concludes visually that the heteroscedastic model 'better represents the black-box.' Section 3.2 says the deep GP predictions are 'not significantly different' between datasets without any statistical test. Section 4.4 reports that IMSPE and contour-SUR 'both succeed' without quantitative metrics. That language implies more evidentiary weight than a single illustrative run can carry. It would be straightforward to add repeated-seed variability, quantitative comparisons (e.g., coverage, RMSE, budget used), or at least hedge the claims. The absence of those doesn't break the survey, because the existence of these methods does not depend on the example, but it does weaken the authors' practical orientation.

The paper is best read as an entry point for practitioners and graduate students who want a concise map of the noisy-GP landscape. It is not a research paper and should not be judged as one. I would accept it for peer review as a review article, with the recommendation that the empirical illustration be strengthened or explicitly labeled as illustrative only.

Referee Report

2 major / 7 minor

Summary. This chapter surveys Gaussian process (GP) surrogates for stochastic simulators with complex, input-dependent noise. It reviews homoscedastic and heteroscedastic GP regression, stochastic kriging with replication, quantile GPs, deep GPs, and adaptations of sequential design criteria (IMSPE, SUR, EI/UCB variants, replication allocation) to noisy settings. A running 1D SIR example with two data sets, one with replication and one without, illustrates the models and design strategies. The paper's central assertion is that each of these modeling and design frameworks has been developed in the literature and can be deployed depending on data availability and replication structure.

Significance. If taken as a survey, the chapter fills a useful niche by organizing a scattered literature around the replication/no-replication distinction and by presenting predictive equations in a unified notation. Its strengths are breadth, accurate attributions (I found no mis-attributed equations), and practical pointers to implementations such as hetGP, deepgp, and GPyTorch. The chapter does not introduce new methodology, and its original comparative evidence is anecdotal, relying on visual inspection of a single simulator without quantitative metrics or repeated runs. Thus the paper is a competent review whose practical recommendations would need more evidence to be considered established.

major comments (2)

[§2.3, §3.2, §4.4] The original illustrative experiments are the only place where the paper moves beyond cataloguing existing methods, yet all performance claims are made from visual inspection of a single 1D SIR example. In §2.3 the conclusion that the heteroscedastic model "better represents the black-box at hand" is not supported by any quantitative metric (e.g., RMSE, coverage, log predictive density) or by variability across random seeds; in §3.2 the statement that the latent quantile model "manages to fit the actual quantiles well, except at the origin" and the claim that deep GP predictions are "not significantly different" across data sets lack statistical or numerical support; in §4.4 "both succeed in improving the initial estimate towards the goal" is not accompanied by a reported improvement. These claims should either be backed by quantitative results or be explicitly framed as anecdotal; as written, the paper's practical recommendations rest on an unvalidated example.
[§4.4] The sequential design illustration reports only absolute outcomes (≈120 unique designs for IMSPE after 900 evaluations, ≈30 iterations for SUR to reach 1000 evaluations) and does not compare against simple baselines such as a space-filling design without replication, a homoscedastic GP, or a constant-replication strategy. Without such comparisons, the reader cannot judge whether replication-aware strategies actually provide the claimed benefit in this example. Adding a baseline, even for one example, would substantially strengthen the section's conclusions.

minor comments (7)

[§2.3] There is a typo: "heteroscedasic" should be "heteroscedastic".
[§3.1.1] The phrase "it as been proposed" should be "it has been proposed".
[§4.2] The sentence "dedicated criteria have also been obtained in closed form, for multi- van der Herten et al. (2016)" is truncated; it should read "for multi-objective optimization".
[§1] The phrase "distinguish between aleatoric uncertainty – arising from noise in observations, from epistemic uncertainty" reads awkwardly; consider inserting "and" before "from epistemic uncertainty" or restructuring the sentence.
[§2.3] The sentence "This even shows for the reference on the variance and skewness estimation" is unclear; clarify what "this" refers to.
[§2.1] Equations (1)–(2) would be easier to follow if c(x) and C_N were explicitly defined as the correlation vector and correlation matrix, since the notation c(x,x) as a variance function may be confusing when paired with the outer noise term r(x).
[Figure 4] The label "IMPSE infill criterion value" should be "IMSPE infill criterion value" in the figure and caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the chapter is a survey whose methods and equations are attributed to prior literature, and the self-citations are not load-bearing.

full rationale

This paper is a review chapter, not an original derivation. Every modeling framework and sequential design criterion is explicitly attributed to prior work: the GP predictive equations are standard (Rasmussen and Williams 2006; Ankenman et al. 2010), heteroscedastic latent-variance models are traced to Goldberg et al. (1998) and later works, quantile GP methods are credited to Plumlee and Tuo (2014) and Picheny et al. (2022), and sequential design criteria such as IMSPE and SUR are cited to Ankenman et al. (2010), Binois et al. (2019), and Lyu et al. (2021). The authors' own prior papers appear, but they are used as references for existing methods (e.g., the hetGP R package, replication-vs-exploration strategies), not as unverified premises that force the chapter's conclusions. The illustrative SIR examples are demonstrations of these published methods, not predictions derived from fitted parameters in a way that would be circular; the chapter makes no claim to have derived new theory from its own assumptions. The main weakness is evidentiary—the qualitative, single-example illustrations provide limited support for the practical recommendations—but that is a correctness/robustness concern, not circularity. Therefore no step in the paper reduces, by construction or by self-citation, to its own input.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The chapter makes no new derivations. Its instructional content rests on standard GP predictive equations and on the fidelity of its representation of cited methods. The illustrative SIR example involves fitted GP hyperparameters, but these are not load-bearing for the review's claims.

assumptions (2)

standard math The standard GP predictive equations (1)-(2) correctly describe noisy GP regression with input-dependent noise variance.
Invoked as background in Section 2.1; correctness is established in the GP literature and is not derived in this chapter.
domain assumption The cited primary sources accurately report the methods and properties summarized in this chapter.
The chapter is a survey and performs no independent verification; its reliability depends on faithful representation of Goldberg et al. (1998), Ankenman et al. (2010), Binois et al. (2018), Plumlee and Tuo (2014), Picheny et al. (2022), and others.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Gearing Gaussian process modeling and sequential design towards stochastic simulators." pith.science (2026). https://pith.science/paper/PKNWSBJK

@misc{pith2026241207306,
  author       = {Pith},
  title        = {Pith review of: Gearing Gaussian process modeling and sequential design towards stochastic simulators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKNWSBJK}},
  note         = {Machine review of arXiv:2412.07306}
}

read the original abstract

This chapter presents specific aspects of Gaussian process modeling in the presence of complex noise. Starting from the standard homoscedastic model, various generalizations from the literature are presented: input varying noise variance, non-Gaussian noise, or quantile modeling. These approaches are compared in terms of goal, data availability and inference procedure. A distinction is made between methods depending on their handling of repeated observations at the same location, also called replication. The chapter concludes with the corresponding adaptations of the sequential design procedures. These are illustrated in an example from epidemiology.

Figures

Figures reproduced from arXiv: 2412.07306 by the authors.

**Figure 2.** Illustration on a 1d SIR simulator. Homoscedastic versus heteroscedastic models [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Quantile GP models: directly on empirical quantiles (top-left) and using latent [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Sequential learning illustration. Left: initial state, right: state after using IMSPE [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

A portfolio approach to massively parallel Bayesian optimization

Ankenman, B., Nelson, B. L., and Staum, J. (2010). Stochastic kriging for simulation metamodeling. Operations research, 58(2):371–382. Aravkin, A. Y., Bell, B. M., Burke, J. V., and Pillonetto, G. (2015). The connec- tion between Bayesian estimation of a Gaussian random field and RKHS. IEEE transactions on neural networks and learning systems , 26(7):1518...

work page Pith review arXiv 2010
[15]

W., Williams, C

Goldberg, P. W., Williams, C. K., and Bishop, C. M. (1998). Regression with input- dependent noise: A Gaussian process treatment. Advances in neural information processing systems, pages 493–499. 16 Gonzalez, S. R., Jalali, H., and Van Nieuwenhuyse, I. (2020). A multiobjective stochas- tic simulation optimization algorithm. European Journal of Operational...

work page 1998
[16]

Lawrence, N. D. and Moore, A. J. (2007). Hierarchical Gaussian process latent variable models. In Proceedings of the 24th International Conference on Machine Learning , ICML ’07, page 481–488, New York, NY, USA. Association for Computing Machin- ery. L´ azaro-Gredilla, M. (2012). Bayesian warped Gaussian processes.Advances in Neural Information Processing...

work page 2007
[22]

Vazquez, E., Villemonteix, J., Sidorkiewicz, M., and Walter, E. (2008). Global op- timization based on noisy evaluations: An empirical study of two statistical ap- proaches. Journal of Physics: Conference Series , 135:012100. Wang, W. and Haaland, B. (2019). Controlling sources of inaccuracy in stochastic kriging. Technometrics. Wang, Z., Shi, J. Q., and ...

work page 2008
[25]

V., Smola, A

Le, Q. V., Smola, A. J., and Canu, S. (2005). Heteroscedastic Gaussian process re- gression. In Proceedings of the 22nd international conference on Machine learning , pages 489–496. ACM. Lederer, A., Umlauft, J., and Hirche, S. (2019). Uniform error bounds for Gaussian process regression with application to safe control. Advances in Neural Information Pro...

work page 2005
[28]

T., Notz, W

Huang, D., Allen, T. T., Notz, W. I., and Zeng, N. (2006). Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of global optimization, 34:441–466. Hunter, S. R., Applegate, E. A., Arora, V., Chong, B., Cooper, K., Rinc´ on-Guevara, O., and Vivas-Valencia, C. (2017). An introduction to multi-objective simulation ...

work page 2006
[31]

Garnett, R. (2023). Bayesian Optimization . Cambridge University Press. Ginsbourger, D. (2018). Sequential Design of Computer Experiments, pages 1–9. Amer- ican Cancer Society. Girard, A., Rasmussen, C., Candela, J. Q., and Murray-Smith, R. (2002). Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting. Ad...

work page 2023
[32]

Letham, B., Karrer, B., Ottoni, G., and Bakshy, E. (2019). Constrained Bayesian optimization with noisy experiments. Bayesian Analysis, 14(2):495. Lyu, X., Binois, M., and Ludkovski, M. (2021). Evaluating Gaussian process metamod- els and sequential designs for noisy level set estimation. Statistics and Computing , 31(4):1–21. Makarova, A., Usmanova, I., ...

arXiv 2019

Show all 12 references

[46]

P., and John, S

Dutordoir, V., Salimbeni, H., Hambro, E., McLeod, J., Leibfried, F., Artemev, A., van der Wilk, M., Hensman, J., Deisenroth, M. P., and John, S. (2021). GPflux: A library for deep Gaussian processes. arXiv preprint arXiv:2104.05674 . El Amri, M. R., Helbert, C., Zuniga, M. M.,...

2021 arXiv
[230]

Klein, A., Bartels, S., Falkner, S., Hennig, P., and Hutter, F

Springer. Klein, A., Bartels, S., Falkner, S., Hennig, P., and Hutter, F. (2015). Towards efficient Bayesian optimization for big data. In NIPS 2015 Bayesian Optimization Workshop . Kozubowski, T. J., Podg´ orski, K., and Rychlik, I. (2013). Multivariate generalized Laplace di...

2015
[301]

and Le Riche, R

Janusevskis, J. and Le Riche, R. (2013). Simultaneous kriging-based estimation and optimization of mean response. Journal of Global Optimization , 55:313–336. Kanagawa, M., Hennig, P., Sejdinovic, D., and Sriperumbudur, B. K. (2018). Gaussian processes and kernel methods: A re...

2013 arXiv
[2009]

Ninth IEEE International Conference on , pages 938–943

ICDM’09. Ninth IEEE International Conference on , pages 938–943. IEEE. Quan, N., Yin, J., Ng, S. H., and Lee, L. H. (2013). Simulation optimization via kriging: a sequential search using expected improvement with computing budget constraints. Iie Transactions, 45(7):763–780. 1...

2013 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

A portfolio approach to massively parallel Bayesian optimization

Ankenman, B., Nelson, B. L., and Staum, J. (2010). Stochastic kriging for simulation metamodeling. Operations research, 58(2):371–382. Aravkin, A. Y., Bell, B. M., Burke, J. V., and Pillonetto, G. (2015). The connec- tion between Bayesian estimation of a Gaussian random field and RKHS. IEEE transactions on neural networks and learning systems , 26(7):1518...

work page Pith review arXiv 2010

[2] [15]

W., Williams, C

Goldberg, P. W., Williams, C. K., and Bishop, C. M. (1998). Regression with input- dependent noise: A Gaussian process treatment. Advances in neural information processing systems, pages 493–499. 16 Gonzalez, S. R., Jalali, H., and Van Nieuwenhuyse, I. (2020). A multiobjective stochas- tic simulation optimization algorithm. European Journal of Operational...

work page 1998

[3] [16]

Lawrence, N. D. and Moore, A. J. (2007). Hierarchical Gaussian process latent variable models. In Proceedings of the 24th International Conference on Machine Learning , ICML ’07, page 481–488, New York, NY, USA. Association for Computing Machin- ery. L´ azaro-Gredilla, M. (2012). Bayesian warped Gaussian processes.Advances in Neural Information Processing...

work page 2007

[4] [22]

Vazquez, E., Villemonteix, J., Sidorkiewicz, M., and Walter, E. (2008). Global op- timization based on noisy evaluations: An empirical study of two statistical ap- proaches. Journal of Physics: Conference Series , 135:012100. Wang, W. and Haaland, B. (2019). Controlling sources of inaccuracy in stochastic kriging. Technometrics. Wang, Z., Shi, J. Q., and ...

work page 2008

[5] [25]

V., Smola, A

Le, Q. V., Smola, A. J., and Canu, S. (2005). Heteroscedastic Gaussian process re- gression. In Proceedings of the 22nd international conference on Machine learning , pages 489–496. ACM. Lederer, A., Umlauft, J., and Hirche, S. (2019). Uniform error bounds for Gaussian process regression with application to safe control. Advances in Neural Information Pro...

work page 2005

[6] [28]

T., Notz, W

Huang, D., Allen, T. T., Notz, W. I., and Zeng, N. (2006). Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of global optimization, 34:441–466. Hunter, S. R., Applegate, E. A., Arora, V., Chong, B., Cooper, K., Rinc´ on-Guevara, O., and Vivas-Valencia, C. (2017). An introduction to multi-objective simulation ...

work page 2006

[7] [31]

Garnett, R. (2023). Bayesian Optimization . Cambridge University Press. Ginsbourger, D. (2018). Sequential Design of Computer Experiments, pages 1–9. Amer- ican Cancer Society. Girard, A., Rasmussen, C., Candela, J. Q., and Murray-Smith, R. (2002). Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting. Ad...

work page 2023

[8] [32]

Letham, B., Karrer, B., Ottoni, G., and Bakshy, E. (2019). Constrained Bayesian optimization with noisy experiments. Bayesian Analysis, 14(2):495. Lyu, X., Binois, M., and Ludkovski, M. (2021). Evaluating Gaussian process metamod- els and sequential designs for noisy level set estimation. Statistics and Computing , 31(4):1–21. Makarova, A., Usmanova, I., ...

arXiv 2019