pith. sign in

arxiv: 1907.11313 · v1 · pith:CVIY3DMQnew · submitted 2019-07-25 · 📊 stat.ML · cs.LG· stat.AP

Towards Scalable Gaussian Process Modeling

Piyush Pandita , Jesper Kristensen , Liping Wang This is my paper

Pith reviewed 2026-05-24 15:43 UTC · model grok-4.3

classification 📊 stat.ML cs.LGstat.AP
keywords Gaussian Processsurrogate modelingAdaptive Sequential Monte Carlohyperparameter estimationscalable Bayesian modelingindustrial applicationsMarkov chain Monte Carlo
0
0 comments X

The pith

Adaptive Sequential Monte Carlo replaces MCMC in GEBHM to train Gaussian Processes on large datasets faster while preserving prediction quality.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that swapping Markov chain Monte Carlo for Adaptive Sequential Monte Carlo when estimating Gaussian Process hyperparameters inside the GEBHM framework cuts computation time on large problems without hurting accuracy. This matters for industrial settings where datasets exceed 1000 points and hundreds of thousands of expensive simulations are needed. The authors show the change works on four mathematical test functions plus two real engineering applications of varying size. A reader would care because it removes a practical barrier to using probabilistic surrogate models on bigger, higher-dimensional problems.

Core claim

The paper claims that an Adaptive Sequential Monte Carlo methodology implemented in GEBHM for training Gaussian Processes enables modeling of large-scale industry problems. This implementation saves computational time especially for large-scale problems while not sacrificing predictability over the current MCMC implementation, as demonstrated on mathematical benchmarks and challenging industry applications.

What carries the argument

Adaptive Sequential Monte Carlo (ASMC) procedure for estimating Gaussian Process hyperparameters inside the GEBHM framework.

If this is right

  • GEBHM becomes usable on datasets larger than the previous 1000-point limit.
  • Hyperparameter training time drops for high-dimensional or high-volume engineering data.
  • Bayesian hybrid surrogate modeling retains its accuracy advantages at industrial scales.
  • The same GP models remain reliable for downstream optimization or insight tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar ASMC replacements could be tried in other Gaussian Process libraries that currently rely on MCMC.
  • The approach might combine with sparse approximation methods to push scalability even further.
  • Industry teams could test the method on problems with millions of points to map remaining bottlenecks.

Load-bearing premise

That hyperparameter estimates from ASMC produce Gaussian Process models whose predictive performance on held-out data matches or exceeds the performance obtained from MCMC.

What would settle it

A side-by-side test on a held-out set from one of the large industry problems where the ASMC-trained model shows clearly higher prediction error or worse uncertainty calibration than the MCMC-trained model.

Figures

Figures reproduced from arXiv: 1907.11313 by Jesper Kristensen, Liping Wang, Piyush Pandita.

Figure 1
Figure 1. Figure 1: Subfigures (a), (b) and (c) show the scalability, predictive accuracy and [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Root mean squared error versus time taken to build the GP model for the two [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The setup of the torsion vibration problem. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: The convergence for the workstation based runs can be seen in Fig. 5 (a), where [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Subfigure (a) Number of particles on a workstation (red dots) are 6, 12, 30, [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Subfigure (a) Number of particles on a workstation (red dots) are 6, 12, 30, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Subfigure (a) Number of particles on a workstation (red dots) are 6, 12, 30, [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Subfigure (a) Number of particles on a workstation (red dots) are 6, 12, 30, [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Subfigure (a) Number of particles on a workstation (red dots) are 6, 12, 30, [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
read the original abstract

Numerous engineering problems of interest to the industry are often characterized by expensive black-box objective experiments or computer simulations. Obtaining insight into the problem or performing subsequent optimizations requires hundreds of thousands of evaluations of the objective function which is most often a practically unachievable task. Gaussian Process (GP) surrogate modeling replaces the expensive function with a cheap-to-evaluate data-driven probabilistic model. While the GP does not assume a functional form of the problem, it is defined by a set of parameters, called hyperparameters. The hyperparameters define the characteristics of the objective function, such as smoothness, magnitude, periodicity, etc. Accurately estimating these hyperparameters is a key ingredient in developing a reliable and generalizable surrogate model. Markov chain Monte Carlo (MCMC) is a ubiquitously used Bayesian method to estimate these hyperparameters. At the GE Global Research Center, a customized industry-strength Bayesian hybrid modeling framework utilizing the GP, called GEBHM, has been employed and validated over many years. GEBHM is very effective on problems of small and medium size, typically less than 1000 training points. However, the GP does not scale well in time with a growing dataset and problem dimensionality which can be a major impediment in such problems. In this work, we extend and implement in GEBHM an Adaptive Sequential Monte Carlo (ASMC) methodology for training the GP enabling the modeling of large-scale industry problems. This implementation saves computational time (especially for large-scale problems) while not sacrificing predictability over the current MCMC implementation. We demonstrate the effectiveness and accuracy of GEBHM with ASMC on four mathematical problems and on two challenging industry applications of varying complexity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript extends the GEBHM Gaussian Process framework by implementing Adaptive Sequential Monte Carlo (ASMC) for hyperparameter estimation. The central claim is that ASMC reduces computational time (especially for datasets >1000 points) relative to the existing MCMC implementation while preserving predictive performance, with direct side-by-side timing and predictive metrics reported on four mathematical test problems and two industry applications.

Significance. If the reported empirical equivalence in downstream predictions holds, the work addresses a practical scalability barrier in an industry-validated GP tool, enabling modeling of larger engineering problems. The provision of direct timing and predictive comparisons on six problems, rather than purely theoretical arguments, is a positive aspect of the evaluation.

minor comments (3)
  1. [Abstract] Abstract: the claim of preserved predictability and time savings is stated without any quantitative metrics, dataset sizes, or baseline values; including one or two key numbers (e.g., wall-clock ratios and held-out error) would make the abstract self-contained.
  2. [Experiments] Experiments section: the precise definition of the predictability metric (RMSE, negative log predictive density, etc.) and whether all comparisons use the same held-out test sets should be stated explicitly once, rather than assumed from context.
  3. [Results] Notation: the distinction between the original MCMC hyperparameters and the ASMC point estimates (or posterior summaries) used for final prediction is not always clear in the result tables.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the constructive review and the recommendation of minor revision. The positive assessment of the empirical timing and predictive comparisons on the six test problems is appreciated. Since no specific major comments were raised in the report, we have no point-by-point responses to provide at this time but remain ready to address any additional points the editor or referee may identify.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The manuscript implements the standard external ASMC algorithm inside the pre-existing GEBHM framework and validates it via direct empirical timing and held-out predictive metrics on six problems. No derivation step reduces by construction to its own inputs, no parameter is fitted on a subset and then relabeled a prediction, and no load-bearing premise rests on a self-citation chain. The central claim is therefore an empirical performance comparison rather than a self-referential derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no information on free parameters, axioms, or invented entities.

pith-pipeline@v0.9.0 · 5828 in / 938 out tokens · 66127 ms · 2026-05-24T15:43:16.333455+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · 2 internal anchors

  1. [1]

    Andrieu, A

    C. Andrieu, A. Doucet, and R. Holenstein. Particle markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 72(3):269–342, 2010

    work page 2010

  2. [2]

    Andrieu, A

    C. Andrieu, A. Doucet, and E. Punskaya. Sequential monte carlo methods for optimal filtering. In Sequential Monte Carlo Methods in Practice , pages 79–95. Springer, 2001

    work page 2001

  3. [3]

    Bilionis, B

    I. Bilionis, B. A. Drewniak, and E. M. Constantinescu. Crop physiology calibration in the clm. Geoscientific Model Development, 8(4):1071–1083, 2015

    work page 2015

  4. [4]

    Bilionis and P.-S

    I. Bilionis and P.-S. Koutsourelakis. Free energy computations by minimization of kullback–leibler divergence: An efficient adaptive biasing potential method for sparse representations. Journal of Computational Physics , 231(9):3849–3870, 2012

    work page 2012

  5. [5]

    R. P. Brent. An improved monte carlo factorization algorithm. BIT Numerical Mathematics, 20(2):176–184, 1980

    work page 1980

  6. [6]

    C. M. Carlo. Markov chain monte carlo and gibbs sampling. Lecture notes for EEB , 581, 2004

    work page 2004

  7. [7]

    M. K. Cowles and B. P. Carlin. Markov chain monte carlo convergence diagnostics: a comparative review. Journal of the American Statistical Association , 91(434):883– 904, 1996

    work page 1996

  8. [8]

    Diaconis

    P. Diaconis. Sequential monte carlo methods in practice, 2003

    work page 2003

  9. [9]

    Doucet, N

    A. Doucet, N. De Freitas, and N. Gordon. An introduction to sequential monte carlo methods. In Sequential Monte Carlo methods in practice , pages 3–14. Springer, 2001

    work page 2001

  10. [10]

    Flournoy

    N. Flournoy. A clinical experiment in bone marrow transplantation: Estimating a percentage point of a quantal response curve. In case studies in Bayesian Statistics , pages 324–336. Springer, 1993

    work page 1993

  11. [11]

    Gelman, J

    A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian data analysis . Chapman and Hall/CRC, 1995

    work page 1995

  12. [12]

    Gelman, G

    A. Gelman, G. O. Roberts, W. R. Gilks, et al. Efficient metropolis jumping rules. Bayesian statistics, 5(599-608):42, 1996

    work page 1996

  13. [13]

    Ghahramani and C

    Z. Ghahramani and C. E. Rasmussen. Bayesian monte carlo. In Advances in neural information processing systems, pages 505–512, 2003

    work page 2003

  14. [14]

    Ghosh, I

    S. Ghosh, I. Asher, J. Kristensen, Y. Ling, K. Ryan, and L. Wang. Bayesian multi- source modeling with legacy data. In 2018 AIAA Non-Deterministic Approaches Conference, page 1663, 2018

    work page 2018

  15. [15]

    W. R. Gilks, S. Richardson, and D. Spiegelhalter. Markov chain Monte Carlo in practice. CRC press, 1995. 14

    work page 1995

  16. [16]

    N. J. Gordon, D. J. Salmond, and A. F. Smith. Novel approach to nonlinear/non- gaussian bayesian state estimation. In IEE Proceedings F (Radar and Signal Pro- cessing), volume 140, pages 107–113. IET, 1993

    work page 1993

  17. [17]

    P. J. Green. Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika, 82(4):711–732, 1995

    work page 1995

  18. [18]

    R. F. Gunst. Response surface methodology: process and product optimization using designed experiments, 1996

    work page 1996

  19. [19]

    Haario, M

    H. Haario, M. Laine, M. Lehtinen, E. Saksman, and J. Tamminen. Markov chain monte carlo methods for high dimensional inversion in remote sensing. Journal of the Royal Statistical Society: series B (statistical methodology) , 66(3):591–607, 2004

    work page 2004

  20. [20]

    Gaussian Processes for Big Data

    J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. arXiv preprint arXiv:1309.6835, 2013

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  21. [21]

    E. T. Jaynes. Information theory and statistical mechanics. Physical review , 106(4):620, 1957

    work page 1957

  22. [22]

    M. C. Kennedy and A. O’Hagan. Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 63(3):425–464, 2001

    work page 2001

  23. [23]

    Kristensen, I

    J. Kristensen, I. Bilionis, and N. Zabaras. Relative entropy as model selection tool in cluster expansions. Physical Review B, 87(17):174112, 2013

    work page 2013

  24. [24]

    Kristensen, I

    J. Kristensen, I. Bilionis, and N. Zabaras. Adaptive simulation selection for the discovery of the ground state line of binary alloys with a limited computational budget. In Recent Progress and Modern Challenges in Applied Mathematics, Modeling and Computational Science , pages 185–211. Springer, 2017

    work page 2017

  25. [25]

    Kristensen, Y

    J. Kristensen, Y. Ling, I. Asher, and L. Wang. Expected-improvement-based meth- ods for adaptive sampling in multi-objective optimization problems. In ASME 2016 International Design Engineering Technical Conferences and Computers and Infor- mation in Engineering Conference , pages V02BT03A024–V02BT03A024. American Society of Mechanical Engineers, 2016

    work page 2016

  26. [26]

    Kullback

    S. Kullback. Information theory and statistics . Courier Corporation, 1997

    work page 1997

  27. [27]

    B.-J. Lee, J. Lee, and K.-E. Kim. Hierarchically-partitioned gaussian process approximation. In Artificial Intelligence and Statistics , pages 822–831, 2017

    work page 2017

  28. [28]

    W. E. Leithead and Y. Zhang. O (n 2)-operation approximation of covariance matrix inverse in gaussian process regression based on quasi-newton bfgs method. Communications in Statistics—Simulation and Computation R⃝, 36(2):367–380, 2007

    work page 2007

  29. [29]

    When Gaussian Process Meets Big Data: A Review of Scalable GPs

    H. Liu, Y.-S. Ong, X. Shen, and J. Cai. When gaussian process meets big data: A review of scalable gps. arXiv preprint arXiv:1807.01065 , 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  30. [30]

    A. O’Hagan. Monte carlo is fundamentally unsound. The Statistician, pages 247–249, 1987

    work page 1987

  31. [31]

    Pan and K.-T

    J.-X. Pan and K.-T. Fang. Maximum likelihood estimation. In Growth curve models and statistical diagnostics , pages 77–158. Springer, 2002

    work page 2002

  32. [32]

    H. Peng, S. Zhe, X. Zhang, and Y. Qi. Asynchronous distributed variational gaussian process for regression. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2788–2797. JMLR. org, 2017. 15

    work page 2017

  33. [33]

    Qui˜ nonero-Candela and C

    J. Qui˜ nonero-Candela and C. E. Rasmussen. A unifying view of sparse approximate gaussian process regression. Journal of Machine Learning Research , 6(Dec):1939– 1959, 2005

    work page 1939

  34. [34]

    Robert and G

    C. Robert and G. Casella. Monte Carlo statistical methods . Springer Science & Business Media, 2013

    work page 2013

  35. [35]

    Schonlau

    M. Schonlau. Computer experiments and global optimization. 1997

    work page 1997

  36. [36]

    Snelson and Z

    E. Snelson and Z. Ghahramani. Local and global sparse gaussian process approxi- mations. In Artificial Intelligence and Statistics , pages 524–531, 2007

    work page 2007

  37. [37]

    C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning. the MIT Press, 2(3):4, 2006

    work page 2006