pith. sign in

arxiv: 2605.23016 · v1 · pith:JQFE5TZPnew · submitted 2026-05-21 · 📊 stat.ME · stat.CO

Sample correlation adjustments for robust Multi-fidelity Monte Carlo under limited pilot sampling

Michael Stanley , Thomas Coons , Geoffrey Bomarito , Patrick Leser , Joshua Pribe , James Warner This is my paper

Pith reviewed 2026-05-25 05:32 UTC · model grok-4.3

classification 📊 stat.ME stat.CO
keywords multi-fidelity Monte Carlosample correlation estimationpilot samplingvariance reductiondiscrepancy functionlimited samplesrobust estimation
0
0 comments X

The pith

A discrepancy function selects correlation estimates minimizing worst-case suboptimality in multi-fidelity Monte Carlo with limited pilots.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a way to adjust correlation estimates between fidelity models in MFMC when only a small number of pilot samples can be afforded. Standard sample correlations often produce suboptimal variance reduction because they ignore the sampling variability in the estimates themselves. The authors define a discrepancy function that quantifies the worst-case expected suboptimality of the resulting MFMC estimator and then choose the correlation value that minimizes this quantity using available probabilistic information on the sample covariance. Demonstrations on a bivariate Gaussian case and a NASA entry-descent-landing application show the adjusted estimators achieve lower variance than the unadjusted sample correlations under the same total budget. The result matters for any setting where computational cost limits the pilot data used to tune a variance-reduction scheme.

Core claim

Leveraging probabilistic information about the sample covariance matrix, the authors construct a discrepancy function that measures the worst-case expected suboptimality of an MFMC estimator arising from correlation estimation error; the correlation estimator is then chosen to minimize this expected discrepancy, producing MFMC estimators with lower variance than those based on the ordinary sample correlation when pilot sample sizes are small.

What carries the argument

The discrepancy function, which quantifies worst-case expected suboptimality of the MFMC estimator with respect to pilot sampling variability and is minimized to select an improved correlation estimate.

If this is right

  • MFMC estimators achieve lower variance for the same total computational budget when pilot samples are few.
  • The adjustment improves performance in applications such as the NASA EDL multi-fidelity model.
  • The bivariate Gaussian example confirms analytically that the method reduces expected suboptimality relative to the sample correlation.
  • The approach applies whenever correlations must be estimated from limited offline samples before the main MFMC run.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same discrepancy construction could be applied to other variance-reduction techniques that depend on estimated parameters from small pilot sets.
  • Practitioners facing budget constraints might routinely replace sample correlations with this adjusted version in existing MFMC codes.
  • The method highlights a general pattern of using sampling distributions to guard against estimation error in Monte Carlo tuning parameters.

Load-bearing premise

Accurate probabilistic information about the sample covariance matrix is available to build the discrepancy function.

What would settle it

Repeated draws of small pilot samples from a setting with known true correlations, followed by a direct comparison showing that the average variance of the MFMC estimator using the adjusted correlations exceeds the average variance obtained with the standard sample correlations.

Figures

Figures reproduced from arXiv: 2605.23016 by Geoffrey Bomarito, James Warner, Joshua Pribe, Michael Stanley, Patrick Leser, Thomas Coons.

Figure 2.1
Figure 2.1. Figure 2.1: An illustration of the suboptimality challenge this paper addresses. The minimizer resulting from [PITH_FULL_IMAGE:figures/full_fig_p005_2_1.png] view at source ↗
Figure 3.1
Figure 3.1. Figure 3.1: Expected true estimator variances for each multi-fidelity estimator, across different pilot sample [PITH_FULL_IMAGE:figures/full_fig_p009_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Figure 3.2: Expected estimator discrepancies for each multi-fidelity estimator, across different pilot sample [PITH_FULL_IMAGE:figures/full_fig_p010_3_2.png] view at source ↗
Figure 3.3
Figure 3.3. Figure 3.3: The expected ratio of true estimator variances, [PITH_FULL_IMAGE:figures/full_fig_p011_3_3.png] view at source ↗
Figure 3.4
Figure 3.4. Figure 3.4: Stacked Shapley values for each component of the sample covariance matrix, for the variance of [PITH_FULL_IMAGE:figures/full_fig_p012_3_4.png] view at source ↗
Figure 5.1
Figure 5.1. Figure 5.1: (Left) EDD and (Right) %EDD across ρ ∈ (0, 1) the bivariate Gaussian problem, showing DDMM’s superior performance over the sample correlation for a collection of pilot sample sizes (N). DDMM dominates the sample correlation up to at least N = 15. robustness of its superior performance to assumption violations. Computationally, we use the toy scenario in Section 2.1 for which all assumptions hold and show… view at source ↗
Figure 5.2
Figure 5.2. Figure 5.2: Average (%)EDD values (over ρ) across a range of pilot sample sizes for the bivariate Gaussian problem. For N < 75, %EDD shows that the DDMM procedure procedures an improvement over the sample correlation, on average. level, α, is set to 0.253 as determined by the procedure detailed in Section E.3. The left panel of [PITH_FULL_IMAGE:figures/full_fig_p018_5_2.png] view at source ↗
Figure 5.3
Figure 5.3. Figure 5.3: Expected variance reduction for all EDL QoIs where the expectation is respect to the randomness [PITH_FULL_IMAGE:figures/full_fig_p020_5_3.png] view at source ↗
Figure 5.4
Figure 5.4. Figure 5.4: (Left) EDD and (Center) %EDD across ρ ∈ (0, 1) show theoretical and empirical values from the EDL data. (Right) similarly shows mean MSE percent changes (unadjusted to adjusted) across the dataset QoIs. Each point corresponds to a QoI. These three plots show that DDMM’s superior performance holds across the dataset QoIs with respect to both (%)EDD and mean MSE metrics. The first two plots further indicat… view at source ↗
Figure 5.5
Figure 5.5. Figure 5.5: (Left) Lofi versus Hifi outputs for the Terminal Velocity QoI. (Center) Lofi versus Hifi output for the Maximum Acceleration QoI. (Right) Sampling correlation sampling distributions along with their theoretical densities (under the bivariate Gaussian assumption) for both QoIs. Although both Terminal Velocity and Maximum Acceleration are non-Gaussian in a similar qualitative way, their resulting sample co… view at source ↗
Figure 5.6
Figure 5.6. Figure 5.6: DDMM behavior for the Terminal Velocity QoI. [PITH_FULL_IMAGE:figures/full_fig_p022_5_6.png] view at source ↗
read the original abstract

Multi-fidelity Monte Carlo (MFMC) is a variance reduction method that leverages a multi-fidelity ensemble of models of varying cost and accuracy levels. Constructing an MFMC estimator with optimal variance requires knowledge of the correlation coefficients between the different fidelity models which are not usually known in practice. The correlations are typically estimated using offline pilot samples and the sample correlation formula, after which the MFMC method proceeds as if the estimated correlations are the true correlations. Computational cost often restricts the number of pilot samples used leading to poor correlation estimates and suboptimal estimators. Leveraging the MFMC problem setting and probabilistic information about the sample covariance matrix, we present a method to improve standard sample-based correlation estimates in the presence of limited pilot samples. We define a novel discrepancy function quantifying the estimator suboptimality which in turn facilitates selecting a correlation estimator minimizing the worst-case expected discrepancy, where the expectation is taken with respect to the pilot sampling variability. Through a simple bivariate Gaussian example and a multi-fidelity modeling application from a NASA Entry, Descent, and Landing (EDL) problem, we show that this method produces better MFMC estimators than the standard sample covariance under small pilot sample sizes and limited total budgets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes a method to adjust sample correlation estimates for multi-fidelity Monte Carlo (MFMC) under limited pilot sampling. It defines a discrepancy function that quantifies worst-case expected suboptimality of the MFMC estimator by leveraging probabilistic information about the sample covariance matrix, then selects the correlation estimator minimizing the expected discrepancy over pilot-sample variability. The approach is demonstrated via a bivariate Gaussian example and a NASA Entry, Descent, and Landing (EDL) multi-fidelity modeling application, with the claim that it yields better MFMC estimators than the standard sample correlation for small pilot sizes and limited total budgets.

Significance. If validated, the adjustment could improve robustness of MFMC variance reduction in resource-limited settings common to engineering applications. The grounding in external probabilistic properties of the sample covariance (rather than data-dependent fitting) is a methodological strength. However, the current demonstrations provide limited quantitative support, so the practical significance remains provisional pending further analysis.

major comments (3)
  1. [Abstract] Abstract: the central claim that the method 'produces better MFMC estimators' rests on two examples, yet the abstract supplies no derivation steps for the discrepancy function, no quantitative metrics (e.g., variance ratios or MSE), and no error analysis; this leaves the improvement unsubstantiated beyond qualitative assertion.
  2. [Bivariate Gaussian example and NASA EDL application] Bivariate Gaussian example and NASA EDL application: the discrepancy function is constructed from the distribution (or moments) of the sample covariance; when fidelity outputs are non-Gaussian or the joint distribution is misspecified, the expectation is taken under the wrong measure, so the selected estimator need not reduce—and may increase—the true MFMC variance relative to the plain sample correlation.
  3. [Method description (discrepancy function construction)] The method's optimality guarantee is conditional on accurate specification of the sample-covariance distribution; no sensitivity analysis or robustness check against distributional misspecification is reported, which is load-bearing for the 'robust' claim under limited pilot sampling.
minor comments (2)
  1. The explicit mathematical form of the discrepancy function and the procedure for minimizing its expectation should be stated in a dedicated section or appendix for reproducibility.
  2. Figure captions and table headings would benefit from clearer indication of which estimator (adjusted vs. sample) is being compared and the precise pilot-sample sizes used.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments highlighting issues with the abstract, distributional assumptions, and lack of sensitivity analysis. We address each major comment below and indicate planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the method 'produces better MFMC estimators' rests on two examples, yet the abstract supplies no derivation steps for the discrepancy function, no quantitative metrics (e.g., variance ratios or MSE), and no error analysis; this leaves the improvement unsubstantiated beyond qualitative assertion.

    Authors: We agree the abstract is concise and omits key quantitative details and derivation references. In the revision we will expand the abstract to include a one-sentence description of the discrepancy function, report the observed variance-reduction ratios from both examples, and reference the relevant sections for the derivation. revision: yes

  2. Referee: [Bivariate Gaussian example and NASA EDL application] Bivariate Gaussian example and NASA EDL application: the discrepancy function is constructed from the distribution (or moments) of the sample covariance; when fidelity outputs are non-Gaussian or the joint distribution is misspecified, the expectation is taken under the wrong measure, so the selected estimator need not reduce—and may increase—the true MFMC variance relative to the plain sample correlation.

    Authors: The discrepancy function is derived under the known Wishart distribution of the sample covariance for jointly Gaussian outputs (Section 3). The NASA EDL example applies the same framework. We acknowledge that misspecification could degrade performance relative to the sample correlation; we will add an explicit limitations paragraph noting this and outlining a bootstrap-based extension for non-Gaussian settings. revision: partial

  3. Referee: [Method description (discrepancy function construction)] The method's optimality guarantee is conditional on accurate specification of the sample-covariance distribution; no sensitivity analysis or robustness check against distributional misspecification is reported, which is load-bearing for the 'robust' claim under limited pilot sampling.

    Authors: We concur that the optimality claim is conditional on correct specification of the sample-covariance distribution and that the absence of sensitivity checks weakens the robustness assertion. In the revised manuscript we will add a new subsection with sensitivity experiments that perturb the assumed distribution parameters and compare resulting MFMC performance. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper introduces a discrepancy function that quantifies worst-case expected suboptimality of the MFMC estimator, with the expectation taken over pilot-sample variability using external probabilistic information about the sample covariance matrix (e.g., scaled Wishart under bivariate Gaussian). This construction relies on known distributional properties independent of the target MFMC result rather than defining the adjustment in terms of itself. No steps reduce by construction to fitted inputs, self-citations, or renamed known results; the improvement is shown via explicit examples without tautological equivalence. The method is self-contained against the stated assumptions.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the MFMC problem setting plus probabilistic properties of the sample covariance matrix; no free parameters or invented entities are visible from the abstract.

axioms (1)
  • domain assumption Probabilistic information about the sample covariance matrix is available and sufficient to define a discrepancy function for worst-case expected suboptimality
    Invoked to construct the novel discrepancy and the selection rule.

pith-pipeline@v0.9.0 · 5751 in / 1256 out tokens · 29272 ms · 2026-05-25T05:32:14.709581+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages

  1. [1]

    M., Eldred, M

    Adams, B. M., Eldred, M. S., Geraci, G., Portone, T., Ridgway, E. M., Stephens, J. A., and Wildey, T. M. (2022). Deployment of Multifidelity Uncertainty Quantification for Thermal Bat- tery Assessment Part I: Algorithms and Single Cell Results. Technical report, Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)

    work page 2022

  2. [2]

    Anderson, T. W. (2003).An Introduction to Multivariate Statistical Analysis. Wiley Inter- science, Hoboken, N.J

    work page 2003

  3. [3]

    and Chatterjee, S

    Azadkia, M. and Chatterjee, S. (2021). A simple measure of conditional dependence.The Annals of Statistics, 49(6):3070–3102

    work page 2021

  4. [4]

    Berger, J. O. (1985).Statistical Decision Theory and Bayesian Analysis. Springer-Verlag, New York

    work page 1985

  5. [5]

    Bomarito, G., Leser, P., Warner, J., and Leser, W. (2022). On the optimization of approxi- mate control variates with parametrically defined estimators.Journal of Computational Physics, 451(C)

    work page 2022

  6. [6]

    (2017).Theδ—Importance Measure, pages 163–180

    Borgonovo, E. (2017).Theδ—Importance Measure, pages 163–180. Springer International Publishing, Cham. 37

    work page 2017

  7. [7]

    Broto, B., Bachoc, F., and Depecker, M. (2020). Variance Reduction for Estimation of Shapley Effects and Adaptation to Unknown Input Distribution.SIAM/ASA Journal on Uncertainty Quantification, 8(2):693–716

    work page 2020

  8. [8]

    Busing, F. M. T. A. (2022). Monotone Regression: A Simple and Fast O(n) PAVA Implemen- tation.Journal of Statistical Software, Code Snippets, 102(1):1–25

    work page 2022

  9. [9]

    and Lugosi, G

    Cesa-Bianchi, N. and Lugosi, G. (2006).Prediction, Learning, and Games. Wiley Interscience

    work page 2006

  10. [10]

    Chen, X., Lin, Q., and Xu, G. (2022). Distributionally robust optimization with confidence bands for probability density functions.INFORMS Journal on Optimization, 4(1):65–89

    work page 2022

  11. [11]

    E., Jivani, A., and Huan, X

    Coons, T. E., Jivani, A., and Huan, X. (2025). Bayesian Covariance Uncertainty for Adap- tive Pilot-Sampling Termination in Multi-fidelity Uncertainty Quantification.arXiv preprint arXiv:2508.18490 [stat.ME]

    work page arXiv 2025

  12. [12]

    Dixon, T., Gorodetsky, A., Jakeman, J., Narayan, A., and Xu, Y. (2026). Optimally balancing exploration and exploitation to automate multi-fidelity statistical estimation.arXiv preprint arXiv:2505.09828 [stat.CO]

    work page arXiv 2026

  13. [13]

    Giles, M. B. (2015). Multilevel Monte Carlo methods.Acta Numerica, 24:259–328

    work page 2015

  14. [14]

    A., Geraci, G., Eldred, M

    Gorodetsky, A. A., Geraci, G., Eldred, M. S., and Jakeman, J. D. (2020). A generalized approximate control variate framework for multifidelity uncertainty quantification.Journal of Computational Physics, 408:109257

    work page 2020

  15. [15]

    A., Jakeman, J

    Gorodetsky, A. A., Jakeman, J. D., and Eldred, M. S. (2024). Grouped approximate control variate estimators.arXiv preprint arXiv:2402.14736 [stat.CO]

    work page arXiv 2024

  16. [16]

    Gramacy, R. B. (2020).Surrogates: Gaussian Process Modeling, Design and Optimization for the Applied Sciences. Chapman Hall/CRC.https://bookdown.org/rbg/surrogates/

    work page 2020

  17. [17]

    Hotelling, H. (1953). New light on the correlation coefficient and its transforms.Journal of the Royal Statistical Society. Series B (Methodological), 15(2):193–232

    work page 1953

  18. [18]

    Jin, C., Netrapalli, P., and Jordan, M. (2020). What is local optimality in nonconvex- nonconcave minimax optimization? InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 4880–4889. PMLR

    work page 2020

  19. [19]

    I., M¨ uhlemann, A., and Ziegel, J

    Jordan, A. I., M¨ uhlemann, A., and Ziegel, J. F. (2020). Optimal solutions to the isotonic regression problem.arXiv preprint arXiv:1904.04761 [math.ST]

    work page arXiv 2020

  20. [20]

    Kosorok, M. R. (2008).Introduction to Empirical Processes and Semiparametric Inference. Springer Series in Statistics. Springer, New York, NY

    work page 2008

  21. [21]

    Kossaifi, J., Panagakis, Y., Anandkumar, A., and Pantic, M. (2019). TensorLy: Tensor Learning in Python.Journal of Machine Learning Research (JMLR), 20(26):1–6

    work page 2019

  22. [22]

    and Xia, Y

    Krishnamoorthy, K. and Xia, Y. (2007). Inferences on correlation coefficients: One-sample, independent and correlated cases.Journal of Statistical Planning and Inference, 137:2362–2379

    work page 2007

  23. [23]

    S., Moeller, T

    Lavenberg, S. S., Moeller, T. L., and Welch, P. D. (1982). Statistical results on control variables with applications to queueing network simulation.Operations Research, 30(1):182–202. 38

    work page 1982

  24. [24]

    Lin, T., Jin, C., and Jordan, M. (2020). On gradient descent ascent for nonconvex-concave minimax problems. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 6083–6093. PMLR

    work page 2020

  25. [25]

    Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. (2009). Robust stochastic approximation approach to stochastic programming.SIAM Journal on Optimization, 19(4):1574–1609

    work page 2009

  26. [26]

    Owen, A. B. (2014). Sobol’ Indices and Shapley Value.SIAM/ASA Journal on Uncertainty Quantification, 2(1):245–251

    work page 2014

  27. [27]

    Owen, A. B. and Prieur, C. (2017). On Shapley Value for Measuring Importance of Dependent Inputs.SIAM/ASA Journal on Uncertainty Quantification, 5(1):986–1002

    work page 2017

  28. [28]

    Peherstorfer, B., Willcox, K., and Gunzburger, M. (2016). Optimal Model Management for Multifidelity Monte Carlo Estimation.SIAM Journal on Scientific Computation, 38(5):A3163– A3194

    work page 2016

  29. [29]

    Pollard, D. (1989). Asymptotics via Empirical Processes.Statistical Science, 4(4):341–354

    work page 1989

  30. [30]

    Rasmussen, C. E. and Williams, C. K. I. (2006).Gaussian Processes for Machine Learning. MIT Press, Cambridge, Mass

    work page 2006

  31. [31]

    (1991).Functional Analysis

    Rudin, W. (1991).Functional Analysis. International series in pure and applied mathematics. McGraw-Hill, New York, 2nd edition

    work page 1991

  32. [32]

    and Ullmann, E

    Schaden, D. and Ullmann, E. (2020). On Multilevel Best Linear Unbiased Estimators. SIAM/ASA Journal on Uncertainty Quantification, 8(2):601–635

    work page 2020

  33. [33]

    Shapley, L. S. (2016).A Value for n-Person Games, chapter 17, pages 307–318. Princeton University Press, Princeton

    work page 2016

  34. [34]

    Sobol’, I. (2001). Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates.Mathematics and Computers in Simulation, 55(1):271–280. The Second IMACS Seminar on Monte Carlo Methods

    work page 2001

  35. [35]

    L., and Staum, J

    Song, E., Nelson, B. L., and Staum, J. (2016). Shapley Effects for Global Sensitivity Analysis: Theory and Computation.SIAM/ASA Journal on Uncertainty Quantification, 4(1):1060–1083

    work page 2016

  36. [36]

    P., Eldred, M

    Thompson, M., Geraci, G., Bomarito, G., Warner, J., Leser, P., Leser, W. P., Eldred, M. S., Jakeman, J., and Gorodetsky, A. (2023). Strategies for automation of model tuning in multi- fidelity trajectory uncertainty propagation.AIAA SCITECH 2023 Forum

    work page 2023

  37. [37]

    E., Bomarito, G

    Warner, J. E., Bomarito, G. F., Geraci, G., and Eldred, M. S. (2026). Automated Model Tuning for Multifidelity Uncertainty Propagation in Trajectory Simulation.arXiv preprint arXiv:2509.16007 [stat.CO]

    work page arXiv 2026

  38. [38]

    E., Niemoeller, S

    Warner, J. E., Niemoeller, S. C., Morrill, L., Bomarito, G. F., Leser, P. E., Leser, W. P., Williams, R. A., and Dutta, S. (2021). Multi-Model Monte Carlo Estimators for Trajectory Simulation.AIAA SciTech Forum. 39

    work page 2021