{"id":"aff79c09-31e6-45c2-9470-75916ff39ca7","arxiv_id":"2411.13361","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MCMC-guided active learning for Gaussian process surrogates outperforms a priori training for Bayesian calibration, and the forward model is the limiting factor, not the MCMC algorithm.","lead":"This paper compares ways to speed up Bayesian calibration of mechanical properties by coupling Gaussian process surrogates with MCMC sampling. It finds that an active learning strategy, which trains the surrogate along the MCMC path, beats pre-trained surrogates, and that the forward model, not the sampler, is the real bottleneck.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Active-learning superiority rests on the GP predictive variance being a trustworthy error signal; Section 5.4.1 shows this signal silently fails (an overconfident surrogate stops collecting data), and Algorithm 1 offers no calibration safeguard or diagnostic.","rationale":"The reader's weakest_assumption identifies exactly the concern I consider most load-bearing: the active learning strategy's guarantee rests on the GP predictive variance being a faithful error signal, and the paper's own Section 5.4.1 documents precisely the failure mode in which this signal breaks down, with no safeguard provided. My read therefore agrees with the reader, and the existing CONDITIONAL verdict already captures the needed remedy (either demonstrate calibration or add a safeguard and temper the claims). I considered but did not elevate three other concerns. First, the infeasibility/bottleneck claim extrapolates training-data counts from d = 2-8 to d = 15 and to a general research-priority recommendation; this is a real scoping limitation, but the exponential growth of required data for a stationary-kernel GP is expected, the extrapolation is transparent, and the authors explicitly call for multi-fidelity and dimension-reduction alternatives, so the claim is adequately scoped to the tested framework. Second, the 'no significant influence of the MCMC algorithm' statement lacks formal statistical testing; this is a secondary claim and the reader already flagged it without it changing the verdict. Third, the thresholds γv and γL are tuned on the five-dimensional problem and applied across dimensions; this is relevant but secondary to the calibration concern, since the failure appears even at the strict threshold in a fraction of MALA runs. Credit is due where the paper supports itself: the code is available, the reference posterior is computed with 200,000 forward-model MCMC samples, the active-learning versus a-priori comparison is carefully designed with equalized training counts in Fig. 10, and the paper honestly reports its own failure mode. My verdict recommendation is UNCHANGED because the concern reinforces the reader's weakest assumption rather than introducing a new one; the concrete test above would settle whether the concern lands by directly measuring predictive-variance calibration along the MCMC path.","tokens_in":22730,"tokens_out":8127,"duration_ms":93731,"concrete_test":"Run a MALA experiment at γv = 1.0 (the strict threshold) and, at the end of burn-in, take 100-200 proposal locations that did NOT trigger data collection (V < γv) and evaluate the true log-likelihood at those points with the forward model. Compute the empirical coverage of the true values within the GP's 95% predictive intervals (µ ± 1.96·√V). If coverage is far below 95% (say < 70%) and this subset of runs coincides with the high-W2 outliers in Fig. 12b, the variance signal is miscalibrated and the active-learning stopping criterion is the cause. As a follow-up, re-run the same experiments with the GP length scale capped at the value estimated from the RWM-trained surrogate; if the outliers disappear, an overestimated length scale is confirmed and a calibration safeguard (e.g., a validation batch during burn-in) is a required component of the method, not an optional refinement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central methodological claim — that the MCMC-guided active learning strategy of Algorithm 1 reliably builds superior surrogates — depends on the GP predictive variance V[L(θ*)] (Eq. 38) being an accurate proxy for surrogate error in unexplored regions. The paper's own results demonstrate that this assumption fails. Section 5.4.1 attributes the high-W2 MALA outliers under relaxed thresholds (γv = 5.0, 20.0) to initial MCMC samples lying far from the posterior high-density region, which yields an overestimated GP length scale, an overconfident surrogate, and a cessation of training data collection. This is a failure of the acquisition mechanism itself, not merely a sampler property: once V < γv holds in a wrongly confident region, no forward evaluations are ever triggered, and the RWM recovers only by chance exploration. Algorithm 1 contains no check that the predictive variance is calibrated (e.g., no coverage test of true log-likelihood values against GP credible intervals) and no fallback; the retrain threshold γL re-estimates hyperparameters on the same biased dataset and can reinforce the overconfidence. Because the superiority claim is reported as an average over seeds (Figs. 9-10) while the failure is silent and produces confidently wrong posteriors, the paper's recommendation to prioritize surrogate construction over MCMC algorithm choice inherits this risk. The paper reports this failure mode transparently, which is to its credit, but it frames the issue as a robustness property of MALA rather than as a limitation of the active-learning mechanism, and no safeguard is proposed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies surrogate-accelerated Bayesian calibration on a scalable one-dimensional bar problem with a non-linear constitutive model and a random-field parameterization of the initial stiffness. It compares GP-based active learning driven by MCMC trajectories (Algorithm 1) with three offline training strategies (grid, LHS, prior) and compares RWM and MALA both for constructing the surrogate and for sampling from a fixed surrogate. Accuracy is measured by Wasserstein-2 distance against a brute-force reference posterior. The main claims are that offline surrogates are inaccurate except in low dimensions, that the proposed active learning strategy is superior, that the MCMC choice has little effect on training-data quantity and no significant effect on accuracy, and that therefore the forward model, not the sampler, is the bottleneck.","tokens_in":23054,"tokens_out":7535,"duration_ms":79083,"significance":"The paper provides a clean experimental design: a reference posterior computed by long forward-model MCMC runs, repeated random seeds (10 to 50 per configuration), Wasserstein comparisons, and released code. The finding that a priori surrogate training becomes unreliable as dimension grows and that active learning along the MCMC path concentrates data more efficiently is practically useful and likely to influence how practitioners combine surrogates with MCMC. The paper also deserves credit for reporting the overconfident-GP failure mode in Section 5.4.1 rather than hiding it. However, the significance is tempered by the fact that the central superiority claim rests on an uncalibrated acquisition criterion, and by several abstract claims that go beyond the presented evidence.","major_comments":[{"comment":"The active-learning acquisition rule uses the GP predictive variance V[L(θ*)] in Eq. (38) as the sole trigger for adding training data. The paper itself reports in Section 5.4.1 that, under relaxed thresholds (γv = 5.0, 20.0), initial MALA samples far from the posterior high-density region can produce an overestimated GP length scale, an overconfident surrogate, and a cessation of data collection. This is a failure of the acquisition mechanism itself, not merely a property of the sampler: once V < γv in a wrongly confident region, no forward evaluations are triggered. Algorithm 1 contains no check that the predictive variance is calibrated (e.g., a coverage test of true log-likelihood values against GP credible intervals) and no fallback; the retrain criterion γL re-estimates hyperparameters on the same biased dataset and can reinforce the overconfidence. Because the paper's central recommendation—prioritize surrogate construction over sampler choice—depends on the active-learning strategy being reliable, this silent failure mode needs a diagnostic and a safeguard, and the conditions for the superiority claim need to be stated.","section":"Section 5.4.1, Algorithm 1, Eq. (38)"},{"comment":"The abstract states that a priori training \"introduces large errors in the posterior estimation even in low to moderate dimensions,\" but Fig. 9 shows nearly equal Wasserstein distances across all strategies for d=2 and d=3, with clear superiority of active learning emerging only at d=4 and d=5. The phrase \"even in low to moderate dimensions\" overstates the evidence and should be revised to reflect this dimension-dependent crossover.","section":"Abstract, Section 5.3, Fig. 9"},{"comment":"The claim that the MCMC algorithm has \"no significant influence\" on accuracy is asserted without a statistical test. Fig. 14 shows overlapping box distributions for d≤8 but an increasing number of high-W2 outliers from d=10 onward, with the cross-marked runs indicating chains that left the high-density region. A formal comparison over the 50 seeds (e.g., paired tests or outlier-rate reporting) is needed before claiming no significant influence.","section":"Abstract, Section 5.4.3, Fig. 14"},{"comment":"The \"70 days\" infeasibility calculation extrapolates training-data counts from Fig. 12 up to d=15 and assumes a 10-minute forward evaluation. The manuscript does not report the fitted scaling relation or its uncertainty, and the d=15 panel in Fig. 14 uses a pre-trained surrogate rather than active learning, so the extrapolation is not directly supported by the experiments. The conclusion should be presented as a conditional estimate with the scaling law stated.","section":"Section 6, Fig. 12"}],"minor_comments":[{"comment":"The caption contains typos: \"strateries\" should be \"strategies\" and \"scatterd\" should be \"scattered.\"","section":"Fig. 10 caption"},{"comment":"The text uses \"RMW\" instead of \"RWM\" in several places; please standardize the abbreviation.","section":"Section 5.4.1 and Fig. 13"},{"comment":"The retrain criterion |Lnew/Lold| > γL is confusing because L is defined in Eq. (37) as the logarithm of the marginal likelihood; a ratio of logarithms is not a standard relative-change measure. Please define the criterion explicitly, for example as a relative change of the log marginal likelihood or as a difference, and state which threshold values correspond to which definition.","section":"Algorithm 1, Eqs. (36)-(37)"},{"comment":"The MALA surrogate proposal in Eq. (39) uses M^{-1} in the drift term, while the exact MALA proposal in Eq. (30) uses M. Please reconcile the preconditioner convention, since this affects reproducibility.","section":"Eq. (39) vs. Eq. (30)"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with transparent failure reporting and reusable code. The main risk is overgeneralization from the specific GP/MCMC setup; I would ask for the calibration safeguard and moderated claims rather than new experiments of a different scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid empirical study on combining GP surrogates with MCMC for Bayesian calibration. The main news: with a spatial-field test problem, a priori trained surrogates become unreliable as dimension grows, and the forward model — not the sampler — is the bottleneck. The scaling law for training data versus dimension is the strongest contribution; the estimate that a 15-dimensional problem needs ~70 days of forward solves makes the point concrete.\n\nThe experimental design is genuinely careful: repeated seeds, an independent brute-force reference posterior, Wasserstein-2 distances, and released code. The active learning algorithm is simple and clearly laid out. The comparison of LHS/grid/prior against the MCMC-guided strategy is useful, especially the same-budget comparison in Fig. 10, where active learning wins at every d.\n\nSoft spots, in rough order of importance. First, the abstract says a priori training introduces large errors 'even in low to moderate dimensions,' but Fig. 9 shows parity at d=2 and d=3; that claim should be tempered. Second, 'no significant influence' of the MCMC algorithm is asserted without any formal test; it looks plausible from the plots, but the wording overreaches. Third, the active learning thresholds (γv, γL, N0) are tuned on d=5 and then carried over; Fig. 7 shows strong sensitivity to γv, so this needs a sensitivity check or justification. Fourth — and this is the one that matters most — the entire active-learning mechanism trusts the GP predictive variance as an error signal. The paper's own Section 5.4.1 shows that with relaxed thresholds the MALA produces an overconfident surrogate that stops collecting data and never recovers. That is a failure of the acquisition mechanism, not just a sampler property. The paper reports it honestly, but frames it as a robustness property of RWM vs MALA rather than a limitation of the active-learning principle. A calibration check on the GP variance (e.g., coverage of held-out log-likelihood values) or a fallback trigger would address this. With stringent γv the method works well, so this is a manageable limitation, not a fatal flaw.\n\nWho it's for: computational mechanics researchers doing Bayesian inverse problems with expensive solvers, and methodologists working on surrogate-accelerated MCMC. The empirical results are reproducible and the central comparison is credible. Send it to peer review, but ask the authors to tone down the abstract, add a statistical test for the sampler effect, and discuss the GP-variance failure as a limitation of active learning.","headline":"A careful empirical study that makes a strong case that forward-model cost dominates MCMC choice in surrogate-based calibration, but the active-learning mechanism has a known silent failure mode that the paper underplays.","tokens_in":23585,"tokens_out":3381,"would_cite":true,"duration_ms":35144,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that in surrogate-accelerated Bayesian calibration, the accuracy of the posterior is determined by how surrogate training data are selected, not by the choice of MCMC sampler, and that the forward model itself is the…","keywords":["Bayesian calibration","active learning","Gaussian process surrogate","Markov chain Monte Carlo","computational mechanics","random field inference","surrogate modelling","Metropolis-adjusted Langevin algorithm"],"falsifier":"A concrete test: on the same bar problem at dimension d=10, run the active learning rule with the GP's predictive variance threshold but with the GP length scale deliberately overestimated by a factor of two, and compare the resulting posterior to the reference; if the posterior remains accurate, the method survives overconfidence, and if it diverges, the uncertainty signal is the load-bearing component. Alternatively, count the number of training points collected under a threshold that scales with dimension and check whether the divergence rate matches the paper's observed outlier pattern.","tokens_in":22546,"feed_emoji":"⚙️","tokens_out":4741,"duration_ms":48137,"temperature":0.7,"pith_summary":"This paper argues that in surrogate-accelerated Bayesian calibration of mechanical models, the accuracy of the posterior depends on how the surrogate's training data are chosen, not on which MCMC sampler is used. It introduces a one-dimensional nonlinear bar problem with spatially varying stiffness, discretized through radial basis functions, to create a family of inverse problems of increasing stochastic dimension. The authors show that a priori training strategies — grid, Latin hypercube, and prior sampling — fail to place points in regions of high posterior density even at moderate dimension. They propose an active learning strategy that follows the MCMC chain and adds a training point whenever the Gaussian process surrogate's predictive variance exceeds a threshold. Their central claim is that the forward model, not the sampler, is the bottleneck: the training data needed to build a reliable surrogate become prohibitive before the gains of gradient-based MCMC on cheap models can be realized.","feed_headline":"Bayesian calibration's bottleneck is training data, not the sampler","feed_subtitle":"MCMC-guided active learning beats grid, LHS, and prior sampling — yet forward-model cost swamps sampler gains.","key_machinery":"The central object is a Gaussian process surrogate for the log-likelihood, trained online by an active learning rule that uses the GP's predictive variance $\\mathbb{V}[L(\\theta^*)]$ as a reject threshold: new posterior proposals that the GP cannot predict confidently trigger a forward-model evaluation, and the retrain threshold $\\gamma_L$ decides when GP hyperparameters are re-estimated. The learning rule is driven by the path of the MCMC chain (Algorithm 1), so training points concentrate in regions of high posterior density. The comparison metric is the Wasserstein 2-distance between the approximate and reference posterior sample clouds.","core_discovery":"The central claim is that surrogate-based Bayesian inference is dominated by the cost of collecting training data for the surrogate, and that a simple uncertainty-threshold active learning rule based on the MCMC path is superior to all a priori trained models. In the authors' experiments, the choice between random-walk Metropolis and the gradient-based Metropolis-adjusted Langevin algorithm has little effect on the accuracy of the resulting posterior; MALA needs somewhat fewer forward evaluations but can fail badly when the GP becomes overconfident under relaxed uncertainty thresholds. Even a surrogate tailored to the posterior does not guarantee that the chain stays in regions it knows well. The paper concludes that the forward model is the bottleneck in the inference process, not the MCMC algorithm.","pith_inferences":["The result suggests that any surrogate-accelerated Bayesian workflow should report training-data cost explicitly as a function of dimension and prior-posterior shift, not just sampler efficiency.","A testable extension would be to replace the variance threshold with a calibrated or conservative uncertainty estimate; the paper's observed MALA failures show the threshold as stated is fragile.","The scalable bar problem could serve as a benchmark for multi-fidelity or dimension-reduced approaches, since it controls the cost of the forward model while isolating the effect of the surrogate and sampler choices."],"forward_implications":["A priori surrogate training (grid, LHS, or prior sampling) should be avoided for Bayesian calibration in dimensions as low as 4 to 5; active learning with the MCMC path is more accurate at equal training budget.","The choice of MCMC sampler is secondary: MALA saves some forward evaluations but does not improve posterior accuracy, and random-walk Metropolis is more robust when the uncertainty threshold is relaxed.","Research effort should shift from advanced samplers to surrogate construction, multi-fidelity schemes, and dimension reduction, because training-data cost dominates the overall inference expense.","Even a well-tuned surrogate can let the chain wander into unexplored regions, so the surrogate's uncertainty estimate should be monitored throughout sampling rather than only during training."],"supporting_citations":[{"why":"Supplies the on-the-fly GP surrogate construction framework that the MCMC-guided active learning strategy builds on.","marker":"[42]"},{"why":"Provides the Gaussian process regression machinery and squared exponential kernel used for the likelihood surrogate.","marker":"[36]"},{"why":"Establishes the optimal scaling and acceptance rate for random-walk Metropolis, used to tune that algorithm.","marker":"[11]"},{"why":"Establishes the optimal scaling and acceptance rate for the Metropolis-adjusted Langevin algorithm, used to tune MALA.","marker":"[12]"},{"why":"Defines Latin hypercube sampling, one of the three a priori training strategies compared against active learning.","marker":"[44]"},{"why":"Justifies the adaptive proposal covariance update used in the MCMC implementations.","marker":"[40]"},{"why":"Represents a related work using a priori surrogate training that the paper argues misidentifies the MCMC algorithm as the bottleneck.","marker":"[16]"},{"why":"Represents a related work with a deep surrogate and Hamiltonian Monte Carlo that the paper contrasts with its own findings.","marker":"[19]"}],"fun_headline_variants":["Surrogate training data, not MCMC, is the real bottleneck","Active learning beats all a priori trained surrogates","Forward model cost swamps MCMC gains","MCMC path guides active learning to best posterior","Surrogate data cost dominates Bayesian calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole active learning method rests on the Gaussian process's predictive variance being a truthful measure of its own error; when the GP becomes overconfident, as the authors observe in MALA runs with a relaxed threshold, no safeguard stops the chain from trusting a biased surrogate.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate training data, not MCMC, is the real bottleneck","Active learning beats all a priori trained surrogates","Forward model cost swamps MCMC gains","MCMC path guides active learning to best posterior","Surrogate data cost dominates Bayesian calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1353,"prompt_tokens":957,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":573,"tokens_out":396,"duration_ms":4317,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:30:10.576592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on the same bar problem at dimension d=10, run the active learning rule with the GP's predictive variance threshold but with the GP length scale deliberately overestimated by a factor of two, and compare the resulting posterior to the reference; if the posterior remains accurate, the method survives overconfidence, and if it diverges, the uncertainty signal is the load-bearing component. Alternatively, count the number of training points collected under a threshold that scales with dimension and check whether the divergence rate matches the paper's observed outlier pattern.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the on-the-fly GP surrogate construction framework that the MCMC-guided active learning strategy builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Gaussian process regression machinery and squared exponential kernel used for the likelihood surrogate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the optimal scaling and acceptance rate for the Metropolis-adjusted Langevin algorithm, used to tune MALA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the adaptive proposal covariance update used in the MCMC implementations."}],"review_version":1}