Adaptive sequential Monte Carlo for structured cross validation in Bayesian hierarchical models

Andrew Gelman; Geonhee Han

REVIEW 3 major objections 7 minor 72 references

Adaptive sequential Monte Carlo for structured cross validation in Bayesian hierarchical models

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adaptive sequential Monte Carlo can approximate structured cross-validation estimands in Bayesian hierarchical models—leave-group-out, group K-fold, and backward sequential leave-end-out—from a single baseline…

desk verdict A practical and mostly sound adaptive-SMC package for structured CV in hierarchical models; the unproven ESS monotonicity assumption is a real but addressable gap. read the letter →

arxiv 2501.07685 v2 pith:6ECNUFYL submitted 2025-01-13 stat.CO stat.AP

Geonhee Han , Andrew Gelman This is my paper · ORCID

classification stat.COstat.AP MSC 62F1565C05

keywords crossvalidationBayesianhierarchicalmodelssequentialMonteCarloadaptiveSMCleave-group-outCVleave-end-outimportancesamplingpredictivemodelassessment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that structured cross validation in Bayesian hierarchical models need not require re-estimating the posterior once per deleted group or subset. It claims an adaptive sequential Monte Carlo sampler can walk from the full-data posterior through automatically chosen intermediate distributions to each case-deleted posterior, reweighting and occasionally rejuvenating a single batch of baseline MCMC draws. If true, this makes predictive checks such as leave-group-out, group K-fold, and backward sequential leave-end-out practical for hierarchical and spatial-temporal models, where standard importance sampling is unstable and brute-force MCMC re-runs are too slow. The payoff is a Bayesian workflow where model comparison and out-of-sample diagnostics are by-products of one initial fit.

What carries the argument

The central object is the adaptive sequential Monte Carlo sampler over an augmented space of posterior draws, with incremental importance weight $w_{k,\ell}(\Theta_{\ell-1}) = \gamma_{k,\ell}(\Theta_{\ell-1})/\gamma_{k,\ell-1}(\Theta_{\ell-1})$ computed at the previous particle positions. Case deletions are encoded as continuous paths: either a geometric (power-scaled) tempering path $\rho_g(n) = p(y_g \mid \theta_g,\phi)^{\varphi_g(n)}$ for dropping whole groups, or ordered within-group factors $\varphi_{g,i}(n) = \min\{\max\{0, i-n\},1\}$ for sequential end-deletion. The mechanism that carries the argument is the automatic construction of intermediate distributions: given a monotone decrease of effective sample size in the deletion parameter, bisection or Brent's method solves for the next parameter value that keeps ESS at a target level, so the user never specifies the path.

What would settle it

A concrete test would be to run the sampler on a Bayesian hierarchical model with a mixture or otherwise multimodal posterior, and check whether the effective sample size as a function of the deletion parameter is non-monotone. If it is, the adaptive root-finding for the next intermediate distribution will pick a parameter value that does not actually meet the ESS threshold, visible as a degenerate resampling step or a stall in the loop.

Watch

Extended reading notes

Core claim

The central claim is that case-deletion paths between the baseline and case-deleted posteriors can be parameterized continuously—by power-scaling likelihood contributions or by ordered within-group deletion—and that the adaptive SMC sampler can discover the bridging distributions automatically. At each step the sampler chooses the next deletion parameter so that the effective sample size of the incremental importance weights meets a target threshold, invoking a Markov kernel (an MCMC move targeting an intermediate distribution) only when the Pareto shape diagnostic indicates the importances are unreliable. The result is an approximation of the log predictive density of each structured CV scheme that closely tracks brute-force MCMC in the paper's three real-data examples, while largely avoiding repeated long MCMC runs.

Load-bearing premise

The sampler's automatic path construction assumes that the effective sample size of the importance weights falls monotonically as more data are deleted, which is stated without proof in the paper (Section 3.3.1); if this monotonicity fails, as it can in multimodal or strongly non-log-concave targets, the adaptive bridge may select invalid intermediate distributions or fail to converge.

Editorial extensions

If this is right

Leave-group-out, group K-fold, and backward sequential leave-end-out predictive densities can be computed from a single baseline MCMC run with accuracy close to brute-force re-runs.
The workflow removes the manual design of tempering schedules: the path of intermediate distributions is chosen by the ESS criterion, with the Markov kernel invoked only when PSIS diagnostics indicate instability.
The same machinery covers multiple CV designs, and it degrades gracefully to plain Pareto-smoothed importance sampling when the deletion is mild.
Because rejuvenation steps are independent across particles and across folds, the method parallelizes, making it substantially faster than re-running MCMC for each fold or group.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same adaptive bridging could be applied to likelihood power-scaling sensitivity analysis, replacing the manual grids currently used in prior and likelihood sensitivity checks.
Editorial inference: for models where conditional independence fails, the ordered within-group deletion path is not available; a covariance-aware or Cholesky-parameterized deletion path would be a natural extension.
Editorial inference: the paper reports empirical accuracy against MCMC but does not give a worst-case bound on the number of intermediate distributions; a complexity analysis relating the number of steps to the $\chi^2$ divergence between baseline and target would sharpen the practical claims.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take on arXiv:2501.07685 (Han and Gelman). The paper proposes adaptive SMC to approximate structured cross-validation (LGO, group K-fold, LEO) in Bayesian hierarchical models, starting from one baseline MCMC run. The building blocks are all prior work — SMC samplers, adaptive tempering, PSIS, Pareto-k diagnostics — but the combination is new and genuinely useful: it automates the path sequence across a broader family of case-deletion schemes, and it embeds the "rerun MCMC only when needed" heuristic into a principled SMC framework. Three real-data examples (radon, yield curves, M5 sales) benchmark against brute-force MCMC; aSMC tracks MCMC where PSIS degrades, and code is public. That is real evidence.

The main caveat is exactly the one your reader flags: the automatic bridge in Section 3.3.1 assumes ESS decreases monotonically in the deletion parameter n, with no proof. In regular, roughly log-concave posteriors this is likely to hold; in multimodal or non-log-concave targets it can fail, and bisection could jump past a useful intermediate. The paper doesn't test that regime, so the "wide range of models" claim in Section 1 outruns the evidence. That said, this is an addressable gap — a monotonicity check or safeguarded search would fix it — and it doesn't undermine the empirical results presented.

Two lesser points: the number of MCMC kernel iterations is still manually set (the paper says so in Section 5), and there is no repeated-run uncertainty quantification on the aSMC estimates, so we don't see how noisy the approximation is. Both are minor relative to the central contribution.

All in, I read this as a solid within-subfield improvement: practitioners doing structured CV in hierarchical models get a cheaper workflow with accuracy close to rerunning MCMC. The reader's CONDITIONAL verdict is about right. I'd send it to peer review; it deserves a serious referee. And I'd cite it if we work on Bayesian workflow or efficient CV.

Referee Report

3 major / 7 minor

Summary. The paper introduces an adaptive sequential Monte Carlo (aSMC) sampler for approximating case-deleted posteriors in structured cross-validation for Bayesian hierarchical models. The authors define several structured case-deletion schemes (leave-group-out, leave-end-out, leave-subset-out including group K-fold), parameterize continuous paths between the baseline posterior and the target case-deleted posterior via likelihood power scaling, and adaptively choose intermediate distributions using an effective sample size (ESS) threshold and a Pareto-k diagnostic. The sampler is designed to reuse a single baseline MCMC run and to minimize additional MCMC re-runs. The method is illustrated on three real-data examples (radon county-level LGO, Dynamic Nelson-Siegel sequential LEO, and M5 retail group K-fold CV), where it is compared against brute-force MCMC and standard PSIS. The paper claims that the sampler automatically constructs the bridging sequence and yields approximations close to MCMC at lower computational cost.

Significance. If the results hold, the method would be a practically useful extension of existing importance-sampling-based cross-validation tools (e.g., PSIS) to structured hierarchical models with distant case-deleted posteriors. The paper explicitly benchmarks against brute-force MCMC and provides reproducible Julia code and data, which is a notable strength. The core idea of adaptively bridging to case-deleted posteriors along a continuous path is natural and well motivated, and the three examples cover interesting CV designs. However, the central 'automatic' path construction relies on an unproven monotonicity assumption on the ESS, and the empirical evaluation does not report uncertainty in either the approximations or the MCMC reference. These issues need to be addressed before the method's advertised generality and accuracy can be fully accepted.

major comments (3)

[Section 3.3.1] The adaptive path construction solves for the next case-deletion parameter n such that the ESS meets a threshold, relying on the statement "as ESS decreases in n". No proof or regularity condition is given for this monotonicity, and it is not guaranteed for general power-scaled likelihoods, especially in multimodal or non-log-concave targets. If the ESS is non-monotone, a bisection or Brent root-finder may converge to a root beyond the first threshold crossing, producing an excessively large bridge step and potentially causing weight collapse or an invalid intermediate distribution. Please either prove monotonicity under stated conditions (e.g., log-concavity) or replace the unguarded root-finding with a safeguarded step-halving procedure with a diagnostic, and discuss when non-monotonicity can occur. This assumption is load-bearing because the paper's claim of automatically constructing the sequence of intermediate distributions depends directly on it.
[Section 4 (Figures 3-8)] The accuracy comparisons against MCMC report only point estimates (absolute error or relative error) without accounting for Monte Carlo variability in either the brute-force MCMC reference or the aSMC approximation. Consequently, the statement that aSMC produces approximations "highly close" to MCMC is not fully supported; the observed differences could be within sampling noise, particularly for groups with small Ng or for early time points in the LEO example. Please report standard errors or credible intervals for the estimands, for instance across independent SMC runs or by using multiple particle replicates, and explicitly state the magnitude of Monte Carlo error in the reference MCMC estimates.
[Section 5 and Algorithm 1] The manuscript acknowledges that the number of MCMC kernel iterations per rejuvenation must be set manually (1-3 for HMC, 5 for Gibbs), but this parameter is not chosen adaptively and can strongly affect the cost-accuracy trade-off. This undercuts the abstract's claim of "automating path construction and systematically minimizing MCMC intervention". Please clarify precisely which components of the workflow are automatic and which remain user-specified, and discuss possible data-driven heuristics or sensitivity analyses for the kernel iteration count. This is a load-bearing point because the advertised streamlining of the workflow depends on minimizing user intervention.

minor comments (7)

[Section 3.1, Eq. (5)] The derivation of the incremental weight formula could be made more explicit by showing the cancellation of the backward kernel ratio after substituting the time-reversal definition; the current text moves from the Radon-Nikodym identity to Eq. (5) with a brief "we obtain" that some readers will find hard to follow.
[Section 2.2.3] The notation p_{-(g,t+1:T_g)}(y*_{t+1:T_g}, Θ) is overloaded and difficult to parse; please define the subscript and the conditioning more carefully, for instance by explicitly stating which observations are deleted and which are treated as future replications.
[Algorithm 1] In the pseudocode, the line "Solve nℓ ∈ (nℓ−1, Nk]" does not reflect the LEO case where the solver should stop at the next predetermined checkpoint rather than at Nk; the text in Section 3.3.1 mentions this but the algorithm would be clearer if it included the checkpoint in the notation.
[Section 3.3.1] The paper refers to an "ESS ratio threshold of 0.5" without clearly defining whether this is ESS/R or an absolute ESS value; please define the threshold as a ratio to the number of particles R to avoid ambiguity.
[Figure 2] The caption describes blocks as "gray" and "white", but these colors may be difficult to distinguish in print; please add explicit labels or a legend to the figure.
[References] There is a typo in the reference list: the title of Gelfand and Dey (1994) reads "ealculations" instead of "calculations".
[Section 4.2.2] Figure 6 appears to contain two panels (cumulative and running average), but the caption does not label them; please add panel labels to aid interpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported CV estimates are weighted posterior expectations benchmarked against brute-force MCMC, and the adaptive path is chosen by ESS/Pareto-k rather than by fitting the target scores.

full rationale

The paper's derivation chain is self-contained. The baseline and target unnormalized posteriors (gamma_0 and gamma_k) are both defined from the same joint model, and the intermediate distributions in Sections 3.2.1 and 3.2.2 are power-scaled versions of the likelihood contributions. The incremental weight in Eq. (5), w_{k,ell}(Theta_{ell-1}) = gamma_{k,ell}(Theta_{ell-1}) / gamma_{k,ell-1}(Theta_{ell-1}), follows algebraically from the Radon-Nikodym and time-reversal construction in Section 3.1; no estimand is used to define it. The adaptive bridge in Section 3.3.1 selects the next case-deletion parameter n using only the effective sample size and, in Section 3.3.2, the Pareto-k diagnostic; it never optimizes against the log predictive density quantities that are subsequently reported. The reported quantities are therefore not fitted constants renamed as predictions: they are importance-weighted averages of user-chosen target functions f_g(Theta) evaluated at the resulting particles. The empirical comparisons in Figures 3, 4, 6, and 8 use brute-force MCMC re-runs as an external reference, so the close agreement with aSMC cannot be an artifact of the method being equivalent to its own inputs. Citations to prior work are used for standard ingredients (PSIS, SMC samplers, power-scaling) rather than to import an unverified uniqueness theorem that would force the paper's choice of path. The only flagged gap is in Section 3.3.1, where the text states that ESS decreases in the case-deletion parameter and then recommends bisection or Brent's method; monotonicity of ESS in n is asserted, not proved. If violated in multimodal targets, adaptive bridging could fail, but this is a correctness and robustness concern, not a circularity: the monotonicity claim concerns a numerical solver, and it is not an estimand defined in terms of the paper's own output. The manuscript itself acknowledges in Section 5 that some tuning parameters, such as the number of MCMC kernel iterations, remain manual, which further indicates that the workflow is not packaged as a self-justifying prediction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard SMC theory, conditional independence of observations, evaluable likelihoods, and two practical assumptions that are not proven in the paper: ESS monotonicity in the deletion parameter and the availability of efficient invariant kernels for intermediate power-scaled targets. The user-set computational parameters (ESS threshold, k-hat threshold, number of kernel iterations, particle count) are tuning constants rather than fitted scientific parameters.

free parameters (4)

ESS threshold ratio = 0.5
Used in Algorithm 1 to stop the automatic path construction; chosen as 'typical' in Section 4.
Pareto k-hat threshold = 0.7
Determines whether the final deletion step is handled by PSIS or particle rejuvenation; follows Vehtari et al. (2024).
Invariant kernel iterations = 1-3 HMC, 5 for Gibbs
Set manually by the user; Section 5 notes the choice affects the quality/runtime trade-off and that optimality is unclear.
Baseline particle count R = 1000
Number of posterior draws used as initial particles in all examples; influences Monte Carlo accuracy.

assumptions (5)

standard math Standard sequential Monte Carlo sampler consistency results (Del Moral et al. 2006; Dai et al. 2022) apply to the weighted particle approximations.
Section 3.1 builds on SMC samplers, time-reversal kernels, and the identity in Eq. (4).
domain assumption Conditional independence of observations within groups holds, allowing importance weights to factor into a product of per-observation likelihood reciprocals.
Used in Sections 2.2.2, 2.2.4 and 3.2 to derive w_g = product p(y_g,i | theta_g, phi)^-1.
ad hoc to paper Effective sample size decreases monotonically as the case-deletion parameter n increases, so bisection root-finding can select the next intermediate distribution.
Section 3.3.1 states 'as ESS decreases in n' without proof; the adaptive bridge construction depends on this monotonicity.
domain assumption An MCMC kernel invariant to each intermediate tempered posterior is available and can be run for a small fixed number of iterations.
Algorithm 1 invokes the kernel at each intermediate step; Section 3.2.1 notes that existing baseline-tuned kernels may become unsuitable.
domain assumption The conditional likelihoods p(y | theta, phi) can be evaluated pointwise for all observations.
Section 2.1 assumes this; it is needed for both the importance weights and the predictive scores.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adaptive sequential Monte Carlo for structured cross validation in Bayesian hierarchical models." pith.science (2026). https://pith.science/paper/6ECNUFYL

@misc{pith2026250107685,
  author       = {Pith},
  title        = {Pith review of: Adaptive sequential Monte Carlo for structured cross validation in Bayesian hierarchical models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ECNUFYL}},
  note         = {Machine review of arXiv:2501.07685}
}

abstract

Importance sampling (IS) is commonly used for cross validation (CV) in Bayesian models, because it only involves reweighting existing posterior draws without needing to re-estimate the model by re-running Markov chain Monte Carlo (MCMC). For hierarchical models, standard IS can be unreliable; the out-of-sample generalization hypothesis may involve structured case-deletion schemes which significantly alter the posterior geometry. This can force costly MCMC re-runs and make CV impractical. As a principled alternative, we tailor adaptive sequential Monte Carlo to sample along a path of posteriors that leads to the case-deleted posterior. The sampler is designed to support various hypotheses by accommodating diverse CV designs, and to streamline the workflow by automating path construction and systematically minimizing MCMC intervention. We demonstrate its utility with three types of predictive model assessment: longitudinal leave-group-out CV, group $K$-fold CV, and sequential one-step-ahead validation.

Figures

Figures reproduced from arXiv: 2501.07685 by the authors.

**Figure 2.** Diagrammatic comparison of three strategies for approximate [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

**Figure 3.** Hierarchical example. Comparison of posterior distribution function approximations of LGO log predictive likelihood. Note: Displayed in the top row are the three most populous groups in the data; the bottom row shows the three least. The solid line corresponds to the MCMC approximation. The dotted vertical line indicates the point at which the PSIS approximation ends. size. Note that we are taking the logarithm; t… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Left: Error comparison for LGO log predictive likelihood estimates, benchmarked against the brute-force strategy of re-running MCMC for each group (county). Right: Realized trajectories of distributions automatically determined by aSMC. Note: Annotated numbers denote…

**Figure 5.** Figure 5: Comparison of run time. Counties are ordered by MCMC run time. The num [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

**Figure 6.** Figure 6: Time-series example. Comparison of log predictive likelihood (LPL) approximation. Both are computed backwards as observations are sequentially deleted backwards. The leftmost value corresponds to the estimate (a) using the least amount of in-sample data points from th…

**Figure 7.** Figure 7: Top: Intermediate distributions adaptively determined by the aSMC sampler; the power coefficient on the corresponding pseudo-latest likelihood contribution is indicated. The (red) dashed line represents the backward running average LPL for aSMC as in [PITH_FULL_IMAGE…

**Figure 8.** Figure 8: Spatial example. Summary of group K-fold results. Left: Relative error of the log predictive likelihood approximations. Lower relative error indicates that the estimates were closer to the second-best strategy of re-running MCMC to re-approximate the posterior. Center…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 68 canonical work pages

[1]

A., Lachos, V

Abanto-Valle, C. A., Lachos, V. H., and Ghosh, P. (2012). A B ayesian approach to term structure modeling using heavy-tailed distributions. Applied Stochastic Models in Business and Industry , 28(5):430--447

work page 2012
[2]

T., Lenzi, A., Liu, Z., Martínez-Minaya, J., and Rue, H

Adin, A., Krainski, E. T., Lenzi, A., Liu, Z., Martínez-Minaya, J., and Rue, H. (2024). Automatic cross-validation in structured models: Is it time to leave out leave-one-out? Spatial Statistics , 62:100843

work page 2024
[3]

Agapiou, S., Papaspiliopoulos, O., Sanz-Alonso, D., and Stuart, A. M. (2017). Importance sampling: Intrinsic dimension and computational cost . Statistical Science , 32(3):405--431

work page 2017
[4]

and Greco, L

Agostinelli, C. and Greco, L. (2013). A weighted strategy to handle likelihood uncertainty in Bayesian inference. Computational Statistics , 28(1):319–339

work page 2013
[5]

and Celisse, A

Arlot, S. and Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys , 4:40--79

work page 2010
[6]

Beskos, A., Jasra, A., Kantas, N., and Thiery, A. (2016). On the convergence of adaptive sequential Monte Carlo methods. Annals of Applied Probability , 26(2):1111--1146

work page 2016
[7]

Bornn, L., Doucet, A., and Gottardo, R. (2010). An efficient computational approach for prior sensitivity analysis and cross-validation. Canadian Journal of Statistics , 38(1):47--64

work page 2010
[8]

Broderick, T., Giordano, R., and Meager, R. (2023). An automatic finite-sample robustness metric: When can dropping a little data make a big difference? https://arxiv.org/abs/2011.14999

arXiv 2023

Show all 72 references

[9]

Bürkner, P.-C., Gabry, J., and Vehtari, A. (2020). Approximate leave-future-out cross-validation for bayesian time series models. Journal of Statistical Computation and Simulation , 90(14):2499--2523

2020
[10]

Bürkner, P.-C., Gabry, J., and Vehtari, A. (2021). Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models . Computational Statistics , 36(2):1243--1261

2021
[11]

D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A

Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017). Stan: A probabilistic programming language . Journal of Statistical Software , 76(1):1–32

2017
[12]

C., Li, X., Xu, S., Yao, H.-R., Porcino, J., and Chow, C

Chang, J. C., Li, X., Xu, S., Yao, H.-R., Porcino, J., and Chow, C. (2024). Gradient-flow adaptive importance sampling for Bayesian leave one out cross-validation with application to sigmoidal classification models. https://arxiv.org/abs/2402.08151

2024
[13]

Cornebise, J., Moulines, E., and Olsson, J. (2008). Adaptive methods for sequential importance sampling with application to state space models. Statistics and Computing , 18:461--480

2008
[14]

E., and Whiteley, N

Dai, C., Heng, J., Jacob, P. E., and Whiteley, N. (2022). An invitation to sequential Monte Carlo samplers . Journal of the American Statistical Association , 117(539):1587--1600

2022
[15]

Del Moral, P., Doucet, A., and Jasra, A. (2006). Sequential Monte Carlo samplers . Journal of the Royal Statistical Society: Series B , 68(3):411--436

2006
[16]

Del Moral, P., Doucet, A., and Jasra, A. (2012). On adaptive resampling strategies for sequential monte carlo methods. Bernoulli , 18(1)

2012
[17]

Diebold, F. X. and Li, C. (2006). Forecasting the term structure of government bond yields . Journal of Econometrics , 130(2):337--364

2006
[18]

N., and Peruggia, M

Epifani, I., MacEachern, S. N., and Peruggia, M. (2008). Case-deletion importance sampling estimators: Central limit theorems and related results. Electronic Journal of Statistics , 2

2008
[19]

and Hardouvelis, G

Estrella, A. and Hardouvelis, G. (1991). The term structure as a predictor of real economic activity . Journal of Finance , 46(2):555--76

1991
[20]

and Wright, J

Faust, J. and Wright, J. H. (2013). Forecasting inflation. In Elliott, G. and Timmermann, A., editors, Handbook of Economic Forecasting , volume 2 of Handbook of Economic Forecasting , pages 2--56. Elsevier

2013
[21]

Geisser, S. (1975). The predictive sample reuse method with applications . Journal of the American Statistical Association , 70(350):320--328

1975
[22]

and Eddy, W

Geisser, S. and Eddy, W. F. (1979). A predictive approach to model selection . Journal of the American Statistical Association , 74(365):153--160

1979
[23]

Gelfand, A. E. and Dey, D. K. (1994). Bayesian model choice: Asymptotics and exact ealculations . Journal of the Royal Statistical Society. Series B , 56(3):501--514

1994
[24]

Gelman, A., Carlin, J., Stern, H., Dunson, D., Vehtari, A., and Rubin, D. (2014a). Bayesian Data Analysis . Chapman and Hall/CRC, 3 edition

2014
[25]

and Hill, J

Gelman, A. and Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models . Cambridge University Press

2006
[26]

Gelman, A., Hwang, J., and Vehtari, A. (2014b). Understanding predictive information criteria for Bayesian models. Statistics and Computing , 24(6):997–1016

2014
[27]

and Meng, X.-L

Gelman, A. and Meng, X.-L. (1998). Simulating normalizing constants: From importance sampling to bridge sampling to path sampling . Statistical Science , 13(2):163--185

1998
[28]

C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., and Modrák, M

Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., and Modrák, M. (2020). Bayesian workflow . https://arxiv.org/abs/2011.01808

2020 arXiv
[29]

D., Deshpande, S., and Broderick, T

Ghosh, S., Stephenson, W., Nguyen, T. D., Deshpande, S., and Broderick, T. (2020). Approximate cross-validation for structured models . In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33,...

2020
[30]

and Raftery, A

Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and Eestimation . Journal of the American Statistical Association , 102(477):359--378

2007
[31]

Guttman, I. (1967). The use of the concept of a future observation in goodness-of-fit problems . Journal of the Royal Statistical Society. Series B , 29(1):83--100

1967
[32]

and Kim, D

Hamilton, J. and Kim, D. H. (2002). A Reexamination of the Predictability of Economic Activity Using the Yield Spread . Journal of Money, Credit and Banking , 34(2):340--60

2002
[33]

Hoffman, M. D. and Gelman, A. (2014). The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo . Journal of Machine Learning Research , 15(47):1593--1623

2014
[34]

Y., Burt, D

Huang, J. Y., Burt, D. R., Nguyen, T. D., Shen, Y., and Broderick, T. (2024). Approximations to worst-case data dropping: Unmasking failure modes . https://arxiv.org/abs/2408.09008

2024 arXiv
[35]

and Nemoto, K

Hukushima, K. and Nemoto, K. (1996). Exchange Monte Carlo method and application to spin glass simulations . Journal of the Physical Society of Japan , 65(6):1604–1608

1996
[36]

A., Doucet, A., and Tsagaris, T

Jasra, A., Stephens, D. A., Doucet, A., and Tsagaris, T. (2011). Inference for Lévy-driven stochastic volatility models via adaptive sequential Monte Carlo . Scandinavian Journal of Statistics , 38(1):1--22

2011
[37]

Kallioinen, N., Paananen, T., Bürkner, P.-C., and Vehtari, A. (2023). Detecting and diagnosing prior and likelihood sensitivity with power-scaling. Statistics and Computing , 34(1)

2023
[38]

S., and Wong, W

Kong, A., Liu, J. S., and Wong, W. H. (1994). Sequential Imputations and Bayesian missing data problems . Journal of the American Statistical Association , 89(425):278--288

1994
[39]

Koop, G., Korobilis, D., and Pettenuzzo, D. (2019). Bayesian compressed vector autoregressions. Journal of Econometrics , 210(1):135--154

2019
[40]

Laurini, M. P. and Hotta, L. K. (2010). Bayesian extensions to Diebold-Li term structure model . International Review of Financial Analysis , 19(5):342--350

2010
[41]

Li, L., Qiu, S., Zhang, B., and Feng, C. X. (2015). Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC . Statistics and Computing , 26(4):881–897

2015
[42]

and Rue, H

Liu, Z. and Rue, H. (2023). Leave-group-out cross-validation for latent Gaussian models. https://arxiv.org/abs/2210.04482

2023 arXiv
[43]

G., Fonseca, T

Lobo, V. G., Fonseca, T. C., and Moura, F. A. (2020). Bayesian cross-validation of geostatistical models. Spatial Statistics , 35:100394

2020
[44]

Makridakis, S., Spiliotis, E., and Assimakopoulos, V. (2022). M5 accuracy competition: Results, findings, and conclusions . International Journal of Forecasting , 38(4):1346--1364

2022
[45]

C., Hoffman, M

Margossian, C. C., Hoffman, M. D., Sountsov, P., Riou-Durand, L., Vehtari, A., and Gelman, A. (2024). Nested R : Assessing the convergence of Markov chain Monte Carlo When running many short chains . Bayesian Analysis

2024
[46]

and Parisi, G

Marinari, E. and Parisi, G. (1992). Simulated tempering: A new Monte Carlo scheme. Europhysics Letters , 19(6):451–458

1992
[47]

and Rue, H

Martínez-Minaya, J. and Rue, H. (2024). A flexible Bayesian tool for CoDa mixed models: Logistic-normal distribution with Dirichlet covariance. Statistics and Computing , 34(3)

2024
[48]

C., Furr, D., and Rabe-Hesketh, S

Merkle, E. C., Furr, D., and Rabe-Hesketh, S. (2019). Bayesian comparison of latent variable models: Conditional versus marginal likelihoods . Psychometrika , 84(3):802–829

2019
[49]

Millar, R. B. (2018). Conditional vs.\ marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation. Statistics and Computing , 28(2):375–385

2018
[50]

Neal, R. M. (1993). Probabilistic Inference Using Markov Chain Monte Carlo Methods . Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto

1993
[51]

Neal, R. M. (2001). Annealed importance sampling. Statistics and Computing , 11:125--139

2001
[52]

D., Giordano, R., Meager, R., and Broderick, T

Nguyen, T. D., Giordano, R., Meager, R., and Broderick, T. (2024). Sensitivity of MCMC -based analyses to small-data removal. https://arxiv.org/abs/2408.07240

2024 arXiv
[53]

Paananen, T., Bürkner, P., Vehtari, A., and Gabry, J. (2024). Leave-one-out cross-validation for non-factorizable models. https://mc-stan.org/loo/articles/loo2-non-factorizable.html

2024
[54]

Paananen, T., Piironen, J., Bürkner, P.-C., and Vehtari, A. (2021). Implicitly adaptive importance sampling . Statistics and Computing , 31(2)

2021
[55]

and Casella, G

Park, T. and Casella, G. (2008). The Bayesian lasso . Journal of the American Statistical Association , 103(482):681--686

2008
[56]

Peruggia, M. (1997). On the variability of case-deletion importance sampling weights in the Bayesian linear model . Journal of the American Statistical Association , 92(437):199--207

1997
[57]

and Vehtari, A

Piironen, J. and Vehtari, A. (2016). Comparison of Bayesian predictive methods for model selection . Statistics and Computing , 27(3):711–735

2016
[58]

N., Nero, A., and Gelman, A

Price, P. N., Nero, A., and Gelman, A. (1996). Bayesian prediction of mean indoor radon concentrations for Minnesota counties. Health Physics , 71(6):922--936

1996
[59]

R., Bahn, V., Ciuti, S., Boyce, M

Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F., and Dormann, C. F. (2017). Cross-validation strategies for data with temporal, spatia...

2017
[60]

Roberts, H. V. (1965). Probabilistic prediction. Journal of the American Statistical Association , 60(309):50--62

1965
[61]

Silva, L. A. and Zanella, G. (2023). Robust leave-one-out cross-validation for high-dimensional Bayesian models . Journal of the American Statistical Association , 119(547):2369–2381

2023
[62]

Stone, M. (1976). Cross‐Validatory Choice and Assessment of Statistical Predictions . Journal of the Royal Statistical Society, Series B , 36(2):111--133

1976
[63]

Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion . Journal of the Royal Statistical Society. Series B , 39(1):44--47

1977
[64]

and Wang, J.-S

Swendsen, R. and Wang, J.-S. (1986). Replica Monte Carlo simulation of spin-glasses . Physical Review Letters , 57:2607--2609

1986
[65]

Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC . Statistics and Computing , 27:1413--1432

2017
[66]

Vehtari, A., Mononen, T., Tolvanen, V., Sivula, T., and Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models . Journal of Machine Learning Research , 17(103):1--38

2016
[67]

and Ojanen, J

Vehtari, A. and Ojanen, J. (2012). A survey of bayesian predictive methods for model assessment, selection and comparison. Statistics Surveys , 6:142--228

2012
[68]

Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2024). Pareto smoothed importance sampling . Journal of Machine Learning Research , 25(72):1--58

2024
[69]

Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory . Journal of Machine Learning Research , 11(116):3571--3594

2010
[70]

and Harrison, P

West, M. and Harrison, P. J. (1997). Bayesian Forecasting and Dynamic Models . Springer, 2nd edition

1997
[71]

J., and Migon, H

West, M., Harrison, P. J., and Migon, H. S. (1985). Dynamic generalised linear models and Bayesian forecasting (with discussion) . Journal of the American Statistical Association , 80(389):73--97

1985
[72]

J., Li, C., and Bao, L

Zhang, A., Daniels, M. J., Li, C., and Bao, L. (2024). Approximate cross-validated mean estimates for Bayesian hierarchical regression models . Journal of Computational and Graphical Statistics

2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

A., Lachos, V

Abanto-Valle, C. A., Lachos, V. H., and Ghosh, P. (2012). A B ayesian approach to term structure modeling using heavy-tailed distributions. Applied Stochastic Models in Business and Industry , 28(5):430--447

work page 2012

[2] [2]

T., Lenzi, A., Liu, Z., Martínez-Minaya, J., and Rue, H

Adin, A., Krainski, E. T., Lenzi, A., Liu, Z., Martínez-Minaya, J., and Rue, H. (2024). Automatic cross-validation in structured models: Is it time to leave out leave-one-out? Spatial Statistics , 62:100843

work page 2024

[3] [3]

Agapiou, S., Papaspiliopoulos, O., Sanz-Alonso, D., and Stuart, A. M. (2017). Importance sampling: Intrinsic dimension and computational cost . Statistical Science , 32(3):405--431

work page 2017

[4] [4]

and Greco, L

Agostinelli, C. and Greco, L. (2013). A weighted strategy to handle likelihood uncertainty in Bayesian inference. Computational Statistics , 28(1):319–339

work page 2013

[5] [5]

and Celisse, A

Arlot, S. and Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys , 4:40--79

work page 2010

[6] [6]

Beskos, A., Jasra, A., Kantas, N., and Thiery, A. (2016). On the convergence of adaptive sequential Monte Carlo methods. Annals of Applied Probability , 26(2):1111--1146

work page 2016

[7] [7]

Bornn, L., Doucet, A., and Gottardo, R. (2010). An efficient computational approach for prior sensitivity analysis and cross-validation. Canadian Journal of Statistics , 38(1):47--64

work page 2010

[8] [8]

Broderick, T., Giordano, R., and Meager, R. (2023). An automatic finite-sample robustness metric: When can dropping a little data make a big difference? https://arxiv.org/abs/2011.14999

arXiv 2023