{"id":"803abb9a-4018-4e51-8029-8386cee43718","arxiv_id":"2501.07685","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adaptive sequential Monte Carlo with automatically constructed intermediate posteriors approximates structured leave-group, leave-subset, and leave-end-out cross-validation without full MCMC reruns.","lead":"The paper introduces an adaptive sequential Monte Carlo sampler that automatically builds a bridge from the full-data posterior to case-deleted posteriors, so structured cross-validation in Bayesian hierarchical models can be approximated without repeated MCMC runs. A generalist reader may care because it turns a computationally expensive model-checking step into a faster, largely automatic workflow with code and demonstrations on radon, yield-curve, and retail-sales data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ESS monotonicity in n is unproven and the automatic path construction can fail in multimodal targets; this is the load-bearing assumption behind the paper's central 'automatically constructs a sequence of auxiliary intermediate distributions' claim.","rationale":"The reader's verdict is CONDITIONAL and its weakest assumption is exactly the ESS monotonicity assumption in Section 3.3.1. My independent reading confirms this is the most load-bearing and least-supported step. The paper's central claim is that the sampler 'automatically constructs a sequence of auxiliary intermediate distributions' with no user input; the automatic construction is implemented precisely by solving for n such that ESS hits a threshold, assuming ESS decreases in n. The three application examples are unimodal / conditionally conjugate and do not stress this assumption. The concern does not invalidate the empirical demonstrations, but it does weaken the generalization claim in the abstract and Section 1. The concrete test (fine-grid ESS(n) on a multimodal target) would settle whether the monotonicity assumption is reliable; if it fails, the paper needs a safeguarded adaptive scheme (e.g., direct ESS evaluation at grid points or a bisection that also checks local monotonicity). This is a moderate, addressable limitation, so the verdict remains CONDITIONAL rather than REJECT. I agree with the reader's identification of the same weakest point.","tokens_in":19710,"tokens_out":1719,"duration_ms":16229,"concrete_test":"Run the adaptive bridging procedure from Section 3.3.1 on a deliberately multimodal target, e.g., a two-component Gaussian mixture model with a case-deletion scheme that removes a discriminating observation, and record ESS as a function of the power/continuum parameter n. Plot ESS(n) on a fine grid and check whether it is monotone decreasing. Then run the paper's bisection/Brent root-finding with threshold 0.5 and compare the resulting intermediate distributions with the grid-computed ESS; if the root-finder lands at an n past a non-monotone bump and the SMC weight variance or k-hat exceeds acceptable limits, the monotonicity assumption fails in a regime relevant to the paper's automation claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central automation claim depends on the adaptive bridging step in Section 3.3.1, which solves for the next case-deletion parameter n by requiring that ESS meets a threshold. The text states 'as ESS decreases in n' and invokes bisection or Brent's method, but provides no proof or monotonicity argument. For geometric power-scaled posteriors (Section 3.2.1), ESS is a one-function of n, not necessarily monotone: as n increases, the weighted average of likelihood ratios can be non-monotone in multimodal or non-log-concave targets. If ESS is non-monotone, bisection on the interval (n_old, N] can converge to a root that is not the first crossing of the threshold, potentially skipping a wide temperature gap and causing weight collapse or an invalid bridge. The same issue arises for ordered continuous deletions (Section 3.2.2). This concern is not merely theoretical: the authors' own examples use HMC/Gibbs kernels that are initialized from resampled particles; if the intermediate chain is too far, particle diversity can collapse. The paper does not test this against a multimodal or non-log-concave example, and the three applications are all reasonably unimodal (hierarchical linear, DNS with conditionally conjugate structure, MVN/IW). The claim that the sampler is 'applicable to a wide range of models' (Section 1) therefore rests on this unproven monotonicity. The reader's weakest assumption identified the same issue; the manuscript explicitly flags no missing proof for this step, but the statement 'as ESS decreases in n' is used without proof. The concern is real but addressable: add a monotonicity check or a safeguarded search that verifies ESS is actually decreasing between consecutive accepted n values.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces an adaptive sequential Monte Carlo (aSMC) sampler for approximating case-deleted posteriors in structured cross-validation for Bayesian hierarchical models. The authors define several structured case-deletion schemes (leave-group-out, leave-end-out, leave-subset-out including group K-fold), parameterize continuous paths between the baseline posterior and the target case-deleted posterior via likelihood power scaling, and adaptively choose intermediate distributions using an effective sample size (ESS) threshold and a Pareto-k diagnostic. The sampler is designed to reuse a single baseline MCMC run and to minimize additional MCMC re-runs. The method is illustrated on three real-data examples (radon county-level LGO, Dynamic Nelson-Siegel sequential LEO, and M5 retail group K-fold CV), where it is compared against brute-force MCMC and standard PSIS. The paper claims that the sampler automatically constructs the bridging sequence and yields approximations close to MCMC at lower computational cost.","tokens_in":20164,"tokens_out":4821,"duration_ms":49279,"significance":"If the results hold, the method would be a practically useful extension of existing importance-sampling-based cross-validation tools (e.g., PSIS) to structured hierarchical models with distant case-deleted posteriors. The paper explicitly benchmarks against brute-force MCMC and provides reproducible Julia code and data, which is a notable strength. The core idea of adaptively bridging to case-deleted posteriors along a continuous path is natural and well motivated, and the three examples cover interesting CV designs. However, the central 'automatic' path construction relies on an unproven monotonicity assumption on the ESS, and the empirical evaluation does not report uncertainty in either the approximations or the MCMC reference. These issues need to be addressed before the method's advertised generality and accuracy can be fully accepted.","major_comments":[{"comment":"The adaptive path construction solves for the next case-deletion parameter n such that the ESS meets a threshold, relying on the statement \"as ESS decreases in n\". No proof or regularity condition is given for this monotonicity, and it is not guaranteed for general power-scaled likelihoods, especially in multimodal or non-log-concave targets. If the ESS is non-monotone, a bisection or Brent root-finder may converge to a root beyond the first threshold crossing, producing an excessively large bridge step and potentially causing weight collapse or an invalid intermediate distribution. Please either prove monotonicity under stated conditions (e.g., log-concavity) or replace the unguarded root-finding with a safeguarded step-halving procedure with a diagnostic, and discuss when non-monotonicity can occur. This assumption is load-bearing because the paper's claim of automatically constructing the sequence of intermediate distributions depends directly on it.","section":"Section 3.3.1"},{"comment":"The accuracy comparisons against MCMC report only point estimates (absolute error or relative error) without accounting for Monte Carlo variability in either the brute-force MCMC reference or the aSMC approximation. Consequently, the statement that aSMC produces approximations \"highly close\" to MCMC is not fully supported; the observed differences could be within sampling noise, particularly for groups with small Ng or for early time points in the LEO example. Please report standard errors or credible intervals for the estimands, for instance across independent SMC runs or by using multiple particle replicates, and explicitly state the magnitude of Monte Carlo error in the reference MCMC estimates.","section":"Section 4 (Figures 3-8)"},{"comment":"The manuscript acknowledges that the number of MCMC kernel iterations per rejuvenation must be set manually (1-3 for HMC, 5 for Gibbs), but this parameter is not chosen adaptively and can strongly affect the cost-accuracy trade-off. This undercuts the abstract's claim of \"automating path construction and systematically minimizing MCMC intervention\". Please clarify precisely which components of the workflow are automatic and which remain user-specified, and discuss possible data-driven heuristics or sensitivity analyses for the kernel iteration count. This is a load-bearing point because the advertised streamlining of the workflow depends on minimizing user intervention.","section":"Section 5 and Algorithm 1"}],"minor_comments":[{"comment":"The derivation of the incremental weight formula could be made more explicit by showing the cancellation of the backward kernel ratio after substituting the time-reversal definition; the current text moves from the Radon-Nikodym identity to Eq. (5) with a brief \"we obtain\" that some readers will find hard to follow.","section":"Section 3.1, Eq. (5)"},{"comment":"The notation p_{-(g,t+1:T_g)}(y*_{t+1:T_g}, Θ) is overloaded and difficult to parse; please define the subscript and the conditioning more carefully, for instance by explicitly stating which observations are deleted and which are treated as future replications.","section":"Section 2.2.3"},{"comment":"In the pseudocode, the line \"Solve nℓ ∈ (nℓ−1, Nk]\" does not reflect the LEO case where the solver should stop at the next predetermined checkpoint rather than at Nk; the text in Section 3.3.1 mentions this but the algorithm would be clearer if it included the checkpoint in the notation.","section":"Algorithm 1"},{"comment":"The paper refers to an \"ESS ratio threshold of 0.5\" without clearly defining whether this is ESS/R or an absolute ESS value; please define the threshold as a ratio to the number of particles R to avoid ambiguity.","section":"Section 3.3.1"},{"comment":"The caption describes blocks as \"gray\" and \"white\", but these colors may be difficult to distinguish in print; please add explicit labels or a legend to the figure.","section":"Figure 2"},{"comment":"There is a typo in the reference list: the title of Gelfand and Dey (1994) reads \"ealculations\" instead of \"calculations\".","section":"References"},{"comment":"Figure 6 appears to contain two panels (cumulative and running average), but the caption does not label them; please add panel labels to aid interpretation.","section":"Section 4.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of stat.CO and is likely to be of interest to practitioners of Bayesian hierarchical modeling. The code and data are provided, which is good practice. I would encourage the editor to ask for a revision that rigorously addresses the ESS monotonicity issue and adds uncertainty quantification to the empirical comparisons. The manual kernel-iteration setting, while acknowledged, should also be framed more carefully relative to the automation claims. With those changes the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2501.07685 (Han and Gelman). The paper proposes adaptive SMC to approximate structured cross-validation (LGO, group K-fold, LEO) in Bayesian hierarchical models, starting from one baseline MCMC run. The building blocks are all prior work — SMC samplers, adaptive tempering, PSIS, Pareto-k diagnostics — but the combination is new and genuinely useful: it automates the path sequence across a broader family of case-deletion schemes, and it embeds the \"rerun MCMC only when needed\" heuristic into a principled SMC framework. Three real-data examples (radon, yield curves, M5 sales) benchmark against brute-force MCMC; aSMC tracks MCMC where PSIS degrades, and code is public. That is real evidence.\n\nThe main caveat is exactly the one your reader flags: the automatic bridge in Section 3.3.1 assumes ESS decreases monotonically in the deletion parameter n, with no proof. In regular, roughly log-concave posteriors this is likely to hold; in multimodal or non-log-concave targets it can fail, and bisection could jump past a useful intermediate. The paper doesn't test that regime, so the \"wide range of models\" claim in Section 1 outruns the evidence. That said, this is an addressable gap — a monotonicity check or safeguarded search would fix it — and it doesn't undermine the empirical results presented.\n\nTwo lesser points: the number of MCMC kernel iterations is still manually set (the paper says so in Section 5), and there is no repeated-run uncertainty quantification on the aSMC estimates, so we don't see how noisy the approximation is. Both are minor relative to the central contribution.\n\nAll in, I read this as a solid within-subfield improvement: practitioners doing structured CV in hierarchical models get a cheaper workflow with accuracy close to rerunning MCMC. The reader's CONDITIONAL verdict is about right. I'd send it to peer review; it deserves a serious referee. And I'd cite it if we work on Bayesian workflow or efficient CV.","headline":"A practical and mostly sound adaptive-SMC package for structured CV in hierarchical models; the unproven ESS monotonicity assumption is a real but addressable gap.","tokens_in":20608,"tokens_out":2027,"would_cite":true,"duration_ms":21074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adaptive sequential Monte Carlo can approximate structured cross-validation estimands in Bayesian hierarchical models—leave-group-out, group K-fold, and backward sequential leave-end-out—from a single baseline…","keywords":["cross validation","Bayesian hierarchical models","sequential Monte Carlo","adaptive SMC","leave-group-out CV","leave-end-out CV","importance sampling","predictive model assessment"],"falsifier":"A concrete test would be to run the sampler on a Bayesian hierarchical model with a mixture or otherwise multimodal posterior, and check whether the effective sample size as a function of the deletion parameter is non-monotone. If it is, the adaptive root-finding for the next intermediate distribution will pick a parameter value that does not actually meet the ESS threshold, visible as a degenerate resampling step or a stall in the loop.","tokens_in":19484,"feed_emoji":"🧮","tokens_out":6797,"duration_ms":58174,"temperature":0.7,"pith_summary":"The paper is trying to establish that structured cross validation in Bayesian hierarchical models need not require re-estimating the posterior once per deleted group or subset. It claims an adaptive sequential Monte Carlo sampler can walk from the full-data posterior through automatically chosen intermediate distributions to each case-deleted posterior, reweighting and occasionally rejuvenating a single batch of baseline MCMC draws. If true, this makes predictive checks such as leave-group-out, group K-fold, and backward sequential leave-end-out practical for hierarchical and spatial-temporal models, where standard importance sampling is unstable and brute-force MCMC re-runs are too slow. The payoff is a Bayesian workflow where model comparison and out-of-sample diagnostics are by-products of one initial fit.","feed_headline":"Adaptive SMC matches MCMC accuracy for structured cross-validation","feed_subtitle":"Leave-group-out, group K-fold, and sequential leave-end-out from a single baseline run, without MCMC re-runs.","key_machinery":"The central object is the adaptive sequential Monte Carlo sampler over an augmented space of posterior draws, with incremental importance weight $w_{k,\\ell}(\\Theta_{\\ell-1}) = \\gamma_{k,\\ell}(\\Theta_{\\ell-1})/\\gamma_{k,\\ell-1}(\\Theta_{\\ell-1})$ computed at the previous particle positions. Case deletions are encoded as continuous paths: either a geometric (power-scaled) tempering path $\\rho_g(n) = p(y_g \\mid \\theta_g,\\phi)^{\\varphi_g(n)}$ for dropping whole groups, or ordered within-group factors $\\varphi_{g,i}(n) = \\min\\{\\max\\{0, i-n\\},1\\}$ for sequential end-deletion. The mechanism that carries the argument is the automatic construction of intermediate distributions: given a monotone decrease of effective sample size in the deletion parameter, bisection or Brent's method solves for the next parameter value that keeps ESS at a target level, so the user never specifies the path.","core_discovery":"The central claim is that case-deletion paths between the baseline and case-deleted posteriors can be parameterized continuously—by power-scaling likelihood contributions or by ordered within-group deletion—and that the adaptive SMC sampler can discover the bridging distributions automatically. At each step the sampler chooses the next deletion parameter so that the effective sample size of the incremental importance weights meets a target threshold, invoking a Markov kernel (an MCMC move targeting an intermediate distribution) only when the Pareto shape diagnostic indicates the importances are unreliable. The result is an approximation of the log predictive density of each structured CV scheme that closely tracks brute-force MCMC in the paper's three real-data examples, while largely avoiding repeated long MCMC runs.","pith_inferences":["Editorial inference: the same adaptive bridging could be applied to likelihood power-scaling sensitivity analysis, replacing the manual grids currently used in prior and likelihood sensitivity checks.","Editorial inference: for models where conditional independence fails, the ordered within-group deletion path is not available; a covariance-aware or Cholesky-parameterized deletion path would be a natural extension.","Editorial inference: the paper reports empirical accuracy against MCMC but does not give a worst-case bound on the number of intermediate distributions; a complexity analysis relating the number of steps to the $\\chi^2$ divergence between baseline and target would sharpen the practical claims."],"forward_implications":["Leave-group-out, group K-fold, and backward sequential leave-end-out predictive densities can be computed from a single baseline MCMC run with accuracy close to brute-force re-runs.","The workflow removes the manual design of tempering schedules: the path of intermediate distributions is chosen by the ESS criterion, with the Markov kernel invoked only when PSIS diagnostics indicate instability.","The same machinery covers multiple CV designs, and it degrades gracefully to plain Pareto-smoothed importance sampling when the deletion is mild.","Because rejuvenation steps are independent across particles and across folds, the method parallelizes, making it substantially faster than re-running MCMC for each fold or group."],"supporting_citations":[{"why":"Supplies the sequential Monte Carlo sampler framework with forward and backward Markov kernels that the method adapts for bridging posteriors.","marker":"Del Moral et al., 2006"},{"why":"The Pareto-smoothed importance sampling and leave-one-out cross-validation baseline that the method extends to structured deletions.","marker":"Vehtari et al., 2017"},{"why":"A non-adaptive SMC approach for LOO-CV and prior sensitivity that the paper generalizes to adaptive bridging for structured CV.","marker":"Bornn et al., 2010"},{"why":"Defines the effective sample size criterion used to automatically select intermediate distributions.","marker":"Kong et al., 1994"},{"why":"Provides convergence justification for adaptive SMC, supporting the bisection and Brent root-finding used in path construction.","marker":"Beskos et al., 2016"},{"why":"Supplies the generalized Pareto $\\hat{k}$ diagnostic and Pareto-smoothed importance sampling used to decide when the MCMC kernel is needed.","marker":"Vehtari et al., 2024"},{"why":"Introduces approximate leave-future-out cross-validation and the idea of selective MCMC re-runs, extended here into a unified sampler.","marker":"Bürkner et al., 2020"},{"why":"Provides the power-scaling of likelihood contributions used as a tempering parameterization for the bridging path.","marker":"Kallioinen et al., 2023"},{"why":"Gives central limit theory for case-deletion importance sampling estimators, motivating the instability that adaptive SMC addresses.","marker":"Epifani et al., 2008"},{"why":"Defines the leave-one-cluster-out (LGO) cross-validation estimands that the method computes in its hierarchical example.","marker":"Merkle et al., 2019"}],"fun_headline_variants":["Adaptive SMC skips MCMC re-runs for hierarchical CV","One baseline run powers all structured CV checks","Adaptive SMC automates deletion paths for CV","Structured cross-validation without MCMC reruns","SMC samples deletion paths for fast hierarchical CV"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sampler's automatic path construction assumes that the effective sample size of the importance weights falls monotonically as more data are deleted, which is stated without proof in the paper (Section 3.3.1); if this monotonicity fails, as it can in multimodal or strongly non-log-concave targets, the adaptive bridge may select invalid intermediate distributions or fail to converge.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive SMC skips MCMC re-runs for hierarchical CV","One baseline run powers all structured CV checks","Adaptive SMC automates deletion paths for CV","Structured cross-validation without MCMC reruns","SMC samples deletion paths for fast hierarchical CV"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1421,"prompt_tokens":843,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":459,"tokens_out":578,"duration_ms":6066,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:43.510056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to run the sampler on a Bayesian hierarchical model with a mixture or otherwise multimodal posterior, and check whether the effective sample size as a function of the deletion parameter is non-monotone. If it is, the adaptive root-finding for the next intermediate distribution will pick a parameter value that does not actually meet the ESS threshold, visible as a degenerate resampling step or a stall in the loop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sequential Monte Carlo sampler framework with forward and backward Markov kernels that the method adapts for bridging posteriors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Pareto-smoothed importance sampling and leave-one-out cross-validation baseline that the method extends to structured deletions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A non-adaptive SMC approach for LOO-CV and prior sensitivity that the paper generalizes to adaptive bridging for structured CV."},{"cited_title":"S., and Wong, W","cited_arxiv_id":null,"evidence_quote":"Defines the effective sample size criterion used to automatically select intermediate distributions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides convergence justification for adaptive SMC, supporting the bisection and Brent root-finding used in path construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the generalized Pareto $\\hat{k}$ diagnostic and Pareto-smoothed importance sampling used to decide when the MCMC kernel is needed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the power-scaling of likelihood contributions used as a tempering parameterization for the bridging path."},{"cited_title":"N., and Peruggia, M","cited_arxiv_id":null,"evidence_quote":"Gives central limit theory for case-deletion importance sampling estimators, motivating the instability that adaptive SMC addresses."},{"cited_title":"C., Furr, D., and Rabe-Hesketh, S","cited_arxiv_id":null,"evidence_quote":"Defines the leave-one-cluster-out (LGO) cross-validation estimands that the method computes in its hierarchical example."}],"review_version":1}