REVIEW 3 major objections 4 minor 16 references
Sequential Monte Carlo samplers can estimate intractable gradients accurately enough to replace expensive MCMC inner loops in stochastic optimisation, with linear convergence for the idealized recursion.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Sequential Monte Carlo samplers can approximate intractable gradients inside a first-order optimizer, yielding a general SOSMC framework that speeds up reward tuning of energy-based models in the reported settings.
T0 review reviewed 2026-08-03 challenge →
load-bearing objection SOSMC is a clean umbrella over existing SMC-for-optimization methods with honest experiments, but the theory only covers the idealized recursion and the gap to the closed-loop algorithm is admitted, not closed. the 3 major comments →
Efficient Stochastic Optimisation via Sequential Monte Carlo
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper claims that the intractable gradient—an expectation under a parameter-dependent distribution—can be approximated by the SMC-weighted particle average g_k = Σ_i w_k^(i) H_{θ_k}(X_k^(i)), maintained across optimisation steps via a Feynman–Kac identity. With exact expectations, this weighted recursion recovers the true gradient and reduces the update to exact gradient descent, converging linearly at rate (1−γμ)^t under Polyak-Łojasiewicz and smoothness assumptions. The paper argues this motivates the SOSMC algorithm and notes existing particle-based optimisers as special cases. Empirically, on reward tuning of energy-based models, the weighted-particle method outperforms unweighted pe
What carries the argument
The central machinery is the Feynman–Kac identity for SMC samplers (Theorem 1), which expresses the intractable expectation π_{θ_k}(H_{θ_k}) as an importance-weighted average over a sequence of proposals, with incremental weights G_k(x_{k-1}, x_k) = Π_{θ_k}(x_k) L_{k-1}(x_k, x_{k-1}) / (Π_{θ_{k-1}}(x_{k-1}) K_k(x_{k-1}, x_k)). The algorithm (SOSMC, Algorithm 1) maintains N weighted particles across gradient-descent iterations: at each step, particles are propagated through a Markov kernel K_k, reweighted, possibly resampled when the effective sample size falls below a threshold, and the gradient estimate g_k is the normalized weighted average of H_{θ_k}. The idealized analysis (Lemma 1 and P
Load-bearing premise
The practical efficiency claim relies on the unproven assumption that standard SMC approximation errors stay controlled when the target distributions are themselves moved by the particle-weighted gradient estimates, so the optimisation loop does not amplify bias.
What would settle it
Run SOSMC on a small latent-variable model where exact gradients are available by numerical integration and check whether the bias of g_k decays like 1/N and the parameter path tracks exact gradient descent as N grows; if the bias accumulates with outer iterations or fails to vanish, the central claim is false.
If this is right
- If SMC-based gradient estimates are accurate enough, MMLE, EBM training, and reward tuning can replace expensive MCMC inner loops with a single SMC sweep per optimisation step, cutting compute substantially.
- The idealized linear rate (1−γμ)^t means that whenever the loss is Polyak-Łojasiewicz and the gradient estimates are near-exact, the outer optimisation inherits fast convergence without inner-loop convergence requirements.
- The weighted-particle estimator reduces non-equilibrium bias, so persistent particles lag less behind the evolving target; consequently, reliable gradient estimates can be obtained with far shorter (or no) fresh evaluation chains.
- The framework subsumes several existing particle-based optimisers as special cases, giving a unified way to reason about their design (choice of forward/backward kernels, resampling threshold, step size).
- The ESS-based adaptive step-size rule offers a practical, scale-free way to keep successive targets close enough to avoid weight degeneracy during optimisation.
Where Pith is reading between the lines
- A direct test of the paper's logic would be to compare the closed-loop bias of g_k against high-accuracy MCMC gradients on a small problem; if the bias grows with the number of outer iterations rather than staying at the single-step SMC bias, the feedback loop is the weak point.
- The ESS formula implies that step size should shrink in directions of large gradient norm or high Fisher information; a curvature-aware adaptive step rule could outperform the simple multiplicative rule in Remark 4.
- The same construction—reusing weighted particles across parameter updates—should transfer to other optimisation problems of the form (2), such as policy-gradient estimation or stochastic variational inference, provided the kernels can be chosen tractably.
- If the feedback bias is benign, particle counts could be much lower than for fixed-target SMC, because the optimisation dynamics themselves average out particle noise; this is testable by running SOSMC at N=100 versus N=10000 and comparing final objective values.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SOSMC, a sequential Monte Carlo (SMC) framework for stochastic optimisation when the gradient of the loss is an intractable expectation under a parameter-dependent distribution. Algorithm 1 maintains a weighted particle system that is propagated with Markov kernels and periodically resampled; the weighted average (13) is used as a gradient estimate in a generic first-order optimiser. The theoretical section analyses an idealised version of the recursion in which Feynman–Kac expectations are computed exactly: Lemma 1 shows exact gradient recovery and Proposition 2 gives a (1−γμ)^t rate under a Polyak–Łojasiewicz condition. Section 4.2 discusses the effective sample size, with an exact Gaussian formula and local expansions of the χ² divergence. Experiments on reward tuning of Langevin processes, 2D EBMs, and MNIST compare SOSMC-ULA with ImpDiff and SOUL, reporting wall-clock and iteration-level convergence.
Significance. If the practical algorithm inherited the idealised convergence behaviour, the framework would be a useful general alternative to MCMC inner loops for MMLE, EBM training, and reward tuning. The paper is clearly written and the empirical section is carefully controlled: device-synchronised timing, equal numbers of kernel applications per outer step, and separate reporting of particle-based and fresh rewards are genuine strengths. The surrogate-loss derivations in Appendix F are also a useful practical contribution. However, the main theoretical support for Algorithm 1 itself is absent: Remark 3 explicitly defers the particle analysis, and the accompanying ESS analysis contains a substantively incorrect proposition. The central efficiency claim is therefore conditional on an unproved closed-loop bias control.
major comments (3)
- [§4.1, Remark 3, Algorithm 1 (Eq. 13)] The convergence result in Proposition 2 applies to an idealised recursion in which expectations are computed exactly. In Algorithm 1, the gradient estimate g_k in Eq. (13) is computed from the current weighted particles, and the next parameter θ_{k+1} is a function of that same particle system. The target sequence (π_{θ_k}) is therefore not exogenous; Remark 4 concedes that it is “an optimisation-coupled random quantity.” Standard SMC O(1/N) bias/MSE results apply to Feynman–Kac flows with deterministic or prespecified target sequences, and the paper gives no bound on the accumulated feedback bias E[g_k] − ∇ℓ(θ_k) when θ_k itself depends on past particles. This is the load-bearing gap: without such a bound, the practical convergence claim for Algorithm 1 is unsupported. The deferral in Remark 3 does not resolve it, even though the abstract states that convergence results are established
- [Proposition 5 (Appendix B), with Proposition 3 and Eq. (19)] Proposition 5 is mathematically incorrect. It concludes χ²(π_{θ_k}, π_{θ_{k-1}}) = sqrt(γ² ∇ℓ^T I ∇ℓ) + O(γ^{3/2}), whose leading term is O(γ). This contradicts the exact Gaussian formula (19), where χ² = exp(γ² ∥∇ℓ∥²_{Σ^{-1}}) − 1 = O(γ²), and also contradicts Proposition 3 and Proposition 4, which give χ² = γ² ∇ℓ^T I ∇ℓ + o(γ²). The proof applies Cauchy–Schwarz to obtain an L¹-type upper bound; an upper bound of order γ cannot be an asymptotic expansion for a quantity that is O(γ²). The displayed square root is also dimensionally inconsistent as an equality. This proposition should be corrected or removed; as written, it undermines the claimed ESS analysis.
- [§4.2, Eq. (19), Remark 4] The ESS analysis in Section 4.2 assumes the parameter update is θ_k = θ_{k−1} − γ∇ℓ(θ_{k−1}) with a fixed step size and exact gradients. Algorithm 1, however, uses an arbitrary optimiser (including Adam and SGD) and a stochastic, biased gradient estimate g_k. Remark 4 then introduces an adaptive γ_k driven by the ESS itself. Consequently, Eqs. (18)–(19) and Proposition 3 do not quantify the ESS of the actual algorithm; they describe a much simpler idealised recursion. This is acceptable if the section is explicitly framed as intuition for the idealised flow, but the current presentation invites the reader to apply these conclusions to SOSMC directly. Please state clearly that these results are heuristic for the closed-loop algorithm.
minor comments (4)
- [Proof of Proposition 2] The proof refers to “Proposition 1” but the statement it uses is Lemma 1. Please correct the cross-reference.
- [Appendix B, Proposition 5 proof] There is a typo: “Cauchy-Swartz” should be “Cauchy–Schwarz.” More importantly, the proof alternates between t and γ inconsistently; this should be cleaned up regardless of the corrected statement.
- [Algorithm 2] The line “logW_k = logW_k − max_i logW_i” is ambiguous; the subtraction should be applied elementwise per particle. Please clarify the notation.
- [Abstract / Section 1] The abstract says “we establish convergence results for the basic recursions defined by our methodology which SMC samplers approximate.” This is accurate only for the idealised recursions, not the particle algorithm. Rephrase to avoid overclaiming.
Circularity Check
No significant circularity: the idealized Feynman-Kac identity is a standard exact identity, and the closed-loop gap is explicitly acknowledged rather than disguised.
full rationale
The central derivation chain is not circular. Theorem 1 is an external standard Feynman-Kac identity (Del Moral 2004), and Lemma 1 shows only that the importance-weight recursion (8)/(10) gives E[HW]/E[W] = pi(H); this is a consistency property of an importance-sampling construction, not a fitted prediction. Proposition 2 then invokes a standard gradient-descent convergence theorem under A1, and it is explicitly restricted to the idealized exact-expectation recursion (14)-(16). The paper does not claim the particle algorithm inherits these rates: Remark 3 defers a particle-level analysis to future work, and Remark 4 explicitly states that the target sequence is 'an optimisation-coupled random quantity' when theta is updated from the weighted particles. That is an honest, acknowledged gap (a correctness/rigor risk, not circularity). The self-citations to Cuin et al. (2025) and Carbone et al. (2024) are used to identify special cases and to match algebraic weight expressions in Appendix A; they are not load-bearing for the convergence claims, and Appendix A derives the weights directly. Separate issues such as the dimensional inconsistency in Proposition 5 and the missing 'Proposition 1' reference are mathematical/correctness concerns, not circularity. The score of 2 reflects only minor non-load-bearing self-citation and the acknowledged closed-loop gap, not derivation-equivalent-to-input circularity.
Axiom & Free-Parameter Ledger
free parameters (5)
- ESS resampling threshold τ =
0.9 (chosen)
- Adaptive threshold τ_adapt =
0.95 (chosen)
- Adaptive step-size multiplier c =
unspecified
- ULA step-size γ =
0.1 (Langevin exp.) or 5e-3 (EBM exp.), adapted
- Optimizer learning rate η =
0.1 / 2e-4 / 1e-4
axioms (6)
- domain assumption Gradient can be written as ∇ℓ(θ) = E_{X∼πθ}[Hθ(X)] with πθ(x) ∝ e^{-Uθ(x)} and Zθ < ∞.
- domain assumption A1: ℓ is μ-PL and L-smooth.
- standard math Forward/backward kernels K_n, L_{n-1} have densities so the incremental weights (8) are well-defined.
- ad hoc to paper Standard SMC error bounds (bias/MSE O(1/N)) extend to the closed-loop, algorithm-dependent target sequence (πθk).
- domain assumption Local bounded score/Hessian and exponential moment conditions for the χ² expansions.
- domain assumption ULA kernels mix sufficiently fast and importance weights have finite variance in the experiments.
Cite this review
Pith. "Pith review of Efficient Stochastic Optimisation via Sequential Monte Carlo." pith.science (2026). https://pith.science/paper/Y6IW5HIE
@misc{pith2026260122003,
author = {Pith},
title = {Pith review of: Efficient Stochastic Optimisation via Sequential Monte Carlo},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y6IW5HIE}},
note = {Machine review of arXiv:2601.22003}
}
read the original abstract
The problem of optimising functions with intractable gradients frequently arises in machine learning and statistics, ranging from maximum marginal likelihood estimation procedures to fine-tuning of generative models. Stochastic approximation methods for this class of problems typically require inner sampling loops to obtain (biased) stochastic gradient estimates, which rapidly becomes computationally expensive. In this work, we develop sequential Monte Carlo (SMC) samplers for optimisation of functions with intractable gradients. Our approach replaces expensive inner sampling methods with efficient SMC approximations, which can result in significant computational gains. We establish convergence results for the basic recursions defined by our methodology which SMC samplers approximate. We demonstrate the effectiveness of our approach on the reward-tuning of energy-based models within various settings.
Figures
Reference graph
Works this paper leans on
-
[1]
(2024, Proposition 1)
and Carbone et al. (2024, Proposition 1). B Proofs We give some additional results on the Chi-square distance betweenπθk andπ θk−1 as a function of the step sizeγin this section. First we present a result with a result with local boundedness assumptions on the gradient and the Hessian. Recall that χ2(πθk , πθk−1 ) = Z x∈Rdx πθk (x) πθk−1 (x) −1 2 πθk−1 (x...
2024
-
[12]
, K−1do w(i) k−1 = 1/N Computeg k−1(wk−1)as in (24) for forward-KL, or as in (25) for reverse-KL
fork= 1, . . . , K−1do w(i) k−1 = 1/N Computeg k−1(wk−1)as in (24) for forward-KL, or as in (25) for reverse-KL. θk = OPT(θk−1, gk−1(wk−1)) fori= 1, . . . , Ndo X (i) k =X (i) k−1 −γ∇ x Eθk−1 (X (i) k−1) + √2γ σ ε(i) k−1,whereε (i) k−1 ∼ N(0, I). end for end for Returntuned parametersθ K, and optionally history of other quantities. E Experimental Details ...
2025
-
[13]
Regarding initialisation, weights are drawn fromN(0, σ2 N ), withσ N = 0.02, and biases are initialised to zero
after each hidden layer, defined byσ(z) =z /(1 + exp(−z)), to help ensure∇ xEθ(x)is differentiable everywhere, for the Langevin dynamics. Regarding initialisation, weights are drawn fromN(0, σ2 N ), withσ N = 0.02, and biases are initialised to zero. T raining Objective.We train the energy model by minimising the negative log-likelihood (NLL) of the data,...
2002
-
[15]
Reward F unctions.Here, we consider indicator rewards, for half-planes, of the form Rleft =1 {x1<0}, R right =1 {x1>0}, R lower =1 {x2<0}, R upper =1 {x2>0},(36) since not only do these rewards highlight thatR(x)may be non-differentiable, but these rewards also admit a closed form for the unique maximiser of (35). A standard variational argument shows tha...
2008
-
[16]
Although the second term is indeed zero here, weleavethisinthecovarianceformtomatchMarionetal.(2025)andtherespectivecodeimplementation
Now, we note that ∇θEπθ [R(X)] =E πθ [R(X)∇ θ logπ θ(X)] =−E πθ [R(X)∇ θV(X, θ)] +Eπθ [R(X)]E πθ [∇θV(X, θ)].(46) where, in the second equality, we have substituted in (45). Although the second term is indeed zero here, weleavethisinthecovarianceformtomatchMarionetal.(2025)andtherespectivecodeimplementation. Now, sinceπ ref does not depend onθ, we have th...
2025
-
[32]
Pre-training details.As was the case for in the setting of 2D datasets, pre-training of the EBM proceeds viaPCD, albeit with a sampler that differs from our tuning kernel
To emphasise the dimensionality of the problem, recall inputs to this network are single-channel images,x∈R 28×28×1. Pre-training details.As was the case for in the setting of 2D datasets, pre-training of the EBM proceeds viaPCD, albeit with a sampler that differs from our tuning kernel. Since many details are the same, we refer to Appendix E.2.1 for the ...
2025
-
[2003]
Handbook of convergence theorems for (stochastic) gradient methods.arXiv preprint arXiv:2301.11235,
Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods.arXiv preprint arXiv:2301.11235,
-
[2008]
ISSN 1935-8237. doi: 10.1561/ 2200000001. URLhttps://doi.org/10.1561/2200000001. Sinan Yildirim, Sumeetpal S Singh, Tom Dean, and Lan Jiang. A monte carlo expectation maximisation algorithm for multiple target tracking. In2012 15th International Conference on Information Fusion, pages 2094–2101. IEEE,
-
[2010]
13 Paul Felix Valsecchi Oliva, O Deniz Akyildiz, and Andrew Duncan. Uniform-in-time convergence bounds for persistent contrastive divergence algorithms.arXiv preprint arXiv:2510.01944,
-
[2012]
14 Equivalently, the log-incremental weight is logG k(xk−1, xk) =−U θk (xk) +U θk−1 (xk−1) − 1 4h ∥xk−1 −x k +h∇U θk (xk)∥2 + 1 4h xk −x k−1 +h∇U θk−1 (xk−1) 2
A Derivation of the weights for the ULA kernel When the forward kernel is chosen as theULAkernel with step-sizeh >0: Kk(xk−1, xk) = (4πh)−d/2 exp − 1 4h ∥xk −x k−1 +h∇U θk−1 (xk−1)∥2 , and Lk−1(xk, xk−1) = (4πh)−d/2 exp − 1 4h ∥xk−1 −x k +h∇U θk (xk)∥2 , the incremental weight in (8) becomes Gk(xk−1, xk) = Πθk (xk)L k−1(xk, xk−1) Πθk−1 (xk−1)K k(xk−1, xk)...
2025
-
[2015]
Non-asymptotic analysis of biased stochastic approximation scheme
Belhal Karimi, Blazej Miasojedow, Eric Moulines, and Hoi-To Wai. Non-asymptotic analysis of biased stochastic approximation scheme. InConference on Learning Theory, pages 1944–1974. PMLR,
1944
-
[2017]
Christoph Schönle, Davide Carbone, Marylou Gabrié, Tony Lelièvre, and Gabriel Stoltz
URL https://arxiv.org/abs/1710.05941. Christoph Schönle, Davide Carbone, Marylou Gabrié, Tony Lelièvre, and Gabriel Stoltz. Efficient monte-carlo sampling of metastable systems using non-local collective variable updates.arXiv preprint arXiv:2512.16812, 2025a. Christoph Schönle, Marylou Gabrié, Tony Lelièvre, and Gabriel Stoltz. Sampling metastable system...
-
[2020]
Gersende Fort and Eric Moulines
URL https://arxiv.org/abs/1903.08689. Gersende Fort and Eric Moulines. Convergence of the monte carlo expectation maximization for curved exponential families.The Annals of Statistics, 31(4):1220–1259,
Pith/arXiv arXiv 1903
-
[2021]
Training restricted boltzmann machines using approximations to the likelihood gra- dient
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gra- dient. InProceedings of the 25th International Conference on Machine Learning, ICML ’08, page 1064–1071, New York, NY, USA, 2008a. Association for Computing Machinery. ISBN 9781605582054. doi: 10.1145/1390156.1390290. URLhttps://doi.org/10.1145/1390156.1390...
-
[2023]
Implicit diffusion: Efficient optimization through stochastic sampling
PierreMarion, AnnaKorba, PeterBartlett, MathieuBlondel, ValentinDeBortoli, ArnaudDoucet, Felipe Llinares-López, Courtney Paquette, and Quentin Berthet. Implicit diffusion: Efficient optimization through stochastic sampling. InInternational Conference on Artificial Intelligence and Statistics, pages 1999–2007. PMLR,
1999
-
[2025]
Joanna Marks, Tim YJ Wang, and O Deniz Akyildiz. Learning latent energy-based models via interacting particle langevin dynamics.arXiv preprint arXiv:2510.12311,
This paper was first reviewed by deepseek-v4-flash on August 3, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.