Pith. sign in

REVIEW 4 major objections 5 minor 30 references

Training a neural sampler and a sequential Monte Carlo particle filter in a closed loop samples hard multi-modal target distributions better than either method alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:03 UTC pith:MPVYBK2O

load-bearing objection A solid, well-tested empirical paper that couples SMC as an off-policy behavior policy with importance-weighted replay for training neural samplers; the theory is mostly reframing, but the combination is new and works on the benchmarks, with caveats about stability, tempering bias, and missing code. the 4 major comments →

arxiv 2510.11711 v3 pith:MPVYBK2O submitted 2025-10-13 cs.LG stat.ML

Reinforced sequential Monte Carlo for amortised sampling

Sanghyeok Choi , Sarthak Mittal , V\'ictor Elvira , Jinkyoo Park , Esmeralda S. Whitammer This is my paper
classification cs.LG stat.ML MSC 65C0560J22
keywords sequential Monte Carloamortised samplingoff-policy reinforcement learningtrajectory balancediffusion samplersimportance samplingexperience replayunnormalised density sampling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that amortised neural samplers and sequential Monte Carlo (SMC) are two sides of the same object, and that they can be trained in a closed loop. The learned sampler supplies SMC with proposal kernels and intermediate target densities; SMC, in turn, supplies the sampler with off-policy trajectories that have explored the target better than the sampler could on its own. To make the loop practical, the paper adds an importance-weighted replay buffer and an adaptive tempering scheme that keep training-signal variance bounded. If correct, this gives a recipe for sampling from unnormalised densities in continuous, discrete, and molecular settings with better mode coverage than on-policy training alone.

Core claim

The central claim is that the log-importance weights accumulated by SMC/AIS are the same quantities that appear in the trajectory-balance and subtrajectory-balance training losses of the amortised sampler. A sampler trained to zero those losses is, by construction, an optimal SMC proposal, and its learned flow functions become the optimal intermediate targets. The paper then closes the loop by using SMC with the current sampler as proposal to generate target-exploring trajectories and training the sampler on those SMC samples off-policy. Empirically, this beats both on-policy diffusion samplers and SMC with manually chosen proposals and targets on multi-modal synthetic benchmarks and on the

What carries the argument

The central object is the identity between the logarithm of the sequential importance weight and the squared log-ratio in the trajectory-balance (TB) and subtrajectory-balance (SubTB) objectives. TB matches the full-trajectory flow against the target density times the backward kernel; SubTB does the same for every segment and introduces learned flow functions as intermediate targets. This identity means optimising the sampler's loss is also optimising the SMC procedure's proposal and intermediate distributions, and vice versa. Around this identity, the paper builds an adaptive weight-tempering scheme and an importance-weighted experience replay buffer whose batch-level weights are particle e

Load-bearing premise

The mutual-improvement loop is assumed to be self-reinforcing and stable: SMC trajectories generated by the current imperfect sampler are informative enough to train it, and the trained sampler improves the next SMC proposal even before convergence.

What would settle it

A concrete check is to run the full method on a target whose modes are separated by near-zero density and measure whether early SMC-discovered modes keep appearing in the replay buffer with high weight as training proceeds; if effective sample size stays near 1 across many epochs and buffer diversity does not rise, the closed loop has failed to discover new modes.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On gradient-free diffusion benchmarks, the proposed SMC and IW-Buf variants improve EUBO and Sinkhorn distance over on-policy TB, log-variance, and DDS, indicating the off-policy SMC data mitigates mode collapse.
  • Using the learned flows as intermediate targets is necessary for stability: fixed linear or cosine interpolation schedules with SMC training diverge on complex targets, while learned flows keep weights balanced.
  • In the gradient-based setting, the combined SMC + IW-Buf system is competitive or better on Funnel, Robot4, GMM40, MoS, and ManyWell; the SMC-only variant without the buffer is unstable on Robot4 and GMM40.
  • For discrete sequence targets (QM9, sEH, TFbind8, L14-RNA1), importance-weighted replay improves EUBO and the Pearson correlation between model log-probability and reward over uniform, reward-prioritised, and loss-prioritised buffers.
  • On alanine dipeptide, the full SMC + IW-Buf method gives ELBO -180.3 and EUBO -166.0, where several baselines failed with numerical errors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same SMC-as-behaviour-policy coupling should apply to any off-policy objective whose loss can be written as a second moment of log-importance weights; the paper's construction is not wedded to the specific TB/SubTB losses.
  • Because batch-level buffer weights estimate the normalising constant, the importance-weighted replay scheme could transfer to other off-policy RL settings where past episodes come from different policies and need to be reweighted by quality.
  • A natural stress test is to push the loop to higher-dimensional molecular targets: if SMC degeneracy worsens with dimension, the adaptive tempering and learned flows may need to be made stronger to keep the mutual-improvement cycle intact.
  • Targeting SMC at the exact target density may not be optimal for exploration; the paper itself points to loss-aware targets that would direct particles toward regions where the current sampler is most wrong.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework that combines amortised samplers (e.g., diffusion samplers, GFlowNets) with sequential Monte Carlo (SMC) and annealed importance sampling (AIS). The authors establish connections between hierarchical variational inference, maximum-entropy RL objectives (TB/SubTB), and SMC/AIS sample weights. They introduce an off-policy training scheme in which SMC uses the current learned sampler as a proposal and learned flow functions as intermediate targets, and the resulting SMC trajectories are used to train the sampler. They also propose an importance-weighted replay buffer that weights historical batches by estimated normalizing constants. Experiments on synthetic continuous targets, alanine dipeptide, and discrete sequence design report improvements over on-policy baselines and alternative buffer prioritisation schemes.

Significance. If the claimed results hold, this is a useful contribution to amortised sampling, bridging Monte Carlo and learned samplers in a principled way. The paper provides a clear derivation of the replay-buffer weights (Eq. 13) and extensive experiments with ablations, multiple metrics, and 5 seeds. The central algorithmic idea—using SMC as an off-policy behaviour policy and weighting replay batches by estimated normalising constants—is well motivated. However, the evidence is strongest for the full combination of SMC and IW-Buf; the SMC component alone is unstable on several targets, and the adaptive tempering introduces an acknowledged but unquantified bias. These issues make the broad claims in the abstract and Section 1 somewhat stronger than what the experiments demonstrate.

major comments (4)
  1. [§3.4, Algorithm 6, Fig. 6] Adaptive weight tempering biases both SMC resampling and replay-buffer sampling when λ<1. The paper acknowledges this bias but does not quantify its effect on the learned sampler. The motivation in §3.2 relies on SMC yielding samples that approximate the target; with tempering, the behaviour policy is deliberately flattened. Fig. 6 shows that γ is a delicate hyperparameter: too high flattens the weights, too low causes instability. This makes the practical operating regime a heuristic compromise. Please either provide a theoretical justification for why training on tempered off-policy trajectories still minimises the intended objective, or add an experiment on stable low-dimensional tasks comparing the final sampler at γ=0 (unbiased) versus the chosen γ=0.05.
  2. [Table 2, §3.2] The SMC-only variant ('TB/SubTB + SMC') is unstable or markedly worse in the gradient-based setting: Robot4 MMD is 0.778±0.339 and GMM40 (d=50) is reported as '×'. Only when combined with IW-Buf does the method achieve strong results. The abstract and Introduction claim that 'the MC method benefits from the learnt components of the amortised sampler', but the experiments indicate this synergy is not realised for SMC alone. The authors should either reframe the contribution as a combined SMC+replay system or characterise the conditions under which SMC alone is beneficial. This is load-bearing because the paper's central mutual-cycle narrative depends on it.
  3. [§3.1, Eq. (8), Algorithm 2/4] The statement that the TB loss 'equals second moment of the log-AIS weights' is only exact when trajectories are sampled from the on-policy forward process p0 pθ. When training with SMC-generated trajectories, the expectation in the loss is taken under a different distribution, which also involves tempered resampling and learned intermediate flows. While TB is a valid off-policy objective, this particular interpretation linking it to log-AIS weight variance no longer holds for the SMC batches. The paper does not clarify how this affects the theoretical motivation for using TB on SMC trajectories. A short note on the intended optimisation target under the off-policy mixture would strengthen the paper.
  4. [§3.3, Algorithm 4] The replay buffer combines batches from different model versions, each with its own batch-level weight \(\hat Z_m\). The adaptive tempering in Algorithm 4 (line 10) is applied to the entire buffer as if it were a single importance sample. This is not justified by the group-importance-sampling argument used for Eq. (13). The resulting sampling distribution over history is a heuristic that may over-emphasise stale high-weight samples. Please clarify whether this is intentional and discuss potential effects on training stability.
minor comments (5)
  1. [Algorithm 7, line 14] The symbol \(\bar\)\(w_N\) is not explicitly defined in the main text; please state that it is the final weight used for the replay buffer and relate it to the normalized weights \(W_N\) and \(\hat Z\).
  2. [§F.2] The text mentions 'IW-ELBO' but no IW-ELBO results are reported in the tables. Either report the metric or remove the mention.
  3. [Table 1] The row label 'SMC-R WM' likely should be 'SMC-RWM' to match standard terminology; please check for consistency.
  4. [§3.4] The sentence 'thereby mitigating the instability' in the ablation discussion would benefit from a formal statement about what 'instability' means (e.g., loss divergence, ESS collapse).
  5. [§H, Table 7] The high variance of 'TB/SubTB + SMC + IW-Buf' on ALDP (e.g., EUBO -166±0.8) is not commented on; a brief note on the source of this variance would help.

Circularity Check

0 steps flagged

No significant circularity: the fixed target R and external ground-truth evaluation anchor the derivation, and the SMC-as-behaviour-policy loop is an off-policy training scheme for a fixed objective, not a fitted identity.

full rationale

The derivation chain is anchored to an external, fixed target: the unnormalised density R(x) and the fixed reverse kernel ←p. The TB/SubTB losses (Eqs. 7–8) are defined against R and ←p, and their optimum enforces Zθ pθ(τ) = R(x)←p(τ|x), an external condition. SMC (Algorithm 7) uses the current pθ and Fφ only to generate off-policy training trajectories and weights for that same fixed loss; it is a behaviour policy, not the definition of the objective. The paper's key observation that the log-AIS weight (10) matches the TB objective (8) is a mathematical identity used to motivate the algorithm, not a prediction that reduces to a fit. The adaptive weight tempering of §3.4 is explicitly acknowledged as introducing bias ('this also introduces bias to our approximations'), and the instability of SMC-only variants (Tables 2–3, Fig. 6) is presented as empirical limitation; these are stability/correctness risks, not circularity. Evaluation uses ground-truth samples and external metrics (EUBO, MMD, Sinkhorn) computed on the trained sampler without SMC. Self-citations such as Malkin et al. (2022), Madan et al. (2023), and Sendera et al. (2024) support standard components and are peer-reviewed results, not load-bearing devices that forbid alternatives or define away the target. I therefore find no step where a claimed prediction is equivalent by construction to an input.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim is an empirical, method-level contribution. It relies on standard SMC/importance-sampling mathematics plus several user-chosen hyperparameters (γ, κ, I, L, σ), and on the unproven but empirically supported assumption that the SMC-amortised feedback loop is stable. No new physical entities are posited.

free parameters (5)
  • γ (adaptive tempering ESS threshold) = 0.05
    Controls how much importance weights are flattened; chosen from sensitivity analysis in Fig. 6. Too small can cause instability, too large dilutes importance weighting. Central to the method's stability claim.
  • κ (adaptive resampling ESS threshold) = 0.2
    Sets when SMC resampling is triggered; affects effective sample size and training stability; chosen by hand uniformly across experiments.
  • I (off-policy ratio) = 2
    Number of off-policy epochs per on-policy epoch; sensitivity analysis in Fig. 8 shows high values degrade performance; directly controls the mix of SMC/buffer data in training.
  • L (subtrajectory/chunk length) = 4
    Length of chunks for the proposed chunk-based SubTB and the SMC resampling interval; ablation in Fig. 7 shows smaller L generally better but more expensive; affects flow training and computational cost.
  • σ (diffusion noise scale) = target-dependent (20, 20, 1, 1, 1, 2, 40, 15, 1)
    Noise scale in the OU process for each target, taken from prior work. The choice differs between gradient-free and gradient-based settings and between targets, and strongly influences the difficulty of the sampling task.
axioms (5)
  • standard math Full support and absolute continuity of target and proposals (footnote 1)
    Needed for importance weights and KL divergences to be well-defined; standard in the sampling literature.
  • standard math SMC asymptotic correctness: particle approximation converges weakly to target as K→∞ (Del Moral et al., 2006)
    Invoked in §3.3 to justify that the weighted replay buffer is an approximating measure of π; this is a standard cited theorem.
  • standard math TB/SubTB losses enforce target matching at optimality when enforced on all trajectories with full-support behaviour policy (Malkin et al., 2022; Madan et al., 2023)
    Used throughout to justify that the trained sampler and flows, at zero loss, make the SMC proposal and intermediate targets optimal. This is an established result in the GFlowNet literature.
  • ad hoc to paper The SMC behaviour policy, built from the current imperfect sampler and learnt flows, has full support and yields a useful training signal
    This is the weakest assumption: the self-training loop is assumed to be beneficial without a convergence proof. The paper's own instability results (Tables 2 and 3) indicate this is only empirically true with the full recipe.
  • domain assumption Sufficient model capacity of the diffusion MLP and flow MLP
    The zero-loss optimum (exact target matching) requires the neural networks to be expressive enough; no capacity analysis is given.

pith-pipeline@v1.3.0-alltime-deepseek · 32996 in / 18282 out tokens · 155085 ms · 2026-08-04T10:03:57.685729+00:00 · methodology

0 comments
read the original abstract

This paper proposes a synergy of amortised and particle-based methods for sampling from distributions defined by unnormalised density functions. We state a connection between sequential Monte Carlo (SMC) and neural sequential samplers trained by maximum-entropy reinforcement learning (MaxEnt RL), wherein learnt sampling policies and value functions define proposal kernels and twist functions. Exploiting this connection, we introduce an off-policy RL training procedure for the sampler that uses samples from SMC -- using the learnt sampler as a proposal -- as a behaviour policy that better explores the target distribution. We describe techniques for stable joint training of proposals and twist functions and an adaptive weight tempering scheme to reduce training signal variance. Furthermore, building upon past attempts to use experience replay to guide the training of neural samplers, we derive a way to combine historical samples with annealed importance sampling weights within a replay buffer. On synthetic multi-modal targets (in both continuous and discrete spaces) and the Boltzmann distribution of alanine dipeptide conformations, we demonstrate improvements in approximating the true distribution as well as training stability compared to both amortised and Monte Carlo methods.

Figures

Figures reproduced from arXiv: 2510.11711 by Esmeralda S. Whitammer, Jinkyoo Park, Sanghyeok Choi, Sarthak Mittal, V\'ictor Elvira.

Figure 1
Figure 1. Figure 1: Visualisation of generated samples for GMM40 ( [PITH_FULL_IMAGE:figures/full_fig_p025_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualisation of generated samples for ManyWell ( [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualisation of generated samples for Robot4 ( [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualisation of generated samples for ManyWell ( [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualisation of intermediate marginals for GMM40 ( [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of γ of adaptive importance weight tempering. Here and in other figures, the error bars show the standard deviation from 5 runs [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Results obtained from different values of subtrajectory length [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Results obtained from different values of replay ratio [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Ramachandran plots generated with 100,000 samples drawn from samplers trained with each algorithm [PITH_FULL_IMAGE:figures/full_fig_p030_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 4 linked inside Pith

  1. [1]

    Albergo, M. S. and Vanden-Eijnden, E. (2025). NETS: A non-equilibrium transport sampler.International Conference on Machine Learning (ICML). Arenz, O., Zhong, M., and Neumann, G. (2020). Trust-region variational inference with Gaussian mixture models.Journal of Machine Learning Re- search (JMLR), 21(163):1–60. Bengio, E., Jain, M., Korablyov, M., Precup, ...

    arXiv 2025

  2. [2]

    Springer. Rust, J. (1987). Optimal replacement of GMC bus engines: An empirical model of Harold Zurcher. Econometrica: Journal of the Econometric Society, pages 999–1033. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized experience replay.International Conference on Learning Representations (ICLR). Sendera, M., Kim, M., Mittal, S., ...

    Pith/arXiv arXiv 1987

  3. [5]

    values are omitted. H APPLICATION TO ALANINE DIPEPTIDE We evaluate our methods for approximating the Boltzmann distribution of alanine dipeptide (ALDP), a 22-atom molecule in an implicit solvent at a temperature of 300K. This challenging molecular sampling task is one of the standard benchmarks for Boltzmann generators (No´ e et al., 2019; Wu et al., 2020...

    arXiv 2019

  4. [10]

    Loula, J., LeBrun, B., Du, L., Lipkin, B., Pasti, C., Grand, G., Liu, T., Emara, Y., Freedman, M., Eis- ner, J., Cotterell, R., Mansinghka, V., Lew, A

    Springer. Loula, J., LeBrun, B., Du, L., Lipkin, B., Pasti, C., Grand, G., Liu, T., Emara, Y., Freedman, M., Eis- ner, J., Cotterell, R., Mansinghka, V., Lew, A. K., Vieira, T., and O’Donnell, T. J. (2025). Syntactic and semantic control of large language models via sequential Monte Carlo.International Conference on Learning Representations (ICLR). Madan,...

    2025

  5. [13]

    (2023) (further developed in Zhang et al

    and that of GFlowNets in the continuous space in works including Lahlou et al. (2023) (further developed in Zhang et al. (2024); Sendera et al. (2024); Kim et al. (2025b)). A unifying perspective on all objectives and continuous-time limit analysis is provided in Berner et al. (2025). Various off-policy behavior policies for diffusion samplers have been p...

    2023

  6. [14]

    and an additional trained policy guided by the loss of the sampler (Kim et al., 2025b) show promise in this regard. Our simple and principled approach of using Monte Carlo methods for training trajectory selection is inspired by these works and shows further improvements in the training of diffusion samplers. Several works have explored the connection bet...

    2022

  7. [15]

    By setting the latent variables asx n =y tn , we can see that the diffusion models belong to the family of hierarchical latent variable models

    This yields a discrete-time Markov process with transition kernel: − →p θ(ytn+1 |ytn ) =N(y tn+1 ;y tn +u θ(ytn , tn)∆tn, σ2(tn)∆tnId),(17) where ∆tn =t n+1 −t n. By setting the latent variables asx n =y tn , we can see that the diffusion models belong to the family of hierarchical latent variable models. Note that we used a different discretisation schem...

    2023

  8. [16]

    4:forj= 1, . . . , N/Ldo 5:Sample subtrajectories from{x k (j−1)L}K k=1 using − →p θ ⇒ {τk (j−1)L:jL }K k=1 6:Update weights for each subtrajectories using ← −p, − →p θ andF ϕ ⇒ {xk jL, wk jL}K k=1.▷Equation (9) 7: ˆZjL ← ˆZ(j−1)L · PK k=1 wk jL. 8:if dESS {wk jL}K k=1 < κKandj < N/Lthen 9:{x k jL, wk jL}K k=1 ←Resampling({x k jL:jL , wk jL}K k=1, γ).▷Alg...

    2022

  9. [18]

    The Double Well energy is: EDW(x1, x2) =x 4 1 −6x 2 1 − 1 2 x1 + 1 2 x2

    is constructed as the product of d/2 independent copies of a 2-dimensional Double Well distribution (No´ e et al., 2019). The Double Well energy is: EDW(x1, x2) =x 4 1 −6x 2 1 − 1 2 x1 + 1 2 x2

    2019

  10. [19]

    (Arenz et al., 2020; Chen et al.,

    2020

  11. [20]

    We generate DNA sequences of length 8, where each token is a DNA nucleotide (A, G, C, or T)

    •TFbind8(|X |= 65,536). We generate DNA sequences of length 8, where each token is a DNA nucleotide (A, G, C, or T). The reward function is the binding affinity to a human transcription factor, approximated via a proxy model from (Trabucco et al., 2022). We useC=

    2022

  12. [21]

    Lett 2 be the density of Student’s t-distribution with degree 2 andν i be the shift of componenti

    is ad-dimensional mixture of Student’s t-distributions with 10 com- ponents. Lett 2 be the density of Student’s t-distribution with degree 2 andν i be the shift of componenti. Each dimension ofν i is randomly sampled fromU[−10,10]. The unnormalised density of MoS is defined as: RMoS(x) = 1 10 10X i=1 t2(x−ν i). F.2 Evaluation Metrics In this section, we i...

    2024

  13. [24]

    We adopt the visualisation method in Chen et al

    target in gradient-based setting. We adopt the visualisation method in Chen et al. (2025). Ground Truth SCLD w/o MCMC SCLD TB + IW-Buf TB/SubTB + SMC + IW-Buf Figure 4: Visualisation of generated samples for ManyWell (d=

    2025

  14. [26]

    Similar to QM9, we create string representations of small molecular graphs with 6 blocks, each from a predefined set of 18 building blocks with 2 stems

    •sEH(|X |= 34,012,224). Similar to QM9, we create string representations of small molecular graphs with 6 blocks, each from a predefined set of 18 building blocks with 2 stems. The reward is the binding affinity to soluble epoxide hydrolase (sEH), approximated using a pretrained model from Bengio et al. (2021). We set C=

    2021

  15. [28]

    We generate RNA sequences of length 14, where each token is an RNA nucleotide (A, G, C, or U)

    •L14-RNA1(|X |= 268,435,456). We generate RNA sequences of length 14, where each token is an RNA nucleotide (A, G, C, or U). The reward function is the binding affinity to a human transcription factor, approximated using a model from (Sinai et al., 2020). We setC=

    2020

  16. [40]

    Model architecture and hyperparameters.We implement prepend/append models (in§B.2) with edge- flow parameterisation following Shen et al. (2023). We use MLP as a backbone architecture, with a hidden dimension of 1,024 for chemical tasks and 128 for biological tasks. We use the Adam (Kingma and Ba,

    2023

  17. [79]

    Capp´ e, O., Guillin, A., Marin, J.-M., and Robert, C. P. (2004). Population monte carlo.Journal of Compu- tational and Graphical Statistics, 13(4):907–929. Cardoso, G., el idrissi, Y. J., Corff, S. L., and Moulines, E. (2024). Monte Carlo guided denois- ing diffusion models for bayesian linear inverse prob- lems.International Conference on Learning Repre...

    2004

  18. [401]

    Martino, L., Elvira, V., M´ιguez, J., Art´ es-Rodr´ιguez, A., and Djuri´ c, P. (2018b). A comparison of clipping strategies for importance sampling. In2018 IEEE Statistical Signal Processing Workshop (SSP), pages 558–562. IEEE. Maruyama, G. (1955). Continuous markov processes and stochastic equations.Rendiconti del Circolo Matematico di Palermo, 4:48–90. ...

    Pith/arXiv arXiv 1955

  19. [470]

    K., Zhi-Xuan, T., Grand, G., and Mans- inghka, V

    Lew, A. K., Zhi-Xuan, T., Grand, G., and Mans- inghka, V. K. (2023). Sequential Monte Carlo steer- ing of large language models using probabilistic pro- grams.arXiv preprint arXiv:2306.03081. Liu, J. S. and Liu, J. S. (2001).Monte Carlo strategies in scientific computing, volume

    Pith/arXiv arXiv 2023

  20. [812]

    Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal transport.Neural Informa- tion Processing Systems (NIPS). Cuturi, M., Meng-Papaxanthos, L., Tian, Y., Bunne, C., Davis, G., and Teboul, O. (2022). Optimal trans- port tools (ott): A jax toolbox for all things Wasser- stein.arXiv preprint arXiv:2201.12324. Del Moral, P., Doucet, A., a...

    Pith/arXiv arXiv 2013

  21. [2003]

    •ManyW ell(d∈[32,64]) (N¨ usken and Richter, 2021; Midgley et al.,

    is a funnel-shaped distribution, whose unnormalised density is defined as: RFunnel(x) =N(x 1; 0,32) 10Y i=2 N(x i; 0,exp(x1)), wherex= (x i)10 i=1. •ManyW ell(d∈[32,64]) (N¨ usken and Richter, 2021; Midgley et al.,

    2021

  22. [2012]

    Note that EUBO, MMD, and Sinkhorn distance calculations require unbiased samples from the target distribution

    for synthetic targets in continuous spaces. Note that EUBO, MMD, and Sinkhorn distance calculations require unbiased samples from the target distribution. For all metrics, we useK eval = 2000 samples. ELBO.The ELBO is defined as: ELBO =E τ=(x 0→···→xN =x)∼− →p θ log R(x)← −p(τ 0:N−1 |x) − →p θ(τ) , wherexis the final state of trajectoryτ. The ELBO is a lo...

    2000

  23. [2013]

    rely on the REINFORCE esti- mator (Williams, 1992), which is often enhanced with advanced control variates (Titsias and L´ azaro-Gredilla, 2014; Mnih and Gregor, 2014; Mnih and Rezende, 2016; Richter et al.,

    1992

  24. [2015]

    We do not apply learning rate scheduling

    optimizer with a learning rate of 0.0001 for policy − →p θ, and SGD with a learning rate of 0.1 and a momentum coefficient of 0.8 for logZ θ. We do not apply learning rate scheduling. We train for 3,000 epochs on QM9 and 6,000 epochs on other tasks, with a batch size ofK= 100 and a buffer size of 100,000. Other algorithm-specific hyperparameters follow th...

    2021

  25. [2017]

    and amortised variational inference. Since MaxEnt RL methods learn a stochastic policy that samples actions proportionally to the expected soft returnsat each state, they face a double-counting problem when multiple action sequences lead to the same object (Bengio et al., 2021), limiting their use as a general amortised inference method. GFlowNets effecti...

    2021

  26. [2020]

    Generative flow networks (GFlowNets; Bengio et al., 2021,

    to mitigate the high variance issue in REINFORCE. Generative flow networks (GFlowNets; Bengio et al., 2021,

    2021

  27. [2022]

    to compute the Sinkhorn distance, following Chen et al. (2025). MMD.We compute the MMD between batches ofK eval samples from the ground truth and from the trained sampler. Following Blessing et al. (2024) (§A.2), we use an exponential kernel with a heuristically determined scaling parameter. F.3 A Note on Evaluation Protocol In§4, we report the moving ave...

    2025

  28. [2023]

    GFlowNets bridge the gap between maximum entropy reinforcement learning (MaxEnt RL) algorithms (Haarnoja et al., 2017, 2018; Nachum et al.,

    were initially introduced as algorithms to learn policies that compositionally sample discrete objects proportionally to their rewards. GFlowNets bridge the gap between maximum entropy reinforcement learning (MaxEnt RL) algorithms (Haarnoja et al., 2017, 2018; Nachum et al.,

    2017

  29. [2024]

    These approaches provide valuable off-policy training examples that improve training efficiency and/or mode coverage

    and Monte Carlo exploration in the target space (Zhang et al., 2022; Kim et al., 2024b; Sendera et al., 2024; Kim et al., 2025b) as a guided exploration. These approaches provide valuable off-policy training examples that improve training efficiency and/or mode coverage. In the next section, we provide a detailed review of works on diffusion samplers for ...

    2022

  30. [2025]

    defines the distribution over joint configurations of a planar robot with 10 degrees of freedom. The target unnormalised density is defined as: RRobot4(x) =R conf(x)Rcart(x), whereR conf encourages smooth configurations by placing a Gaussian prior andR cart attracts the robot’s end-effector to one of four possible goal locations: Rconf(x) =N(x 1; 0,12) 10...

    2020