DIME:Diffusion-Based Maximum Entropy Reinforcement Learning

Daniel Palenicek; Denis Blessing; Ge Li; Georgia Chalvatzaki; Gerhard Neumann; Jan Peters; Onur Celik; Zechu Li

REVIEW 3 major objections 4 minor 5 cited by

DIME:Diffusion-Based Maximum Entropy Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims diffusion policies can be trained in maximum entropy RL by maximizing a variational lower bound on the intractable entropy, so each policy improvement step becomes a path-space KL fit to the energy policy $\exp(Q)/Z$.

desk verdict A genuinely new diffusion MaxEnt RL method with strong empirical work, but the advertised convergence theorem has a real proof gap that the authors need to fix or retract. read the letter →

arxiv 2502.02316 v2 pith:TCMMLHA7 submitted 2025-02-04 cs.LG

Onur Celik , Zechu Li , Denis Blessing , Ge Li , Daniel Palenicek , Jan Peters , Georgia Chalvatzaki , Gerhard Neumann This is my paper · ORCID

classification cs.LG

keywords maximumentropyreinforcementlearningdiffusionpoliciesvariationalinferencepolicyiterationenergy-basedcontinuouscontrolapproximatedenoisingmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Maximum entropy reinforcement learning rewards policies for acting randomly as well as for collecting reward, but its standard implementations use Gaussian policies because their entropy is available in closed form. This paper claims diffusion-model policies, which are far more expressive, can be used in the same framework: instead of computing the intractable marginal entropy of the diffusion policy, DIME maximizes a variational lower bound on it. Under this bound, each policy improvement step becomes a KL-divergence minimization between the denoising diffusion process and a noising process that ends in the energy-based policy $\exp(Q)/Z$. The paper further claims this policy iteration provably converges to the optimal policy within the diffusion class, and that the resulting algorithm outperforms all diffusion-based RL baselines on 13 continuous-control benchmarks while matching or beating Gaussian-policy state-of-the-art methods at a lower update-to-data ratio. If the claims hold, DIME gives MaxEnt-RL the expressive action distributions of diffusion models without adding Gaussian exploration noise.

What carries the argument

The load-bearing object is the path-space KL divergence between the forward noising process $\bar{\pi}_{0:N}(a_{0:N}|s)$ and the backward denoising process $\pi_{0:N}(a_{0:N}|s)$ over the discretized Ornstein-Uhlenbeck diffusion chain. Minimizing this KL aligns the two joint distributions, so the denoising chain ends in the target energy policy $\exp(Q_{\bar{\pi}}(s,a_0))/Z_{\bar{\pi}}(s)$; expanding it gives the tractable objective in Eq. 27, in which the partition function drops out and gradients flow to the score network through reparameterization. The same KL acts as the policy improvement operator in the iteration, which is why the lower-bound derivation in Eq. 15, not score matching, is the mechanism that makes entropy-based training of diffusion policies possible.

What would settle it

Take a small exactly solvable MDP (or a finely discretized finite MDP) in which the optimal diffusion policy and its Q-function can be computed to high precision directly from the path-space KL objective. Run DIME's policy iteration exactly, without neural approximation; if the Q-values do not increase monotonically or the iterates do not approach the computed optimal policy, the convergence claim is false. A sharper check targets a specific proof step: Proposition 4.2's derivation writes the target density as $\exp(Q(s,a_0))/Z$ but the displayed algebra replaces the action by $a_N$; testing whether $Q_{\pi_{\mathrm{new}}}\ge Q_{\pi_{\mathrm{old}}}$ survives in a two-action case where that replacement changes the inequality would settle the proof's validity.

Watch

Extended reading notes

Core claim

At the heart of the paper is the observation that the MaxEnt objective for a diffusion policy can be maximized indirectly. The marginal entropy $H(\bar{\pi}_0(a_0|s))$ of the denoising process is intractable, but it is bounded below by $\ell_{\bar{\pi}}(a_0,s)=E_{\bar{\pi}_{0:N}}\!\left[\log\big(\bar{\pi}_{1:N|0}(a_{1:N}|a_0,s)/\pi_{0:N}(a_{0:N}|s)\big)\right]$, where $\bar{\pi}_{0:N}$ is the discretized noising process and $\pi_{0:N}$ the denoising process. Substituting this lower bound into the maximum entropy objective turns the policy improvement step into the path-space KL problem $\arg\min_{\pi\in\bar{\Pi}} D_{\mathrm{KL}}(\pi_{0:N}(a_{0:N}|s)\,\|\,\bar{\pi}_{0:N}(a_{0:N}|s))$, with target marginal $\bar{\pi}_0(a_0|s)=\exp(Q_{\bar{\pi}}(s,a_0))/Z_{\bar{\pi}}(s)$. Unlike score matching, this objective can be optimized from evaluations of the Q-function alone, using reparameterized samples from the diffusion policy, so no samples from the energy policy are required. The paper states this gives a policy iteration scheme that converges monotonically to the optimal diffusion policy (Propositions 4.1-4.3), and the practical variant uses distributional Q-learning with batch renormalization while learning the temperature and the diffusion coefficient.

Load-bearing premise

The load-bearing assumption is that at every policy improvement step the parameterized diffusion policy can represent the target density $\exp(Q_{\bar{\pi}})/Z_{\bar{\pi}}$ closely enough that the KL minimization is effectively solved, and that the policy iteration guarantee, which the paper proves only for finite action spaces, still holds in the continuous action spaces used in the experiments.

Editorial extensions

If this is right

Diffusion policies become usable in online maximum entropy RL without score matching or samples from the energy policy, so the exploration distribution comes from the policy itself rather than from injected Gaussian noise.
The exploration-exploitation trade-off is controlled by the learned temperature and diffusion coefficient, so the method requires less manual tuning than diffusion-based baselines that must add noise for exploration.
The policy improvement step is a path-space inference problem, so any diffusion sampler that defines forward and backward joint distributions can be substituted into the same objective, as the paper's general-bridge extension suggests.
The convergence result, if valid, upgrades diffusion-based RL from a heuristic policy class to one with a monotone policy-iteration guarantee within the diffusion family.
On the paper's benchmarks, DIME outperforms all diffusion-based baselines on all 13 tasks and is competitive with the best Gaussian-policy methods at a much smaller update-to-data ratio and roughly half the training time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The same lower-bound trick should transfer to offline RL and imitation learning, where the Q-function is fixed or estimated from a static dataset, giving a variational objective that needs no online rollouts.
Editorial inference: The exploration claim predicts a measurable signature: over training, DIME's action distribution should remain visibly non-Gaussian on tasks that reward multi-modal behavior, while a fair Gaussian-policy comparison cannot produce the same diversity; logging action statistics would test this directly.
Editorial inference: The convergence guarantee as stated is open at its joints: the proofs are written for finite action spaces and one displayed step appears to replace the action $a_0$ in the target with $a_N$; computing the exact Q-iteration on a small solvable MDP would settle whether the monotone improvement claim survives in continuous action spaces.
Editorial inference: Because the objective is fully differentiable in the diffusion coefficient and the prior, end-to-end learning of these quantities could be pushed further into learned forward processes, potentially replacing the fixed Ornstein-Uhlenbeck prior with an adaptive one.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about arXiv:2502.02316. The method itself is a real new combination: DIME replaces the intractable marginal entropy of a diffusion policy with a variational lower bound and casts MaxEnt policy improvement as a path-space KL against the energy policy exp(Q)/Z. The empirical evaluation is broad—13 continuous-control tasks, comparisons with the main diffusion-RL baselines plus CrossQ and BRO—and the results are strong. Second thing: the headline theoretical claim, that the proposed policy iteration provably converges to the optimal diffusion policy, is not supported as written. The proof of Proposition 4.2 in Appendix B puts the exponentiated Q at the noised action a^N, while the Q-function, the rewards, and the policy loss are all evaluated at the denoised action a^0. The inequalities in (47)-(50) all track a^N, so monotone improvement is established, at best, for the noised argument—not for the action that actually enters the MDP. No step connects the two. Proposition 4.3 inherits this gap and also just assumes the policy sequence converges. The finite-action-space assumption is dropped without proof when moving to the practical algorithm. The lower-bound derivation itself is clean and standard; the authors also honestly note that the sampler is an instance of Berner et al., which is the right way to handle that lineage. The ablations—alpha sensitivity, diffusion-step count, distributional Q—are useful, and the diffusion-vs-Gaussian comparison under identical implementation details is the right way to exhibit the benefit of a diffusion policy. So the empirical story is the real contribution. The main soft spot beyond the proof is that no code or data is shipped, which limits independent verification of the 13-benchmark claims. The baseline tuning is handled transparently enough, though it remains the usual soft spot. Bottom line: this is a serious paper, worth the time of a careful reviewer and likely to be useful to people working with diffusion policies in online RL. But the "provably converges" claim needs a real fix or an explicit downgrade to "heuristic policy improvement." I would send it to review, with the instruction to focus on Appendix B, and expect the authors to either patch the proof or weaken the theorem.

Referee Report

3 major / 4 minor

Summary. The paper proposes DIME, a method for maximum-entropy reinforcement learning with diffusion-model policies. Because the marginal entropy of a diffusion policy is intractable, the authors derive a variational lower bound on the entropy (Section 4.1, Eq. 15) and cast the MaxEnt objective as a path-space KL minimization between the denoising process and an energy-based target policy exp(Q)/Z. They then state a policy-iteration scheme with monotone improvement (Propositions 4.1-4.3) and a practical actor-critic algorithm using a score network, distributional Q-learning, and automatic temperature tuning. Experiments on 13 continuous-control tasks report that DIME outperforms diffusion-based baselines on all environments and is competitive with state-of-the-art Gaussian-policy methods such as CrossQ and BRO.

Significance. If the theoretical claims hold, DIME would be a principled way to train expressive diffusion policies in online MaxEnt RL without adding Gaussian exploration noise, which is a genuine gap in the current literature. The variational lower-bound derivation in Section 4.1 is clean and standard, and the paper is commendably thorough in its empirical validation: 10 seeds, IQM with bootstrap confidence intervals, ablations on the temperature, diffusion steps, distributional Q, and the learned diffusion coefficient, plus runtime comparisons. The connection to approximate-inference samplers (Berner et al.; Richter & Berner) is clearly laid out and the method is well positioned relative to prior diffusion-RL baselines. The main body of the paper is readable and the practical recipe is concrete and reproducible from the appendices. The load-bearing theoretical claims, as written, are not fully supported, which limits what can be concluded from the convergence propositions.

major comments (3)

[Appendix B, Eq. 45 and Section 4.2] The proof of Proposition 4.2 defines the candidate policy as pi^(i+1)(a^{0:N}|s) = exp(Q^{pi^i}(s,a^N))/Z * pi^i(a^{0:N-1}|a^N,s), putting the energy factor at the noised action a^N. In the main text, however, the MDP action is a^0: the policy improvement target in Eqs. 22-23, the rewards in Eq. 16, the Q-function in Eq. 17, and the actor loss in Eq. 27 all evaluate Q at a^0. The inequalities (47)-(57) and the recursive definition of Q in Eq. 52 all track a^N, so the chain establishes at most an improvement statement for the noised action argument, not for the action a^0 that enters the environment. No step in the proof connects the a^N-marginal of exp(Q)/Z to the required a^0-marginal. As a result, Proposition 4.2 and the convergence claim in Proposition 4.3 are not supported by the presented proof.
[Section 4.2 and Section 4.3] Section 4.2 states 'Throughout this section, we assume finite action spaces to enable theoretical analysis, but relax this assumption in Section 4.3.' Section 4.3 then introduces the practical algorithm without any additional regularity conditions, convergence argument, or density-assumption check for continuous action spaces. Since the proof of Proposition 4.2 relies on KL divergences between densities and on exact minimization over the policy class (Eq. 22), the formal guarantees do not transfer to the continuous setting in which DIME is defined and evaluated. The paper should either supply conditions under which the finite-action proofs extend, or explicitly scope the convergence theorems to the finite-action case and state what is only heuristic for the continuous case.
[Appendix B, proof of Proposition 4.3] The proof of Proposition 4.3 is conditional on existence of the limit: it reads 'If for lim_{k->infty} pi^k = pi*, then it must hold that Q^{pi*}(s,a) >= Q^{pi}(s,a)'. The proof does not show that the sequence pi^i converges, nor that the fixed point is unique and optimal within the restricted class \bar{Pi}. Additionally, the idealized policy-iteration step in Eq. 22 assumes an exact argmin over \bar{pi}, whereas the practical algorithm minimizes the amortized loss in Eq. 27 by stochastic gradients, so the connection between the formal policy-iteration theorem and the empirical procedure is not established. These gaps leave the statement 'provably converges to the optimal diffusion policy' unsupported as written.

minor comments (4)

[Appendix A, Eqs. (29)-(32)] The lower-bound derivation has inconsistent notation between \vec pi and \bar pi in the displayed equations; for instance, Eq. (29) uses \vec pi for the denoising joint, while Eq. (32) returns a bound involving \bar pi, and Eq. (33) states the factorisation π0 = \vecπ0:N / \vecπ1:N|0 without aligning the superscripts with the preceding lines. The argument is standard, but the notation should be made uniform.
[Proposition 4.3 statement] The statement uses (s,a) without a superscript on the action, whereas the policy and Q-function elsewhere in Section 4.2 are defined in terms of a^0. The statement should read Q^{pi*}(s,a^0) for consistency with Eq. 17.
[Figure 2 caption] The caption for 'Diffusion Policy Benefit' refers to 'on the (a) humanoid-run and (b) dog-run tasks', but the corresponding panels are (c) and (d); this makes the caption confusing when read alongside the temperature panels (a)-(b).
[Appendix D, Eq. (59)] The tanh change-of-variables log-likelihood correction uses u^N_i in the final term, but the correction is applied at the last diffusion step n = 0, so the term should involve u^0_i; the superscript N appears to be a typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the derivation is self-contained, though the convergence proof contains a correctness gap.

full rationale

The paper's central derivation chain is not circular. The lower bound in Eq. 15 is a standard auxiliary-variable bound derived from the data processing inequality and the non-negativity of KL divergence (Appendix A); it does not presuppose the maximum-entropy objective it is used to maximize. The policy improvement target exp(Q)/Z in Eqs. 18-23 is the standard MaxEnt control-as-inference target, with Q defined through Eq. 17 including the entropy lower bound, and the parameterized diffusion policy is trained by minimizing the resulting KL divergence against a learned critic. This is normal actor-critic feedback rather than a fitted input being renamed as a prediction. Self-citations to the authors' approximate-inference work (Richter & Berner, Blessing et al.) are used as optional samplers and ablations; the objective itself is derived by elementary identities in Appendix A, so these citations are not load-bearing for the central claim. The main theoretical weakness is a correctness gap, not circularity: the proof of Proposition 4.2 in Appendix B, Eq. 45, places the energy exp(Q)/Z at the noised action a^N rather than the MDP action a^0 used in Eqs. 17, 22-23 and 27, and Proposition 4.3 assumes convergence of the policy sequence rather than establishing it. That gap undermines the 'provably converges' claim as written, but it does not make any prediction reduce to its inputs by construction. The empirical comparison is against external baselines, so the paper is self-contained against outside benchmarks. Honest non-finding is therefore appropriate.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or latent entities. It relies on six background assumptions, the most fragile being representational sufficiency of the diffusion policy class and convergence of the iterative scheme. The main free parameters are the exploration temperature, diffusion step count, prior variance, per-dimension diffusion scaling, and distributional Q bounds.

free parameters (5)

Temperature alpha / target entropy Htarget = Auto-tuned via Eq. 28; Htarget = 4*dim(A) for DIME
Controls exploration-exploitation; chosen per benchmark rather than derived, and varies across baselines.
Number of diffusion steps N = 16 (2 to 32 tested in Fig. 3)
Accuracy/computation trade-off; end performance saturates around 16, so the value is a design choice.
Prior distribution variance eta^2 = N(0, 2.5)
Initial noise scale of the OU process; set by hand and influences exploration and target coverage.
Per-dimension diffusion coefficient scaling beta = Learned end-to-end (Appendix G, Fig. 7)
A learnable scaling parameter multiplying the cosine schedule; reduces manual tuning but is an additional model parameter.
Distributional Q bin bounds vmin/vmax = -1600/1600 Gym, -200/200 DMC, -3600/3600 MyoSuite
Chosen per benchmark suite; affects critic scale and can shift results.

assumptions (6)

domain assumption The OU noising process converges to a Gaussian prior at T: pi_T approx N(0, eta^2 I).
Used in Section 3.2 to justify starting the backward process from a Gaussian; exact only for suitable beta and large T.
standard math Time-reversal of the OU diffusion (Anderson/Nelson) gives the reverse process in Eq. 8.
Invoked in Section 3.2; this is a standard stochastic calculus result.
standard math The variational lower bound in Eq. 15 is valid for any joint distributions placed in the log-ratio.
Derivation in Appendix A, Eqs. 29-32, follows from non-negativity of KL.
standard math Standard policy evaluation convergence (Sutton and Barto) applies to the entropy-augmented reward r + ell.
Used in the proof of Proposition 4.1 in Appendix B.
ad hoc to paper The restricted diffusion policy class Pi_bar contains, or can approximate, the optimal solution of the KL minimization in Eq. 22.
Needed for Propositions 4.2 and 4.3; no approximation-error bound is given, and finite action spaces are assumed then dropped in Section 4.3.
ad hoc to paper The sequence of policies generated by the practical actor-critic updates converges to the fixed point of the idealized policy iteration.
Proposition 4.3's proof assumes the limit exists; practical updates use stochastic gradients and a learned critic, so the guarantee is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DIME:Diffusion-Based Maximum Entropy Reinforcement Learning." pith.science (2026). https://pith.science/paper/TCMMLHA7

@misc{pith2026250202316,
  author       = {Pith},
  title        = {Pith review of: DIME:Diffusion-Based Maximum Entropy Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TCMMLHA7}},
  note         = {Machine review of arXiv:2502.02316}
}

read the original abstract

Maximum entropy reinforcement learning (MaxEnt-RL) has become the standard approach to RL due to its beneficial exploration properties. Traditionally, policies are parameterized using Gaussian distributions, which significantly limits their representational capacity. Diffusion-based policies offer a more expressive alternative, yet integrating them into MaxEnt-RL poses challenges-primarily due to the intractability of computing their marginal entropy. To overcome this, we propose Diffusion-Based Maximum Entropy RL (DIME). \emph{DIME} leverages recent advances in approximate inference with diffusion models to derive a lower bound on the maximum entropy objective. Additionally, we propose a policy iteration scheme that provably converges to the optimal diffusion policy. Our method enables the use of expressive diffusion-based policies while retaining the principled exploration benefits of MaxEnt-RL, significantly outperforming other diffusion-based methods on challenging high-dimensional control benchmarks. It is also competitive with state-of-the-art non-diffusion based RL methods while requiring fewer algorithmic design choices and smaller update-to-data ratios, reducing computational complexity.

Figures

Figures reproduced from arXiv: 2502.02316 by the authors.

**Figure 2.** Reward Scaling Sensitivity (a)-(b). The α parameter controls the exploration-exploitation trade-off. (a) shows the learning curves for varying values on DMC’s dog-run task. Too high α values (α = 0.1) do not incentivize learning whereas too small α values (α ≤ 10−5 ) converge to suboptimal behavior. (b) shows the aggregated end performance for each learning curve in (a). For increasing α values, the end performance … view at source ↗

**Figure 3.** Varying the Number of diffusion steps (a)-(b). The number of diffusion steps might affect the performance and the computation time. (a) shows DIME’s learning curves for varying diffusion steps. Two diffusion steps perform badly, whereas four and eight diffusion steps perform similar but still worse than 16 and 32 diffusion steps which perform similarly. (b) shows the computation time for 1MIO steps of the correspond… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Training curves on DMC’s dog, humanoid tasks, and the hand environments from the MYO Suite. DIME performs favorably on the high-dimensional dog tasks, where it significantly outperforms all baselines (dog-run) or converges faster to the final performance. On the humano…

**Figure 5.** Figure 5: Considered environments. The Humanoid-v3 and the Ant-v3 are environments from the mujoco gym benchmark (Brockman et al., 2016). The three environmentshumanoid-run,humanoid-walk and humanoid-stand are from the deepmind control suite (DMC) benchmark (Tunyasuvunakool et a…

**Figure 6.** Figure 6: Preliminary results for the GB sampler on the dog run (a) and humanoid run (b) environments from DMC. Comparison to BRO on the humanoid run for 3 million steps. 0 0.2 0.4 0.6 0.8 1 ·106 1.2 1.4 1.6 1.8 Number Env Interactions IQM β0 (a) β0 on the Dog Run 0 0.2 0.4 0.6 …

**Figure 7.** Figure 7: Learned β parameters. DIME’s policy improvement objective (Eq. 27) allows to train various parameters end-to-end, such as the scaling for the diffusion coefficient β. More concretely, we train a scaling parameter βk per dimension k, that scales the cosine schedule. We …

**Figure 8.** Figure 8: Comparison to Diffusion Baselines with (a)-b)) and without Distributional Q (c)-d)) on the Ant-v3 and Humanoid-v3 tasks. We provide the learning curves for distributional versions for Diff-QL and Consistency-AC alongside DACER, which employs distributional Q by default…

**Figure 9.** Figure 9: Ablation on Distributional Q. Comparison of DIME and DIME without employing distributional Q (dashed line). While there is a small improvement when using distributional Q, DIME w/o Distributional Q still performs on par, or better than BRO, which employs quantile distr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning
cs.LG 2026-03 conditional novelty 6.5 of 10

GeMPO unifies diffusion RL reweighting as measure matching to a regularized target, enabling flexible and negative weights that improve exploration and performance.
Efficient Online Reinforcement Learning for Diffusion Policy
cs.LG 2025-02 conditional novelty 6.0 of 10

Diffusion policies can be trained online with reweighted score matching using only Q-functions, and the resulting DPMD and SDAC algorithms beat SAC and prior diffusion-policy RL on most MuJoCo tasks.
Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
cs.LG 2025-12 conditional novelty 5.0 of 10

Diffusion policies can be inserted into maximum-entropy RL by minimizing an upper bound on reverse KL, yielding DiffPPO, DiffSAC, and DiffWPO.
Towards Adaptive External Communication in Autonomous Vehicles: A Conceptual Design Framework
cs.HC 2025-08 unverdicted novelty 5.0 of 10

A three-layer framework (input, processing, output) for adaptive external human-machine interfaces in autonomous vehicles is introduced to systematize design and analysis.
Exploratory Diffusion Model for Unsupervised Reinforcement Learning
cs.LG 2025-02 conditional novelty 5.0 of 10

A diffusion-model denoising loss serves as an intrinsic reward to guide unsupervised RL exploration, plus an alternating fine-tuning scheme for diffusion policies.

Reference graph

Works this paper leans on

79 extracted references · 37 canonical work pages · cited by 5 Pith papers

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Agakov, F. V. and Barber, D. An auxiliary variational method. In Neural Information Processing: 11th International Conference, ICONIP 2004, Calcutta, India, November 22-25, 2004. Proceedings 11, pp.\ 561--566. Springer, 2004

work page 2004
[3]

S., Courville, A

Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021

2021
[4]

J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., et al

Akhound-Sadegh, T., Rector-Brooks, J., Bose, A. J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., et al. Iterated denoising energy matching for sampling from boltzmann densities. arXiv preprint arXiv:2402.06121, 2024

arXiv 2024
[5]

Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12 0 (3): 0 313--326, 1982

1982
[6]

Efficient gradient-free variational inference using policy search

Arenz, O., Neumann, G., and Zhong, M. Efficient gradient-free variational inference using policy search. In International conference on machine learning, pp.\ 234--243. PMLR, 2018

work page 2018
[7]

G., Dabney, W., and Munos, R

Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International conference on machine learning, pp.\ 449--458. PMLR, 2017

2017
[8]

An optimal control perspective on diffusion-based generative modeling

Berner, J., Richter, L., and Ullrich, K. An optimal control perspective on diffusion-based generative modeling. Transactions on Machine Learning Research

work page

Show all 79 references

[9]

Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity

Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024

2024
[10]

Underdamped diffusion bridges with applications to sampling

Blessing, D., Berner, J., Richter, L., and Neumann, G. Underdamped diffusion bridges with applications to sampling. In The Thirteenth International Conference on Learning Representations, 2025 a

2025
[11]

End-to-end learning of gaussian mixture priors for diffusion sampler

Blessing, D., Jia, X., and Neumann, G. End-to-end learning of gaussian mixture priors for diffusion sampler. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=iXbUquaWbl

2025
[12]

Openai gym, 2016

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016

2016
[13]

Myosuite -- a contact-rich simulation suite for musculoskeletal motor control, 2022

Caggiano, V., Wang, H., Durandau, G., Sartori, M., and Kumar, V. Myosuite -- a contact-rich simulation suite for musculoskeletal motor control, 2022. arXiv preprint arXiv:2205.00588

2022 arXiv
[14]

Offline reinforcement learning via high-fidelity generative behavior modeling

Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023

2023
[15]

Sequential controlled langevin diffusions

Chen, J., Richter, L., Berner, J., Blessing, D., Neumann, G., and Anandkumar, A. Sequential controlled langevin diffusions. arXiv preprint arXiv:2412.07081, 2024

2024 arXiv
[16]

Sequential controlled langevin diffusions

Chen, J., Richter, L., Berner, J., Blessing, D., Neumann, G., and Anandkumar, A. Sequential controlled langevin diffusions. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=dImD2sgy86

2025
[17]

Diffusion policy: Visuomotor policy learning via action diffusion

Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023

2023
[18]

Cover, T. M. Elements of information theory. John Wiley & Sons, 1999

1999
[19]

A stochastic control approach to reciprocal diffusion processes

Dai Pra, P. A stochastic control approach to reciprocal diffusion processes. Applied mathematics and Optimization, 23 0 (1): 0 313--329, 1991

1991
[20]

Sequential monte carlo samplers

Del Moral, P., Doucet, A., and Jasra, A. Sequential monte carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68 0 (3): 0 411--436, 2006

2006
[21]

Diffusion-based reinforcement learning via q-weighted variational policy optimization

Ding, S., Hu, K., Zhang, Z., Ren, K., Zhang, W., Yu, J., Wang, J., and Shi, Y. Diffusion-based reinforcement learning via q-weighted variational policy optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

2024
[22]

and Jin, C

Ding, Z. and Jin, C. Consistency models as a rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024

2024
[23]

G., and Strathmann, H

Doucet, A., Grathwohl, W., Matthews, A. G., and Strathmann, H. Score-based diffusion meets annealed importance sampling. Advances in Neural Information Processing Systems, 35: 0 21482--21494, 2022

2022
[24]

Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning

Fang, L., Liu, R., Zhang, J., Wang, W., and Jing, B. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025

2025
[25]

and Domke, J

Geffner, T. and Domke, J. Langevin diffusion variational inference. In International Conference on Artificial Intelligence and Statistics, pp.\ 576--593. PMLR, 2023

2023
[26]

Reinforcement learning with deep energy-based policies

Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp.\ 1352--1361. PMLR, 2017

2017
[27]

Latent space policies for hierarchical reinforcement learning

Haarnoja, T., Hartikainen, K., Abbeel, P., and Levine, S. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp.\ 1851--1860. PMLR, 2018 a

2018
[28]

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018 b

2018
[29]

Soft actor-critic algorithms and applications

Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018 c

2018 arXiv
[30]

G., and Levine, S

Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

2023 arXiv
[31]

Haussmann, U. G. and Pardoux, E. Time reversal of diffusions. The Annals of Probability, pp.\ 1188--1205, 1986

1986
[32]

Denoising diffusion probabilistic models

Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

2020
[33]

o dinger-f \

Huang, J., Jiao, Y., Kang, L., Liao, X., Liu, J., and Liu, Y. Schr \"o dinger-f \"o llmer sampler: sampling without ergodicity. arXiv preprint arXiv:2106.10880, 1, 2021

2021 arXiv
[34]

and Dayan, P

Hyv \"a rinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6 0 (4), 2005

2005
[35]

N., and Precup, D

Ishfaq, H., Wang, G., Islam, S. N., and Precup, D. Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=FvQsk3la17

2025
[36]

Planning with diffusion for flexible behavior synthesis

Janner, M., Du, Y., Tenenbaum, J., and Levine, S. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp.\ 9902--9915. PMLR, 2022

2022
[37]

Efficient diffusion policies for offline reinforcement learning

Kang, B., Ma, X., Du, C., Pang, T., and Yan, S. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023

2023
[38]

Elucidating the design space of diffusion-based generative models

Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 0 26565--26577, 2022

2022
[39]

Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

2013 arXiv
[40]

Batch reinforcement learning

Lange, S., Gabel, T., and Riedmiller, M. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pp.\ 45--73. Springer, 2012

2012
[41]

Offline reinforcement learning: Tutorial, review, and perspectives on open problems

Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

2005 arXiv
[42]

TOP - ERL : Transformer-based off-policy episodic reinforcement learning

Li, G., Tian, D., Zhou, H., Jiang, X., Lioutikov, R., and Neumann, G. TOP - ERL : Transformer-based off-policy episodic reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025

2025
[43]

Learning multimodal behaviors from scratch with diffusion policy gradient

Li, Z., Krohn, R., Chen, T., Ajay, A., Agrawal, P., and Chalvatzaki, G. Learning multimodal behaviors from scratch with diffusion policy gradient. arXiv preprint arXiv:2406.00681, 2024

2024 arXiv
[44]

Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pp.\ 22825--22855. PMLR, 2023

2023
[45]

K., S nderby, S

Maal e, L., S nderby, C. K., S nderby, S. K., and Winther, O. Auxiliary deep generative models. In International conference on machine learning, pp.\ 1445--1453. PMLR, 2016

2016
[46]

Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning

Mao, L., Xu, H., Zhan, X., Zhang, W., and Zhang, A. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

2024
[47]

S 2 ac: Energy-based reinforcement learning with stein soft actor critic

Messaoud, S., Mokeddem, B., Xue, Z., Pang, L., An, B., Chen, H., and Chawla, S. S 2 ac: Energy-based reinforcement learning with stein soft actor critic. In The Twelfth International Conference on Learning Representations
[48]

and Cygan, M

Nauman, M. and Cygan, M. On the theory of risk-aware agents: Bridging actor-critic and economics. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2023

2024
[49]

Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control

Nauman, M., Ostaszewski, M., Jankowski, K., Mi o \'s , P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

2024
[50]

Dynamical theories of Brownian motion, volume 101

Nelson, E. Dynamical theories of Brownian motion, volume 101. Princeton university press, 2020

2020
[51]

A unified view of entropy-regularized markov decision processes

Neu, G., Jonsson, A., and G \'o mez, V. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798, 2017

2017 arXiv
[52]

The primacy bias in deep reinforcement learning

Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022

2022
[53]

Noble, M., Grenioux, L., Gabri \'e , M., and Durmus, A. O. Learned reference-based diffusion sampling for multi-modal distributions. arXiv preprint arXiv:2410.19449, 2024

2024 arXiv
[54]

Transport meets variational inference: Controlled monte carlo diffusions

Nusken, N., Vargas, F., Padhy, S., and Blessing, D. Transport meets variational inference: Controlled monte carlo diffusions. In The Twelfth International Conference on Learning Representations: ICLR 2024, 2024

2024
[55]

Learning a diffusion model policy from rewards via q-score matching

Psenka, M., Escontrela, A., Abbeel, P., and Ma, Y. Learning a diffusion model policy from rewards via q-score matching. In Forty-first International Conference on Machine Learning, 2024

2024
[56]

Hierarchical variational models

Ranganath, R., Tran, D., and Blei, D. Hierarchical variational models. In International conference on machine learning, pp.\ 324--333. PMLR, 2016

2016
[57]

Goal conditioned imitation learning using score-based diffusion policies

Reuss, M., Li, M., Jia, X., and Lioutikov, R. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023

2023
[58]

and Berner, J

Richter, L. and Berner, J. Improved sampling via learned diffusions. In The Twelfth International Conference on Learning Representations
[59]

and Solin, A

S \"a rkk \"a , S. and Solin, A. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019

2019
[60]

Deep unsupervised learning using nonequilibrium thermodynamics

Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.\ 2256--2265. PMLR, 2015

2015
[61]

P., Kumar, A., Ermon, S., and Poole, B

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

2021
[62]

Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. Robotica, 17 0 (2): 0 229--235, 1999

1999
[63]

Robot trajectory optimization using approximate inference

Toussaint, M. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pp.\ 1049--1056, 2009

2009
[64]

Tran, D., Ranganath, R., and Blei, D. M. The variational gaussian process. arXiv preprint arXiv:1511.06499, 2015

2015 arXiv
[65]

dm\_control: Software and tasks for continuous control

Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, 2020. ISSN 2665-9638

2020
[66]

and Raginsky, M

Tzen, B. and Raginsky, M. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In Conference on Learning Theory, pp.\ 3084--3114. PMLR, 2019

2019
[67]

S., and Doucet, A

Vargas, F., Grathwohl, W. S., and Doucet, A. Denoising diffusion samplers. In The Eleventh International Conference on Learning Representations
[68]

u sken, N. Bayesian learning via neural schr \

Vargas, F., Ovsianas, A., Fernandes, D., Girolami, M., Lawrence, N. D., and N \"u sken, N. Bayesian learning via neural schr \"o dinger--f \"o llmer flows. Statistics and Computing, 33 0 (1): 0 3, 2023

2023
[69]

A connection between score matching and denoising autoencoders

Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23 0 (7): 0 1661--1674, 2011

2011
[70]

and Liu, Q

Wang, D. and Liu, Q. Learning to draw samples: With application to amortized mle for generative adversarial learning. arXiv preprint arXiv:1611.01722, 2016

2016 arXiv
[71]

Wang, Y., Wang, L., Jiang, Y., Zou, W., Liu, T., Song, X., Wang, W., Xiao, L., WU, J., Duan, J., and Li, S. E. Diffusion actor-critic with entropy regulator. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum...

2024
[72]

J., and Zhou, M

Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. International Conference on Learning Representations, 2023

2023
[73]

and Teh, Y

Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.\ 681--688. Citeseer, 2011

2011
[74]

h., Zhong, Y., Yang, Y., Fang, C., Wen, S., Zhou, B., and Lin, Z

Yang, L., Huang, Z., Lei, F. h., Zhong, Y., Yang, Y., Fang, C., Wen, S., Zhou, B., and Lin, Z. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023

2023 arXiv
[75]

T., Liu, C.-H., Courville, A., and Bengio, Y

Zhang, D., Chen, R. T., Liu, C.-H., Courville, A., and Bengio, Y. Diffusion generative flow samplers: Improving learning signals through partial trajectory optimization. arXiv preprint arXiv:2310.02679, 2023

2023 arXiv
[76]

and Chen, Y

Zhang, Q. and Chen, Y. Path integral sampler: a stochastic control approach for sampling. arXiv preprint arXiv:2111.15141, 2021

2021 arXiv
[77]

Variational distillation of diffusion policies into mixture of experts

Zhou, H., Blessing, D., Li, G., Celik, O., Jia, X., Neumann, G., and Lioutikov, R. Variational distillation of diffusion policies into mixture of experts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id...

2024
[78]

Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010

2010
[79]

D., Maas, A

Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp.\ 1433--1438. Chicago, IL, USA, 2008

2008

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Agakov, F. V. and Barber, D. An auxiliary variational method. In Neural Information Processing: 11th International Conference, ICONIP 2004, Calcutta, India, November 22-25, 2004. Proceedings 11, pp.\ 561--566. Springer, 2004

work page 2004

[3] [3]

S., Courville, A

Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021

2021

[4] [4]

J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., et al

Akhound-Sadegh, T., Rector-Brooks, J., Bose, A. J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., et al. Iterated denoising energy matching for sampling from boltzmann densities. arXiv preprint arXiv:2402.06121, 2024

arXiv 2024

[5] [5]

Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12 0 (3): 0 313--326, 1982

1982

[6] [6]

Efficient gradient-free variational inference using policy search

Arenz, O., Neumann, G., and Zhong, M. Efficient gradient-free variational inference using policy search. In International conference on machine learning, pp.\ 234--243. PMLR, 2018

work page 2018

[7] [7]

G., Dabney, W., and Munos, R

Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International conference on machine learning, pp.\ 449--458. PMLR, 2017

2017

[8] [8]

An optimal control perspective on diffusion-based generative modeling

Berner, J., Richter, L., and Ullrich, K. An optimal control perspective on diffusion-based generative modeling. Transactions on Machine Learning Research

work page