Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

Kaustubh Patil; Sebastian Sanokowski

REVIEW 4 major objections 4 minor 1 cited by

The paper claims that maximum-entropy RL with a diffusion policy can be re-expressed as a tractable sum of per-diffusion-step KL divergences on an augmented MDP, yielding practical diffusion-based variants of PPO, SAC, and WPO.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:04 UTC pith:F75ZDXRV

load-bearing objection A promising diffusion-MaxEnt RL framework whose central KL derivation has an indexing bug that needs fixing before the theory can be trusted. the 4 major comments →

arxiv 2512.02019 v3 pith:F75ZDXRV submitted 2025-12-01 cs.LG cs.AIstat.ML

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

Sebastian Sanokowski , Kaustubh Patil This is my paper

classification cs.LG cs.AIstat.ML

keywords diffusion policiesmaximum entropy reinforcement learningreverse KL divergencediffusion-augmented MDPsoft actor-criticproximal policy optimizationWasserstein policy optimizationcontinuous control

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Maximum-entropy RL can be viewed as sampling from a reward-weighted Boltzmann distribution over action sequences, and this paper claims that when the policy is a diffusion model the problem becomes tractable. By applying the data-processing inequality twice, the authors turn the reverse KL between the diffusion policy and the optimal trajectory distribution into an upper bound that decomposes into per-diffusion-step KL terms (Eq. 15). They encode this as a Diffusion-Augmented MDP—each denoising step becomes an MDP transition—so existing RL algorithms such as PPO, SAC, and WPO can adopt diffusion policies with only small changes. The resulting DiffPPO, DiffSAC, and DiffWPO match or beat their Gaussian-policy baselines on humanoid continuous-control benchmarks, and the authors argue this matters because diffusion policies can express multimodal and heavy-tailed action distributions that Gaussian policies cannot.

Core claim

The central claim is that the intractable maximum-entropy RL objective for a diffusion policy can be written as a sum, over time steps and diffusion steps, of KL divergences between the learned reverse transition q_theta(·|s_tilde_t) and the target pi(a^k|·,s_t) exp(alpha Q_Diff(s_tilde_t,·))/Z(s_tilde_t) (Eq. 15). The argument has three load-bearing parts: the environment dynamics cancel in the joint KL, the policy-gradient theorem applies because the modified reward becomes independent of theta after a stop-gradient treatment of log q_theta, and a second data-processing-inequality step upper-bounds the marginal action divergence by a joint divergence over states, actions, and diffusion int

What carries the argument

The key object is the Diffusion-Augmented Markov Decision Process (DA-MDP), which flattens the original time steps t=0..T and reverse-diffusion steps k=K..0 into a single augmented index t_tilde, with augmented state s_tilde_t=(s_t,a^k_t,k) and augmented action a_tilde_t=a^k_t. The environment reward is paid only at k=0, and the transition kernel is deterministic for k>0 (moving to the next lower diffusion index) and the environment transition plus a fresh diffusion prior draw at k=0. The identity that carries the argument is Eq. 15: the reverse-KL objective reduces to a sum over diffusion steps of KL(q_theta(·|s_tilde_t) || pi(a^k|·,s_t) exp(alpha Q_Diff(s_tilde_t,·))/Z), where Q_Diff inclu

Load-bearing premise

The load-bearing premise is that minimizing the tractable joint upper bound on the reverse KL—obtained by two applications of the data-processing inequality—actually drives down the intractable marginal KL between the diffusion policy's action distribution and the optimal action distribution; the paper gives no bound on the gap between the two.

What would settle it

Take a small MDP with a known optimal Boltzmann action distribution and a diffusion policy; after each update, compute both D_KL(q_theta(a^0)||pi(a^0)) on the marginal action distribution (by high-resolution Monte Carlo or numerical integration) and the joint upper bound D_KL(q_theta(a^{0:K},s)||pi(a^{0:K},s)) that the paper optimizes. If the upper bound decreases while the marginal KL increases over any stretch of training, the surrogate is not faithful and the central claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

DiffPPO reduces exactly to the previously proposed zero-temperature diffusion-policy PPO when the temperature is set to 0, and generalizes it to arbitrary temperatures.
DiffSAC and DiffWPO are new diffusion-based variants of SAC and WPO that follow from the same surrogate objective, so SAC- and WPO-style training can use expressive diffusion policies without redesigning the RL loop.
Because the diffusion steps are part of an augmented MDP rather than backpropagated through the whole chain, training avoids the memory cost of differentiating through the full diffusion process.
Empirically on Humanoid, Humanoid-Run, and Humanoid-Standup benchmarks, diffusion-augmented policies achieve higher average returns and better sample efficiency than their Gaussian-policy counterparts, and performance improves with more diffusion steps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If Eq. 15 is correct, the same DA-MDP construction should transfer to any maximum-entropy RL method that relies on a reverse-KL surrogate, not just the three tested; trust-region or natural-gradient algorithms could be augmented in the same way.
The framework's dependence on the data-processing inequality leaves an open quantitative question: the paper minimizes an upper bound, so a direct comparison of the marginal-action KL gap before and after training on a small MDP would reveal whether the surrogate is tight in practice.
The same augmentation could plausibly be carried to discrete action spaces, as the authors note, connecting diffusion-based MaxEnt RL to combinatorial optimization and diffusion language models; this is an extension the paper sketches but does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A promising diffusion-MaxEnt RL framework whose central KL derivation has an indexing bug that needs fixing before the theory can be trusted.

read the letter

Quick take: this is worth a serious referee, but not in its current form. The core idea — treat the diffusion policy's reverse chain as an augmented MDP and optimize a per-step KL surrogate — is the right kind of extension, and the recognition that DPPO is the T=0 limit is correct. The authors also make two genuinely new contributions: the entropy-aware value function that absorbs the forward/reverse log-ratio, and the MaxEnt extension of WPO. The claimed equivalence between the off-policy SAC surrogate and the Log-Variance/Trajectory Balance loss is plausible and, if correct, a nice unification.

The soft spots are real and one is load-bearing. Eq. 14 is written with the sum over k=1..K but places the environment reward under an indicator 1{k=0}; inside that range the indicator is identically zero, so the RHS contains no reward at all. The boundary term -log π(a^0|s_t) — which is where the environment reward should enter — is missing. Eq. 16 and Eq. 28 then freely use a^{-1}_t at k=0, which is undefined in the reverse chain. This is not a nitpick: it means the derivation of the main surrogate objective (Eq. 15) does not actually follow from the KL decomposition as written. The issue looks repairable — add the k=0 boundary term outside the sum and define the value function/reward carefully at k=0 — but as it stands the central theoretical claim is not established.

The experiments don't rescue the theory. Only Humanoid variants; four seeds; no comparisons against DPPO or the original WPO; and DiffPPO does K× more gradient updates per environment interaction, which confounds the sample-efficiency claim. No code is released, so the numbers are uncorroborated. That said, the ablation showing performance improves with more diffusion steps is at least consistent with the framework.

Who's this for? People working on diffusion policies in RL, especially those building on DPPO and DiME. A reading group could get a good discussion out of the LV/SAC connection and the MDP flattening. I would not cite it in its current state, because the main derivation has a hole and the empirical evidence is too thin to back the marketing-style claims.

Recommendation: send to peer review, with a strong request to fix the indexing in Eq. 14/16/28, add the missing boundary term, and rerun the experiments against the proper baselines. The idea is good enough that a careful revision could make this a solid paper.

Referee Report

4 major / 4 minor

Summary. The paper proposes a framework, DA-MDP, for applying maximum-entropy RL to diffusion policies. It derives an upper bound on the trajectory KL divergence via the data-processing inequality and claims that the resulting objective can be written as a sum of per-diffusion-step KL terms (Eq. 15), leading to three diffusion-based algorithms: DiffPPO, DiffSAC, and DiffWPO. The authors also claim a connection between the off-policy MaxEnt RL surrogate and the log-variance (trajectory balance) loss, and a maximum-entropy extension of Wasserstein policy optimization. Experiments on Humanoid-v4, Humanoid-run-v4, and Humanoid-standup-v4 compare the diffusion methods against vanilla PPO and SAC.

Significance. If made rigorous, the contribution is significant: the DPI-based upper bound provides a principled reason to augment existing MaxEnt RL algorithms with diffusion policies, and the proposed algorithms are simple modifications of PPO, SAC, and WPO. The claimed equivalence between the SAC surrogate and the log-variance loss is also of interest. However, the central derivation contains load-bearing indexing inconsistencies; the empirical comparison omits the closest baselines, and no code is supplied. The ideas are promising, but the paper in its current form does not substantiate the main theoretical and empirical claims.

major comments (4)

[§3, Eq. (14)] Eq. (14) as written is internally inconsistent: the sum runs over k=1..K, yet the environment reward term is multiplied by 1{k=0}, which is identically zero on that range. The KL decomposition of the diffusion joint also contains a boundary term at k=0, -log π(a^0|s_t) (equivalently -α R_env plus a constant), which is omitted entirely. Under neither reading (k=1..K or k=0..K) does Eq. (14) produce the reward-dependent Q_Diff that is inserted into Eq. (15). This is a load-bearing issue: the derivation of the paper's central surrogate objective does not follow from the stated equations.
[§3, Eqs. (16) and (28)] Eq. (16) defines Q_Diff(s̃t, a^{k-1}_t) and V_Diff with arguments a^{k-1} for k=0, where a^{-1} is undefined. Likewise, Eq. (28) writes the k=0 branch of the modified reward as R_env - T log(qθ(a^{-1}|a^0,s)/π(a^0|a^{-1},s)), but there is no reverse transition at k=0 and π(a^0|a^{-1},s) is not defined. This directly affects the DiffPPO implementation in App. E.1, which uses Eq. (28) as its reward. The k=0 log-ratio term must be removed and the boundary reward handled separately.
[App. D, Eq. (29)] The policy-gradient statement in Eq. (29) uses ∇θ log qθ(a^k_t | a^{k+1}_t, s_t), but the actual reverse transition is qθ(a^{k-1}_t | a^k_t, s_t). The index is shifted by one. This makes Eq. (29) inconsistent with both Eq. (14) and the surrogate loss derived later in App. D.2, where the correct transition appears. The policy-gradient theorem can be applied to the diffusion chain, but as written the expression does not match the distribution being differentiated.
[§4.2 and §5] The main empirical comparison omits the two closest baselines. DiffPPO is claimed to reduce to DPPO when T=0, yet DPPO is not compared in Fig. 2. DiffWPO is proposed as a maximum-entropy extension of WPO, yet WPO is not included as a baseline. The paper's claim that the diffusion methods 'match or outperform' baselines is therefore only supported for PPO and SAC, not for the methods most directly related to the proposed algorithms. Additionally, the results are averaged over only four seeds and no statistical significance testing is reported.

minor comments (4)

[§3, Eq. (15)] The notation qθ(·|s̃t) is not defined. From the text, s̃t=(s_t,a^k_t,k), and the policy transition is qθ(a^{k-1}|a^k,s_t). Please make the argument of qθ explicit and align the notation with the transition kernels in Eq. (16).
[§3.1] The augmented MDP defines the action as ã_{t̃}=a^k_t, but the transition p(s̃_{t̃+1}|s̃,ã) maps to (s_t,a^{k-1}_t,k-1) for k>0. The action should be a^{k-1}_t, not a^k_t, for consistency with the policy and Q-function definitions.
[App. D.2] The derivation of the surrogate loss uses DKL in a form where the target contains exp(αQ_Diff), but the signs of the log-ratio and reward terms are not checked against the policy-gradient statement in Eq. (29); after fixing the indices, verify that the surrogate gradient equals the policy-gradient direction.
[General] There are several typographical issues, e.g. 'donw' in Sec. 2.5 and 'evaluatable' in Sec. 2. The notation α=1/T is introduced in Sec. 2 but the same symbol T is used for the horizon and temperature; clarify.

Circularity Check

0 steps flagged

No significant circularity: the diffusion surrogate is a standard reverse-KL/soft-Bellman construction, and the DPPO special case is disclosed; the Eq. 14 boundary-index flaw is a correctness issue, not an equivalence.

full rationale

The central derivation chain is not circular. Eq. 13 applies the data-processing inequality to the trajectory KL, and Eq. 14 decomposes the log ratio into per-diffusion-step terms. There is, as printed, an indexing defect: the sum runs k=1..K, so the term -R_env(s_t,a^0_t)1{k=0} is identically zero, and the boundary term -log pi(a^0|s_t) is absent; Eq. 28 likewise references a_t^{-1} at k=0. This is a proof-correctness concern, not a circularity, since the intended decomposition is not equivalent to its input by construction; it is a missing boundary term. Eq. 15 then follows from the reverse log-derivative trick, with Q_Diff defined through a soft-Bellman self-consistency condition that uses q_theta*. That self-referential value function is the standard MaxEntRL fixed-point structure, not a fitted parameter presented as a prediction. DPPO being the T=0 special case is stated explicitly in Secs. 3.1 and 5. Self-citations to Sanokowski et al. (2025a, 2025b, 2023) support related-work context, LV-gradient/DPI caveats, and annealing details, but none supplies the load-bearing diffusion surrogate; no uniqueness theorem or ansatz is imported from the authors. Experimental claims are benchmarked against independent vanilla PPO/SAC implementations. Therefore no significant circularity; score 2 reflects only minor non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 0 invented entities

The central derivation uses standard variational RL identities plus hand-set diffusion hyperparameters; no new physical entities are introduced. DA-MDP is a mathematical MDP construction rather than an invented entity.

free parameters (6)

Diffusion noise schedule = β: 3 to 0.05 (linear); prior variance ν=2.2
Hand-set for all diffusion experiments; changes chain noise and target sharpness.
Number of diffusion steps K = swept (values not enumerated in text)
Ablation shows higher K improves returns; central hyperparameter.
Temperature schedule for DiffPPO = T_start = k/dim(A), k∈[0.1,0.3], halved every 10% of training
Chosen by hand; anneals to the DPPO limit T=0.
Target entropy for DiffSAC/DiffWPO = -10·dim(A)
Hand-set for the chosen noise schedule; controls exploration via automatic temperature tuning.
Learning rates = three values swept, not reported in text
Best per method selected; baselines swept over more values.
Discount γ and GAE λ = γ=0.999^(1/K), λ=0.95
Fixed to avoid tuning; affects value estimation.

axioms (7)

standard math The data processing inequality applies to f-divergences and gives a valid upper bound.
Used in Eq. 1 and Eq. 13 to marginalize away states and diffusion steps.
standard math Environment dynamics and initial-state distribution cancel in the trajectory KL.
Eq. 2/App. A.1; requires shared transition kernels in qθ and π.
domain assumption Optimal policy can be represented as a Boltzmann distribution over Q-values, π(a|s)∝exp(α Q).
Standard MaxEntRL/Ziebart assumption; introduced in Sec. 2.
domain assumption Reverse diffusion transition is a Gaussian with fixed variance (score network only shifts the mean).
Sec. 2.6; makes log qθ tractable.
ad hoc to paper Stop-gradient replacement of log qθ in the modified reward preserves the true gradient after the first update.
App. D.1; exact at θ=θ*, approximate afterward.
domain assumption For the LV loss, ω is absolutely continuous w.r.t. π and qθ.
Sec. 2.4; needed for the zero-iff-equality statement.
standard math Tail-decay/regularity for integration by parts in WPO projection.
App. C; needed to drop boundary terms in Eq. 26.

pith-pipeline@v1.3.0-alltime-deepseek · 20575 in / 22982 out tokens · 234714 ms · 2026-08-03T19:04:08.813157+00:00 · methodology

0 comments

read the original abstract

Diffusion models excel at sampling from complex, unnormalized distributions. In this work, we extend Maximum Entropy Reinforcement Learning (ME-RL) to diffusion processes, enabling sampling from the optimal policy trajectory distribution. By minimizing a tractable upper bound on the reverse KL divergence between the diffusion policy and the optimal policy trajectory distributions, we derive a modified surrogate objective and introduce Diffusion-Augmented Markov Decision Processes (DA-MDPs). DA-MDPs allow for seamless integration of diffusion policies into any ME-RL method with minimal modifications. We demonstrate its effectiveness by adapting Proximal Policy Optimization (PPO), Wasserstein Policy Optimization (WPO), and Relative Entropy Pathwise Policy Optimization (REPPO) into their diffusion-based variants: DA-MDP: PPO, DA-MDP: WPO, and DA-MDP: REPPO. Empirical results on standard continuous-control benchmarks show that our approach matches or outperforms baseline methods, while experiments on multimodal benchmarks confirm its ability to model multimodal action distributions.

Figures

Figures reproduced from arXiv: 2512.02019 by Kaustubh Patil, Sebastian Sanokowski.

**Figure 1.** Figure 1: Effect of varying the number of diffusion steps K on the Humanoid-v4 environment for DiffPPO, DiffSAC and DiffWPO. The average and standard deviation are taken over four seeds. 4.2. Main Results [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗

**Figure 2.** Figure 2: compares DiffPPO, DiffSAC, and DiffWPO against respective baselines, PPO and SAC, on the Humanoid, Humanoid-Run, and Humanoid-Standup benchmarks. Across all three tasks, diffusion-augmented policies exhibit consistently higher sample efficiency and achieve a higher average return. In particular, our experiments show that DiffPPO substantially increases sample efficiency in terms of environment interaction… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Guided Discovery of New Behaviors using Diffusion Policies
cs.RO 2026-06 unverdicted novelty 6.0

A framework combining Feynman-Kac correctors with a guiding potential mines and repairs novel trajectories to enable diffusion policies to discover diverse executable behaviors in robotic manipulation.

Reference graph

Works this paper leans on

23 extracted references · 1 canonical work pages · cited by 1 Pith paper

[8]

Levine, S

URLhttps://proceedings.neurips.cc/ paper/2020/hash/49f85a9ed090b20c8bed85a5923c669f-Abstract.html. Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review.arXiv preprint arXiv:1805.00909,

Pith/arXiv arXiv 2020
[12]

Variational annealing on graphs for combinatorial optimization

Sanokowski, S., Berghammer, W., Hochreiter, S., and Lehner, S. Variational annealing on graphs for combinatorial optimization. InAdvances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 - 16, 2023,

2023
[13]

Sanokowski, S., Hochreiter, S., and Lehner, S

URL http://papers.nips.cc/paper_files/paper/2023/hash/ c9c54ac0dd5e942b99b2b51c297544fd-Abstract-Conference.html. Sanokowski, S., Hochreiter, S., and Lehner, S. A diffusion model framework for unsupervised neural combinatorial optimization. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning...

2023
[16]

instead of the value function as defined in Sec. 2.1. For automatic temperature tuning as in (Haarnoja et al., 2018), we follow (Celik et al.,

2018
[17]

Song, Y ., Durkan, C., Murray, I., and Ermon, S

URL http://proceedings.mlr.press/v37/sohl-dickstein15.html. Song, Y ., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models.Advances in neural information processing systems, 34:1415–1428, 2021a. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling th...

2021
[18]

URL https://openreview.net/forum?id=PxTIG12RRHS

OpenReview.net, 2021b. URL https://openreview.net/forum?id=PxTIG12RRHS. Sun, H., Guha, E. K., and Dai, H. Annealed training for combinatorial optimization on graphs. InOPT 2022: Opti- mization for Machine Learning (NeurIPS 2022 Workshop),

2022
[20]

Substituting˜r(st, at) =R env(st, at)− Tlogqθ(at |s t)yields the expression stated in the main text

applies directly: ∇θDKL(qθ∥π) = TX t=0 Est,at∼qθ [−˜r(st, at)∇ θ logq θ(at |s t)]. Substituting˜r(st, at) =R env(st, at)− Tlogqθ(at |s t)yields the expression stated in the main text. A.3. Reverse Log-Derivative Trick Define the unnormalized Boltzmann policy: π(a|s)∝exp(βQ qθ (s, a)). The KL divergence DKL qθ(at|st)∥π(a t|st) has gradient ∇θDT KL =E qθ [(...

2015
[21]

Let ω denote an arbitrary (possibly off-policy) distribution over full trajectories(a 0:T , s0:T+1 )

TY t=0 π(at |s t)p(s t+1 |s t, at), but the key requirement is only that π does not depend on θ. Let ω denote an arbitrary (possibly off-policy) distribution over full trajectories(a 0:T , s0:T+1 ). We assumeωis fixed (does not depend onθ). Define the per-trajectory log-ratio ℓθ(a0:T , s0:T+1 ) := log qθ(a0:T , s0:T+1 ) π(a0:T , s0:T+1 ) = TX t=0 log qθ(a...

2023
[22]

The mixed Fisher block is Ftθ = Z ∇θ logq θ(a|s) ∂qθ(a|s) ∂t da.(24) Insert the flow from Eq

the induced flow can be projected on the parametric familyqθ by minimizing the local KL between the flowed density and the parametric perturbation. The mixed Fisher block is Ftθ = Z ∇θ logq θ(a|s) ∂qθ(a|s) ∂t da.(24) Insert the flow from Eq. 23: Ftθ =− Z ∇θ logq θ(a|s)∇ a · qθ(a|s)v(a) da. Expand the divergence and apply integration by parts. Using∇ a ·(q...

2025
[28]

The MDP is augmented following Section 3.1, but DiffPPO incorporates the diffusion terms directly into the reward rather than absorbing them into the value function

During rollout, we therefore treat each reverse-diffusion step k as part of an augmented trajectory, and importantly, since the diffusion log ratios are already incorporated in the reward, we do not use these log ratios within the value function. The MDP is augmented following Section 3.1, but DiffPPO incorporates the diffusion terms directly into the rew...

2021
[1967]

Dong, X., Cheng, J., and Zhang, X. S. Maximum entropy reinforcement learning with diffusion policy.arXiv preprint arXiv:2502.11612,

Pith/arXiv arXiv
[2000]

An optimal control perspective on diffusion-based generative modeling.arXiv preprint arXiv:2211.01364,

Berner, J., Richter, L., and Ullrich, K. An optimal control perspective on diffusion-based generative modeling.arXiv preprint arXiv:2211.01364,

Pith/arXiv arXiv
[2008]

Denoising diffusion samplers.arXiv preprint arXiv:2302.13834,

Vargas, F., Grathwohl, W., and Doucet, A. Denoising diffusion samplers.arXiv preprint arXiv:2302.13834,

Pith/arXiv arXiv
[2012]

and Loukas, A

Karalias, N. and Loukas, A. Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs. InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual,

2020
[2017]

13 A Diffusion Model Framework for Maximum Entropy Reinforcement Learning Sohl-Dickstein, J., Weiss, E

URLhttp://arxiv.org/abs/1707.06347. 13 A Diffusion Model Framework for Maximum Entropy Reinforcement Learning Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. InProceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015...

Pith/arXiv arXiv 2015
[2018]

Reinforcement learning with discrete diffusion policies for combinatorial action spaces.arXiv preprint arXiv:2509.22963,

Ma, H., Nabati, O., Rosenberg, A., Dai, B., Lang, O., Szpektor, I., Boutilier, C., Li, N., Mannor, S., Shani, L., et al. Reinforcement learning with discrete diffusion policies for combinatorial action spaces.arXiv preprint arXiv:2509.22963,

Pith/arXiv arXiv
[2019]

Dime: Diffusion-based maximum entropy reinforcement learning.arXiv preprint arXiv:2502.02316,

Celik, O., Li, Z., Blessing, D., Li, G., Palenicek, D., Peters, J., Chalvatzaki, G., and Neumann, G. Dime: Diffusion-based maximum entropy reinforcement learning.arXiv preprint arXiv:2502.02316,

Pith/arXiv arXiv
[2020]

Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D

URL https://proceedings.neurips.cc/paper/2020/hash/ 4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html. Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models.Advances in neural information processing systems, 35:8633–8646,

2020
[2021]

URL https://doi.org/10.1038/ s42256-021-00401-3

doi: 10.1038/s42256-021-00401-3. URL https://doi.org/10.1038/ s42256-021-00401-3. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. InAdvances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual,

work page doi:10.1038/s42256-021-00401-3 2020
[2022]

Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.arXiv preprint arXiv:1902.05605,

Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.arXiv preprint arXiv:1902.05605,

Pith/arXiv arXiv 1902
[2023]

Large language diffusion models

Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y ., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992,

Pith/arXiv arXiv
[2024]

press/v235/sanokowski24a.html

URL https://proceedings.mlr. press/v235/sanokowski24a.html. Sanokowski, S., Berghammer, W., Ennemoser, M., Wang, H. P., Hochreiter, S., and Lehner, S. Scalable discrete diffusion samplers: Combinatorial optimization and statistical physics. InThe Thirteenth International Conference on Learning Representations, 2025a. URLhttps://openreview.net/forum?id=peN...

arXiv
[2025]

Trajectory balance: Improved credit assignment in gflownets

Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y . Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955–5967, 2022a. Malkin, N., Lahlou, S., Deleu, T., Ji, X., Hu, E., Everett, K., Zhang, D., and Bengio, Y . Gflownets and variational inference. arXiv preprint arXiv:2210.00580, 2022...

Pith/arXiv arXiv

[1] [8]

Levine, S

URLhttps://proceedings.neurips.cc/ paper/2020/hash/49f85a9ed090b20c8bed85a5923c669f-Abstract.html. Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review.arXiv preprint arXiv:1805.00909,

Pith/arXiv arXiv 2020

[2] [12]

Variational annealing on graphs for combinatorial optimization

Sanokowski, S., Berghammer, W., Hochreiter, S., and Lehner, S. Variational annealing on graphs for combinatorial optimization. InAdvances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 - 16, 2023,

2023

[3] [13]

Sanokowski, S., Hochreiter, S., and Lehner, S

URL http://papers.nips.cc/paper_files/paper/2023/hash/ c9c54ac0dd5e942b99b2b51c297544fd-Abstract-Conference.html. Sanokowski, S., Hochreiter, S., and Lehner, S. A diffusion model framework for unsupervised neural combinatorial optimization. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning...

2023

[4] [16]

instead of the value function as defined in Sec. 2.1. For automatic temperature tuning as in (Haarnoja et al., 2018), we follow (Celik et al.,

2018

[5] [17]

Song, Y ., Durkan, C., Murray, I., and Ermon, S

URL http://proceedings.mlr.press/v37/sohl-dickstein15.html. Song, Y ., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models.Advances in neural information processing systems, 34:1415–1428, 2021a. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling th...

2021

[6] [18]

URL https://openreview.net/forum?id=PxTIG12RRHS

OpenReview.net, 2021b. URL https://openreview.net/forum?id=PxTIG12RRHS. Sun, H., Guha, E. K., and Dai, H. Annealed training for combinatorial optimization on graphs. InOPT 2022: Opti- mization for Machine Learning (NeurIPS 2022 Workshop),

2022

[7] [20]

Substituting˜r(st, at) =R env(st, at)− Tlogqθ(at |s t)yields the expression stated in the main text

applies directly: ∇θDKL(qθ∥π) = TX t=0 Est,at∼qθ [−˜r(st, at)∇ θ logq θ(at |s t)]. Substituting˜r(st, at) =R env(st, at)− Tlogqθ(at |s t)yields the expression stated in the main text. A.3. Reverse Log-Derivative Trick Define the unnormalized Boltzmann policy: π(a|s)∝exp(βQ qθ (s, a)). The KL divergence DKL qθ(at|st)∥π(a t|st) has gradient ∇θDT KL =E qθ [(...

2015

[8] [21]

Let ω denote an arbitrary (possibly off-policy) distribution over full trajectories(a 0:T , s0:T+1 )

TY t=0 π(at |s t)p(s t+1 |s t, at), but the key requirement is only that π does not depend on θ. Let ω denote an arbitrary (possibly off-policy) distribution over full trajectories(a 0:T , s0:T+1 ). We assumeωis fixed (does not depend onθ). Define the per-trajectory log-ratio ℓθ(a0:T , s0:T+1 ) := log qθ(a0:T , s0:T+1 ) π(a0:T , s0:T+1 ) = TX t=0 log qθ(a...

2023

[9] [22]

The mixed Fisher block is Ftθ = Z ∇θ logq θ(a|s) ∂qθ(a|s) ∂t da.(24) Insert the flow from Eq

the induced flow can be projected on the parametric familyqθ by minimizing the local KL between the flowed density and the parametric perturbation. The mixed Fisher block is Ftθ = Z ∇θ logq θ(a|s) ∂qθ(a|s) ∂t da.(24) Insert the flow from Eq. 23: Ftθ =− Z ∇θ logq θ(a|s)∇ a · qθ(a|s)v(a) da. Expand the divergence and apply integration by parts. Using∇ a ·(q...

2025

[10] [28]

The MDP is augmented following Section 3.1, but DiffPPO incorporates the diffusion terms directly into the reward rather than absorbing them into the value function

During rollout, we therefore treat each reverse-diffusion step k as part of an augmented trajectory, and importantly, since the diffusion log ratios are already incorporated in the reward, we do not use these log ratios within the value function. The MDP is augmented following Section 3.1, but DiffPPO incorporates the diffusion terms directly into the rew...

2021

[11] [1967]

Dong, X., Cheng, J., and Zhang, X. S. Maximum entropy reinforcement learning with diffusion policy.arXiv preprint arXiv:2502.11612,

Pith/arXiv arXiv

[12] [2000]

An optimal control perspective on diffusion-based generative modeling.arXiv preprint arXiv:2211.01364,

Berner, J., Richter, L., and Ullrich, K. An optimal control perspective on diffusion-based generative modeling.arXiv preprint arXiv:2211.01364,

Pith/arXiv arXiv

[13] [2008]

Denoising diffusion samplers.arXiv preprint arXiv:2302.13834,

Vargas, F., Grathwohl, W., and Doucet, A. Denoising diffusion samplers.arXiv preprint arXiv:2302.13834,

Pith/arXiv arXiv

[14] [2012]

and Loukas, A

Karalias, N. and Loukas, A. Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs. InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual,

2020

[15] [2017]

13 A Diffusion Model Framework for Maximum Entropy Reinforcement Learning Sohl-Dickstein, J., Weiss, E

URLhttp://arxiv.org/abs/1707.06347. 13 A Diffusion Model Framework for Maximum Entropy Reinforcement Learning Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. InProceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015...

Pith/arXiv arXiv 2015

[16] [2018]

Reinforcement learning with discrete diffusion policies for combinatorial action spaces.arXiv preprint arXiv:2509.22963,

Ma, H., Nabati, O., Rosenberg, A., Dai, B., Lang, O., Szpektor, I., Boutilier, C., Li, N., Mannor, S., Shani, L., et al. Reinforcement learning with discrete diffusion policies for combinatorial action spaces.arXiv preprint arXiv:2509.22963,

Pith/arXiv arXiv

[17] [2019]

Dime: Diffusion-based maximum entropy reinforcement learning.arXiv preprint arXiv:2502.02316,

Celik, O., Li, Z., Blessing, D., Li, G., Palenicek, D., Peters, J., Chalvatzaki, G., and Neumann, G. Dime: Diffusion-based maximum entropy reinforcement learning.arXiv preprint arXiv:2502.02316,

Pith/arXiv arXiv

[18] [2020]

Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D

URL https://proceedings.neurips.cc/paper/2020/hash/ 4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html. Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models.Advances in neural information processing systems, 35:8633–8646,

2020

[19] [2021]

URL https://doi.org/10.1038/ s42256-021-00401-3

doi: 10.1038/s42256-021-00401-3. URL https://doi.org/10.1038/ s42256-021-00401-3. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. InAdvances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual,

work page doi:10.1038/s42256-021-00401-3 2020

[20] [2022]

Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.arXiv preprint arXiv:1902.05605,

Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.arXiv preprint arXiv:1902.05605,

Pith/arXiv arXiv 1902

[21] [2023]

Large language diffusion models

Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y ., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992,

Pith/arXiv arXiv

[22] [2024]

press/v235/sanokowski24a.html

URL https://proceedings.mlr. press/v235/sanokowski24a.html. Sanokowski, S., Berghammer, W., Ennemoser, M., Wang, H. P., Hochreiter, S., and Lehner, S. Scalable discrete diffusion samplers: Combinatorial optimization and statistical physics. InThe Thirteenth International Conference on Learning Representations, 2025a. URLhttps://openreview.net/forum?id=peN...

arXiv

[23] [2025]

Trajectory balance: Improved credit assignment in gflownets

Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y . Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955–5967, 2022a. Malkin, N., Lahlou, S., Deleu, T., Ji, X., Hu, E., Everett, K., Zhang, D., and Bengio, Y . Gflownets and variational inference. arXiv preprint arXiv:2210.00580, 2022...

Pith/arXiv arXiv