Pith. sign in

REVIEW 4 major objections 7 minor 4 cited by

Efficient Online Reinforcement Learning for Diffusion Policy

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reweighted score matching lets diffusion policies train online without sampling from the optimal policy, and yields algorithms that outperform SAC on MuJoCo benchmarks.

desk verdict The RSM identity is real and the two losses are genuine contributions; the paper's main soft spots are empirical overclaiming and an unanalyzed SDAC critic approximation, not the core math. read the letter →

arxiv 2502.00361 v4 pith:WN3EH2EA submitted 2025-02-01 cs.LG

Haitong Ma , Tianyi Chen , Kai Wang , Na Li , Bo Dai This is my paper · ORCID
classification cs.LG
keywords diffusionpolicyonlinereinforcementlearningreweightedscorematchingenergy-basedmirrordescentmax-entropydenoisingsoftactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that denoising score matching can be reweighted by any strictly positive function without changing its optimum, and that this freedom removes the main obstacle to training diffusion policies in online reinforcement learning, namely the need to sample from the optimal policy. With a Q-function-derived reweighting, the loss becomes a tractable expectation over the current policy or any full-support sampling distribution, yielding two algorithms: DPMD for policy mirror descent and SDAC for max-entropy policies. This matters because diffusion policies can represent the multimodal energy-based policies that arise as closed-form solutions in these settings, while Gaussian actors such as SAC cannot. The paper reports that the resulting algorithms outperform recent diffusion-policy RL methods on most MuJoCo tasks, with more than 120% improvement over SAC on Humanoid and Ant.

What carries the argument

The central object is the reweighted score-matching loss $L_g(\theta;s,t) = \int g(a_t;s) \|s_\theta(a_t;s,t) - \nabla_{a_t} \log p_t(a_t|s)\|^2 \, da_t$, whose minimizer is the noise-perturbed score for any strictly positive $g$. The argument rides on Tweedie's identity, which expresses the unknown score $\nabla_{a_t} \log p_t(a_t|s)$ as an expectation of the Gaussian corruption score over the posterior $q_{0|t}$, and on the reverse sampling trick, which notes that $\phi_{0|t}(\tilde{a}_0|a_t) = \mathcal{N}(\tilde{a}_0; a_t/\sqrt{\bar\alpha_t}, ((1-\bar\alpha_t)/\bar\alpha_t)I)$ is proportional to the forward kernel $q_{t|0}(a_t|a_0)$ and shares its score with respect to $a_t$. These identities let importance weights $\exp(Q(s,a_0)/\lambda)$ be evaluated on samples that are not from the optimal policy, converting intractable score matching against the optimal policy into tractable expectations over the current policy or a chosen density $h_t$.

What would settle it

On a small MDP with a known soft-optimal policy, run SDAC with the Gaussian-noise surrogate for $\log \pi(a|s)$ and with an unbiased estimate of the diffusion policy's true log-likelihood, and compare both learned action distributions to Boltzmann($Q/\lambda$); disagreement beyond sampling error, or a mismatch between Algorithm 2's empirical loss and the expectation in Eq (14), would show the implemented objective is not the exact reweighted loss.

Watch

Extended reading notes

Core claim

The paper establishes that, for any strictly positive weighting $g(a_t;s)$, minimizing $L_g(\theta;s,t) = \int g(a_t;s) \|s_\theta(a_t;s,t) - \nabla_{a_t} \log p_t(a_t|s)\|^2 \, da_t$ has the same optimum as standard denoising score matching, namely $s_{\theta^*}(a_t;s,t) = \nabla_{a_t} \log p_t(a_t|s)$. With $g = Z_{MD}(s) p_t(a_t|s)$, the loss equals, up to constants, the DPMD objective $\mathbb{E}_{a_0 \sim \pi_{\mathrm{old}}, a_t \sim q_{t|0}}[\exp(Q(s,a_0)/\lambda) \|s_\theta(a_t;s,t) - \nabla_{a_t} \log q_{t|0}(a_t|a_0)\|^2]$, which is estimable from samples of the current policy. With $g = h_t(a_t|s) Z(s) p_t(a_t|s)$ and the reverse sampling trick, where $\phi_{0|t}(\tilde{a}_0|a_t) \propto q_{t|0}(a_t|\tilde{a}_0)$ with matching scores, the SDAC objective becomes an expectation over any full-support $h_t$ and the reverse Gaussian. The paper claims this preserves the optimal solution and the low cost of denoising score matching while eliminating the need to sample from the target policy, and validates the two algorithms on MuJoCo.

Load-bearing premise

The load-bearing premise is that the Q-function used as the energy oracle is reliable, and that SDAC's practical approximations, namely the Gaussian-noise stand-in for $\log \pi(a|s)$ and the per-sample logsumexp normalization treated as a state-only reweighting, do not change the learned optimum.

Editorial extensions

If this is right

  • Diffusion policies can be trained for online RL with roughly the memory and compute cost of denoising score matching, avoiding backpropagation through the reverse diffusion process.
  • DPMD trains a diffusion policy to represent the mirror-descent update $\pi_{MD}$ in (2), and SDAC trains one to represent the max-entropy policy in (5), without projecting these energy-based policies onto Gaussians.
  • Because any strictly positive $g$ works, RSM also applies to general known-energy distributions; the paper demonstrates this on Gaussian-mixture and Two-Moon Boltzmann sampling tasks.
  • Combined with batch action sampling and tuned exploration noise, the reweighted losses keep exploitation while preserving the expressiveness of diffusion policies.
  • On MuJoCo benchmarks the best DPMD/SDAC variants outperform all compared baselines, exceeding SAC by more than 120% on Humanoid and Ant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same reweighting construction should transfer to any setting where a reliable energy or Q-function is available, such as offline RL with learned critics, removing the need for expert action samples for diffusion-policy fitting.
  • Since the SDAC loss's sampling distribution $h_t$ is arbitrary subject to full support, it could be used as an explicit exploration knob: shaping $h_t$ changes which actions are weighted in training while the optimum stays fixed, an avenue the paper does not fully explore.
  • A natural stress test of the identity would be to apply RSM to other energy-based policy objectives, such as KL-regularized objectives beyond the two studied here, and check numerically that the learned score still matches the target Boltzmann distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Reweighted Score Matching (RSM), a family of losses for training diffusion policies in online RL. The core idea is that the denoising score matching objective can be reweighted by any strictly positive function without changing the pointwise optimal score, and that two particular reweightings yield tractable losses for policy mirror descent (DPMD, Eq. 13) and max-entropy RL (SDAC, Eq. 14), avoiding both sampling from the target policy and backpropagation through the reverse diffusion process. The authors provide a toy Boltzmann-sampling experiment, a MuJoCo benchmark study against SAC, PPO, TD3, and several recent diffusion-policy RL methods, and a comparison of GPU memory and training time.

Significance. The RSM identity is clean and useful: if it holds, diffusion policies can be trained online with cost comparable to denoising score matching while preserving the expressiveness of energy-based policies. The derivations of Eqs. (13) and (14) are internally consistent under the stated support assumptions, and the mathematical core is not circular: the reweighted losses follow from the definitions of the target policies and standard score-matching identities. The release of code and the memory/time comparisons are also valuable. The main risk is the gap between the exact RSM losses and the implemented SDAC algorithm, specifically the Gaussian surrogate for log pi(a|s) in the critic, plus an overstatement in the empirical summary. These issues are load-bearing for the SDAC max-entropy claim and for the claimed all-task superiority, respectively.

major comments (4)
  1. [Appendix C.1.2, Eq. (29); Section 4.2 (Log likelihood computation)] The soft policy evaluation for SDAC replaces log pi(a|s) in Eq. (29) with the log density of the additive Gaussian exploration noise. The policy actually executed is the batch-argmax selector (16) composed with additive Gaussian noise and, at the next state, reverse diffusion sampling; the log-density of this composed policy is not the Gaussian log-density. Consequently the learned Q is not the soft Q of the executed or learned policy, and exp(Q/lambda) is not the energy function of the max-entropy policy (5) that Eq. (14) is designed to match. The paper provides no analysis or empirical validation of this surrogate; it should either estimate the true log pi(a|s) of the composed policy, prove that the surrogate introduces negligible bias in the Bellman target, or explicitly reposition SDAC's max-entropy claim as heuristic.
  2. [Algorithm 2, line 11; Section 4.2 (Numerical stability)] The logsumexp normalization replaces the weight exp(Q(s, a0)/lambda) in Eq. (14) with a self-normalized weight. For finite K this estimator is biased for the expectation in Eq. (14), and because the sampled a0^(i) are drawn from phi0|t(·|at), the normalizer depends on at; calling it 'another reweighting on the s space' is inaccurate. In the infinite-K limit the normalized objective is a different but still valid RSM loss (it reweights by h_t rather than h_t Z(s) p_t), so the pointwise optimum is preserved in the infinite-capacity limit, but the paper should state this reasoning and either analyze the finite-K bias or acknowledge it as an unmodeled approximation.
  3. [Section 5.2.2, Table 1] The claim that the proposed algorithms 'outperform all the baselines in all OpenAI Gym MuJoCo environments' is contradicted by Table 1. On Pusher, DPMD (-30.43 ± 0.37) and SDAC (-32.53 ± 5.27) are worse than TD3 (-25.07 ± 1.01) and PPO (-25.52 ± 2.60); on Swimmer, DPMD (79.3 ± 52.5) is worse than DPPO (106.1 ± 6.5) and DACER (103.0 ± 45.8). The abstract's more guarded statement, 'outperform recent diffusion-policy online RLs on most tasks,' is consistent with the table, but the main-text overclaim should be corrected.
  4. [Algorithm 1, line 10; Section 4.2 (Numerical stability)] The EMA normalization of Q changes the exponent in Eq. (13) from Q/lambda to (Q - mu)/(sigma lambda). This is not a pure constant shift: it changes the effective KL coefficient to sigma lambda and introduces a batch-dependent scaling of the loss. The paper should either state that this is a heuristic stabilization separate from the exact derivation of Eq. (13), or show that the normalized objective preserves the optimum of Eq. (13).
minor comments (7)
  1. [Algorithm 2, line 11] The displayed loss uses a single exp term without an explicit average over i; as written it is ambiguous whether the K samples are averaged, and the parentheses around Q_e(s, a0^(i))/lambda_e are misplaced. It should read something like (1/K) sum_i exp(Q_e(s, a0^(i))/lambda_e - log sum_j exp(Q_e(s, a0^(j))/lambda_e)) ||s_theta(at;s,t) - nabla_{at} log phi_t(a0^(i)|at)||^2.
  2. [Section 4.2] The text says 'sample a batch of cations and choose the one with the highest Q-value'; 'cations' should be 'actions'.
  3. [Eq. (7)] The summation runs from t=0 to T, but the forward process and noise schedule are defined for t=1,...,T; the sum should presumably start at t=1.
  4. [Remark 4.1] The integral J(s) uses p0(a|s) where the action variable should be a0, and the proportionality factor between phi0|t and qt|0 is not written out; this makes the reverse sampling trick harder to follow than necessary.
  5. [Section 5.2.1] The GitHub URL contains spaces ('diffusion policy online rl') and will not resolve as a URL; it should be a single hyphenated or percent-encoded string.
  6. [Table 1 and main text] The environment names are inconsistent: the table uses 'Inverted2Pendulum' while the text uses 'InvertedDoublePendulum'; the naming should be unified.
  7. [Proposition 3.1] In the main-text display of Eq. (11), the notation 'at ~ q0|t(·|at)' uses at on both sides of the conditioning bar; this should be a0 ~ q0|t(·|at).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RSM optimum and the DPMD/SDAC losses follow from Tweedie and DSM identities plus change of measure; the practical SDAC mismatches are correctness risks, not circular derivations.

full rationale

The central derivation is self-contained. Proposition 3.1 proves that the DSM optimum matches the noise-perturbed score via Tweedie's identity and the standard expansion in which the DSM objective differs from the explicit-score objective by a θ-independent constant; no target result is assumed. The RSM family in Eq. (12) preserves the optimum for any strictly positive g by pointwise weighting. Eq. (13) is obtained by substituting π_MD(a0|s) = π_old(a0|s) exp(Q(s,a0)/λ)/Z_MD(s) into the reweighted loss and canceling Z_MD(s), leaving an expectation over the old policy π_old with an exponential importance weight. Eq. (14) follows by the same substitution for π_MaxEnt plus the reverse-sampling identity φ_0|t(a0|at) ∝ q_t|0(at|a0), which the paper proves in Remark 4.1 and Appendix B.2. The Q-function is learned from data by Bellman residuals, which is standard actor-critic, not a fitted constant renamed as a prediction. The only overlapping-author citation (Shribak et al., 2024, for the diffusion-as-perturbed-EBM viewpoint) is non-load-bearing because Proposition 3.1 supplies the proof in the paper itself. Two practical SDAC implementation choices are flagged in the text: approximating log π(a|s) with the additive Gaussian log-density in Appendix C.1.2, and using a logsumexp normalization in Algorithm 2 described as 'another reweighting on the s space' even though the normalization is computed over a0 samples drawn conditionally on at. These are potential bias or correctness risks for the max-entropy claim, not circular steps: they do not make Eq. (14) reduce to its own input. Accordingly, no circular step is identified.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central RSM identity uses standard score matching and Gaussian perturbation facts. The load-bearing extra assumptions are about the practical algorithms: exact Q, Gaussian log-likelihood, and the logsumexp normalization. No new physical entities are introduced.

free parameters (5)
  • Diffusion steps T = 20
    Selected from sensitivity analysis on Ant-v4 (Figure 4); affects both policy sampling and training, and is fixed for all MuJoCo tasks.
  • Noise schedule = cosine
    Selected from sensitivity analysis on Ant-v4 (Figure 4); changes the noisy score targets p_t.
  • KL/entropy coefficient lambda = not reported (lambda0 annealed to lambda_target)
    Sets the temperature in both target policies (2) and (5); not listed in Table 4, so reimplementation requires guessing.
  • Exploration noise scale after batch action sampling = not reported
    Used for behavior policy and for the Gaussian log-likelihood approximation in SDAC's critic; magnitude and schedule omitted.
  • SDAC K sample count = K, value not reported
    Number of reverse-sampled candidate actions in Algorithm 2; controls variance of the logsumexp reweighting.
assumptions (7)
  • standard math Tweedie's identity and the standard equivalence between denoising score matching and explicit score matching (Vincent 2011)
    Used in Proposition 3.1 to equate the optimum of the DDPM loss with the noise-perturbed score; requires standard regularity conditions.
  • domain assumption Target policies (2) and (5) are the optimal solutions to policy mirror descent and max-entropy RL
    Taken from prior theory; the RSM losses are built to represent exactly these Boltzmann forms.
  • standard math Reweighted squared-error loss with strictly positive g has the same global minimizer as unweighted score matching
    Core to RSM; true in function space when the model class contains the target score and g has full support.
  • domain assumption Full support of sampling distribution h_t in SDAC
    Required so the reweighted objective constrains the score on the whole action space; stated in Section 4.1.2.
  • domain assumption The learned Q-function is an exact energy function for the target policies
    Eqs (13) and (14) treat Q as known; in practice it is an approximate critic learned by Bellman residuals (Eqs (28) and (29)).
  • ad hoc to paper Gaussian additive noise log-probability approximates log pi(a|s) for SDAC
    Appendix C.1.2 uses this without bias analysis; load-bearing for the soft critic update.
  • ad hoc to paper Logsumexp normalization in SDAC preserves the reweighted objective
    Section 4.2 calls it another reweighting, but the normalization depends on the sampled actions, so the empirical objective differs from Eq (14).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Online Reinforcement Learning for Diffusion Policy." pith.science (2026). https://pith.science/paper/WN3EH2EA

@misc{pith2026250200361,
  author       = {Pith},
  title        = {Pith review of: Efficient Online Reinforcement Learning for Diffusion Policy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WN3EH2EA}},
  note         = {Machine review of arXiv:2502.00361}
}
read the original abstract

Diffusion policies have achieved superior performance in imitation learning and offline reinforcement learning (RL) due to their rich expressiveness. However, the conventional diffusion training procedure requires samples from target distribution, which is impossible in online RL since we cannot sample from the optimal policy. Backpropagating policy gradient through the diffusion process incurs huge computational costs and instability, thus being expensive and not scalable. To enable efficient training of diffusion policies in online RL, we generalize the conventional denoising score matching by reweighting the loss function. The resulting Reweighted Score Matching (RSM) preserves the optimal solution and low computational cost of denoising score matching, while eliminating the need to sample from the target distribution and allowing learning to optimize value functions. We introduce two tractable reweighted loss functions to solve two commonly used policy optimization problems, policy mirror descent and max-entropy policy, resulting in two practical algorithms named Diffusion Policy Mirror Descent (DPMD) and Soft Diffusion Actor-Critic (SDAC). We conducted comprehensive comparisons on MuJoCo benchmarks. The empirical results show that the proposed algorithms outperform recent diffusion-policy online RLs on most tasks, and the DPMD improves more than 120% over soft actor-critic on Humanoid and Ant.

Figures

Figures reproduced from arXiv: 2502.00361 by the authors.

Figure 1
Figure 1. Diffusion model aims to match score network sθ(xt, t) with noise-perturbed score function ∇xt log pt(xt) by minimizing the expectation of error L2-norm ∥sθ(xt, t) − ∇xt log pt(xt)∥ 2 over distribution pt. RSM generalize to other weight function gt to enable diffusion policy training in online RL. the additional input of states sθ(at; s, t). The data distribu￾tion p0(·|s) refers to the policy in (2) or (5), and we wi… view at source ↗
Figure 2
Figure 2. The scatter plots of generating 2D Gaussian mixture, the histograms show the partition on each axis [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Average return over 20 evaluation episodes every 25k iterations (125k for Humanoid) during training. We select the top 5 baselines ranked by average performance over all tasks for clarity. The error bars are standard deviations over 5 random seeds. 5.2. OpenAI Gym MuJoCo Tasks 5.2.1. EXPERIMENTAL SETUP We implemented the proposed DPMD and SDAC algorithms with the JAX package3 and evaluated the performance on 10 Open… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Sensitivity analysis on diffusion steps and diffusion noise schedule on Ant-v4. 6. Conclusion In this paper, we proposed Reweighted Score Match￾ing (RSM), an efficient diffusion policy training algorithm 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Results to Fit Two Moon distribution, a commonly used Boltzmann sampler benchmark. We compare RSM, DDPM, iDEM, and FAB. RSM, DDPM, and iDEM all recover two separate modes, while FAB shows connections between the two modes. C.5. Baselines We include two families of meth…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Expressivity and Statistical Trade-offs in Diffusion Policy Learning

    stat.ML 2026-07 accept novelty 7.0 of 10

    Drift Lipschitz budget K yields 1/K value approximation for diffusion policies, with matching lower bound, and finite-sample rates Õ(n^{-2/(m+6)}) (generic) or Õ(n^{-2/(m+4)}) (dissipative).

  2. GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning

    cs.LG 2026-03 conditional novelty 6.5 of 10

    GeMPO unifies diffusion RL reweighting as measure matching to a regularized target, enabling flexible and negative weights that improve exploration and performance.

  3. Training Diffusion Policies via Prior-Mapping Co-Evolution

    cs.LG 2025-12 conditional novelty 6.0 of 10

    GoRL outperforms Gaussian and generative RL baselines on continuous control by optimizing a Gaussian latent policy while a separately trained diffusion or flow decoder maps fixed noise to actions.

  4. Exploratory Diffusion Model for Unsupervised Reinforcement Learning

    cs.LG 2025-02 conditional novelty 5.0 of 10

    A diffusion-model denoising loss serves as an intrinsic reward to guide unsupervised RL exploration, plus an alternating fine-tuning scheme for diffusion policies.

Reference graph

Works this paper leans on

48 extracted references · 12 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., et al

    Akhound-Sadegh, T., Rector-Brooks, J., Bose, A. J., Mittal, S., Lemos, P., Liu, C.-H., Sendera, M., Ravanbakhsh, S., Gidel, G., Bengio, Y., et al. Iterated denoising energy matching for sampling from boltzmann densities. arXiv preprint arXiv:2402.06121, 2024

    arXiv 2024

  3. [3]

    Dime: Diffusion-based maximum entropy reinforcement learning

    Celik, O., Li, Z., Blessing, D., Li, G., Palanicek, D., Peters, J., Chalvatzaki, G., and Neumann, G. Dime: Diffusion-based maximum entropy reinforcement learning. arXiv preprint arXiv:2502.02316, 2025

    arXiv 2025

  4. [4]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548, 2022

    arXiv 2022

  5. [5]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp.\ 02783649241273668, 2023

    2023

  6. [6]

    Diffusion-based reinforcement learning via q-weighted variational policy optimization

    Ding, S., Hu, K., Zhang, Z., Ren, K., Zhang, W., Yu, J., Wang, J., and Shi, Y. Diffusion-based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv:2405.16173, 2024 a

    arXiv 2024

  7. [7]

    Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning

    Ding, Z., Zhang, A., Tian, Y., and Zheng, Q. Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03570, 2024 b

    arXiv 2024

  8. [8]

    Learning universal policies via text-guided video generation

    Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

Show all 48 references
  1. [9]

    Tweedie’s formula and selection bias

    Efron, B. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106 0 (496): 0 1602--1614, 2011

    2011

  2. [10]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  3. [11]

    Reinforcement Learning with Deep Energy-Based Policies , July 2017

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement Learning with Deep Energy-Based Policies , July 2017

    2017

  4. [12]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  5. [13]

    G., and Levine, S

    Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  6. [14]

    Denoising Diffusion Probabilistic Models

    Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Probabilistic Models . In Advances in Neural Information Processing Systems , volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020

    2020

  7. [15]

    B., Shao, S., Nikolic, B., and Sreenath, K

    Huang, X., Chi, Y., Wang, R., Li, Z., Peng, X. B., Shao, S., Nikolic, B., and Sreenath, K. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets. arXiv preprint arXiv:2404.19264, 2024

    2024 arXiv

  8. [16]

    and Dayan, P

    Hyv \"a rinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6 0 (4), 2005

    2005

  9. [17]

    Sampling from energy-based policies using diffusion

    Jain, V., Akhound-Sadegh, T., and Ravanbakhsh, S. Sampling from energy-based policies using diffusion. arXiv preprint arXiv:2410.01312, 2024

    2024 arXiv

  10. [18]

    B., and Levine, S

    Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022

    2022 arXiv

  11. [19]

    Elucidating the Design Space of Diffusion-Based Generative Models , October 2022

    Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the Design Space of Diffusion-Based Generative Models , October 2022

    2022

  12. [20]

    3d diffuser actor: Policy diffusion with 3d scene representations

    Ke, T.-W., Gkanatsios, N., and Fragkiadaki, K. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885, 2024

    2024 arXiv

  13. [21]

    Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes

    Lan, G. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming, 198 0 (1): 0 1059--1106, 2023

    2023

  14. [22]

    On the global convergence rates of softmax policy gradient methods

    Mei, J., Xiao, C., Szepesvari, C., and Schuurmans, D. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pp.\ 6820--6829. PMLR, 2020

    2020

  15. [23]

    I., Stimper, V., Simm, G

    Midgley, L. I., Stimper, V., Simm, G. N., Sch \"o lkopf, B., and Hern \'a ndez-Lobato, J. M. Flow annealed importance sampling bootstrap. arXiv preprint arXiv:2208.01893, 2022

    2022 arXiv

  16. [24]

    Bridging the gap between value and policy based reinforcement learning

    Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  17. [25]

    Neal, R. M. et al. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo, 2 0 (11): 0 2, 2011

    2011

  18. [26]

    A unified view of entropy-regularized markov decision processes

    Neu, G., Jonsson, A., and G \'o mez, V. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798, 2017

    2017 arXiv

  19. [27]

    Relative entropy policy search

    Peters, J., Mulling, K., and Altun, Y. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp.\ 1607--1612, 2010

    2010

  20. [28]

    Learning a diffusion model policy from rewards via q-score matching

    Psenka, M., Escontrela, A., Abbeel, P., and Ma, Y. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752, 2023

    2023 arXiv

  21. [29]

    Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  22. [30]

    Zero-shot text-to-image generation

    Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International conference on machine learning, pp.\ 8821--8831. Pmlr, 2021

    2021

  23. [31]

    Z., Lidard, J., Ankile, L

    Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., Burchfiel, B., Dai, H., and Simchowitz, M. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  24. [32]

    World models via policy-guided trajectory diffusion

    Rigter, M., Yamada, J., and Posner, I. World models via policy-guided trajectory diffusion. arXiv preprint arXiv:2312.08533, 2023

    2023 arXiv

  25. [33]

    L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35...

    2022

  26. [34]

    M., Schreiber, N., Haas, C., Freymuth, N., Neumann, G., Lioutikov, R., and Mathis-Ullrich, F

    Scheikl, P. M., Schreiber, N., Haas, C., Freymuth, N., Neumann, G., Lioutikov, R., and Mathis-Ullrich, F. Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects. IEEE Robotics and Automation Letters, 2024

    2024

  27. [35]

    Trust region policy optimization

    Schulman, J. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  28. [36]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  29. [37]

    Diffusion spectral representation for reinforcement learning

    Shribak, D., Gao, C.-X., Li, Y., Xiao, C., and Dai, B. Diffusion spectral representation for reinforcement learning. arXiv preprint arXiv:2406.16121, 2024

    2024 arXiv

  30. [38]

    Deep Unsupervised Learning using Nonequilibrium Thermodynamics

    Sohl-Dickstein , J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics . In Proceedings of the 32nd International Conference on Machine Learning , pp.\ 2256--2265. PMLR, June 2015

    2015

  31. [39]

    Denoising Diffusion Implicit Models , October 2022

    Song, J., Meng, C., and Ermon, S. Denoising Diffusion Implicit Models , October 2022

    2022

  32. [40]

    and Ermon, S

    Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  33. [41]

    and Kingma, D

    Song, Y. and Kingma, D. P. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021

    2021 arXiv

  34. [42]

    Sliced score matching: A scalable approach to density and score estimation

    Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp.\ 574--584. PMLR, 2020

    2020

  35. [43]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl-Dickstein , J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score- Based Generative Modeling through Stochastic Differential Equations , February 2021

    2021

  36. [44]

    Mirror descent policy optimization, 2021

    Tomar, M., Shani, L., Efroni, Y., and Ghavamzadeh, M. Mirror descent policy optimization, 2021. URL https://arxiv.org/abs/2005.09814

    2021 arXiv

  37. [45]

    A Connection Between Score Matching and Denoising Autoencoders

    Vincent, P. A Connection Between Score Matching and Denoising Autoencoders . Neural Computation, 23 0 (7): 0 1661--1674, July 2011. ISSN 0899-7667, 1530-888X. doi:10.1162/NECO_a_00142

    2011 doi

  38. [46]

    Wang, Y., Wang, L., Jiang, Y., Zou, W., Liu, T., Song, X., Wang, W., Xiao, L., Wu, J., Duan, J., and Li, S. E. Diffusion Actor-Critic with Entropy Regulator , December 2024

    2024

  39. [47]

    J., and Zhou, M

    Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022

    2022 arXiv

  40. [48]

    Policy representation via diffusion probability model for reinforcement learning

    Yang, L., Huang, Z., Lei, F., Zhong, Y., Yang, Y., Fang, C., Wen, S., Zhou, B., and Lin, Z. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.