Pith. sign in

REVIEW 22 cited by

Policy Representation via Diffusion Probability Model for Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.13122 v1 pith:7CDRWVVI submitted 2023-05-22 cs.LG

Long Yang , Zhixiong Huang , Fenghao Lei , Yucun Zhong , Yiming Yang , Cong Fang , Shiting Wen , Binbin Zhou
show 1 more author
Zhouchen Lin
This is my paper · ORCID
classification cs.LG
keywords policydiffusionmodeldipomodel-freeonlineprobabilitycomplicated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration. The diffusion probability model is powerful to learn complicated multimodal distributions, which has shown promising and potential applications to RL. In this paper, we formally build a theoretical foundation of policy representation via the diffusion probability model and provide practical implementations of diffusion policy for online model-free RL. Concretely, we character diffusion policy as a stochastic process, which is a new approach to representing a policy. Then we present a convergence guarantee for diffusion policy, which provides a theory to understand the multimodality of diffusion policy. Furthermore, we propose the DIPO which is an implementation for model-free online RL with DIffusion POlicy. To the best of our knowledge, DIPO is the first algorithm to solve model-free online RL problems with the diffusion model. Finally, extensive empirical results show the effectiveness and superiority of DIPO on the standard continuous control Mujoco benchmark.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Expressivity and Statistical Trade-offs in Diffusion Policy Learning

    stat.ML 2026-07 accept novelty 7.0 of 10

    Drift Lipschitz budget K yields 1/K value approximation for diffusion policies, with matching lower bound, and finite-sample rates Õ(n^{-2/(m+6)}) (generic) or Õ(n^{-2/(m+4)}) (dissipative).

  2. Q-VGM: Q-Value-Gradient Matching for Off-Policy Reinforcement Learning of Flow-Matching VLA

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    Q-VGM offline RL fine-tuning converts critic Q-gradients into residual velocity targets for flow-matching VLAs, raising LIBERO success from 75.0% to 92.5%.

  3. Diffusion Guidance Is a Controllable Policy Improvement Operator

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Diffusion guidance with a tunable weight is a controllable policy improvement operator, improving offline and goal-conditioned policies beyond the data without retraining and often without a value function.

  4. GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning

    cs.LG 2026-03 conditional novelty 6.5 of 10

    GeMPO unifies diffusion RL reweighting as measure matching to a regularized target, enabling flexible and negative weights that improve exploration and performance.

  5. Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners

    cs.LG 2026-07 conditional novelty 6.0 of 10

    In a large empirical study on a water-treatment PID control task, bounded beta policies with adaptive critic updates were the most reliable actor-critic configuration, while common defaults like Gaussian policies with...

  6. Mean Flow Policy Optimization

    cs.LG 2026-04 conditional novelty 6.0 of 10

    Mean Flow Policy Optimization (MFPO) uses few-step flow-based models for RL policies and achieves performance on par with or better than diffusion-based methods while substantially lowering training and inference time...

  7. FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning

    cs.LG 2025-10 conditional novelty 6.0 of 10

    An online imitation-learning method uses a flow-matching teacher's class-conditional loss as a reward and a regularizer to train a simple MLP policy, beating cloning and adversarial-imitation baselines on five of six tasks.

  8. Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Diffusion Tree Sampling is a Monte Carlo tree search over denoising trajectories that propagates terminal rewards backward to sample from reward-aligned distributions, showing up to 10x compute savings on tested benchmarks.

  9. Habitizing Diffusion Planning for Efficient and Effective Decision Making

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A variational-Bayes distillation framework (Habi) turns slow diffusion planners into fast feedforward policies that match their performance at orders-of-magnitude higher decision frequency on D4RL benchmarks.

  10. UniDB: A Unified Diffusion Bridge Framework via Stochastic Optimal Control

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A stochastic optimal control formulation of diffusion bridges, where Doob's h-transform is the infinite-penalty limit and a finite penalty yields a tunable detail-preserving bridge.

  11. Flow Q-Learning

    cs.LG 2025-02 accept novelty 6.0 of 10

    FQL trains a one-step policy to maximize Q-values while distilling a flow-matching behavioral cloning policy, outperforming many offline RL baselines.

  12. DIME:Diffusion-Based Maximum Entropy Reinforcement Learning

    cs.LG 2025-02 conditional novelty 6.0 of 10

    DIME derives a variational lower bound on the maximum entropy RL objective for diffusion policies and shows strong continuous-control benchmark results.

  13. Efficient Online Reinforcement Learning for Diffusion Policy

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Diffusion policies can be trained online with reweighted score matching using only Q-functions, and the resulting DPMD and SDAC algorithms beat SAC and prior diffusion-policy RL on most MuJoCo tasks.

  14. Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

    cs.LG 2024-12 conditional novelty 6.0 of 10

    PA-RL replaces the policy-gradient step in actor-critic RL with supervised learning on critic-optimized action samples, letting one algorithm fine-tune Gaussian, diffusion, transformer, and 7B VLA policies.

  15. RLMM-Flow: A Flow-based Mobile Manipulation Framework with Latent-Space Reinforcement Learning

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Steering a frozen flow-based whole-body motion policy in latent space with a warmed-up critic and coarse-to-fine residuals improves mobile manipulation success and safety.

  16. A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

    cs.RO 2026-07 conditional novelty 5.0 of 10

    One diffusion policy trained via energy-guided RL solves multi-shape block pushing without demos and transfers zero-shot to real robots under varied conditions.

  17. Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

    cs.AI 2026-02 conditional novelty 5.0 of 10

    A diffusion-policy multi-agent RL framework substitutes an ELBO for intractable joint entropy and reports 2.5–5× sample-efficiency gains on 10 MPE/MAMuJoCo tasks.

  18. Diffusion-RL Based Air Traffic Conflict Detection and Resolution Method

    cs.AI 2025-09 conditional novelty 5.0 of 10

    Diffusion-AC, a diffusion-policy RL agent with dual-Q guidance and a density curriculum, beats PPO/TD3/DQN baselines in simulated 3D conflict resolution, cutting near-collisions by about 60% in dense traffic.

  19. Exploratory Diffusion Model for Unsupervised Reinforcement Learning

    cs.LG 2025-02 conditional novelty 5.0 of 10

    A diffusion-model denoising loss serves as an intrinsic reward to guide unsupervised RL exploration, plus an alternating fine-tuning scheme for diffusion policies.

  20. Diffusion Model for Multiple Antenna Communications

    eess.SP 2025-02 conditional novelty 4.0 of 10

    A unified diffusion-model framework for generation and decision-making tasks in multi-antenna communications, with a beamforming case study showing gains over fully-connected and graph neural networks.

  21. Unpacking the Individual Components of Diffusion Policy

    cs.LG 2024-11 conditional novelty 4.0 of 10

    An ablation study shows that observation sequences, action sequences, receding horizon control, U-Net backbones, and FiLM conditioning each help Diffusion Policy in task-dependent ways, with absolute-control and hard ...

  22. Spatial-Temporal Aware Visuomotor Diffusion Policy Learning

    cs.RO 2025-07 conditional novelty 3.0 of 10

    A diffusion-based visuomotor policy gains 3D and 4D scene awareness from a dynamic Gaussian world model, improving simulated and real robot manipulation success rates.

Pith tools