REVIEW 6 cited by
DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Constrained policy search (CPS) is a fundamental problem in offline reinforcement learning, which is generally solved by advantage weighted regression (AWR). However, previous methods may still encounter out-of-distribution actions due to the limited expressivity of Gaussian-based policies. On the other hand, directly applying the state-of-the-art models with distribution expression capabilities (i.e., diffusion models) in the AWR framework is intractable since AWR requires exact policy probability densities, which is intractable in diffusion models. In this paper, we propose a novel approach, $\textbf{Diffusion-based Constrained Policy Search}$ (dubbed DiffCPS), which tackles the diffusion-based constrained policy search with the primal-dual method. The theoretical analysis reveals that strong duality holds for diffusion-based CPS problems, and upon introducing parameter approximation, an approximated solution can be obtained after $\mathcal{O}(1/\epsilon)$ number of dual iterations, where $\epsilon$ denotes the representation ability of the parametrized policy. Extensive experimental results based on the D4RL benchmark demonstrate the efficacy of our approach. We empirically show that DiffCPS achieves better or at least competitive performance compared to traditional AWR-based baselines as well as recent diffusion-based offline RL methods. The code is now available at https://github.com/felix-thu/DiffCPS.
Forward citations
Cited by 6 Pith papers
-
VINE: Taming Generative Control Policies for Reinforcement Learning
Reconstructing a fresh noisy interpolation state at every denoising step stabilizes end-to-end value-gradient training of multi-step flow-matching policies and yields state-of-the-art offline and real-robot results.
-
Diffusion Guidance Is a Controllable Policy Improvement Operator
Diffusion guidance with a tunable weight is a controllable policy improvement operator, improving offline and goal-conditioned policies beyond the data without retraining and often without a value function.
-
Flow Q-Learning
FQL trains a one-step policy to maximize Q-values while distilling a flow-matching behavioral cloning policy, outperforming many offline RL baselines.
-
Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation
HyDo combines diffusion-model policies with maximum entropy RL in a hybrid discrete/continuous action space, improving success rates on non-prehensile manipulation tasks.
-
RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures
RoboTALES uses hierarchical LLM subgoals and VLM reward feedback to keep video-model futures task-aligned, then trains robot policies that beat baselines on RoboCasa and LIBERO10 long-horizon tasks.
-
Flow-Based Policy for Online Reinforcement Learning
FlowRL learns online RL policies as flow-matching models regularized by a Wasserstein-2 constraint toward behavior-optimal replay-buffer actions.
Discussion (0). Continue with ORCID to comment.