REVIEW 7 cited by
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Safe offline RL is a promising way to bypass risky online interactions towards safe policy learning. Most existing methods only enforce soft constraints, i.e., constraining safety violations in expectation below thresholds predetermined. This can lead to potentially unsafe outcomes, thus unacceptable in safety-critical scenarios. An alternative is to enforce the hard constraint of zero violation. However, this can be challenging in offline setting, as it needs to strike the right balance among three highly intricate and correlated aspects: safety constraint satisfaction, reward maximization, and behavior regularization imposed by offline datasets. Interestingly, we discover that via reachability analysis of safe-control theory, the hard safety constraint can be equivalently translated to identifying the largest feasible region given the offline dataset. This seamlessly converts the original trilogy problem to a feasibility-dependent objective, i.e., maximizing reward value within the feasible region while minimizing safety risks in the infeasible region. Inspired by these, we propose FISOR (FeasIbility-guided Safe Offline RL), which allows safety constraint adherence, reward maximization, and offline policy learning to be realized via three decoupled processes, while offering strong safety performance and stability. In FISOR, the optimal policy for the translated optimization problem can be derived in a special form of weighted behavior cloning. Thus, we propose a novel energy-guided diffusion model that does not require training a complicated time-dependent classifier to extract the policy, greatly simplifying the training. We compare FISOR against baselines on DSRL benchmark for safe offline RL. Evaluation results show that FISOR is the only method that can guarantee safety satisfaction in all tasks, while achieving top returns in most tasks.
Forward citations
Cited by 7 Pith papers
-
Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning
The paper proposes a constraint-aware Bellman operator formed by composing the optimal Bellman operator with a proximal projection, and claims it stays contractive while enforcing convex domain constraints exactly.
-
Semi-gradient DICE for Offline Constrained Reinforcement Learning
Semi-gradient DICE outputs a policy correction instead of a stationary distribution correction, and CORSDICE recovers the latter to enable accurate cost estimation and safe offline constrained RL.
-
Skill Expansion and Composition in Parameter Space
PSEC shows that weighting and summing LoRA skill modules inside a diffusion policy network outperforms composing the same skills in action or noise space across D4RL, DSRL, DMC, and Meta-World tasks.
-
From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Control
SafeDiffCon adapts a diffusion model for PDE control by adding a conformal-prediction uncertainty quantile to its safety penalty, and reports zero safety violations on three control benchmarks.
-
Constrained Diffusers for Safe Planning and Control
Constrained Diffusers enforces trajectory constraints on pre-trained diffusion models without retraining by replacing the reverse process with constrained Langevin sampling.
-
Offline Safe Reinforcement Learning Using Trajectory Classification
TraC trains an offline safe RL policy by classifying trajectories as desirable (safe, high-reward) versus undesirable (unsafe or low-reward) using a logistic loss on a policy-ratio score.
-
FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning
FAWAC adds a cost-advantage penalty to advantage weighted regression to keep offline-trained policies within a safety budget, with variants for standard and high-reward-but-unsafe datasets.
Discussion (0). Continue with ORCID to comment.