D4RL supplies new offline RL benchmarks and datasets from expert and mixed sources to expose weaknesses in existing algorithms and standardize evaluation.
hub Mixed citations
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Mixed citation behavior. Most common role is background (60%).
57
Pith papers citing it
3,500
external citations · Pith
Background
60% of classified citations
abstract
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
hub tools
citation-role summary
background 3
baseline 1
method 1
citation-polarity summary
representative citing papers
Deep reinforcement learning achieves real-time cooling of single-atom motion with a 388 microsecond time constant using cavity feedback, outperforming a linear differentiator controller.
Flow Reversal Steering steers flow matching generalist policies by reversing suboptimal actions to nearby better modes, enabling improved zero-shot control, quick distillation, and RL bootstrapping in robotic manipulation.
PromptPO shows LLMs can act as black-box policy optimizers for sequential RL when leveraging prior knowledge, matching baselines in exploration and robotics but underperforming in MuJoCo.
A new marginalized reparameterization estimator allows low-variance training of mixture policies in entropy-regularized actor-critic algorithms, matching or exceeding Gaussian policy performance in several continuous control benchmarks.
CODA augments offline multi-agent RL with on-policy diffusion trajectories that evolve with the joint policy to enable coordination.
SmoothCruiser achieves O~(1/epsilon^4) problem-independent sample complexity for value estimation in entropy-regularized MDPs and games via a generative model.
A litmus test based on reachset-conformant model identification and correlation analysis of uncertainties predicts if RL-based control is superior to model-based control without any RL training.
DreamerV3 uses world models and robustness techniques to solve over 150 tasks across domains with a single configuration, including Minecraft diamond collection from scratch.
Dreamer learns to control from images by imagining and optimizing behaviors in a learned latent world model, outperforming prior methods on 20 visual tasks in data efficiency and final performance.
Introduces a benchmark suite of over 18 MBRL environments, evaluates multiple algorithms under consistent settings, and identifies three core challenges: dynamics bottleneck, planning horizon dilemma, and early-termination dilemma.
POPLIN combines policy networks with model-predictive planning by optimizing either action sequences or policy parameters, yielding 3x better sample efficiency than PETS, TD3 and SAC on MuJoCo locomotion tasks.
Proposes Prior Random Testing (PRT) that leverages task difficulty to prioritize failure-prone test cases for DRL agents, achieving over 50% lower testing cost than random testing while preserving diversity on four benchmarks.
A typed-contract harness with containerized 'chambers' and robotics-specific agent skills lets a coding LLM turn a single natural-language prompt into working reproduction, evaluation, and deployment workflows for robot learning.
Mutual Reinforcement Learning allows heterogeneous LLMs to exchange experience through mechanisms like Peer Rollout Pooling, Cross-Policy GRPO Advantage Sharing, and Success-Gated Transfer, with outcome-level sharing identified as favorable on the stability-support trade-off.
LQL turns n-step action-sequence lower bounds into a practical hinge-loss stabilizer for off-policy Q-learning without extra networks or forward passes.
MetaRL pre-trained on GBWM problems delivers near-optimal dynamic strategies in 0.01s achieving 97.8% of DP optimal utility and handles larger problems where DP fails.
MADDPG-K scales centralized critics in multi-agent RL by limiting each critic to k-nearest neighbors under Euclidean distance, yielding constant input size and competitive performance.
Rank correlation (Kendall tau) of price forecasts, not mean absolute error, determines intraday dispatch value for multi-market battery storage, with tau above 0.85-0.95 capturing 97-100% of perfect-foresight revenue.
A multi-agent RL system using Independent Soft Actor-Critic and a local-inflow surrogate for damage-equivalent loads learns policies that raise wind-farm power while respecting explicit load-increase limits.
A deep RL vulnerability-prediction policy trained in semantic embedding space finds up to 23% more unique robot manipulation failures than vision-language baselines and enables more efficient fine-tuning.
DIPPER uses bi-level optimization and DPO to train the higher-level policy from stationary preference comparisons and value regularization, claiming up to 40% gains on robotic navigation and manipulation tasks while introducing metrics for non-stationarity and infeasible subgoals.
Behavior-regularized actor-critic methods achieve strong offline RL results with simple regularization, rendering many recent technical additions unnecessary.
Disentangled Skill Embeddings (DSE) is a variational inference framework for multi-task RL using shared parameters and task-specific latent embeddings for generalization to unseen conditions and as skills in hierarchical RL.
citing papers explorer
-
D4RL: Datasets for Deep Data-Driven Reinforcement Learning
D4RL supplies new offline RL benchmarks and datasets from expert and mixed sources to expose weaknesses in existing algorithms and standardize evaluation.
-
Deep Reinforcement Learning for Individual Atomic Control and Cooling
Deep reinforcement learning achieves real-time cooling of single-atom motion with a 388 microsecond time constant using cavity feedback, outperforming a linear differentiator controller.
-
Improving Robotic Generalist Policies via Flow Reversal Steering
Flow Reversal Steering steers flow matching generalist policies by reversing suboptimal actions to nearby better modes, enabling improved zero-shot control, quick distillation, and RL bootstrapping in robotic manipulation.
-
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
PromptPO shows LLMs can act as black-box policy optimizers for sequential RL when leveraging prior knowledge, matching baselines in exploration and robotics but underperforming in MuJoCo.
-
Revisiting Mixture Policies in Entropy-Regularized Actor-Critic
A new marginalized reparameterization estimator allows low-variance training of mixture policies in entropy-regularized actor-critic algorithms, matching or exceeding Gaussian policy performance in several continuous control benchmarks.
-
CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning
CODA augments offline multi-agent RL with on-policy diffusion trajectories that evolve with the joint policy to enable coordination.
-
Planning in entropy-regularized Markov decision processes and games
SmoothCruiser achieves O~(1/epsilon^4) problem-independent sample complexity for value estimation in entropy-regularized MDPs and games via a generative model.
-
To Learn or Not to Learn: A Litmus Test for Using Reinforcement Learning in Control
A litmus test based on reachset-conformant model identification and correlation analysis of uncertainties predicts if RL-based control is superior to model-based control without any RL training.
-
Mastering Diverse Domains through World Models
DreamerV3 uses world models and robustness techniques to solve over 150 tasks across domains with a single configuration, including Minecraft diamond collection from scratch.
-
Dream to Control: Learning Behaviors by Latent Imagination
Dreamer learns to control from images by imagining and optimizing behaviors in a learned latent world model, outperforming prior methods on 20 visual tasks in data efficiency and final performance.
-
Benchmarking Model-Based Reinforcement Learning
Introduces a benchmark suite of over 18 MBRL environments, evaluates multiple algorithms under consistent settings, and identifies three core challenges: dynamics bottleneck, planning horizon dilemma, and early-termination dilemma.
-
Exploring Model-based Planning with Policy Networks
POPLIN combines policy networks with model-predictive planning by optimizing either action sequences or policy parameters, yielding 3x better sample efficiency than PETS, TD3 and SAC on MuJoCo locomotion tasks.
-
Failure-Based Testing for Deep Reinforcement Learning Agents
Proposes Prior Random Testing (PRT) that leverages task difficulty to prioritize failure-prone test cases for DRL agents, achieving over 50% lower testing cost than random testing while preserving diversity on four benchmarks.
-
Nautilus: From One Prompt to Plug-and-Play Robot Learning
A typed-contract harness with containerized 'chambers' and robotics-specific agent skills lets a coding LLM turn a single natural-language prompt into working reproduction, evaluation, and deployment workflows for robot learning.
-
Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models
Mutual Reinforcement Learning allows heterogeneous LLMs to exchange experience through mechanisms like Peer Rollout Pooling, Cross-Policy GRPO Advantage Sharing, and Success-Gated Transfer, with outcome-level sharing identified as favorable on the stability-support trade-off.
-
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
LQL turns n-step action-sequence lower bounds into a practical hinge-loss stabilizer for off-policy Q-learning without extra networks or forward passes.
-
A Meta Reinforcement Learning Approach to Goals-Based Wealth Management
MetaRL pre-trained on GBWM problems delivers near-optimal dynamic strategies in 0.01s achieving 97.8% of DP optimal utility and handles larger problems where DP fails.
-
Scalable Neighborhood-Based Multi-Agent Actor-Critic
MADDPG-K scales centralized critics in multi-agent RL by limiting each critic to k-nearest neighbors under Euclidean distance, yielding constant input size and competitive performance.
-
When Forecast Accuracy Fails: Rank Correlation and Decision Quality in Multi-Market Battery Storage Optimization
Rank correlation (Kendall tau) of price forecasts, not mean absolute error, determines intraday dispatch value for multi-market battery storage, with tau above 0.85-0.95 capturing 97-100% of perfect-foresight revenue.
-
Load constrained wind farm flow control through multi-objective multi-agent reinforcement learning
A multi-agent RL system using Independent Soft Actor-Critic and a local-inflow surrogate for damage-equivalent loads learns policies that raise wind-farm power while respecting explicit load-increase limits.
-
RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields
A deep RL vulnerability-prediction policy trained in semantic embedding space finds up to 23% more unique robot manipulation failures than vision-language baselines and enables more efficient fine-tuning.
-
Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
DIPPER uses bi-level optimization and DPO to train the higher-level policy from stationary preference comparisons and value regularization, claiming up to 40% gains on robotic navigation and manipulation tasks while introducing metrics for non-stationarity and infeasible subgoals.
-
Behavior Regularized Offline Reinforcement Learning
Behavior-regularized actor-critic methods achieve strong offline RL results with simple regularization, rendering many recent technical additions unnecessary.
-
Disentangled Skill Embeddings for Reinforcement Learning
Disentangled Skill Embeddings (DSE) is a variational inference framework for multi-task RL using shared parameters and task-specific latent embeddings for generalization to unseen conditions and as skills in hierarchical RL.
-
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
Two plug-and-play strategies — per-timestep advantage weighting and advantage-based trajectory replay — improve diffusion RLHF sample efficiency up to 6× across five reward functions.
-
VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
VLM-AR3L learns absolute and relative reward models from VLM preference labels to improve RL on control, manipulation, and Minecraft tasks.
-
FactorLibrary: From Polynomials to Circuits via Recursive Subgoals
FactorLibrary stores reusable subexpressions to help RL agents (especially PPO+MCTS top-down) find certified optimal arithmetic circuits for polynomials up to complexity 8 at 91.8% success rate.
-
InSight: Self-Guided Skill Acquisition via Steerable VLAs
InSight enables autonomous acquisition of manipulation primitives in VLAs via automated segmentation for steerability and a VLM-guided data flywheel that generates and integrates new demonstrations for tasks like pouring and sweeping.
-
Forecasting what Matters: Decision-Focused RL for Controlled EV Charging with Unknown Departure Times
Decision-focused RL framework trains forecaster end-to-end with RL agent for EV charging under unknown departure times, yielding up to 14% higher reward and 55% less unsupplied energy than separate training.
-
Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
Space-sampled Value Decay is proposed as a simple forgetting mechanism for DQN and SAC modifications that shows positive but limited effects on returns in non-stationary RL environments.
-
Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models
MLLMs show stochastic collapse with top-1 probabilities up to 97% and low randomness indices when choosing among equivalent options.
-
Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning
Eligibility traces in deep RL create a peak bias by amplifying distal TD errors into gradient shocks that fixed-step SGD cannot normalize, leading to overestimation of peak-reward trajectories and a mechanistic account of the peak-end rule.
-
Coordination Architecture Shapes Continuous Demand Response Outcomes in Building Districts
In a 25-building district simulation, the hybrid MPC-SAC architecture delivered the strongest balance of load tracking accuracy (4.8% NMBE), thermal comfort (16.8% exceedance), and lowest spatial variability compared to centralized MPC, decentralized SAC, MAPPO, and rule-based control.
-
A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
PD-RSAC, a distributionally robust SAC variant with GCN encoder and MILP constraint projection, reports $1.22M net profit on an NYC taxi-based EV simulator while achieving zero feeder violations, outperforming heuristic and other RL baselines.
-
Hierarchical RL-MPC Control for Dynamic Wake Steering in Wind Farms
A hierarchical RL-MPC framework for dynamic wake steering in wind farms delivers 23% power gain over baseline on a three-turbine case while outperforming idealized MPC with perfect state knowledge and offering safer training than direct RL.
-
Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations
Pretraining Soft Actor-Critic agents via behavior cloning on PyWake-generated expert trajectories in WindGym simulations eliminates the initial learning phase for 2x2 wind farm control and yields final performance exceeding a lookup-table baseline.
-
Hierarchical DLO Routing with Reinforcement Learning and In-Context Vision-language Models
Hierarchical framework pairs in-context VLMs for high-level plan synthesis with RL-trained low-level skills and failure recovery to reach 92% success on long-horizon DLO routing across varied scenes and language inputs.
-
Polychromic Objectives for Reinforcement Learning
Introduces polychromic objectives adapted into PPO via vine sampling and modified advantages, showing higher success rates and better coverage under perturbations on BabyAI, Minigrid, and algorithmic tasks.
-
Gymnasium: A Standard Interface for Reinforcement Learning Environments
Gymnasium establishes a standardized API for RL environments to improve interoperability, reproducibility, and ease of development in reinforcement learning.
-
Potential-Based Advice for Stochastic Policy Learning
Potential-based reward shaping preserves optimality of stochastic policies and accelerates learning when added to soft Q-learning and advantage actor-critic algorithms.
-
Uncertainty-aware Model-based Policy Optimization
Introduces a framework that learns an uncertainty-aware dynamics model and optimizes the policy via automatic differentiation through the model, reporting competitive asymptotic performance with significantly lower sample complexity than baselines on continuous control benchmarks.
-
Calibrated Model-Based Deep Reinforcement Learning
Augmenting model-based RL agents with calibrated predictive uncertainties improves planning, sample efficiency, and exploration on continuous control tasks.
-
Dynamic multi-agent deep reinforcement learning-based pricing and incentivization approach in multimodal transportation networks
Multi-agent DRL framework shows dynamic incentives and pricing can cut commuter costs ~20%, emissions ~10%, and double public transport profit in simulated morning peak scenarios.
-
Multi-Head Attention-Based Feature Extractor Integration with Soft Actor-Critic for Porosity Prediction and Process Parameter Optimization in Additive Manufacturing
Integrates multi-head attention with SAC for faster convergence in optimizing additive manufacturing parameters to minimize porosity, outperforming DQN, PPO, TD3, and vanilla SAC.
-
Learning to Adapt: Representation-Based Reinforcement Learning for Multi-Task Skill Transfer
RepMT-SAC uses spectral MDP decomposition to build a task-agnostic value-function core plus minimal task adjustment, yielding up to 30% better performance than baselines on quadcopter trajectory tasks with zero-shot in-distribution and few-shot out-of-distribution transfer.
-
Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
ULPS integrates A*-generated symbolic trajectories, fine-tuned BERT priors, MC dropout uncertainty, and entropy-based blending into PPO, reporting over 9% accuracy gains and better sample efficiency on the MiniGridUnlockPickup benchmark.
-
Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient
SDPG is a new on-policy visual RL algorithm that estimates gradients via stochastic perturbations of rollouts, achieving faster training and lower memory use than baselines on visual MuJoCo tasks while adding new robotics benchmarks and sim-to-real results.
-
An Aircraft Upset Recovery System with Reinforcement Learning
A SAC-based reinforcement learning controller for aircraft upset recovery is judged by domain experts to produce more desirable behavior than conventional control methods.
-
Transformer-Guided Deep Reinforcement Learning for Optimal Takeoff Trajectory Design of an eVTOL Drone
Transformer-guided DRL cut training time steps to 25% of vanilla DRL while reaching 97.2% of optimal energy consumption for eVTOL takeoff versus 96.1%.
-
Neural Embedding for Physical Manipulations
Generative model with normalized pairwise distance constraint discovers output space topologies from sparse data and outperforms GANs and VAEs by avoiding mode collapse.