REVIEW 9 cited by
DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Visual reinforcement learning (RL) has shown promise in continuous control tasks. Despite its progress, current algorithms are still unsatisfactory in virtually every aspect of the performance such as sample efficiency, asymptotic performance, and their robustness to the choice of random seeds. In this paper, we identify a major shortcoming in existing visual RL methods that is the agents often exhibit sustained inactivity during early training, thereby limiting their ability to explore effectively. Expanding upon this crucial observation, we additionally unveil a significant correlation between the agents' inclination towards motorically inactive exploration and the absence of neuronal activity within their policy networks. To quantify this inactivity, we adopt dormant ratio as a metric to measure inactivity in the RL agent's network. Empirically, we also recognize that the dormant ratio can act as a standalone indicator of an agent's activity level, regardless of the received reward signals. Leveraging the aforementioned insights, we introduce DrM, a method that uses three core mechanisms to guide agents' exploration-exploitation trade-offs by actively minimizing the dormant ratio. Experiments demonstrate that DrM achieves significant improvements in sample efficiency and asymptotic performance with no broken seeds (76 seeds in total) across three continuous control benchmark environments, including DeepMind Control Suite, MetaWorld, and Adroit. Most importantly, DrM is the first model-free algorithm that consistently solves tasks in both the Dog and Manipulator domains from the DeepMind Control Suite as well as three dexterous hand manipulation tasks without demonstrations in Adroit, all based on pixel observations.
Forward citations
Cited by 9 Pith papers
-
Value-Based Deep RL Scales Predictably
For SAC, BRO, and PQL, data and compute requirements follow fitted power laws in the updates-to-data ratio, and these laws extrapolate to larger runs.
-
V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
V-Simba, a visual RL architecture combining layer normalization, weight decay, and a distributional critic, matches or outperforms complex baselines on 29 continuous control tasks while using less compute.
-
The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Introduces LEAST, an adaptive early-episode-stopping rule for off-policy deep RL that improves learning efficiency on MuJoCo and DeepMind Control benchmarks.
-
Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulation
By summing multi-view features and adding single-view features as actor-critic augmentations, MAD produces manipulation policies that learn faster and tolerate missing cameras in simulation.
-
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
A learnable wavelet activation with dynamic capacity injection and slope regularization improves plasticity retention in continual learning.
-
Reinforcement Learning for the Full Strawberry Harvesting Process: Obstacle Separation, Detachment, and Placement
A single PPO policy, trained with domain randomization and heuristic phase gating, performs obstacle separation, detachment, and placement, reaching 82% real-world strawberry-harvest success with zero-shot sim-to-real...
-
A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
Forget and Grow (FoG) combines decaying replay weights for old experiences with progressive critic-network expansion to improve continuous-control reinforcement learning, beating BRO, SimBa, and TD-MPC2 on most of 41 ...
-
Residual Reward Models for Preference-based Reinforcement Learning
Combining a hand-designed or learned prior reward with a preference-trained residual improves sample efficiency and final performance in preference-based reinforcement learning.
-
Zero-Shot Sim-to-Real Reinforcement Learning for Fruit Harvesting
A sim-to-real reinforcement learning pipeline with domain randomization lets a Franka Panda robot pick a target strawberry from dense clusters in a lab, reaching a 50% success rate with five distractors.
Discussion (0). Continue with ORCID to comment.