Single-timescale actor-critic with STORM momentum and a recent-sample buffer achieves optimal O(ε^{-2}) sample complexity for ε-optimal policies in finite discounted MDPs.
Elementary analysis of policy gradient methods
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
citation-role summary
other 1
citation-polarity summary
fields
cs.LG 4roles
other 1polarities
unclear 1representative citing papers
Generalizes non-uniform smoothness to affine curvature in objective value and derives linear convergence rates for sign GD, RMSProp, and Adam under gradient domination, plus a lower bound showing they beat AdaGrad and GD.
Certain errors in proxy rewards for policy gradient methods can be benign or beneficial by preventing policies from stalling on outputs with mediocre ground truth rewards, enabling improved RLHF metrics and reward design insights.
Shows entropy coupling limits DSAC on discrete tasks and introduces a generalized actor-critic framework with m-step critics and novel entropy-regularized objectives that perform robustly on Atari.
citing papers explorer
-
Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum
Single-timescale actor-critic with STORM momentum and a recent-sample buffer achieves optimal O(ε^{-2}) sample complexity for ε-optimal policies in finite discounted MDPs.
-
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
Generalizes non-uniform smoothness to affine curvature in objective value and derives linear convergence rates for sign GD, RMSProp, and Adam under gradient domination, plus a lower bound showing they beat AdaGrad and GD.
-
When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Certain errors in proxy rewards for policy gradient methods can be benign or beneficial by preventing policies from stalling on outputs with mediocre ground truth rewards, enabling improved RLHF metrics and reward design insights.
-
Dissecting Discrete Soft Actor-Critic: Limitations and Principled Alternatives
Shows entropy coupling limits DSAC on discrete tasks and introduces a generalized actor-critic framework with m-step critics and novel entropy-regularized objectives that perform robustly on Atari.