A trajectory-wise control variate estimator removes both action-level and future-trajectory variance in policy gradients, and the natural time-ordering is proven optimal under exact critic assumptions.
Policy Optimization with Stochastic Mirror Descent
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency. We prove that the proposed $\mathtt{VRMPO}$ needs only $\mathcal{O}(\epsilon^{-3})$ sample trajectories to achieve an $\epsilon$-approximate first-order stationary point, which matches the best sample complexity for policy optimization. The extensive experimental results demonstrate that $\mathtt{VRMPO}$ outperforms the state-of-the-art policy gradient methods in various settings.
fields
cs.LG 1years
2019 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods
A trajectory-wise control variate estimator removes both action-level and future-trajectory variance in policy gradients, and the natural time-ordering is proven optimal under exact critic assumptions.