RECORD
Hindsight Experience Replay Accelerates Proximal Policy Optimization
Machine Learning · Submitted 29 Oct 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Here, we show that HER can dramatically accelerate proximal policy optimization (PPO), an on-policy reinforcement learning algorithm, when tested on a custom predator-prey environment.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Hindsight experience replay (HER) accelerates off-policy reinforcement learning algorithms for environments that emit sparse rewards by modifying the goal of the episode post-hoc to be some state achieved during the episode. Because post-hoc modification of the observed goal violates the assumptions of on-policy algorithms, HER is not typically applied to on-policy algorithms. Here, we show that HER can dramatically accelerate proximal policy optimization (PPO), an on-policy reinforcement learning algorithm, when tested on a custom predator-prey environment.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Hindsight experience replay (HER) accelerates off-policy reinforcement learning algorithms for environments that emit sparse rewards by modifying the goal of the episode post-hoc to be some state achieved during the episode. | background 0.99 | 0.07 | 0.07 | asserted 1.00 |
| s1 | Because post-hoc modification of the observed goal violates the assumptions of on-policy algorithms, HER is not typically applied to on-policy algorithms. | background 0.78 | 0.05 | 0.03 | asserted 0.95 |
| s2 | Here, we show that HER can dramatically accelerate proximal policy optimization (PPO), an on-policy reinforcement learning algorithm, when tested on a custom predator-prey environment. | result 0.62 | 0.96 | 0.93 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Hindsight experience replay (HER) accelerates off-policy reinforcement learning algorithms for environments that emit sparse rewards by modifying the goal of the episode post-hoc to be some state achieved during the episode. Because post-hoc modification of the observed goal violates the assumptions of on-policy algorithms, HER is not typically applied to on-policy algorithms. Here, we show that HER can dramatically accelerate proximal policy optimization (PPO), an on-policy reinforcement learning algorithm, when tested on a custom predator-prey environment.
Human review
References
How the literature uses it
1 arXiv paper cite this in 5 sentences
Papers citing it · 4 sentences without a confident label
Uses its method or tool 1 paper
-
We used a high-quality implementation of PPO [ref] with a custom HER implementation, as describe previously [ref] .
Maximum Entropy Hindsight Experience Replay · Reinforcement Learning · 1.00
What it cites · 5 sentences without a confident label
Cites as background 4 papers
-
[ref] derived a version of HER for policy gradient methods and applied their algorithm in environments that had discrete action spaces.
Hindsight policy gradients · Previous Work · 0.88
-
[ref] combined HER with TRPO.
Hindsight Trust Region Policy Optimization · Previous Work · 0.92
-
Reinforcement learning (RL) has been remarkably successful at solving complex tasks [ref] .
Playing Atari with Deep Reinforcement Learning · Introduction · 0.99
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.