RECORD
Utilizing Skipped Frames in Action Repeats via Pseudo-Actions
Machine Learning · Submitted 7 May 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. For discrete control tasks, pseudo-actions are computed from learned action embeddings. This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. | background 1.00 | 0.06 | 0.02 | asserted 0.99 |
| s1 | This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. | background 0.87 | 0.09 | 0.03 | asserted 0.98 |
| s2 | Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. | background 0.93 | 0.11 | 0.03 | hedged possibility 0.53 |
| s3 | In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. | contribution 1.00 | 0.81 | 0.12 | asserted 0.99 |
| s4 | The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. | method 0.59 | 0.72 | 0.20 | asserted 1.00 |
| s5 | Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. | method 0.99 | 0.79 | 0.06 | asserted 1.00 |
| s6 | For discrete control tasks, pseudo-actions are computed from learned action embeddings. | method 1.00 | 0.72 | 0.04 | asserted 1.00 |
| s7 | This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. | method 0.75 | 0.22 | 0.06 | asserted 0.53 |
| s8 | We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym. | result 0.65 | 0.95 | 0.41 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. For discrete control tasks, pseudo-actions are computed from learned action embeddings. This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 1 sentence without a confident label
Cites as background 1 paper
-
Soft Actor-Critic (SAC) [ref] is a popular algorithm for continuous control tasks.
Soft Actor-Critic Algorithms and Applications · Soft Actor-Critic · 0.89
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.