RECORD
Sequential memory improves sample and memory efficiency in Episodic Control
Machine Learning · Submitted 29 Dec 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Here, we demonstrate that including a bias in the acquired memory content derived from the order of episodic sampling improves both the sample and memory efficiency of an episodic control algorithm.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
State of the art deep reinforcement learning algorithms are sample inefficient due to the large number of episodes they require to achieve asymptotic performance. Episodic Reinforcement Learning (ERL) algorithms, inspired by the mammalian hippocampus, typically use extended memory systems to bootstrap learning from past events to overcome this sample-inefficiency problem. However, such memory augmentations are often used as mere buffers, from which isolated past experiences are drawn to learn from in an offline fashion (e.g., replay). Here, we demonstrate that including a bias in the acquired memory content derived from the order of episodic sampling improves both the sample and memory efficiency of an episodic control algorithm. We test our Sequential Episodic Control (SEC) model in a foraging task to show that storing and using integrated episodes as event sequences leads to faster learning with fewer memory requirements as opposed to a standard ERL benchmark, Model-Free Episodic Control, that buffers isolated events only. We also study the effect of memory constraints and forgetting on the sequential and non-sequential version of the SEC algorithm. Furthermore, we discuss how a hippocampal-like fast memory system could bootstrap slow cortical and subcortical learning subserving habit formation in the mammalian brain.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | State of the art deep reinforcement learning algorithms are sample inefficient due to the large number of episodes they require to achieve asymptotic performance. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| s1 | Episodic Reinforcement Learning (ERL) algorithms, inspired by the mammalian hippocampus, typically use extended memory systems to bootstrap learning from past events to overcome this sample-inefficiency problem. | background 1.00 | 0.04 | 0.03 | asserted 0.96 |
| s2 | However, such memory augmentations are often used as mere buffers, from which isolated past experiences are drawn to learn from in an offline fashion (e.g., replay). | gap 0.70 | 0.05 | 0.02 | asserted 0.70 |
| s3 | Here, we demonstrate that including a bias in the acquired memory content derived from the order of episodic sampling improves both the sample and memory efficiency of an episodic control algorithm. | contribution 0.72 | 0.96 | 0.92 | asserted 1.00 |
| s4 | We test our Sequential Episodic Control (SEC) model in a foraging task to show that storing and using integrated episodes as event sequences leads to faster learning with fewer memory requirements as opposed to a standard ERL benchmark, Model-Free Episodic Control, that buffers isolated events only. | result 0.74 | 0.97 | 0.40 | asserted 1.00 |
| s5 | We also study the effect of memory constraints and forgetting on the sequential and non-sequential version of the SEC algorithm. | contribution 0.84 | 0.91 | 0.03 | unclear 0.65 |
| s6 | Furthermore, we discuss how a hippocampal-like fast memory system could bootstrap slow cortical and subcortical learning subserving habit formation in the mammalian brain. | interpretation 0.95 | 0.41 | 0.04 | hedged possibility 0.98 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
State of the art deep reinforcement learning algorithms are sample inefficient due to the large number of episodes they require to achieve asymptotic performance. Episodic Reinforcement Learning (ERL) algorithms, inspired by the mammalian hippocampus, typically use extended memory systems to bootstrap learning from past events to overcome this sample-inefficiency problem. However, such memory augmentations are often used as mere buffers, from which isolated past experiences are drawn to learn from in an offline fashion (e.g., replay). Here, we demonstrate that including a bias in the acquired memory content derived from the order of episodic sampling improves both the sample and memory efficiency of an episodic control algorithm. We test our Sequential Episodic Control (SEC) model in a foraging task to show that storing and using integrated episodes as event sequences leads to faster learning with fewer memory requirements as opposed to a standard ERL benchmark, Model-Free Episodic Control, that buffers isolated events only. We also study the effect of memory constraints and forgetting on the sequential and non-sequential version of the SEC algorithm. Furthermore, we discuss how a hippocampal-like fast memory system could bootstrap slow cortical and subcortical learning subserving habit formation in the mammalian brain.
Human review
How the literature uses it
1 arXiv paper cite this in 8 sentences
Papers citing it · 5 sentences without a confident label
Uses its method or tool 1 paper
-
This study employs the Sequential Episodic Control (SEC) algorithm to model foraging agents capable of storing past experienced events and learning from them [ref] .
High-fidelity social learning via shared episodic memories enhances collaborative foraging through mnemonic convergence · Sequential Episodic Control · 0.99
What it cites · 12 sentences without a confident label
Uses the method or tool of 2 papers
-
As in previous approaches [ref] , we use an embedding function $ϕ$ to build compressed feature representations $s_t$ of the observations $o_t$ obtained from the environment $s_t=ϕ(o_t)$.
Model-Free Episodic Control · Sequential Episodic Control · 0.98
-
We have implemented the SEC model on the AnimalAI testbed [ref] .
The Animal-AI Environment: Training and Testing Animal-Like Artificial Cognition · Results · 0.98
Compares with 1 paper
-
We ran 20 simulations of 5000 episodes comparing the SEC algorithm against a control version of the model (NSEC) that does not incorporate the sequential bias in action selection (i.e., $B$ matrix being all 1's and not being updated), and against a baseline episodic control benchmark [ref] , Model-Free Episodic Control (MFEC).
Model-Free Episodic Control · Results · 0.95
Cites as background 3 papers
-
Episodic Control algorithms, on the other hand, propose an alternative approach that gets rid of the gradient-based methods and directly uses the stored events in memory for action selection [ref] .
Model-Free Episodic Control · Introduction · 0.91
-
Episodic Backward Update propagates the value of a state to its previous states after sampling a complete episode [ref] .
Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update · Related work · 0.90
-
Beating world-champions in games like chess and Go [ref] , or more recently, in complex real-time games like Starcraft [ref] and DOTA [ref] are signatures of such human-like capacity.
Dota 2 with Large Scale Deep Reinforcement Learning · Introduction · 0.86
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.