Pith. sign in

REVIEW 4 major objections 6 minor 126 references

Towards General Purpose Robots at Scale: Lifelong Learning and Learning to Use Memory

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This thesis argues that two mechanisms—timestep-conditioned deep generative replay and attention supervision from memory dependency pairs—let robot policies learn continually and use long-term memory, and it reports state-of-the-art…

desk verdict An honest, well-documented thesis compiling two prior papers; the new experiments are real but thin, and the replay-labeling assumption deserves a control. read the letter →

arxiv 2501.10395 v1 pith:E2PEKZAS submitted 2024-12-28 cs.LG cs.AIcs.RO

William Yue This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords lifelonglearningdeepgenerativereplaycatastrophicforgettingtimestep-conditionedgenerationmemorydependencypairsattentionsupervisionpartiallyobservableenvironmentsimitation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis targets two obstacles to robots that operate in homes for hours or days: they must keep learning new skills without forgetting old ones, and they must use memory of past observations when deciding. For the first obstacle, it introduces t-DGR, a deep generative replay method whose diffusion generator produces state observations conditioned on the trajectory timestep, giving every timestep equal replay coverage without autoregressive error compounding. For the second, it introduces memory dependency pairs (p, q) saying that an observation at time p must be recalled when acting at time q, and AttentionTuner, which turns those pairs into a binary expert attention matrix and trains a Transformer's first-layer attention head to match it. The thesis reports that t-DGR achieves state-of-the-art average success rate on Continual World CW10, CW20, and a blurry-boundary variant, and that AttentionTuner raises success rates from 20.8% to 99.8% on Mortar Mayhem and from 53.2% to 99.9% on Hallway, with benefits visible even when only 0.1% of trajectories are annotated.

What carries the argument

The central object for t-DGR is the timestep-conditioned trajectory generator: a diffusion U-net that takes the trajectory timestep j, via a sinusoidal positional embedding added at each resolution level, and outputs a state observation, with actions supplied by the previous policy; its role is to give equal sample coverage at every timestep, reducing sample complexity from Θ(n log n + mn log log n) for i.i.d. sampling to Θ(mn). For AttentionTuner the central object is the memory dependency pair (p, q), converted into an expert self-attention matrix E ∈ {0,1}^{n×n} with E[q,p]=1, and a binary cross-entropy memory loss L(A,E) applied to one head of the first Transformer layer, combined with the imitation loss as L = L_IL + λ L(A,E) with λ=10. The Long-term Memory Benchmark (LTMB), built on Minigrid, provides Hallway, Ordering, and Counting tasks that require recalling specific past observations.

What would settle it

Train a behavioral-cloning policy on a task where the correct action depends on the immediately previous state, for example a velocity-dependent control task or tracking a moving object, once on real trajectories and once on t-DGR replayed trajectories of the same tasks, and compare success rates; a large drop for the replay-trained policy would falsify the claim that temporally incoherent replay suffices.

Watch

Extended reading notes

Core claim

The core discovery is that both lifelong learning and memory use in sequential decision-making can be advanced by replacing unstructured replay and undirected attention with structure that matches how the learner consumes data. t-DGR trains a timestep-conditioned diffusion model to generate each state observation of a past trajectory independently, then labels each state with the old policy's action; because the learner is a behavioral-cloning policy trained on state-action pairs, the replayed trajectories need not be temporally coherent, only well-covered per timestep. This yields state-of-the-art average success rate on Continual World CW10, CW20, and the blurry-boundary BB10, and it outperforms i.i.d. generative replay (DGR) and autoregressive trajectory generation (CRIL) when a diffusion generator is used. AttentionTuner extends imitation learning demonstrations with memory dependency pairs (p, q), encodes them as a binary self-attention matrix, and adds a cross-entropy loss that pulls one head of the first Transformer layer toward the expert's recall pattern; together with the new Long-term Memory Benchmark (LTMB), it shows reliable gains on four of six memory tasks, often turning bimodal near-zero-or-perfect outcomes into consistently near-perfect ones.

Load-bearing premise

For t-DGR, the whole method rests on the premise that a policy trained on state-action pairs whose states are generated independently per timestep, with no realistic transitions between them, can learn as well as it would from real trajectories; if the correct action at a step depends on dynamics or the preceding state, the replayed signal is corrupted.

Editorial extensions

If this is right

  • If t-DGR is right, pseudo-rehearsal for decision-making no longer needs autoregressive trajectory generation: a diffusion generator conditioned on the trajectory timestep gives state-of-the-art average success rate on Continual World CW10, CW20, and the blurry-boundary BB10.
  • If AttentionTuner is right, imitation learning in POMDPs can include a human-articulable memory signal: labeling which past observations should be recalled at each decision improved success rates from 20.8% to 99.8% on Mortar Mayhem and from 53.2% to 99.9% on Hallway.
  • AttentionTuner's benefit persists when only 0.1% of demonstration trajectories carry memory annotations, and performance holds with annotations on 10% of trajectories, making the annotation cost low enough for practical use.
  • The new Long-term Memory Benchmark (LTMB) gives the community three procedurally generated tasks, Hallway, Ordering, and Counting, for measuring long-term episodic memory in decision-making agents.
  • Because t-DGR handles blurry task boundaries and bounded memory, it is positioned as compatible with real-world continual learning conditions rather than only clean task segments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the thesis does not test: applying t-DGR's timestep-conditioned replay to vision-based policies or to latent state representations, where the diffusion generator would operate in a learned latent space rather than on raw proprioceptive vectors.
  • The attention-supervision idea could generalize beyond Transformers: the paper itself suggests RNNs and state-space models as targets, so a concrete testable step would be converting memory dependency pairs into a hidden-state reconstruction loss for an LSTM on the same Memory Gym and LTMB tasks; if the gains persist, the mechanism is about memory supervision, not the Transformer's attention matri
  • The two contributions are complementary: t-DGR could use the memory mechanism learned by AttentionTuner to decide which past experiences are worth replaying, which the thesis names as future work; one testable integration is to weight replay trajectories by the agent's learned attention importance.
  • AttentionTuner's framing of memory dependency pairs as a human-feedback modality suggests a practical path for real robots: a human annotating a few demonstrations with 'remember this' links could steer a deployed policy's attention without extra robot trials, though the thesis only simulates expert annotations in gridworlds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript is a bachelor's thesis presenting two methods for robot learning at scale. The first, t-DGR, is a trajectory-based deep generative replay method for continual imitation learning. Instead of generating full trajectories autoregressively or sampling states independently, t-DGR trains a diffusion generator conditioned on the trajectory timestep and labels generated states with the previous policy's actions. It is evaluated on Continual World CW10, CW20, and a modified blurry-boundary benchmark BB10, where the authors claim state-of-the-art average success rates. The second contribution, AttentionTuner, introduces memory dependency pairs (p, q) that indicate which past observation should be recalled when choosing an action at a later time. These pairs are converted into an expert self-attention matrix, and a binary cross-entropy memory loss is added to the imitation learning loss of a causal Transformer. AttentionTuner is evaluated on Memory Gym and the newly proposed Long-term Memory Benchmark (LTMB), with claims of significant improvements on four of six tasks and reduced annotation cost. The thesis also discusses future directions and includes extensive appendices on hyperparameters, architectures, ablations, and statistical tests.

Significance. If the results hold, the thesis makes two useful contributions. t-DGR offers a simple, non-autoregressive replay mechanism that avoids the compounding-error problem of autoregressive trajectory generation, and the sample-complexity argument using the Double Dixie Cup theorem is a principled motivation for timestep-conditioned generation. AttentionTuner is one of the first attempts to inject human-specified memory dependencies into Transformer-based policies, and the annotation-cost ablations, which show benefits with as few as 0.1% of trajectories annotated, are practically informative. The manuscript is unusually transparent: code is released for both methods, hyperparameters and architectures are documented in detail, and statistical tests are reported. These strengths are significant, but the evaluation has important gaps: the replay-label reliability premise of t-DGR is not directly tested, the CW10 state-of-the-art claim is weakened by overlapping confidence intervals, the CW20 comparison treats methods asymmetrically, and AttentionTuner is compared only to a vanilla Transformer and fails to improve two of six tasks.

major comments (4)
  1. [Algorithm 1, Lines 10-11; Section 2.4] The replay signal used by t-DGR is generated by the learner's own policy. In each generated trajectory, states are sampled from the timestep-conditioned diffusion generator, which is trained only on state observations, and then labeled by the previous policy pi(t-1). If the policy has already drifted on a previous task, the replayed actions drift with it, providing no corrective signal toward expert behavior. This is the exact failure mode replay is meant to prevent, and it is not isolated experimentally. The comparisons against DGR and CRIL share the same policy-labeling scheme, so they measure only timestep-coverage effects. Please add a controlled experiment that separates state-generation quality from label reliability, for example by comparing t-DGR with replay labels from the expert policy against t-DGR with replay labels from the evolving learner policy on the same generated states.
  2. [Section 2.4, 'temporally coherent' paragraph] The central premise that a policy trained on temporally incoherent generated state-action pairs can match a policy trained on real trajectories is asserted but not validated. The paper argues that this is acceptable because the learner is trained on state-action pairs rather than full trajectories, but if action selection depends on dynamics or on temporal context encoded across consecutive states, the replay signal is corrupted. The reader's concern is concrete: there is no experiment comparing a policy trained on generated incoherent trajectories against one trained on real trajectories, or against a policy trained on generated states that are also temporally consistent. Please include such an experiment or clearly state the assumption as a limitation with supporting evidence.
  3. [Table 2.1(a); Section 2.5.3] The state-of-the-art claim on CW10 is not supported by the reported confidence intervals. t-DGR achieves 81.9±3.3 while PackNet achieves 81.4±3.7; the 90% intervals overlap substantially. The significant improvement over existing pseudo-rehearsal methods is only demonstrated on CW20, where t-DGR's 83.9±3.0 does not overlap DGR's 74.1±4.1. Additionally, the CW20 comparison is asymmetric: PackNet is evaluated with N=20 (tasks 1-20 with the second repetition treated as distinct), while t-DGR and other methods are evaluated with N=10. Reporting an average over 10 tasks for one method and over 20 tasks for another undermines the head-to-head comparison. Please report a consistent evaluation protocol or justify why the asymmetry does not affect the conclusion.
  4. [Table 3.1; Appendix A.10] The claimed general improvement of AttentionTuner is based on only a single baseline. AttentionTuner significantly improves success rates on Mortar Mayhem, Mystery Path, Hallway, and Ordering, but the improvements on Searing Spotlights (p=0.546) and Counting (p=0.261) are not significant, and the Counting result is only 6.5 vs 6.0. Since the vanilla Transformer is the only comparator, and since two of six tasks show no significant gain, the conclusion that AttentionTuner is an effective general method for learning memory mechanisms is stronger than the evidence. At minimum, discuss the two non-significant tasks in the main text with the p-values, and consider adding a second baseline such as a recurrent policy or a Transformer with an auxiliary memory objective.
minor comments (6)
  1. [Section 3.5.3] The sentence referencing robustness to perturbed endpoints says 'Figure 3.2, 3.3' but should refer to Table 3.2 and Table 3.3, which contain the imprecise annotation results.
  2. [Appendix A.7.3] The description of Searing Spotlights contains a duplicated sentence: 'A single coin was used to unlock the exit' appears twice with slightly different wording.
  3. [Section 3.5.2] There is a typo in the heading 'AttentionT uner promotes convergence to solutions with better generalizabiltiy'; it should be 'generalizability'. Similar spacing issues appear as 'AttentionT uner' in several places.
  4. [Appendix A.7] The phrase 'this process can be made relatively seemless for human experts' contains a typo; it should be 'seamless'.
  5. [Section 2.2.2] In the citation for CRIL, the page range appears as '6747-5754', which is likely a typo for '6747-6754'.
  6. [Table 3.1] In the Counting row, the vanilla Transformer result is listed as '6 ±0.7' without a trailing decimal; for consistency with the other entries it should be '6.0 ± 0.7'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the replay-labeling and attention-supervision mechanisms are evaluated against external environment rollouts, making the central claims empirically grounded rather than self-referential.

full rationale

I examined both derivation chains in the thesis. For t-DGR, the replay procedure is a training mechanism, not a derivation of the reported results: average success rates are measured by executing the learned policy in the Continual World environments (Section 2.5.1), which is external to the replay loss. The generator is trained on state observations from past tasks via the diffusion loss (Eq. 2.6), and the policy labels generated states with its own actions (Algorithm 1, Lines 10-11). This is a potential validity concern because replay labels may encode policy drift rather than expert behavior, but it is not circular: the claimed performance metric is not defined in terms of the replay data, and the construction does not force a particular success rate. The sample-complexity argument uses the Double Dixie Cup theorem, an external mathematical result. For AttentionTuner, the expert attention matrix E is defined from human/task-specific memory dependency pairs, not from the model's own outputs; the memory loss (Eq. 3.4) supervises attention directly, and success rates are again measured by environment rollouts. The choice of lambda=10 based on observed performance is a hyperparameter-selection concern, not a self-definitional equivalence. The self-citations [117,118] merely state the provenance of the chapters and do not supply load-bearing uniqueness or correctness arguments. No step was found where a claimed result equals its input by construction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard ML practices (diffusion models, behavioral cloning, transformer attention) plus two domain assumptions: that temporally incoherent replay states suffice for policy learning, and that simulated memory-dependency annotations represent human annotation. Free parameters include the replay ratio, the memory-loss weight, and the oEWC Fisher multiplier.

free parameters (3)
  • replay ratio r = 0.9
    Chosen for all pseudo-rehearsal methods; ablation across 0.5-0.9 shows monotonic improvement, but the operating point is a hand-set hyperparameter.
  • memory loss scaling lambda = 10
    Set based on 'robust performance observed across various benchmark tasks' (Section 3.4); a hand-tuned balancing coefficient.
  • oEWC Fisher multiplier = 10^2
    Tuned over 10^-2 to 10^6 on the success rate metric (Appendix A.1.3); affects the EWC baseline comparison, not the proposed methods.
assumptions (4)
  • domain assumption Generated state observations need not be temporally coherent for the learner to benefit.
    Section 2.4 argues the learner is trained on state-action pairs, so trajectory coherence is unnecessary. If the policy must act in an environment where state transitions matter, this assumption could break.
  • standard math Standard diffusion probabilistic models approximate the reverse process well enough to sample useful replay states.
    Assumes DDPM training (Equation 2.6) yields a generator whose samples are on-distribution for replay; the paper's diffusion loss plot (Figure 2.3) shows generation quality degrades but not enough to hurt the learner.
  • domain assumption The expert attention matrix E derived from task structure represents the memory mechanism a human would use.
    Appendix A.7 simulates both expert demonstrations and memory dependency pair annotations via a computer program; human annotation is only measured for timing in A.13, not used in main experiments.
  • domain assumption Behavioral cloning from observation-action pairs is a valid proxy for learning in these POMDP tasks.
    Used to train both vanilla Transformer and AttentionTuner; the Counting task's near-zero success indicates this assumption is strained for that task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards General Purpose Robots at Scale: Lifelong Learning and Learning to Use Memory." pith.science (2026). https://pith.science/paper/E2PEKZAS

@misc{pith2026250110395,
  author       = {Pith},
  title        = {Pith review of: Towards General Purpose Robots at Scale: Lifelong Learning and Learning to Use Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E2PEKZAS}},
  note         = {Machine review of arXiv:2501.10395}
}
read the original abstract

The widespread success of artificial intelligence in fields like natural language processing and computer vision has not yet fully transferred to robotics, where progress is hindered by the lack of large-scale training data and the complexity of real-world tasks. To address this, many robot learning researchers are pushing to get robots deployed at scale in everyday unstructured environments like our homes to initiate a data flywheel. While current robot learning systems are effective for certain short-horizon tasks, they are not designed to autonomously operate over long time horizons in unstructured environments. This thesis focuses on addressing two key challenges for robots operating over long time horizons: memory and lifelong learning. We propose two novel methods to advance these capabilities. First, we introduce t-DGR, a trajectory-based deep generative replay method that achieves state-of-the-art performance on Continual World benchmarks, advancing lifelong learning. Second, we develop a framework that leverages human demonstrations to teach agents effective memory utilization, improving learning efficiency and success rates on Memory Gym tasks. Finally, we discuss future directions for achieving the lifelong learning and memory capabilities necessary for robots to function at scale in real-world settings.

Figures

Figures reproduced from arXiv: 2501.10395 by the authors.

Figure 1.1
Figure 1.1. A humanoid operating in a living room. Fields like language and vision have achieved remarkable progress in advancing artificial intelligence, largely through training massive neural networks on extensive and diverse datasets [43, 52, 15, 83]. However, these successes have not fully translated to robotics, where robots are often limited to impressive demos but in controlled environments. This contrasts sharply with … view at source ↗
Figure 1.2
Figure 1.2. The data flywheel for robotics. A long-standing goal for robot learning researchers is to ignite a data flywheel (Fig￾ure 1.2) through large-scale deployment of general-purpose robots [63, 123]. With more users, robots would collect more diverse data, enabling better models and more capable robots, which would attract even more users [25, 30]. But what does it take to achieve this data engine? To get general-purpose… view at source ↗
Figure 1.3
Figure 1.3. Roomba robot vacuuming the carpet of a residential game room. [PITH_FULL_IMAGE:figures/full_fig_p008_1_3.png] view at source ↗
Figures from the paper (11 more)
Figure 2.1
Figure 2.1. Figure 2.1: The first row presents a comparison of three generative methods for imitating [PITH_FULL_IMAGE:figures/full_fig_p012_2_1.png]
Figure 2.2
Figure 2.2. Figure 2.2: The deep generative replay paradigm. The algorithm learns to generate [PITH_FULL_IMAGE:figures/full_fig_p016_2_2.png]
Figure 2.3
Figure 2.3. Figure 2.3: This table illustrates the ability of the diffusion model in t-DGR to generate [PITH_FULL_IMAGE:figures/full_fig_p026_2_3.png]
Figure 3.1
Figure 3.1. Figure 3.1: The red arrows indicate episodic memory dependencies labeled by an expert. [PITH_FULL_IMAGE:figures/full_fig_p029_3_1.png]
Figure 3.2
Figure 3.2. Figure 3.2: Architecture of the causal Transformer for sequential decision making modeling [PITH_FULL_IMAGE:figures/full_fig_p033_3_2.png]
Figure 3.3
Figure 3.3. Figure 3.3: Overview of LTMB and MemGym tasks, each represented by a sequence of [PITH_FULL_IMAGE:figures/full_fig_p036_3_3.png]
Figure 3.4
Figure 3.4. Figure 3.4: Median learning curves with interquartile range for Memory Gym and LTMB [PITH_FULL_IMAGE:figures/full_fig_p039_3_4.png]
Figure 3.5
Figure 3.5. Figure 3.5: Success rates and 90% confidence intervals for AttentionTuner training on [PITH_FULL_IMAGE:figures/full_fig_p041_3_5.png]
Figure 3
Figure 3. Figure 3: shows that it can take anywhere from 2 to 4 times more demonstrations to [PITH_FULL_IMAGE:figures/full_fig_p042_3.png]
Figure 3.6
Figure 3.6. Figure 3.6: Success rates for vanilla Transformer with different training data sizes. The [PITH_FULL_IMAGE:figures/full_fig_p043_3_6.png]
Figure 3
Figure 3. Figure 3: illustrates these results in a bar plot. [PITH_FULL_IMAGE:figures/full_fig_p077_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

126 extracted references · 46 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Uncertainty-based continual learning with adaptive regularization

    Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-based continual learning with adaptive regularization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alch´ e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.,

  3. [3]

    April: Active preference-learning based reinforcement learning, 2012

    Riad Akrour, Marc Schoenauer, and Mich` ele Sebag. April: Active preference-learning based reinforcement learning, 2012

    2012

  4. [4]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Computer Vision–ECCV 2018: 15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part III 15 , pages 144–161. Springer, 2018

    2018

  5. [5]

    Gradient based sample selection for online continual learning

    Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alch´ e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.,

  6. [6]

    Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy

    Cameron Allen, Aaron Kirtland, Ruo Yu Tao, Sam Lobel, Daniel Scott, Nicholas Petrocelli, Omer Gottesman, Ronald Parr, Michael L Littman, and George Konidaris. 47 Mitigating partial observability in sequential decision processes via the lambda discrepancy. arXiv preprint arXiv:2407.07333 , 2024

    work page Pith review arXiv 2024

  7. [7]

    URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ e562cd9c0768d5464b64cf61da7fc6bb-Paper.pdf

    2019

  8. [8]

    Rainbow memory: Continual learning with a memory of diverse samples

    Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8218–8227, 2021

    2021

Show all 126 references
  1. [9]

    Working memory

    Alan Baddeley. Working memory. Science, 255(5044):556–559, 1992

    1992

  2. [10]

    Learning long-term dependen- cies with gradient descent is difficult

    Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependen- cies with gradient descent is difficult. IEEE transactions on neural networks , 5(2): 157–166, 1994

    1994

  3. [11]

    Credit assignment through time: Alternatives to backpropagation

    Yoshua Bengio and Paolo Frasconi. Credit assignment through time: Alternatives to backpropagation. Advances in neural information processing systems , 6, 1993

    1993

  4. [12]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  5. [13]

    Recurrent memory decision transformer

    Arkadii Bessonov, Alexey Staroverov, Huzhenyu Zhang, Alexey K Kovalev, Dmitry Yudin, and Aleksandr I Panov. Recurrent memory decision transformer. arXiv preprint arXiv:2306.09459, 2023

    2023

  6. [14]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018

    2018 arXiv

  7. [15]

    π0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  8. [16]

    Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V

    Mikhail S. Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V. Sapunov. Memory transformer, 2021

    2021

  9. [17]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 48

    2005 arXiv

  10. [18]

    An interactive framework for learning continuous actions policies based on corrective feedback

    Carlos Celemin and Javier Ruiz del Solar. An interactive framework for learning continuous actions policies based on corrective feedback. Journal of Intelligent & Robotic Systems, pages 1–21, 2019. URL https://api.semanticscholar.org/ CorpusID:115908814

    2019

  11. [19]

    Dark experience for general continual learning: a strong, simple baseline

    Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems , 33:15920–15930, 2020

    2020

  12. [20]

    Yuying Chen, Congcong Liu, Lei Tai, Ming Liu, and Bertram E. Shi. Gaze training by modulated dropout improves imitation learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7756–7761, 2019. doi: 10.1109/IROS40897.2019.8967843

    2019

  13. [21]

    Abbeel, A

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org...

    2021

  14. [22]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS) , 2023

    2023

  15. [23]

    Mini- grid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Mini- grid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023

    2023 arXiv

  16. [24]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. 49

    2023

  17. [25]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014

    Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014

    2014

  18. [26]

    What are the differences between long-term, short-term, and working memory? Progress in brain research, 169:323–338, 2008

    Nelson Cowan. What are the differences between long-term, short-term, and working memory? Progress in brain research, 169:323–338, 2008

    2008

  19. [27]

    Turning the Flywheel: A Monograph to Accompany Good to Great

    Jim Collins. Turning the Flywheel: A Monograph to Accompany Good to Great . HarperCollins, New York, 2019. ISBN 978-0062933799

    2019

  20. [28]

    Deep transformer q-networks for partially observable reinforcement learning, 2022

    Kevin Esslinger, Robert Platt, and Christopher Amato. Deep transformer q-networks for partially observable reinforcement learning, 2022

    2022

  21. [29]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780–8794, 2021

    2021

  22. [30]

    Andrej karpathy on Tesla AI data en- gine

    Lex Fridman and Andrej Karpathy. Andrej karpathy on Tesla AI data en- gine. YouTube video, October 2022. URL https://www.youtube.com/watch? v=zPH5O8hRfMA. Clip from the Lex Fridman Podcast

    2022

  23. [31]

    Gener- alization of reinforcement learners with working and episodic memory, 2020

    Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adri` a Puigdom` enech Badia, Gavin Buttimore, Charlie Deck, Joel Z Leibo, and Charles Blundell. Gener- alization of reinforcement learners with working and episodic memory, 2020

    2020

  24. [32]

    Cril: Continual robot imitation learning via generative and prediction model

    Chongkai Gao, Haichuan Gao, Shangqi Guo, Tianren Zhang, and Feng Chen. Cril: Continual robot imitation learning via generative and prediction model. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6747–5754. IEEE, 2021

    2021

  25. [33]

    Humanplus: Humanoid shadowing and imitation from humans

    Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454 , 2024

    2024 arXiv

  26. [34]

    Neural turing machines, 2014

    Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014

    2014

  27. [35]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139–144, 2020

    2020

  28. [36]

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches, 2023

    Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, and Ted Xiao. Rt-trajectory: Robotic task generaliza...

    2023

  29. [37]

    Hybrid computing using a neural network with dynamic external memory

    Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwi´ nska, Sergio G´ omez Colmenarejo, Edward Grefenstette, Tiago Ra- 50 malho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626...

    2016

  30. [38]

    Learning latent dynamics for planning from pixels, 2019

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels, 2019

    2019

  31. [39]

    Improved training of wasserstein gans

    Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017

    2017

  32. [40]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Ad- vances in neural information processing systems , 29, 2016

    2016

  33. [41]

    Deep recurrent q-learning for partially observable mdps, 2017

    Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps, 2017

    2017

  34. [42]

    Long short-term memory.Neural Compu- tation, 9:1735–1780, 1997

    Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neural Compu- tation, 9:1735–1780, 1997. URL https://api.semanticscholar.org/CorpusID: 1915014

    1997

  35. [43]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840–6851, 2020

    2020

  36. [44]

    Re-evaluating continual learning scenarios: A categorization and case for strong baselines, 2019

    Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines, 2019. 51

    2019

  37. [45]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  38. [46]

    Optimizing agent behavior over long time scales by transporting value

    Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Fed- erico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10(1):5223, 2019

    2019

  39. [47]

    Different ways to cue a coherent memory system: A theory for episodic, semantic, and procedural tasks

    Michael S Humphreys, John D Bain, and Ray Pike. Different ways to cue a coherent memory system: A theory for episodic, semantic, and procedural tasks. Psychological Review, 96(2):208, 1989

    1989

  40. [48]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:235313679

    2021

  41. [49]

    Deep variational reinforcement learning for pomdps, 2018

    Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps, 2018

    2018

  42. [50]

    Continual learning with node-importance based adaptive group sparse regularization

    Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. Advances in neural information processing systems , 33:3647–3658, 2020

    2020

  43. [51]

    Inferring algorithmic patterns with stack- augmented recurrent nets, 2015

    Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack- augmented recurrent nets, 2015

    2015

  44. [52]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  45. [53]

    Think before you act: Decision transformers with working memory

    Jikun Kang, Romain Laroche, Xingdi Yuan, Adam Trischler, Xue Liu, and Jie Fu. Think before you act: Decision transformers with working memory. In Forty-first International Conference on Machine Learning , 2023

    2023

  46. [54]

    Mozer, Chris Pal, and Yoshua Bengio

    Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Michael C. Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding, 2018. 52

    2018

  47. [55]

    Qmdp-net: Deep learning for planning under partial observability, 2017

    Peter Karkus, David Hsu, and Wee Sun Lee. Qmdp-net: Deep learning for planning under partial observability, 2017

    2017

  48. [56]

    Auto-encoding variational {Bayes}

    Diederik P Kingma and Max Welling. Auto-encoding variational {Bayes}. In Int. Conf. on Learning Representations

  49. [57]

    Droid: A large-scale in-the-wild robot manipulation dataset

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yun- liang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403...

    2024 arXiv

  50. [58]

    Bradley Knox and Peter Stone

    W. Bradley Knox and Peter Stone. Interactively shaping agents via human re- inforcement: The tamer framework. In The Fifth International Conference on Knowledge Capture, September 2009. URL http://www.cs.utexas.edu/users/ ai-lab?KCAP09-knox

    2009

  51. [59]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting i...

    2017

  52. [60]

    Towards mental time travel: a hierarchical memory for reinforcement learning agents

    Andrew Lampinen, Stephanie Chan, Andrea Banino, and Felix Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents. Advances in Neural Information Processing Systems , 34:28182–28195, 2021

    2021

  53. [61]

    McClelland

    Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512–534, 2016. ISSN 1364-6613. doi: https://doi. org/10.1016/j.tics.2016.05.004. URL ...

    2016 doi

  54. [62]

    Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training, 2021

    Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training, 2021. 53

    2021

  55. [63]

    Gradient-based learning applied to document recognition

    Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  56. [64]

    Psychobiology of active and inactive memory

    Donald J Lewis. Psychobiology of active and inactive memory. Psychological bulletin, 86(5):1054, 1979

    1979

  57. [65]

    The promise of generalist robotic policies

    Sergey Levine. The promise of generalist robotic policies. https://sergeylevine. substack.com/p/the-promise-of-generalist-robotic , 2024. Blog post on Sub- stack. Accessed: 2024-11-16

    2024

  58. [66]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023

    2023

  59. [67]

    Longhorn: State space models are amortized online learners, 2024

    Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners, 2024. URL https://arxiv.org/ abs/2407.14207

    2024 arXiv

  60. [68]

    Con- tinual learning of diffusion models with generative distillation

    Sergi Masip, Pau Rodriguez, Tinne Tuytelaars, and Gido M van de Ven. Con- tinual learning of diffusion models with generative distillation. arXiv preprint arXiv:2311.14028, 2023

    2023 arXiv

  61. [69]

    Packnet: Adding multiple tasks to a single network by iterative pruning

    Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 7765–7773, 2018

    2018

  62. [70]

    McClelland, Bruce L

    James L. McClelland, Bruce L. McNaughton, and Randall C. O’Reilly. Comple- mentary learning systems within the hippocampus: A neural network modeling approach to understanding episodic memory consolidation. Psychological Review, 102(3):419–457, 1995

    1995

  63. [71]

    Why there are com- plementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory

    James Mcclelland, Bruce Mcnaughton, and Randall O’Reilly. Why there are com- plementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102:419–57, 08 1995. doi: 1...

    1995 doi

  64. [72]

    Spectral normalization for generative adversarial networks

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018

    2018 arXiv

  65. [73]

    Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation , pages 109–165. Academic Press, 1989. doi: https://doi.org/10. 54 1016/S0079-7421(08)60536-8. URL http...

    1989

  66. [74]

    Donald J. Newman. The double dixie cup problem. The American Mathematical Monthly, 67(1):58–61, 1960. ISSN 00029890, 19300972. URL http://www.jstor. org/stable/2308930

    1960

  67. [75]

    Some new aspects of the coupon collector’s problem

    Amy N Myers and Herbert S Wilf. Some new aspects of the coupon collector’s problem. SIAM review, 48(3):549–565, 2006

    2006

  68. [76]

    Variational continual learning

    Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. In International Conference on Learning Representations , 2018

    2018

  69. [77]

    Ng and Stuart J

    Andrew Y. Ng and Stuart J. Russell. Inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning (ICML-2000) , pages 663–670, 2000

    2000

  70. [78]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162–

  71. [79]

    When do transformers shine in rl? decoupling memory from credit assignment, 2023

    Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in rl? decoupling memory from credit assignment, 2023

    2023

  72. [80]

    O’Reilly and Kenneth A

    Randall C. O’Reilly and Kenneth A. Norman. Hippocampal and neocortical contri- butions to memory: Advances in the complementary learning systems framework. Trends in Cognitive Sciences, 6(12):505–510, December 2002. ISSN 1364-6613. doi: 10.1016/S1364-6613(02)02005-3. 55

    2002 doi

  73. [81]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864 , 2023

    arXiv

  74. [82]

    Memory gym: Towards endless tasks to benchmark memory capabilities of agents, 2024

    Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory gym: Towards endless tasks to benchmark memory capabilities of agents, 2024

    2024

  75. [83]

    Memory gym: Partially observable challenges to memory-based agents

    Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory gym: Partially observable challenges to memory-based agents. In International Conference on Learning Representations, 2023. URL https://openreview.net/ forum?id=jHc8dCx6DDr

    2023

  76. [84]

    Continual Learning in Reinforcement Environments

    Mark Ring. Continual Learning in Reinforcement Environments . PhD thesis, University of Texas at Austin, 1994. URL https://www.cs.utexas.edu/~ring/ Ring-dissertation.pdf

    1994

  77. [85]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Interna- tional conference on machine learning...

    2021

  78. [86]

    Gordon, and J

    Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011

    2011

  79. [87]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Mu- nich, Germany, October 5-9, 2015, proceedings, par...

    2015

  80. [88]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  81. [89]

    Dragan, S

    Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems ,

  82. [90]

    Trial without error: Towards safe reinforcement learning via human intervention, 2017

    William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention, 2017

    2017

  83. [91]

    Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233–242, 1999

    Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233–242, 1999

    1999

  84. [92]

    Ef- ficiently guiding imitation learning algorithms with human gaze

    Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum. Ef- ficiently guiding imitation learning algorithms with human gaze. arXiv preprint arXiv:2002.12500, 2020. 56

    2002 arXiv

  85. [93]

    Continual learning with deep generative replay

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems , 30, 2017

    2017

  86. [94]

    Continual diffusion: Continual customization of text-to-image diffusion with c-lora

    James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.06027 , 2023

    2023 arXiv

  87. [95]

    Progress & compress: A scalable framework for continual learning

    Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pages 4528–4537. PMLR, 2018

    2018

  88. [96]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022

    2022

  89. [97]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2 edition, 2018

    2018

  90. [98]

    Weiss, Niru Maheswaranathan, and Surya Gan- guli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Gan- guli. Deep unsupervised learning using nonequilibrium thermodynamics. CoRR, abs/1503.03585, 2015. URL http://arxiv.org/abs/1503.03585

    2015 arXiv

  91. [99]

    Procedural and declarative knowledge: An evolutionary perspective

    Timon Ten Berge and Ren´ e Van Hezewijk. Procedural and declarative knowledge: An evolutionary perspective. Theory & Psychology , 9(5):605–624, 1999. 57

    1999

  92. [100]

    Behavioral cloning from observation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence , 2018. URL https: //api.semanticscholar.org/CorpusID:23206414

    2018

  93. [101]

    Temporal credit assignment in reinforcement learning

    Richard Stuart Sutton. Temporal credit assignment in reinforcement learning . University of Massachusetts Amherst, 1984

    1984

  94. [102]

    Elements of Episodic Memory

    Endel Tulving. Elements of Episodic Memory . Oxford University Press, 1983

    1983

  95. [103]

    Memory and consciousness

    Endel Tulving. Memory and consciousness. Canadian Psychology/Psychologie canadienne, 26(1):1, 1985

    1985

  96. [104]

    Episodic and semantic memory

    Endel Tulving. Episodic and semantic memory. Organization of memory , pages 381–403, 1972

    1972

  97. [105]

    van de Ven and Andreas S

    Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning, 2019

    2019

  98. [106]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023

    2023

  99. [107]

    Contributions of memory circuits to language: The declara- tive/procedural model

    Michael T Ullman. Contributions of memory circuits to language: The declara- tive/procedural model. Cognition, 92(1-2):231–270, 2004

    2004

  100. [108]

    Deep tamer: Interactive agent shaping in high-dimensional state spaces, 2018

    Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces, 2018

    2018

  101. [109]

    Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Ag- nieszka Grabska-Barwinska, Jack Rae, Piotr Mirowski, Joel Z. Leibo, Adam Santoro, Mevlana Gemici, Malcolm Reynolds, Tim Harley, Josh Abramson, Shakir Mohamed, Danilo Rezende, David Saxton, Adam Cain, Chloe Hi...

    2018

  102. [110]

    A comprehensive survey of continual learning: Theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  103. [111]

    A bayesian approach for policy learning from trajectory preference queries

    Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Neural Information Processing Systems, 2012. URL https://api.semanticscholar.org/CorpusID:6019958

    2012

  104. [112]

    Model-free preference- based reinforcement learning

    Christian Wirth, Johannes F¨ urnkranz, and Gerhard Neumann. Model-free preference- based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016

    2016

  105. [113]

    Memory networks, 2015

    Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks, 2015. 58

    2015

  106. [114]

    Rabe, DeLesley Hutchins, and Christian Szegedy

    Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers, 2022

    2022

  107. [115]

    Lifelong learning with dynamically expandable networks

    Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018 . International Conference on Learning Representations, ICLR, 2018

    2018

  108. [116]

    Continual world: A robotic benchmark for continual reinforcement learning

    Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Mi- los. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems , 34:28496–28510, 2021

    2021

  109. [117]

    Learning memory mechanisms for decision making through demonstrations, 2024

    William Yue, Bo Liu, and Peter Stone. Learning memory mechanisms for decision making through demonstrations, 2024. URL https://arxiv.org/abs/2411.07954

    2024 arXiv

  110. [118]

    t-dgr: A trajectory-based deep generative replay method for continual learning in decision making

    William Yue, Bo Liu, and Peter Stone. t-dgr: A trajectory-based deep generative replay method for continual learning in decision making. 3rd Conference on Lifelong Learning Agents (CoLLAs), 2024

    2024

  111. [119]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020

    2020

  112. [120]

    Query-efficient imitation learning for end-to- end simulated driving

    Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to- end simulated driving. In AAAI Conference on Artificial Intelligence , 2017. URL https://api.semanticscholar.org/CorpusID:5929487

    2017

  113. [121]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023

    2023 arXiv

  114. [122]

    Tomczak, Tomasz Trzcinski, Florian Shkurti, and Piotr Milo’s

    Michal Zajac, Kamil Deja, Anna Kuzina, Jakub M. Tomczak, Tomasz Trzcinski, Florian Shkurti, and Piotr Milo’s. Exploring continual learning of diffusion models. ArXiv, abs/2303.15342, 2023. 59

    2023 arXiv

  115. [123]

    searing spotlights

    Yuke Zhu. Pathway to generalist robots: Scaling law, data flywheel, and humanlike embodiment. YouTube video, November 2023. URL https://www.youtube.com/ watch?v=hFPZSBnJeLc. Talk presented at Conference on Robot Learning (CoRL). 60 Appendix A Appendix A.1 Hyperparameters A.1.1...

    2023

  116. [125]

    On improving deep reinforcement learning for pomdps, 2018

    Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learning for pomdps, 2018

    2018

  117. [2017]

    URL https://api.semanticscholar.org/CorpusID:12226563

  118. [2019]

    URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ 2c3ddf4bf13852db711dd1901fb517fa-Paper.pdf

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.