Pith. sign in

REVIEW 2 major objections 6 minor 40 references

Replay Can Provably Increase Forgetting

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replaying old samples can provably increase forgetting in continual learning, even in benign noiseless linear settings.

desk verdict A genuinely new worst-case result and a solid d=3 average-case construction, but the high-dimensional average-case theorem as stated is not proven. read the letter →

arxiv 2506.04377 v1 pith:FC4GZYJU submitted 2025-06-04 cs.LG

Yasaman Mahdaviyeh , James Lucas , Mengye Ren , Andreas S. Tolias , Richard Zemel , Toniann Pitassi This is my paper
classification cs.LG MSC 68T0568Q32
keywords continuallearningcatastrophicforgettingsamplereplayover-parameterizedlinearregressiontasksubspacesprincipalanglesimplicitbiasnon-monotonicity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether sample replay, the standard continual-learning remedy for forgetting, is always beneficial, and answers that it is not. In an over-parameterized linear regression setting, where each task has fewer samples than dimensions and all tasks share a common solution, the authors show that replaying a single stored sample can turn vanishing forgetting (of order 1/T) into catastrophic, constant forgetting, and that this can even happen in expectation when tasks and replay samples are chosen at random. The mechanism is geometric: replay changes the effective null space of the current task, and forgetting depends non-monotonically on the angle between task null spaces. The paper also proves simple conditions under which replay is guaranteed harmless, identifying exactly when the popular remedy cannot backfire.

What carries the argument

The load-bearing identity is the projection recursion $w_t - w^* = P_t(w_{t-1} - w^*)$, where $P_t$ is the orthogonal projection onto the null space of task $t$'s sample matrix; it holds because gradient-based training converges to the minimum-distance-to-initialization solution. Forgetting is then expressed as a norm of the projection chain applied to $w^*$, so it depends on the principal angles between task null spaces. Replay enters the recursion by shrinking the null space of the current task: the combined set of current and replayed samples spans a larger subspace, so the projection $\tilde{P}_2$ acts on a smaller null space, changing the angle that controls the forgetting expression $x(1-x)$. This single change, plus the transfer of error between non-orthogonal samples, is what makes forgetting increase.

What would settle it

Train a linear model with full-batch gradient descent until convergence on the two-task 3D construction of Theorem C.1, measure expected forgetting with and without a randomly replayed sample; if replay does not increase expected forgetting across many draws, the lower-dimensional average-case theorem is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that sample replay is not a monotone intervention: there are realizable, noiseless task sequences in which replaying a remembered example strictly increases forgetting, and does so to the maximum possible extent. Theorem 3.2 exhibits a worst-case sample whose replay lifts forgetting from O(1/T) to Θ(1); Theorems 3.5 and C.1 show the same phenomenon in an average-case sense, where both task samples and replay samples are drawn at random and expected forgetting increases. The mechanism is that replay shrinks the null space of the current combined task, which can move the angle between task null spaces closer to π/4, the point of maximal forgetting; a minimal three-sample two-task example shows replay can also simply transfer error from a replayed sample onto an unreplayed sample that appears in many earlier tasks.

Load-bearing premise

The theorems all assume that training on each task converges to the minimum-distance-to-initialization solution; if real training deviates from that path through finite steps, momentum, weight decay, nonconvex losses, or early stopping, the replay constructions are no longer guaranteed to hold.

Editorial extensions

If this is right

  • Replay buffers should be chosen with the geometry of task subspaces in mind; a random or class-balanced buffer is not automatically safe.
  • Forgetting can first increase and then decrease as more replay samples are added, so evaluating replay at a single buffer size can mislead.
  • When forgetting without replay is already zero, replay is guaranteed not to hurt, and the same holds when the smallest principal angle between task null spaces is at least π/4.
  • The average-case result shows that even a non-adversarial continual-learning pipeline, with random samples from a task distribution and random replay choices, can suffer from replay.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The angle mechanism suggests a practical buffer-selection rule: prefer replay samples whose projection onto the new task's null space is far from the old null-space direction, so the angle moves away from π/4 rather than toward it.
  • The non-monotonicity in replay count implies that conflicting empirical reports on replay could simply reflect the buffer sizes tested; a systematic sweep over the number of replay samples is needed to characterize a method.
  • The authors note a resemblance to retrieval-induced forgetting in psychology; the paper's linear construction gives a minimal analytic model of how rehearsing some items in a category can hurt unrehearsed items in the same category.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper studies whether sample replay can increase forgetting in a continual learning setting based on over-parameterized linear regression. Under the minimum-distance implicit bias of GD/SGD, the parameter error evolves by orthogonal projections onto task null spaces (Eq. (2)). The authors prove a worst-case result (Theorem 3.2) in which forgetting without replay is O(1/T) but becomes Theta(1) when a single carefully chosen sample is replayed. They also prove average-case results: Theorem C.1 for d=3 and Theorem 3.5 for high dimension, claiming that replaying randomly chosen samples can increase expected forgetting for two tasks with nearby subspaces. Experiments on linear models, MLPs, Rotated MNIST, and Split MNIST show non-monotone effects of replay and support the possibility of harmful replay. The central thesis is that replay is not universally beneficial, even in a noiseless realizable linear setting.

Significance. If established, the paper's results would be a valuable theoretical counterpoint to the common assumption that replay is always helpful. The worst-case construction (Theorem 3.2) and the three-dimensional average-case construction (Theorem C.1) appear essentially correct and are accompanied by a clear geometric explanation and supporting experiments. The paper is self-contained given standard implicit-bias results from Gunasekar et al. (2018) and Zhang et al. (2021), and the projection recursion from Evron et al. (2022). However, the high-dimensional average-case theorem (Theorem 3.5), which is one of the two main theoretical contributions, has a proof gap in its final constant choice and expectation bound. Because the core claim is still supported by the other theorems, the paper has potential, but the high-dimensional result needs to be repaired or restated.

major comments (2)
  1. [C.3, Eq. (91)] The proof claims that inequality (91), 96(d-1)/m + 2 < exp(m), follows from the stated hypotheses with c1=120, c2=15, c3=97. It does not. The hypotheses only give d-1 < m^m/97, which permits d-1 far larger than m(exp(m)-2)/96. For example, m=10 allows d-1 up to about 1.03e8 under the hypotheses, while (91) requires d-1 < about 2300. Thus there are parameter points satisfying the theorem's assumptions for which the key lower bound (81) is not established. The theorem should be restated with a sufficient condition that actually implies the needed inequality (e.g., 96(d-1)/m + 2 < exp(m)), or with a different proof that does not rely on this implication; the quantifier over d and m also needs to be made explicit.
  2. [C.3, Eq. (80)] The lower bound on the expected value uses the term exp(-m) f(1-epsilon^2) on the failure event, but the manuscript does not justify why the failure event has probability exp(-m) or why f(X) >= f(1-epsilon^2) there. On the complement of the concentration event, X can be outside [1/2,1], where f(X) can be smaller than f(1-epsilon^2). A correct lower bound must either handle the full 2exp(-m) failure probability or use a different decomposition. This gap is part of the same argument needed for Theorem 3.5.
minor comments (6)
  1. [C.1, Eq. (24)] The value x1^T tilde-P_T u u^T w* is -(1/2) sqrt(6/7) a, not -6/28 a; the squared value is 3/14 a^2, not 9/196 a^2. The Theta(1) conclusion is unaffected, but the constant should be corrected.
  2. [C.3] The proof treats the m without-replacement replay samples as i.i.d. Gaussian rows. This is in fact valid when the n1 training samples are i.i.d. continuous (by exchangeability of the random index set), but the manuscript should state this justification, since the current text appears to rely on an unstated independence assumption.
  3. [3.2.2 and 3.3] Theorem 3.5 and Proposition 3.6 say 'Assumptions 2.1, 2.1, and 2.2'; the duplicate reference should be to Assumption 3.3 (or whichever was intended).
  4. [C.2] Appendix C labels Claim C.2 and Claim C.3 as 'Theorem C.2' and 'Theorem C.3' in the proof of Theorem C.1; the labels should be consistent.
  5. [C.4] The proof of Proposition 3.4 is headed 'Proof of Theorem 3.4'; the heading should match the proposition numbering.
  6. [Figure 1 caption] The phrase 'standard mean error' should be 'standard error of the mean'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the replay-forgetting theorems are derived from explicit constructions and standard external results, not from their conclusions.

full rationale

The derivation chain is self-contained. Equation (2) is derived in Section A.1 from the minimum-distance-to-initialization solution in Equation (1), which is an independently established implicit-bias result cited to Gunasekar et al. (2018) and Zhang et al. (2021); the paper includes the derivation for completeness. The forgetting expression in Equation (4) follows algebraically from the projection recursion, and the task constructions in Theorems 3.2, C.1, and 3.5 are explicit rather than fitted. For instance, the worst-case proof fixes concrete vectors x1, x2, x3, computes forgetting without replay as 3a^2/(28(T-1)) and with replay as 9a^2/196, exhibiting the claimed O(1/T) versus Theta(1) gap by direct calculation. The average-case results use Gaussian sample generation, explicit subspaces, and the standard Dasgupta-Gupta concentration lemma (Lemma C.4); the constants c1, c2, c3 and the value of epsilon are fixed universal choices, not parameters fitted to data, and no replay result is assumed as an input. The only self-citation with author overlap is van de Ven et al. (2020), which includes co-author Tolias; it is cited in the introduction as an example of generative replay and is not load-bearing for any theorem. The skeptic-identified issue in Theorem 3.5, namely that inequality (91) does not follow from the stated parameter conditions and that without-replacement replay is treated as i.i.d., is a proof-gap or correctness concern, not a circular reduction: it does not make the theorem's conclusion equivalent to its hypotheses or to a fitted quantity. Therefore no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard implicit-bias results for gradient descent on linear regression, explicit domain assumptions (linearity, realizability, over-parameterization, sample availability), and a Gaussian distributional model for the average case. The only hand-tuned construction parameter is epsilon. No new physical or mathematical entities are introduced.

free parameters (1)
  • epsilon = 1/sqrt(63) in Theorem C.1; 0 < epsilon < 1/2 in Theorem 3.5
    Hand-chosen constant controlling the angle between the first task's null space and the second task's subspace. The proofs require the initial angle to be small and the post-replay angle to approach pi/4. No data fitting, but the construction depends on it.
assumptions (6)
  • standard math GD/SGD in over-parameterized linear regression converges to the minimum-norm solution (Eq 1).
    Used in Section 2.2 to derive the projection recursion Eq (2). This is a theorem from Gunasekar et al. 2018 and Zhang et al. 2021; the paper relies on it without proof.
  • domain assumption Assumption 2.1: tasks are linear and over-parameterized (rank(X_t) < d).
    Defines the setting; ensures the null space is nontrivial for each task.
  • domain assumption Assumption 2.2: realizability, a shared w* lies in the span of all tasks and y_t = X_t w*.
    Without a common solution, the forgetting measure and the projection dynamics would not have the simple form Eq (4).
  • domain assumption Assumption 3.3: n_t >= k_t so that each task's samples span its subspace.
    Ensures rank(X_t)=k_t and the null space is exactly I - W_t W_t^T; used in Proposition 3.4 and the average case proofs.
  • domain assumption Average case: task samples follow X_tj = W_t Z_tj with Z_tj iid N(0, I/k_t) (Eq 5).
    Defines the distributional setting for the average case results; the Gaussianity is used to compute expected forgetting and to control the distribution of replayed-sample projections.
  • standard math Dasgupta-Gupta concentration lemma (Lemma C.4) for norms of random projections.
    Used in the proof of Theorem 3.5 to bound the norm of the projection of a fixed vector onto a random m-dimensional subspace.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Replay Can Provably Increase Forgetting." pith.science (2026). https://pith.science/paper/FC4GZYJU

@misc{pith2026250604377,
  author       = {Pith},
  title        = {Pith review of: Replay Can Provably Increase Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FC4GZYJU}},
  note         = {Machine review of arXiv:2506.04377}
}
read the original abstract

Continual learning seeks to enable machine learning systems to solve an increasing corpus of tasks sequentially. A critical challenge for continual learning is forgetting, where the performance on previously learned tasks decreases as new tasks are introduced. One of the commonly used techniques to mitigate forgetting, sample replay, has been shown empirically to reduce forgetting by retaining some examples from old tasks and including them in new training episodes. In this work, we provide a theoretical analysis of sample replay in an over-parameterized continual linear regression setting, where each task is given by a linear subspace and with enough replay samples, one would be able to eliminate forgetting. Our analysis focuses on sample replay and highlights the role of the replayed samples and the relationship between task subspaces. Surprisingly, we find that, even in a noiseless setting, forgetting can be non-monotonic with respect to the number of replay samples. We present tasks where replay can be harmful with respect to worst-case settings, and also in distributional settings where replay of randomly selected samples increases forgetting in expectation. We also give empirical evidence that harmful replay is not limited to training with linear models by showing similar behavior for a neural networks equipped with SGD. Through experiments on a commonly used benchmark, we provide additional evidence that, even in seemingly benign scenarios, performance of the replay heavily depends on the choice of replay samples and the relationship between tasks.

Figures

Figures reproduced from arXiv: 2506.04377 by the authors.

Figure 1
Figure 1. Forgetting versus the number of replayed samples. Each plot shows forgetting of a linear model and a [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Class of the replayed sample affects forgetting. The x-axis shows the class of the replay sample while the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. A histogram of differences in forgetting without replay and with replay of a digit [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Role of tasks in replay for Split MNIST. For each task sequence, we have plotted the average forgetting against [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: A simple example to demonstrate how the angle between tasks with one dimensional null spaces affects [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Replay can transfer error between samples. The first task consists of two samples [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: An illustration of forgetting in the average case construction. The red plane in (a) shows the null space of [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Same experiment as in Figure 1b with a network of width 4d instead D.1.1 THE EFFECT OF THE ANGLE BETWEEN TASKS WHILE TRAINING WITH A MLP We discussed in Section 3.2 how replay changes the angle between the two tasks in a way that increases forgetting on average. To und…
Figure 9
Figure 9. Figure 9: When training using a MLP, the angle between task null spaces mostly has a similar effect on forgetting as the [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 24 canonical work pages

  1. [1]

    Online continual learning with maximal interfered retrieval

    Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019 a

    work page 2019

  2. [2]

    Gradient based sample selection for online continual learning

    Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019 b

    2019

  3. [3]

    M. C. Anderson, R. A. Bjork, and E. L. Bjork. Remembering can cause forgetting: Retrieval dynamics in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 0 (20(5)): 0 1063--1087, 1994

    work page 1994

  4. [4]

    Retrieval-induced forgetting: evidence for a recall-specific mechanism

    M C Anderson, E L Bjork, and R A Bjork. Retrieval-induced forgetting: evidence for a recall-specific mechanism. Journal of Experimental Psychology: Learning, Memory, and Cognition, 7 0 (3): 0 522--530, Sep 2000

    work page 2000

  5. [5]

    Theoretical insights into overparameterized models in multi-task and replay-based continual learning, 2024

    Mohammadamin Banayeeanzade, Mahdi Soltanolkotabi, and Mohammad Rostami. Theoretical insights into overparameterized models in multi-task and replay-based continual learning, 2024

    work page 2024

  6. [6]

    New insights on reducing abrupt representation change in online continual learning

    Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025, 2021

    arXiv 2021

  7. [7]

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv: Learning, 2019

    work page 2019

  8. [8]

    Memory bounds for continual learning

    Xi Chen, Christos Papadimitriou, and Binghui Peng. Memory bounds for continual learning. 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pp.\ 519--530, 2022

    work page 2022

Show all 40 references
  1. [9]

    An elementary proof of a theorem of johnson and lindenstrauss

    Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms, 22, 2003

    2003

  2. [10]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 0 (7): 0 33...

    2022

  3. [11]

    A theoretical analysis of catastrophic forgetting through the ntk overlap matrix

    Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artif...

    2021

  4. [12]

    Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Po-Ling Loh and Maxim Raginsky (eds.), Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of M...

    2022

  5. [13]

    Continual learning in linear classification on separable data

    Itay Evron, Edward Moroshko, Gon Buzaglo, Maroun Khriesh, Badea Marjieh, Nathan Srebro, and Daniel Soudry. Continual learning in linear classification on separable data. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  6. [14]

    Analysis of catastrophic forgetting for random orthogonal transformation tasks in the overparameterized regime

    Daniel Goldfarb and Paul Hand. Analysis of catastrophic forgetting for random orthogonal transformation tasks in the overparameterized regime. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Int...

    2023

  7. [15]

    Characterizing implicit bias in terms of optimization geometry

    Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Le...

    2018

  8. [16]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985

    1985

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014

    2014 arXiv

  10. [18]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  11. [19]

    Fixed design analysis of regularization-based continual learning

    Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of regularization-based continual learning. In Proceedings of the 2nd Conference on Lifelong Learning Agents (CoLLAs), pp.\ 513--533, 2023. arXiv:2303.10263

    2023 arXiv

  12. [20]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40 0 (12): 0 2935--2947, 2017

    2017

  13. [21]

    Reinforcement learning for robots using neural networks

    Long-Ji Lin. Reinforcement learning for robots using neural networks. Carnegie Mellon University, 1992

    1992

  14. [22]

    Theory on forgetting and generalization of continual learning

    Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  15. [23]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 6470–6479, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964

    2017

  16. [24]

    Catastrophic interference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp.\ 109--165. Elsevier, 1989

    1989

  17. [25]

    Upgrading the sleeping brain with targeted memory reactivation

    Delphine Oudiette and Ken Paller. Upgrading the sleeping brain with targeted memory reactivation. Trends in Cognitive Science, 3 0 (17): 0 pp. 142--149, 2013

    2013

  18. [26]

    Parisi, Ronald Kemker, Jose L

    German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113: 0 54--71, 2019. ISSN 0893-6080. doi:https://doi.org/10.1016/j.neunet.2019.01.012

    2019 doi

  19. [27]

    Giampouras, and Ren\' e Vidal

    Liangzu Peng, Paris V. Giampouras, and Ren\' e Vidal. The ideal continual learner: an agent that never forgets. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  20. [28]

    Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision -- ECCV 2020, pp.\ 524--540, Cham, 2020. Springer In...

    2020

  21. [29]

    Maintaining memories by reactivation

    Bjorn Rasch and Jan Born. Maintaining memories by reactivation. Current Opinions in Neurobiology, 6 0 (17): 0 pp. 698--703, 2007

    2007

  22. [30]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.\ 2001--2010, 2017

    2001

  23. [31]

    Experience replay for continual learning

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  24. [32]

    Order parameters and phase transitions of continual learning in deep neural networks

    Haozhe Shan, Qianyi Li, and Haim Sompolinsky. Order parameters and phase transitions of continual learning in deep neural networks. arXiv preprint arXiv:2407.10315, 2024

    2024 arXiv

  25. [33]

    Gcr: Gradient coreset based replay buffer selection for continual learning

    Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Gcr: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 99--108, 2022

    2022

  26. [34]

    An empirical study of example forgetting during deep neural network learning

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018

    2018 arXiv

  27. [35]

    van de Ven, Hava T

    Gido M. van de Ven, Hava T. Siegelmann, and Andreas S. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11 0 (1): 0 4069, 2020. doi:10.1038/s41467-020-17866-2

    2020 doi

  28. [36]

    Rehearsal revealed: The limits and merits of revisiting samples in continual learning

    Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 9365--9374, 2021. doi:10.1109/ICCV48922.2021.00925

    2021

  29. [37]

    A comprehensive survey of continual learning: Theory, method and application, 2024

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application, 2024

    2024

  30. [38]

    Large scale incremental learning

    Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 374--382, 2019

    2019

  31. [39]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp.\ 3987--3995. PMLR, 2017

    2017

  32. [40]

    Understanding deep learning (still) requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64 0 (3): 0 107–115, feb 2021. ISSN 0001-0782. doi:10.1145/3446776

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.