Teaching AI to Remember: Insights from Brain-Inspired Replay in Continual Learning

Jina Kim

REVIEW 4 major objections 4 minor 14 references

Teaching AI to Remember: Insights from Brain-Inspired Replay in Continual Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Internal replay in brain-inspired continual learning preserves earlier tasks but lowers accuracy on the task being learned, the paper's CIFAR-100 ablations show.

desk verdict Worth a look for the retention/initial-accuracy trade-off, but the top-line representational-overlap claim is contradicted by the paper's own near-zero silhouette scores. read the letter →

arxiv 2509.00047 v1 pith:WFSGZ7ZR submitted 2025-08-24 cs.LG cs.AI

Jina Kim This is my paper · ORCID

classification cs.LGcs.AI MSC 68T07

keywords continuallearningcatastrophicforgettinginternalreplaybrain-inspiredSynapticIntelligenceCIFAR-100class-incrementallatentrepresentations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what the internal replay component actually does in a brain-inspired generative replay model for continual learning, and answers by turning it on and off in four configurations on CIFAR-100 class-incremental learning. Its main claim is a trade-off: internal replay, especially when combined with Synaptic Intelligence, clearly mitigates catastrophic forgetting and improves the model's fit to the training data, but it also lowers initial accuracy on the task being learned. The paper also claims the benefit is not accompanied by clean task representations: all variants show heavily overlapping latent clusters. If this is right, current brain-inspired replay improves memory stability at a real plasticity cost, and future continual-learning methods need a way to keep the retention without sacrificing new-task accuracy.

What carries the argument

The central mechanism is internal replay, a component of the brain-inspired generative replay model in which previously learned classes are reactivated at the hidden or latent level rather than replayed as input pixels. The paper's argument is carried by four ablation conditions (brain-inspired replay with and without internal replay, each with and without Synaptic Intelligence), compared through retention ratio, forgetting score, initial/final accuracy, log-likelihood, reconstruction error, silhouette score, and UMAP projections. Internal replay is the switch that separates the conditions, so all observed trade-offs are attributed to it. Synaptic Intelligence plays the supporting role of pr

What would settle it

Re-run the four ablations with the original model implementation and exact training settings on CIFAR-100; if internal replay no longer raises retention while lowering initial accuracy, or if the separation between with- and without-internal-replay conditions disappears, the trade-off claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that turning on internal replay in the brain-inspired replay model, with or without Synaptic Intelligence, changes the stability-plasticity balance in a specific way: it increases retention ratio and decreases forgetting score across all ten tasks, and it improves log-likelihood and lowers reconstruction error, but it reduces initial test accuracy on each task, so the upper bound of achievable performance is lowered. When internal replay is combined with Synaptic Intelligence, retention is strongest and forgetting smallest, yet this combination performs worst on the final task, indicating Synaptic Intelligence's contribution is stability rather than better learni

Load-bearing premise

The results assume the author's re-implementation of the original brain-inspired replay model is faithful; the paper gives no training protocol or code, and the only consistency check is a rough agreement that final accuracy is around 0.2.

Editorial extensions

If this is right

Adding internal replay to a generative-replay continual learner will improve retention of earlier tasks and data fit while lowering accuracy on the task currently being learned.
Combining internal replay with Synaptic Intelligence yields the strongest retention and lowest forgetting, but the stability gain comes at the cost of reduced plasticity on new tasks.
The method as proposed does not create well-separated task representations in latent space; near-zero silhouette scores persist with or without internal replay.
Improving the model requires recovering initial-task accuracy without giving up the retention benefit, rather than focusing on forgetting alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the trade-off is general, then varying where internal replay is applied (which layer) or how sparse the context gate is could recover current-task accuracy; the paper itself lists layer location and mask sparsity as unexplored levers.
The low silhouette scores hint that these models rely on shared or overlapping features across tasks; a direct test would be to add a separation regularizer on latent embeddings and see whether forgetting or final accuracy moves.
Because the report omits training details and code, the quantitative trade-off should be treated as evidence about this particular re-implementation until reproduced with the original training protocol.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a short analysis report re-running the brain-inspired replay (BIR) model from Van de Ven et al. (2020) on CIFAR-100 class-incremental, ablating internal replay with and without SI. The central empirical trade-off is real and worth reporting: w/ IR variants show higher retention and lower initial accuracy, consistently across tasks, and BIR+SI (w/ IR) clearly has the best retention and worst initial accuracy. That part is a clean ablation and the figures tell a consistent story.

What's new: per-task retention and forgetting curves, initial/final accuracy splits, log-likelihood and reconstruction distributions, and silhouette scores for the four variants. Those specific measurements aren't in the cited prior work. The paper is honest about its scope, describes the model in an appendix, and acknowledges limitations in the conclusion.

The soft spots are in the interpretative overlay. The abstract says internal replay 'increases representational overlap in latent space, potentially limiting task-specific differentiation,' but Figure 4(a) shows essentially identical near-zero silhouette scores for all four variants, with the same tendency across tasks. The paper itself says all models show poorly separated clusters. That's a null result for any differential effect of internal replay, not evidence that internal replay increases overlap. The UMAP plots are from the single highest-silhouette task and are subjective. This is an internal inconsistency, not just a missing baseline: even if the re-implementation is perfect, the stated effect isn't in the measurements. The conclusion is more careful—it says all models exhibit high overlap—so the abstract overreaches relative to the paper's own data.

Also worth noting: all results are single runs with no error bars, seeds, or standard deviations, and no code or training protocol is provided, so the ablation contrasts can't be independently checked. The lack of code matters more than usual here because the paper is a re-implementation of a published method, and any drift in architecture or hyperparameters changes which system the trade-off describes. I'd also like to see a standard continual learning baseline or two to contextualize the trade-off, but that's a nice-to-have, not a load-bearing flaw.

Overall, the retention/accuracy trade-off is a useful empirical data point for the BIR framework, but the representational-overlap claim should be cut or heavily qualified. This is the kind of short paper that could be a solid workshop contribution after revision. Yes, send it to a serious referee—the central ablation deserves scrutiny and the overlap claim needs to be caught. But I wouldn't cite it in its current form.

Referee Report

4 major / 4 minor

Summary. The paper investigates the internal replay (IR) component of the brain-inspired replay (BIR) model of Van de Ven et al. (2020), using CIFAR-100 in a class-incremental setting. It compares four variants—BIR with/without IR, with and without Synaptic Intelligence (SI)—on accuracy retention, forgetting, initial/final task accuracy, log-likelihood, reconstruction error, silhouette scores, and UMAP embeddings. The central claim is that IR mitigates catastrophic forgetting, especially with SI, but lowers initial-task accuracy, and that IR increases representational overlap in latent space, potentially limiting task-specific differentiation.

Significance. If established, the retention–plasticity trade-off would be a useful diagnostic for the BIR framework and would caution against adding internal replay without considering its cost. The paper also showcases useful analysis metrics (retention ratio, forgetting score, distributional diagnostics). However, the current evidence is not sufficient: the experiments appear to be single runs with no training protocol or code, and the representational-overlap claim is contradicted by the paper's own silhouette-score results. The analysis is a reasonable starting point, but the manuscript is not yet at the standard required for a journal publication.

major comments (4)

[Section 3.1] The experimental setup is critically underspecified. The paper does not report epochs, optimizer, learning rate, batch size, latent dimensionality, generator/classifier architecture details, or any random seeds, and no code is provided. All conclusions rest on the author's re-implementation of the BIR model, but the only consistency check is a rough statement that final accuracy 'ranges around 0.2.' Without a reproducible protocol, the w/ IR vs w/o IR contrasts cannot be verified or transferred to the original method. This is load-bearing for every claim in the paper.
[Section 3.2.3, Figure 4] The abstract states that internal replay 'increases representational overlap in latent space,' but the reported silhouette scores are near zero for all four variants with 'the same tendency across tasks,' and the UMAP panels show overlap in all models. No quantitative comparison of overlap between w/ IR and w/o IR is provided, and the UMAP is only shown for the single task with the highest silhouette score. These data do not support a differential effect of internal replay on representational separation; at most they show that all variants have poorly separated clusters. This is an internal inconsistency and the representational-overlap claim should be either removed or supported with appropriate quantitative cluster-separation metrics.
[Figures 1 and 2] All results are single-run plots with no error bars, confidence intervals, or multiple seeds. The paper repeatedly uses the word 'significant' (e.g., 'internal replay significantly mitigates forgetting'), but no statistical testing or variance information is reported. Given the small margins and overlapping trends visible in the plots, the authors should provide multi-seed results and either report uncertainty or tone down the claims.
[Section 3.2.2] The log-likelihood and reconstruction-error analysis is only shown for BIR models, while the conclusion states that IR 'improves the model’s ability to fit the data' in general. The paper says BIR+SI results are 'consistent,' but no evidence is shown. This is a gap in the supporting evidence, and the claim should be restricted to the BIR variants or the BIR+SI results should be included.

minor comments (4)

[Throughout] There are several typos and spacing issues, e.g., 'CIF AR-100' in Section 3.1 and 'V AE' in Appendix B. The manuscript would benefit from a careful proofread.
[Figure 1 caption] The left panel is labeled 'Retention Ratio' but the y-axis in the figure is labeled 'Retention Ratio'; the right panel is labeled 'T est Accuracy' in the axis while the caption describes forgetting score. The dashed-line explanation is also unclear—does the dashed line represent average accuracy per task or the average over tasks?
[Section 3.2.1] The text refers to '2(right)' for the performance degradation during training, but Figure 2's right panel is final accuracy; the initial-accuracy panel is the left panel of Figure 2. This cross-reference should be corrected.
[Section 2 / Appendix B] The description of the brain-inspired components is present only in Appendix B, but the main text in Section 2 refers to it without sufficient detail. Since the paper focuses on internal replay, a more explicit definition of 'internal replay' in the main text would help readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper reports ablations of an external published model with no fitted inputs and no self-citation chain; the unsupported overlap claim is an evidentiary defect, not circularity.

full rationale

The paper contains no derivation chain in the sense of predicting an output from fitted inputs. It re-implements (or relies on) the published BIR model of Van de Ven et al. (2020) and compares w/IR vs w/o IR under CIFAR-100 class-IL. The metrics used (retention ratio = final/initial accuracy, forgetting score = initial − final accuracy, silhouette score) are operational definitions; reporting that one condition scores higher on these metrics is an empirical measurement, not a tautology. No parameter is fitted to a subset and then called a prediction. All load-bearing citations to Van de Ven et al. are external to this paper's authorship, so the self-citation patterns do not apply. The abstract's statement that internal replay 'increases representational overlap' is not supported by the paper's own Fig. 4(a), where all silhouette scores are near zero and 'with the same tendency across tasks'; however, this is an internal-validity/overclaim problem, not a circularity problem. The absence of training hyperparameters and code affects reproducibility, not circularity. Hence score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters and no invented entities; it is an ablation of the existing BIR model, so the brain-inspired components (internal replay, gating, distillation) are borrowed from Van de Ven et al. (2020) rather than postulated here. The load-bearing unstated inputs are the faithfulness of the re-implementation and the validity of the silhouette/UMAP diagnostics for the representational-overlap claim. The neuroscience background in Appendix A is motivational and not load-bearing.

assumptions (4)

domain assumption The BIR model of Van de Ven et al. (2020) is re-implemented faithfully, including the internal replay mechanism at the hidden layer
Entered in Section 3.1 and Appendix B: all ablation contrasts presuppose the reproduction matches the original model, yet no code, hyperparameters, or quantitative check against the original paper's numbers is given beyond rough agreement on final accuracy.
domain assumption Silhouette score and UMAP on layer fcE.fcLayer2.linear activations validly measure task-specific representational separation
Entered in Section 3.2.3: this is the interpretive bridge for the claim that internal replay increases representational overlap; UMAP projections can distort inter-cluster distances, and near-zero silhouette for all conditions limits what can be concluded.
domain assumption CIFAR-100 class-incremental learning is an adequate testbed for catastrophic forgetting
Entered in Section 3.1: follows the original paper's choice of benchmark and scenario; standard but still a single dataset and a single scenario.
domain assumption Hippocampal replay supports memory consolidation in humans
Entered in the introduction and Appendix A via Bear et al. (2020): motivational neuroscience background that motivates but does not by itself constrain the experimental claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Teaching AI to Remember: Insights from Brain-Inspired Replay in Continual Learning." pith.science (2026). https://pith.science/paper/WFSGZ7ZR

@misc{pith2026250900047,
  author       = {Pith},
  title        = {Pith review of: Teaching AI to Remember: Insights from Brain-Inspired Replay in Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFSGZ7ZR}},
  note         = {Machine review of arXiv:2509.00047}
}

read the original abstract

Artificial neural networks (ANNs) continue to face challenges in continual learning, particularly due to catastrophic forgetting, the loss of previously learned knowledge when acquiring new tasks. Inspired by memory consolidation in the human brain, we investigate the internal replay mechanism proposed by~\citep{brain_inspired_replay1}, which reactivates latent representations of prior experiences during learning. As internal replay was identified as the most influential component among the brain-inspired mechanisms in their framework, it serves as the central focus of our in-depth investigation. Using the CIFAR-100 dataset in a class-incremental setting, we evaluate the effectiveness of internal replay, both in isolation and in combination with Synaptic Intelligence (SI). Our experiments show that internal replay significantly mitigates forgetting, especially when paired with SI, but at the cost of reduced initial task accuracy, highlighting a trade-off between memory stability and learning plasticity. Further analyses using log-likelihood distributions, reconstruction errors, silhouette scores, and UMAP projections reveal that internal replay increases representational overlap in latent space, potentially limiting task-specific differentiation. These results underscore the limitations of current brain-inspired methods and suggest future directions for balancing retention and adaptability in continual learning systems.

Figures

Figures reproduced from arXiv: 2509.00047 by the authors.

**Figure 1.** Retention ratio and Forgetting score per task. Retention ratio comparison (left) and forgetting score comparison (right) between BIR(w/ IR), BIR(w/o IR), BIR+SI(w/ IR), BIR+SI(w/o IR) for all tasks. Dashed lines refer to as the average test accuracy throughout all the tasks for each model. As shown in 1, the BIR+SI (w/ IR) model achieves the highest retention ratio and lowest forgetting score across all tasks. Alth… view at source ↗

**Figure 2.** Initial accuracy and Final accuracy per task. Initial test accuracy comparison (left) and final test accuracy comparison (right) between BIR(w/ IR), BIR(w/o IR), BIR + SI(w/ IR), BIR + SI(w/o IR) for all tasks. Dashed lines refer to the average test accuracy throughout all the tasks for each model. 3.2.2 Log likelihood distribution and reconstruction error distribution Here we present the result of log-likelihood d… view at source ↗

**Figure 3.** Log likelihood distribution and Reconstruction error distribution. Comparison of log likelihood distribution (left) and reconstruction error distribution (right) between model with internal replay (BIR(w/ IR)) and model without internal replay (BIR(w/o IR)). method of Van de Ven et al. [2020] is not yet optimal which also aligns with the fact that the final accuracy is not that high(2(right)). 4 Conclusion In this… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Silhouette score and UMAP visualization of embeddings on layer fcE.fcLayer2.linear. (a) Comparison of silhouette score between model with internal replay (BIR(w/ IR)) and model without internal replay (BIR(w/o IR)). UMAP visualization of embeddings on task 7 for (b) B…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

Neuroscience: exploring the brain, enhanced edition: exploring the brain

Mark Bear, Barry Connors, and Michael A Paradiso. Neuroscience: exploring the brain, enhanced edition: exploring the brain. Jones & Bartlett Learning, 2020

work page 2020
[2]

Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval

Margaret F Carr, Shantanu P Jadhav, and Loren M Frank. Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nature neuroscience, 14 0 (2): 0 147--153, 2011

work page 2011
[3]

Coordinated memory replay in the visual cortex and hippocampus during sleep

Daoyun Ji and Matthew A Wilson. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature neuroscience, 10 0 (1): 0 100--107, 2007

work page 2007
[4]

Brain-inspired feature exaggeration in generative replay for continual learning

Jack Millichamp and Xi Chen. Brain-inspired feature exaggeration in generative replay for continual learning. arXiv preprint arXiv:2110.15056, 2021

work page Pith review arXiv 2021
[5]

Upgrading the sleeping brain with targeted memory reactivation

Delphine Oudiette and Ken A Paller. Upgrading the sleeping brain with targeted memory reactivation. Trends in cognitive sciences, 17 0 (3): 0 142--149, 2013

work page 2013
[6]

Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles

Yu-Lin Qin, Bruce L Mcnaughton, William E Skaggs, and Carol A Barnes. Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 352 0 (1360): 0 1525--1533, 1997

work page 1997
[7]

Brain-inspired continual pre-trained learner via silent synaptic consolidation

Xuming Ran, Juntao Yao, Yusong Wang, Mingkun Xu, and Dianbo Liu. Brain-inspired continual pre-trained learner via silent synaptic consolidation. arXiv preprint arXiv:2410.05899, 2024

arXiv 2024
[8]

Maintaining memories by reactivation

Bj \"o rn Rasch and Jan Born. Maintaining memories by reactivation. Current opinion in neurobiology, 17 0 (6): 0 698--703, 2007

work page 2007

Show all 14 references

[9]

Loss of recent memory after bilateral hippocampal lesions

William Beecher Scoville and Brenda Milner. Loss of recent memory after bilateral hippocampal lesions. Journal of neurology, neurosurgery, and psychiatry, 20 0 (1): 0 11, 1957

1957
[10]

Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans

Larry R Squire. Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychological review, 99 0 (2): 0 195, 1992

1992
[11]

Hippocampal offline reactivation consolidates recently formed cell assembly patterns during sharp wave-ripples

Gido M Van de Ven, Stephanie Trouche, Colin G McNamara, Kevin Allen, and David Dupret. Hippocampal offline reactivation consolidates recently formed cell assembly patterns during sharp wave-ripples. Neuron, 92 0 (5): 0 968--974, 2016

2016
[12]

Brain-inspired replay for continual learning with artificial neural networks

Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11 0 (1): 0 4069, 2020

2020
[13]

Reactivation of hippocampal ensemble memories during sleep

Matthew A Wilson and Bruce L McNaughton. Reactivation of hippocampal ensemble memories during sleep. Science, 265 0 (5172): 0 676--679, 1994

1994
[14]

Continual learning through synaptic intelligence, 2017

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence, 2017. URL https://arxiv.org/abs/1703.04200

2017 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Neuroscience: exploring the brain, enhanced edition: exploring the brain

Mark Bear, Barry Connors, and Michael A Paradiso. Neuroscience: exploring the brain, enhanced edition: exploring the brain. Jones & Bartlett Learning, 2020

work page 2020

[2] [2]

Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval

Margaret F Carr, Shantanu P Jadhav, and Loren M Frank. Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nature neuroscience, 14 0 (2): 0 147--153, 2011

work page 2011

[3] [3]

Coordinated memory replay in the visual cortex and hippocampus during sleep

Daoyun Ji and Matthew A Wilson. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature neuroscience, 10 0 (1): 0 100--107, 2007

work page 2007

[4] [4]

Brain-inspired feature exaggeration in generative replay for continual learning

Jack Millichamp and Xi Chen. Brain-inspired feature exaggeration in generative replay for continual learning. arXiv preprint arXiv:2110.15056, 2021

work page Pith review arXiv 2021

[5] [5]

Upgrading the sleeping brain with targeted memory reactivation

Delphine Oudiette and Ken A Paller. Upgrading the sleeping brain with targeted memory reactivation. Trends in cognitive sciences, 17 0 (3): 0 142--149, 2013

work page 2013

[6] [6]

Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles

Yu-Lin Qin, Bruce L Mcnaughton, William E Skaggs, and Carol A Barnes. Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 352 0 (1360): 0 1525--1533, 1997

work page 1997

[7] [7]

Brain-inspired continual pre-trained learner via silent synaptic consolidation

Xuming Ran, Juntao Yao, Yusong Wang, Mingkun Xu, and Dianbo Liu. Brain-inspired continual pre-trained learner via silent synaptic consolidation. arXiv preprint arXiv:2410.05899, 2024

arXiv 2024

[8] [8]

Maintaining memories by reactivation

Bj \"o rn Rasch and Jan Born. Maintaining memories by reactivation. Current opinion in neurobiology, 17 0 (6): 0 698--703, 2007

work page 2007