REVIEW 3 major objections 4 minor 23 references
Replay to Remember: Retaining Domain Knowledge in Streaming Language Models
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Replay buffers, even minimal ones, steady a streaming language model's domain memory.
desk verdict A well-intentioned pilot study whose central replay claim is not supported because the reported experiments lack any no-replay control condition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The operative mechanism is a two-part continual-learning loop. LoRA (Low-Rank Adaptation) freezes the base model and trains only small low-rank matrices added to attention layers, cutting computational cost; the replay buffer periodically reintroduces previously seen question-answer chunks into the training stream. The paper measures the loop's behavior with perplexity as predictive uncertainty, cosine similarity of generated answers to baseline outputs as semantic drift, and GPT-4 ratings of answer quality on a 1-10 scale. The replay buffer is the part claimed to carry the stabilization and recovery, while LoRA is the part claimed to make the whole procedure feasible under memory and compute constraints.
What would settle it
Run the same streaming protocol twice, once with the replay buffer enabled and once with it disabled, holding LoRA, dataset order, chunk sizes, and evaluation prompts fixed; if the no-replay run shows the same perplexity trends and recovery, the replay mechanism is not the cause of the stabilization.
Extended reading notes
Core claim
The central claim is that a lightweight combination of LoRA and a minimal replay buffer mitigates catastrophic forgetting in a realistic streaming setting. Sequentially exposing a transformer LLM to medical question answering, genetic-disease Q&A, and legal dictionary data, the authors observe domain-specific perplexity spikes, such as MedQuAD rising from 121.42 to 20402.01 and genetics rising to over 326,000, while replay yields partial stabilization and recovery in semantic similarity and GPT-4 ratings. The consistent ordering, with genetics suffering the most forgetting and showing the clearest post-replay recovery and law remaining stable throughout, leads the paper to conclude that replay's benefit is real but domain-dependent, and that LoRA-equipped models resist total collapse even without continuous replay.
Load-bearing premise
The argument rests on the assumption that the observed stabilization and partial recovery are caused by the replay buffer, but the experiments report no comparison run without replay, so the effect could in principle come from LoRA's fine-tuning dynamics or from re-exposure to similar data.
Editorial extensions
If this is right
- Replay, even at minimal fixed size, can keep a streaming LoRA model from collapsing on earlier domains, so continual deployments may not need large memory stores.
- Domain difficulty matters: genetics-like volatile content will need stronger memory mechanisms, while law-like stable content may need little or no replay.
- A single metric can mislead: perplexity spikes while GPT-4 ratings hold, so multi-metric evaluation is necessary for judging real-time adaptation.
- The approach works under modest compute, making continual adaptation plausible for edge or resource-constrained LLM applications.
- Since LoRA alone appears to provide some resilience, future systems might combine adapter-level modularity with replay for better retention.
Reading between the lines
- The paper does not isolate replay's contribution from LoRA's; a direct replay-on/replay-off ablation is the most obvious next experiment and would also tell us how large the replay benefit actually is.
- The Law domain's stability could stem from pretraining overlap or from legal language being structurally uniform; testing on additional domains with controlled pretraining exposure would show whether replay or domain regularity drives the result.
- Because the replay buffer is fixed and unprioritized, priority-based sampling, such as re-rehearsing the most mistaken or most informative examples, is a natural extension that could improve recovery efficiency.
- The reported recovery in genetics suggests that even a small amount of rehearsal can trigger re-learning; this could be tested by varying buffer size and measuring recovery slope across chunks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a lightweight continual-learning method that combines LoRA fine-tuning with a minimal replay buffer for streaming domain adaptation of LLMs, and evaluates it on three knowledge domains (MedQuAD, genetics, and law). The model is trained on sequential chunks from these domains, and performance is tracked with perplexity, cosine similarity to baseline outputs, and GPT-4-based ratings. The authors report that catastrophic forgetting occurs, but that replay 'significantly stabilizes and partially restores' domain knowledge, with law being the most stable and genetics the most volatile domain. The manuscript concludes that adaptive retention and domain-sensitive stability are achievable without catastrophic forgetting, while acknowledging several limitations such as a fixed replay size and limited evaluation prompts.
Significance. If the causal claim were established, the practical value would be real: a resource-constrained, streaming continual-learning recipe that preserves some domain knowledge with modest hardware and simple components. The paper has explicit strengths: it uses realistic domain datasets, employs three complementary evaluation metrics, and is honest about its limitations. However, the central claim that replay is the operative mechanism is not supported by the experimental design, because no no-replay control condition is reported. The evaluation is also a single-run, single-configuration trajectory with no statistical assessment, so the title claim of 'significantly stabilizes' is not grounded in the data as presented. The paper is better viewed as a preliminary observational study than as a demonstrated causal comparison.
major comments (3)
- [§3, Tables 1–3; §4] The central causal claim that replay stabilizes and restores knowledge is unsupported because the paper reports no no-replay control condition. Tables 1 through 3 show metrics from a single replay-augmented run only; the observed stability in Law, the partial recovery in MedQuAD (cosine similarity 0.72 to 0.78), and the Genetic rebound (GPT-4 rating 3.2 to 5.0) could all be produced by LoRA fine-tuning dynamics, domain difficulty, chunk order, or pretraining exposure. Section 4 explicitly weakens the replay attribution by stating that the system can 'prevent complete degradation even without continuous replay' and that 'LoRA-based adapters have an inherent capacity to preserve structural or factual patterns.' To support the abstract and conclusion claims, the authors must add an otherwise identical ablation with replay disabled, across the same chunk order and domains, and compare the trajectories statistically.
- [Abstract; §3] The word 'significantly' is used to describe the effect of replay, but no significance test, confidence interval, seed variation, or repeated trial is reported. Tables 1 and 2 contain a single value per chunk per domain, and Table 3 contains a single rating per chunk per domain. Without multiple independent runs or at least a held-out evaluation with quantified variance, the reader cannot determine whether the observed differences (e.g., MedQuAD similarity 0.72 vs. 0.78, Genetic rating 3.2 vs. 5.0) are meaningful or within run-to-run variability. Please provide repeated-seed results and a statistical comparison, or else soften the causal and significance language throughout.
- [§3.1, Table 1] The perplexity values in Table 1 span an enormous range, from 121.42 at chunk 0 to 326,263.46 at chunk 4. Such extreme spikes suggest either that the model becomes essentially random on some held-out sets or that the perplexity is computed over different numbers of tokens or different prompts across chunks. This makes the 'recovery' claims in Tables 2 and 3 hard to interpret: it is unclear whether the same evaluation set is used each time, whether the baseline answers were generated from the chunk-0 checkpoint only, and how a model with perplexity of 326K can simultaneously show a semantic similarity of 0.70. Please specify the exact held-out prompts, the tokenization, and the length normalization for perplexity, and verify that the evaluation protocol is identical across chunks.
minor comments (4)
- [§2.2, §2.5] The replay buffer size, chunk size, number of training steps, LoRA rank/alpha, and the exact 'modest computational hardware' are not specified. Please report these values to make the experiments reproducible.
- [Figures 1–3] The text repeatedly refers to Figures 1, 2, and 3, but no figure images are included in the manuscript. Either include the figures or replace the figure references with explicit descriptions of the corresponding table rows.
- [§3.2, text after Table 2] The text states that MedQuAD semantic similarity 'remains above 0.8,' but Table 2 lists MedQuAD cosine similarity as 0.72 at chunk 3. This is an internal inconsistency that should be corrected.
- [References] Several citations are to Medium posts, blog entries, or vendor pages rather than peer-reviewed sources, including the citations for semantic similarity, GPT-based evaluation, and catastrophic forgetting in LLMs. Where available, please replace these with archival or peer-reviewed references.
Circularity Check
No circularity: empirical measurements are external to the replay claim; missing no-replay control is a validity gap, not a derivation loop.
full rationale
The paper contains no derivation chain whose conclusion is equivalent to its premises. The method is empirical: LoRA with a replay buffer is run on streaming chunks, and perplexity, cosine similarity to chunk-0 baseline outputs, and GPT-4 ratings are measured externally. None of these metrics is defined in terms of the claim that replay stabilizes or restores knowledge; that claim is a causal interpretation of the measurements. The absence of a no-replay control arm weakens attribution and significance, but this is an internal-validity or experimental-design concern, not circularity. No fitted parameter is renamed as a prediction, no load-bearing self-citation appears (the reference list contains no works by the author), no uniqueness theorem is imported from prior work by the same authors, and no ansatz is smuggled in via citation. The evaluation is therefore self-contained as an empirical study, even though the replay-specific causal conclusion would need a control comparison to be fully supported.
Assumptions & free parameters
free parameters (2)
- Replay buffer size =
Not reported
- LoRA rank and alpha =
Not reported
assumptions (4)
- domain assumption Perplexity is a valid measure of domain knowledge retention
- domain assumption GPT-4 ratings approximate human judgment of answer quality
- domain assumption The named datasets are representative of their domains
- standard math Standard deep learning and LoRA training assumptions
Cite this review
Pith. "Pith review of Replay to Remember: Retaining Domain Knowledge in Streaming Language Models." pith.science (2026). https://pith.science/paper/JKMVSCVZ
@misc{pith2026250417780,
author = {Pith},
title = {Pith review of: Replay to Remember: Retaining Domain Knowledge in Streaming Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/JKMVSCVZ}},
note = {Machine review of arXiv:2504.17780}
}
read the original abstract
Continual learning in large language models (LLMs) typically encounters the critical challenge of catastrophic forgetting, where previously acquired knowledge deteriorates upon exposure to new data. While techniques like replay buffers and parameter-efficient tuning (e.g., Low-Rank Adaptation or LoRA) have been proposed, few studies investigate real-time domain adaptation under strict computational and data-stream constraints. In this paper, we demonstrate a lightweight method combining LoRA and a minimal replay mechanism in a realistic streaming setting across three diverse knowledge domains: medical question answering, genetics, and law. Using perplexity, semantic similarity, and GPT-based human-like evaluation metrics, we quantify the model's adaptation, forgetting, and recovery over time. Our experiments reveal that while catastrophic forgetting naturally occurs, even minimal replay significantly stabilizes and partially restores domain-specific knowledge. This study contributes practical insights for deploying adaptable LLMs in resource-constrained, real-world scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
Reliably evaluate llm models with semantic similarity
Aguru. Reliably evaluate llm models with semantic similarity. https://aguru.com/blog/evaluate-llm-models/, 2024
work page 2024
-
[2]
Catastrophic forgetting: The essential guide
Nightfall AI. Catastrophic forgetting: The essential guide. https://www.nightfall.ai/ai-security-101/catastrophic-forgetting, 2024
work page 2024
-
[3]
A question-entailment approach to question answering
Asma Ben Abacha and Dina Demner-Fushman. A question-entailment approach to question answering. BMC Bioinformatics, 20 0 (1): 0 511, 2019. doi:10.1186/s12859-019-3119-4. URL https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3119-4
-
[4]
Semantic similarity evaluation of llms
Vlad Bogolin. Semantic similarity evaluation of llms. https://www.linkedin.com/pulse/semantic-similarity-evaluation-llms-vlad-bogolin-8tzre, 2023
work page 2023
-
[5]
Llm evaluation metrics: The ultimate llm evaluation guide
Confident AI Team . Llm evaluation metrics: The ultimate llm evaluation guide. https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation, 2023
work page 2023
-
[6]
Efficient fine-tuning with lora for llms
Databricks. Efficient fine-tuning with lora for llms. https://www.databricks.com/blog/efficient-fine-tuning-lora-guide-llms, 2023
work page 2023
-
[7]
Llm evaluation metrics and methods
Evidently AI Team . Llm evaluation metrics and methods. https://www.evidentlyai.com/llm-guide/llm-evaluation-metrics, 2025
work page 2025
-
[8]
Bryan A. Garner. Black's Law Dictionary . Thomson Reuters, St. Paul, MN, 11th edition, 2019
work page 2019
Show all 23 references
-
[9]
Evaluating llms with semantic similarity
Alex Geronimo. Evaluating llms with semantic similarity. https://medium.com/@geronimo7/semscore-evaluating-llms-with-semantic-similarity-2abf5c2fadb9, 2023
2023
-
[10]
Catastrophic forgetting in llms
Cobus Greyling. Catastrophic forgetting in llms. https://cobusgreyling.medium.com/catastrophic-forgetting-in-llms-bf345760e6e2, 2024
2024
-
[11]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021
2021 arXiv
-
[12]
Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal
Jia Huang, Ling Cui, Alan Wang, Chun Yang, Xinyi Liao, Liang Song, Jing Yao, and Jia Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244, 2024
2024 arXiv
-
[13]
Lora explained: Low-rank adaptation for fine-tuning llms
Zilliz Learn. Lora explained: Low-rank adaptation for fine-tuning llms. https://medium.com/@zilliz_learn/lora-explained-low-rank-adaptation-for-fine-tuning-llms-066c9bdd0b32, 2024
2024
-
[14]
Contextual experience replay for continual learning of language agents
Yi Liu, Chao Si, Kalpesh R Narasimhan, and Shiyu Yao. Contextual experience replay for continual learning of language agents. OpenReview, 2024
2024
-
[15]
An empirical study of catastrophic forgetting in large language models during continual fine-tuning
Yifan Luo, Zhen Yang, Fan Meng, Yang Li, Jie Zhou, and Yichao Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023
2023 arXiv
-
[16]
List of metrics for evaluating llm-generated content
Microsoft. List of metrics for evaluating llm-generated content. https://learn.microsoft.com/en-us/ai/playbook/technology-guidance/generative-ai/working-with-llms/evaluation/list-of-eval-metrics, 2024
2024
-
[17]
Parameter-efficient llm finetuning with low-rank adaptation (lora)
Sebastian Raschka. Parameter-efficient llm finetuning with low-rank adaptation (lora). https://sebastianraschka.com/blog/2023/llm-finetuning-lora.html, 2023 a
2023
-
[18]
Practical tips for finetuning llms using lora (low-rank adaptation)
Sebastian Raschka. Practical tips for finetuning llms using lora (low-rank adaptation). https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms, 2023 b
2023
-
[19]
Adaptive memory replay for continual learning
Andrew Smith and Ben Jones. Adaptive memory replay for continual learning. arXiv preprint arXiv:2404.12526, 2024
2024 arXiv
-
[20]
Catastrophic forgetting in large language models
UnfoldAI. Catastrophic forgetting in large language models. https://unfoldai.com/catastrophic-forgetting-llms/, 2024
2024
-
[21]
Llm evaluations: Metrics, frameworks, and best practices
Wandb. Llm evaluations: Metrics, frameworks, and best practices. https://wandb.ai/onlineinference/genai-research/reports/LLM-evaluations-Metrics-frameworks-and-best-practices--VmlldzoxMTMxNjQ4NA, 2025
2025
-
[22]
Using llms for evaluation
Cameron R Wolfe. Using llms for evaluation. https://cameronrwolfe.substack.com/p/llm-as-a-judge, 2024
2024
-
[23]
An integrative knowledge graph for rare diseases, derived from the genetic and rare diseases information center (gard)
Qian Zhu, Dac-Trung Nguyen, Ivan Grishagin, Noel Southall, Eric Sid, and Anne Pariser. An integrative knowledge graph for rare diseases, derived from the genetic and rare diseases information center (gard). Journal of Biomedical Semantics, 11 0 (1): 0 13, 2020. doi:10.1186/s13...
2020 doi
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.