Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Chained Tuning Leads to Biased Forgetting

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chained fine-tuning of LLMs causes order-dependent, group-uneven loss of safety knowledge.

desk verdict Chained tuning order matters: safety tasks forget more when trained first, consistently across nine pairs; the biased-forgetting magnitudes need error bars and a neutral control before the group-level claims are treated as firm. read the letter →

arxiv 2412.16469 v2 pith:WH4UEIG6 submitted 2024-12-21 cs.CL

Megan Ung , Alicia Sun , Samuel J. Bell , Bhaktipriya Radharapu , Levent Sagun , Adina Williams This is my paper · ORCID
classification cs.CL
keywords catastrophicforgettingcontinuallearningsafetytuningbiasedtaskorderinglargelanguagemodelsHessianspectralradiusdemographicgroups
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that when a large language model is fine-tuned first on a safety or bias task and then on an ordinary capability task, the safety behavior is forgotten much more heavily than the capability is forgotten in the reverse order. It introduces a metric called biased forgetting to capture a second, less visible effect: the safety knowledge that is lost is not spread evenly across demographic groups, and some groups lose considerably more than the average. The authors argue that the asymmetry is tied to the sharpness of the loss landscape at the end of the first training task, with safety tasks converging to sharper minima that are more easily overwritten, and they show that a brief replay of a small slice of safety data can recover most of the lost behavior. A sympathetic reader would care because real deployment pipelines routinely chain safety tuning and downstream capability tuning, so the claim implies that task order is a safety-relevant decision and that safety evaluation needs to be reported per group.

What carries the argument

The central measuring device is the forgetting metric $\mathrm{Forgetting}_{AB} = \mathrm{Acc}_A(\theta^*_A) - \mathrm{Acc}_A(\theta^*_{AB})$, with a group-level version and the biased-forgetting gap $\mathrm{BiasedForgetting}_{A,g} = \mathrm{Forgetting}_{A,g} - \mathrm{Forgetting}_A$. The explanatory mechanism is the curvature of the first-task minimum: using the Taylor bound $L_A(\theta^*_{AB})-L_A(\theta^*_A) \approx \frac12 \Delta\theta^{*\top} H \Delta\theta^* \le \frac12 \rho(H) \|\Delta\theta^*\|^2$, the paper measures the spectral radius $\rho(H)$ of the first task's Hessian by power iteration and finds that safety tasks like ToxiGenQA and BBQ end in sharper minima than capability tasks, so a fixed parameter move in the second task costs them more loss.

What would settle it

A decisive test: retrain the safety→capability chains, then evaluate on ToxiGenQA and BBQ with the response format re-supplied, e.g., reminding the model to end with 'The best answer is...' or 'This is toxic'; if accuracy recovers to near pre-capability levels whenever the format is restored, the observed forgetting is mostly template loss rather than safety-knowledge loss.

Watch

Extended reading notes

Core claim

On a 7-billion-parameter open language model, the paper reports that for every one of nine pairs of one capability task and one safety task, forgetting of the first task is far larger when the safety task was trained first. Representative numbers from Table 1: ARC→BBQ forgetting is 1.02 percentage points while BBQ→ARC forgetting is 43.45, and CQA2→ToxiGenQA is 9.29 while ToxiGenQA→CQA2 is 38.05. The authors define forgetting as the accuracy drop on the first task after the second task, and biased forgetting as the gap between a demographic group's forgetting and the average forgetting. They report maximum biased forgetting values up to 10.09 on ToxiGenQA→CQA2, with Muslim, Jewish, and Native American groups consistently among the worst-affected in ToxiGenQA and Age and Disability Status among the worst in BBQ. They also show that the Hessian spectral radius after the first task correlates with later forgetting, and that a third fine-tuning stage on as little as 5% of the safety data, or about five optimization steps, restores most of the lost safety accuracy.

Load-bearing premise

The central numbers assume that a drop on ToxiGenQA, BBQ, and SaFeRDialogues measures loss of safety knowledge, not simply loss of the response template or scoring format the model was using for those tasks.

Editorial extensions

If this is right

  • Chained tuning should be treated as a safety decision: for all nine (capability, safety) task pairs, safety-task forgetting is larger when the safety task is tuned first, with BBQ→ARC reaching 43.45% forgetting versus 1.02% for ARC→BBQ.
  • Safety evaluations need group-level reporting: maximum biased forgetting reaches 10.09 points on ToxiGenQA→CQA2, and the groups losing most are not the average groups.
  • First-task minima width predicts forgetting: tasks that end in sharper minima, such as BBQ and ToxiGenQA, are forgotten more, so curvature measurements could guide task ordering.
  • Rehearsal is a cheap mitigation: re-tuning on 5% of the safety data or about five optimization steps nearly restores safety accuracy with only a minor capability drop.
  • Capability tuning after safety tuning can degrade safety even when the capability task is benign, so downstream fine-tuning pipelines should audit for this rather than only tracking final-task accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If format loss rather than content loss drives the numbers, the true safety deficit from chained tuning may be smaller than the headline table suggests; the same confound could inflate the apparent group disparity if some groups' examples are easier to answer from surface patterns.
  • The curvature result suggests a practical rule the paper does not state outright: when chaining tunings, measure the first task's Hessian spectral radius and put the wider-minimum task first; this could be tested cheaply before committing to a full fine-tuning run.
  • A natural extension is to repeat the ordering experiment on a more recent base model and on non-QA capabilities like code or summarization; persistence of the ordering asymmetry outside QA formats would make the safety implication much broader.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies catastrophic forgetting in chained supervised fine-tuning of LLaMA-2 7B, comparing sequences in which a capability task (ARC, CommonsenseQA, CommonsenseQA 2.0) is followed by a safety/bias task (ToxiGenQA, BBQ, SaFeRDialogues) with the reverse order. It introduces a metric called biased forgetting, defined as the gap between group-level and average forgetting, and reports three main findings: (i) safety tasks are forgotten more when they are tuned first and then followed by a capability task than capability tasks are forgotten when the order is reversed; (ii) forgetting is unevenly distributed across demographic groups, with certain groups (e.g., Muslim, Jewish, Native American in ToxiGenQA; Age and Disability Status in BBQ) consistently suffering more; and (iii) first-task loss curvature (spectral radius of the Hessian) correlates with downstream forgetting, and replaying a small portion of the safety data or using a low second-task learning rate mitigates forgetting. The paper frames these results as evidence that task ordering is a safety-relevant training decision and that safety evaluations should be group-disaggregated.

Significance. If the central claims hold, the paper is a valuable empirical contribution to LLM continual learning and safety: it systematically compares nine task pairs with fixed hyperparameters, uses an openly available base model, repeats experiments with three seeds, and proposes a concrete group-level metric. The finding that safety tuning is disproportionately forgotten by subsequent capability tuning, and that the effect varies by demographic group, would have direct implications for how open-weight models are fine-tuned downstream. The paper also gives a practical mitigation (replay with as little as 5% of safety data) and connects the results to flat-minima theory. However, the strength of the conclusions currently depends on ruling out a format/content confound and on making the group-level metric statistically robust; these are addressable within the manuscript's scope.

major comments (4)
  1. [§3.2, §7; Table 1] The central asymmetry may partly reflect forgetting of the response template rather than loss of safety-relevant knowledge. ToxiGenQA is reformatted with a unique verbalizer ('This is toxic/not toxic') while ARC/CQA/BBQ use 'The best answer is [letter]' and CQA2 uses 'The answer is yes/no'; under exact-match evaluation, a model that still recognizes toxicity but emits the wrong format is scored as unsafe. The authors acknowledge task similarity as a confound in §7 but do not isolate it experimentally: SaFeRDialogues, the only generative task scored by a classifier, shows much smaller forgetting (0.84–4.36%) than ToxiGenQA (3.98–38.05%) or BBQ (13.52–46.85%), which is consistent with a format-based account. The reverse ordering is not an adequate control because it compares different tasks. I request a matched neutral-content control with the same template, number of choices, and difficulty, or a robustness analysis that evaluates ToxiGenQA/BBQ with semantic equivalence or accepts both template variants.
  2. [§3.2, Eq. (2); Figure 3] The biased forgetting metric is an absolute difference between group forgetting and average forgetting, with no normalization by baseline accuracy. Groups with higher baseline accuracy have more room to drop, so the reported BiasedForgettingmax values could partly be a floor/ceiling artifact. The paper should report per-group baseline accuracies and the group-level relative forgetting (defined in §3.2 but not used for groups), and provide confidence intervals via bootstrapping with a multiple-comparison correction across the 13 ToxiGenQA groups and 9 BBQ groups. Without these, the claim that forgetting disproportionately impacts certain groups is not yet quantified in a way that rules out trivial scale effects.
  3. [§4.1, Table 1] The headline ordering asymmetry is reported as point estimates averaged over three seeds, without standard deviations or confidence intervals. This is a comparative claim across nine task pairs, and the paper should report variability; a paired or mixed-effects test of the ordering effect would strengthen the claim. The absence of error bars in Figures 5–7 also makes it difficult to assess whether the learning-rate trends are reliable, especially where the paper itself states the picture is unclear for the first-task learning rate.
  4. [§4.3, Eqs. (3)–(4); Figure 4(b)] The curvature explanation is a post-hoc regression fitted to the same six tasks used in the main experiments, not an independent test of the hypothesis. With six task-level points, or with repeated observations if each task contributes multiple downstream pairs, the reported OLS p ≤ 0.01 and R² = 0.85 are fragile; the spectral radius is estimated from 50 training samples with no variance or convergence diagnostics reported. This section should be explicitly framed as exploratory, and the predictive claim should be validated, e.g., by leave-one-task-out prediction or by pre-registering the curvature hypothesis on held-out task pairs.
minor comments (5)
  1. [§4.3] The text 'Section 4.3 shows a markedly sharper minima...' should refer to Figure 4(a), not to the current section; the same self-reference appears later in the paragraph.
  2. [§3.2] The formula for relative forgetting is ambiguous: it writes RelativeForgettingAB with a denominator AccA(θ∗A,g), but the left-hand side has no group subscript. Please clarify whether relative forgetting is defined per group or overall.
  3. [Table 3] The table title says 'ToxigenQA accuracy across different settings' but the table lists accuracy for ToxiGenQA, BBQ, and SaFeRDialogues; the title should reflect all three tasks.
  4. [Figure 10] The caption states that the blue dotted vertical line denotes the average forgetting on ToxiGenQA, but Figure 10 shows BBQ results; it should say BBQ.
  5. [Footnote 2] The footnote says 'see footnote 1' for the ToxiGen classifier, but footnote 1 is about the definition of safety; the reference to the classifier should be a separate footnote describing the model and scoring procedure.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the core claims are direct measurements, the SaFeRDialogues self-citation is not load-bearing, and the curvature analysis is a post-hoc fit rather than a prediction.

full rationale

The paper's principal claims—that safety-tuned models forget more when subsequently capability-tuned than in the reverse order, and that forgetting is uneven across demographic groups—are direct accuracy measurements expressed through the operational definitions in Eqs. (1) and (2). Biased forgetting is defined as a deviation-from-average metric, but the observed values, group rankings, and Kendall's tau consistency are data-dependent and not forced by the definition. The only self-citation is the SaFeRDialogues dataset (Ung et al., 2022), and the headline asymmetry is driven by ToxiGenQA and BBQ; SaFeRDialogues actually shows the smallest forgetting, so the self-citation is not load-bearing. The curvature analysis in Section 4.3 is an in-sample OLS regression on the six studied tasks, presented as a possible explanation rather than a held-out prediction; this is a fitted correlation with limited inferential strength, but it does not reduce the forgetting claim to its inputs by construction. The paper explicitly acknowledges task-format similarity as a confound in Section 7, which is a validity limitation and not a circularity. No definition, fitted parameter, or self-citation is used both as premise and conclusion in a way that forces the central empirical result.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on experimental measurements rather than new theoretical constructs. The only fitted quantity is the small-N curvature-forgetting regression. The main assumptions are measurement-level (what the safety benchmarks capture, Hessian approximation) rather than mathematical axioms. No new entities are postulated.

free parameters (1)
  • OLS slope and intercept for forgetting vs spectral radius = not reported
    Section 4.3 fits a linear regression of mean downstream forgetting on Hessian spectral radius across the 6 tasks (R-squared 0.85, p at most 0.01); this is a fitted correlation, not a predictive model, and no held-out tasks validate it.
assumptions (3)
  • domain assumption Taylor expansion (Eq. 3) applies because the first-task model theta*_A is at or near a local minimum of L_A
    The Hessian-based bound in Eqs. (3)-(4) is used in Section 4.3 to link minima curvature to forgetting; the authors say they validate this by inspecting training losses and gradient norms, but the validation details are not shown.
  • domain assumption The Hessian spectral radius estimated with 50 random training samples and power iteration approximates the true curvature of the first-task loss
    Section 4.3 states the Hessian is computed using 50 random samples to improve efficiency; no convergence checks or comparison with full-Hessian estimates are reported, so estimates may be noisy.
  • domain assumption Accuracy on ToxiGenQA and BBQ, and ToxiGen-classifier scores of SaFeRDialogues continuations, measure the safety-relevant behavior that is claimed to be forgotten
    Safety tasks are reformatted into QA templates (Section 3.2, Section 8.2); if models forget the output template rather than safety content, the forgetting numbers overstate safety loss. The paper notes task-format similarity as a confound in Section 7 but does not disentangle format and content forgetting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chained Tuning Leads to Biased Forgetting." pith.science (2026). https://pith.science/paper/WH4UEIG6

@misc{pith2026241216469,
  author       = {Pith},
  title        = {Pith review of: Chained Tuning Leads to Biased Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WH4UEIG6}},
  note         = {Machine review of arXiv:2412.16469}
}
read the original abstract

Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has important potential implications for the safety of deployed models. In this work, we first show that models trained on downstream tasks forget their safety tuning to a greater extent than models trained in the opposite order. Second, we show that forgetting disproportionately impacts safety information about certain groups. To quantify this phenomenon, we define a new metric we term biased forgetting. We conduct a systematic evaluation of the effects of task ordering on forgetting and apply mitigations that can help the model recover from the forgetting observed. We hope our findings can better inform methods for chaining the finetuning of LLMs in continual learning settings to enable training of safer and less toxic models.

Figures

Figures reproduced from arXiv: 2412.16469 by the authors.

Figure 1
Figure 1. Task ordering experimental set up. Finetuning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Relative Forgetting (%) of the capability task [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Forgetting by groups in ToxiGenQA (a, b, c) and BBQ (d, e, f) followed by finetuning on a capability [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: (a) Minima curvature (i.e. approx. spectral radius of the Hessian) obtained after training on the first task. (b) Mean downstream forgetting as a function of first task curvature. Curvature explains a significant proportion of downstream forgetting. Error bars are stan…
Figure 5
Figure 5. Figure 5: Forgetting when varying first task learning rate for different safety task [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Forgetting when varying second task learning rate for different safety task [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: We perform a third stage of finetuning (Safety [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Forgetting on the first task the model is fine-tuned on across learning rates for first task and second task. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Biased Forgetting by groups in SFT ToxiGenQA followed by SFT on a capability task (ARC, CQA, [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Biased Forgetting by groups in SFT BBQ followed by SFT on a capability task (ARC, CQA, CQA2). All [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Safety task accuracy after third stage of [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Safety task forgetting after third stage of [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The Claude 3 Model Family: Opus, Sonnet, Haiku

    work page 2024

  4. [4]

    Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dar...

    arXiv 2022

  5. [5]

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul R \"o ttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875

    arXiv 2023

  6. [6]

    Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of 'bias' in nlp. Association for Computational Linguistics (ACL)

    work page 2020

  7. [7]

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2018. https://openreview.net/forum?id=Hkf2_sC5FX&utm_referrer=https

    work page 2018

  8. [8]

    Dokania, Philip H

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc'Aurelio Ranzato. 2019. https://doi.org/10.48550/arXiv.1902.10486 On Tiny Episodic Memories in Continual Learning . arXiv preprint. ArXiv:1902.10486 [cs, stat]

    arXiv doi:10.48550/arxiv.1902.10486 2019

Show all 47 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    2018 arXiv

  2. [10]

    Cyprien de Masson d Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/f8d2e80c1458ea2501f98a2cafadb397-Paper.pdf Episodic memory in lifelong language learning . In Advances in Neural Information Processi...

    2019

  3. [11]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3356--3369

    2020

  4. [12]

    Noah Golmant, Zhewei Yao, Amir Gholami, Michael Mahoney, and Joseph Gonzalez. 2018. https://github.com/noahgolmant/pytorch-hessian-eigenthings pytorch-hessian-eigenthings: efficient pytorch hessian eigendecomposition

    2018

  5. [13]

    Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2014. An empirical investigation of catastrophic forgetting in gradient-based neural networks. 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings. ArXiv...

    2014 arXiv

  6. [14]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://arxiv.org/abs/2203.09509 Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection . Preprint, arXiv:2203.09509

    2022 arXiv

  7. [15]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. https://doi.org/10.1162/neco.1997.9.1.1 Flat minima . Neural Computation, 9(1):1--42

    1997 doi

  8. [16]

    Saghar Hosseini, Hamid Palangi, and Ahmed Hassan Awadallah. 2023. https://arxiv.org/abs/2301.09211 An empirical study of metrics to measure representational harms in pre-trained language models . Preprint, arXiv:2301.09211

    2023 arXiv

  9. [17]

    Essa Jan, Nouar AlDahoul, Moiz Ali, Faizan Ahmad, Fareed Zaffar, and Yasir Zaki. 2024. https://arxiv.org/abs/2409.15361 Multitask mayhem: Unveiling and mitigating safety gaps in llms fine-tuning . Preprint, arXiv:2409.15361

    2024 arXiv

  10. [18]

    Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. 2018. https://doi.org/10.48550/arXiv.1711.04623 Three Factors Influencing Minima in SGD . arXiv preprint. ArXiv:1711.04623 [cs, stat]

    2018 doi arXiv

  11. [19]

    Hayes, and Christopher Kanan

    Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. 2018. https://doi.org/10.1609/AAAI.V32I1.11651 Measuring catastrophic forgetting in neural networks . In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18)...

    2018 doi

  12. [21]

    Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A

    James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska - Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. https://arxiv.org/abs/161...

    2016 arXiv

  13. [22]

    Springer, and Aditi Raghunathan

    Suhas Kotha, Jacob M. Springer, and Aditi Raghunathan. 2023. https://api.semanticscholar.org/CorpusID:262054014 Understanding catastrophic forgetting in language models via implicit inference . ArXiv, abs/2309.10105

    2023 arXiv

  14. [23]

    Chen-An Li and Hung-Yi Lee. 2024. https://api.semanticscholar.org/CorpusID:266844262 Examining forgetting in continual pre-training of aligned large language models . ArXiv, abs/2401.03129

    2024 arXiv

  15. [24]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  16. [25]

    David Lopez-Paz and Marc' Aurelio Ranzato. 2017. https://proceedings.neurips.cc/paper/2017/hash/f87522788a2be2d171666752f97ddebb-Abstract.html Gradient Episodic Memory for Continual Learning . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc

    2017

  17. [26]

    Ilya Loshchilov and Frank Hutter. 2017. https://api.semanticscholar.org/CorpusID:53592270 Decoupled weight decay regularization . In International Conference on Learning Representations

    2017

  18. [27]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. https://api.semanticscholar.org/CorpusID:261031244 An empirical study of catastrophic forgetting in large language models during continual fine-tuning . ArXiv, abs/2308.08747

    2023 arXiv

  19. [28]

    Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2024. Keeping LLMs aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402.18540

    2024 arXiv

  20. [29]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  21. [30]

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. 2020 a . http://arxiv.org/abs/2010.04495 Linear Mode Connectivity in Multitask and Continual Learning . ArXiv: 2010.04495

    2020 arXiv

  22. [31]

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. 2020 b . https://api.semanticscholar.org/CorpusID:219636010 Understanding the role of training regimes in continual learning . ArXiv, abs/2006.06958

    2020 arXiv

  23. [32]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

    2022 doi

  24. [33]

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nathan McAleese, and Geoffrey Irving. 2022. https://api.semanticscholar.org/CorpusID:246634238 Red teaming language models with language models . In Conference on Empirical Methods...

    2022

  25. [34]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. https://api.semanticscholar.org/CorpusID:263671523 Fine-tuning aligned language models compromises safety, even when users do not intend to! ArXiv, abs/2310.03693

    2023 arXiv

  26. [35]

    Ramasesh, Ethan Dyer, and Maithra Raghu

    Vinay V. Ramasesh, Ethan Dyer, and Maithra Raghu. 2020. http://arxiv.org/abs/2007.07400 Anatomy of Catastrophic Forgetting : Hidden Representations and Task Semantics . pages 1--26. ArXiv: 2007.07400

    2020 arXiv

  27. [36]

    Roger Ratcliff. 1990. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285

    1990

  28. [37]

    Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. https://openreview.net/pdf?id=UJTgQBc91\_ Progressive prompts: Continual learning for language models . In The Eleventh International Conference on Learning Representations, ICL...

    2023

  29. [38]

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc

    2019

  30. [39]

    Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.410 Fine-tuned language models are continual learners . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107--6122, Abu D...

    2022 doi

  31. [40]

    i'm sorry to hear that

    Eric Michael Smith, Melissa Hall Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. https://api.semanticscholar.org/CorpusID:248887683 "i'm sorry to hear that": finding bias in language models with a holistic descriptor dataset . ArXiv, abs/2205.09209

    2022 arXiv

  32. [41]

    Tony Sun, Akhila Gaut, Shang-Wen Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, ..., and Kai-Wei Chang. 2019. Mitigating gender bias in natural language processing: Literature review. In arXiv preprint arXiv:1906.08976

    2019 arXiv

  33. [42]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://api.semanticscholar.org/CorpusID:53296520 Commonsenseqa: A question answering challenge targeting commonsense knowledge . ArXiv, abs/1811.00937

    2019 arXiv

  34. [43]

    Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. https://api.semanticscholar.org/CorpusID:237263476 Commonsenseqa 2.0: Exposing the limits of ai through gamification . ArXiv, abs/2201.05320

    2021 arXiv

  35. [44]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  36. [45]

    Megan Ung, Jing Xu, and Y-Lan Boureau. 2022. https://arxiv.org/abs/2110.07518 Saferdialogues: Taking feedback gracefully after conversational safety failures . Preprint, arXiv:2110.07518

    2022 arXiv

  37. [46]

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2024. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  38. [47]

    Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Christopher Griffin, Kamilė Huang, John Mellor, ..., and Iason Gabriel. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  39. [48]

    Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. https://proceedings.mlr.press/v70/zenke17a.html Continual Learning Through Synaptic Intelligence . In Proceedings of the 34th International Conference on Machine Learning , pages 3987--3995. PMLR. ISSN: 2640-3498

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.