pith. sign in

arxiv: 2606.00132 · v1 · pith:LWCFZTKOnew · submitted 2026-05-28 · 💻 cs.LG · cs.AI

Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization

Dongjun Kim , Adrian de Wynter , Huancheng Chen , Heasung Kim , Haris Vikalo This is my paper

Pith reviewed 2026-06-29 08:36 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords FoLoRALoRA adaptationforgetting penaltyRayleigh quotientfoundation model preservationparameter-efficient fine-tuninggated Adam updatespreservation-adaptation trade-off
0
0 comments X

The pith

FoLoRA scores LoRA update directions via a generalized Rayleigh quotient of task utility over forgetting penalty to preserve non-target capabilities during adaptation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Foundation Preserving LoRA (FoLoRA), a method that regulates the adaptation-preservation trade-off dynamically during low-rank fine-tuning. It defines a forgetting penalty from first-order changes in pretraining-proxy activations and a task utility from downstream activations. These quantities form a generalized Rayleigh quotient whose eigenvectors supply a coordinate system for direction-wise gated Adam updates that down-weight low-utility directions. Calibration data for the penalty is obtained by sampling the pretrained model itself rather than a fixed proxy set. On math, code, and instruction-following tasks the resulting updates improve target performance while yielding the strongest aggregate retention of non-target skills among tested baselines.

Core claim

FoLoRA constructs a spectral coordinate system from the generalized Rayleigh quotient of task utility divided by forgetting penalty, then performs direction-wise gated Adam updates that attenuate directions with low utility-to-penalty ratios, thereby improving target-task performance while achieving the best measured preservation of non-target capabilities.

What carries the argument

The generalized Rayleigh quotient that ranks candidate update directions by the ratio of downstream task utility to pretraining-proxy forgetting penalty, supplying the basis for gated Adam steps.

If this is right

  • Directions that deliver high task utility per unit of estimated forgetting receive larger effective learning rates.
  • Sampling calibration activations directly from the pretrained model removes dependence on any single external proxy dataset.
  • The same spectral gating procedure applies across math, code, and instruction-following adaptation settings.
  • Aggregate preservation of non-target capabilities improves even as target performance rises.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the first-order penalty correlates reliably with downstream forgetting, the same quotient construction could be tested on full fine-tuning or other adapter families.
  • The approach implicitly treats preservation as a per-direction resource constraint rather than a global regularization term.
  • Extending the calibration sampling to include synthetic or out-of-distribution prompts might further tighten the penalty estimate.

Load-bearing premise

A first-order preservation condition computed on activations sampled from the pretrained model is sufficient to define a forgetting penalty that tracks actual degradation of non-target capabilities.

What would settle it

An experiment in which FoLoRA updates produce equal or lower non-target task scores than standard LoRA while matching or exceeding target-task gains would refute the claimed advantage of the Rayleigh-quotient gating.

read the original abstract

While finetuning effectively adapts foundation models to specialized downstream tasks, it can degrade nontarget capabilities acquired during pretraining. Existing forgetting aware methods typically seek safer updates through specialized initialization or fixed constraints, but do not regulate the adaptation preservation trade-off during training. We propose Foundation Preserving LoRA (FoLoRA), a forgetting aware optimization framework. Guided by a first order preservation condition, FoLoRA defines a forgetting penalty over pretraining-proxy activations and a task utility over downstream task activations. It then scores update directions by task utility per unit forgetting penalty via a generalized Rayleigh quotient. The resulting spectral coordinate system enables direction wise gated Adam updates, attenuating low utility to penalty directions during training. To estimate the forgetting penalty, FoLoRA constructs pretraining proxy calibration data by sampling from the pretrained model rather than relying on a single proxy dataset. Experiments on math, code, and instruction following adaptation show that FoLoRA achieves the strongest preservation adaptation balance over baselines, improving target task performance with best aggregate preservation of non target capabilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes Foundation-Preserving LoRA (FoLoRA), a forgetting-aware optimization method for adapting foundation models. It constructs a forgetting penalty from a first-order preservation condition on activations sampled from the pretrained model (pretraining-proxy data) and a task utility on downstream activations. Update directions are scored via a generalized Rayleigh quotient of utility per unit penalty; the resulting spectral basis is used for direction-wise gated Adam updates that attenuate low-utility directions. Experiments on math, code, and instruction-following adaptation tasks are reported to show that FoLoRA attains the best aggregate preservation of non-target capabilities while improving target-task performance relative to baselines.

Significance. If the first-order preservation condition reliably ranks directions by their effect on non-target capabilities, the work supplies a training-time mechanism for regulating the adaptation-preservation trade-off that is more flexible than fixed initialization or constraint methods. The use of a generalized Rayleigh quotient to induce a spectral coordinate system for gated updates is a technically interesting construction that could apply to other constrained fine-tuning settings. The proxy-data construction by direct sampling from the pretrained model avoids dependence on a single external calibration set and is a concrete strength. The significance is limited by the absence of an explicit derivation or quantitative validation that the linear penalty dominates higher-order effects.

major comments (3)
  1. [Abstract / method] Abstract and method description: the first-order preservation condition is invoked to define the forgetting penalty, yet no explicit statement of the condition, the precise form of the generalized Rayleigh quotient, or any supporting derivation is supplied. Without these, it is impossible to verify whether the quotient reduces to an implicit utility-penalty hyperparameter or whether the linear term suffices to proxy non-target capability degradation.
  2. [§4] §4 (experiments): the headline claim that FoLoRA achieves the 'strongest preservation-adaptation balance' is stated without quantitative results, error bars, ablation tables, or statistical tests. This absence makes the empirical superiority load-bearing for the central contribution impossible to evaluate.
  3. [Method / proxy data] Proxy-data construction paragraph: sampling from the pretrained model is presented as avoiding a single proxy dataset, but the procedure for choosing the sampling distribution is unspecified. If the distribution itself depends on the model under adaptation, the forgetting penalty estimate risks circularity that would undermine the preservation guarantee.
minor comments (2)
  1. [Method] Notation for the generalized Rayleigh quotient and the gated Adam update rule should be introduced with explicit equations rather than descriptive prose.
  2. [Abstract] The abstract would benefit from a single-sentence statement of the first-order condition and the exact Rayleigh-quotient objective.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments. We address each major point below and will incorporate clarifications and additional material in the revised manuscript.

read point-by-point responses

  1. Referee: [Abstract / method] Abstract and method description: the first-order preservation condition is invoked to define the forgetting penalty, yet no explicit statement of the condition, the precise form of the generalized Rayleigh quotient, or any supporting derivation is supplied. Without these, it is impossible to verify whether the quotient reduces to an implicit utility-penalty hyperparameter or whether the linear term suffices to proxy non-target capability degradation.

    Authors: We agree that an explicit statement and derivation are needed. In the revision we will insert a dedicated subsection (new §3.2) that states the first-order preservation condition as the linear term in the Taylor expansion of activation change, defines the forgetting penalty matrix B from the expected squared norm of this change on pretraining-proxy activations, and presents the generalized Rayleigh quotient as the solution to max_u (u^T A u) / (u^T B u) where A encodes downstream utility. The resulting eigenvectors supply the spectral basis for the gated updates; this is not equivalent to a scalar hyperparameter. We will also add a short paragraph discussing the linear approximation as a computationally tractable proxy. revision: yes

  2. Referee: [§4] §4 (experiments): the headline claim that FoLoRA achieves the 'strongest preservation-adaptation balance' is stated without quantitative results, error bars, ablation tables, or statistical tests. This absence makes the empirical superiority load-bearing for the central contribution impossible to evaluate.

    Authors: We accept that the experimental claims require fuller quantitative support. The revision will expand §4 with complete tables reporting mean and standard deviation over five random seeds for all tasks, ablation tables isolating the contribution of the Rayleigh-quotient gating, and paired statistical tests (t-tests with Bonferroni correction) comparing FoLoRA against each baseline on both target and non-target metrics. revision: yes

  3. Referee: [Method / proxy data] Proxy-data construction paragraph: sampling from the pretrained model is presented as avoiding a single proxy dataset, but the procedure for choosing the sampling distribution is unspecified. If the distribution itself depends on the model under adaptation, the forgetting penalty estimate risks circularity that would undermine the preservation guarantee.

    Authors: The sampling distribution is generated once from the frozen pretrained model using a fixed collection of generic prompts drawn from public corpora that contain no downstream-task examples. Because the model parameters remain unchanged during sampling, the procedure is independent of the subsequent adaptation. The revision will add an explicit paragraph (new §3.4) describing the prompt sources, temperature, and number of samples, together with a short argument confirming the absence of circularity. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The derivation introduces a first-order preservation condition to define a forgetting penalty on sampled pretraining-proxy activations, pairs it with a task utility on downstream activations, and applies a generalized Rayleigh quotient to rank update directions for gated Adam. This constructs an explicit optimization objective and spectral coordinate system rather than reducing any claimed prediction or result to a fitted parameter or input by definition. No self-citations are invoked as load-bearing uniqueness theorems, no ansatz is smuggled, and no renaming of known results occurs. The proxy sampling procedure is presented as an explicit design choice, not a closed loop. The central claims rest on empirical comparisons, which are externally falsifiable and do not collapse the method to its inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on an unstated first-order preservation condition and on the assumption that sampling from the pretrained model yields a faithful proxy for nontarget capability degradation.

free parameters (1)
  • utility-penalty trade-off implicit in the generalized Rayleigh quotient
    The quotient itself encodes a scaling between task utility and forgetting penalty; its effective value is chosen or normalized during training.
axioms (1)
  • domain assumption A first-order preservation condition exists that can be evaluated on pretraining-proxy activations.
    Invoked to define the forgetting penalty term.

pith-pipeline@v0.9.1-grok · 5720 in / 1182 out tokens · 22656 ms · 2026-06-29T08:36:55.802225+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 16 canonical work pages · 9 internal anchors

  1. [1]

    Kingma and Jimmy Ba , editor =

    Diederik P. Kingma and Jimmy Ba , editor =. Adam:. 3rd International Conference on Learning Representations,. 2015 , url =

    2015

  2. [2]

    ArXiv , year=

    Language Models are Few-Shot Learners , author=. ArXiv , year=

  3. [3]

    ArXiv , year=

    Training language models to follow instructions with human feedback , author=. ArXiv , year=

  4. [4]

    2018 , url=

    Improving Language Understanding by Generative Pre-Training , author=. 2018 , url=

    2018

  5. [5]

    Annual Meeting of the Association for Computational Linguistics , year=

    COMET: Commonsense Transformers for Automatic Knowledge Graph Construction , author=. Annual Meeting of the Association for Computational Linguistics , year=

  6. [6]

    ArXiv , year=

    Fine-Tuning Language Models from Human Preferences , author=. ArXiv , year=

  7. [7]

    ArXiv , year=

    Fine-Tuning is Fine, if Calibrated , author=. ArXiv , year=

  8. [8]

    ArXiv , year=

    Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! , author=. ArXiv , year=

  9. [9]

    Conference on Empirical Methods in Natural Language Processing , year=

    I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses , author=. Conference on Empirical Methods in Natural Language Processing , year=

  10. [10]

    2025 , url=

    Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning , author=. 2025 , url=

    2025

  11. [11]

    North American Chapter of the Association for Computational Linguistics , year=

    MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning , author=. North American Chapter of the Association for Computational Linguistics , year=

  12. [12]

    ArXiv , year=

    CorDA: Context-Oriented Decomposition Adaptation of Large Language Models , author=. ArXiv , year=

  13. [13]

    AAAI Conference on Artificial Intelligence , year=

    Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge , author=. AAAI Conference on Artificial Intelligence , year=

  14. [14]

    AAAI Conference on Artificial Intelligence , year=

    OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning , author=. AAAI Conference on Artificial Intelligence , year=

  15. [15]

    Transactions of the Association for Computational Linguistics , year=

    Natural Questions: A Benchmark for Question Answering Research , author=. Transactions of the Association for Computational Linguistics , year=

  16. [16]

    ArXiv , year=

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. ArXiv , year=

  17. [17]

    Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,

    Jonathan Berant and Andrew Chou and Roy Frostig and Percy Liang , title =. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,. 2013 , url =. doi:10.18653/V1/D13-1160 , timestamp =

    work page doi:10.18653/v1/d13-1160 2013

  18. [18]

    MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

    Metamath: Bootstrap your own mathematical questions for large language models , author=. arXiv preprint arXiv:2309.12284 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Mark Chen and Heewoo Jun and Lukasz Kaiser and Matthias Plappert and Jerry Tworek and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman , title =. CoRR , volume =. 2021 , url =. 2110.14168 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  20. [20]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    ArXiv , year=

    Evaluating Large Language Models Trained on Code , author=. ArXiv , year=

  22. [22]

    Program Synthesis with Large Language Models

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Opencodeinterpreter: Integrating code generation with execution and refinement , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  24. [24]

    International Conference on Learning Representations , year=

    WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions , author=. International Conference on Learning Representations , year=

  25. [25]

    Instruction-Following Evaluation for Large Language Models

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  27. [27]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Qwen3 Technical Report

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Journal of Machine Learning Research , volume=

    Scaling instruction-finetuned language models , author=. Journal of Machine Learning Research , volume=

  30. [30]

    Manning and Stefano Ermon and Chelsea Finn , editor =

    Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn , editor =. Direct Preference Optimization: Your Language Model is Secretly a Reward Model , booktitle =. 2023 , url =

    2023

  31. [31]

    Advances in Neural Information Processing Systems , volume=

    Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning , author=. Advances in Neural Information Processing Systems , volume=

  32. [32]

    What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? , booktitle =

    Thomas Wang and Adam Roberts and Daniel Hesslow and Teven Le Scao and Hyung Won Chung and Iz Beltagy and Julien Launay and Colin Raffel , editor =. What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? , booktitle =. 2022 , url =

    2022

  33. [33]

    Proceedings of the 2016 conference on empirical methods in natural language processing , pages=

    Squad: 100,000+ questions for machine comprehension of text , author=. Proceedings of the 2016 conference on empirical methods in natural language processing , pages=

    2016

  34. [34]

    9th International Conference on Learning Representations,

    Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  35. [35]

    Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Truthfulqa: Measuring how models mimic human falsehoods , author=. Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  36. [36]

    Advances in Neural Information Processing Systems , volume=

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack , author=. Advances in Neural Information Processing Systems , volume=

  37. [37]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  38. [38]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2311.12022 , eprinttype =. 2311.12022 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2311.12022 2023

  39. [39]

    Qwen3 Technical Report

    Qwen Team , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.09388 , eprinttype =. 2505.09388 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  40. [40]

    Technometrics , year=

    Principal Component Analysis , author=. Technometrics , year=

  41. [41]

    Forty-first International Conference on Machine Learning , year=

    Dora: Weight-decomposed low-rank adaptation , author=. Forty-first International Conference on Machine Learning , year=

  42. [42]

    Reddi and Sanjiv Kumar , title =

    Chulhee Yun and Srinadh Bhojanapalli and Ankit Singh Rawat and Sashank J. Reddi and Sanjiv Kumar , title =. 8th International Conference on Learning Representations,. 2020 , url =

    2020

  43. [43]

    The Twelfth International Conference on Learning Representations,

    Tokio Kajitsuka and Issei Sato , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  44. [44]

    Approximation Rate of the Transformer Architecture for Sequence Modeling , booktitle =

    Haotian Jiang and Qianxiao Li , editor =. Approximation Rate of the Transformer Architecture for Sequence Modeling , booktitle =. 2024 , url =

    2024

  45. [45]

    Nature Machine Intelligence , year=

    Parameter-efficient fine-tuning of large-scale pre-trained language models , author=. Nature Machine Intelligence , year=

  46. [46]

    Annual Meeting of the Association for Computational Linguistics , year=

    Universal Language Model Fine-tuning for Text Classification , author=. Annual Meeting of the Association for Computational Linguistics , year=

  47. [47]

    CoRR , volume =

    Xiao Liu and Kaixuan Ji and Yicheng Fu and Zhengxiao Du and Zhilin Yang and Jie Tang , title =. CoRR , volume =. 2021 , url =. 2110.07602 , timestamp =

    work page arXiv 2021

  48. [48]

    The Twelfth International Conference on Learning Representations,

    Renrui Zhang and Jiaming Han and Chris Liu and Aojun Zhou and Pan Lu and Yu Qiao and Hongsheng Li and Peng Gao , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  49. [49]

    The Tenth International Conference on Learning Representations,

    Ananya Kumar and Aditi Raghunathan and Robbie Matthew Jones and Tengyu Ma and Percy Liang , title =. The Tenth International Conference on Learning Representations,. 2022 , url =

    2022

  50. [50]

    Dan Biderman and Jacob P. Portes and Jose Javier Gonzalez Ortiz and Mansheej Paul and Philip Greengard and Connor Jennings and Daniel King and Sam Havens and Vitaliy Chiley and Jonathan Frankle and Cody Blakeney and John Patrick Cunningham , title =. Trans. Mach. Learn. Res. , volume =. 2024 , url =

    2024

  51. [51]

    2014 , url=

    Rayleigh Quotient Based Optimization Methods For Eigenvalue Problems , author=. 2014 , url=

    2014

  52. [52]

    Numerische Mathematik , year=

    Eigenvalues of Rayleigh quotient matrices , author=. Numerische Mathematik , year=

  53. [53]

    A Neural Probabilistic Language Model , booktitle =

    Yoshua Bengio and R. A Neural Probabilistic Language Model , booktitle =. 2000 , url =

    2000

  54. [54]

    ArXiv , year=

    Are Transformers universal approximators of sequence-to-sequence functions? , author=. ArXiv , year=

  55. [55]

    CoRR , volume =

    Penghao Yu and Haotian Jiang and Zeyu Bao and Ruoxi Yu and Qianxiao Li , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.06662 , eprinttype =. 2510.06662 , timestamp =

    work page doi:10.48550/arxiv.2510.06662 2025

  56. [56]

    arXiv preprint arXiv:2512.17720 , year=

    Mitigating Forgetting in Low Rank Adaptation , author=. arXiv preprint arXiv:2512.17720 , year=

    work page arXiv

  57. [57]

    Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning , booktitle =

    Wei Huang and Anda Cheng and Yinggui Wang , editor =. Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning , booktitle =. 2025 , url =. doi:10.18653/V1/2025.EMNLP-MAIN.1108 , timestamp =

    work page doi:10.18653/v1/2025.emnlp-main.1108 2025

  58. [58]

    Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting , booktitle =

    Sunny Sanyal and Hayden Prairie and Rudrajit Das and Ali Kavis and Sujay Sanghavi , editor =. Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting , booktitle =. 2025 , url =

    2025

  59. [59]

    Rabinowitz and Joel Veness and Guillaume Desjardins and Andrei A

    James Kirkpatrick and Razvan Pascanu and Neil C. Rabinowitz and Joel Veness and Guillaume Desjardins and Andrei A. Rusu and Kieran Milan and John Quan and Tiago Ramalho and Agnieszka Grabska. Overcoming catastrophic forgetting in neural networks , journal =. 2016 , url =. 1612.00796 , timestamp =

    work page arXiv 2016

  60. [60]

    arXiv preprint arXiv:2406.14026 , year=

    Demystifying language model forgetting with low-rank example associations , author=. arXiv preprint arXiv:2406.14026 , year=

    work page arXiv

  61. [61]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  62. [62]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    Fine-tuned language models are continual learners , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022