Pith. sign in

REVIEW 3 major objections 5 minor 41 references

PLATE constructs a low-rank adapter update from frozen pretrained weights alone, so new-task fine-tuning can reduce functional drift on the old distribution without replaying old data, while matching LoRA on new-task gains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:47 UTC pith:HR2RYY26

load-bearing objection PLATE is a genuinely new data-free continual PEFT method with solid controlled experiments, but the geometric justification has an internal inconsistency and the core theorem rests on an unstated assumption. the 3 major comments →

arxiv 2602.03846 v2 pith:HR2RYY26 submitted 2026-02-03 cs.LG cs.AI

PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

Romain Cosentino This is my paper
classification cs.LG cs.AI
keywords continual learningcatastrophic forgettingparameter-efficient fine-tuninglow-rank adaptersdata-free adaptationfunctional driftneuron redundancylarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that catastrophic forgetting in parameter-efficient adaptation can be cut substantially without any access to old-task data. Its argument isolates functional drift—how much an update changes the model's outputs on old inputs—as the central driver of worst-case forgetting, proves that approximate orthogonality alone leaves a nonzero forgetting floor, and then uses the redundancy of pretrained weights as a data-free proxy for old-task feature directions. From this it builds PLATE, a low-rank adapter ΔW = B A Q^T whose two outer factors are frozen and computed once from pretrained weights; only the small core A trains on the new task. If the argument is right, fine-tuning a foundation model no longer needs replay or the pretraining distribution to protect prior capabilities: the protected subspace is read off frozen weights, and the r and τ knobs explicitly trade retention against plasticity.

Core claim

On the paper's own terms, the central discovery is that protecting pretraining behavior reduces to building a low-rank update family with small first-order functional drift ε(S) on the old distribution, and that this drift can be controlled using only frozen weights. Each adapted layer writes ΔW = B A Q^T: B freezes onto a set of redundant output neurons selected by row-colinearity, Q spans the low-energy subspace of the Gram matrix of the complementary frozen rows, and only A learns on the new task. The theory chain—exact zero-forgetting under per-layer orthogonality, a worst-case forgetting floor as soon as drift is nonzero, and a bound on forgetting by β ε(S)^2—turns the structural choice

What carries the argument

The central object is the per-layer structured adapter update family S_PLATE = {B A Q^T}: B is a frozen selection matrix picking the r output neurons whose rows are most redundant (highest mean absolute cosine similarity to anchor rows, estimated in a random projection); Q is a frozen orthonormal basis for a low-energy subspace of W_frozen^T W_frozen (the bottom eigenspace, computed with a structured randomized Hadamard transform for large d_in); and A is the only trainable matrix. This family is designed to keep the first-order drift radius ε(S) small by restricting updates to degrees of freedom that barely excite the frozen neurons, and Theorem 2 converts small ε(S) into an upper bound on

Load-bearing premise

The load-bearing premise, stated in Section 2.3, is that the most colinear rows of a frozen pretrained layer are a faithful weight-only proxy for dominant old-distribution feature directions; the supporting result comes from a deep linear unconstrained-features model and is not demonstrated for nonlinear pretrained LLMs, so if that correspondence fails, PLATE's frozen B and Q have no proven reason to reduce drift on P0.

What would settle it

Measure the worst-case first-order output drift on old-task inputs, sup_{unit-norm updates in S} E_{x on P0} ||J(theta0,x) Delta||^2, for PLATE's subspace and for a random subspace of the same dimension; if PLATE's drift is not substantially smaller, the claimed drift-reduction mechanism is not what explains the observed retention gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Direct corollary: continual specialization of a pretrained model can proceed without replay buffers or access to the pretraining distribution; the protected subspaces are read once from frozen weights.
  • The hyperparameters r and τ become an explicit retention–plasticity dial, with r as the primary control and τ as a secondary lever that recovers retention with smaller learning cost.
  • PLATE uses rk trainable parameters per layer versus LoRA's r(d_in+d_out), so rank can be enlarged without the parameter and optimizer-state cost growing with hidden width; the complexity study reports lower peak memory at a modest per-epoch time overhead.
  • Theorem 1 implies any approximately orthogonal update family—including PLATE—has a nonzero worst-case forgetting floor, so forgetting can be reduced but not erased without exact invariance.
  • In the reported experiments, PLATE's retention advantage appears at matched new-task performance: on MNIST 0-4→5-9, about 1.85% forgetting versus LoRA's 7–9% at comparable trainable parameters, and on Qwen2.5-7B the math gains are kept while avoiding most of LoRA's 16-point instruction-following drop.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit: ε(S) could be measured directly on old-task holdout data for PLATE and for LoRA subspaces, which would test whether the weight-only construction lowers drift itself or merely rides on a smaller trainable parameter count.
  • Because B and Q are frozen and computed once per model, PLATE should stack naturally with replay- or regularization-based continual learning even though the paper does not test that combination.
  • The paper notes in Section 4.1 that PLATE is more involved to configure than LoRA; an untested follow-up is an automatic rule for choosing r and τ from a small validation proxy, since the current method sweeps them.
  • If the redundancy premise holds broadly, PLATE-style weight-only protected subspaces could serve as an initialization for memory-protected fine-tuning in settings where old-task data cannot be shared, such as federated adaptation; this is speculative and not in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PLATE, a data-free parameter-efficient adapter for continual learning in pretrained models. The method computes, once per layer and without access to the old-task distribution, a frozen output selector B that picks redundant output neurons and a frozen low-energy input basis Q derived from the remaining rows of the pretrained weight matrix; only a small core A is trained, giving the structured update ΔW = B A Q^T. The paper develops a theoretical framework in which approximate orthogonality to old-task features yields a worst-case forgetting floor, while restricted curvature is controlled by first-order functional drift, leading to a design principle of keeping drift on P0 small. Experimental results on LLM specialization (Qwen2.5-7B and OLMo-2-7B) and on controlled in-distribution benchmarks (MNIST, AG News→IMDB, WikiText-2→Middle English, synthetic regression) show that PLATE matches LoRA on new-task performance while reducing forgetting at similar or lower trainable-parameter budgets.

Significance. If the mechanism is sound, PLATE is a practically relevant contribution: it offers a fully weight-only, data-free way to build protected update subspaces, explicit knobs (r and τ) to navigate the plasticity–retention trade-off, and a parameter-efficient design whose trainable count is rk rather than r(din+dout). The manuscript ships code and the controlled experiments are a genuine strength: they suggest that the structured update family does reduce forgetting at comparable capacity. However, several load-bearing theoretical and conceptual steps are not yet established: the construction in Algorithm 1 is inconsistent with the stated geometric rationale, Proposition 4 contains an unstated assumption that is dropped in the proof, and the central proxy—that weight colinearity faithfully identifies old-task feature directions—is never directly validated. The empirical retention advantage could therefore stem from capacity effects or from the input projection rather than from the claimed redundancy mechanism. These issues are fixable, but they currently prevent the paper's theoretical claims from supporting its empirical narrative.

major comments (3)
  1. [§2.3, §4.3, Algorithm 1] There is an internal inconsistency between the claimed geometric rationale and the actual construction. Section 2.3 states that highly colinear (redundant) neurons are the proxy for dominant pretraining-era feature directions and that the protected subspace should be the orthogonal complement of their span. Algorithm 1 and Section 4.3, however, build Q from W_frozen, the submatrix of rows not selected by B, i.e., Q spans the bottom eigenspace of W_frozen^T W_frozen, which is approximately the nullspace of the non-selected rows. For Q^T h to be small on supp(P0), old input features must lie in the row space of W_frozen; no argument or experiment establishes this. If the intent is that selected redundant rows are linearly dependent on the remaining rows so that removing them does not change the row span, this must be stated and verified; otherwise the method is not implementing the protect
  2. [Appendix A.5, Proposition 4, Theorem 2] The proof of Proposition 4 is incomplete as written. Equation (9) decomposes the per-sample Hessian into a Gauss–Newton-like term and a residual term involving ∂ℓ/∂f_i times ∇^2_θ f_θ,i. The proof states "we assume this residual contribution is uniformly bounded... which we will assume to be 0 for clarity," but no such assumption appears in the statement of Proposition 4. Consequently, the claimed inequality λ(S) ≤ β ε(S)^2 is not established, and Theorem 2, which relies on Proposition 4, does not follow. Additionally, Theorem 1, Proposition 3, and Theorem 2 all use the informal approximation g0 ≈ 0 without stating a formal hypothesis; if the gradient at θ0 is not exactly zero, the linear term in the Taylor expansion can dominate the quadratic bound for small ρ, invalidating the stated orders. These assumptions must be stated explicitly and included in the theorem statements or the claim
  3. [§5.2–§5.3, Proposition 2] The load-bearing premise that neuron redundancy provides a faithful weight-only proxy for dominant pretraining-era feature directions, and that restricting updates to the complement of such directions reduces ε(S) on P0, is not directly validated. Proposition 2 is borrowed from a deep linear unconstrained-features model (reference [7]) and is not demonstrated for the nonlinear LLMs used in the experiments. No experiment in the paper measures ε(S) on P0; Figure 2 measures restricted curvature using H0, which requires old data and does not decompose whether the improvement comes from smaller drift or from reduced parameter count. I recommend adding an explicit measurement of ε(S) under P0 for PLATE vs. LoRA at matched trainable budgets, and an ablation where B selects random neurons instead of redundancy-ranked neurons (with Q fixed), to isolate the geometric mechanism from capacity effect
minor comments (5)
  1. [§4.1] Typo: "hyperparameters that have provide an explicit trade-off" should be "that provide an explicit trade-off".
  2. [§5.2.1] Awkward phrasing: "we fine-tune on it the AM-DeepSeek-R1 distilled reasoning corpus" should be reworded, e.g., "we fine-tune on the AM-DeepSeek-R1 distilled reasoning corpus".
  3. [§4.2] The phrase "a set of anchor rows" appears as "anchorrows" in one place; fix spacing.
  4. [§5.2.2/Figure 5 caption] The heading and caption use "T ulu-3" with an odd space; should be "Tulu-3".
  5. [Algorithm 1] The algorithm does not state how k is capped by kmax in the pseudocode, though the text mentions a cap. Adding this detail would improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the drift-forgetting bounds are formal conditional inequalities; PLATE's redundancy-to-protection link is an unverified premise, and the self-citations are motivational rather than load-bearing.

full rationale

The derivation chain runs: exact per-layer orthogonality gives zero forgetting (Prop. 1); approximate orthogonality gives a worst-case floor in ε(S) (Thm. 1); restricted curvature λ(S) upper-bounds forgetting (Prop. 3); and λ(S) is bounded by βε(S)^2 under a Hessian-regularity assumption (Prop. 4), yielding Thm. 2. These are Taylor-expansion and quadratic-form manipulations with explicit assumptions (Assumption 1, ∇²_f ℓ ⪯ βI). They do not assume the conclusion and are not equivalent to their inputs by construction. The real load-bearing step is the claim that colinear pretrained rows are a weight-only proxy for dominant P0 feature directions, so that the PLATE subspaces make ε(S) small. That step is asserted with citations to [7] and [38] and is not demonstrated for nonlinear LLMs; it is an empirical/assumption gap, not a circular reduction. The internal inconsistency noted between Section 2.3 (orthogonal complement of the colinear span) and Section 4.3 (Q built from the complement of the selected rows) is a correctness risk, not a circularity. In addition, the proof of Proposition 4 in Appendix A.5 drops the residual Hessian term 'assume to be 0 for clarity' without making it a formal hypothesis, so Theorem 2 is not fully established; again, this is an omitted-proof/unsupported-step concern, not a case of a prediction reducing to a fit. The empirical results are measured on independent benchmarks (IFEval, MATH-500, WikiText-2, MNIST, AG News→IMDB), so PLATE's retention advantage is not a fitted parameter renamed as a prediction. The only author self-citation is [2], used for geometric intuition about redundant neurons; it is motivational and not load-bearing for the formal inequalities or the benchmark outcomes. Overall, the central theoretical statements are self-contained conditional results, and the method's claimed benefit is evaluated externally; there is no significant circularity, though there are genuine unsupported premises that affect correctness, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

PLATE introduces no new physical or structural entity; it selects existing neurons and computes matrix bases. The central claim rests on several assumptions: the curvature link, a discarded Hessian residual term, the transfer of deep neural collapse to nonlinear LLMs, and a heuristic redundancy geometry. These are the main reasons the theoretical guarantees are conditional.

free parameters (5)
  • r (number of selected redundant output neurons) = varies by experiment: 32-350 in benchmarks; 256 for Qwen; 32-1024 in OLMo sweep
    Controls the plasticity budget; selected by hyperparameter sweep, not derived from theory.
  • tau (input energy threshold) = varies: 0.6-0.98 across experiments
    Controls dimension k of Q; selected by sweep and acts as the retention knob.
  • adapter scale rho = 0.5 (fixed)
    Set to 0.5 to mirror LoRA's alpha/r scaling; an arbitrary calibration choice.
  • anchor indices and projection dimension d' = not specified
    The redundancy score s_i in Eq. 3 depends on an anchor set and random projection; no principled selection is given.
  • kmax cap for basis dimension = 256/512 (Table 2)
    Ad hoc upper bound on the width of Q.
axioms (5)
  • ad hoc to paper Curvature link Assumption 1: DeltaTheta^T H0 DeltaTheta >= mu0 E[||J_theta0(x) DeltaTheta||^2] for all DeltaTheta in S (Eq. 4).
    Introduced in Appendix A.2 specifically to make Theorem 1's lower bound go through; not derived or validated.
  • ad hoc to paper Residual curvature term in the Hessian decomposition is zero (second term of Eq. 9).
    Assumed '0 for clarity' in the proof of Proposition 4 but absent from the proposition statement; Theorem 2 inherits it.
  • domain assumption Deep neural collapse: layer weights lie in the span of K class prototypes [7].
    Borrowed from a deep linear unconstrained-features model and used to justify weight redundancy as a data-direction proxy in Proposition 2.
  • domain assumption g0 ≈ 0 at the old-task optimum.
    Used in the proofs of Theorem 1 and Proposition 3; reasonable for well-trained models but limits the scope of the bounds.
  • domain assumption Redundant neurons have colinear hyperplanes and modifying them does not induce new input-space partitions [38, 2].
    Output-side safety in Section 3.3 relies on this geometric claim; it is not proven in this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 18853 in / 11246 out tokens · 112096 ms · 2026-08-03T04:47:45.180705+00:00 · methodology

0 comments
read the original abstract

We develop a continual learning method for pretrained models that \emph{requires no access to old-task data}, addressing a practical barrier in foundation model adaptation where pretraining distributions are often unavailable. Our key observation is that pretrained networks exhibit substantial \emph{geometric redundancy}, and that this redundancy can be exploited in two complementary ways. First, redundant neurons provide a proxy for dominant pretraining-era feature directions, enabling the construction of approximately protected update subspaces directly from pretrained weights. Second, redundancy offers a natural bias for \emph{where} to place plasticity: by restricting updates to a subset of redundant neurons and constraining the remaining degrees of freedom, we obtain update families with reduced functional drift on the old-data distribution and improved worst-case retention guarantees. These insights lead to \textsc{PLATE} (\textbf{Pla}sticity-\textbf{T}unable \textbf{E}fficient Adapters), a continual learning method requiring no past-task data that provides explicit control over the plasticity-retention trade-off. PLATE parameterizes each layer with a structured low-rank update $\Delta W = B A Q^\top$, where $B$ and $Q$ are computed once from pretrained weights and kept frozen, and only $A$ is trained on the new task. The code is available at https://github.com/SalesforceAIResearch/PLATE.

Figures

Figures reproduced from arXiv: 2602.03846 by Romain Cosentino.

Figure 1
Figure 1. Figure 1: Local-geometry view of forgetting on a continual learning 2-dimensional binary classification problem: Blue points denote the old-task dataset P0 and yellow points the new-task dataset P1; decision boundaries are shown when trained on P0 (blue curve) and after training on P1 (yellow curve). The background heatmap visualizes how the training on P1 change the model’s local input-output linearization, ∆(x) :=… view at source ↗
Figure 2
Figure 2. Figure 2: Restricted-curvature forgetting: We train an MLP on MNIST digits 0-4 to obtain parameters θ0. For each method, we perturb the trained model by θ0 + ρv and measure the resulting forgetting F0(θ0, θ0 + ρv) = L0(θ0 + ρv) − L0(θ0) where v is the unit vector in each subspace that maximizes v T H0v, i.e., highest-curvature direction. PLATE exhibits the smallest slope, indicating substantially reduced restricted … view at source ↗
Figure 3
Figure 3. Figure 3: PLATE exhibits a controllable forgetting-plasticity spectrum via (r, τ ): We sweep PLATE’s hyperparameters on a two-moons continual-learning toy: the number of adapted (redundant) output neurons r (rows) and the input energy threshold τ (columns), where larger τ enforces a stricter input-side constraint (smaller k). Each panel overlays Dataset 1 (blue) and Dataset 2 (yellow), and visualizes how adaptation … view at source ↗
Figure 4
Figure 4. Figure 4: Qwen2.5-7B on DeepSeek-R1 reasoning: (Left) Learning capabilities on maths/reasoning dataset. (Right) Forgetting on instruction following dataset. PLATE (green) matches LoRA (blue) on math/reasoning benchmarks while preserving instruction-following (IFE￾val), whereas LoRA exhibits substantial OOD forgetting relative to the base model. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: OLMo-2-7B on Tulu-3: (Left) IFEval accuracy vs. percentage of trainable parameters. The red dashed line is the base model capabilities on IFEval. (Right) MATH forgetting (drop from base) versus trainable parameters. PLATE (green) improves IFEval roughly linearly with parameter budget while keeping forgetting almost flat, whereas LoRA (blue) quickly saturates on IFEval and accumulates much larger MATH forge… view at source ↗
Figure 6
Figure 6. Figure 6: Qwen 2.5-3B on Middle English (EN-ME): Perplexity over training steps on WikiText-2 (top: retention/forgetting) and EN-ME (bottom: learning) Columns fix the PLATE output rank r ∈ {32, 64, 128, 256} and sweep τ ∈ {0.70, 0.80, 0.90, 0.98} (green, solid) against LoRA baselines with varying ranks (blue, dashed). Top row reports WikiText-2 perplexity (forgetting) and bottom row reports Middle English perplexity… view at source ↗
Figure 7
Figure 7. Figure 7: Synthetic regression with tunable task dissimilarity: (Left) forgetting on Task 1 (increase in MSE) versus task dissimilarity D2 (α); (Right) Task 2 test loss versus D2 (α). Forgetting for full FT (red) and LoRA (blue) grows roughly linearly with D2 (α), while PLATE (green) remains an order of magnitude smaller even for dissimilar tasks, with only a modest increase in Task 2 loss. To explicitly vary task r… view at source ↗
Figure 8
Figure 8. Figure 8: In distribution Vision and text benchmarks: (Top): MNIST 0-4→5-9, showing task 2 performance and task 1 forgetting as a function of trainable parameters. (Bottom): AG News→IMDB, with all methods achieving near-perfect task 2 accuracy while differing in how much they forget task 1. For this experiment, task 1 is a classification problem on MNIST digits {0, . . . , 4}, and task 2 is the classification on MNI… view at source ↗
Figure 9
Figure 9. Figure 9: Training efficiency of PLATE vs. LoRA on DistilBERT: We adapt all linear layers and measure forward/backward time per epoch and peak GPU memory as a function of the total number of trainable adapter parameters (log scale). (i) For a fixed output rank r, PLATE trains only A ∈ R r×k and therefore uses rk trainable parameters per layer, whereas LoRA trains two matrices and uses r(din + dout); across the sweep… view at source ↗
Figure 10
Figure 10. Figure 10: PLATE initialization computational details [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 18 linked inside Pith

  1. [1]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. InProceedings of the European conference on computer vision (ECCV), pages 139–154, 2018

    2018

  2. [2]

    The geometry of deep networks: Power diagram subdivision.Advances in Neural Information Processing Systems, 32, 2019

    Randall Balestriero, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision.Advances in Neural Information Processing Systems, 32, 2019

    2019

  3. [3]

    Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420, 2018

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420, 2018

    Pith/arXiv arXiv 2018

  4. [4]

    Orthogonal gradient descent for continual learning

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. InInternational conference on artificial intelligence and statistics, pages 3762–3773. PMLR, 2020

    2020

  5. [5]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    Pith/arXiv arXiv 2018

  6. [6]

    Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4):128–135, 1999

    Robert M French. Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4):128–135, 1999

    1999

  7. [7]

    The persistence of neural collapse despite low-rank bias

    Connall Garrod and Jonathan P Keating. The persistence of neural collapse despite low-rank bias. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  8. [8]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks.arXiv preprint arXiv:1312.6211, 2013

    Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks.arXiv preprint arXiv:1312.6211, 2013

    Pith/arXiv arXiv 2013

  9. [9]

    Olmes: A standard for language model evaluations, 2025

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations, 2025. URL https://arxiv.org/abs/ 2406.08446

    Pith/arXiv arXiv 2025

  10. [10]

    Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

    Pith/arXiv arXiv 2015

  11. [11]

    Learning both weights and connections for efficient neural network.Advances in neural information processing systems, 28, 2015

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network.Advances in neural information processing systems, 28, 2015

    2015

  12. [12]

    Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366, 2021

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

    Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1–124, 2021. 21

    2021

  14. [14]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  15. [15]

    Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    2022

  16. [16]

    Scaling laws for forgetting when fine-tuning large language models

    Damjan Kalajdzievski. Scaling laws for forgetting when fine-tuning large language models. arXiv preprint arXiv:2401.05605, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    2017

  18. [18]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Theory on mixture-of- experts in continual learning.arXiv preprint arXiv:2406.16437, 2024

    Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B Shroff. Theory on mixture-of- experts in continual learning.arXiv preprint arXiv:2406.16437, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    Pith/arXiv arXiv 2021

  21. [21]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  22. [22]

    Catastrophic interference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. InPsychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989

    1989

  23. [23]

    Understanding the role of training regimes in continual learning.Advances in Neural Information Processing Systems, 33:7308–7320, 2020

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning.Advances in Neural Information Processing Systems, 33:7308–7320, 2020

    2020

  24. [24]

    Insights on representational similarity in neural networks with canonical correlation.Advances in neural information processing systems, 31, 2018

    Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation.Advances in neural information processing systems, 31, 2018

    2018

  25. [25]

    Olmo 3.arXiv preprint arXiv:2512.13961, 2025

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3.arXiv preprint arXiv:2512.13961, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Ji...

    Pith/arXiv arXiv 2025

  27. [27]

    Continual lifelong learning with neural networks: A review.Neural networks, 113:54–71, 2019

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review.Neural networks, 113:54–71, 2019

    2019

  28. [28]

    Learn more, but bother less: parameter efficient continual learning.Advances in Neural Information Processing Systems, 37:97476–97498, 2024

    Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning.Advances in Neural Information Processing Systems, 37:97476–97498, 2024

    2024

  29. [29]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  30. [30]

    Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.Psychological review, 97(2):285, 1990

    Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.Psychological review, 97(2):285, 1990

    1990

  31. [31]

    Progressive neural networks.arXiv preprint arXiv:1606.04671, 2016

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks.arXiv preprint arXiv:1606.04671, 2016

    Pith/arXiv arXiv 2016

  32. [32]

    Gradient projection memory for continual learning

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762, 2021

    Pith/arXiv arXiv 2021

  33. [33]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL https://arxiv.org/abs/1910.01108

    Pith/arXiv arXiv 2020

  34. [34]

    Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958, 2014

    1929

  35. [35]

    Orthogonal subspace learning for language model continual learning

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10658–10671, 2023

    2023

  36. [36]

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning.Advances in Neural Information Processing Systems, 37:71768–71791, 2024

    2024

  37. [37]

    Lifelong learning with dynamically expandable networks.arXiv preprint arXiv:1708.01547, 2017

    Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks.arXiv preprint arXiv:1708.01547, 2017

    Pith/arXiv arXiv 2017

  38. [38]

    Max-affine spline insights into deep network pruning.arXiv preprint arXiv:2101.02338, 2021

    Haoran You, Randall Balestriero, Zhihan Lu, Yutong Kou, Huihong Shi, Shunyao Zhang, Shang Wu, Yingyan Celine Lin, and Richard Baraniuk. Max-affine spline insights into deep network pruning.arXiv preprint arXiv:2101.02338, 2021. 23

    Pith/arXiv arXiv 2021

  39. [39]

    Continual learning of context-dependent processing in neural networks.Nature Machine Intelligence, 1(8):364–372, 2019

    Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks.Nature Machine Intelligence, 1(8):364–372, 2019

    2019

  40. [40]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. InInternational conference on machine learning, pages 3987–3995. PMLR, 2017

    2017

  41. [41]

    Safe: Slow and fast parameter-efficient tuning for continual learning with pre-trained models.Advances in Neural Information Processing Systems, 37:113772–113796, 2024

    Linglan Zhao, Xuerui Zhang, Ke Yan, Shouhong Ding, and Weiran Huang. Safe: Slow and fast parameter-efficient tuning for continual learning with pre-trained models.Advances in Neural Information Processing Systems, 37:113772–113796, 2024. 24 A Proofs A.1 Proof of Proposition 1 Proof.Recall that for each layerℓwe have z(ℓ) θ (x) =W (ℓ)h(ℓ−1) θ (x), h (ℓ) θ ...

    2024