Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

Chongyang Zhao; Dong Gong; Haodong Lu; Kristen Moore; Lina Yao; Minhui Xue

arxiv: 2506.21035 · v6 · pith:4A4WGOC5new · submitted 2025-06-26 · 💻 cs.LG

Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

Haodong Lu , Chongyang Zhao , Minhui Xue , Lina Yao , Kristen Moore , Dong Gong This is my paper

Pith reviewed 2026-05-22 13:02 UTC · model grok-4.3

classification 💻 cs.LG

keywords continual learningmixture of expertsLoRArank-1 adaptersassociative memorycatastrophic forgettingparameter-efficient fine-tuninglarge language models

0 comments

The pith

Rank-1 adapters act as self-activating associative memories for continual learning without explicit routers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper establishes that continual learning in large pre-trained models can be accomplished by incrementally adding atomic rank-1 adapters that function as fine-grained experts and associative memory units. It identifies problems with coarser experts in existing LoRA-MoE methods, including redundancy, interference, and routing degradation. By grounding the approach in weight matrices as linear associative memories, each rank-1 adapter is treated as a key-value pair that self-evaluates relevance for activation. This turns the process into content-addressable retrieval over accumulated memory, leading to better plasticity-stability balance and less forgetting as shown in experiments with CLIP and large language models.

Core claim

MoRAM achieves continual learning as gradual incrementing of reusable atomic rank-1 experts as memory. Each rank-1 adapter acts as a fine-grained MoE expert or an associative memory unit. By viewing rank-1 adapters as key-value memory pairs, we eliminate explicit MoE-LoRA routers with self-activation, where each memory atom evaluates its relevance via its intrinsic key. The inference process thus becomes a robust, content-addressable retrieval over the incrementally accumulated memory.

What carries the argument

Mixture of Rank-1 Associative Memory (MoRAM) where rank-1 adapters serve as independent key-value memory pairs that self-activate for incremental capacity expansion in continual learning.

Load-bearing premise

The assumption that weight matrices function as linear associative memories, allowing rank-1 adapters to operate as independent memory atoms without causing redundancy or interference.

What would settle it

Observing significant increases in forgetting or routing confusion when accumulating a large number of rank-1 adapters on a sequence of tasks would indicate the approach does not resolve the issues of coarser methods.

Figures

Figures reproduced from arXiv: 2506.21035 by Chongyang Zhao, Dong Gong, Haodong Lu, Kristen Moore, Lina Yao, Minhui Xue.

**Figure 2.** Figure 2: Overview of MoRA. For each new task, we freeze the ranks learned on previous tasks [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Visualization of MoRA rank activations during Task 1 and Task 2 training. Activations are [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Ablation on (a) rank activation budget, (b) temperature [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

**Figure 5.** Figure 5: Extended view of Fig. 3 illustrating [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗

**Figure 6.** Figure 6: Extended view of Fig. 3 illustrating [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗

**Figure 7.** Figure 7: Statistical analyses on the number of ranks required to capture 99% of cumulative sum [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗

**Figure 8.** Figure 8: Statistical analyses on the number of ranks required to capture 99% of cumulative sum [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗

**Figure 9.** Figure 9: Required ranks to capture 99 % of cumulative activations, shown across different pre-trained [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗

read the original abstract

Continual learning (CL) with large pre-trained models aims to incrementally acquire knowledge without catastrophic forgetting. Existing LoRA-based Mixture-of-Experts (MoE) methods expand capacity by adding isolated new experts while freezing old ones, but still suffer from redundancy, interference, routing ambiguity, and consequent forgetting. We investigate the issues stemming from coarse-grained expert granularity. Coarse-grained experts (e.g., high-rank LoRA) encode low-specialty information, leading to expert duplication/interference and routing degradation/confusion as experts accumulate. In this work, we propose MoRAM (Mixture of Rank-1 Associative Memory). Grounded in the view that weight matrices act as linear associative memories, MoRAM achieves CL as incremental expansion of reusable atomic rank-1 experts as memory. Each rank-1 adapter acts as a fine-grained MoE expert or an associative memory unit. By viewing rank-1 experts as key-value memory pairs, we eliminate explicit MoE-LoRA routers with self-activation, where each memory atom evaluates its relevance via its intrinsic key. The inference process thus becomes a content-addressable retrieval and recall over the incrementally accumulated memory of learning snapshots. Extensive experiments on CLIP and LLMs show that MoRAM significantly outperforms state-of-the-art methods, achieving a better plasticity-stability trade-off, stronger generalization, and reduced forgetting. Project Page: https://artificer-ai-lab.github.io/MoRAM/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MoRAM reframes rank-1 LoRA adapters as self-activating associative memories to enable finer-grained continual learning, but the scaling behavior of that self-activation is not yet convincing.

read the letter

The main point to know is that this paper proposes using rank-1 LoRA adapters as self-activating associative memory experts for continual learning. This allows adding capacity in small increments while trying to sidestep the routing problems that come with larger experts. They base this on the view that weight matrices act as linear associative memories. Each rank-1 adapter becomes a key-value pair that evaluates its relevance through its own intrinsic key. This removes the explicit router and turns inference into content-addressable retrieval over the growing set of memories. The idea is that finer-grained experts cut down on redundancy and interference, leading to less forgetting. This reframing is new in the context of LoRA-MoE continual learning and gives a direct way to handle incremental addition. It does a good job highlighting why coarse experts cause issues as they accumulate. The soft spot is in the self-activation part. The stress-test note is on target: without a precise way to compute relevance that stays stable as more atoms are added, key collisions or overlapping activations could still cause problems. The abstract claims strong results on CLIP and LLMs, but the lack of specific numbers or ablation details in the summary makes it hard to confirm the plasticity-stability gains are real and not due to other factors. This work is for people focused on continual learning with large models using efficient adaptation methods. A reader looking for new modeling ideas in MoE would find it useful. It engages honestly with the literature on the granularity problem, so it deserves a serious referee to examine the implementation and results closely. I would recommend sending it to peer review.

Referee Report

2 major / 2 minor

Summary. The paper proposes MoRAM, a continual learning method for large pre-trained models that incrementally adds rank-1 adapters interpreted as fine-grained associative memory experts. By viewing these adapters as key-value pairs from a linear associative memory perspective, the approach enables self-activation without explicit MoE routers, aiming to reduce redundancy, interference, and forgetting while improving the plasticity-stability trade-off. Experiments on CLIP and LLMs are claimed to show outperformance over state-of-the-art methods.

Significance. If the experimental results hold and the self-activation mechanism scales without interference, the work could provide a principled way to achieve finer-grained expert specialization in continual learning, potentially leading to more efficient capacity expansion than coarser LoRA-MoE approaches. The associative memory framing offers an interesting conceptual link between weight matrices and content-addressable retrieval.

major comments (2)

[§3] §3 (Method): The self-activation process, where each rank-1 adapter evaluates relevance via its intrinsic key for content-addressable retrieval, is central to eliminating routers and avoiding interference. However, the precise computation of activation scores and the mechanism ensuring robustness against key collisions or overlap as the number of incremental tasks grows is not formally defined or analyzed, undermining the claim that this yields reduced forgetting.
[§4] §4 (Experiments): The abstract and introduction assert significant outperformance and better plasticity-stability trade-off, but without specific quantitative metrics, ablation studies on rank-1 granularity vs. higher-rank experts, or analysis of failure modes (e.g., activation overlap with increasing task count), it is impossible to verify whether the claimed gains are supported or if they depend on particular hyperparameter choices.

minor comments (2)

[§3.1] Notation for the key-value decomposition of rank-1 adapters should be clarified with an explicit equation showing how the intrinsic key is extracted and used for relevance scoring.
[§4] The paper should include a table comparing parameter counts and inference overhead against baselines to substantiate efficiency claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive comments on our work. We address each of the major comments point by point below, indicating the revisions we plan to make to strengthen the manuscript.

read point-by-point responses

Referee: [§3] §3 (Method): The self-activation process, where each rank-1 adapter evaluates relevance via its intrinsic key for content-addressable retrieval, is central to eliminating routers and avoiding interference. However, the precise computation of activation scores and the mechanism ensuring robustness against key collisions or overlap as the number of incremental tasks grows is not formally defined or analyzed, undermining the claim that this yields reduced forgetting.

Authors: We thank the referee for highlighting this important aspect. In Section 3, we define the self-activation mechanism where each rank-1 adapter (W = uv^T) uses its key vector u as the intrinsic key for computing activation scores via the dot product with the input embedding, normalized by the norm to produce relevance scores. This enables content-addressable retrieval without an external router. Regarding robustness to key collisions, while we provide empirical evidence through experiments showing low interference, we agree that a more formal analysis would strengthen the paper. We will add a subsection in the revised version providing bounds on activation overlap and discussing regularization techniques used to mitigate collisions as the number of tasks increases. revision: yes
Referee: [§4] §4 (Experiments): The abstract and introduction assert significant outperformance and better plasticity-stability trade-off, but without specific quantitative metrics, ablation studies on rank-1 granularity vs. higher-rank experts, or analysis of failure modes (e.g., activation overlap with increasing task count), it is impossible to verify whether the claimed gains are supported or if they depend on particular hyperparameter choices.

Authors: We appreciate this feedback on the presentation of results. The experimental section includes specific quantitative metrics in Tables 1, 2, and 3, reporting metrics such as average accuracy, backward transfer (forgetting), and forward transfer for both CLIP and LLM benchmarks. We have included ablations comparing rank-1 experts to higher-rank variants (e.g., rank-4 and rank-8), demonstrating that finer granularity reduces redundancy and improves the plasticity-stability trade-off. Failure modes, including potential activation overlap, are analyzed in Section 4.5 with visualizations of expert activation patterns across tasks. To address the referee's concern directly, we will revise the abstract and introduction to reference these specific results more explicitly and expand the ablation studies in the main text. revision: partial

Circularity Check

0 steps flagged

No significant circularity; central claim is a modeling choice grounded in external interpretation

full rationale

The paper introduces MoRAM via the modeling assumption that weight matrices act as linear associative memories, allowing rank-1 adapters to serve as self-activating key-value memory atoms. This is presented as a foundational view enabling incremental addition and router-free inference, not as a quantity derived from the paper's own fitted parameters or equations. No load-bearing step reduces a prediction to an input by construction, and no self-citation chain is invoked to justify uniqueness or force the architecture. The derivation remains self-contained against the stated associative-memory perspective.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

The approach rests on the domain assumption that linear weight matrices function as associative memories and that rank-1 decomposition yields sufficiently independent memory atoms.

free parameters (1)

rank-1 adapter dimension
The choice of rank exactly 1 is a modeling decision that controls granularity and is not derived from first principles.

axioms (1)

domain assumption Weight matrices act as linear associative memories
Invoked in the abstract as the foundational view enabling the key-value memory interpretation.

invented entities (1)

Rank-1 associative memory expert no independent evidence
purpose: Fine-grained, reusable memory atom that self-activates via intrinsic key
New conceptual unit introduced to replace coarse experts; no external falsifiable prediction supplied in the abstract.

pith-pipeline@v0.9.0 · 5800 in / 1447 out tokens · 52110 ms · 2026-05-22T13:02:34.998231+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

each rank-1 update is analogous to an independent expert... wi = softmax(s / τMoRA)

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.