Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Learning Mamba as a Continual Learner: Meta-learning Selective State Space Models for Efficient Continual Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mamba, a selective state space model with a fixed-size hidden state, can be meta-learned as a continual learner that matches or beats Transformers at lower memory and compute.

desk verdict A credible empirical case that Mamba works for MCL, held back by an under-specified regularizer and missing code. read the letter →

arxiv 2412.00776 v4 pith:UY4KWLYE submitted 2024-12-01 cs.LG

Chongyang Zhao , Dong Gong This is my paper · ORCID
classification cs.LG
keywords meta-continuallearningstatespacemodelsMambasequencepredictionlinearattentioncontinualselectivityregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continual learning can be framed as sequence prediction: a meta-learned model reads a stream of (x, y) pairs and predicts the label for a query. Transformer-based meta-continual learners do this well but must keep a growing cache of all past representations, which conflicts with continual learning's goal of not storing everything. This paper asks whether Mamba, an attention-free state space model with a fixed-size hidden state, can play the same role. It proposes MambaCL, which meta-learns Mamba across continual-learning episodes, plus a selectivity regularizer that steers training by linking Mamba's internal parameters to the attention patterns of Transformers. The paper reports that MambaCL matches or exceeds Transformer accuracy on most benchmark and generalization settings while using roughly half the parameters and running more than twice as fast.

What carries the argument

The central object is the associative indicator q^Mamba_{2t+1} = [C_{2t+1} B_j^T]_{j=1}^{2t}, built from Mamba's input-dependent output and input projection parameters C and B. It is used exactly like Transformer attention weights: a ground-truth indicator vector marks which past samples share the query's class, and a KL divergence steers Mamba's selectivity pattern toward it. This works because, under the duality identified between Mamba and linear attention, C_t plays the role of the query and B_j the role of a key, so C_t B_j^T encodes how much past token j should be retrieved when predicting at step t. The A_t recurrence that also shapes the hidden state is set aside in this proxy, and the fixed-size hidden state H_t itself is what gives Mamba its constant memory.

What would settle it

Compare the learned association maps of a model trained with the paper's regularizer against a version trained with a full-path regularizer that accumulates each past token's influence through the actual recurrence by summing over all paths the product of intervening A matrices times C_t B_j^T (computed by auto-differentiation or by ablating individual B_j). If the two maps disagree sharply, or if the proxy-regularized model cannot predict which individual past token most changes the output (test by zeroing each B_j in turn and measuring the output change), the proxy assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that selective state space models can serve as effective meta-continual learners once they are properly formulated and regularized. Concretely, the paper treats each class-incremental episode as a token sequence (x1, y1, ..., xt, yt, xtest) -> ytest, meta-learns a Mamba model over many such episodes, and introduces a selectivity regularization loss that forces the model's input-dependent SSM parameters to associate each query with the samples of the same class. The regularizer works by identifying the cross-products C_t B_j^T of Mamba's selective parameters with the query-key products of linear attention, which in turn approximate softmax attention; the KL divergence between this association pattern and the ground-truth same-class indicator stabilizes meta-training, which otherwise fails to converge for Mamba. Across general, fine-grained, and domain-shifted image classification plus regression tasks, the meta-learned Mamba matches or beats a vanilla Transformer on meta-test accuracy while using a fixed-size hidden state, and degrades far less than Transformers when meta-test episodes are longer, noisier, or from a held-out domain.

Load-bearing premise

The load-bearing assumption is that the cross-products C_t B_j^T of Mamba's input-dependent SSM parameters faithfully represent how much Mamba actually retrieves each past token, even though the recurrent A_t matrices that multiply earlier B_j values are dropped from the regularization; if that proxy is unfaithful, the selectivity regularizer may teach the wrong association pattern and the reported gains may not transfer to other MCL definitions or architectures.

Editorial extensions

If this is right

  • MambaCL matches or exceeds Transformer meta-test accuracy on general, fine-grained, and domain-shift benchmarks while keeping a fixed-size state, removing the memory growth that conflicts with continual learning's goal of not storing all seen samples.
  • It generalizes to meta-test episodes with ten times as many shots as training (50 versus 5) with only about a 10% accuracy drop, while Transformer and Linear Transformer degrade sharply at untrained episode lengths.
  • Under Gaussian input noise up to σ = 10, Mamba's accuracy remains far more stable than Transformer's, and on the DomainNet held-out-domain test it reaches 55.6% on quickdraw versus 50.2% for Transformer.
  • The selectivity regularizer is necessary for Mamba to converge at all: without it the meta-training loss oscillates and stops decreasing, while varying the regularization weight λ across 0.1 to 2.0 leaves results stable.
  • The efficiency claim is concrete: MambaCL uses 5.4M parameters and runs 858 episodes per second versus the Transformer's 9.2M parameters and 325 episodes per second, at comparable or better accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the C_t B_j^T proxy is faithful, the same KL-regularizer recipe should extend to any recurrent or linear-attention model whose state update can be written as an outer-product accumulation; testing it on other recurrent architectures with fixed-size states would show whether Mamba's advantage is specific to selective SSMs or a general property of recurrent compression.
  • The strong length generalization suggests that a meta-learned recurrent learner may be inherently less prone to positional overfitting than attention models, because it must represent content in a capacity-limited state; this predicts that MambaCL should also transfer to streaming settings with unpredictable task boundaries, which the paper does not test.
  • The random-token vocabulary trick, where class identities are arbitrary symbols within each episode, means the learner is effectively trained to handle any label permutation; a natural extension the paper does not explore is using MambaCL as a task-agnostic few-shot classifier on streams where the label set changes by example rather than by task.
  • The paper's stated limitations leave offline CL and larger-scale datasets unexplored; one concrete implication of the results is that Mamba's fixed-state compression could bring replay-free offline CL within reach if the selectivity regularizer transfers to multi-epoch streams.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MambaCL, a meta-continual-learning method that trains a selective state-space model (Mamba) as a sequence-prediction continual learner with a fixed-size hidden state. To stabilize meta-training, the authors introduce a selectivity regularizer that encourages Mamba's C_t B_j^T scores to match ground-truth same-class association patterns, drawing an analogy between Mamba and linear/softmax attention. Extensive experiments on general image classification, fine-grained recognition, domain shift, regression, and length/noise generalization settings compare Mamba with Transformers, Linear Transformers, Performer, and OML, reporting that Mamba matches or exceeds Transformer performance at lower parameter count and higher inference speed. The paper also includes ablations on regularization strength, state size, architecture variants, and learning rate.

Significance. If the empirical claims hold, the paper is a useful demonstration that attention-free SSMs with fixed-size hidden states can be competitive with Transformers in meta-continual learning, and the length/domain/noise generalization studies are valuable for the MCL community. The paper's strengths include extensive tables with means and standard deviations, several ablation families, and the explicit convergence comparison with and without the regularizer in Fig. 6. However, the load-bearing selectivity regularizer is under-specified as a loss, and its Mamba association proxy ignores the A_t recurrence; these issues make the method description and the mechanistic interpretation currently unsupported, so the significance is conditional on resolving them.

major comments (3)
  1. [§3.2.2 and Appendix C.1 (Fig. 6)] The selectivity regularizer is not a well-defined loss as written. The paper defines ℓ_slct((x,y)) = KL(p_idx, q*_idx), where p is an unnormalized indicator vector in {0,1}^{2t} and q is, for Transformers, an unnormalized softmax score vector (the text explicitly says normalization is omitted) and, for Mamba, the raw real-valued inner-product vector [C_{2t+1}B_j^T]. KL divergence is undefined for such inputs, so the reader cannot tell what objective is actually minimized. This is not a cosmetic issue because Fig. 6 shows that Mamba does not converge without ℓ_slct, making the regularizer load-bearing. Please specify the exact computation: the normalization/softmax procedure over which positions, the handling of zero entries in p, which Mamba layer(s) provide C and B, and whether the same convention is used for the Transformer, Linear Transformer, and Performer baselines.
  2. [§3.2.2 and Eq. (3)] The association proxy qMamba = [C_{2t+1}B_j^T] omits the A_t factors in the state recurrence. Unrolling Eq. (3), the contribution of token z_j to the output at step 2t+1 is C_{2t+1}(A_{2t+1}A_{2t}...A_{j+1})B_j z_j, so C_{2t+1}B_j^T equals the true retrieval weight only if all A_l are identities, which contradicts the selectivity/gating role of A in Mamba and the paper's own description in §3.1. Since Appendix C.1 makes the regularizer necessary for convergence, the method may be training Mamba to align a surrogate association pattern that its hidden state does not actually implement; the visualizations in Appendix D inherit this issue. I request a validation of the proxy (for example, comparing C_tB_j^T with the A-inclusive influence or with the actual gradient contribution on a trained model, or ablating against an A-aware weighting). Without such validation, the mechanistic interpretation and the generalization claims tied to Mamba's selectivity are not established.
  3. [§4.3, Table 6, and Tables 1–5] The main results are not clearly tied to a specific Mamba architecture version. In Table 6, the row labeled Mamba-1 reports 59.7±0.5 on Cifar-100 and 90.1±0.3 on ImageNet-1K, while the row labeled Mamba-2 reports 67.1±0.4 and 93.6±0.2; the 'Mamba' entries in Table 2 (67.1±0.4 on Cifar-100 and 93.6±0.2 on ImageNet-1K) match the Mamba-2 row exactly. The body text, however, describes a Mamba block with 1-D convolution and Table 8 lists a convolution kernel size of 4, which reads like Mamba-1, and the version used for the main tables is never stated explicitly. Please state which architecture all reported results correspond to and adjust the method description and configuration table accordingly.
minor comments (4)
  1. [§3.2.2] The indexing in the definition of p_{2t+1} is inconsistent: the token at step 2t+1 is x_{t+1}, so its label is y_{t+1}, not y_{2t+1}; please fix the notation.
  2. [Appendix D] The captions of Figs. 10–15 describe the plots as showing Mamba's 'associations,' but the quantities plotted are the C_tB_j^T surrogate whose validity is questioned in Major Comment 2; the captions should acknowledge this.
  3. [Table 8] The row label 'Delta Convolution 4' is unclear; if this denotes the convolution kernel size or dilation used in the Mamba block, please state the terminology precisely.
  4. [§4.1 and Tables 1–5] The tables indicate that best and second-best results are highlighted in red and blue, but the text-only rendering does not show these colors; please ensure the final formatted version is unambiguous.

Circularity Check

1 steps flagged · score 2.0 of 10

One minor self-referential element: the paper visualizes the exact association score that the selectivity regularizer is trained to align, but the central empirical comparisons against external benchmarks are independent.

  1. other [Sec. 3.2.2 (selectivity regularization) and Appendix D.1 (visualization of attention and selectivity)]
    "Relying on the linear attention as the bridge, we can obtain the associative indicators of Mamba as qMamba_{2t+1}=[C_{2t+1}B^⊤_j]^{2t}_{j=1}. To regularize the models' selection behavior in meta-training, for a query sample (x,y) in a sequence, we apply a selectivity regularization: ℓslct((x,y)) = KL(pidx((x,y)), q∗_{idx((x,y))}). ... Note that Mamba does not have explicit attention weights, we compute the scores relying on the connection between Mamba and Transformers described in 3.2.2 ... Specifically, we computed the parameters C_test and B (C_testB^⊤) within its SSMs."

    The selectivity regularizer directly trains q = C_tB_j^T to match the ground-truth same-class indicator p via a KL loss. The same score C_testB^⊤ is then reported in Appendix D as evidence that Mamba effectively associates with relevant shots and exhibits selective behavior. The visualized association pattern is therefore not an emergent, independently measured property of the meta-learned model; it is the quantity that the training objective explicitly encouraged. This makes the mechanistic interpretation partially self-referential. However, the accuracy results in Tables 1-5 are external benchmark evaluations, and the visualization is not required to compute them, so the central empirical claims remain independent.

full rationale

The paper's main contribution is an empirical study of Mamba as a meta-continual learner, evaluated with held-out accuracy on standard benchmarks such as Cifar-100, ImageNet-1K, fine-grained datasets, DomainNet, and regression tasks. These results are not derived from the paper's own assumptions; they are comparisons against external baselines and held-out test sets. The selectivity regularizer is an auxiliary training loss, and removing it changes convergence as shown in Appendix C.1, but the final performance numbers still measure generalization to unseen episodes rather than the regularizer's target. The only self-referential element that can be pinned to specific text is the use of the regularized association score C_tB_j^T as evidence of Mamba's associative selectivity in the visualization analyses in Appendix D. Since this score is exactly what ℓslct is trained to align with same-class indicators, observing that it aligns is partly by construction; nevertheless, the visualizations are on meta-testing episodes with unseen classes and longer lengths, so they retain some generalization content. I do not find load-bearing self-citation, uniqueness arguments imported from the authors, or renamed known results. The concern that the proxy C_tB_j^T omits the recurrent A_t factors is an assumption-validity issue about the regularizer's fidelity, not circularity. Overall, a minor self-referential analysis accompanies an otherwise independent empirical derivation, so the circularity score is 2.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical results depend on a handful of hyperparameters selected by ablation on the evaluation datasets, and on two modeling assumptions: the linear-attention duality from the literature and the simplified C_t B_j^T association proxy. No new physical or architectural entities are introduced.

free parameters (4)
  • selectivity regularization weight λ = 0.5
    Chosen by ablation over {0.1, 0.2, 0.5, 1.0, 2.0} (Fig. 5a, Fig. 7); the paper reports performance is stable across the range but fixes λ=0.5 for all experiments.
  • SSM state size = 128
    Chosen by ablation over {16, 32, 64, 128, 256} (Fig. 5b) as a balance of accuracy and efficiency.
  • initial learning rate = 1e-4
    Chosen from ablation over {5e-5, 1e-4, 2e-4, 5e-4} (Fig. 8), with decay 0.5 every 10,000 steps.
  • target vocabulary size = 200
    Set for generalization experiments; the prediction head outputs over this fixed universal vocabulary, and the size constrains how many classes an episode can contain.
assumptions (3)
  • domain assumption The linear-attention duality between selective SSMs and linear attention (Dao and Gu, 2024) is valid for Mamba's trained parameters in MCL.
    Invoked in Section 3.2.2 to define q_Mamba = [C_t B_j^T]. The paper relies on this external result without re-deriving it for the MCL setting.
  • ad hoc to paper C_t B_j^T, without the A_t decay factors, is a sufficient association proxy for selective SSM behavior.
    Introduced in Section 3.2.2; the paper simplifies the SSM unrolling by dropping the recurrent A_t terms and gives no empirical or theoretical check of the proxy's fidelity.
  • domain assumption Pre-trained frozen CLIP-ViT-B16 features are an adequate representation for evaluating continual learning behavior.
    Used by default for most datasets (Appendix B); the paper notes results may differ when training from scratch, which it only tests on four datasets in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Mamba as a Continual Learner: Meta-learning Selective State Space Models for Efficient Continual Learning." pith.science (2026). https://pith.science/paper/UY4KWLYE

@misc{pith2026241200776,
  author       = {Pith},
  title        = {Pith review of: Learning Mamba as a Continual Learner: Meta-learning Selective State Space Models for Efficient Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UY4KWLYE}},
  note         = {Machine review of arXiv:2412.00776}
}
read the original abstract

Continual learning (CL) aims to efficiently learn from a non-stationary data stream, without storing or recomputing all seen samples. CL enables prediction on new tasks by incorporating sequential training samples. Building on this connection between CL and sequential modeling, meta-continual learning (MCL) aims to meta-learn an efficient continual learner as a sequence prediction model, with advanced sequence models like Transformers being natural choices. However, despite decent performance, Transformers rely on a linearly growing cache to store all past representations, conflicting with CL's objective of not storing all seen samples and limiting efficiency. In this paper, we focus on meta-learning sequence-prediction-based continual learners without retaining all past representations. While attention-free models with fixed-size hidden states (e.g., Linear Transformers) align with CL's essential goal and efficiency needs, they have shown limited effectiveness in MCL in previous literature. Given Mamba's strong sequence modeling performance and attention-free nature, we explore a key question: Can attention-free models like Mamba perform well on MCL? By formulating Mamba and the SSM for MCL tasks, we propose MambaCL, a meta-learned continual learner. To enhance MambaCL's training, we introduce selectivity regularization, leveraging the connection between Mamba and Transformers to guide its behavior over sequences. Furthermore, we study how Mamba and other models perform across various MCL scenarios through extensive and well-designed experiments. Our results highlight the promising performance and strong generalization of Mamba and attention-free models in MCL, demonstrating its potential for efficient continual learning and adaptation.

Figures

Figures reproduced from arXiv: 2412.00776 by the authors.

Figure 1
Figure 1. The overall framework of our proposed methods. We meta-train a Mamba Learner [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The Mamba block in MambaCL. On the data stream, the meta-learned Mamba fθ() recognizes the asso￾ciation relationship between x and y through the sequence, recurrently updating the hidden state Ht for prediction, as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Final-layer associations in a 20-task 5-shot meta-testing episode of MambaCL (meta-trained [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Generalization analysis on ImageNet-1K, with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Training loss curves for (a, e) Mamba, (b, f) Transformer, (c, g) Linear Transformer, and [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Ablation studies on regularization strength [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Ablation studies on learning rates ({5×10−5 , 1×10−4 , 2×10−4 , 5×10−4}) during meta￾testing of 20-task, 5-shot models (meta-trained on 20-task, 5-shot) for (a) Mamba, (b) Transformer, (c) Linear Transformer, and (d) Performer. that regularization strengths of 0.1 (9) …
Figure 9
Figure 9. Figure 9: Generalization Analysis on ImageNet-1K with regularization strength [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: 20-task 5-shot in meta-testing: visualization of the final layer associations between various [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: More shots in meta-testing: visualization of the final layer associations between various [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: More tasks in meta-testing: visualization of the final layer associations between various [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Noise inputs in meta-testing: visualization of the final layer associations between various [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Noise inputs in meta-testing: visualization of the final layer associations between various [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Noise inputs in meta-testing: visualization of the final layer associations between various [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

    cs.LG 2025-06 unverdicted novelty 6.0 of 10

    MoRAM learns continually by adding small rank-1 adapters that act as associative memory items, using input-key similarity to retrieve and mix only the relevant adapters at test time.

Reference graph

Works this paper leans on

74 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018

    work page 2018

  2. [2]

    Learning to continually learn

    Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.09571, 2020

    arXiv 2002

  3. [3]

    Transformers for supervised online continual learning

    Jorg Bornschein, Yazhe Li, and Amal Rannen-Triki. Transformers for supervised online continual learning. arXiv preprint arXiv:2403.01554, 2024

    arXiv 2024

  4. [4]

    Class-incremental continual learning into the extended der-verse

    Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the extended der-verse. IEEE transactions on pattern analysis and machine intelligence, 45(5):5497–5512, 2022

    work page 2022

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    Dark experience for general continual learning: a strong, simple baseline

    Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930, 2020

    2020

  7. [7]

    On tiny episodic memories in continual learning

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019

    arXiv 1902

  8. [8]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020

    arXiv 2009

Show all 74 references
  1. [9]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  2. [10]

    Novel datasets for fine-grained image categorization

    E Dataset. Novel datasets for fine-grained image categorization. First Workshop on Fine Grained Visual Categorization, CVPR. Citeseer. Citeseer. Citeseer, 5(1):2, 2011

    2011

  3. [11]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7): 3366–3...

    2021

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2019

    2019 arXiv

  5. [13]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (120):1–39, 2022

    2022

  6. [14]

    Model-agnostic meta-learning for fast adap- tation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  7. [15]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  8. [16]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  9. [17]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 10

    2021 arXiv

  10. [18]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021

    2021

  11. [19]

    Ms-celeb-1m: A dataset and benchmark for large-scale face recognition

    Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 87–...

    2016

  12. [20]

    Dealing with cross-task class discrimination in online continual learning

    Yiduo Guo, Bing Liu, and Dongyan Zhao. Dealing with cross-task class discrimination in online continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11878–11887, 2023

    2023

  13. [21]

    Look-ahead meta learning for continual learning

    Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 33:11588–11598, 2020

    2020

  14. [22]

    Demystify mamba in vision: A linear attention perspective

    Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605, 2024

    2024 arXiv

  15. [23]

    Remind your neural network to prevent catastrophic forgetting

    Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European conference on computer vision, pages 466–483. Springer, 2020

    2020

  16. [24]

    Openclip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021

    2021

  17. [25]

    Meta-learning representations for continual learning

    Khurram Javed and Martha White. Meta-learning representations for continual learning. Ad- vances in neural information processing systems, 32, 2019

    2019

  18. [26]

    A new approach to linear filtering and prediction problems

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 1960

    1960

  19. [27]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  20. [28]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  21. [29]

    3d object representations for fine- grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013

    2013

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  23. [31]

    Human-level concept learning through probabilistic program induction

    Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015

    2015

  24. [32]

    Recasting continual learning as sequence mod- eling

    Soochan Lee, Jaehyeon Son, and Gunhee Kim. Recasting continual learning as sequence mod- eling. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 70433–70452, 2023

    2023

  25. [33]

    Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting

    Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International conference on machine learning, pages 3925–3934. PMLR, 2019

    2019

  26. [34]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017. 11

    2017

  27. [35]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  28. [36]

    Casia online and offline chinese handwriting databases

    Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Casia online and offline chinese handwriting databases. In Proceedings of the 2011 International Conference on Document Analysis and Recognition, pages 37–41, 2011

    2011

  29. [37]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  30. [38]

    Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision-language models with dynamic rank-selective lora

    Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, and Dong Gong. Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision-language models with dynamic rank-selective lora. arXiv preprint arXiv:2412.01004, 2024

    2024 arXiv

  31. [39]

    Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning

    Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3589–3599, 2021

    2021

  32. [40]

    Fine- grained visual classification of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  33. [41]

    Metaicl: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021

    2021 arXiv

  34. [42]

    Continual learning using a kernel-based method over foundation models

    Saleh Momeni, Sahisnu Mazumder, and Bing Liu. Continual learning using a kernel-based method over foundation models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 19528–19536, 2025

    2025

  35. [43]

    Variational continual learning

    Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10628, 2017

    2017 arXiv

  36. [44]

    Continual learning via local module composition

    Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298– 30312, 2021

    2021

  37. [45]

    Can mamba learn how to learn? a comparative study on in-context learning tasks

    Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04248, 2024

    2024 arXiv

  38. [46]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019

    2019

  39. [47]

    Moe-mamba: Efficient selective state space models with mixture of experts

    Maciej Pioro, Kamil Ciebiera, Krystian Krol, Jan Ludziejewski, and Sebastian Jaszczur. Moe-mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.04081, 2024

    2024 arXiv

  40. [48]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  41. [49]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017

    2001

  42. [50]

    Learning to learn without forgetting by maximizing transfer and minimizing interference

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018

    arXiv

  43. [51]

    Scalable rec- ollections for continual lifelong learning

    Matthew Riemer, Tim Klinger, Djallel Bouneffouf, and Michele Franceschini. Scalable rec- ollections for continual lifelong learning. Proceedings of the AAAI conference on artificial intelligence, 33(01):1352–1359, 2019. 12

    2019

  44. [52]

    Complementary learning for overcoming catastrophic forgetting using experience replay

    Mohammad Rostami, Soheil Kolouri, and Praveen K Pilly. Complementary learning for overcoming catastrophic forgetting using experience replay. arXiv preprint arXiv:1903.04566, 2019

    1903 arXiv

  45. [53]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015

    2015

  46. [54]

    Neural machine translation of rare words with subword units

    Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  47. [55]

    Learning equi-angular representations for online continual learning

    Minhyuk Seo, Hyunseo Koh, Wonje Jeung, Minjae Lee, San Kim, Hankook Lee, Sungjun Cho, Sungik Choi, Hyunwoo Kim, and Jonghyun Choi. Learning equi-angular representations for online continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  48. [56]

    Overcoming catastrophic forgetting with hard attention to the task

    Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pages 4548–4557. PMLR, 2018

    2018

  49. [57]

    Continual learning with deep generative replay

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017

    2017

  50. [58]

    When meta-learning meets online and continual learning: A survey

    Jaehyeon Son, Soochan Lee, and Gunhee Kim. When meta-learning meets online and continual learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  51. [59]

    Efficient transformers: A survey

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint cs.LG/2009.06732, 2020

    2009 arXiv

  52. [60]

    Efficient transformers: A survey

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28, 2022

    2022

  53. [61]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  54. [62]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011

    2011

  55. [63]

    Self-expansion of pre-trained models with mixture of adapters for continual learning

    Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. Self-expansion of pre-trained models with mixture of adapters for continual learning. arXiv preprint arXiv:2403.18886, 2024

    2024 arXiv

  56. [64]

    A comprehensive survey of continual learning: theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  57. [65]

    Attention is all you need

    A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and I Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  58. [66]

    Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction

    Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Representations, 2024

    2024

  59. [67]

    Der: Dynamically expandable representation for class incremental learning

    Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3014–3023, 2021

    2021

  60. [68]

    Self-evolved dynamic expansion model for task-free continual learning

    Fei Ye and Adrian G Bors. Self-evolved dynamic expansion model for task-free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22102–22112, 2023

    2023

  61. [69]

    Lifelong learning with dynamically expandable networks

    Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017. 13

    2017 arXiv

  62. [70]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR, 2017

    2017

  63. [71]

    Class-incremental learning via deep model consolidation

    Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 1131–1140, 2020

    2020

  64. [72]

    Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm

    Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint arXiv:2403.07487, 2024

    2024 arXiv

  65. [73]

    Deep class-incremental learning: A survey

    Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incremental learning: A survey. arXiv preprint arXiv:2302.03648, 2023

    2023 arXiv

  66. [74]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 14 A Datasets A.1 General Image Classification Tasks Cifar-1...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.