Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Continual Learning Using a Kernel-Based Method Over Foundation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A kernelized classifier over frozen foundation-model features matches the joint-training upper bound in class-incremental learning.

desk verdict Simple, strong empirical recipe for replay-free CIL on frozen features, but the headline claim leans on hyperparameters tuned on the test sets and misses the obvious RanPAC comparison. read the letter →

arxiv 2412.15571 v1 pith:64DZEE2X submitted 2024-12-20 cs.LG cs.AIcs.CLcs.CV

Saleh Momeni , Sahisnu Mazumder , Bing Liu This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CV
keywords continuallearningclass-incrementalfoundationmodelsrandomFourierfeatureskernelmethodslineardiscriminantanalysiscatastrophicforgettinginter-taskclassseparation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that class-incremental learning, the setting where a model must keep recognizing old classes while learning new ones, can match the accuracy of joint training on all classes without replaying any old data. The proposed method, Kernel Linear Discriminant Analysis (KLDA), keeps a foundation model frozen and enhances its latent features with a random Fourier approximation of the RBF kernel, then incrementally updates a shared covariance matrix and class means. Classification is done by linear discriminant analysis in that kernelized space, so no network parameters are ever updated. On four text and four image benchmarks, KLDA reports final accuracies equal to or slightly better than joint fine-tuning of the same foundation model, which the field treats as the upper bound for class-incremental learning.

What carries the argument

The machinery is Random Fourier Features (RFF), a Monte Carlo approximation of the Radial Basis Function kernel: $z(x) = \sqrt{2/D}\cos(Wx+\beta)$ with $W$ drawn from $N(0, \sigma^{-2}I)$ and $\beta$ uniform on $[0, 2\pi)$, so that $z(x_i)^\top z(x_j)$ approximates $K(x_i,x_j) = \exp(-\|x_i-x_j\|^2 / (2\sigma^2))$. RFF turns the infinite-dimensional kernel map into a finite $D$-dimensional feature vector, avoiding the infeasible $N\times N$ kernel matrix and making incremental mean/covariance updates possible. The other central object is the shared covariance matrix $\Sigma$, updated online when each new class arrives, which together with the class means defines the LDA scoring rule. With $D=5000$, the method stores roughly 29.5M additional parameters for a BART-base model, dominated by the fixed RFF matrix and the covariance matrix.

What would settle it

On a dataset whose classes are visibly multi-modal in the RFF-transformed feature space (for instance, each class consists of two widely separated clusters), run KLDA and compare final accuracy to joint fine-tuning; a large gap would falsify the shared-covariance Gaussian assumption, while near-equal accuracy would support it.

Watch

Extended reading notes

Core claim

KLDA's central discovery is that the features of a frozen foundation model, which are only weakly linearly separable on their own, become sufficient for near-optimal class-incremental classification once they pass through a random Fourier feature map approximating the RBF kernel. In that transformed space the paper models every class as a Gaussian with a shared covariance matrix, so the optimal classifier is linear: weight vectors $w_m = \Sigma^{-1}\mu_m$ and biases $b_m = -\frac{1}{2}\mu_m^\top \Sigma^{-1}\mu_m$. Because KLDA only accumulates first- and second-order statistics, catastrophic forgetting is impossible by construction, and the shared-covariance Gaussian per class provides a global decision rule that the paper argues eliminates inter-task class separation. The empirical claim is that this simple statistical model reaches the joint-training upper bound, with the ensemble version KLDA-E even exceeding it on three of four text datasets, while using no replay buffer and only a fixed random feature matrix.

Load-bearing premise

The load-bearing premise is that after the random Fourier transform, the features of every class are well described by one Gaussian with a common covariance matrix, so the linear discriminant scores are near-optimal; if real features are far from Gaussian or have very different covariances, the claimed natural separation between old and new classes is not guaranteed.

Editorial extensions

If this is right

  • Adding the RFF kernel map over frozen foundation-model features closes most of the accuracy gap that plain NCM and plain LDA leave to joint training, so the bottleneck in CIL is linear separability of features rather than forgetting itself.
  • No replay buffer is needed: KLDA stores only class means, a shared covariance matrix, and the fixed random feature matrix, so its memory does not grow with the number of tasks beyond the class means.
  • The method transfers across model families and sizes, from MiniLM and BART to Mistral-7b in text and DINOv2 in vision, suggesting the result is not tied to one foundation model.
  • Training is only moment accumulation, so KLDA trains in about 10 seconds on CLINC with BART-base, compared with 4 minutes for joint fine-tuning and 11-23 minutes for fine-tuning baselines.
  • Averaging probabilities across five random feature draws (KLDA-E) further improves accuracy, indicating that part of the remaining error is Monte Carlo variance of the kernel approximation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If KLDA's results generalize, the practical recipe for CIL becomes: pick a strong frozen feature extractor, kernelize its outputs, and fit a Gaussian classifier; the large literature on replay, regularization, and prompt tuning may be unnecessary when such features are available.
  • The ensemble gain in KLDA-E suggests that increasing the random feature dimension $D$ might substitute for ensembling; comparing the accuracy of a single KLDA model with $D=25{,}000$ against KLDA-E with five models of $D=5{,}000$ would isolate the Monte Carlo approximation error.
  • Because KLDA only needs running means and a running covariance, it can be applied directly to online or streaming class-incremental learning where samples arrive one at a time, a setting the paper does not evaluate.
  • The shared-covariance Gaussian assumption is the most fragile part; extending KLDA to per-class covariance shrinkage or a mixture of Gaussians per class would test whether the upper-bound match survives when classes are strongly multi-modal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes KLDA, a class-incremental learning method that keeps a foundation model frozen, applies a radial-basis-function kernel approximated by random Fourier features (RFF) to the extracted features, and incrementally maintains class means and a shared covariance matrix, classifying via linear discriminant analysis (LDA). An ensemble variant, KLDA-E, averages softmax scores over several RFF draws. Experiments on four text intent/classification datasets (CLINC, Banking, DBpedia, HWU) and four image datasets (CIFAR-10/100, TinyImageNet, Stanford Cars) compare against fine-tuning, prompting, pseudo-replay, and prototype-based baselines, with joint fine-tuning of the frozen or fine-tuned foundation model as the upper bound. The central empirical claim is that KLDA(-E), without replay, reaches or exceeds the joint-training accuracy on most benchmarks.

Significance. If the reported numbers are robust, this is a practically valuable result: a simple, replay-free, non-parametric CIL method that leverages frozen foundation-model features and reaches the joint-training upper bound. The paper provides public code, repeated runs with random seeds, multiple language and vision backbones, and a consistent set of baselines, which are strengths. The main weakness is that the two hyperparameters of KLDA, the RFF dimension D and the kernel scale sigma, are selected after seeing the same benchmarks whose final accuracies are reported, with no held-out validation or explicit selection rule. Because the headline claim is parity with joint training, the risk of selection artifacts is the load-bearing issue. The theoretical justification for avoiding inter-task class separation is also asserted rather than proven.

major comments (4)
  1. [Implementation Details and Figure 1] The manuscript does not specify how sigma and D are selected. It states that D=5000 was found through "empirical testing" and that sigma is "empirically determined within range [10^-2, 10^-6] for each FM," and Figure 1 sweeps these hyperparameters on the same datasets that appear in Tables 1-3. No held-out validation split, selection criterion, or pre-registered configuration is described. In a real CIL deployment, future tasks are not available at configuration time, so the reported parity with Joint Fine-tuning could partly reflect choosing hyperparameters after observing test-set outcomes. To support the central claim, the authors should either (a) select hyperparameters on a disjoint validation set and report the resulting accuracies, or (b) demonstrate that the Figure 1 curves are sufficiently flat that the choice is immaterial across all reported datasets. Without this, the main empirical claim is not yet established.
  2. [Theoretical Justification] The paragraph titled "Theoretical Justification" does not constitute a proof of the claim that KLDA avoids inter-task class separation. It asserts that because each class is represented as a Gaussian, "effectively each task has only one class," so within-task prediction is always correct and the Gaussian serves as an OOD detector. This does not verify the conditions of Kim et al. (2022, 2023) and does not address the fact that the RFF-transformed features are not necessarily Gaussian; if they are not, the LDA decision boundary in Eq. (4) need not be optimal or safe against class confusion. If the theoretical argument is retained, it should be stated formally with explicit assumptions on the feature distribution and the sufficiency conditions from the cited work. Otherwise, the corresponding contribution claim should be weakened to an empirical observation.
  3. [Related Work and Baselines] The paper positions KLDA as an extension of streaming LDA (Hayes and Kanan 2020) and mentions random projections and pre-trained models (McDonnell et al. 2023) in the introduction, but it does not include either SLDA or RanPAC as empirical baselines. Since the novelty of KLDA relative to plain LDA is the RBF/RFF feature transformation, a direct comparison against SLDA on the same frozen features is needed to isolate the contribution of the kernel component. Without such a comparison, the improvement over NCM and LDA reported in Table 1 cannot be attributed specifically to the kernelized features rather than to the incremental LDA mechanism itself.
  4. [Evaluation on Image Datasets] The image experiments in Table 3 do not use task splits; the text says "task splits are not required for this evaluation." This means the image results compare one-by-one class prototype accumulation against joint training, which is a valid feature-extraction benchmark but not a class-incremental learning protocol with a task sequence. The claim that KLDA is "evaluated" on image datasets should be qualified accordingly, or the experiments should be run under the same CIL protocol used for the text datasets, to avoid over-generalizing the image results to the incremental setting.
minor comments (5)
  1. [Datasets] The references for DBpedia and HWU appear to be swapped: the paper cites Auer et al. (2007) for HWU, but that is the DBpedia paper, and it cites Liu et al. (2021b) for DBpedia, but that reference is the conversational-agent benchmark that is usually associated with HWU. Please correct these citations.
  2. [Table 1] NCM and LDA report zero variance (±0.00). This should be explained, for example by noting that their final prototypes and covariance are invariant to task order, so the only randomness across runs would come from the data split, which is held fixed.
  3. [Table 3] The text states that ensembles did not improve image results but does not explain why. A brief comment on this difference between text and image domains would help the reader interpret the role of the ensemble in KLDA-E.
  4. [Conclusion / Terminology] The term "upper bound" is used for Joint Fine-tuning. This is a standard CIL convention, but it should be described as the empirical joint-training baseline rather than a theoretical upper bound, since a fixed frozen feature extractor with a better classifier could in principle exceed it (as KLDA-E does on three datasets in Table 1).
  5. [Algorithm 1] In the covariance update on line 17, the notation is slightly confusing because Ntotal is updated before being used in the coefficient Nprev/Ntotal. The formula is correct, but renaming the variables (e.g., T for total before update) would improve readability.

Circularity Check

1 steps flagged · score 4.0 of 10

KLDA's headline accuracy claim is partly a product of test-set hyperparameter selection; the method's derivation itself is not circular.

  1. fitted input called prediction [Implementation Details (Experimental Setup) and Figure 1 (Analysis of Hyperparameters)]
    "Given the CIL setup, where tasks are learned incrementally, the system does not see all tasks at the same time, and validation sets are not typically available. Therefore, it is hard to optimize the parameters for all tasks. Through empirical testing, we found that setting D to 5000 offers a balanced trade-off between memory usage and performance. The σ parameter is also empirically determined within range [10−2, 10−6] for each FM."

    The central empirical claim—that KLDA/KLDA-E matches or surpasses Joint Fine-tuning on the four text datasets—is reported after D=5000 and σ were 'empirically determined' using the same datasets whose final accuracies appear in Table 1. Figure 1 then sweeps σ and D against exactly those datasets, so the chosen configuration is the one that looked best on the reported test sets. The headline result is therefore a post-hoc selection outcome rather than an independent held-out prediction: the reported accuracy is partly forced by the tuning procedure, not produced by a fixed a-priori configuration.

full rationale

The method itself is a standard, self-contained derivation: frozen FM features are mapped through Random Fourier Features, class means and a shared covariance are accumulated incrementally, and classification uses LDA scores (Eqs. 3–6, 11–13). No equation in the paper defines a predicted quantity in terms of the target accuracy, and no fitted parameter is renamed as a theoretical result. The Kim et al. (2022, 2023) citations used in the Theoretical Justification are self-citations by the author group, but they are general theorems about CIL conditions and do not assume KLDA's conclusion, so they are not load-bearing circularity. The main circular-content concern is the hyperparameter selection: D and σ are tuned on the same benchmarks whose final accuracy is then reported as evidence that KLDA reaches the joint-training upper bound. This makes the headline empirical claim partially a selection artifact. However, the method's derivation is not circular, and the reported numbers would still be meaningful if a proper held-out validation protocol were applied. Score 4 reflects this partial, empirical-selection circularity rather than a mathematically forced derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or structural entities. It relies on standard kernel approximation math and on several domain assumptions about the quality of foundation-model features and the validity of the shared-covariance Gaussian model in the transformed space. The two main free parameters, sigma and D, are selected using the evaluation benchmarks themselves, which is the most significant caveat in the ledger.

free parameters (3)
  • RFF dimension D = 5000
    Chosen empirically as a balance between memory usage and accuracy; affects the fidelity of the kernel approximation and the size of the covariance matrix.
  • RBF kernel scale sigma = Selected per foundation model from [10^-2, 10^-6]
    Determined empirically on the evaluation benchmarks; directly controls the width of the Gaussian kernel and the spread of the random Fourier features.
  • Ensemble size E = 5
    Number of KLDA models averaged in KLDA-E; fixed design choice to stabilize scores, not extensively tuned.
assumptions (5)
  • standard math Bochner's theorem guarantees that the random Fourier feature map z(x) satisfies E[z(x_i)^T z(x_j)] = K(x_i, x_j) for the RBF kernel.
    Invoked in Section 'Approximating the Kernel with Random Fourier Features' to justify replacing the full kernel matrix with random features.
  • domain assumption The foundation model is frozen and its features are sufficiently discriminative for the target classes.
    The method does not adapt the backbone; the authors state in Limitations that if FM features are not well-suited to a domain, accuracy may suffer.
  • domain assumption Class-conditional distributions in the RFF space are approximately Gaussian with a shared covariance matrix.
    The LDA scoring rule and the theoretical justification for avoiding inter-task class separation both rely on this Gaussian assumption.
  • domain assumption The approximation error of random Fourier features at D=5000 is small enough not to hurt classification.
    The paper chooses D=5000 empirically and does not quantify the approximation error or its effect on final accuracy.
  • domain assumption The Kim et al. theory that within-task prediction and out-of-distribution detection are sufficient for good CIL applies to this method.
    The Theoretical Justification paragraph cites Kim et al. without showing rigorously how the Gaussian-per-class construction satisfies the theory's conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Learning Using a Kernel-Based Method Over Foundation Models." pith.science (2026). https://pith.science/paper/64DZEE2X

@misc{pith2026241215571,
  author       = {Pith},
  title        = {Pith review of: Continual Learning Using a Kernel-Based Method Over Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64DZEE2X}},
  note         = {Machine review of arXiv:2412.15571}
}
read the original abstract

Continual learning (CL) learns a sequence of tasks incrementally. This paper studies the challenging CL setting of class-incremental learning (CIL). CIL has two key challenges: catastrophic forgetting (CF) and inter-task class separation (ICS). Despite numerous proposed methods, these issues remain persistent obstacles. This paper proposes a novel CIL method, called Kernel Linear Discriminant Analysis (KLDA), that can effectively avoid CF and ICS problems. It leverages only the powerful features learned in a foundation model (FM). However, directly using these features proves suboptimal. To address this, KLDA incorporates the Radial Basis Function (RBF) kernel and its Random Fourier Features (RFF) to enhance the feature representations from the FM, leading to improved performance. When a new task arrives, KLDA computes only the mean for each class in the task and updates a shared covariance matrix for all learned classes based on the kernelized features. Classification is performed using Linear Discriminant Analysis. Our empirical evaluation using text and image classification datasets demonstrates that KLDA significantly outperforms baselines. Remarkably, without relying on replay data, KLDA achieves accuracy comparable to joint training of all classes, which is considered the upper bound for CIL performance. The KLDA code is available at https://github.com/salehmomeni/klda.

Figures

Figures reproduced from arXiv: 2412.15571 by the authors.

Figure 1
Figure 1. Hyperparameter impact on KLDA: (Left) Effect of [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 38 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abati, D.; Tomczak, J.; Blankevoort, T.; Calderara, S.; Cucchiara, R.; and Bejnordi, B. E. 2020. Conditional channel gated networks for task-aware continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3931--3940

    work page 2020

  4. [4]

    Aljundi, R.; Lin, M.; Goujaud, B.; and Bengio, Y. 2019. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32

    work page 2019

  5. [5]

    Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. 2007. Dbpedia: A nucleus for a web of open data. In international semantic web conference, 722--735

    work page 2007

  6. [6]

    Casanueva, I.; Tem c inas, T.; Gerz, D.; Henderson, M.; and Vuli \'c , I. 2020. Efficient Intent Detection with Dual Sentence Encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, 38--45

    work page 2020

  7. [7]

    Chen, Z.; and Liu, B. 2018. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3): 1--207

    work page 2018

  8. [8]

    De Lange, M.; Aljundi, R.; Masana, M.; Parisot, S.; Jia, X.; Leonardis, A.; Slabaugh, G.; and Tuytelaars, T. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7): 3366--3385

    2021

Show all 59 references
  1. [9]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

  2. [10]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations

    2020

  3. [11]

    Geng, B.; Yuan, F.; Xu, Q.; Shen, Y.; Xu, R.; and Yang, M. 2021. Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL/IJCNLP)

    2021

  4. [12]

    A.; and Zettlemoyer, L

    Gururangan, S.; Lewis, M.; Holtzman, A.; Smith, N. A.; and Zettlemoyer, L. 2022. DEMix Layers: Disentangling Domains for Modular Language Modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, 5557--5576

    2022

  5. [13]

    Hastie, T. 2009. The elements of statistical learning: data mining, inference, and prediction

    2009

  6. [14]

    L.; and Kanan, C

    Hayes, T. L.; and Kanan, C. 2020. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 220--221

    2020

  7. [15]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  8. [16]

    He, X.; and Jaeger, H. 2018. Overcoming catastrophic interference using conceptor-aided backpropagation. In International Conference on Learning Representations

    2018

  9. [17]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  10. [18]

    Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, 2790--2799. PMLR

    2019

  11. [19]

    Huang, Y.; Zhang, Y.; Chen, J.; Wang, X.; and Yang, D. 2021. Continual Learning for Text Classification with Information Disentanglement Based Regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2021

  12. [20]

    Izenman, A. J. 2013. Linear discriminant analysis. In Modern multivariate statistical techniques: regression, classification, and manifold learning, 237--280. Springer

    2013

  13. [21]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

    2023 arXiv

  14. [22]

    Ke, Z.; and Liu, B. 2022. Continual learning of natural language processing tasks: A survey. arXiv preprint arXiv:2211.12701

    2022 arXiv

  15. [23]

    Ke, Z.; Liu, B.; Ma, N.; Xu, H.; and Shu, L. 2021. Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning. Advances in Neural Information Processing Systems, 34

    2021

  16. [24]

    Kenton, J. D. M.-W. C.; and Toutanova, L. K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, 4171--4186

    2019

  17. [25]

    Kim, G.; Xiao, C.; Konishi, T.; Ke, Z.; and Liu, B. 2022. A theoretical study on solving continual learning. Advances in neural information processing systems, 35: 5065--5079

    2022

  18. [26]

    Kim, G.; Xiao, C.; Konishi, T.; and Liu, B. 2023. Learnability and algorithm for continual learning. In International Conference on Machine Learning, 16877--16896. PMLR

    2023

  19. [27]

    A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al

    Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13): 3521--3526

    2017

  20. [28]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  21. [29]

    J.; Clarke, C.; Lee, A.; Hill, P.; Kummerfeld, J

    Larson, S.; Mahendran, A.; Peper, J. J.; Clarke, C.; Lee, A.; Hill, P.; Kummerfeld, J. K.; Leach, K.; Laurenzano, M. A.; Tang, L.; et al. 2019. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In Proceedings of the 2019 Conference on Empirical Metho...

    2019

  22. [30]

    Le, Y.; and Yang, X. 2015. Tiny imagenet visual recognition challenge. CS 231N, 7: 7

    2015

  23. [31]

    Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461

    2019 arXiv

  24. [32]

    Lin, H.; Shao, Y.; Qian, W.; Pan, N.; Guo, Y.; and Liu, B. 2024. Class incremental learning via likelihood ratio based task prediction. ICML-2024

    2024

  25. [33]

    Lin, S.; Yang, L.; Fan, D.; and Zhang, J. 2022. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35: 16165--16177

    2022

  26. [34]

    Liu, Q.; Yu, X.; He, S.; Liu, K.; and Zhao, J. 2021 a . Lifelong intent detection via multi-strategy rebalancing. arXiv preprint arXiv:2108.04445

    2021 arXiv

  27. [35]

    Liu, X.; Eshghi, A.; Swietojanski, P.; and Rieser, V. 2021 b . Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction: 10th International Workshop on Spoken Dialogue Syst...

    2021

  28. [36]

    Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  29. [37]

    McCloskey, M.; and Cohen, N. J. 1989. Catastrophic interference in connectionist networks: The sequential learning problem . In Psychology of learning and motivation, volume 24, 109--165. Elsevier

    1989

  30. [38]

    D.; Gong, D.; Parvaneh, A.; Abbasnejad, E.; and van den Hengel, A

    McDonnell, M. D.; Gong, D.; Parvaneh, A.; Abbasnejad, E.; and van den Hengel, A. 2023. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems (NeurIP-2023), 36

    2023

  31. [39]

    V.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; HAZIZA, D.; Massa, F.; El-Nouby, A.; et al

    Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H. V.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; HAZIZA, D.; Massa, F.; El-Nouby, A.; et al. 2023. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research

    2023

  32. [40]

    Qin, Y.; Zhang, J.; Lin, Y.; Liu, Z.; Li, P.; Sun, M.; and Zhou, J. 2022. ELLE: Efficient Lifelong Pre-training for Emerging Data. In Findings of the Association for Computational Linguistics: ACL 2022, 2789--2810

    2022

  33. [41]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140)

    2020

  34. [42]

    Rahimi, A.; and Recht, B. 2007. Random features for large-scale kernel machines. Advances in neural information processing systems, 20

    2007

  35. [43]

    Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)

    2019

  36. [44]

    Rudin, W. 2017. Fourier analysis on groups. Courier Dover Publications

    2017

  37. [45]

    Serra, J.; Suris, D.; Miron, M.; and Karatzoglou, A. 2018. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, 4548--4557. PMLR

    2018

  38. [46]

    Shao, Y.; Guo, Y.; Zhao, D.; and Liu, B. 2023. Class-Incremental Learning based on Label Generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Short Papers), 1263--1276

    2023

  39. [47]

    K.; Kim, J.; and Kim, J

    Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017. Continual learning with deep generative replay. Advances in neural information processing systems, 30

    2017

  40. [48]

    Sun, F.-K.; Ho, C.-H.; and Lee, H.-Y. 2019. LAMOL: LAnguage MOdeling for Lifelong Language Learning. In International Conference on Learning Representations

    2019

  41. [49]

    M.; and Tolias, A

    Van de Ven, G. M.; and Tolias, A. S. 2019. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734

    2019 arXiv

  42. [50]

    Wang, F.-Y.; Zhou, D.-W.; Liu, L.; Ye, H.-J.; Bian, Y.; Zhan, D.-C.; and Zhao, P. 2022 a . BEEF: Bi-compatible class-incremental learning via energy-based expansion and fusion. In The Eleventh International Conference on Learning Representations

    2022

  43. [51]

    Wang, L.; Zhang, X.; Su, H.; and Zhu, J. 2024. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  44. [52]

    Wang, Z.; Liu, Y.; Ji, T.; Wang, X.; Wu, Y.; Jiang, C.; Chao, Y.; Han, Z.; Wang, L.; Shao, X.; et al. 2023. Rehearsal-free continual language learning via efficient parameter isolation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics,...

    2023

  45. [53]

    Wang, Z.; Zhang, Z.; Lee, C.-Y.; Zhang, H.; Sun, R.; Ren, X.; Su, G.; Perot, V.; Dy, J.; and Pfister, T. 2022 b . Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 139--149

    2022

  46. [54]

    Wortsman, M.; Ramanujan, V.; Liu, R.; Kembhavi, A.; Rastegari, M.; Yosinski, J.; and Farhadi, A. 2020. Supermasks in superposition. Advances in Neural Information Processing Systems, 33: 15173--15184

    2020

  47. [55]

    Yan, S.; Xie, J.; and He, X. 2021. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3014--3023

    2021

  48. [56]

    Yang, L.; Luo, P.; Change Loy, C.; and Tang, X. 2015. A large-scale car dataset for fine-grained categorization and verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3973--3981

    2015

  49. [57]

    Yang, Y.; Zhou, J.; Ding, X.; Huai, T.; Liu, S.; Chen, Q.; Xie, Y.; and He, L. 2024. Recent advances of foundation language models-based continual learning: a survey. ACM Computing Surveys

    2024

  50. [58]

    Zenke, F.; Poole, B.; and Ganguli, S. 2017. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 3987--3995. PMLR

    2017

  51. [59]

    Zhou, D.-W.; Sun, H.-L.; Ning, J.; Ye, H.-J.; and Zhan, D.-C. 2024. Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.