Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Parameter-Efficient Continual Fine-Tuning: A Survey

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large pre-trained models can keep adapting to new tasks with tiny parameter updates while retaining old knowledge.

desk verdict A useful qualitative map of PECFT that is undermined by a comparative evaluation with no experimental protocol and a parameter table with clear unit errors; worth revising, not rejecting. read the letter →

arxiv 2504.13822 v3 pith:7CIXSQ7M submitted 2025-04-18 cs.LG cs.AI

Eric Nuertey Coleman , Luigi Quarantiello , Ziyue Liu , Qinwen Yang , Samrat Mukherjee , Julio Hurtado , Vincenzo Lomonaco This is my paper · ORCID
classification cs.LGcs.AI
keywords continuallearningparameter-efficientfine-tuningcatastrophicforgettinglow-rankadaptationprompttuningadapterspre-trainedmodelssurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey tries to establish that parameter-efficient continual fine-tuning (PECFT) is a coherent and useful paradigm: large pre-trained models can be adapted to an endless stream of tasks by updating only a small set of parameters, without sacrificing old knowledge. It argues that continual learning supplies the sequential-learning machinery while parameter-efficient fine-tuning supplies the cheap adaptation, and that their combination is what makes lifelong deployment of large models practical. The survey organizes existing methods into adapter-based, LoRA-based, prompt-based, and unified frameworks, and compares them on accuracy and forgetting across three benchmarks. A sympathetic reader would care because the claim, if right, means a single large model can keep evolving in the real world without prohibitive retraining costs.

What carries the argument

The organizing machinery is a taxonomy of PECFT methods plus the design principles that separate them: parameter separation (modular separation, subspace separation, and input space separation) and interference prevention (orthogonality constraints, architectural modularity, and compositional isolation). The taxonomy takes each PEFT family—adapters, LoRA, prompts, and unified frameworks—and shows how continual-learning mechanisms such as routing, merging, masking, prompt pools, and orthogonal projection attach to it. The comparative tables use Average Accuracy and Average Forgetting as the load-bearing metrics that make the paradigm's trade-offs visible.

What would settle it

Running the ten methods compared in Tables 4 and 5 on CIFAR-100 with 10 tasks under a single, fixed protocol (identical epochs, learning rate, LoRA rank, prompt length, and backbone) and showing that the accuracy ranking or the forgetting ranking changes materially would overturn the survey's comparative conclusions. Likewise, correcting Table 3 so that all parameter counts are in the same units would settle whether the efficiency comparison is meaningful.

Watch

Extended reading notes

Core claim

The paper's central claim is that PECFT methods enable large pre-trained models to continuously evolve and adapt to new tasks while preserving past knowledge and maintaining computational efficiency. Concretely, the survey reports that adapter-based methods achieve the highest average accuracy, LoRA-based methods offer the most consistent cross-dataset performance, prompt-based methods are the lightest but lag in accuracy, and subspace-constrained methods such as InfLoRA yield the lowest forgetting (4.61% on CIFAR-100 with 10 tasks). It also finds that high accuracy and low forgetting are decoupled: no method tops both tables, and the choice of parameter-separation and interference-prevention strategy matters more than the choice of PEFT module alone.

Load-bearing premise

The comparative rankings in Tables 4 and 5 assume that every method was evaluated under a consistent experimental protocol with the same backbone, hyperparameters, and evaluation conditions; if the numbers came from heterogeneous original setups, the reported ordering is not reliable.

Editorial extensions

If this is right

  • Practitioners who need peak accuracy on a specific domain should prefer adapter-based PECFT methods, which top the averaged accuracy comparison at the cost of more trainable parameters.
  • For multi-domain deployment, LoRA-based methods such as LoRAC-IPC and SD-LoRA give the most consistent accuracy across CIFAR-100, CUB-200, and Tiny ImageNet, with moderate, predictable degradation when the task count doubles.
  • Stability-critical applications can use subspace-constrained methods like InfLoRA, whose 4.61% average forgetting on CIFAR-100 is the best in the comparison.
  • Prompt-based methods, despite being the most parameter-light, underperform by 3-6% and scale poorly, so they are a reasonable choice only when memory is the dominant constraint.
  • Unified frameworks such as LAE and APER decouple the continual-learning strategy from the underlying PEFT module, so any adapter, LoRA, or prompt method can be turned into a continual learner; LAE loses only 1.32% accuracy when moving from 10 to 20 CIFAR-100 tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported trade-offs persist under a truly unified protocol, a natural next step is hybrid designs that combine orthogonal-subspace updates (low forgetting) with adapter expansion (high accuracy), a combination no single method in the survey currently offers.
  • The survey's future-direction section points toward model merging; a testable extension is to treat each task's LoRA or adapter weights as a task vector and merge them sequentially with interference-reducing operators, which would make PECFT methods memory-constant rather than growing with tasks.
  • Because the comparison only covers image classification, the claimed rankings may not transfer to language or reasoning tasks; a natural extension is to rerun the same methods on sequential question answering or instruction-tuning streams, where representation drift behaves differently.
  • Before relying on the efficiency comparison, a reader should re-derive the parameter counts in Table 3 from the original papers, since entries like 'BERT Base Uncased (109B)' appear to use inconsistent units.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This survey defines and reviews the emerging area of Parameter-Efficient Continual Fine-Tuning (PECFT), which combines Continual Learning (CL) with Parameter-Efficient Fine-Tuning (PEFT) to adapt large pre-trained models to sequential tasks while limiting forgetting and computational cost. The paper first presents background on CL scenarios and classical CL algorithms, then reviews PEFT methods (adapters, prompts, reparameterization, selective methods), and then proposes a taxonomy of PECFT methods organized into adapter-based, LoRA-based, prompt-based, and unified frameworks. It defines evaluation metrics (average accuracy, average forgetting, forward/backward transfer, model size efficiency), reports a comparative performance evaluation in Tables 4 and 5, distills design principles (parameter separation, interference prevention), and discusses future directions including multi-modality, model merging, reasoning tasks, and realistic settings.

Significance. If the survey's central claims are accepted, it provides a useful systematization of a rapidly growing intersection between CL and PEFT, with a clear taxonomy (Figure 2) and a structured catalog of representative methods. The qualitative coverage is broad and largely recognizable to a reader familiar with the literature, and the synthesis of design principles in Section 5.7 is a genuine contribution that goes beyond a simple enumeration of methods. However, the paper's quantitative contribution is load-bearing for its claim to characterize the state of the art and to derive practical recommendations; those numbers are currently not supported by any described experimental protocol, and Table 3 contains unit and consistency errors. The paper also includes self-acknowledged limitations in Section 5.6 that are not resolved in the presentation. The qualitative survey value is real, but the quantitative evidence must be repaired or explicitly downgraded before the paper can be accepted.

major comments (4)
  1. [Section 5.6, Tables 4 and 5] The comparative evaluation that drives the recommendations in Section 5.6 and the design principles in Section 5.7 is not reproducible or verifiable as reported. No experimental protocol is given: there is no specification of the backbone model, the task-splitting rule, the optimizer, learning rate, epochs, LoRA rank, prompt length, batch size, number of seeds, or whether rehearsal/memory buffers were used. The section itself concedes that 'individual methods often rely on different experimental settings,' yet the rows of Tables 4 and 5 are treated as directly comparable. Because accuracy and forgetting values depend strongly on these choices, the rankings and the conclusions drawn from them (e.g., 'adapter methods for peak performance,' 'LoRA methods for consistent multi-domain deployment') are not established. Please either provide the full protocol and state whether the numbers were produced by the authors under a unified setup, or reposition Tables 4 and 5 as a literature-reported comparison with clear per-source attribution and explicit caveats that no cross-method ranking should be inferred.
  2. [Table 3] Table 3 contains unit errors and internal inconsistencies that undermine its quantitative content. BERT Base Uncased is labeled (109B), ALBERT Base V2 (11B), RoBERTa Base (124B), ViT B16 (85B), and CLIP ViT B16 (149B); these are roughly three orders of magnitude too large and should be in millions. The reported percentages are also not internally consistent with the listed total and trainable parameter counts: for example, ViT B16 lists QLoRA trainable parameters as 43,906 with 0.51%, while LoRA lists 442,368 with 0.52% on the same model, and CLIP ViT B16 lists QLoRA as citation [88] rather than [22] or [23]. Please correct the units, recompute all percentages from a single consistent source, and verify the citations in each row.
  3. [Section 5.6.2, Table 5] The forgetting analysis in Section 5.6.2 makes strong comparative claims on the basis of a single scenario (CIFAR-100, 10 tasks) with no protocol or error-bar derivation described. Statements such as 'L2P suffers from severe forgetting (10.61%), nearly doubling the forgetting of any other competing method' and the conclusion that 'subspace-constrained approaches like InfLoRA offer the best stability-performance balance' depend on the comparability of the table entries. Please clarify how the standard deviations were computed, how many runs were performed, and whether the same backbone, optimizer, and task order were used for all ten methods; otherwise these claims should be substantially softened.
  4. [References [22] and [23]] References [22] and [23] are the same QLoRA paper (Dettmers et al.), which creates confusion in the text and in Table 3, where some rows cite [22] and one row cites [88] for QLoRA. Please deduplicate the citation and ensure every QLoRA row points to the single correct reference.
minor comments (5)
  1. [Section 2.1, Eq. (1)] Equation (1) uses the summation index t for both the task index and the inner sum over samples, which is notationally confusing; please replace the inner index with i or n.
  2. [Section 5.5, Eq. (18)] The definition of Model Size Efficiency (MS) is unclear: the formula 'min(1, (sum_i Mem(theta1)/Mem(theta_i))/N)' appears to average ratios but the min operator and the placement of N are ambiguous, and no reference is given for this exact formulation. Please clarify the definition or cite a source.
  3. [Section 4.2] The text describes QLoRA with reference [22] while the reference list contains an identical duplicate [23]; after deduplication, please also check the in-text citation for QLoRA in Table 3, which currently cites [88] in the CLIP block.
  4. [Section 4.1.1] The description of Mix-and-Match adapters states that they 'match full fine-tuning performance on some benchmarks while fine-tuning only about 6.7% of parameters,' but the cited source [43] is listed as a parallel conference version of [42]; please reconcile the two citations and clarify which paper is being described.
  5. [Throughout] Several sentences contain grammatical errors or missing words, e.g., 'On one hand, CL enables models to continually adapt whiles PEFT ensures that efficiency whiles the model adapts' in Section 1 and 'The organization of this paper as follows' in the same section. A thorough language edit is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the survey's claims are descriptive and its comparative tables are inherited external numbers; the sole self-citation is non-load-bearing.

full rationale

This paper is a survey, not a derivation. Its central claim that PECFT methods enable continual adaptation while preserving knowledge and efficiency is an organizing descriptive statement supported by the cited literature, not a quantity derived from its own inputs. There is no fitted parameter later relabeled as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in through citation; equations in the paper (Eqs. 1-18) are standard definitions of CL objectives, memory updates, attention, prompt templates, LoRA decomposition, and evaluation metrics, and they are not used to generate the survey's conclusions by construction. The comparative evaluation in Section 5.6 is the closest thing to a quantitative contribution, but the absence of a common experimental protocol and the likely copying of numbers from heterogeneous original papers is a reproducibility and correctness weakness, not circularity: the table values are external inputs, not outputs that reduce back to their own assumptions. The paper even concedes this limitation in the text: 'individual methods often rely on different experimental settings,' which further shows the authors are not presenting the table as a derivation from a unified protocol. Table 3's unit errors (e.g., BERT Base labeled 109B) are factual mistakes, not circular reasoning. The only self-citation is reference [20], the authors' own Adaptive LoRA merging paper, mentioned in Section 6.2 as one of several future model-merging directions ('While Adaptive LoRA Merging for Domain Incremental Learning[20] highlights the limitations of fixed-weight merging'); this mention is illustrative, not load-bearing for the survey's central taxonomies or recommendations, and the direction is independently supported by other cited merging works such as TIES-MERGING and DARE. No circular step meets the evidentiary bar of quoted text plus exhibited reduction, so the appropriate finding is no significant circularity, with a low score reflecting only the presence of a non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The survey's central claims depend on the validity of its taxonomy, the reliability of pre-trained representations, and the comparability of the reported benchmarks. The last two are assumed rather than demonstrated, and the paper contains quantitative errors in Table 3.

assumptions (4)
  • domain assumption Pre-trained models provide generic representations that are reusable across tasks.
    Section 2.2 and throughout; the entire PECFT premise relies on frozen backbones retaining general knowledge.
  • domain assumption The CL scenarios (TIL, CIL, DIL, OCL, CIR, RM) are a complete and standard taxonomy.
    Section 2.1.2; the survey's organization depends on these categories being canonical.
  • ad hoc to paper The reported accuracy and forgetting numbers in Tables 4 and 5 were obtained under directly comparable experimental settings.
    Section 5.6 claims 'our evaluation' but gives no protocol; comparability of numbers is assumed.
  • ad hoc to paper The parameter counts in Table 3 are correct and use consistent units.
    Table 3; unit errors undermine this assumption.
invented entities (1)
  • PECFT (Parameter-Efficient Continual Fine-Tuning)
    purpose: Name for the intersection of CL and PEFT, used as the survey's organizing concept.
    The term is introduced by the authors ('which we called PECFT', Section 1); no independent benchmark or falsifiable prediction is tied to the name itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parameter-Efficient Continual Fine-Tuning: A Survey." pith.science (2026). https://pith.science/paper/7CIXSQ7M

@misc{pith2026250413822,
  author       = {Pith},
  title        = {Pith review of: Parameter-Efficient Continual Fine-Tuning: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CIXSQ7M}},
  note         = {Machine review of arXiv:2504.13822}
}
read the original abstract

The emergence of large pre-trained networks has revolutionized the AI field, unlocking new possibilities and achieving unprecedented performance. However, these models inherit a fundamental limitation from traditional Machine Learning approaches: their strong dependence on the \textit{i.i.d.} assumption hinders their adaptability to dynamic learning scenarios. We believe the next breakthrough in AI lies in enabling efficient adaptation to evolving environments -- such as the real world -- where new data and tasks arrive sequentially. This challenge defines the field of Continual Learning (CL), a Machine Learning paradigm focused on developing lifelong learning neural models. One alternative to efficiently adapt these large-scale models is known Parameter-Efficient Fine-Tuning (PEFT). These methods tackle the issue of adapting the model to a particular data or scenario by performing small and efficient modifications, achieving similar performance to full fine-tuning. However, these techniques still lack the ability to adjust the model to multiple tasks continually, as they suffer from the issue of Catastrophic Forgetting. In this survey, we first provide an overview of CL algorithms and PEFT methods before reviewing the state-of-the-art on Parameter-Efficient Continual Fine-Tuning (PECFT). We examine various approaches, discuss evaluation metrics, and explore potential future research directions. Our goal is to highlight the synergy between CL and Parameter-Efficient Fine-Tuning, guide researchers in this field, and pave the way for novel future research directions.

Figures

Figures reproduced from arXiv: 2504.13822 by the authors.

Figure 1
Figure 1. Classification of CL Method to the specific characteristics of the task while using the pertaining as a good starting point. Parameter-Efficient Fine-Tuning has emerged as a significant advancement in large PTMs studies. It refines the capabilities of pre-trained models by strate￾gically adjusting a limited subset of parameters during fine-tuning. This ap￾proach differs significantly from comprehensive fine-tuning, … view at source ↗
Figure 2
Figure 2. Taxonomy of Parameter-Efficient Continual Fine-Tuning methods [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Parameter Separation Strategies Architectural Modularity. This aims to prevent interference through physical separation of parameters, where each task uses dedicated modules while the backbone remains frozen. This architectural separation provides interference resistance through construction rather than restraint, since tasks often use en￾tirely different parameters and they cannot interfere with each other. This in… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Parts Are Greater Than the Sum: Automated Task Sequencing for Efficient Training of Multi-Policy LLMs

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Automatically grouping and sequencing tasks into multiple QLoRA adapters improves continual fine-tuning performance over a single shared adapter at matched trainable capacity.

Reference graph

Works this paper leans on

175 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [23]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

    2023

  2. [88]

    Dora: Weight-decomposed low-rank adaptation, 2024

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024

    2024

  3. [1]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929 , 2020

    arXiv 2010

  4. [2]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018

    2018

  5. [3]

    Expert gate: Lifelong learning with a network of experts

    Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3366–3375, 2017

    2017

  6. [4]

    Task-free con- tinual learning

    Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free con- tinual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11254–11263, 2019

    2019

  7. [5]

    Beyond su- pervised continual learning: a review

    Benedikt Bagus, Alexander Gepperth, and Timoth´ ee Lesort. Beyond su- pervised continual learning: a review. arXiv preprint arXiv:2208.14307 , 2022

    arXiv 2022

  8. [6]

    Rainbow memory: Continual learning with a memory of diverse samples, 2021

    Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples, 2021

    2021

Show all 175 references
  1. [7]

    Stanley, Jeff Clune, and Nick Cheney

    Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn, 2020

    2020

  2. [8]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  3. [9]

    New insights on reducing abrupt rep- resentation change in online continual learning, 2022

    Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt rep- resentation change in online continual learning, 2022

    2022

  4. [10]

    Online fast adaptation and knowl- edge accumulation: a new approach to continual learning, 2021

    Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, and Laurent Charlin. Online fast adaptation and knowl- edge accumulation: a new approach to continual learning, 2021

    2021

  5. [11]

    Riemannian walk for incremental learning: Understanding forgetting and intransigence

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV) , pages 532–547, 2018. 42

    2018

  6. [12]

    Efficient lifelong learning with a-gem

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mo- hamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018

    2018 arXiv

  7. [13]

    Efficient lifelong learning with a-gem, 2019

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mo- hamed Elhoseiny. Efficient lifelong learning with a-gem, 2019

    2019

  8. [14]

    Dual low-rank adaptation for continual learning with pre-trained models, 2024

    Huancheng Chen, Jingtao Li, Nidham Gazagnadou, Weiming Zhuang, Chen Chen, and Lingjuan Lyu. Dual low-rank adaptation for continual learning with pre-trained models, 2024

    2024

  9. [15]

    Cat: Continual adapter tuning for aspect sentiment classification

    Qiangpu Chen, Jiahua Huang, Wushao Wen, Qingling Li, Rumin Zhang, and Jinghui Qin. Cat: Continual adapter tuning for aspect sentiment classification. Neurocomputing, 580:127423, 2024

    2024

  10. [16]

    Adaptformer: Adapting vision transformers for scalable visual recognition, 2022

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition, 2022

    2022

  11. [17]

    Semi-supervised and unsupervised deep visual learning: A survey

    Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, and Zeynep Akata. Semi-supervised and unsupervised deep visual learning: A survey. IEEE transactions on pattern analysis and machine intelligence , 2022

    2022

  12. [18]

    Lifelong machine learning , volume 1

    Zhiyuan Chen and Bing Liu. Lifelong machine learning , volume 1. Springer, 2018

    2018

  13. [19]

    Task arithmetic with loRA for continual learning

    Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Santosh Ghotkar. Task arithmetic with loRA for continual learning. In Workshop on Ad- vancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023) , 2023

    2023

  14. [20]

    Adaptive LoRA merging for efficient domain incremental learning

    Eric Nuertey Coleman, Luigi Quarantiello, Julio Hurtado, and Vincenzo Lomonaco. Adaptive LoRA merging for efficient domain incremental learning. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024

    2024

  15. [21]

    Flattening sharpness for dynamic gradient projection memory benefits continual learning, 2021

    Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng- Ann Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning, 2021

    2021

  16. [24]

    Bert: Pre-training of deep bidirectional transformers for language under- standing

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American 43 Chapter of the Association for Computational Linguistics: Human ...

    2019

  17. [25]

    Learning without memorizing, 2019

    Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing, 2019

    2019

  18. [26]

    Learning without memorizing

    Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5138–5146, 2019

    2019

  19. [27]

    Don’t forget, there is more than forgetting: new metrics for continual learning

    Natalia D´ ıaz-Rodr´ ıguez, Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don’t forget, there is more than forgetting: new metrics for continual learning. arXiv preprint arXiv:1810.13166 , 2018

    2018 arXiv

  20. [28]

    Neural logic machines

    Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. arXiv preprint arXiv:1904.11694 , 2019

    1904 arXiv

  21. [29]

    An image is worth 16x16 words: Transformers for image recog- nition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recog- nitio...

    2021

  22. [30]

    Podnet: Pooled outputs distillation for small-tasks incre- mental learning, 2020

    Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incre- mental learning, 2020

    2020

  23. [31]

    Hat-cl: A hard-attention-to-the-task pytorch library for continual learning, 2024

    Xiaotian Duan. Hat-cl: A hard-attention-to-the-task pytorch library for continual learning, 2024

    2024

  24. [32]

    Orthogonal gradient descent for continual learning, 2019

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning, 2019

    2019

  25. [33]

    Pathnet: Evo- lution channels gradient descent in super neural networks

    Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evo- lution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017

    2017 arXiv

  26. [34]

    The lottery ticket hypoth- esis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypoth- esis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  27. [35]

    On the effectiveness of parameter-efficient fine-tuning

    Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence , vol- ume 37, pages 12799–12807, 2023. 44

    2023

  28. [36]

    A unified continual learning framework with general parameter-efficient tuning, 2023

    Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning, 2023

    2023

  29. [37]

    Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning, 2024

    Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning, 2024

    2024

  30. [38]

    Continual learning via neural pruning, 2019

    Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning, 2019

    2019

  31. [39]

    Federated class-incremental learning with prototype guided transformer

    Haiyang Guo, Fei Zhu, Wenzhuo Liu, Xu-Yao Zhang, and Cheng-Lin Liu. Federated class-incremental learning with prototype guided transformer. arXiv preprint arXiv:2401.02094 , 2024

    2024 arXiv

  32. [40]

    On the domain adaptation and generalization of pretrained language models: A survey, 2022

    Xu Guo and Han Yu. On the domain adaptation and generalization of pretrained language models: A survey, 2022

    2022

  33. [41]

    La-maml: Look-ahead meta learning for continual learning, 2020

    Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: Look-ahead meta learning for continual learning, 2020

    2020

  34. [42]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 , 2021

    2021 arXiv

  35. [43]

    Towards a unified view of parameter-efficient transfer learning

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Towards a unified view of parameter-efficient transfer learning. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022

    2022

  36. [44]

    Mera: Merging pretrained adapters for few-shot learning, 2023

    Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Mera: Merging pretrained adapters for few-shot learning, 2023

    2023

  37. [45]

    Class- incremental learning with repetition, 2023

    Hamed Hemati, Andrea Cossu, Antonio Carta, Julio Hurtado, Lorenzo Pellegrini, Davide Bacciu, Vincenzo Lomonaco, and Damian Borth. Class- incremental learning with repetition, 2023

    2023

  38. [46]

    Distilling the knowledge in a neural network, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015

    2015

  39. [47]

    Learning a unified classifier incrementally via rebalancing

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 831–839, 2019

    2019

  40. [48]

    Learning a unified classifier incrementally via rebalancing

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 45

    2019

  41. [49]

    Parameter-efficient transfer learning for nlp, 2019

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019

    2019

  42. [50]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Syl- vain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th In-...

    2019

  43. [51]

    Universal language model fine- tuning for text classification, 2018

    Jeremy Howard and Sebastian Ruder. Universal language model fine- tuning for text classification, 2018

    2018

  44. [52]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  45. [53]

    Pop: Prompt of prompts for continual learning

    Zhiyuan Hu, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Pop: Prompt of prompts for continual learning. arXiv preprint arXiv:2306.08200, 2023

    2023 arXiv

  46. [54]

    Expand and merge: Continual learning with the guidance of fixed text embedding space

    Yujun Huang, Wentao Zhang, and Ruixuan Wang. Expand and merge: Continual learning with the guidance of fixed text embedding space. In 2024 International Joint Conference on Neural Networks (IJCNN) , pages 1–8, 2024

    2024

  47. [55]

    Memory population in con- tinual learning via outlier elimination

    Julio Hurtado, Alain Raymond-S´ aez, Vladimir Araujo, Vincenzo Lomonaco, Alvaro Soto, and Davide Bacciu. Memory population in con- tinual learning via outlier elimination. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3481–3490, 2023

    2023

  48. [56]

    Optimizing reusable knowledge for continual learning via metalearning, 2021

    Julio Hurtado, Alain Raymond-Saez, and Alvaro Soto. Optimizing reusable knowledge for continual learning via metalearning, 2021

    2021

  49. [57]

    Continual learning for predictive maintenance: Overview and challenges

    Julio Hurtado, Dario Salvati, Rudy Semola, Mattia Bosio, and Vincenzo Lomonaco. Continual learning for predictive maintenance: Overview and challenges. Intelligent Systems with Applications , page 200251, 2023

    2023

  50. [58]

    Editing models with task arithmetic, 2023

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Guru- rangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023

    2023

  51. [59]

    Meta-learning representations for continual learning, 2019

    Khurram Javed and Martha White. Meta-learning representations for continual learning, 2019

    2019

  52. [60]

    Helpful or harmful: Inter-task associa- tion in continual learning

    Hyundong Jin and Eunwoo Kim. Helpful or harmful: Inter-task associa- tion in continual learning. In Shai Avidan, Gabriel Brostow, Moustapha Ciss´ e, Giovanni Maria Farinella, and Tal Hassner, editors,Computer Vi- sion – ECCV 2022 , pages 519–535, Cham, 2022. Springer Nature Sw...

    2022

  53. [61]

    Meta-consolidation for con- tinual learning, 2020

    K J Joseph and Vineeth N Balasubramanian. Meta-consolidation for con- tinual learning, 2020

    2020

  54. [62]

    Con- tinual learning with node-importance based adaptive group sparse regu- larization, 2021

    Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Con- tinual learning with node-importance based adaptive group sparse regu- larization, 2021

    2021

  55. [63]

    Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D. Yoo. Forget-free continual learning with winning subnetworks. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and...

    2022

  56. [64]

    Class-Incremental Learn- ing by Knowledge Distillation with Adaptive Feature Consolidation

    Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class-Incremental Learn- ing by Knowledge Distillation with Adaptive Feature Consolidation. In CVPR, 2022

    2022

  57. [65]

    Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020

    2020

  58. [66]

    Achieving for- getting prevention and knowledge transfer in continual learning

    Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving for- getting prevention and knowledge transfer in continual learning. ArXiv, abs/2112.02706, 2021

    2021 arXiv

  59. [67]

    Introducing language guidance in prompt-based continual learning

    Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-based continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11...

    2023

  60. [68]

    On the stability-plasticity dilemma of class-incremental learning, 2023

    Dongwan Kim and Bohyung Han. On the stability-plasticity dilemma of class-incremental learning, 2023

    2023

  61. [69]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guil- laume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ra- malho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of s...

    2017

  62. [70]

    Rusu, Kieran Milan, John Quan, Tiago Ra- malho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guil- laume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ra- malho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forget- ...

    2017

  63. [71]

    Kopiczko, Tijmen Blankevoort, and Yuki M

    Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024

    2024

  64. [72]

    Hierarchical mo- tion understanding via motion programs

    Sumith Kulal, Jiayuan Mao, Alex Aiken, and Jiajun Wu. Hierarchical mo- tion understanding via motion programs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6568– 6576, 2021

    2021

  65. [73]

    Overcoming catas- trophic forgetting with unlabeled data in the wild

    Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catas- trophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 312– 321, 2019

    2019

  66. [74]

    Do pre-trained mod- els benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6485–6493, 2023

    Kuan-Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained mod- els benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6485–6493, 2023

    2023

  67. [75]

    Overcoming catastrophic forgetting by incremental moment matching

    Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems , 30, 2017

    2017

  68. [76]

    Symbolic replay: Scene graph as prompt for continual learning on vqa task

    Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1250–1259, 2023

    2023

  69. [77]

    The power of scale for parameter-efficient prompt tuning, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021

    2021

  70. [78]

    Atlas: Adapter- based multi-modal continual learning with a two-stage learning strategy, 2024

    Hong Li, Zhiquan Tan, Xingyu Li, and Weiran Huang. Atlas: Adapter- based multi-modal continual learning with a two-stage learning strategy, 2024

    2024

  71. [79]

    Prefix-tuning: Optimizing continuous prompts for generation, 2021

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021

    2021

  72. [80]

    Vb-lora: Extreme parameter efficient fine-tuning with vector banks, 2024

    Yang Li, Shaobo Han, and Shihao Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks, 2024

    2024

  73. [81]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE trans- actions on pattern analysis and machine intelligence , 40(12):2935–2947, 2017

    2017

  74. [82]

    Inflora: Interference-free low-rank adap- tation for continual learning, 2024

    Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adap- tation for continual learning, 2024

    2024

  75. [83]

    Trgp: Trust region gradient projection for continual learning, 2022

    Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning, 2022. 48

    2022

  76. [84]

    The clear bench- mark: Continual learning on real-world imagery

    Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear bench- mark: Continual learning on real-world imagery. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2), 2021

    2021

  77. [85]

    Lora-based continual learning with constraints on critical parameter changes, 2025

    Shimou Ling, Liang Zhang, Jiangwei Zhao, Lili Pan, and Hongliang Li. Lora-based continual learning with constraints on critical parameter changes, 2025

    2025

  78. [86]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    1950

  79. [87]

    Parameter-efficient fine-tuning for continual learn- ing: A neural tangent kernel perspective, 2025

    Jingren Liu, Zhong Ji, YunLong Yu, Jiale Cao, Yanwei Pang, Jungong Han, and Xuelong Li. Parameter-efficient fine-tuning for continual learn- ing: A neural tangent kernel perspective, 2025

    2025

  80. [89]

    Rotate your networks: Better weight consolidation and less catastrophic forgetting

    Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262–2268. IEEE, 2018

    2018

  81. [90]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, 2022

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, 2022

    2022

  82. [91]

    Gpt understands, too

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 5:208–215, 2024

    2024

  83. [92]

    Adaptive aggregation net- works for class-incremental learning

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation net- works for class-incremental learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, June 2021

    2021

  84. [93]

    Rmm: Reinforced memory management for class-incremental learning, 2023

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class-incremental learning, 2023

    2023

  85. [94]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  86. [95]

    Learning to describe scenes with programs

    Yunchao Liu and Zheng Wu. Learning to describe scenes with programs. In International conference on learning representations, 2019. 49

    2019

  87. [96]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  88. [97]

    Gradient episodic memory for continual learning, 2022

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning, 2022

    2022

  89. [98]

    Visual prompt tuning in null space for contin- ual learning

    Yue Lu, Shizhou Zhang, De Cheng, Yinghui Xing, Nannan Wang, Peng Wang, and Yanning Zhang. Visual prompt tuning in null space for contin- ual learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information ...

    2024

  90. [99]

    Univl: A unified video and lan- guage pre-training model for multimodal understanding and generation

    Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and lan- guage pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 , 2020

    2002 arXiv

  91. [100]

    Language semantic graph guided data-efficient learning

    Wenxuan Ma, Shuang Li, Jingxuan Kang, et al. Language semantic graph guided data-efficient learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  92. [101]

    Online continual learning in image classification: An em- pirical survey

    Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An em- pirical survey. Neurocomputing, 469:28–51, 2022

    2022

  93. [102]

    Piggyback: Adapting a single network to multiple tasks by learning to mask weights

    Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Pro- ceedings of the European conference on computer vision (ECCV) , pages 67–82, 2018

    2018

  94. [103]

    Packnet: Adding multiple tasks to a single network by iterative pruning, 2018

    Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning, 2018

    2018

  95. [104]

    Deepproblog: Neural probabilistic logic pro- gramming

    Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas De- meester, and Luc De Raedt. Deepproblog: Neural probabilistic logic pro- gramming. Advances in neural information processing systems , 31, 2018

    2018

  96. [105]

    Neuro-symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal

    Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso. Neuro-symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal. arXiv preprint arXiv:2302.01242, 2023

    2023 arXiv

  97. [106]

    Magmax: Leveraging model merging for seamless continual learning, 2024

    Daniel Marczak, Bart lomiej Twardowski, Tomasz Trzci´ nski, and Sebas- tian Cygert. Magmax: Leveraging model merging for seamless continual learning, 2024. 50

    2024

  98. [107]

    Krisp: Integrating implicit and symbolic knowledge for open- domain knowledge-based vqa

    Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open- domain knowledge-based vqa. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 14111–14121, 2021

    2021

  99. [108]

    Catastrophic interference in con- nectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in con- nectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109–165. Elsevier, 1989

    1989

  100. [109]

    Byol for audio: Exploring pre-trained general-purpose audio representations

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Exploring pre-trained general-purpose audio representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:137–151, 2022

    2022

  101. [110]

    Continual lifelong learning with neural networks: A review

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019

    2019

  102. [111]

    Miller, and Sebastian Riedel

    Fabio Petroni, Tim Rockt¨ aschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases?, 2019

    2019

  103. [112]

    Adapterfusion: Non-destructive task composition for transfer learning, 2021

    Jonas Pfeiffer, Aishwarya Kamath, Andreas R¨ uckl´ e, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning, 2021

    2021

  104. [113]

    MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

    Jonas Pfeiffer, Ivan Vuli´ c, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceed- ings of the 2020 Conference on Empirical Methods in Natural Lang...

    2020

  105. [114]

    Online continual learning without the storage constraint

    Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXiv preprint arXiv:2305.09253 , 2023

    2023 arXiv

  106. [115]

    Adaptive adapter routing for long-tailed class-incremental learning, 2024

    Zhi-Hong Qi, Da-Wei Zhou, Yiran Yao, Han-Jia Ye, and De-Chuan Zhan. Adaptive adapter routing for long-tailed class-incremental learning, 2024

    2024

  107. [116]

    Learn more, but bother less: parameter efficient continual learning

    Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  108. [117]

    Prompt gradient projection for continual learning

    Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, Yuan Xie, et al. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations . 51

  109. [118]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Explor- ing the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  110. [119]

    itaml: An incremental task-agnostic meta- learning approach, 2020

    Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta- learning approach, 2020

    2020

  111. [120]

    Encoder based lifelong learning

    Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuyte- laars. Encoder based lifelong learning. In Proceedings of the IEEE inter- national conference on computer vision , pages 1320–1328, 2017

    2017

  112. [121]

    Progressive prompts: Continual learning for language models

    Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In International Conference on Learning Represen- tations, 2023

    2023

  113. [122]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017

    2001

  114. [123]

    An- alyzing and reducing catastrophic forgetting in parameter efficient tuning, 2024

    Weijieying Ren, Xinlong Li, Lei Wang, Tianxiang Zhao, and Wei Qin. An- alyzing and reducing catastrophic forgetting in parameter efficient tuning, 2024

    2024

  115. [124]

    Learning to learn without forgetting by maximizing transfer and minimizing interference, 2019

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference, 2019

    2019

  116. [125]

    Progressive neural networks

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Had- sell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016

    2016 arXiv

  117. [126]

    Gradient projection memory for continual learning

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Represen- tations

  118. [127]

    Gradient projection memory for continual learning, 2021

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning, 2021

    2021

  119. [128]

    Dis- tilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Dis- tilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019

    1910 arXiv

  120. [129]

    Are emergent abil- ities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abil- ities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024. 52

    2024

  121. [130]

    Overcoming catastrophic forgetting with hard attention to the task

    Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning , pages 4548–4557. PMLR, 2018

    2018

  122. [131]

    A unified approach to domain incremental learning with memory: Theory and algorithm, 2023

    Haizhou Shi and Hao Wang. A unified approach to domain incremental learning with memory: Theory and algorithm, 2023

    2023

  123. [132]

    Continual diffusion with stamina: Stack-and-mask incremental adapters

    James Seale Smith, Yen-Chang Hsu, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion with stamina: Stack-and-mask incremental adapters. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1744–1754, 2023

    2024

  124. [133]

    Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IE...

    2023

  125. [134]

    Zero-shot learning through cross-modal transfer

    Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013

    2013

  126. [135]

    A compre- hensive empirical evaluation on online continual learning

    Albin Soutif-Cormerais, Antonio Carta, Andrea Cossu, Julio Hurtado, Vincenzo Lomonaco, Joost Van de Weijer, and Hamed Hemati. A compre- hensive empirical evaluation on online continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages...

    2023

  127. [136]

    Steitz and Stefan Roth

    Jan-Martin O. Steitz and Stefan Roth. Adapters strike back, 2024

    2024

  128. [137]

    Energy and policy considerations for deep learning in nlp, 2019

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp, 2019

    2019

  129. [138]

    Merging models on the fly without retraining: A sequential approach to scalable continual model merging, 2025

    Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging, 2025

    2025

  130. [139]

    Layerwise optimization by gradient decomposition for continual learning, 2021

    Shixiang Tang, Dapeng Chen, Jinguo Zhu, Shijie Yu, and Wanli Ouyang. Layerwise optimization by gradient decomposition for continual learning, 2021

    2021

  131. [140]

    Three scenarios for continual learning

    Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 , 2019

    1904 arXiv

  132. [141]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. 53

    2023

  133. [142]

    Efficient con- tinual learning with modular networks and task-driven priors, 2021

    Tom Veniat, Ludovic Denoyer, and Marc’Aurelio Ranzato. Efficient con- tinual learning with modular networks and task-driven priors, 2021

    2021

  134. [143]

    Continual learning: Appli- cations and the road forward

    Eli Verwimp, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke H¨ ullermeier, Christopher Kanan, Dhiree- sha Kudithipudi, Christoph H Lampert, et al. Continual learning: Appli- cations and the road forward. arXiv preprint arXiv:2311.11908 , 2023

    2023 arXiv

  135. [144]

    Clad: A realistic continual learning benchmark for autonomous driving

    Eli Verwimp, Kuo Yang, Sarah Parisot, Lanqing Hong, Steven McDonagh, Eduardo P´ erez-Pellitero, Matthias De Lange, and Tinne Tuytelaars. Clad: A realistic continual learning benchmark for autonomous driving. Neural Networks, 161:659–669, 2023

    2023

  136. [145]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018

    2018 arXiv

  137. [146]

    Self-expansion of pre-trained models with mixture of adapters for continual learning, 2024

    Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. Self-expansion of pre-trained models with mixture of adapters for continual learning, 2024

    2024

  138. [147]

    A comprehensive survey of continual learning: Theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  139. [148]

    Orthogonal subspace learning for language model continual learning, 2023

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning, 2023

    2023

  140. [149]

    S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning

    Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. Advances in Neural Information Processing Systems , 35:5682– 5695, 2022

    2022

  141. [150]

    Adamix: Mixture-of- adaptations for parameter-efficient model tuning, 2022

    Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of- adaptations for parameter-efficient model tuning, 2022

    2022

  142. [151]

    Multitask prompt tuning enables parameter-efficient transfer learning, 2023

    Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multitask prompt tuning enables parameter-efficient transfer learning, 2023

    2023

  143. [152]

    Dualprompt: Complementary prompting for rehearsal-free contin- ual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free contin- ual learning. In European Conference on Computer Vision, pages 631–648. Sprin...

    2022

  144. [153]

    Learn- ing to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learn- ing to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 139–149, 2022

    2022

  145. [154]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Met- zler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  146. [155]

    Mixture of lora experts, 2024

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts, 2024

    2024

  147. [156]

    Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning, 2025

    Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning, 2025

    2025

  148. [157]

    Large scale incremental learning, 2019

    Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yan- dong Guo, and Yun Fu. Large scale incremental learning, 2019

    2019

  149. [158]

    Parameter-efficient fine-tuning for pre-trained vision models: A survey, 2024

    Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey, 2024

    2024

  150. [159]

    Ties-merging: Resolving interference when merging models, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models, 2023

    2023

  151. [160]

    Representation surgery for multi-task model merging, 2024

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xing- wei Wang, and Dacheng Tao. Representation surgery for multi-task model merging, 2024

    2024

  152. [161]

    Neural-symbolic vqa: Disentangling reasoning from vision and language understanding

    Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information pro- cessing systems, 31, 2018

    2018

  153. [162]

    Boosting continual learning of vision-language models via mixture-of-experts adapters, 2024

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters, 2024

    2024

  154. [163]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch, 2024

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch, 2024

    2024

  155. [164]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Jun- suk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023– 6032, 2019. 55

    2019

  156. [165]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models. arXiv preprint arXiv:2106.10199 , 2021

    2021

  157. [166]

    Continual learn- ing of context-dependent processing in neural networks

    Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learn- ing of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364–372, August 2019

    2019

  158. [167]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR, 2017

    2017

  159. [168]

    Adapter learning in pretrained feature extractor for continual learning of diseases, 2023

    Wentao Zhang, Yujun Huang, Tong Zhang, Qingsong Zou, Wei-Shi Zheng, and Ruixuan Wang. Adapter learning in pretrained feature extractor for continual learning of diseases, 2023

    2023

  160. [169]

    Vqacl: A novel visual question answering continual learning setting

    Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqacl: A novel visual question answering continual learning setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19102–19112, 2023

    2023

  161. [170]

    Mask- ing as an efficient alternative to finetuning for pretrained language models

    Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Sch¨ utze. Mask- ing as an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004.12406 , 2020

    2004 arXiv

  162. [171]

    Revisiting class-incremental learning with pre-trained models: Gener- alizability and adaptivity are all you need

    Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Gener- alizability and adaptivity are all you need. Int. J. Comput. Vision , 133(3):1012–1032, August 2024

    2024

  163. [172]

    Continual learning with pre-trained models: A survey

    Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey. In Kate Lar- son, editor, Proceedings of the Thirty-Third International Joint Confer- ence on Artificial Intelligence, IJCAI-24 , pages 8363–8371. Inter...

    2024

  164. [173]

    Ex- pandable subspace ensemble for pre-trained model-based class-incremental learning, 2024

    Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Ex- pandable subspace ensemble for pre-trained model-based class-incremental learning, 2024

    2024

  165. [174]

    Deep class-incremental learning: A survey

    Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incremental learning: A survey. arXiv preprint arXiv:2302.03648, 2023

    2023 arXiv

  166. [175]

    Co-transport for class- incremental learning

    Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Co-transport for class- incremental learning. In Proceedings of the 29th ACM International Con- ference on Multimedia, pages 1645–1654, 2021. 56

    2021

  167. [176]

    A comprehensive survey on trans- fer learning, 2020

    Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on trans- fer learning, 2020. 57

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.