Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a vision-language model can be taught new classification tasks one after another and come out better at recognizing domains it has never seen, by splitting the job between a task-invariant contrastive adapter and a…

desk verdict AFA is a solid, practical MTIL framework, but the headline 'beyond CLIP' claim is an averaging artifact: restrict both sides to the same datasets and AFA (≈68.4) sits below CLIP (69.4). read the letter →

arxiv 2505.07690 v1 pith:ODAVCNAF submitted 2025-05-12 cs.CV

Songlin Dong , Chenhao Ding , Jiangyang Li , Jizhou Han , Qiang Wang , Yuhang He , Yihong Gong This is my paper · ORCID
classification cs.CV
keywords continuallearningvision-languagemodelszero-shottransfercatastrophicforgettingmixture-of-expertsadaptersLoRAmulti-domaintaskincrementalparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes aim at a specific failure of continual learning for vision-language models: when a model like CLIP is trained on a stream of new classification tasks, its pre-trained zero-shot ability erodes. Previous fixes handed unseen-domain test samples back to the untouched original CLIP, which stops the bleeding but cannot improve generalization. The paper proposes AFA, a two-track adapter framework in which a task-shared adapter trained with contrastive learning accumulates task-invariant knowledge to push zero-shot transfer above the original CLIP's score, while a mixture-of-experts adapter with multiple low-rank heads preserves each learned task. On the standard 11-dataset MTIL benchmark, AFA reports a transfer score of 70.3%, above the 69.4% zero-shot baseline, while retaining 87.2% accuracy on the last task.

What carries the argument

The load-bearing mechanism is a pair of adapters mounted on a frozen CLIP backbone with a router between them. AFFA (against-forward-forgetting adapter) is one LoRA module shared across every task, trained on each incoming dataset with an image-text contrastive loss; its job is to accumulate task-invariant knowledge so that unseen-domain samples become easier, not harder, to recognize. ABFA (against-backward-forgetting adapter) is a dynamically expanding mixture-of-experts: a shared pool of experts, each a multi-head LoRA with one A matrix and several B matrices to limit interference among sub-concepts within a task, plus a task-specific router that is frozen once each task ends to stop forgetting. A parameter-free domain distribution selector compares a test sample's cosine similarity to K-means prototypes of each learned task and, if the top score falls below a threshold (0.75), sends the sample to AFFA, otherwise to the matching ABFA router.

What would settle it

Run the MTIL sequence again with the domain-selection threshold fixed at a value chosen without ever looking at the test set (for example, cross-validated on a held-out seen-domain split) and with the 11 datasets reordered so that visually similar domains are adjacent; if the Transfer score in either variant falls back below CLIP's 69.4% zero-shot reference, the claim that AFA enhances rather than merely routes around zero-shot generalization is undercut.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that forward forgetting is not merely preventable but reversible: knowledge gathered during incremental learning can be turned into an asset for zero-shot generalization. The AFFA module, a LoRA adapter shared across all tasks and fine-tuned with an image-text contrastive loss on every sequential dataset, extracts task-invariant visual-linguistic structure, and at inference an unseen-domain sample is routed to this adapter rather than to the frozen CLIP. The result is a Transfer score of 70.3% on the 11-dataset MTIL benchmark, surpassing frozen CLIP's 69.4% zero-shot transfer by 0.9%, a first among MTIL methods. The companion ABFA module, a sparsely activated mixture of multi-head LoRA experts with progressively expanding task routers, carries the incremental side of the workload, reaching a Last accuracy of 87.2%, and the same benefits appear in 5-shot MTIL, class-incremental, and domain-incremental settings.

Load-bearing premise

The routing threshold that decides whether a test image comes from a seen or an unseen domain was chosen by tuning on the benchmark's test set, and the whole transfer claim depends on that threshold reliably sending genuinely unseen samples to the forward-forgetting adapter rather than accidentally to a task-specific expert.

Editorial extensions

If this is right

  • Each newly learned task can improve zero-shot transfer to unseen datasets instead of merely preserving it, directly contradicting the prior assumption that continual learning and generalization must trade off.
  • In the 5-shot regime the multi-head LoRA experts buy their largest gains on fine-grained and complex datasets such as Aircraft, DTD, and SUN397, making the framework a candidate for data-scarce continual learning.
  • The reported state-of-the-art scores come with lower training cost than the closest competitor: 30.1 million trainable parameters against BCL's 59.8 million, with 19.21% lower GPU memory.
  • The same two-adapter split carries over to class-incremental learning on CIFAR100 and TinyImageNet and to domain-incremental learning on DomainNet and CORe50, suggesting the mechanism is not tied to the MTIL protocol.
  • Ablations show the pair is complementary: a single plain adapter drops transfer by 24.4 points, ABFA alone loses only 0.1 point, and only the combination exceeds CLIP's zero-shot reference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper does not run is to plot the Transfer score after each of the 11 tasks: if AFFA genuinely accumulates task-invariant knowledge, transfer should rise monotonically with task count, whereas a flat trajectory would suggest the gain is mostly benign regularization.
  • Since the routing threshold (0.75) and prototype count (5) were tuned with the benchmark test sets in view, re-running the protocol with those hyperparameters fixed on seen-domain validation data alone would show whether the 0.9-point transfer gain survives a stricter protocol.
  • The multi-head LoRA design is motivated by interference among implicit sub-concepts; measuring whether individual B heads specialize on identifiable sub-classes of fine-grained datasets would test that mechanism directly and predict where few-shot gains concentrate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes AFA (Against Forward and Backward Forgetting Adapter), a continual learning framework for vision-language models (VLMs) under multi-domain task incremental learning (MTIL). AFA combines two modules: AFFA, a task-shared LoRA adapter trained with contrastive loss to preserve and enhance zero-shot transfer, and ABFA, a mixture-of-experts adapter with multiple low-rank B matrices per expert to mitigate backward forgetting and improve few-shot learning. A domain distribution selector (DDS) routes test samples either to task-specific ABFA experts or to the AFFA path based on cosine similarity to task prototypes. Experiments are reported on MTIL, few-shot MTIL, CIL, and DIL benchmarks, with the central claim that AFA's Transfer score (70.3%) exceeds the frozen CLIP zero-shot average (69.4%).

Significance. If the claims were fully supported, the paper would make a meaningful contribution to continual learning of VLMs, particularly the idea of learning task-invariant information to improve zero-shot generalization and the multi-head LoRA MoE design for few-shot incremental learning. The reported improvements over prior MTIL methods on the Last metric (e.g., 87.2% vs. 85.0% for BCL in Table 1) are noteworthy, and the paper includes supplementary code and detailed ablations. However, the central novelty—exceeding CLIP's zero-shot capability—is currently not supported due to an inconsistency in the evaluation protocol. The method still warrants consideration, but the headline claim must be corrected and the experimental evidence strengthened.

major comments (3)
  1. [§4.2, Table 1 and Table 2, Appendix Table 8] The claim that AFA's Transfer (70.3%) surpasses frozen CLIP's zero-shot (69.4%) is based on an invalid comparison of different dataset sets. The footnote states that Aircraft is excluded when evaluating CLIP's transfer capability, but AFA's Transfer average includes Aircraft (88.6% in Table 1). Excluding Aircraft from AFA's Transfer values yields approximately 68.4% on the shared 10 datasets, which is below CLIP's 69.4% on the same 10 datasets. The same arithmetic inconsistency affects Table 2 (70.2% includes Aircraft 88.5%; the common-10 average is about 68.3%) and the Order-II results in Table 8. Consequently, the paper's headline conclusion that AFA 'surpasses the inherent zero-shot performance of CLIP' is unsupported by the reported data and must be corrected or withdrawn.
  2. [§4.3 and Appendix §9.4, Fig. 3 and Fig. 4] Key hyperparameters of the proposed method are tuned directly on the benchmark test set, and the paper reports no multiple-seed runs or error bars. Specifically, the DDS threshold (0.75), the number of prototypes K (5), the number of experts (22), the top-k activated experts (2), and the LoRA rank (16) are all selected based on test-set performance curves (Fig. 3 and Appendix Fig. 4). Since the claimed margins over baselines are small (e.g., 1.8–2.2% in Average, 0.9% over CLIP), the absence of variance estimates makes it impossible to assess whether the reported improvements are statistically meaningful. The authors should provide results over multiple seeds and clarify which choices were made before or after seeing test-set performance.
  3. [§3.5, Table 11] The Transfer metric depends critically on the DDS routing threshold: unseen-domain samples must be correctly sent to the AFFA path for the transfer score to reflect genuine zero-shot generalization. The paper reports high task-selection accuracy (Table 11), but this accuracy is measured on the same benchmark used to tune the threshold, and the threshold selection in Appendix Fig. 4(b) is based on the MTIL test set. Without an evaluation on a held-out task sequence or a principled threshold-selection procedure, it is unclear whether the reported transfer ability is robust or an artifact of test-set tuning. This concern is especially relevant because the transfer gains are small in magnitude.
minor comments (3)
  1. [Abstract and §1] There are typographical errors and redundancies: 'that that strengthens' appears in the abstract, and 'the final output of the AFFA module' in Eq. (10) should refer to the ABFA module.
  2. [Table 1, Table 2, and Appendix Table 8] The table formatting in the submitted text is inconsistent: several rows appear to have 11 numbers where 12 columns are expected (e.g., the AFA Transfer row in Table 1), making it difficult to verify the reported averages. Please ensure the final version presents all dataset-specific values and the average unambiguously.
  3. [§4.1, Evaluation metrics] The definition of the 'Average' metric is described as the mean of 'Transfer' and 'Last', but Table 1 reports 'Avg.' values that appear to be per-dataset averages rather than a simple mean of the two metrics. Please clarify the computation and state the formula explicitly.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'beyond CLIP' Transfer claim is an artifact of averaging different dataset sets: AFA's 70.3% includes the trained Aircraft task, while CLIP's 69.4% excludes it; on the shared 10 datasets AFA is 68.4%.

  1. fitted input called prediction [Section 4.1 (Evaluation metrics), Section 4.2 (Table 1 and footnote), Appendix Table 9 (Order-I sequence)]
    "The ‘Transfer’ metric evaluates the model’s zero-shot transfer capability on unseen data... To ensure a fair comparison, we exclude Aircraft when evaluating frozen CLIP’s transfer capability. ... AFA (Ours) × 88.6 67.9 45.7 54.8 71.3 88.5 64.4 89.7 64.7 67.0 70.3 ... AFA’s Transfer performance even exceeds CLIP’s zero-shot capability (69.4%, an improvement of 0.9%)"

    The 0.9% “improvement” is forced by the evaluation sets, not by the model. CLIP’s 69.4% excludes Aircraft (24.3), while AFA’s 70.3% includes Aircraft (88.6); Aircraft is the first task of Order-I, so AFFA was trained on it. Recomputing AFA’s Transfer over the same ten non-Aircraft datasets gives 684.3/10 = 68.4%, below CLIP’s 69.4%. Thus a trained-dataset accuracy is counted in a metric defined as “zero-shot transfer capability on unseen data,” and the headline comparison reduces to an arithmetic artifact of including a fitted task in AFA’s average but excluding it from the CLIP baseline.

full rationale

Most of AFA’s derivation is self-contained and not circular. AFFA is a LoRA adapter trained on each task with a contrastive loss (Eqs. 1–5); ABFA is a multi-head LoRA MoE with task routers (Eqs. 6–10); the DDS routing rule is defined from K-Means prototypes and a score threshold. These components are not defined in terms of the target Transfer numbers, and the CIL/DIL/Last/Average results are external benchmark comparisons with independent content. The one load-bearing circular/forced step is in the headline Transfer comparison. The paper defines Transfer as “zero-shot transfer capability on unseen data,” but its Table 1 AFA row averages 11 datasets including Aircraft at 88.6, while the CLIP zero-shot baseline is averaged over 10 datasets after the footnote “we exclude Aircraft when evaluating frozen CLIP’s transfer capability.” Summing AFA’s other ten entries gives 68.4%, below CLIP’s 69.4%. Hence the claimed 0.9% “beyond CLIP” gain is not a measured property of the model; it is produced by including a trained dataset in the numerator and excluding it from the denominator. This is a fitted/trained input relabeled as a zero-shot prediction. No load-bearing self-citation chain is present; the authors’ prior works appear only in related work and ablations. The threshold of 0.75 for the domain selector is tuned on the benchmark test set (Appendix Fig. 4b), which is a validation leak but not a definitional circularity. Taking everything together, the central “surpasses CLIP” claim is partially forced by the averaging construction, while the method’s incremental-learning results remain independent.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. All free parameters are hyperparameters tuned on the benchmark, which is the main source of selection bias. The assumptions listed are the load-bearing premises for transfer improvement and routing behavior.

free parameters (6)
  • DDS threshold = 0.75
    Chosen via accuracy sweep on the MTIL benchmark test set (Appendix Fig. 4b) to classify samples as seen or unseen; directly controls the transfer versus seen-task routing.
  • Number of prototypes K (DDS) = 5
    Selected from KNN-N selection accuracy sweep (Appendix Fig. 4a); higher K saturates near 5.
  • Number of experts N_E = 22
    Set to 2T based on ablation in Fig. 3a; performance plateaus beyond 2T.
  • Top-k activated experts = 2
    Peak on ablation Fig. 3b; larger k degrades due to task interference.
  • LoRA rank (AFFA and ABFA) = 16
    Best transfer in Table 13 sweep over ranks 1, 2, 4, 8, 16, 32.
  • Number of B matrices per expert M = 4
    Best Last accuracy in Table 6 (1A/4B); larger M (6B) slightly worse.
assumptions (4)
  • domain assumption Contrastive training on each task's dataset with CLIP text labels improves zero-shot accuracy on future unseen datasets.
    The entire AFFA design depends on this transfer premise (Section 3.3). It is not proven and is precisely the claim tested by the Transfer metric.
  • domain assumption K-Means on frozen CLIP features produces stable, separable prototypes for each task's domain.
    DDS routing (Section 3.5) relies on this; no analysis of prototype stability is given.
  • domain assumption Frozen CLIP plus task-shared AFFA and task-specific ABFA do not interfere during training because they are trained alternately with the other frozen.
    Section 3.5 asserts mutual freezing avoids interference; no direct empirical verification beyond final metrics.
  • standard math CLIP contrastive loss (Eq. 3-5) is a valid objective for aligning image and text features.
    Background from CLIP [30] and standard contrastive learning; not original to this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models." pith.science (2026). https://pith.science/paper/ODAVCNAF

@misc{pith2026250507690,
  author       = {Pith},
  title        = {Pith review of: Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODAVCNAF}},
  note         = {Machine review of arXiv:2505.07690}
}
read the original abstract

This study aims to address the problem of multi-domain task incremental learning~(MTIL), which requires that vision-language models~(VLMs) continuously acquire new knowledge while maintaining their inherent zero-shot recognition capability. Existing paradigms delegate the testing of unseen-domain samples to the original CLIP, which only prevents the degradation of the model's zero-shot capability but fails to enhance the generalization of the VLM further. To this end, we propose a novel MTIL framework, named AFA, which comprises two core modules: (1) an against forward-forgetting adapter that learns task-invariant information for each dataset in the incremental tasks to enhance the zero-shot recognition ability of VLMs; (2) an against backward-forgetting adapter that strengthens the few-shot learning capability of VLMs while supporting incremental learning. Extensive experiments demonstrate that the AFA method significantly outperforms existing state-of-the-art approaches, especially in few-shot MTIL tasks, and surpasses the inherent zero-shot performance of CLIP in terms of transferability. The code is provided in the Supplementary Material.

Figures

Figures reproduced from arXiv: 2505.07690 by the authors.

Figure 1
Figure 1. Comparison between existing work and our approach:(a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of AFA method. (a) AFFA module (left) is trained on D t using Laffa to boost the generalization of VLMs. (b) the AFBA module (right) expands a task-specific router h t at stage t to train D t while freezing h 1 , . . . , ht−1 to prevent catastrophic forgetting. In addition, the expert component comprises multi-head LoRA, designed to enhance the model’s few-shot learning ability. different classes in the fea… view at source ↗
Figure 3
Figure 3. Impact of numbers of experts and top-k activated ex [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Impact of different Numbers of KNN-N and the thresh [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BPG: Balancing Plasticity and Generalization for Domain Incremental Learning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    BPG sizes per-domain adapters inversely to a feature-separability score and replaces hard domain selection with confidence-weighted logit fusion, setting state-of-the-art accuracy and near-zero forgetting on three dom...

Reference graph

Works this paper leans on

60 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and et al. Memory aware synapses: Learning what (not) to forget. InECCV, pages 139–154,

    work page

  2. [2]

    Dark experience for general continual learning: a strong, simple baseline.NIPS, 33:15920–15930, 2020

    Pietro Buzzega, Matteo Boschini, Angelo Porrello, and et al. Dark experience for general continual learning: a strong, simple baseline.NIPS, 33:15920–15930, 2020. 2

    work page 2020

  3. [3]

    Tem- adapter: Adapting image-text pretraining for video question answer

    Guangyi Chen, Xiao Liu, Guangrun Wang, Kun Zhang, Philip HS Torr, Xiao-Ping Zhang, and Yansong Tang. Tem- adapter: Adapting image-text pretraining for video question answer. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 13945–13955, 2023. 2

    work page 2023

  4. [4]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PmLR, 2020. 3

    2020

  5. [5]

    Adamv-moe: Adaptive multi-task vision mixture-of- experts

    Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamv-moe: Adaptive multi-task vision mixture-of- experts. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 17346–17357, 2023. 2, 4

    work page 2023

  6. [6]

    A continual learning survey: Defying for- getting in classification tasks.IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale ˇs Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying for- getting in classification tasks.IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021. 1

    2021

  7. [7]

    Learning without mem- orizing

    Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without mem- orizing. InCVPR, pages 5138–5146, 2019. 2, 5, 6, 7, 1

    work page 2019

  8. [8]

    Few-shot class- incremental learning via relation knowledge distillation

    Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong. Few-shot class- incremental learning via relation knowledge distillation. In AAAI, pages 1255–1263, 2021. 2

    work page 2021

Show all 60 references
  1. [9]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, and N. Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. 2020. 5, 6

    2020

  2. [10]

    Dytox: Transformers for continual learn- ing with dynamic token expansion

    Arthur Douillard, Alexandre Ram ´e, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learn- ing with dynamic token expansion. InCVPR, pages 9285– 9295, 2022. 2, 3, 7, 1

    2022

  3. [11]

    Clip-adapter: Better vision-language models with feature adapters.International Journal of Computer Vision, 132(2): 581–595, 2024

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters.International Journal of Computer Vision, 132(2): 581–595, 2024. 2

    2024

  4. [12]

    Dkt: Diverse knowledge transfer transformer for class incremental learning

    Xinyuan Gao, Yuhang He, Songlin Dong, Jie Cheng, Xing Wei, and Yihong Gong. Dkt: Diverse knowledge transfer transformer for class incremental learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24236–24245, 2023. 2

    2023

  5. [13]

    Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning

    Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning. InEuropean Conference on Computer Vision, pages 89–106. Springer,

  6. [14]

    Knowit vqa: Answering knowledge-based ques- tions about videos

    Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit vqa: Answering knowledge-based ques- tions about videos. InProceedings of the AAAI conference on artificial intelligence, pages 10826–10834, 2020. 2

    2020

  7. [15]

    Learning a unified classifier incrementally via rebalancing

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. InCVPR, pages 831–839, 2019. 6

    2019

  8. [16]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022. 1, 2, 3

    2022

  9. [17]

    Scaling up visual and vision-language representation learning with noisy text su- pervision

    Chao Jia, Yinfei Yang, Ye Xia, et al. Scaling up visual and vision-language representation learning with noisy text su- pervision. InInternational conference on machine learning, pages 4904–4916. PMLR, 2021. 2

    2021

  10. [18]

    Overcoming catastrophic forgetting in neu- ral networks.National Academy of Sciences, 114(13):3521– 3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks.National Academy of Sciences, 114(13):3521...

    2017

  11. [19]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021. 2, 3

    2021 arXiv

  12. [20]

    Enhancing visual document understanding with con- trastive learning in large visual-language models

    Xin Li, Yunfei Wu, Xinghua Jiang, Zhihao Guo, Mingming Gong, Haoyu Cao, Yinsong Liu, Deqiang Jiang, and Xing Sun. Enhancing visual document understanding with con- trastive learning in large visual-language models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision ...

    2024

  13. [21]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 40(12):2935–2947, 2017. 2, 5, 6, 7, 8, 1

    2017

  14. [22]

    Inflora: Interference-free low-rank adaptation for continual learning

    Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024. 3

    2024

  15. [23]

    Rotate your networks: Better weight consolidation and less catastrophic forgetting

    Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Wei- jer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. InInternational Conference on Pattern Recogni- tion, pages 2262–2268. IEEE, 2018. 2

    2018

  16. [24]

    Model behavior preserving for class-incremental learning.IEEE Transactions on Neural Networks and Learning Systems, 2022

    Yu Liu, Xiaopeng Hong, Xiaoyu Tao, Songlin Dong, Jin- gang Shi, and Yihong Gong. Model behavior preserving for class-incremental learning.IEEE Transactions on Neural Networks and Learning Systems, 2022. 2

    2022

  17. [25]

    Core50: a new dataset and benchmark for continuous object recognition

    Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17–26. PMLR, 2017. 6 9

    2017

  18. [26]

    Packnet: Adding mul- tiple tasks to a single network by iterative pruning

    Arun Mallya and Svetlana Lazebnik. Packnet: Adding mul- tiple tasks to a single network by iterative pruning. InCVPR, pages 7765–7773, 2018. 2

    2018

  19. [27]

    Piggy- back: Adapting a single network to multiple tasks by learn- ing to mask weights

    Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggy- back: Adapting a single network to multiple tasks by learn- ing to mask weights. InECCV, pages 67–82, 2018. 2

    2018

  20. [28]

    Class-incremental learning: survey and performance evalu- ation on image classification.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513–5533, 2022

    Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evalu- ation on image classification.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513–5533, 2022. 1

    2022

  21. [29]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 1406–1415,

  22. [30]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  23. [31]

    Learning multiple visual domains with residual adapters.Ad- vances in neural information processing systems, 30, 2017

    Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters.Ad- vances in neural information processing systems, 30, 2017. 2, 3

    2017

  24. [32]

    icarl: Incremental classi- fier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classi- fier and representation learning. InCVPR, pages 2001–2010,

    2001

  25. [33]

    Experience replay for continual learning.NIPS, 32, 2019

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lil- licrap, and Gregory Wayne. Experience replay for continual learning.NIPS, 32, 2019. 2

    2019

  26. [34]

    Progress & compress: A scalable framework for continual learning

    Jonathan Schwarz, Wojciech Czarnecki, and et al. Progress & compress: A scalable framework for continual learning. InICML, pages 4528–4537. PMLR, 2018. 2

    2018

  27. [35]

    Overcom- ing catastrophic forgetting with hard attention to the task

    Joan Serra, Didac Suris, Marius Miron, and et al. Overcom- ing catastrophic forgetting with hard attention to the task. In ICML, pages 4548–4557. PMLR, 2018. 2

    2018

  28. [36]

    Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer.arXiv preprint arXiv:1701.06538, 2017. 2, 4

    2017 arXiv

  29. [37]

    Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning. InProceedings of the IE...

    2023

  30. [38]

    Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models

    Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Han- tao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. In European Conference on Computer Vision, page...

    2024

  31. [39]

    Few-shot class- incremental learning

    Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class- incremental learning. InCVPR, pages 12183–12192, 2020. 2

    2020

  32. [40]

    Hydralora: An asymmetric lora architecture for efficient fine-tuning.Advances in Neural Information Pro- cessing Systems, 37:9565–9584, 2025

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng- Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning.Advances in Neural Information Pro- cessing Systems, 37:9565–9584, 2025. 2, 4

    2025

  33. [41]

    Three types of incremental learning.Nature Machine Intelligence, 4(12):1185–1197, 2022

    Gido M Van de Ven, Tinne Tuytelaars, and Andreas S To- lias. Three types of incremental learning.Nature Machine Intelligence, 4(12):1185–1197, 2022. 1

    2022

  34. [42]

    Foster: Feature boosting and compression for class- incremental learning

    Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class- incremental learning. InECCV, pages 398–414. Springer,

  35. [43]

    Non-exemplar domain incremental learning via cross-domain concept integration

    Qiang Wang, Yuhang He, Songlin Dong, Xinyuan Gao, Shaokun Wang, and Yihong Gong. Non-exemplar domain incremental learning via cross-domain concept integration. InEuropean Conference on Computer Vision, pages 144–

  36. [44]

    Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix

    Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix. InIn- ternational Conference on Machine Learning, pages 22680– 22690. PMLR, 2022. 2

    2022

  37. [45]

    S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning.Advances in Neural Informa- tion Processing Systems, 35:5682–5695, 2022

    Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning.Advances in Neural Informa- tion Processing Systems, 35:5682–5695, 2022. 3, 5, 6, 7, 1

    2022

  38. [46]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. InEuropean Conference on Computer Vision, pages 631–648. Springer,

  39. [47]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, and et al. Learning to prompt for continual learning. InCVPR, pages 139–149, 2022. 2, 3, 5, 7, 1

    2022

  40. [48]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. InProceedings of the IEEE/CVF Conference on Computer Visio...

    2022

  41. [49]

    Large scale in- cremental learning

    Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale in- cremental learning. InCVPR, pages 374–382, 2019. 2

    2019

  42. [50]

    Der: Dynam- ically expandable representation for class incremental learn- ing

    Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynam- ically expandable representation for class incremental learn- ing. InCVPR, pages 3014–3023, 2021. 2

    2021

  43. [51]

    Boosting continual learning of vision-language models via mixture-of-experts adapters

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23219–23230, 2024...

    2024

  44. [52]

    Maintaining discrimination and fairness in class incremental learning

    Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu- Tao Xia. Maintaining discrimination and fairness in class incremental learning. InCVPR, pages 13208–13217, 2020. 2

    2020

  45. [53]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xi- angyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. InProceedings of the IEEE/CVF international conference on computer vision, pages 19125–19136, 2023. 1, 3, 5...

    2023

  46. [54]

    Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

  47. [57]

    To further validate its robustness, we also evaluate its performance on ORDER-II

    Additional Experiment Results In this section, we first provide a detailed analysis of the test accuracy of our method across various tasks. To further validate its robustness, we also evaluate its performance on ORDER-II. Detail Test Accuracy.Table 9 presents the performance ...

  48. [58]

    For structural details, we set the rank of both matrixAand matrixBto 16 for the experts in ABFA, and similarly, the adapter in AFFA is configured with a rank of 16 (Table 13)

    Additional Experimental Setup Implementation Detail.All experiments were conducted using the PyTorch framework on an RTX 4090 24GB GPU. For structural details, we set the rank of both matrixAand matrixBto 16 for the experts in ABFA, and similarly, the adapter in AFFA is config...

  49. [59]

    Task Selection Accuracy of DDS Table 11 presents the selection accuracy of our domain dis- tribution selector (DDS) on previously learned tasks. It can be observed that with DDS, we are able to accurately assign each test sample to its corresponding task during the testing pha...

  50. [60]

    transfer

    Additional Ablation Experiments 9.1. Study on Application Strategies of AFFA To appropriately apply the AFFA module, we experimented with three strategies: applying it only to the CLIP text branch (the first row of Table 12), applying it only to the CLIP visual branch (the sec...

  51. [162]

    Springer, 2024. 7, 1

    2024

  52. [2022]

    2 11 Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models Supplementary Material

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.