Pith. sign in

REVIEW 11 cited by

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2207.09248 v2 pith:2P6ZXHSE submitted 2022-07-19 cs.CV

Yuxuan Ding , Lingqiao Liu , Chunna Tian , Jingyuan Yang , Haoxuan Ding This is my paper · ORCID
classification cs.CV
keywords cliplearningmodelcontinualforgettingissueimage-textcatastrophic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The Contrastive Language-Image Pre-training (CLIP) Model is a recently proposed large-scale pre-train model which attracts increasing attention in the computer vision community. Benefiting from its gigantic image-text training set, the CLIP model has learned outstanding capabilities in zero-shot learning and image-text matching. To boost the recognition performance of CLIP on some target visual concepts, it is often desirable to further update the CLIP model by fine-tuning some classes-of-interest on extra training data. This operation, however, raises an important concern: will the update hurt the zero-shot learning or image-text matching capability of the CLIP, i.e., the catastrophic forgetting issue? If yes, could existing continual learning algorithms be adapted to alleviate the risk of catastrophic forgetting? To answer these questions, this work conducts a systemic study on the continual learning issue of the CLIP model. We construct evaluation protocols to measure the impact of fine-tuning updates and explore different ways to upgrade existing continual learning methods to mitigate the forgetting issue of the CLIP model. Our study reveals the particular challenges of CLIP continual learning problem and lays a foundation for further researches. Moreover, we propose a new algorithm, dubbed Learning without Forgetting via Replayed Vocabulary (VR-LwF), which shows exact effectiveness for alleviating the forgetting issue of the CLIP model.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

  2. Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Expert routing in multimodal continual instruction tuning is reformulated as soft task-as-class class-incremental learning, and a new 34-task fingerprint-reduced benchmark shows that plugging CIL classifiers into rout...

  3. iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection

    cs.CV 2025-05 conditional novelty 6.0 of 10

    iDPA improves incremental medical object detection by generating instance-level prompts from bounding-box regions and decoupling prompt attention in a frozen GLIP model.

  4. Satellites Reveal Mobility: A Commuting Origin-destination Flow Generator for Global Cities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Satellite imagery plus population is enough to generate commuting origin-destination flows that closely match models using detailed sociodemographic and point-of-interest data.

  5. Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A multimodal class-incremental learning method based on AudioCLIP with MoE adapters, adaptive audio-visual fusion, and a contrastive loss reports gains on three vision-audio-text datasets.

  6. VLMs meet UDA: Boosting Transferability of Open Vocabulary Segmentation with Unsupervised Domain Adaptation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    UDA-FROVSS, the first unsupervised domain adaptation framework for open-vocabulary segmentation, preserves the ability to segment categories never seen in the source dataset.

  7. Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    Textual Inversion is applied to open-vocabulary object detectors to learn a few new tokens while keeping the VLM frozen and preserving zero-shot abilities.

  8. Fresh-CL: Feature Realignment through Experts on Hypersphere in Continual Learning

    cs.LG 2025-01 conditional novelty 5.0 of 10

    Fresh-CL combines fixed equiangular classifier targets with mixture-of-experts projections to reduce feature overlap in multi-task continual learning, reporting about 2% average accuracy gain over a strong baseline.

  9. Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning

    cs.CV 2024-12 conditional novelty 5.0 of 10

    CoDyRA dynamically shrinks the rank of each LoRA update during continual learning, and this rank minimization reduces forgetting while preserving or improving performance on new tasks and unseen data.

  10. MUNBa: Machine Unlearning via Nash Bargaining

    cs.CV 2024-11 conditional novelty 5.0 of 10

    MUNBa is a machine unlearning method that uses Nash bargaining to balance forgetting and preservation gradients, improving unlearning quality, generalization, and robustness in image classification and generation.

  11. ChordPrompt: Orchestrating Cross-Modal Prompt Synergy for Multi-Domain Incremental Learning in CLIP

    cs.AI 2025-06 conditional novelty 4.0 of 10

    Cross-modal prompt sharing with domain-adaptive retrieval improves CLIP's continual learning performance on multi-domain image classification benchmarks.

Pith tools