Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Continual fine-tuning of CLIP can preserve old-task accuracy and zero-shot transfer by projecting each task's gradient into the null space of previous task activations, plus distillation and a modality-alignment loss that keep the…

desk verdict A workmanlike combination of null-space projection, distillation, and alignment loss for continual CLIP fine-tuning; the results are good, but the exact-invariance claim doesn't survive the rho=0.15 threshold. read the letter →

arxiv 2507.19839 v1 pith:EJGYZS4Y submitted 2025-07-26 cs.LG cs.CV

Tiantian Peng , Yuyang Liu , Shuo Yang , Qiuhe Hong , YongHong Tian This is my paper
classification cs.LGcs.CV
keywords continuallearningvision-languagemodelsCLIPgradientnullspaceprojectioncatastrophicforgettingzero-shotgeneralizationmodalitygapcontrastivedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLIP's usefulness comes from a shared image-text embedding space that enables zero-shot transfer, and ordinary continual fine-tuning quietly destroys that space as each new task pulls the two modalities apart. This paper claims that Gradient Null Space Projection (GNSP) can prevent that destruction: the gradient for each new task is projected onto the common null space of previous tasks' feature activations—the set of update directions that leaves those activations unchanged—before being applied, so the update is confined to directions that leave prior task outputs mathematically invariant, with no rehearsal data and no architectural changes. Because that constraint alone does not protect CLIP's original embedding geometry, the paper adds Contrastive Distillation from the untouched CLIP and a Modality Alignment Preservation loss on reference images, which together mimic CLIP's pre-training objective. On the 11-task MTIL benchmark the method reports state-of-the-art Average and Last accuracy in both task orderings while keeping the modality gap stable and cross-modal retrieval close to zero-shot levels, which is what the paper offers as evidence that the vision-language space survives continual learning.

What carries the argument

The load-bearing object is the projection matrix $P_l^{t-1} = V_2 V_2^T$: the right singular vectors belonging to the small singular values of the accumulated gram matrix $\hat{M}_l^{t-1} = \sum_i M_l^i$, where each $M_l^i = (X_l^i)^\top X_l^i / \|(X_l^i)^\top X_l^i\|_F$ is the normalized covariance of task $i$'s activations at layer $l$. Because a gram matrix shares its right null space with the activations themselves, projecting the current gradient as $\Delta W_l^t = P_l^{t-1} G_l^t$ makes $X_l^{t-1}\Delta W_l^t = 0$ in the ideal case, which is exactly the condition that keeps every previous task's layer outputs invariant. The adaptive threshold rule (choose $\sigma_m$ so that the excluded singular values carry at most a fixed fraction $\rho = 0.15$ of the spectrum) is what makes the mechanism usable: with the exact null space ($\rho = 0$) the projection is so low-rank that CLIP cannot learn new tasks, while a fixed rank budget would ignore each layer's spectral shape. Two auxiliary losses carry the generalization half of the argument: Contrastive Distillation (KL divergence between teacher and student image-text similarity matrices) and the Modality Alignment Preservation loss (an in-batch contrastive objective), both evaluated on reference images.

What would settle it

After each task, measure the per-layer residual $\|X_l^{t-1} \Delta W_l^t\|_F$ on held-out previous-task data under the default $\rho = 0.15$; if the residual is not close to zero, then previous-task outputs are changing despite the claimed invariance, and the reported Last accuracy would have to come from the distillation and alignment losses rather than from the null-space constraint. The paper's own ablation—GNSP without those losses drops Last to 80.7—makes this test decisive.

Watch

Extended reading notes

Core claim

The central claim is that continual fine-tuning of CLIP can be made non-destructive: confine the weight update for task $t$ to the common null space of all previous tasks' activations at each feed-forward layer—formally $X_l^{t-1} \Delta W_l^t = 0$—and the model's behavior on every earlier task is preserved by construction, not by remembering data. The paper implements this with a projection matrix $P_l^{t-1} = V_2 V_2^T$ obtained from the SVD of accumulated gram matrices of previous task activations, selecting the singular vectors below an adaptive threshold $\rho = 0.15$ of the spectrum to keep enough plasticity to learn new tasks. The second half of the claim is that preserving old tasks is not the same as preserving the embedding space: Contrastive Distillation matches the current model's image-text similarity structure to that of the initial CLIP teacher on 1,000 ImageNet reference images, and the Modality Alignment Preservation loss re-runs a contrastive objective on those same images to hold the two modalities together. The evidence takes the form of state-of-the-art Average (77.3/76.7) and Last (86.4/87.7) accuracy on MTIL Orders I and II, Transfer accuracy near the zero-shot upper bound, smoothed modality-gap curves, and COCO retrieval recall within a point of zero-shot.

Load-bearing premise

The whole anti-forgetting guarantee depends on the bottom 15% of the singular-value spectrum being close enough to the true common null space that the leftover change $X_l^{t-1}\Delta W_l^t$ is negligible; the paper presents this as an exact mathematical guarantee, but with $\rho = 0.15$ it is an approximation whose size is never measured.

Editorial extensions

If this is right

  • Continual fine-tuning of a VLM no longer forces a stability-plasticity trade: updates are confined to a subspace that leaves earlier tasks' feature responses invariant, so old-task accuracy is retained without replaying data.
  • Zero-shot ability survives the stream: Transfer accuracy stays within about a point of the untouched CLIP upper bound (65.7 vs 65.4 on Order II), so the fine-tuned model remains usable on unseen tasks.
  • The shared embedding space stays usable beyond classification: COCO image-to-text retrieval drops far less than in baselines (R@1 29.0 vs 9.0 for plain continual fine-tuning and 28.1 for the previous best), so retrieval-style tasks do not silently degrade.
  • Scaling to more tasks does not grow the anti-forgetting overhead: only gram matrices of intermediate activations are stored, and only the 12 feed-forward layers of the image encoder need to be trained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reference set is ImageNet, the method assumes access to a generic proxy of CLIP's pre-training distribution; a natural extension is to test whether domain-matched reference data changes the Transfer/Last balance in predictable ways.
  • The modality-gap curves double as a warning for the field: a method can post strong MTIL classification numbers while its embedding geometry drifts badly (as the paper's GIFT comparison shows), so classification accuracy alone is not a trustworthy monitor of a VLM's health during continual learning.
  • Since the guarantee is approximate once $\rho>0$, an obvious follow-up is an adaptive per-layer threshold chosen from the measured residual $\|X\Delta W\|_F$ rather than a fixed 15%, which could tighten protection without sacrificing learnability.
  • Nothing in the mechanism is CLIP-specific: the same null-space projection on feed-forward layers plus distillation should transfer to other dual-encoder VLMs whenever a reference set approximating their pre-training data is available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GNSP (Gradient Null Space Projection) for continual fine-tuning of CLIP. After each task, the method accumulates normalized gram matrices of intermediate FFN activations, computes their SVD, and projects new-task gradients onto the subspace spanned by singular vectors associated with small singular values, selected by a spectral-ratio threshold ρ=0.15. A contrastive distillation loss and a modality alignment preservation loss on ImageNet reference data are added to stabilize the embedding geometry. Experiments on the MTIL benchmark report state-of-the-art Average and Last accuracies for both task orderings, and the paper further analyzes modality-gap stability and COCO retrieval. The central theoretical claim is that the projection mathematically prevents interference with previous tasks.

Significance. If the exact-invariance claim were quantitatively supported, GNSP would be a clean and useful recipe for continual VLM fine-tuning: it avoids rehearsal and architectural changes, maintains zero-shot transfer, and directly targets the cross-modal embedding space. The paper also makes a worthwhile empirical contribution by measuring modality-gap drift and retrieval degradation, and it ships code. However, the mathematical guarantee is currently asserted for a thresholded projection that does not satisfy the exact null-space condition, and the ablations do not cleanly separate the contribution of the projection from the distillation/alignment regularizers. With residual measurements and consistent ablations, the method's contribution could be established; as written, the load-bearing claims exceed the evidence.

major comments (4)
  1. [Section 3.2, Eqs. (4), (8)-(10)] The claim that the update ΔW_l^t = P_l^{t-1} G_l^t 'guarantees that Eq.4 holds true' is not correct for the implemented ρ=0.15. Eq.4 requires X_l^{t-1} ΔW_l^t = 0, but because V2 contains singular vectors with nonzero singular values Σ2, X_l^{t-1} V2 = U2 Σ2 ≠ 0. The residual R = X_l^{t-1} P_l^{t-1} G_l^t is generally nonzero, with operator norm bounded by σ_m ‖G_l^t‖. The paper never reports σ_m, the spectral distribution, or the measured residual magnitude, so the 'mathematical prevention of interference' is unsubstantiated. Please report this residual at each layer and task, compare it with ‖G_l^t‖, or replace the exact-invariance claim with an explicitly approximate statement whose error is quantified.
  2. [Section 4.3, Table 3 vs. Table 2] The Continual Fine-tune baseline is not consistent across the two tables. Table 2 reports Transfer 56.5, Avg 59.9, Last 63.0, while Table 3 reports Transfer 44.6, Avg 55.9, Last 77.3 for the same named method. These are very different numbers, so the component-wise gains in Table 3 cannot be compared with the main SOTA comparison. Please specify the exact training protocol for each baseline and use one consistent baseline for both the main results and the ablation.
  3. [Section 4.3, Table 3] The ablation does not support the text's emphasis on GNSP as the main forgetting-prevention mechanism. GNSP alone improves Last from 77.3 to 80.7, whereas CD alone improves Last to 86.5, a much larger gain, and the full method reaches 87.7. Thus most of the Last-accuracy retention is attributable to distillation rather than to the null-space projection. The paper should report a full 2×2 ablation (with/without GNSP and with/without CD+MAP) under the same baseline and quantify the marginal contribution of and interaction between the components.
  4. [Section 4.2, Tables 1-2; Section 4.3, Table 4] All SOTA and hyperparameter claims are based on single runs, with differences as small as 0.1-0.3 points (e.g., Avg 76.67 vs. 76.62, Last 87.65 vs. 87.50 in Table 4). No error bars, multiple seeds, or significance tests are reported. Because several MTIL comparisons are separated by less than one percentage point, the claimed improvements cannot be distinguished from training noise. Please report means and standard deviations over at least three seeds.
minor comments (6)
  1. [Section 3.2] Typo: 'guarantees that Eq.4 holds ture' should be 'holds true'; also 'wil be preserved' should be 'will be preserved'.
  2. [Section 4.3, Table 3] The sentence 'CD alone ... suffers from forgetting, as shown by a lower Last score' is misleading because the CD-only row reports Last 86.5, which is higher than the GNSP-only Last 80.7; please rephrase to compare against the appropriate reference row.
  3. [Section 4.3, Table 3] The checkmark notation in Table 3 is ambiguous because there is no legend stating which columns are active for each row; please add explicit on/off labels (e.g., 'GNSP only', 'CD only').
  4. [Section 4.4, Eq. (21)] The quantity in Eq. (21) is the average cosine similarity between paired image-text embeddings, not the standard 'modality gap' in the literature, which is typically the distance between modality centroids; please use a different term or justify why this definition is appropriate here.
  5. [Section 4.3] Typos: 'Souore' in Table 4(b), 'Singluar' in Figure 3 and its caption, 'Contrastie Distillation', and 'lagerρ' should be corrected.
  6. [Figure 3 and Section 4.3] Figure 3 shows singular values of gram matrices computed on 100k ImageNet images, while the reference-data experiments use 1k images; please clarify the relationship between these two uses of ImageNet.

Circularity Check

1 steps flagged · score 3.0 of 10

Mild circularity: MAP loss directly optimizes the modality-gap quantity, making the gap-stability analysis partly self-confirmatory; the central MTIL and COCO retrieval claims remain externally benchmarked.

  1. self definitional [Section 3.4, Eqs. 17-19; Section 4.4, Eq. 21 and Fig. 4]
    "The final Modality Alignment Preservation loss is: L_MAP = L_MAP_I2T + L_MAP_T2I. ... Δgap = 1/N Σ_i cos(f(I_i), f(T_i)). ... With MAP component, the modality gap is further stabilized, demonstrating the effectiveness of our method in maintaining the shared embedding space."

    MAP (Eqs. 17-19) is an in-batch contrastive/InfoNCE loss on the reference-data cosine similarities S_t_{i,j} = cos(f_t(I_i), f_t(T_j)), whose diagonal terms are exactly the quantities averaged by the modality-gap metric Δgap in Eq. 21. Minimizing L_MAP therefore directly increases the diagonal similarities and suppresses off-diagonal ones on the reference data, so a smaller or more stable Δgap with MAP in Fig. 4 is substantially enforced by the loss itself rather than being an emergent consequence of the continual-learning mechanism.

full rationale

The MTIL benchmark comparisons (Tables 1-2) and COCO image-to-text retrieval (Table 5) are external to the method's construction, so the SOTA and retrieval claims are not circular. The core null-space construction (Eqs. 4-10) is a standard derivation: if P were exactly the projector onto the common null space of previous-task activations, Eq. 4 would follow by definition. In practice, the threshold ρ=0.15 makes the 'guarantee' inexact, but that is a mathematical-support gap rather than a circular or self-referential derivation. The only identifiable circularity is the MAP / modality-gap analysis: L_MAP and Δgap are built from the same per-pair cosine similarities, so the observed stabilization is partly by construction. The claim that Eq. 4 'holds true' is also deferred to a supplementary that is not included in the manuscript, leaving the exact-invariance assertion unsupported; this is a missing-proof concern, not a circularity. No load-bearing self-citation chain appears, and no fitted parameter is renamed as a prediction. The score of 3 reflects one partially self-confirmatory auxiliary analysis while the central benchmark results remain independent.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method relies on standard linear algebra for the null-space construction, plus several domain assumptions about FFN localization and ImageNet representativeness. The main free parameters are tuned by ablation. No new physical or conceptual entities are introduced.

free parameters (4)
  • rho (singular value ratio) = 0.15
    Controls the threshold sigma_m in Eq.9; ablated across 0.1, 0.15, 0.2 and 0.15 chosen as default.
  • lambda (distillation weight) = 1.0
    Weight on LCD in Eq.20; set by ablation.
  • beta (MAP loss weight) = 0.75
    Weight on LMAP in Eq.20; ablated over 0.5, 0.75, 1.0.
  • reference image count = 1000
    Number of ImageNet images used for CD and MAP; ablated over 0.5k, 1k, 10k.
assumptions (5)
  • standard math Gram matrix M = X^T X / ||X^T X||_F has the same right null space as X.
    Standard linear algebra; used in Eq.6.
  • standard math The null space of the sum of positive semidefinite gram matrices equals the intersection of their individual null spaces.
    Used to combine previous task covariance matrices in Eq.7.
  • domain assumption Preserving the outputs of the 12 FFN layers of the image encoder is sufficient to preserve the model's behavior on previous tasks.
    The authors fine-tune only FFN layers (Section 4.1) and project gradients for those layers; the claim that previous tasks are protected relies on this localization assumption.
  • domain assumption A 1000-image subset of ImageNet is representative of CLIP's pretraining distribution and sufficient to stabilize the embedding space.
    Used for CD and MAP losses; the paper justifies this as 'close to CLIP's pretraining data' but gives no measure of representativeness.
  • ad hoc to paper The spectral truncation with rho=0.15 yields a null-space projection whose residual interference is negligible.
    This is the key approximation in Eq.8-9; the paper provides no bound on ||X Delta W|| after projection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning." pith.science (2026). https://pith.science/paper/EJGYZS4Y

@misc{pith2026250719839,
  author       = {Pith},
  title        = {Pith review of: GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJGYZS4Y}},
  note         = {Machine review of arXiv:2507.19839}
}
read the original abstract

Contrastive Language-Image Pretraining has demonstrated remarkable zero-shot generalization by aligning visual and textual modalities in a shared embedding space. However, when continuously fine-tuned on diverse tasks, CLIP suffers from catastrophic forgetting and degradation of its embedding alignment, undermining its zero-shot capabilities. In this work, we propose Gradient Null Space Projection (GNSP), an efficient continual learning method that projects task-specific gradients onto the null space of previously learned knowledge. This orthogonal projection mathematically prevents interference with previous tasks without relying on rehearsal or architectural modification. Furthermore, to preserve the inherent generalization property of CLIP, we introduce knowledge distillation and combine it with a modality alignment preservation loss inspired by CLIP pre-training to stabilize the structure of the multimodal embedding space during fine-tuning. On the MTIL benchmark consisting of 11 tasks, our method achieved SOTA performance on both the Average and Last key metrics. More importantly, experiments show that our method successfully maintains the original modality gap and cross-modal retrieval performance of CLIP, confirming its effectiveness in maintaining a robust visual-language space throughout the continual learning process.

Figures

Figures reproduced from arXiv: 2507.19839 by the authors.

Figure 1
Figure 1. Orthogonal Gradient Projection (OGP) projects the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework overview of GNSP. (a) Gradient Null Space Projection. The updated gradient Gt l will be projected onto the shared null space Nˆ t−1 of previous tasks to obtain ∆Wt l to update CLIP θ t . After training task t, the null space N t will be computed to update the shared null space; (b) Distillation and Alignment. Contrastive Distillation LCD is used for matching CLIP θ t and Initial CLIP θ 0 on reference data,… view at source ↗
Figure 3
Figure 3. Singluar values of gram matrix at different layers [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The variation of modality gap when fine-tuning using different methods, with the vertical dashed line representing [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: T-SNE visualization. For StanfordCars, “source” [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

Reference graph

Works this paper leans on

64 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bossard, L.; Guillaumin, M.; and Van Gool, L. 2014. Food-101 -- Mining Discriminative Components with Random Forests. In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., Computer Vision -- ECCV 2014, 446--461. Cham: Springer International Publishing

    work page 2014

  4. [4]

    Chaudhry, A.; Ranzato, M.; Rohrbach, M.; and Elhoseiny, M. 2018. Efficient Lifelong Learning with A-GEM. arXiv:1812.00420

    arXiv 2018

  5. [5]

    Chen, X.; Fang, H.; Lin, T.-Y.; Vedantam, R.; Gupta, S.; Doll \'a r, P.; and Zitnick, C. L. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325

    arXiv 2015

  6. [6]

    Chen, X.; Zhang, J.; Wang, X.; Zhang, N.; Wu, T.; Wang, Y.; Wang, Y.; and Chen, H. 2023. Continual multimodal knowledge graph construction. arXiv preprint arXiv:2305.08698

    arXiv 2023

  7. [7]

    Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; and Vedaldi, A. 2014. Describing Textures in the Wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, 3606--3613

    work page 2014

  8. [8]

    Cui, Z.; Peng, Y.; Wang, X.; Zhu, M.; and Zhou, J. 2024. Continual vision-language retrieval via dynamic knowledge rectification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 11704--11712

    work page 2024

Show all 64 references
  1. [9]

    D'Alessandro, M.; Alonso, A.; Calabr \'e s, E.; and Galar, M. 2023. Multimodal parameter-efficient few-shot class incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3393--3403

    2023

  2. [10]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

  3. [11]

    Deng, L. 2012. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Processing Magazine, 29(6): 141--142

    2012

  4. [12]

    Douillard, A.; Rame, A.; Couairon, G.; and Cord, M. 2021. DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2021

  5. [13]

    Fang, J.; Jiang, H.; Wang, K.; Ma, Y.; Shi, J.; Wang, X.; He, X.; and Chua, T.-S. 2025. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. In The Thirteenth International Conference on Learning Representations

    2025

  6. [14]

    Fei-Fei, L.; Fergus, R.; and Perona, P. 2004. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, 178--178

    2004

  7. [15]

    Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2021

  8. [16]

    J.; Mirza, M.; Xiao, D.; Courville, A.; and Bengio, Y

    Goodfellow, I. J.; Mirza, M.; Xiao, D.; Courville, A.; and Bengio, Y. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv:1312.6211

    2013 arXiv

  9. [17]

    Guo, H.; Zeng, F.; Xiang, Z.; Zhu, F.; Wang, D.-H.; Zhang, X.-Y.; and Liu, C.-L. 2025. Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. arXiv preprint arXiv:2503.12941

    2025 arXiv

  10. [18]

    Hase, P.; Bansal, M.; Kim, B.; and Ghandeharioun, A. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36: 17643--17668

    2023

  11. [19]

    Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2217--2226

    2019

  12. [20]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv:1503.02531

    2015 arXiv

  13. [21]

    Jha, S.; Gong, D.; and Yao, L. 2024. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. Advances in neural information processing systems, 37: 129146--129186

    2024

  14. [22]

    V.; Sung, Y.-H.; Li, Z.; and Duerig, T

    Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q. V.; Sung, Y.-H.; Li, Z.; and Duerig, T. 2021. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In International Conference on Machine Learning

    2021

  15. [23]

    D.; Xu, Y.; Zeng, B.; and Chilimbi, T

    Jiang, Q.; Chen, C.; Zhao, H.; Chen, L.; Ping, Q.; Tran, S. D.; Xu, Y.; Zeng, B.; and Chilimbi, T. 2023. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2023

  16. [24]

    C.; Veness, J.; Desjardins, G.; Rusu, A

    Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N. C.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2016. Overcoming catastrophic forgetting in neural networks. Proceedings o...

    2016

  17. [25]

    Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013. 3D Object Representations for Fine-Grained Categorization. In 2013 IEEE International Conference on Computer Vision Workshops, 554--561

    2013

  18. [26]

    Krizhevsky, A.; and Hinton, G. 2009. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4)

    2009

  19. [27]

    M.; Sebe, N.; and Lew, M

    Lao, M.; Pu, N.; Liu, Y.; Zhong, Z.; Bakker, E. M.; Sebe, N.; and Lew, M. S. 2023. Multi-domain lifelong visual question answering via self-critical distillation. In Proceedings of the 31st ACM International Conference on Multimedia, 4747--4758

    2023

  20. [28]

    W.; Gao, D.; Wu, J

    Lei, S. W.; Gao, D.; Wu, J. Z.; Wang, Y.; Liu, W.; Zhang, M.; and Shou, M. Z. 2023. Symbolic replay: Scene graph as prompt for continual learning on vqa task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 1250--1259

    2023

  21. [29]

    Li, Z.; and Hoiem, D. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12): 2935--2947

    2017

  22. [30]

    Liang, W.; Zhang, Y.; Kwon, Y.; Yeung, S.; and Zou, J. 2022. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. In NeurIPS

    2022

  23. [31]

    Liang, Y.-S.; and Li, W.-J. 2023. Adaptive Plasticity Improvement for Continual Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7816--7825

    2023

  24. [32]

    Liu, W.; Zhu, F.; and Tian, Q. 2025. C-CLIP: Multimodal continual learning for vision-language model. In The Thirteenth International Conference on Learning Representations

    2025

  25. [33]

    Lopez-Paz, D.; and Ranzato, M. 2017. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, 6470–6479. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781510860964

    2017

  26. [34]

    Lu, H.; Zhao, C.; Xue, J.; Yao, L.; Moore, K.; and Gong, D. 2024 a . Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision-language models with dynamic rank-selective lora. arXiv preprint arXiv:2412.01004

    2024 arXiv

  27. [35]

    Lu, Y.; Zhang, S.; Cheng, D.; Xing, Y.; Wang, N.; Wang, P.; and Zhang, Y. 2024 b . Visual prompt tuning in null space for continual learning. Advances in neural information processing systems, 37: 7878--7901

    2024

  28. [36]

    B.; and Vedaldi, A

    Maji, S.; Rahtu, E.; Kannala, J.; Blaschko, M. B.; and Vedaldi, A. 2013. Fine-Grained Visual Classification of Aircraft. arXiv:1306.5151

    2013 arXiv

  29. [37]

    McCloskey, M.; and Cohen, N. J. 1989. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation, 24: 109--165

    1989

  30. [38]

    Mistretta, M.; Baldrati, A.; Agnolucci, L.; Bertini, M.; and Bagdanov, A. D. 2025. Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion. In The Thirteenth International Conference on Learning Representations

    2025

  31. [39]

    Ni, Z.; Wei, L.; Tang, S.; Zhuang, Y.; and Tian, Q. 2023. Continual vision-language representation learning with off-diagonal information. In International Conference on Machine Learning, 26129--26149. PMLR

    2023

  32. [40]

    Nilsback, M.-E.; and Zisserman, A. 2008. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 722--729

    2008

  33. [41]

    M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C

    Parkhi, O. M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C. V. 2012. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3498--3505

    2012

  34. [42]

    Qiao, J.; Tan, X.; Chen, C.; Qu, Y.; Peng, Y.; Xie, Y.; et al. 2024. Prompt gradient projection for continual learning. In The Twelfth International Conference on Learning Representations

    2024

  35. [43]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning

    2021

  36. [44]

    Saha, G.; Garg, I.; and Roy, K. 2021. Gradient Projection Memory for Continual Learning. In International Conference on Learning Representations

    2021

  37. [45]

    T.; Argus, M.; Fischer, V.; and Brox, T

    Schrodi, S.; Hoffmann, D. T.; Argus, M.; Fischer, V.; and Brox, T. 2025. Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. In The Thirteenth International Conference on Learning Representations

    2025

  38. [46]

    Tang, L.; Tian, Z.; Li, K.; He, C.; Zhou, H.; Zhao, H.; Li, X.; and Jia, J. 2024. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. In European conference on computer vision, 346--365. Springer

    2024

  39. [47]

    M.; and Tolias, A

    van de Ven, G. M.; and Tolias, A. S. 2019. Three scenarios for continual learning. arXiv:1904.07734

    2019 arXiv

  40. [48]

    van der Maaten, L.; and Hinton, G. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86): 2579--2605

    2008

  41. [49]

    Wang, S.; Li, X.; Sun, J.; and Xu, Z. 2021. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 184--193

    2021

  42. [50]

    Wang, Y.; Huang, Z.; and Hong, X. 2022. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. Advances in Neural Information Processing Systems, 35: 5682--5695

    2022

  43. [51]

    W.; Li, M.; Kornblith, S.; Roelofs, R.; Lopes, R

    Wortsman, M.; Ilharco, G.; Kim, J. W.; Li, M.; Kornblith, S.; Roelofs, R.; Lopes, R. G.; Hajishirzi, H.; Farhadi, A.; Namkoong, H.; et al. 2022. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7959--7971

    2022

  44. [52]

    Wu, B.; Shi, W.; Wang, J.; and Ye, M. 2025. Synthetic Data is an Elegant GIFT for Continual Vision-Language Models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2813--2823

    2025

  45. [53]

    A.; Oliva, A.; and Torralba, A

    Xiao, J.; Hays, J.; Ehinger, K. A.; Oliva, A.; and Torralba, A. 2010. SUN database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3485--3492

    2010

  46. [54]

    Yan, S.; Hong, L.; Xu, H.; Han, J.; Tuytelaars, T.; Li, Z.; and He, X. 2022. Generative negative text replay for continual vision-language pretraining. In European Conference on Computer Vision, 22--38. Springer

    2022

  47. [55]

    Yan, S.; Xie, J.; and He, X. 2021. DER: Dynamically Expandable Representation for Class Incremental Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2021

  48. [56]

    Yang, E.; Shen, L.; Wang, Z.; Liu, S.; Guo, G.; Wang, X.; and Tao, D. 2025. Revisiting Flatness-Aware Optimization in Continual Learning With Orthogonal Gradient Projection. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2025

  49. [57]

    Yang, Y.; Wan, F.; Jiang, Q.-Y.; and Xu, Y. 2024. Facilitating Multimodal Classification via Dynamically Learning Modality Gap. In Globerson, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J.; and Zhang, C., eds., Advances in Neural Information Processing Systems,...

    2024

  50. [58]

    Yu, J.; Zhuge, Y.; Zhang, L.; Hu, P.; Wang, D.; Lu, H.; and He, Y. 2024 a . Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23219--23230

    2024

  51. [59]

    Yu, Y.-C.; Huang, C.-P.; Chen, J.-J.; Chang, K.-P.; Lai, Y.-H.; Yang, F.-E.; and Wang, Y.-C. F. 2024 b . Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models. In European Conference on Computer Vision, 219--236. Springer

    2024

  52. [60]

    Zeng, G.; Chen, Y.; Cui, B.; and Yu, S. 2019. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8): 364--372

    2019

  53. [61]

    Zhang, X.; Zhang, F.; and Xu, C. 2023. Vqacl: A novel visual question answering continual learning setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19102--19112

    2023

  54. [62]

    Zheng, M.; Tang, Y.; Hao, Z.; Han, K.; Wang, Y.; and Xu, C. 2024. Adapt without forgetting: Distill proximity from dual teachers in vision-language models. In European Conference on Computer Vision, 109--125. Springer

    2024

  55. [63]

    Zheng, Z.; Ma, M.; Wang, K.; Qin, Z.; Yue, X.; and You, Y. 2023. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, 19125--19136

    2023

  56. [64]

    Zhu, H.; Wei, Y.; Liang, X.; Zhang, C.; and Zhao, Y. 2023. Ctp: Towards vision-language continual pretraining via compatible momentum contrast and topology preservation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22257--22267

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.