Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Efficient Few-Shot Continual Learning in Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Updating only the image encoder of a vision-language model fixes visual-domain errors with 25x less compute and little forgetting.

desk verdict Solid empirical study of a PEFT recipe for continual VQA, but the 25x and 'without sacrificing performance' claims don't survive contact with the paper's own tables. read the letter →

arxiv 2502.04098 v2 pith:BEV2FLEZ submitted 2025-02-06 cs.CV cs.AI

Aristeidis Panos , Rahaf Aljundi , Daniel Olmeda Reino , Richard E. Turner This is my paper · ORCID
classification cs.CVcs.AI
keywords few-shotcontinuallearningvision-languagemodelsparameter-efficientfine-tuninglow-rankadaptationstructuredupdatesimageencoderCLIPvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the most effective and economical way to adapt a vision-language model to a new visual domain is to update only its image encoder, leaving the language model frozen. To do that it introduces LoRSU (Low-Rank Adaptation with Structured Updates), which in each continual-learning session computes the CLIP contrastive-loss gradient on a few dozen target image-text pairs, ranks attention heads and first-MLP-layer parameters by gradient magnitude, and updates only the top-ranked subset through LoRA adapters and binary masks. The authors report that over ten VQA datasets and 5/20/50-shot settings LoRSU improves target-domain accuracy (their Target Improvement metric) more consistently than full fine-tuning, EWC, LoRA, AdaLoRA, and the structured-sparsity baseline SPU, while keeping accuracy on control datasets nearly unchanged. They also report that updating only the image encoder cuts computational overhead by more than 25 times compared with updating the whole VLM. If true, this would make continual visual adaptation practical in resource-constrained settings where replay buffers and large-scale retraining are not available.

What carries the argument

The central object is the optimal binary update mask $p^\ast$ that solves $\max_{p\in\{0,1\}^d} \lVert p \odot \nabla_W L(\theta_0)\rVert^2 / \lVert \nabla_W L(\theta_0)\rVert^2$ subject to disjoint parameter groups and a total sparsity budget $C$. The lemma the paper proves is that this mask keeps exactly the largest-magnitude gradient entries inside each group, which is why the attention-head score in equation (3) sums squared query/key/value gradients and why the MLP gate applies a gradient-magnitude mask. LoRA adapters on the selected heads then make the update low-rank, while the mask keeps all other parameters exactly frozen. This combination of group-structured gradient selection and low-rank reparametrization is what the paper claims carries both the performance gain and the forgetting control.

What would settle it

Run a matched experiment where LoRSU's gradient-based head selection is replaced by random head selection at the same parameter count and the same sparsity, on a dataset where the few-shot CLIP gradients are noisy (for example one image per class per session); if random selection matches LoRSU's Target Improvement, the gradient-magnitude selection rule is not doing the work. A complementary check is to measure TI on a target dataset after removing the attention-head component entirely and keeping only the fc1 mask: the paper's own ablation shows a gap on GTS, so a dataset where that gap disappears would falsify the claim that head selection contributes.

Watch

Extended reading notes

Core claim

The authors claim that the visual failures of VLMs such as LLaVA-v1.5 are usually perception-side failures and can be corrected by targeted, cheap updates to the CLIP image encoder. Concretely, LoRSU takes the gradient of the CLIP loss over a small labelled target set and, for every transformer block, computes an importance score for each attention head as the sum of squared gradients of its query, key, and value matrices; only the top two heads are updated, and their weight matrices are reparametrized with LoRA adapters. In parallel, gradients of the first linear layer of each MLP block are masked so that only the largest-magnitude entries (10 percent sparsity) are updated. The paper proves that this mask choice is the exact solution of a constrained optimization problem that maximizes the norm of the masked gradient under group-wise sparsity constraints. Across five consecutive few-shot sessions, with no replay buffer, LoRSU reports positive Target Improvement on nearly every dataset and near-zero or positive Control Change, and the 25x compute reduction is measured against the full-VLM LoRA baseline.

Load-bearing premise

The load-bearing premise is that CLIP contrastive-loss gradients computed on a few target-domain image-text pairs point to image-encoder parameters whose adjustment improves downstream VQA accuracy once the encoder is reinserted into the frozen-LLM model, even though LoRSU never optimizes the VQA loss itself.

Editorial extensions

If this is right

  • If LoRSU's claims hold, a VLM can be adapted to a new visual domain with more than a 25x reduction in compute relative to full-model updates, using only a handful of labelled images per class.
  • Because the method needs no replay buffer, deployment scenarios that cannot store old images (privacy-sensitive or memory-constrained settings) can still accumulate target improvements across sessions.
  • The near-zero Control Change values imply that repeated adaptation over many domains would preserve generic VLM knowledge while fixing perception errors, weakening the usual continual-learning accuracy/forgetting trade-off.
  • The same recipe transfers to any transformer image encoder, since the selection rule uses only gradient magnitudes of attention and MLP weights, not dataset-specific structure.
  • The perplexity-loss variant LoRSU-Ppl shows that the target/forgetting balance can be shifted further toward target accuracy when the frozen LLM's own text-generation signal is available, at the cost of extra compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stress test not run in the paper is whether the gradient-magnitude selection transfers to other VLM backbones and larger CLIP variants; the authors validate only LLaVA-v1.5 with CLIP-L-14, so the generality claim is an extrapolation.
  • The paper's own Discussion flags that binary-mask structured updates may not scale to LLM-sized parameter spaces; this bounds the method's applicability but does not touch the 25x image-encoder result, which is exactly the claim that matters for the stated use case.
  • An unstated corollary of the metric design is that positive CC means positive backward transfer; a useful follow-up would test whether the attention heads selected for early tasks are re-selected in later tasks, which would predict when forgetting eventually accumulates.
  • The selection rule could be tested outside VLMs, on unimodal image classification or retrieval, where the same 'update only top-gradient-magnitude parameters with CLIP loss' recipe might produce a standalone parameter-efficient learner.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LoRSU, a replay-free parameter-efficient fine-tuning method for continually adapting the CLIP image encoder of a vision-language model (LLaVA-v1.5) under few-shot continual learning. LoRSU selects a small number of attention heads and a structured subset of first-MLP-layer parameters based on gradient magnitudes, applies LoRA to the selected heads, and trains with the CLIP loss. The method is evaluated on ten VQA datasets under CL-5, CL-20, and CL-50 settings, reporting Target Improvement (TI), Control Change (CC), and additional ACC/BWT metrics against six CLIP-loss baselines and several perplexity-loss baselines. The paper also introduces the TSI and DALLE datasets. The central claims are that LoRSU improves target VQA accuracy while preserving generic knowledge, and that it reduces computational overhead by more than 25x compared to full VLM updates without sacrificing performance.

Significance. If the claims were fully supported, LoRSU would be a useful replay-free, compute-efficient method for visual-domain adaptation of VLMs. The paper contains a substantial empirical study: ten datasets, three few-shot continual settings, multiple baselines, ablations over rank and attention-head count, and a robustness study over training epochs, with detailed per-dataset results and error bars in the appendix. The introduction of the TSI and DALLE datasets and the TI/CC evaluation protocol are also useful contributions. However, the headline efficiency claim is confounded by the loss choice, and the 'without sacrificing performance' and 'consistently superior TI' statements are contradicted by the paper's own tables. These issues do not invalidate the method's potential, but they require substantial reframing of the claims and a clear separation of what LoRSU contributes versus what the CLIP loss and encoder-only updating contribute.

major comments (4)
  1. [Abstract; §4.6; Figure 3] The 'over 25x' computational-efficiency claim is confounded by the choice of loss. In Figure 3, LoRSU with the CLIP loss is reported at 0.36 TFLOPs, while LoRSU-Ppl and LoRA-F are reported at 9.1 and 9.0 TFLOPs respectively; the text then states that 'LoRSU requires 25 × fewer computation resources than LoRA-F and LoRSU-Ppl.' Since LoRSU-Ppl uses the same structured-update mechanism as LoRSU and differs only in the loss (perplexity instead of CLIP), the 25x factor measures the cost of backpropagating through the LLM under the perplexity loss, not the benefit of LoRSU's structured parameter selection. The efficiency gain of the structured update itself should be measured against a CLIP-loss baseline that updates the same modules, or the claim should be explicitly rephrased as a property of encoder-only CLIP-loss fine-tuning.
  2. [Abstract; §4.3; Table 3] The abstract's claim that LoRSU reduces computational overhead 'without sacrificing performance' is not supported by Table 3. At CL-50, LoRA-F achieves TI of 17.9 on AIR, 21.0 on VSR, and 12.3 on HM, whereas LoRSU achieves 8.2, 2.3, and 0.3 on the same datasets; at CL-20, LoRA-F leads on AIR (16.3 vs 5.9) and VSR (23.3 vs 0.1). These are large target-accuracy gaps, and the paper's own tables show that LoRSU's advantage is primarily in the CC/forgetting dimension, not in achieving equal or better target improvement. The unqualified 'without sacrificing performance' statement should be removed or replaced with a precise characterization of the TI/CC trade-off.
  3. [§4.3; Table 1] The sentence 'LoRSU consistently achieves superior TI scores across datasets and CL settings' is contradicted by Table 1. On the TSI dataset, F-FT and F-EWC report higher TI than LoRSU in CL-5 (7.4 and 8.5 vs 3.2), CL-20 (15.3 and 16.0 vs 10.6), and CL-50 (F-EWC 22.4 vs 19.1). The claim should be qualified to reflect that LoRSU is competitive or best on most datasets, but not consistently superior on every dataset and setting, especially when target accuracy is considered in isolation from CC.
  4. [§3, Eq. (6); Appendix A] The theoretical justification of the head-importance score is incomplete. Lemma A.2 solves the unconstrained top-C problem, but the optimization in Eq. (6) includes group constraints and per-group budgets c_l; Corollary A.4 merely states that the optimal mask keeps the top c_l entries within each group. The method, however, selects all parameters of the top-k heads according to the cumulative score s_i in Eq. (3), which is an all-or-nothing group-selection rule. The proof does not show that Eq. (3) is the solution to Eq. (6) under the head-group structure, so the claimed theoretical grounding for the attention-head selection mechanism is not established. Either the proof should be extended to the group-level selection rule, or the derivation should be presented as a heuristic justified by the top-gradient intuition.
minor comments (5)
  1. [Tables 1 and 3; Appendix B] The main tables report TI/CC as averages over three runs without error bars, even though Appendix B states that error bars are included for all experiments and the detailed appendix tables do show them. Many headline comparisons are within 1–2 points (e.g., Table 1, CL-5 GTS: LoRSU 6.4 vs SPU 5.4), so the main tables should include error bars or confidence intervals to allow readers to assess significance.
  2. [§4.5] The text refers to 'LoRSU-AHH' when the table and surrounding text use 'LoRSU-AAH'; the abbreviation should be consistent.
  3. [§4.6; Figure 3] The paper says 'floating-point operations per second' but TFLOPs is a count of operations, not a rate; also 'TFlops' in the figure caption should be 'TFLOPs'.
  4. [§4.4] There is a typo in the sentence beginning 'We aevaluate how LoRSU and LoRA perform'; it should read 'We evaluate'.
  5. [§3] In the definition of the attention mechanism, the sentence says 'W(i)_q, W(i)_k, and W(i)_k are the query, key, and value matrices'; the third symbol should be W(i)_v.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the selection rule is derived from a self-contained gradient-norm optimization, and the efficiency and performance claims are empirical comparisons rather than fitted inputs returned as predictions.

full rationale

LoRSU's parameter-selection rule is derived from Eq. (6), a constrained optimization that maximizes the norm of the masked gradient under group-wise sparsity budgets. Appendix A proves the top-C solution via a self-contained binary-knapsack argument (Lemma A.2) that does not assume the target result; when an entire attention head is selected, the importance score in Eq. (3) is exactly the per-head group objective, so the derivation is an optimization identity rather than a fitted prediction. The downstream VQA gains (TI/CC) are measured on held-out test splits after fine-tuning with a CLIP loss, and the 25x efficiency figure in Fig. 3 is a direct TFLOPs comparison (0.36 vs 9.1) between LoRSU with the CLIP loss and full-VLM/perplexity updates. No parameter fitted to the target outputs is relabeled as a prediction, and the only self-citation (Panos et al. 2023 for CL sequence construction) is a dataset-protocol reference, not a load-bearing uniqueness or ansatz claim. The fixed hyperparameters (rank 64, sparsity 10%, 2 heads) are chosen on the same benchmark suite, which raises an empirical-validity concern about possible overfitting, but that is not circularity under the stated tests: the central derivation and the reported comparisons stand independently of those choices.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on a standard convex selection rule plus several domain assumptions. The only hand-set numbers are the three hyperparameters above, which are not fitted per dataset and are matched to baseline parameter budgets.

free parameters (3)
  • sparsity for fc1 mask = 10%
    Chosen globally (Appendix B); ablations in Appendix E.3 test robustness only over epochs, not this value.
  • LoRA rank r = 64
    Global hyperparameter; Appendix E.1 shows peak target accuracy around r=64; not derived.
  • number of attention heads k = 2
    Global hyperparameter; Appendix E.2 shows sensitivity of TI/CC to k.
assumptions (4)
  • standard math The optimal mask for maximizing the norm of masked gradients is the top-k gradient elements per group (Lemma A.2, Corollary A.4).
    Trivial knapsack; proven in Appendix A.
  • domain assumption Updating image-encoder parameters with CLIP contrastive loss on few-shot target pairs improves downstream VQA performance of the frozen-LLM VLM.
    This is the load-bearing premise of the whole pipeline; validated only empirically in Section 4.2.
  • domain assumption The image encoder, not the LLM, is the main source of VLM failures on new visual domains (motivated by Figure 1).
    The paper argues this through one anecdote and then builds the method on it.
  • domain assumption The 5-session class-disjoint few-shot protocol is a fair proxy for real-world continual adaptation.
    Standard in prior CL work (Panos et al., 2023), but the transfer to VQA is new.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Few-Shot Continual Learning in Vision-Language Models." pith.science (2026). https://pith.science/paper/BEV2FLEZ

@misc{pith2026250204098,
  author       = {Pith},
  title        = {Pith review of: Efficient Few-Shot Continual Learning in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BEV2FLEZ}},
  note         = {Machine review of arXiv:2502.04098}
}
read the original abstract

Vision-language models (VLMs) excel in tasks such as visual question answering and image captioning. However, VLMs are often limited by their use of pretrained image encoders, like CLIP, leading to image understanding errors that hinder overall performance. On top of that, real-world applications often require the model to be continuously adapted as new and often limited data continuously arrive. To address this, we propose LoRSU (Low-Rank Adaptation with Structured Updates), a robust and computationally efficient method for selectively updating image encoders within VLMs. LoRSU introduces structured and localized parameter updates, effectively correcting performance on previously error-prone data while preserving the model's general robustness. Our approach leverages theoretical insights to identify and update only the most critical parameters, achieving significant resource efficiency. Specifically, we demonstrate that LoRSU reduces computational overhead by over 25x compared to full VLM updates, without sacrificing performance. Experimental results on VQA tasks in the few-shot continual learning setting, validate LoRSU's scalability, efficiency, and effectiveness, making it a compelling solution for image encoder adaptation in resource-constrained environments.

Figures

Figures reproduced from arXiv: 2502.04098 by the authors.

Figure 1
Figure 1. (Left) Responses of the pretrained LLaVA to samples from TSI dataset (bottom) compared to DALL·E 2 generated im￾ages (top) for the ‘cooking on a stove’ class. (Right) LLaVA’s correct response to the same TSI image after fine-tuning LLaVA using LoRSU. a low-number of samples from TSI dataset compared to the pretrained LLaVA’s (wrong) response. Through extensive experiments, we demonstrate that up￾dating the image enc… view at source ↗
Figure 2
Figure 2. LoRSU mechanism: After computing the gradient ∇θLt(θ) over the target dataset at time t, LoRSU picks a small number of attention heads and a small number of paremeters from the first linear layer of the MLP module in the transformer block based on the magnitude of the gradients of ∇WAttnLt and ∇Wfc1Lt, respectively. Computational efficiency is ensured by introducing LoRA adapters to the attention weight matrices. sp… view at source ↗
Figure 3
Figure 3. TFlops and trainable parameters comparison between LoRSU with CLIP loss (LoRSU), perplexity loss (LoRSU-Ppl), and LoRA-F. Random selection (LoRSU-Rand) fails to generalize well, while fine-tuning all attention heads (LoRSU-AHH) adds unnecessary computational overhead with less effective gen￾eralization. LoRSU outperforms both of the variants in TI while LoRSU-AHH is marginally better in CC. Additional experiments th… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Instances of the ‘Use Laptop’ action. F. TSI vs. DALLE In Figures 4 through 7, we present examples of images from TSI and DALLE for different actions. In general, we observe that TSI comprised of natural, unposed images of senior individuals performing daily tasks, ref…
Figure 5
Figure 5. Figure 5: Instances of the ‘Watching TV’ action. (a) TSI (b) DALLE [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]
Figure 6
Figure 6. Figure 6: Instances of the ‘Use Tablet’ action. (a) TSI (b) DALLE [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Instances of the ‘Use a telephone’ action. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 18 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    K., Ajanthan, T., and Torr, P

    Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp.\ 532--547, 2018

    work page 2018

  3. [3]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., and Elhoseiny, M. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023

    arXiv

  4. [4]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24185--24198, 2024

    2024

  5. [5]

    Can we edit multimodal large language models? In Bouamor, H., Pino, J., and Bali, K

    Cheng, S., Tian, B., Liu, Q., Chen, X., Wang, Y., Chen, H., and Zhang, N. Can we edit multimodal large language models? In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 13877--13888, Singapore, December 2023. Association for Computational Linguistics. doi:10.18653/v...

    doi:10.18653/v1/2023.emnlp-main.856 2023

  6. [6]

    E., et al

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2 0 (3): 0 6, 2023

    2023

  7. [7]

    Knowledge neurons in pretrained transformers

    Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021

    arXiv 2021

  8. [8]

    One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering

    Das, D., Talon, D., Mancini, M., Wang, Y., and Ricci, E. One vlm to keep it learning: Generation and balancing for data-free continual visual question answering. arXiv preprint arXiv:2411.02210, 2024

    work page Pith review arXiv 2024

Show all 48 references
  1. [9]

    Toyota smarthome: Real-world activities of daily living

    Das, S., Dai, R., Koperski, M., Minciullo, L., Garattoni, L., Bremond, F., and Francesca, G. Toyota smarthome: Real-world activities of daily living. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 833--842, 2019

    2019

  2. [10]

    A continual learning survey: Defying forgetting in classification tasks

    De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44 0 (7): 0 3366--3385, 2021

    2021

  3. [11]

    Image N et: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Image N et: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  4. [12]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp.\ 11198--11201, 2024

    2024

  5. [13]

    Calibrating higher-order statistics for few-shot class-incremental learning with pre-trained vision transformers

    Goswami, D., Twardowski, B., and Van De Weijer, J. Calibrating higher-order statistics for few-shot class-incremental learning with pre-trained vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4075--4084, 2024

    2024

  6. [14]

    Sensitivity-aware visual parameter-efficient fine-tuning

    He, H., Cai, J., Zhang, J., Tao, D., and Zhuang, B. Sensitivity-aware visual parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11825--11835, 2023 a

    2023

  7. [15]

    Continual instruction tuning for large multimodal models

    He, J., Guo, H., Tang, M., and Wang, J. Continual instruction tuning for large multimodal models. arXiv preprint arXiv:2311.16206, 2023 b

    2023 arXiv

  8. [16]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019

    2019

  9. [17]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  10. [18]

    Kamoi, R., Zhang, Y., Das, S. S. S., Zhang, R. H., and Zhang, R. Visonlyqa: Large vision language models still struggle with visual perception of geometric information. arXiv preprint arXiv:2412.00947, 2024

    2024 arXiv

  11. [19]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., and Testuggine, D. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33: 0 2611--2624, 2020

    2020

  12. [20]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023

    2023

  15. [23]

    Liu, F., Emerson, G. E. T., and Collier, N. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023

    2023

  16. [24]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024 a

    2024

  17. [25]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  18. [26]

    and Ranzato, M

    Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  19. [27]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  20. [28]

    Fine-grained visual classification of aircraft

    Maji, S., Kannala, J., Rahtu, E., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. Technical report, University of Oxford, 2013

    2013

  21. [29]

    Locating and editing factual associations in gpt

    Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35: 0 17359--17372, 2022

    2022

  22. [30]

    Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=0DcZxeWfOPt

    2022

  23. [31]

    O., Aljundi, R., and Turner, R

    Panos, A., Kobe, Y., Reino, D. O., Aljundi, R., and Turner, R. E. First session adaptation: A strong replay-free baseline for class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 18820--18830, 2023

    2023

  24. [32]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in N eural I nformation P rocessing S ystems , 32, 2019

    2019

  25. [33]

    Fi LM : Visual reasoning with a general conditioning layer

    Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Fi LM : Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  26. [34]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  27. [35]

    Editable neural networks

    Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., and Babenko, A. Editable neural networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HJedXaEtvS

    2020

  28. [36]

    Y., Shrestha, R., and Kanan, C

    Srivastava, S., Harun, M. Y., Shrestha, R., and Kanan, C. Improving multimodal large language models using continual learning. arXiv preprint arXiv:2410.19925, 2024

    2024 arXiv

  29. [37]

    Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32: 0 323--332, 2012

    2012

  30. [38]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9568--9578, 2024

    2024

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [40]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  33. [41]

    L., H \"u llermeier, E., Kanan, C., Kudithipudi, D., et al

    Verwimp, E., Aljundi, R., Ben-David, S., Bethge, M., Cossu, A., Gepperth, A., Hayes, T. L., H \"u llermeier, E., Kanan, C., Kudithipudi, D., et al. Continual learning: Applications and the road forward. arXiv preprint arXiv:2311.11908, 2023

    2023 arXiv

  34. [42]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv prepr...

    2024 arXiv

  35. [43]

    Do clips always generalize better than imagenet models? arXiv preprint arXiv:2403.11497, 2024 b

    Wang, Q., Lin, Y., Chen, Y., Schmidt, L., Han, B., and Zhang, T. Do clips always generalize better than imagenet models? arXiv preprint arXiv:2403.11497, 2024 b

    2024 arXiv

  36. [44]

    Continual learning for large language models: A survey

    Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01364, 2024

    2024 arXiv

  37. [45]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  38. [46]

    Overcoming generic knowledge loss with selective parameter update

    Zhang, W., Janson, P., Aljundi, R., and Elhoseiny, M. Overcoming generic knowledge loss with selective parameter update. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24046--24056, 2024

    2024

  39. [47]

    Safe: Slow and fast parameter-efficient tuning for continual learning with pre-trained models

    Zhao, L., Zhang, X., Yan, K., Ding, S., and Huang, W. Safe: Slow and fast parameter-efficient tuning for continual learning with pre-trained models. arXiv preprint arXiv:2411.02175, 2024

    2024 arXiv

  40. [48]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.