Pith. sign in

REVIEW 4 major objections 4 minor 55 references

Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An instruction-conditioned mixture of visual projector experts lets generative vision-language models learn tasks sequentially without forgetting old instructions or losing zero-shot ability.

desk verdict Solid CL-for-VLM paper with strong gains and a genuinely new mechanism, but the cosine-similarity relevance score is the load-bearing part that needs direct validation; worth peer review after fixes. read the letter →

arxiv 2508.00260 v1 pith:W3U5GC4D submitted 2025-08-01 cs.CV cs.MM

Hyundong Jin , Hyung Jin Chang , Eunwoo Kim This is my paper
classification cs.CVcs.MM
keywords continuallearningvision-languagemodelsmixtureofexpertsvisualprojectorinstructiongroundingcatastrophicforgettingzero-shottransferexpertpruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that the instruction neglect and catastrophic forgetting seen when continually fine-tuning generative vision-language models (VLMs) on new tasks is caused by retraining a single shared visual projector, and that replacing it with a mixture of instruction-conditioned projector experts fixes the problem. The proposed framework, MVP, keeps a set of expert visual projectors and a router that selects experts from both the image and the language instruction, along with a semantic-relevance score that reuses experts from similar earlier tasks, an activation-bias term, expert pruning, and adaptive aggregation with the pre-trained projector at inference. On a sequence of 18 classification, captioning, and question-answering tasks, MVP reports higher Last, Avg, and Transfer scores than existing continual learning methods (e.g., classification Last of 82.81–87.52 versus 65.10–70.01 for MoEAdapter), while in the LLaMa-2 experiments it stays within 0.07 points of the pre-trained zero-shot SEED-Bench score.

What carries the argument

The load-bearing object is the Mixture-of-Visual Projectors (MVP): a router $R$ plus a set of $N_E$ expert visual projectors $\{E_j\}$, where the router output $W=\mathrm{Softmax}(\mathrm{Top\text{-}K}(R(x_{i,\mathrm{img}}, x_{i,\mathrm{text}})))$ determines how the expert outputs are blended with the pre-trained projector $V$ (Eqs. 1–3). The router is a multi-head self-attention block followed by a linear layer, taking concatenated projected image features and POS-filtered instruction embeddings. The continual-learning behavior is carried by four auxiliary components: (i) semantic-relevance scores computed as a sigmoid-scaled, $\alpha$-weighted combination of image and instruction cosine similarities to stored per-task means (Eq. 4); (ii) a contrastive distribution loss $L_{\mathrm{rec}}$ that pushes the router's activation pattern toward those of semantically similar previous tasks (Eq. 5); (iii) an activation-bias loss $L_{\mathrm{bias}}$ penalizing similarity to the cumulative historical activation vector (Eq. 6); and (iv) expert pruning, which learns a vector $E^t$ by matching the pruned and full expert outputs under an L1 sparsity penalty, thresholds it into a binary mask, and reinitializes zero-activation experts from the pre-trained projector (Eq. 7). Inference-time adaptive aggregation (Eq. 8) replaces the training-time fixed blend by weighting the MoE branch with $\lambda_{\mathrm{inf}}$, the maximum semantic-relevance score over stored tasks.

What would settle it

Construct a task sequence containing two tasks with visually near-identical images but opposite instructions (for example, 'Name the object' versus 'Count the objects'), and compare MVP's router decisions and Last/Transfer scores with an oracle router told the true task identity. If the Eq. 4 semantic-relevance scores for the two tasks are not separated (their $s^{t'}$ values are close for both tasks) and MVP's performance on those tasks collapses toward the single-projector baseline, the cosine-similarity proxy is the component that fails.

Watch

Extended reading notes

Core claim

The central claim is that grounding visual translation in the text instruction prevents a continually learning VLM from overwriting old knowledge and from ignoring instructions. Rather than updating one shared visual projector, MVP maintains $N_E=20$ expert projectors; a router $R$ takes the image embedding and the instruction embedding, applies Top-K ($K=2$) softmax weighting, and produces $\tilde{x} = \frac{1}{K+1}(V(x) + \sum_j w_j E_j(x))$, so the pre-trained projector $V$ remains in the loop. Three training-time mechanisms make the routing continual: a semantic-relevance score $s^{t'} = \alpha\sigma(s^{t'}_{\mathrm{img}}) + (1-\alpha)\sigma(s^{t'}_{\mathrm{text}})$ that measures cosine similarity to stored per-task means (Eq. 4); a recommendation loss aligning router activations with those of semantically related previous tasks; and an activation-bias penalty reducing overlap with cumulative historical activations. After each task, expert pruning learns a sparse binary mask via an L1-regularized output-matching objective and reinitializes never-used experts from the pre-trained projector. At inference, adaptive aggregation scales the MoE branch by the highest relevance score, $\lambda_{\mathrm{inf}} = \arg\max_{t'} s^{t'}_{\mathrm{inf}}$, preserving zero-shot behavior. The paper reports that this recipe outperforms LwF, EWC, GMM, EProj, and MoEAdapter on both Vicuna-7B and LLaMa-2-7B, with classification Last rising to 82.81/85.34/87.52 on the three Vicuna task blocks and SEED-Bench retention of 42.59 versus 42.66 zero-shot on the LLaMa-2 model.

Load-bearing premise

The load-bearing premise is that cosine similarity between the current image-and-instruction pair and stored per-task average embeddings reliably indicates which previous task's experts should be reused; if the embedding space does not separate tasks, the router's recommendation loss and the inference-time aggregation weight become miscalibrated and the claimed instruction grounding fails.

Editorial extensions

If this is right

  • On 18 sequential vision-language tasks, MVP reports higher Last, Avg, and Transfer scores than LwF, EWC, GMM, EProj, and MoEAdapter, with classification Last of 82.81–87.52 versus 65.10–70.01 for MoEAdapter on Vicuna.
  • Adaptive knowledge aggregation keeps zero-shot ability nearly intact: after all tasks, SEED-Bench is 42.59 versus 42.66 for the pre-trained LLaMa-2 model, while competing methods drop to 29.69–34.72.
  • Expert pruning prevents negative transfer: removing it drops captioning Last from 76.70 to 71.55 in the ablation, and the activation-bias term is what raises classification Last from 3.31 to 85.92.
  • The method transfers knowledge across task types: captioning performance improves slightly after training on classification tasks, indicating the instruction-grounded experts reuse cross-task knowledge rather than forgetting it.
  • Training remains parameter-efficient, updating only the router, the expert projectors, and the pruning vector, with training time about 20h48min and 23.9GB VRAM (versus 19h58min and 20.5GB for GMM), and prune/finetune stages finishing in under 25 minutes each.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper only varies instruction templates within fixed task types, an unstated extension is whether the same router separates tasks that share identical instruction text but differ only in image content; the dual similarity score in Eq. 4 suggests it could, but the paper does not dissociate the two modalities.
  • The semantic-relevance proxy is a fixed, non-learned estimator of task identity; replacing it with a learned task-identifier or a calibrated $\alpha$ (instead of the fixed 0.3) could make the method robust when stored per-task means drift as more tasks accumulate.
  • With $N_E=20$ fixed experts and pruning-to-reinit, the architecture has bounded capacity; a natural extension is to grow $N_E$ dynamically when all semantic-relevance scores are low, signaling a genuinely novel task rather than reusing or pruning experts.
  • The adaptive aggregation weight $\lambda_{\mathrm{inf}}$ takes the single best task's relevance; a softer aggregation (e.g., weighted by all relevance scores) might improve transfer on inputs that genuinely mix multiple past tasks, but the paper only tests the max.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes MVP, a continual learning method for generative vision-language models that keeps the vision encoder and LLM frozen and trains a mixture of visual projector experts selected by a router conditioned on image and instruction embeddings. An expert recommendation loss aligns router activation with a cosine-similarity relevance score between current inputs and stored per-task mean embeddings; activation bias reduction diversifies expert use; expert pruning removes redundant experts and reinitializes them; and adaptive knowledge aggregation mixes the MoE branch with the frozen pre-trained projector at inference. Experiments on an 18-task sequence comprising ImageNet-R classification, Flickr30K captioning, and COCO-QA, with Vicuna-7B and LLaMa-2-7B, report Last, Avg, and Transfer metrics, with supplementary results for a random task order, generated instructions, computational cost, and SEED-Bench.

Significance. If the reported results are reproducible, MVP is a meaningful advance: it is parameter-efficient, uses only current-task data, and appears to mitigate forgetting and preserve zero-shot performance better than single-projector and MoEAdapter baselines. The paper deserves credit for evaluating two backbones and a random task order, for ablating each component, and for reporting wall-clock time and VRAM usage. The central claim, however, is weakened by an ill-defined aggregation equation, an unvalidated relevance proxy that drives both routing and aggregation, and the absence of variance estimates or code.

major comments (4)
  1. [§3.3, Eq. (8)] In Eq. (8), λ_inf is defined as argmax_{t′∈[1,t]}{s^{t′}_inf}, but argmax returns an index, not a scalar weight in [0,1]; the equation then multiplies the MoE sum by λ_inf and divides by λ_inf·K+1. As written, the adaptive aggregation formula is not computable. Please define λ_inf as the maximum relevance score, or as another explicit function of the scores, and verify that the denominator and the ablation results correspond to that definition.
  2. [§3.2.1, Eqs. (4)–(5) and §3.3, Eq. (8)] The semantic relevance score s^{t′} of Eq. (4) is load-bearing: it serves as the target distribution for the recommendation loss in Eq. (5) and as the inference-time aggregation weight in Eq. (8). The manuscript does not provide evidence that cosine similarity between the current image/instruction embedding and stored per-task mean embeddings is a calibrated relevance measure. Figure 5 shows activation-frequency shifts at task boundaries but not embedding separation or score calibration. A direct test would be to compare the argmax of s^{t′} against the true task identity, or to ablate the method by replacing s^{t′} with oracle task labels. Supplementary Table C, where paraphrased instructions leave Last performance stable but reduce Transfer on captioning and QA, suggests the text-similarity component may not be the driving signal, which weakens the 'instruction-grounded' interpretation. This should be analyzed before the mechanism can be accepted.
  3. [Tables 1–3 and Table A] All experimental tables report a single run with no standard deviations, confidence intervals, or significance tests, and no code is released. Some margins supporting the cross-method claim are small; for example, Table 1 shows a QA Transfer gap of 36.62 vs. 36.34 over MoEAdapter, and Table 2 shows a QA Transfer gap of 57.16 vs. 56.27. The empirical central claim needs multiple seeds and, ideally, public code to rule out optimization noise.
  4. [§3.2.2] The router fine-tuning procedure is under-specified. The text says inputs are 'sampled from a normal distribution defined by the stored mean and covariance of images and text instructions from observed tasks,' but it does not state how the covariance is estimated, how many samples are drawn per task, whether the distribution is per-task or global, or how the label distribution Softmax(L^{t′} ⊙ M^{t′}) is computed. Because the pruning and fine-tuning stage is a claimed component and the ablation attributes substantial gains to it (e.g., captioning Last improves from 71.55 to 76.70 in Table 3 when pruning is added), these details are necessary for reproducibility.
minor comments (4)
  1. [§3.1] There is a typo: 'vision-lanugage task' should be 'vision-language task.'
  2. [Table 1] In the Avg block, the EProj row for question answering contains run-together numbers '40.0650.8850.65' and '42.1748.9948.65'; this formatting error makes the table hard to read and should be corrected.
  3. [Eq. (5)] The cross-entropy notation H(S^t, π^{t,t′}) is ambiguous because S^t is described as a set of scores rather than a probability distribution; please specify how S^t is normalized into a distribution over previous tasks.
  4. [Eq. (7)] The minimization over E^t does not state whether E^t is constrained to be non-negative, nor how the thresholding to the binary mask M^t is applied; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the method is evaluated on held-out tasks and the relevance score is a stated design mechanism, not a fitted target.

full rationale

I found no load-bearing circular step. The MVP training loop (Eqs. 1-8) is self-contained: expert routing is trained by cross-entropy L_ce plus recommendation and bias losses, and the semantic-relevance score s^{t'} (Eq. 4) is a stated mechanism, not a fitted constant. The same score is used as the recommendation target (Eq. 5) and as the inference aggregation weight (Eq. 8); this is an architectural design choice, and the downstream Last/Transfer/Avg numbers are measured on held-out task blocks and SEED-Bench, so the central comparison does not reduce to the score by construction. Expert pruning (Eq. 7) and the router fine-tuning with self-generated labels are internal consistency regularizers, not predictions derived from the target result. The only self-citations ([22], [23]) appear in related-work context and do not justify the method's assumptions. The argmax/lambda_inf notation in Eq. 8 is ambiguous (argmax yields an index, not necessarily a weight in [0,1]), but this is a correctness/ambiguity concern, not circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a set of hand-set hyperparameters and on modeling assumptions about embedding-space similarity and Gaussian sampling; no new physical or conceptual entities are introduced.

free parameters (7)
  • alpha = 0.3
    Balances image vs instruction similarity in Eq. 4; fixed for all experiments and central to expert recommendation and adaptive aggregation.
  • lambda_rec = 1.0
    Weight of the expert recommendation loss in the total training loss L; hand-set.
  • lambda_bias = 1.0
    Weight of the activation bias reduction loss in L; hand-set.
  • K (number of engaged experts) = 2
    Top-K routing parameter in Eq. 2; fixed following prior MoE practice.
  • N_E (number of experts) = 20
    Total visual projector experts; default used in main experiments, varied in Fig. 6.
  • tau (contrastive temperature) = not reported
    Temperature in Eq. 5 softmax; affects how sharply router activations are pushed to match semantic relevance.
  • expert pruning threshold = 1e-3
    Threshold on E_t to form binary mask M_t (Supplementary A); determines which experts are pruned.
assumptions (5)
  • domain assumption Pre-trained vision encoder, language model, and original visual projector are frozen and sufficient as base representations for continual learning.
    All methods update only projectors and the router; if frozen representations cannot support new instruction types, the comparison would not test continual projector adaptation.
  • domain assumption Stored mean image and instruction embeddings per previous task are representative prototypes for semantic relevance scoring.
    Used in Eq. 4 to compute S_t and in adaptive aggregation; no uncertainty or distribution overlap is modeled.
  • domain assumption Cosine similarity in the frozen embedding space is a reliable proxy for visual and textual task relevance.
    The recommendation loss and aggregation both use cosine similarity between router activation patterns and embeddings without learned calibration.
  • ad hoc to paper Task relevance can be summarized by a fixed sigmoid combination of image and text similarities with alpha=0.3.
    Eq. 4 is introduced specifically for this method and no justification or sensitivity analysis for alpha is given.
  • ad hoc to paper Data sampled from a normal distribution defined by stored means and covariances approximates real router fine-tuning inputs.
    Section 3.2.2 uses Gaussian sampling for router fine-tuning; covariance storage is not described in the main text and the approximation is unvalidated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models." pith.science (2026). https://pith.science/paper/W3U5GC4D

@misc{pith2026250800260,
  author       = {Pith},
  title        = {Pith review of: Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W3U5GC4D}},
  note         = {Machine review of arXiv:2508.00260}
}
read the original abstract

Continual learning enables pre-trained generative vision-language models (VLMs) to incorporate knowledge from new tasks without retraining data from previous ones. Recent methods update a visual projector to translate visual information for new tasks, connecting pre-trained vision encoders with large language models. However, such adjustments may cause the models to prioritize visual inputs over language instructions, particularly learning tasks with repetitive types of textual instructions. To address the neglect of language instructions, we propose a novel framework that grounds the translation of visual information on instructions for language models. We introduce a mixture of visual projectors, each serving as a specialized visual-to-language translation expert based on the given instruction context to adapt to new tasks. To avoid using experts for irrelevant instruction contexts, we propose an expert recommendation strategy that reuses experts for tasks similar to those previously learned. Additionally, we introduce expert pruning to alleviate interference from the use of experts that cumulatively activated in previous tasks. Extensive experiments on diverse vision-language tasks demonstrate that our method outperforms existing continual learning approaches by generating instruction-following responses.

Figures

Figures reproduced from arXiv: 2508.00260 by the authors.

Figure 1
Figure 1. (a) Generated responses for classification and captioning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the proposed continual learning framework for the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance of each task across time steps. The gray area represents the time steps preceding training for specific tasks, and the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results of the continual learning methods illustrating changes of responses to randomly selected samples after learning [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Results for MVP with varying numbers of experts. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 40 canonical work pages

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Expert gate: Lifelong learning with a network of experts

    Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017. 2

    work page 2017

  3. [3]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. InEuropean Con- ference on Computer Vision, pages 144–161. Springer, 2018. 2

    work page 2018

  4. [4]

    Generative multi-modal models are good class incremental learners

    Xusheng Cao, Haori Lu, Linlan Huang, Xialei Liu, and Ming-Ming Cheng. Generative multi-modal models are good class incremental learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28706–28717, 2024. 1, 2, 3, 5, 6, 7

    work page 2024

  5. [5]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024. 2

    work page 2024

  6. [6]

    Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajan- than, and Philip HS Torr. Riemannian walk for incremen- tal learning: Understanding forgetting and intransigence. In European Conference on Computer Vision, pages 556–572. Springer, 2018. 2

    work page 2018

  7. [7]

    CoIN: A benchmark of continual instruction tuning for multimodel large language models

    Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Jingkuan Song, and Lianli Gao. CoIN: A benchmark of continual instruction tuning for multimodel large language models. Advances in Neural Information Processing Systems, 37: 57817–57840, 2025. 1

    work page 2025

  8. [8]

    Lifelong language pretraining with distribution-specialized experts

    Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with distribution-specialized experts. InInterna- tional Conference on Machine Learning, pages 5383–5395. PMLR, 2023. 2

    work page 2023

Show all 55 references
  1. [9]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023.URL https://lmsys

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023.URL https://lmsys. org/blog/2023-03-30-vicuna, 3(5),

    2023

  2. [10]

    V ocabulary-free image classification.Advances in Neural Information Processing Systems, 36:30662–30680, 2023

    Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. V ocabulary-free image classification.Advances in Neural Information Processing Systems, 36:30662–30680, 2023. 1

    2023

  3. [11]

    Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning

    Wenliang Dai, Junnan Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning. arxiv 2023.arXiv preprint arXiv:2305.06500, 2,

    2023 arXiv

  4. [12]

    RATT: Recurrent attention to transient tasks for continual image captioning.Advances in Neural Information Processing Systems, 33:16736–16748,

    Riccardo Del Chiaro, Bartłomiej Twardowski, Andrew Bag- danov, and Joost Van de Weijer. RATT: Recurrent attention to transient tasks for continual image captioning.Advances in Neural Information Processing Systems, 33:16736–16748,

  5. [13]

    DyTox: Transformers for continual learning with dynamic token expansion

    Arthur Douillard, Alexandre Ram ´e, Guillaume Couairon, and Matthieu Cord. DyTox: Transformers for continual learning with dynamic token expansion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285–9295, 2022. 1, 2

    2022

  6. [14]

    EV A: Exploring the limits of masked visual representa- tion learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A: Exploring the limits of masked visual representa- tion learning at scale. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition,...

    2023

  7. [15]

    Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning

    Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning. InEuropean Conference on Computer Vision, pages 89–106, 2024. 2

    2024

  8. [16]

    Continual instruction tuning for large multimodal models

    Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. Continual instruction tuning for large multimodal models. arXiv preprint arXiv:2311.16206, 2023. 1, 3, 5, 6, 7

    2023 arXiv

  9. [17]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. InProceedings of the IEEE/CVF internationa...

    2021

  10. [18]

    CLIPScore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pages 7514–7528, 2021. 5

    2021

  11. [19]

    LoRA: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 2

    2021 arXiv

  12. [20]

    Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991. 2

    1991

  13. [21]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. InEuropean Conference on Computer Vision, pages 709–727. Springer, 2022. 2

    2022

  14. [22]

    Helpful or harmful: Inter- task association in continual learning

    Hyundong Jin and Eunwoo Kim. Helpful or harmful: Inter- task association in continual learning. InEuropean confer- ence on computer vision, pages 519–535. Springer, 2022. 1

    2022

  15. [23]

    Growing a brain with sparsity-inducing genera- tion for continual learning

    Hyundong Jin, Gyeong-hyeon Kim, Chanho Ahn, and Eun- woo Kim. Growing a brain with sparsity-inducing genera- tion for continual learning. InProceedings of the IEEE/CVF international conference on computer vision, pages 18961– 18970, 2023. 1

    2023

  16. [24]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. InProceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015. 1

    2015

  17. [25]

    Overcoming catastrophic forgetting in neu- ral networks.Proceedings of the National Academy of Sci- ences of the United States of America, 114(13):3521–3526,

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks.Proceedings of the National Academy of Sci...

  18. [26]

    Quantifying the carbon emissions of machine learning.arXiv preprint arXiv:1910.09700, 2019

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning.arXiv preprint arXiv:1910.09700, 2019. 1

    1910 arXiv

  19. [27]

    GShard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. InIn- ternational Conference on Learning Representations. 4, 7

  20. [28]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023. 8, 3

    2023 arXiv

  21. [29]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 2

    2023

  22. [30]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelli- gence, 40(12):2935–2947, 2017. 5, 6, 7, 1

    2017

  23. [31]

    InfLoRA: Interference-free low-rank adaptation for continual learning

    Yan-Shuo Liang and Wu-Jun Li. InfLoRA: Interference-free low-rank adaptation for continual learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024. 1

    2024

  24. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2

    2024

  25. [33]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 1

    2024

  26. [34]

    Adaptive aggregation networks for class-incremental learning

    Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2544–2553, 2021. 2

    2021

  27. [35]

    Unified-IO 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling autoregressive multimodal models with vision language audio and action. InProceed- ings of the IEEE/CVF Conference on Computer Vision a...

    2024

  28. [36]

    Not all ex- perts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all ex- perts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. InProceedings of the 62nd Annual Meeting of the Association for Compu...

    2024

  29. [37]

    Catastrophic inter- ference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic inter- ference in connectionist networks: The sequential learning problem. InPsychology of learning and motivation, pages 109–165. Elsevier, 1989. 1, 2

    1989

  30. [38]

    Multimodal contrastive learn- ing with limoe: the language-image mixture of experts.Ad- vances in Neural Information Processing Systems, 35:9564– 9576, 2022

    Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learn- ing with limoe: the language-image mixture of experts.Ad- vances in Neural Information Processing Systems, 35:9564– 9576, 2022. 2

    2022

  31. [39]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models.Interna- tional Journal of Computer Vision, 123:74–93, 2017

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models.Interna- tional Journal of Computer Vision, 123:74–93, 2017. 2, 5

    2017

  32. [40]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  33. [41]

    iCaRL: Incremental clas- sifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental clas- sifier and representation learning. InProceedings of the IEEE conference on Computer Vision and Pattern Recogni- tion, pages 2001–2010, 2017. 1, 2

    2001

  34. [42]

    Exploring models and data for image question answering.Advances in neural information processing systems, 28, 2015

    Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering.Advances in neural information processing systems, 28, 2015. 1, 2, 5

    2015

  35. [43]

    Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer.arXiv preprint arXiv:1701.06538, 2017. 4, 7

    2017 arXiv

  36. [44]

    CODA-prompt: Con- tinual decomposed attention-based prompting for rehearsal- free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-prompt: Con- tinual decomposed attention-based prompting for rehearsal- free continual learning. InProceedings of the I...

    2023

  37. [45]

    Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in med- ical images

    Yu Tian, Guansong Pang, Fengbei Liu, Yuanhong Chen, Seon Ho Shin, Johan W Verjans, Rajvinder Singh, and Gus- tavo Carneiro. Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in med- ical images. InMedical Image Computing and Comp...

    2021

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2, 5, 7

    2023 arXiv

  39. [47]

    DualPrompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. DualPrompt: Complementary prompting for rehearsal-free continual learning. InEuropean Conference on Computer Vision, pages 631–648. Springer,

  40. [48]

    Learning to prompt for con- tinual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for con- tinual learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149,

  41. [49]

    mPLUG-OWl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-OWl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  42. [50]

    Boosting continual learning of vision-language models via mixture-of-experts adapters

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23219–23230, 2024...

    2024

  43. [51]

    Contin- ual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. InProceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3987–3995, 2017. 2

    2017

  44. [52]

    Investigating the catastrophic for- getting in multimodal large language models

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic for- getting in multimodal large language models. InNeurIPS 2023 Workshop on Instruction Tuning and Instruction Fol- lowing. 2

    2023

  45. [53]

    Prompt-aware adapter: Towards learning adaptive visual tokens for multimodal large language models.arXiv preprint arXiv:2405.15684, 2024

    Yue Zhang, Hehe Fan, and Yi Yang. Prompt-aware adapter: Towards learning adaptive visual tokens for multimodal large language models.arXiv preprint arXiv:2405.15684, 2024. 5

    2024 arXiv

  46. [54]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xi- angyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19125–19136, 2023. 5

    2023

  47. [55]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 1, 2, 5 Instruction-Grounded Visual Projectors for Continual Learning of Generat...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.