Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

LLaVA-c: Continual Improved Visual Instruction Tuning

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that sequential task-by-task training of a vision-language model can match or beat training on all tasks at once.

desk verdict Solid, practical continual learning recipe for LLaVA-1.5; the headline parity claim is confounded by a LoRA-vs-full-finetuning mismatch between their method and the multitask baseline. read the letter →

arxiv 2506.08666 v2 pith:3IS7HXJD submitted 2025-06-10 cs.CV

Wenzhuo Liu , Fei Zhu , Haiyang Guo , Longhui Wei , Cheng-Lin Liu This is my paper
classification cs.CV
keywords continuallearningmultimodallargelanguagemodelsvisualinstructiontuningcatastrophicforgettingmodelmixingsingularvaluedecompositionLLaVA-1.5basedegradation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the standard way of training vision-language models, mixing every task's data into one joint batch, is not necessary for good performance. On LLaVA-1.5 it proposes LLaVA-c, a continual-learning variant with two modifications: spectral-aware consolidation (SAC) rescales the singular values of each parameter update so old knowledge is kept while new tasks are learned, and unsupervised inquiry regularization (UIR) anchors the model's hidden text representations on unlabeled questions so it does not collapse into overfitting task-specific instructions. In continual pretraining on six domains of the LLaVA-665k dataset and continual fine-tuning on five unseen tasks, the reported numbers match or exceed multitask joint learning on general benchmarks such as MME and POPE. The authors claim this is the first time task-by-task continual learning has matched or surpassed multitask joint learning in this setting.

What carries the argument

The load-bearing object is the parameter update $\Delta\theta = \theta_t - \theta_{t-1}$, decomposed by singular value decomposition into $\Delta\theta = U\Sigma V^{\top}$. Model mixing scales every singular value by the same $\alpha$; SAC replaces this with a sliding-window average over singular values, so directions with large singular values receive small scaling factors while directions with small singular values are preserved, and the window size $k$ is chosen by minimizing $(\tfrac{1}{k}\sum_{i=1}^{k+1} \sigma_i - \alpha\sigma_1)^2$. UIR adds to the training loss the term $L_t^{\mathrm{uir}} = \mathbb{E}_{x\sim D_{\mathrm{inquiry}}} \|h[f_{\theta_t}(x)] - h[f_{\theta_{t-1}}(x)]\|_2$, an L2 distance between current and previous final hidden states on unlabeled inquiry text. The two mechanisms work together: SAC expands the mixing-ratio region where old and new task performance are both high, and UIR prevents the language-feature drift that would otherwise degrade the base model.

What would settle it

Run the continual-pretraining protocol with the same $\alpha$ on several random permutations of the six LLaVA-665k domains and compare SAC to uniform ModelMix at matched per-task accuracy; if SAC never produces an ideal region, where old-task accuracy is improved while new-task accuracy is maintained, or if the parity with joint training breaks on any single order, the central claim is falsified. A second decisive check is to replace the unsupervised inquiry set with random text and measure whether UIR's benefit disappears, which would show the anchor must be semantically meaningful rather than merely a feature-space stabilizer.

Watch

Extended reading notes

Core claim

The central claim is that catastrophic forgetting in visual instruction tuning is not an unavoidable cost of sequential training: a carefully consolidated model can end up equal to or better than one trained on all tasks simultaneously. The paper identifies two separate failure modes. Direct model mixing (averaging old and new weights) suffers from task imbalance, because a single uniform scale factor $\alpha$ cannot both keep old knowledge and absorb new knowledge; SAC addresses this by scaling each singular direction of the parameter update $\Delta\theta$ separately, using a sliding-window average that down-weights large singular values while leaving small ones nearly intact. The second failure mode, base model degradation, is overfitting to the phrasing of a new task (for example, always answering in a single word), which destroys general instruction-following; UIR prevents this by adding an L2 penalty between the current and previous model's final hidden states on unlabeled inquiry text, without touching multimodal training data. With both mechanisms, the final consolidated model reports general benchmarks at or above the multitask LLaVA-1.5 baseline and task-specific averages of 58.3 versus 24.9 for zero-shot on the fine-tuning suite, with LLaVA-Bench retained at about 97% of its original score.

Load-bearing premise

The load-bearing premise is that a sliding-window average of the singular values of each parameter update cleanly separates old-task knowledge from new-task learning, with the window size chosen by a heuristic objective and the single mix ratio $\alpha = 0.2$ set from validation curves.

Editorial extensions

If this is right

  • Continual pretraining on six visual-instruction domains reaches or exceeds multitask joint pretraining on MME (1502.4 vs 1483.5), POPE, MMBench, and SEED-Bench, so sequential training can replace data mixing for building a general multimodal model.
  • Continual fine-tuning on five unseen tasks raises average task accuracy from 24.9 (zero-shot) to 58.3 while keeping general instruction-following at roughly 97% of the original LLaVA-Bench score.
  • Because UIR uses unlabeled text only, it does not suppress new-task learning: the LLaVA loss curve stays aligned with the vanilla curve, unlike replay- or KL-based methods that trade new-task accuracy for old-task retention.
  • SAC adds no new hyperparameters: the window size is solved from the target mix ratio $\alpha$ (default 0.2), and exploring $\alpha$ over the whole range costs minutes on CPU, in contrast to data-mixing search which scales with the number of tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the parity result transfers to larger backbones and full fine-tuning, the practical implication is that model capability expansion no longer requires retaining or re-mixing the original training data; the paper explicitly leaves this scaling question open.
  • The spectral-windowing rule suggests a generic post-hoc consolidation recipe for any fine-tuned model, independent of the training objective; a natural test is whether SAC applied to non-instruction-tuned adaptations, such as standard fine-tuning on classification, shows the same task-balance improvement.
  • A stress test worth running is changing the task order: the paper's claim that large singular values carry old knowledge is most plausible when consecutive tasks are diverse, so orders with highly similar tasks, such as two VQA-style datasets back to back, may narrow or eliminate the reported ideal region.
  • The reported MME gains during continual pretraining hint that sequential training may act as a regularizer that joint mixing does not provide; if confirmed, this would invert the usual assumption that joint training is the ceiling for multitask models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LLaVA-c, a continual learning method for visual instruction tuning of LLaVA-1.5, combining Spectral-Aware Consolidation (SAC), a singular-value-based parameter mixing rule, and Unsupervised Inquiry Regularization (UIR), an auxiliary loss that stabilizes hidden representations on unlabeled text inquiries. The method is evaluated under continual pretraining (six domains from LLaVA-665k) and continual fine-tuning (five downstream tasks), with general benchmarks such as MME, MMBench, POPE, and task-specific benchmarks. The central claim is that task-by-task continual learning with LLaVA-c can match or surpass multitask joint learning. The paper includes ablation studies, comparisons against existing CL methods, full data-replay, and multi-backbone experiments.

Significance. If the headline claim holds, the paper would be significant: it offers a practical recipe for extending an MLLM to new tasks without retraining on all previous data, while preserving general instruction-following ability. The paper also draws attention to 'base model degradation' during continual instruction tuning and provides a simple regularization mechanism (UIR) to mitigate it. The experiments cover both general and task-specific evaluation, and the proposed method appears to give large improvements over existing continual baselines. The method is lightweight and the paper promises code release, which would aid reproducibility. The main significance is conditional on the comparison to the multitask baseline being fair, because the claimed parity with joint training is central to the paper's contribution.

major comments (3)
  1. [Section 4 (Limitations) and Appendix A.1, Table 2] The central claim that continual learning 'matches or surpasses multitask joint learning' is confounded by a LoRA/full fine-tuning mismatch. The proposed method uses LoRA (Appendix A.1, Table 7: LoRA R=128, alpha=256), while the multitask baseline LLaVA-1.5† in Table 2 appears to be the standard full fine-tuned LLaVA-1.5 model from [1]. The Limitations section states 'adopt LoRA instead of full fine-tuning... merge LoRA parameters after each task to maintain structural consistency,' but no control is reported where the joint/multitask baseline uses the identical LoRA configuration. The observed superiority of LLaVA-c on MME and POPE may therefore stem from LoRA's implicit regularization rather than from SAC+UIR. This issue also affects Table 11, where the 'Data-replay (all)' baseline is not clearly stated to use the same LoRA setup. The authors should provide a multitask joint baseline trained with the same LoRA configuration, and state clearly for every baseline whether LoRA is used.
  2. [Section 4.1, Eqs. (6)-(7), and Figure 4] SAC is a heuristic spectral scaling rule whose key hyperparameter, the mix ratio alpha, is chosen via validation sweeps (Figure 4 reports alpha=0.2 by default for LLaVA-1.5c). Eq. (7) determines the window size k by minimizing a squared difference between a top-singular-value average and alpha*sigma_1; this objective has no derivation or theoretical justification for preserving old-task knowledge, and the algorithm recomputes the same k for every i (Algorithm 1, line 9). The paper's claim of 'no extra hyperparameters' is only relative to ModelMix and does not address the sensitivity of the headline result to alpha. The authors should report results across a range of alpha values (at least for the main experiments, not only ablation curves), or provide a principled selection criterion.
  3. [Section 4.2, Eq. (8), and Appendix A.2] The unsupervised inquiry dataset D_inquiry is not sufficiently specified for reproducibility. The text says TQA questions are extracted as the inquiry set in continual pretraining, but it does not state the number of inquiries, whether the same set is used at every task stage and in continual fine-tuning, how the text-only inquiries are fed into the multimodal model (e.g., what image, if any, is paired with the text), or whether the inquiry set overlaps with the task training data. Without these details, the UIR loss in Eq. (8) cannot be reproduced, and its effect may be sensitive to the choice of inquiry text. Please provide the exact inquiry set construction and usage protocol.
minor comments (6)
  1. [Reference [16] and Section 5] The paper cites LWF (Learning without Forgetting) as reference [16], but the entry listed is 'Learning visual n-grams from web data' by Li et al., which is a different paper. The correct citation is to Zhizhong Li and Derek Hoiem, 'Learning without Forgetting', CVPR 2017. This is a load-bearing reference for a baseline method and should be fixed.
  2. [Figure 2] The text in Figure 2 contains scrambled or encoded strings (e.g., '6TFS', '0VST', '$P*/') instead of readable example queries and responses. As printed, the figure cannot be inspected by the reader and does not support the claimed qualitative demonstration of base model degradation.
  3. [Section 4.1, Eq. (6)] Eq. (6) has a redundant nested diag and unclear summation indices: g(Sigma) is written as diag(1/k diag(Sigma_i, ...)). Please define g(Sigma) directly as diag( e_sigma_1, ..., e_sigma_r ) with e_sigma_i = (1/k) sum_{j=i}^{i+k-1} sigma_j (or the stated padding). The current notation makes the sliding-window operation difficult to parse.
  4. [Algorithm 1, line 9] The arg min in Algorithm 1 uses sum_{j=1}^{min(k+1,r)} sigma_j, which is independent of i, while the text says the window size is applied per spectral component. Clarify whether k is a single global value or is recomputed per i, and align the pseudocode with Eq. (6).
  5. [Throughout] The method name is inconsistently written as LLaVA-c, LLaVA-1.5c, and LLaVA-1.5 c. Please standardize the notation to avoid confusion with the baseline LLaVA-1.5.
  6. [Section 5, Tables 2 and 4] The paper reports single runs without error bars or multiple seeds. Given that continual learning results are known to be sensitive to task order and initialization, reporting variance or at least stating the number of runs would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the continual-learning comparison is empirical and the method's components are not defined in terms of the target result.

full rationale

The paper's central claim is an empirical comparison claim: LLaVA-c (SAC + UIR) trained task-by-task matches or surpasses the multitask-trained LLaVA-1.5 on hold-out benchmarks. Nothing in the derivation defines the reported benchmark scores in terms of the method's inputs. SAC (Eq. 6-7) is a post-hoc spectral rescaling of the parameter update Δθ after each task, with the sliding-window size k fit to the heuristic objective in Eq. 7; α is set to 0.2 by default and validated by the α-sweeps in Figure 4. This is ordinary hyperparameter selection, not a fitted parameter renamed as a prediction. UIR (Eq. 8) is an explicit regularization term that pulls current-task hidden representations toward the previous model's representations for unlabeled inquiries; using the previous model as a target is the stated mechanism of the regularizer, not a hidden circular dependency. The comparison to the multitask baseline is not circular: the baseline is LLaVA-1.5 reproduced from [1] and evaluated on the same benchmarks, and the paper's claimed improvement is an outcome that could in principle fail. The concern raised about LoRA versus full fine-tuning is an experimental-control/fairness issue, not a case where the result reduces to its inputs by construction. The only self-citation (ref. [9], by overlapping authors) is used to note that base-model degradation is observed in other methods; this is peripheral and not load-bearing. Therefore no circular step can be exhibited, and the appropriate finding is no significant circularity (score 0).

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method introduces no new entities, but rests on two domain assumptions: that SVD of parameter updates yields task-separable directions, and that stabilizing hidden states on a small set of unlabeled text inquiries prevents base model degradation. One hyperparameter, alpha=0.2, is fitted via validation sweeps.

free parameters (1)
  • SAC mix ratio alpha = 0.2
    Selected via validation sweeps on IconQA and ArxivQA (Figure 4); controls the overall scale of parameter update retained after SVD scaling.
assumptions (3)
  • domain assumption Singular value decomposition of the parameter update exposes task-relevant directions that can be selectively scaled.
    Section 4.1 states the spectral view and motivates SAC; no proof or independent evidence is provided.
  • domain assumption L2 distance of final-layer hidden states on unlabeled inquiries reflects base model degradation.
    Section 4.2 and Figure 6 show this correlation for one example; it is assumed to generalize.
  • standard math LLaVA-1.5 training recipe and 665k dataset are used as-is, including their hyperparameters and splits.
    The method builds directly on LLaVA-1.5 [1] without re-verifying the base model's own claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVA-c: Continual Improved Visual Instruction Tuning." pith.science (2026). https://pith.science/paper/3IS7HXJD

@misc{pith2026250608666,
  author       = {Pith},
  title        = {Pith review of: LLaVA-c: Continual Improved Visual Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IS7HXJD}},
  note         = {Machine review of arXiv:2506.08666}
}
read the original abstract

Multimodal models like LLaVA-1.5 achieve state-of-the-art visual understanding through visual instruction tuning on multitask datasets, enabling strong instruction-following and multimodal performance. However, multitask learning faces challenges such as task balancing, requiring careful adjustment of data proportions, and expansion costs, where new tasks risk catastrophic forgetting and need costly retraining. Continual learning provides a promising alternative to acquiring new knowledge incrementally while preserving existing capabilities. However, current methods prioritize task-specific performance, neglecting base model degradation from overfitting to specific instructions, which undermines general capabilities. In this work, we propose a simple but effective method with two modifications on LLaVA-1.5: spectral-aware consolidation for improved task balance and unsupervised inquiry regularization to prevent base model degradation. We evaluate both general and task-specific performance across continual pretraining and fine-tuning. Experiments demonstrate that LLaVA-c consistently enhances standard benchmark performance and preserves general capabilities. For the first time, we show that task-by-task continual learning can achieve results that match or surpass multitask joint learning. The code will be publicly released.

Figures

Figures reproduced from arXiv: 2506.08666 by the authors.

Figure 1
Figure 1. Results of training LLaVA￾1.5 on LLaVA-665k [1]. For the first time, we demonstrate that task-by-task continual learning can outperform joint learning on six tasks (e.g., coco, ocr). Large Language Models (LLMs) like GPT [2] and LLaMA [3] excel in understanding instructions and generating ac￾curate responses. To handle real-world multimodal data, which combines text, images, and other formats, the devel￾opment of Mu… view at source ↗
Figure 2
Figure 2. Test examples on a recent MLLM-related continual learning method CoIN [ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of LLaVA-c, which extends LLaVA-1.5 with two modifications: [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Impact of ModelMix ratio α in fine-tuning LLaVA-1.5 on IconQA and ArxivQA. SAC improves task balance within [0, α1], while combining UIR and SAC yields the best performance, forming an Ideal Region [˜α2, α˜1] for continual learning. 4.1 Spectral-Aware Consolidation A p…
Figure 5
Figure 5. Figure 5: Spectral comparison of ModelMix and SAC. ModelMix applies uniform scaling to all eigenvectors, whereas SAC adaptively amplifies directions with larger eigenvalues. 0 1000 2000 3000 Iterations 0.0 0.2 0.4 0.6 0.8 1.0 Value Language Feature Shift w/o UIR w/ UIR 0 1000 20…
Figure 7
Figure 7. Figure 7: Comparison of general performance after con [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparison of data replay and forward KL loss in a 1-step incremental learning experi￾ment on ArxivQA. Vanilla Our LWF EWC 0 1 2 3 4 5 6 PFLOPs Computational Cost Vanilla Our LWF EWC 0 2 4 6 8 10 12 Samples/sec Processing Speed [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: α-performance curves for intermediate tasks under continual fine-tuning. Compared to multi-task data mixing, SAC-coefficient tuning uses a single parameter, operates on CPU with minimal overhead, and can explore the entire range in minutes. Data mixing, however, adds …

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GUI-AC: Enhancing Continual Learning in GUI Agents

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.

  2. SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    SAME stabilizes MoE-based continual instruction tuning of vision-language models by constraining router updates to task-relevant subspaces and scaling expert updates with historical input covariance, achieving 66.82% ...

  3. Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Reinforcement fine-tuning largely prevents catastrophic forgetting during continual post-training of a multimodal LLM, while supervised fine-tuning degrades both task and general performance.

  4. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages · cited by 4 Pith papers

  1. [1]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  2. [2]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [3]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  4. [4]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  5. [5]

    Learning transferable visual models from natural language supervision.arXiv preprint arXiv:2103.00020, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision.arXiv preprint arXiv:2103.00020, 2021

    arXiv 2021

  6. [6]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017

    2017

  7. [7]

    CoIN: A benchmark of continual instruction tuning for multimodel large language models

    Cheng Chen, Junchen Zhu, Xu Luo, Heng Tao Shen, Jingkuan Song, and Lianli Gao. CoIN: A benchmark of continual instruction tuning for multimodel large language models. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    work page 2024

  8. [8]

    Orthogonal subspace learning for language model continual learning

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10658–10671, 2023

    2023

Show all 38 references
  1. [9]

    Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models.arXiv preprint arXiv:2410.05849, 2024

    Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models.arXiv preprint arXiv:2410.05849, 2024

    2024 arXiv

  2. [10]

    Palm: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311, 2022

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311, 2022

    2022 arXiv

  3. [11]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  4. [12]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiangyang Wang, and Ming Ding. Minigpt-4: Enhancing vision- language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  5. [13]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    2023 arXiv

  6. [14]

    Flamingo: a visual language model for few-shot learning.arXiv preprint arXiv:2204.14198, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.arXiv preprint arXiv:2204.14198, 2022

    2022 arXiv

  7. [15]

    Emu: Efficient multimodal pretraining with unified masked prediction.arXiv preprint arXiv:2305.04522, 2023

    Chen Sun, Cordelia Schmid, Yuxiong He, Hugo Touvron, Armand Joulin, and Antoine Miech. Emu: Efficient multimodal pretraining with unified masked prediction.arXiv preprint arXiv:2305.04522, 2023

    2023 arXiv

  8. [16]

    Learning visual n-grams from web data

    Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. InICCV, 2017

    2017

  9. [17]

    Podnet: Pooled outputs distillation for small-tasks incremental learning

    Arthur Douillard, Matthieu Cord, Charles Ollion, et al. Podnet: Pooled outputs distillation for small-tasks incremental learning. InProceedings of the European Conference on Computer Vision, pages 86–102, 2020

    2020

  10. [18]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149, 2022. 10

    2022

  11. [19]

    Codaprompt: Co-designing prompt tuning and architecture for continual learning

    Alexander Smith, Yifan Liu, Hanzhang Peng, Dejing Dou, et al. Codaprompt: Co-designing prompt tuning and architecture for continual learning. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  12. [20]

    Kolesnikov, Georg Sperl, and Christoph H

    Sylvestre-Alvise Rebuffi, A. Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5533–5542, 2017

    2017

  13. [21]

    Wu, Yan-Jia Chen, Lijuan Wang, et al

    Y . Wu, Yan-Jia Chen, Lijuan Wang, et al. Large scale incremental learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374–382, 2019

    2019

  14. [22]

    R-dfcil: Relation-guided representation learning for data-free class incremental learning

    Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation-guided representation learning for data-free class incremental learning. InProceedings of the European Conference on Computer Vision, pages 423–439. Springer, 2022

    2022

  15. [23]

    Der: Dynamically expandable representation for class incremental learning

    Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2021

    2021

  16. [24]

    Foster: Feature boosting and compression for class-incremental learning

    Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. InProceedings of the European Conference on Computer Vision, 2022

    2022

  17. [25]

    Dense network expansion for class incremental learning

    Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11858–11867, 2023

    2023

  18. [26]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  19. [27]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  20. [28]

    Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  21. [29]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  22. [30]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  23. [31]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  24. [32]

    Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning

    Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2023

  25. [33]

    Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. InThe 35th Conference on Neural Information Processing Systems (NeurIPS 2021) T...

    2021

  26. [34]

    Clevr-math: A dataset for compositional language, visual and mathematical reasoning

    Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. InInternational Joint Conference on Learning and Reasoning, 16th International Workshop on Neural-Symbolic Learning and Reasoning (NeSy 2022), ...

    2022

  27. [35]

    Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meet...

  28. [36]

    Figureqa: An annotated figure dataset for visual reasoning.arXiv preprint arXiv:1710.07300, 2017

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning.arXiv preprint arXiv:1710.07300, 2017

    2017 arXiv

  29. [37]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 12 A Append...

    2023

  30. [2024]

    Association for Computational Linguistics. 11

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.