Pith. sign in

REVIEW 4 major objections 4 minor 60 references

The paper claims that continual instruction tuning of multimodal LLMs is limited mainly by drift in the vision-language projector, and that a progressive expansion of projector experts anchored to the original projector fixes it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:18 UTC pith:ENR67QO2

load-bearing objection Solid, well-scoped MCIT paper with a plausible new mechanism; the expansion trigger is tuned rather than validated, but the core method holds up under ablation. the 4 major comments →

arxiv 2607.26947 v2 pith:ENR67QO2 submitted 2026-07-29 cs.CV cs.AI

Progressive Multimodal Alignment for Continual Instruction Tuning

Duzhen Zhang , Yahan Yu , Qiaoyi Su , Jiahua Dong , Tielin Zhang This is my paper
classification cs.CV cs.AI
keywords multimodal continual instruction tuningprojector-level forgettingparameter-efficient fine-tuningmixture of expertsvision-language alignmentcatastrophic forgettingLLaVAInternVL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the main overlooked failure in Multimodal Continual Instruction Tuning is projector-level forgetting: the module that maps visual features into language space drifts as tasks are trained sequentially, corrupting earlier cross-modal alignments even when the LLM backbone retains its skills. The authors propose Progressive Multimodal Alignment (PMA), a method-agnostic add-on that detects whether a new task's fused image-plus-instruction representations differ from previous ones, expands a small set of projector experts only when such a shift is detected, and routes each input through the experts without needing task labels. A frozen copy of the original pretrained projector is kept as a stable anchor, so the model always has access to the pretrained alignment. PMA, when combined with existing backbone-level continual learning methods HiDE and DISCO, improves final retained accuracy and reduces backward transfer on the UCIT and MLLM-DCL benchmarks across LLaVA-1.5-7B and InternVL-Chat-7B. The paper claims that this shows projector-level adaptation is a necessary complement to backbone-level continual learning, not an optional extra.

Core claim

The central claim is that projector-level forgetting is a distinct, measurable source of catastrophic forgetting in MCIT, separate from LLM backbone forgetting. PMA treats the projector as the locus of adaptation: for each new task, a lightweight descriptor autoencoder decides whether existing projector experts can reconstruct the new task's fused visual-instruction representation; if fewer than 60% of new-task samples are within 1.4 standard deviations of every previous descriptor, a new projector expert is added. An expandable router soft-mixes experts based on the same fused features, so no task ID is required at inference, and the pretrained projector is always retained as a frozen ancho

What carries the argument

The load-bearing mechanism is the Representation Descriptor (RD), a small MLP autoencoder trained on the concatenation of the global visual feature and the averaged instruction-token embedding. Its reconstruction error, standardized as a z-score against the statistics from the task it was trained on, is the signal that decides expansion versus reuse: if all prior RDs judge fewer than 60% of new-task samples as 'familiar' (z ≤ 1.4), a new projector expert is added; otherwise the most compatible prior expert is reused. The expandable router is a linear layer with softmax that outputs mixing weights from the same fused representation, and the final projected visual feature is a weighted average

Load-bearing premise

The whole expansion mechanism assumes that the reconstruction error of a small autoencoder trained on fused image-plus-instruction features reliably indicates whether the current projector experts can align a new task's multimodal distribution without drift; if that signal is noisy or misaligned with actual projector degradation, the expansion decisions, expert reuse, and claimed sub-linear growth lose their foundation.

What would settle it

Compare PMA's expansion decisions against a direct measure of projector drift: for each new task, compute the change in the projection of held-out images from earlier tasks through the current (fine-tuned) projector versus the pretrained projector, and check whether expansion is triggered exactly when this drift metric exceeds some threshold. If the descriptor's reconstruction error does not correlate with this drift metric across tasks and seeds, the mechanism's core signal is not doing the claimed work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If PMA is right, any PEFT-based MCIT method that only modifies the LLM backbone leaves a ceiling on performance; treating the projector as a first-class continual learning target should be part of standard MCIT pipelines.
  • The sub-linear growth property means that, for long task sequences, the parameter overhead of preserving alignment remains bounded rather than scaling one-to-one with tasks, making projector-level adaptation feasible for many-task deployments.
  • The fact that later tasks reuse earlier experts (e.g., Flickr30k reusing the VizWiz expert, IconQA reusing ArxivQA) suggests that cross-task transfer in multimodal alignment can be exploited without task IDs, potentially lowering the amount of adaptation data needed for related tasks.
  • The BWT improvements imply that projector drift is a significant contributor to the instruction-style collapse observed in MCIT (models answering classification prompts with caption-style responses); eliminating it may improve robustness to instruction variations in deployed MLLMs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One could test whether the descriptor's reconstruction error is really measuring alignment quality by comparing it against direct measures of projection drift (e.g., distance between the projected visual features under the pretrained projector and the fine-tuned projector on held-out data from earlier tasks) — the paper does not report such a correlation, and the entire expansion policy rests on i
  • The 60% threshold and τ=1.4 are set by the authors; a sensitivity analysis across more τ values is shown only for MAA, not for MFN/BWT, so the robustness of the expansion policy to these hyperparameters across diverse task orders and domains is an open question.
  • PMA's routing uses only global average visual features and instruction embeddings; extending the descriptor to operate on token-level or region-level features could make shift detection more sensitive to fine-grained visual changes, a direction the paper does not explore.
  • The method assumes the pretrained projector is a sufficiently good anchor for all tasks; if a task's distribution is far from pretraining (e.g., medical or remote-sensing imagery), the frozen anchor may contribute unhelpful signal, suggesting a possible extension where the anchor is also adapted slowly rather than frozen.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper identifies projector-level forgetting, i.e., drift in the vision-language projector during multimodal continual instruction tuning (MCIT), as an overlooked source of catastrophic forgetting. It proposes Progressive Multimodal Alignment (PMA), a method-agnostic add-on that maintains a frozen pretrained projector as an anchor, trains lightweight MLP-autoencoder Representation Descriptors (RDs) on fused image+instruction features, expands a set of projectors only when reconstruction-error z-scores fall below a threshold for fewer than 60% of samples across all prior descriptors, and routes inputs to the resulting experts through an expandable soft router without task IDs. PMA is integrated with HiDE and DISCO on the UCIT and MLLM-DCL benchmarks with LLaVA-1.5-7B and InternVL-Chat-7B, reporting consistent gains in MFT/MFN/MAA/BWT and sub-linear projector growth (e.g., 3 experts for 6 tasks).

Significance. If the reported gains hold, the paper addresses a genuinely underexplored component of MCIT: prior methods almost exclusively target LLM-side forgetting and treat the projector as shared, so a projector-specific mechanism that can be added to existing PEFT-based approaches would be a useful contribution. The paper's strengths are its clean treatment of PMA as an add-on, the breadth of benchmarks and backbones, component-level ablations, and a promised code release. However, the central mechanism is not yet validated to the standard the claims require: the RD trigger is an unverified proxy, the threshold is tuned on the same benchmark used for the headline results, and the experimental comparisons lack variance information. These issues are load-bearing because the sub-linear growth and expert-reuse behavior are driven by the RD trigger and threshold choice.

major comments (4)
  1. [§3.2–3.3, Eq. (3)–(4)] The expansion trigger is the load-bearing component of the method, but the paper provides no evidence that reconstruction error of an MLP autoencoder on x_fuse is a reliable indicator of projector-level mismatch. Since x_fuse concatenates a global visual feature with an averaged instruction embedding, while each projector P_j operates only on visual features, the reconstruction error can be dominated by instruction-text variation; a task with novel wording but familiar visual content could spuriously trigger expansion, and vice versa. No analysis separates these factors, and no baseline compares the RD trigger against a fixed expansion schedule (e.g., expand every k tasks). Please add such an ablation, or otherwise demonstrate that RD decisions track the actual projector drift rather than text-level novelty.
  2. [§5.3, Fig. 3] The threshold τ=1.4 is selected by maximizing MAA on UCIT and then used to report the UCIT main results (Tables 1–2). This makes the '3 experts, sub-linear growth' result partly a tuning artifact rather than an independent prediction. The 60% coverage rule is likewise hand-set and is not swept in Fig. 3. The paper should either evaluate the threshold on a held-out validation split, report a τ sweep on MLLM-DCL as well, or otherwise show that the expansion decisions are robust to reasonable choices of τ and the 60% rule.
  3. [Tables 1–4] All results appear to come from a single run with no error bars or seed variance. Several headline improvements are small (e.g., HiDE+PMA on UCIT-LLaVA MFT 72.50 vs 70.24; DISCO+PMA on MLLM-DCL-InternVL BWT -2.62 vs -4.24), and the re-implemented DISCO* is often worse than the published DISCO (e.g., Table 2 MFT 73.70 vs 78.92; Table 3 MFN 59.24 vs 60.33). Without multiple seeds and a clarification of why the reimplementation differs from the published numbers, it is difficult to assess whether the improvements are statistically meaningful and fair. Please report means and standard deviations over at least three seeds.
  4. [Eq. (5)] The final projected representation in Eq. (5) is not a normalized mixture. Because the router weights satisfy ∑_j w_j = 1, the coefficients in 1/(1+t') [P0 + ∑_j w_j P_j] sum to 2/(1+t'), which equals 1 only when t'=1. Thus the scale of the projected feature shrinks as experts are added. This confounds the ablation study: 'Always Expand' (t'=6) uses a different output scale than 'No Expansion' (t'=1), so the measured degradation in the Always Expand variant may be partly due to this scaling artifact rather than to reduced cross-task sharing. If a normalized mixture is intended, the denominator should be 2 (or the routing weights should be renormalized); otherwise the design should be justified and shown in the ablations.
minor comments (4)
  1. [§2.2] The sentence 'MCITlib [18] provides a unified framework that consolidates representative approaches such as MoELoRA [5] maintains multiple LoRA experts...' is grammatically incomplete and missing punctuation; please rephrase.
  2. [Tables 1–4] The header 'LLaV A-1.5-7B' contains an inconsistent space in several table captions; use 'LLaVA-1.5-7B' throughout.
  3. [§5.3] The term 'sub-linear growth' should be defined precisely. Three experts for six tasks is sub-linear in this specific instance, but the paper does not establish an asymptotic or mechanism-level guarantee; a precise statement about what is meant by sub-linear would help.
  4. [§3.1–3.2] Eq. (1) uses a single global visual feature obtained by averaging visual token embeddings, whereas common MLLM projectors operate on per-token features. Please clarify why projection of a single averaged feature is sufficient and how this interacts with the LLM's token-level interface.

Circularity Check

1 steps flagged

Hyperparameter τ selected on UCIT is then used to claim sub-linear growth on the same benchmark; central gains still have independent MLLM-DCL support.

specific steps
  1. fitted input called prediction [Section 4.4 (Implementation Details) and Section 5.3 (Hyperparameter Analysis, Figure 3); results in Tables 1–2.]
    "We set the z-score threshold to τ=1.4for distribution-shift detection. In particular, τ=1.4achieves the highest MAA (83.98) with only3experts, demonstrating that PMA attains strong MCIT performance with sub-linear projector growth by expanding capacity only when necessary."

    τ is tuned on the UCIT benchmark by maximizing MAA (Figure 3). The same UCIT configuration (τ=1.4, 3 experts, MAA=83.98) is then reported as the headline DISCO+PMA result (Table 1) and as evidence of 'sub-linear projector growth.' The expert count is threshold-dependent: the paper itself notes τ=1.0 yields 6 experts (linear growth). Thus the UCIT sub-linear-growth claim and the associated MAA gain are not independent predictions; they are the selected outcome of tuning τ on the evaluation benchmark. The MLLM-DCL tables provide partial out-of-sample support, so the central claim does not fully reduce to the fit, but the UCIT-based growth claim is partly an artifact of hyperparameter selection.

full rationale

The derivation chain is mostly self-contained: the projector-expert expansion is defined by an explicit RD reconstruction-error trigger (Eq. 3–4), the router and anchor are trained with standard LM loss, and the main gains are benchmarked against re-implemented baselines. The self-citations in the bibliography are background and not load-bearing; no uniqueness theorem or prior result by the authors is used to force the design. The principal circularity concern is the hyperparameter analysis: τ is selected on UCIT by maximizing MAA, and the same UCIT result is then used to claim sub-linear growth and SOTA gains. This is a fitted-input-called-prediction pattern for that claim. However, the expansion-vs-no-expansion ablation and the MLLM-DCL results (with τ fixed from UCIT) provide independent evidence that addressing projector drift helps, so the central claim has content beyond the tuning artifact. The reconstruction-error proxy itself is unvalidated, but that is a validity risk, not a definitional circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central claim rests on a small set of hand-set or tuned hyperparameters (τ, 60% rule, RD bottleneck, RD epochs) and on several domain assumptions about projector drift being measurable by an autoencoder and fixable by frozen-anchor expert mixing. No standard mathematical axioms beyond gradient-based training are invoked. The RD is the main invented component; its independent validity is not established outside the paper.

free parameters (4)
  • z-score expansion threshold τ = 1.4
    Selected in §5.3 by maximizing MAA on UCIT; directly controls the number of projector experts (3 for UCIT) and hence the claimed sub-linear growth behavior.
  • expansion proportion threshold = 0.60
    Hand-set in §3.3 with no sensitivity analysis; if all prior RD p_j < 60%, a new expert is created. It co-determines expansion vs reuse alongside τ.
  • RD bottleneck dimension = (d1+d2)/4 = 1216
    Chosen by hand in §4.4; no study of how this affects detection accuracy or routing quality.
  • RD training epochs = 1
    Set in §4.4; not varied. Descriptor reliability may depend on this under-trained autoencoder.
axioms (5)
  • domain assumption Projector drift (updating the shared projector across tasks) is a primary cause of MCIT forgetting and is orthogonal to LLM-backbone forgetting.
    Central motivation; supported only by qualitative Figure 1 and indirect ablations, not by direct measurement of projector representation drift.
  • ad hoc to paper Reconstruction error of an MLP autoencoder on fused visual+text features is a reliable signal for whether existing projector experts can model a new task.
    The RD is introduced specifically for this purpose; no theoretical or independent empirical justification beyond in-paper ablations and routing analysis.
  • domain assumption Frozen pretrained projector P0 remains a stable alignment anchor and averaging it with experts (Eq. 5) preserves alignment.
    Assumed from standard transfer-learning practice; empirically supported only by the w/o Anchor ablation in Table 5.
  • domain assumption A softmax router trained only on new columns (with old columns frozen) will route unseen similar tasks to the right experts without task IDs.
    Heuristic; supported by the projector usage analysis in §5.4 on the same benchmark used to tune τ.
  • domain assumption UCIT and MLLM-DCL benchmarks have low information leakage and are valid MCIT evaluation suites.
    Adopted from MCITlib [18]; no leakage checks are run in this paper.
invented entities (1)
  • Representation Descriptor (RD) no independent evidence
    purpose: MLP autoencoder trained to reconstruct fused visual+text features; used to detect multimodal distribution shift and trigger projector expert expansion.
    No evidence outside this paper that reconstruction error on fused features is a valid shift detector for MCIT; the only supporting evidence is the paper's own ablation and routing analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 16507 in / 12718 out tokens · 132738 ms · 2026-08-03T01:18:24.877843+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.

Figures

Figures reproduced from arXiv: 2607.26947 by Duzhen Zhang, Jiahua Dong, Qiaoyi Su, Tielin Zhang, Yahan Yu.

Figure 1
Figure 1. Figure 1: Illustration of projector-level forgetting in MCIT. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of PMA. Lightweight RDs detect multimodal distribution shifts and determine whether to trigger projector [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Projector usage analysis under the DISCO+PMA [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 12 linked inside Pith

  1. [1]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision- language model with versatile abilities.arXiv preprint arXiv:2308.12966(2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923(2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Meng Cao, Yuyang Liu, Yingfei Liu, Tiancai Wang, Jiahua Dong, Henghui Ding, Xiangyu Zhang, Ian Reid, and Xiaodan Liang. 2024. Continual llava: Continual in- struction tuning in large vision-language models.arXiv preprint arXiv:2411.02564 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. 2022. MapQA: A Dataset for Question Answering on Choropleth Maps. In NeurIPS 2022 First Table Representation Workshop

    2022

  5. [5]

    Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao

  6. [6]

    Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023. Vlp: A survey on vision-language pre-training.Machine Intelligence Research20, 1 (2023), 38–56

    2023

  7. [7]

    Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ip, and Sam Kwong. 2025. SEFE: Superficial and Essential Forgetting Elim- inator for Multimodal Continual Instruction Tuning. InForty-second International Conference on Machine Learning

    2025

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision. Springer, 370– 387

    2024

  9. [9]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 24185–24198

    2024

  10. [10]

    Jiahua Dong, Duzhen Zhang, Yang Cong, Wei Cong, Henghui Ding, and Dengxin Dai. 2023. Federated incremental semantic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3934–3943

    2023

  11. [11]

    Dong, Jiahua and Li, Hongliu and Cong, Yang and Sun, Gan and Zhang, Yulun and Van Gool, Luc. 2024. No One Left Behind: Real-World Federated Class-Incremental Learning.IEEE Transactions on Pattern Analysis and Machine Intelligence46, 4 (2024), 2054–2070. https://doi.org/10.1109/TPAMI.2023.3334213

    arXiv 2024

  12. [12]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. 2023. Loramoe: Revolution- izing mixture of experts for maintaining world knowledge in language model alignment.arXiv preprint arXiv:2312.099794, 7 (2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957 (2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Chendi Ge, Xin Wang, Zeyang Zhang, Hong Chen, Jiapei Fan, Longtao Huang, Hui Xue, and Wenwu Zhu. 2025. Dynamic Mixture of Curriculum LoRA Ex- perts for Continual Multimodal Instruction Tuning. InForty-second International Conference on Machine Learning

    2025

  15. [15]

    Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio

  16. [16]

    Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. 2025. HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - A...

    2025

  17. [17]

    Haiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu, Da-Han Wang, Jian Xu, Xu-Yao Zhang, and Cheng-Lin Liu. 2025. Federated continual instruction tuning.ICCV (2025)

    2025

  18. [18]

    Haiyang Guo, Fei Zhu, Hongbo Zhao, Fanhu Zeng, Wenzhuo Liu, Shijie Ma, Da- Han Wang, and Xu-Yao Zhang. 2025. Mcitlib: Multimodal continual instruction tuning library and benchmark.ICCV 2025@Workshop on Multimodal Continual Learning(2025)

    2025

  19. [19]

    Ziyu Guo, Renrui Zhang, Hao Chen, Jialin Gao, Dongzhi Jiang, Jiaze Wang, and Pheng-Ann Heng. 2025. SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems. InFindings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025. Association for Computational L...

    2025

  20. [20]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition. 3608–3617

    2018

  21. [21]

    Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. 2023. Continual in- struction tuning for large multimodal models.arXiv preprint arXiv:2311.16206 (2023)

    Pith/arXiv arXiv 2023

  22. [22]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286(2020)

    Pith/arXiv arXiv 2020

  23. [23]

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. 2021. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization.ICCV(2021)

    2021

  24. [24]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

    2022

  25. [25]

    Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. 2025. CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering. InProceedings of the Computer Vision and Pattern Recognition Conference. 19608– 19617

    2025

  26. [26]

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Ha- jishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. InEuropean conference on computer vision. Springer, 235–251

    2016

  27. [27]

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. InProceedings of the IEEE Conference on Computer Vision and Pattern recognition. 4999–5007

    2017

  28. [28]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences114, 13 (2017), 3521– 3526

    2017

  29. [29]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3045–3059

    2021

  30. [30]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2025. LLaVA-OneVision: Easy Visual Task Transfer.Transactions on Machine Learning Research(2025)

    2025

  31. [31]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. InInternational Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. 19730–19742

    2023

  32. [32]

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024. Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 14369–14387

    2024

  33. [33]

    Adam Dahlgren Lindström and Savitha Sam Abraham. 2022. CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning. In Proceedings of the 16th International Workshop on Neural-Symbolic Learning and Reasoning as part of the 2nd International Joint Conference on Learning & Reasoning (IJCLR 2022), Cumberland Lodge, Windsor Great Park,...

    2022

  34. [34]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 26296–26306

    2024

  35. [35]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowl- edge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  36. [36]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning. InThirty-seventh Conference on Neural Information Processing Systems

    2023

  37. [37]

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2023. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications.arXiv preprint arXiv:2310.18339 MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Duzhen Zhang, Yahan Yu, Qiaoyi Su, Jiahua Dong, and Tielin Zhang (2023)

    Pith/arXiv arXiv 2023

  38. [38]

    Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. 2020. RSVQA: Visual question answering for remote sensing data.IEEE Transactions on Geoscience and Remote Sensing58, 12 (2020), 8555–8566

    2020

  39. [39]

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning. InThe 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks

    2021

  40. [40]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in con- nectionist networks: The sequential learning problem. InPsychology of learning and motivation. Vol. 24. Elsevier, 109–165

    1989

  41. [41]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/

    2024

  42. [42]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hock- enmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to- phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE international conference on computer vision. 2641–2649

    2015

  43. [43]

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. 2024. Drivelm: Driving with graph visual question answering. InEuropean conference on computer vision. Springer, 256–274

    2024

  44. [44]

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. 2023. Orthogonal subspace learning for lan- guage model continual learning. InFindings of the Association for Computational Linguistics: EMNLP 2023. 10658–10671

    2023

  45. [45]

    Ziao Wang, Yuhang Li, Junda Wu, Jaehyeon Soon, and Xiaofeng Zhang. 2023. Finvis-gpt: A multimodal large language model for financial chart analysis.arXiv preprint arXiv:2308.01430(2023)

    Pith/arXiv arXiv 2023

  46. [46]

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu

  47. [47]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models.National Science Review11, 12 (2024), nwae403

    2024

  48. [48]

    Yahan Yu, Duzhen Zhang, Yong Ren, Xuanle Zhao, Xiuyi Chen, and Chenhui Chu

  49. [49]

    Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. 2025. Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models.EMNLP(2025)

    2025

  50. [50]

    Duzhen Zhang, Yong Ren, Zhong-Zhi Li, Yahan Yu, Jiahua Dong, Chenxing Li, Zhilong Ji, and Jinfeng Bai. 2025. Enhancing Multimodal Continual Instruc- tion Tuning with BranchLoRA. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 5743–5756

    2025

  51. [51]

    Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024. Mm-llms: Recent advances in multimodal large language models. InFindings of the Association for Computational Linguistics ACL 2024

    2024

  52. [52]

    Tielin Zhang, Xiang Cheng, Shuncheng Jia, Chengyu T Li, Mu-ming Poo, and Bo Xu. 2023. A brain-inspired algorithm that mitigates catastrophic forgetting of artificial and spiking neural networks with low computational cost.Science Advances9, 34 (2023), eadi2947

    2023

  53. [53]

    Hongbo Zhao, Fei Zhu, Haiyang Guo, Meng Wang, Rundong Wang, Gaofeng Meng, and Zhaoxiang Zhang. 2025. Mllm-cl: Continual learning for multimodal large language models.arXiv preprint arXiv:2506.05453(2025)

    arXiv 2025

  54. [54]

    Junhao Zheng, Xidi Cai, Qiuke Li, Duzhen Zhang, ZhongZhi Li, Yingying Zhang, Le Song, and Qianli Ma. 2025. Lifelongagentbench: Evaluating llm agents as lifelong learners.arXiv preprint arXiv:2505.11942(2025)

    Pith/arXiv arXiv 2025

  55. [55]

    Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. 2026. Lifelong learning of large language model based agents: A roadmap.IEEE Transactions on Pattern Analysis and Machine Intelligence(2026)

    2026

  56. [56]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models.arXiv preprint arXiv:2304.10592(2023)

    Pith/arXiv arXiv 2023

  57. [2013]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks.arXiv preprint arXiv:1312.6211(2013)

    Pith/arXiv arXiv 2013

  58. [2023]

    In2023 IEEE International Conference on Big Data (BigData)

    Multimodal large language models: A survey. In2023 IEEE International Conference on Big Data (BigData). IEEE, 2247–2256

  59. [2024]

    Coin: A benchmark of continual instruction tuning for multimodel large language models.Advances in Neural Information Processing Systems37 (2024), 57817–57840

    2024

  60. [2025]

    InFindings of the Association for Computational Linguistics: ACL 2025

    Progressive lora for multimodal continual instruction tuning. InFindings of the Association for Computational Linguistics: ACL 2025. 2779–2796

    2025