Pith. sign in

REVIEW 10 cited by

Continual Instruction Tuning for Large Multimodal Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.16206 v1 pith:MIFDREEL submitted 2023-11-27 cs.LG cs.AIcs.CV

Jinghan He , Haiyun Guo , Ming Tang , Jinqiao Wang This is my paper
classification cs.LGcs.AIcs.CV
keywords continualinstructionlmmstuningforgettinglearningmethodsmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Instruction tuning is now a widely adopted approach to aligning large multimodal models (LMMs) to follow human intent. It unifies the data format of vision-language tasks, enabling multi-task joint training. However, vision-language tasks are constantly being created in practice. Instead of always re-training LMMs when new tasks arrive, continual learning offers flexibility for models to continually and efficiently exploit the evolving data. This work aims to explore the following two questions: 1) Do LMMs still suffer from catastrophic forgetting in continual instruction tuning? 2) Are the existing three classes of continual learning methods still applicable to the continual instruction tuning of LMMs? An extensive study is conducted to address the above questions. First, we establish the first benchmark in this setting and reveal that catastrophic forgetting is still observed when continually instruction-tuning LMMs. However, the multi-task joint instruction tuning can facilitate the model's continual learning ability and mitigate forgetting. Second, we integrate and adapt classic continual learning methods to our context, demonstrating the efficacy of data replay and model expansion strategies across diverse scenarios. In contrast, regularization-based methods only perform well on models that have been jointly instruction-tuned on multiple tasks. Third, we delve into the correlation and forgetting dynamics between vision-language task pairs and propose task-similarity-informed regularization and model expansion methods for continual instruction tuning of LMMs. Experimental results show that our approach consistently boosts the model's performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Progressive Multimodal Alignment for Continual Instruction Tuning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Progressive Multimodal Alignment expands projector experts only when multimodal distribution shifts are detected, reducing projector-level forgetting and boosting MCIT baselines with sub-linear growth.

  2. Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs

    cs.CV 2025-11 conditional novelty 6.0 of 10

    HPA is a post-training parameter-selection method that keeps safety-aligned multimodal LLMs safe and reduces forgetting during continual visual instruction tuning.

  3. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.

  4. Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A mixture of instruction-conditioned visual projectors with expert recommendation, pruning, and adaptive aggregation improves continual learning and zero-shot retention in generative vision-language models.

  5. Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Bisecle reduces catastrophic forgetting and improves accuracy in sequential VideoQA learning using multi-directional auxiliary losses and contrastive prompt regularization.

  6. Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    D-MoLE dynamically allocates LoRA experts to the most gradient-sensitive layers of a vision-language model for each new task, and routes inputs via per-task autoencoders, reporting near-zero forgetting on a nine-task ...

  7. PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

    cs.LG 2026-01 conditional novelty 5.0 of 10

    Routing MoE-LoRA experts by the strength of their low-rank activation and regularizing important rank directions reduces forgetting in continual instruction tuning of multimodal LLMs.

  8. Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CorDA++ uses data-driven SVD to initialize LoRA adapters, adding per-layer covariance selection and rank allocation that reduce forgetting and speed convergence compared to LoRA, PiSSA, QLoRA, and other baselines.

  9. Enhancing Multimodal Continual Instruction Tuning with BranchLoRA

    cs.CL 2025-05 conditional novelty 5.0 of 10

    BranchLoRA reduces catastrophic forgetting in multimodal continual instruction tuning by using a shared LoRA matrix, task-specific branches, frozen experts, and learned task keys.

  10. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Pith tools