Pith. sign in

REVIEW 5 cited by

HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.12941 v2 pith:KLT4BQSV submitted 2025-03-17 cs.CL cs.LG

Haiyang Guo , Fanhu Zeng , Ziwei Xiang , Fei Zhu , Da-Han Wang , Xu-Yao Zhang , Cheng-Lin Liu This is my paper
classification cs.CLcs.LG
keywords instructiondatasetsexistingmethodsmodelperformancetuningbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Instruction tuning is widely used to improve a pre-trained Multimodal Large Language Model (MLLM) by training it on curated task-specific datasets, enabling better comprehension of human instructions. However, it is infeasible to collect all possible instruction datasets simultaneously in real-world scenarios. Thus, enabling MLLM with continual instruction tuning is essential for maintaining their adaptability. However, existing methods often trade off memory efficiency for performance gains, significantly compromising overall efficiency. In this paper, we propose a task-specific expansion and task-general fusion framework based on the variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets. Furthermore, we analyze the information leakage present in the existing benchmark and propose a new and more challenging benchmark to rationally evaluate the performance of different methods. Comprehensive experiments showcase a significant performance improvement of our method compared to existing state-of-the-art methods. Code and dataset are released at https://github.com/Ghy0501/HiDe-LLaVA.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Aligning images to multi-view caption cores while suppressing orthogonal residual text and disagreement-aware temperature improves robust zero-shot recognition and LVLM transfer.

  2. CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A private-plus-shared LoRA MoE with layer-adaptive momentum transfer enables continual anomaly detection on MLLMs and beats prior continual-learning baselines across class, domain, and modality shifts.

  3. Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Balancing the singular values of LoRA task updates, plus orthogonality to past gradients, reduces backward and forward forgetting in continual adaptation of vision-language models.

  4. GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning

    cs.LG 2025-07 conditional novelty 5.0 of 10

    GNSP combines null-space gradient projection with contrastive distillation and a modality alignment loss to fine-tune CLIP continually while preserving its zero-shot transfer and retrieval.

  5. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Pith tools