Pith. sign in

REVIEW 3 major objections 5 minor 153 references

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AWARe claims that freezing the output neurons with the strongest upstream activations, about 30 percent of self-attention projections, keeps a multimodal model's prior knowledge nearly intact during downstream fine-tuning.

desk verdict A clean, if modest, empirical paper that reuses activation saliency from pruning/quantization to freeze important rows during fine-tuning; the headline numbers are real, but the optimal 30% freeze ratio is tuned on the test set, so the central claim is partly post-selection. read the letter →

arxiv 2608.11758 v1 pith:B3DSA7YX submitted 2026-08-12 cs.CL

Juncheng Liao , Jinfan Lv , Guoming Wang , Jupeng Zheng , Ling Xiao , Siliang Tang This is my paper · ORCID
classification cs.CL
keywords catastrophicforgettingmultimodallargelanguagemodelsactivation-basedsaliencyselectivefreezingcontinualinstructiontuningparameter-efficientfine-tuningknowledgeretentionplasticity-stabilitytrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to solve catastrophic forgetting in multimodal large language models: when a model that already understands images and text is fine-tuned on one new task, it tends to lose its earlier general capabilities. AWARe claims that a small set of output neurons, identified by their activation strength on upstream data, is disproportionately responsible for holding that prior knowledge, and that freezing the top 30% of those neurons during fine-tuning preserves upstream ability while leaving the model enough freedom to learn the new task. The method needs only a forward pass on a few hundred calibration samples, changes no architecture, and can even use a general-purpose multimodal benchmark when upstream data are unavailable. The reported numbers on IconQA, COCO-Caption, and the MLLM-DCL sequence support that claim, with retention $R=98.4$ and harmonic means above every listed baseline. If correct, this gives practitioners a cheap, deployment-compatible way to keep a multimodal model general while specializing it.

What carries the argument

The central object is an activation-weighted saliency score computed per output neuron. Given a target linear layer's activation tensor $A \in \mathbb{R}^{B \times L \times d_{\mathrm{out}}}$, the method first aggregates activations along the sequence length with the $L^2$ norm, then normalizes each sample's vector across the hidden dimension, and finally averages over the batch: $s_k = \frac{1}{B} \sum_{i=1}^B \frac{\sqrt{\sum_{j=1}^L a_{i,j,k}^2}}{\|\mathbf{a}'_i\|_2}$. Neurons are ranked by $s_k$ globally across the target layers, and the top $\rho$ fraction (default $\rho = 0.3$) are frozen by setting the corresponding rows of the gradient mask to zero, while the remaining rows receive the full gradient update. The mask is applied only to the self-attention projections $q\_proj$, $k\_proj$, $v\_proj$ and to the multimodal projector $mm\_projector$, with all other components kept frozen. What carries the argument is the claim, supported by the paper's ablations, that this particular score locates the output dimensions whose preservation is sufficient for upstream knowledge, whereas random or weight-magnitude selection at the same ratio loses it.

What would settle it

Measure the activation vectors of the frozen neurons on the calibration set before and after downstream fine-tuning. If those activations drift substantially while upstream retention stays high, the mask is not what is pinning the retained pathway, and the reported retention would have to come from some other part of the training recipe.

Watch

Extended reading notes

Core claim

On the paper's own terms, AWARe's central claim is that you can mitigate catastrophic forgetting in a multimodal large language model by computing a per-neuron saliency score from forward activations on a small calibration set and freezing the top-ranked output neurons in the self-attention q/k/v projections and the multimodal projector during downstream fine-tuning. This is a selective-freezing rule, not a new architecture and not a replay strategy. The paper reports that with the top 30% of globally salient neurons frozen, LLaVA-v1.5-7B fine-tuned on IconQA keeps upstream retention $R=98.4$ and reaches a harmonic mean $H=103.2$; on COCO-Caption it reaches $H=108.0$; and on the five-task MLLM-DCL continual learning benchmark it reaches an average of $65.58$, with all numbers above the listed baselines. It further claims the same scheme remains effective when the calibration set is a general-purpose benchmark such as MMMU instead of the upstream task data.

Load-bearing premise

The load-bearing premise is that the neurons which fire most strongly on upstream data are exactly the ones whose preservation keeps upstream knowledge intact; if a different model or task distribution stores prior knowledge elsewhere, the frozen mask protects nothing.

Editorial extensions

If this is right

  • Freezing the globally top 30% of salient q/k/v and mm_projector neurons is enough for near-complete upstream retention: on IconQA the paper reports $R=98.4$ with target accuracy 86.92, above the full fine-tuning reference of 80.15.
  • The method transfers without upstream data: using a general-purpose benchmark as the calibration set keeps $H=101.7$ on IconQA and $H=107.9$ on COCO-Caption.
  • In a five-task continual instruction tuning sequence (MLLM-DCL), AWARe reports average and last-task scores of 65.58 and 62.10, both above the listed baselines.
  • Because freezing happens at the neuron-row level, only about 17.5% of the model's parameters are updated in the optimal setting, and the training-time wrapper can be removed after fine-tuning so inference remains standard.
  • Ablations show the choice of saliency signal is load-bearing: random selection of the same 30% of neurons drops average upstream accuracy to 49.94 on IconQA, while activation-based selection keeps it at 60.13.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If high-activation output neurons are where stable knowledge lives, the same saliency score could be reused for model merging or pruning decisions, since it is obtained in one forward pass and requires no labels; the paper does not test this.
  • The paper's layer ablation suggests a modular map: self-attention projections tolerate task-specific updates while MLP blocks are so knowledge-dense that even protected fine-tuning collapses upstream accuracy. One could test that map directly by comparing per-layer freezing masks across different base models and task families.
  • Because the paper evaluates one seven-billion-parameter model and a five-task sequence, a natural extension is to check whether the saliency ordering stays stable over much longer continual learning streams; repeated updates could shift activations so that the original frozen mask becomes stale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AWARe, a fine-tuning method for multimodal LLMs that estimates per-neuron activation saliency on a small calibration set, then freezes the top ρ fraction of output rows in selected q/k/v and mm_projector linear layers during downstream training while updating the remaining rows. The method is evaluated on LLaVA-v1.5-7B for single-task adaptation on IconQA and COCO-Caption, on MLLM-DCL for continual instruction tuning, and on Qwen2.5-VL for two downstream tasks. The central claims are that activation-based selective freezing preserves upstream knowledge with only a modest loss of downstream plasticity, that freezing roughly 30% of the salient attention parameters is often sufficient, and that a general-purpose calibration set such as MMMU can substitute for unavailable upstream data.

Significance. If the empirical claims hold, AWARe is a useful addition to the anti-forgetting toolbox for MLLMs: it is architecture-agnostic, requires no replay buffer or extra adapter modules, and its saliency score is a deterministic function of activations rather than a fitted model. The ablations in Table 3 give direct evidence that activation-based selection beats random and weight-norm selection, and the MMMU-calibration results in Table 1 address a practically important data-availability scenario. The core equations are simple enough to be checked and reimplemented, and a code link is provided. The main weaknesses are methodological: the headline hyperparameters are selected on the evaluation benchmarks, and the main tables lack variance estimates, so the strength of the comparative claims is currently not fully supported.

major comments (3)
  1. [§4.5, Table 4; §D, Table 11; Eq. (6)] The headline stability-plasticity numbers are selected on the evaluation benchmarks rather than on an independent validation split. In Table 4, ρ is swept from 1% to 90% and the value maximizing H (Eq. 6) on IconQA is reported; Table 11 then lists the resulting 30% global-highest configuration as optimal, and Section D states only that results are averaged over three runs. Because R and E are computed from upstream retention and downstream target accuracy on the same test benchmarks, this is hyperparameter selection on the test set. The same issue applies to the choice of target set (q/k/v + mm_projector), which is ablated on IconQA in §4.5/Fig. 5 and then transferred to COCO-Caption, MLLM-DCL, and Qwen2.5-VL. Consequently, the abstract's claim that 'freezing the top 30%... is often sufficient' is a post-selection observation, not a tested prediction. Please add a validation-based selection protocol (e.g., hold out part of the downstream training set or a separate task for choosing ρ and the target set), or clearly label the reported numbers as selected-on-test and show that the choice of ρ is not material to the conclusions.
  2. [§4.3, Eq. (8); Table 1] Equation (8) defines E relative to full fine-tuning and the text calls Accdown,Full-FT the 'performance upper bound achievable by Full-FT', but Table 1 contradicts this: on COCO-Caption, LoRA reaches 110.27, Model Tailor 117.64, LoRASculpt 120.35, and AWARe 120.94, all above the Full-FT value of 95.59, giving E>100 in several rows. The bounded interpretation of E, and hence of H as a harmonic mean of stability and plasticity, is therefore not valid as stated. Please replace the 'upper bound' wording or redefine E, and check whether any conclusions change under an alternative normalization.
  3. [§4, Tables 1–2; §D] Section D reports that all results are averaged over three random runs, but Tables 1 and 2 report only point estimates, and Table 6 reports standard deviations only for the IconQA calibration-size sensitivity analysis. The reported improvements, such as +3.1 in H over LoRASculpt on IconQA and +3.18 in Avg over DISCO on MLLM-DCL, cannot be assessed for statistical significance. Please report mean ± std (or confidence intervals) for the main tables, at least for AWARe and the closest baselines, or provide per-run tables in the appendix.
minor comments (5)
  1. [§3.2, Eq. (1)] The text says the L2-norm is computed 'for each sample j and neuron k', but the equation uses the index i for the sample; please make the indexing consistent.
  2. [§C, Eq. (14)] The text states that the method 'updates only a quarter of the model parameters', but Eq. (14) gives a trainable ratio of about 17.5%. Please correct the wording to match the calculation, and note that the calculation excludes the mm_projector parameters.
  3. [§4.5, Table 4] The 30% global-highest configuration is called optimal with H=103.2, but the 40% row also attains H=103.2; please acknowledge the tie or explain the additional criterion used to select 30%.
  4. [§3.4] The cited finding that 'updating self-attention projections tends to cause significantly less catastrophic forgetting' than MLP blocks is not obviously the rationale for freezing self-attention rows; please clarify the logical connection between the citation and the design choice.
  5. [§4.6 and Table 11] Table 11 specifies a calibration composition of '200 samples each' from four upstream tasks (800 total), while Section 4.6 describes calibration sizes of 200 and 400 samples; please clarify whether these are per-task or total-sample counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AWARe's saliency scores are deterministic functions of activations, and the retention ratio is an openly reported hyperparameter rather than a fitted prediction.

full rationale

AWARe's saliency estimate (Eqs. 1-3) is a closed-form function of forward activations on a calibration set; it contains no parameter fitted to the retention outcome, and the gradient mask (Eqs. 4-5) follows deterministically once the retention ratio rho is chosen. The paper reports rho as a hyperparameter (Table 11) and sweeps it in an ablation (Table 4); selecting rho by harmonic mean on IconQA is a benchmark-selection concern rather than an equation-level circularity, because the claim that the top 30% is sufficient is an empirical post-selection observation, not a quantity derived from the method's own equations. The target-layer choice (q/k/v plus mm_projector) is justified by an external citation and a dedicated ablation (Fig. 5), not by a self-citation chain. The authors do not cite their own prior work as load-bearing evidence, no uniqueness theorem is imported, and no known result is renamed as a derivation. The Limitations section even concedes that 'extremely large distribution shifts may still require tuning the retention ratio,' confirming rho is a tunable hyperparameter rather than a predicted output. The strongest remaining issue is statistical validity of test-set hyperparameter selection, which is outside the circularity rubric; the MMMU and COCO-Caption results provide some independent support for the method's transferability.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the empirical validity of activation magnitude as an importance proxy, on the choice to restrict freezing to q/k/v and the multimodal projector, and on the adequacy of a small calibration set. These are domain assumptions tested only partially. The only free parameter with a chosen value is the retention ratio rho=0.30, tuned on the evaluation task.

free parameters (1)
  • retention ratio rho = 0.30
    Top 30% of globally highest-saliency output neurons are frozen; chosen from the ablation in Table 4 as the best stability-plasticity balance on IconQA.
assumptions (3)
  • domain assumption Per-channel activation magnitude (Eqs. 1-3) identifies output neurons critical for preserving upstream knowledge.
    Borrowed from AWQ/Wanda and validated here by ablation (Table 3), but only for two backbones and three benchmarks.
  • domain assumption Freezing only q_proj, k_proj, v_proj, and mm_projector, with all MLP and other layers frozen, preserves capabilities.
    Based on prior finding (Zhu et al. 2025b) and the ablation in Figure 5; not derived from first principles.
  • domain assumption A calibration set of 200 samples per task (or 600 MMMU samples) yields stable saliency estimates.
    Supported by the seed sensitivity analysis (Table 6), but only for IconQA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention." pith.science (2026). https://pith.science/paper/B3DSA7YX

@misc{pith2026260811758,
  author       = {Pith},
  title        = {Pith review of: AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3DSA7YX}},
  note         = {Machine review of arXiv:2608.11758}
}
read the original abstract

Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.

Figures

Figures reproduced from arXiv: 2608.11758 by the authors.

Figure 1
Figure 1. Overall Performance of AWARe. Top: radar plots showing comparison between our approach and various baselines under Single-Task Downstream Adap￾tation. Bottom: results on the MLLM-DCL continuous learning benchmark. their impressive zero-shot capabilities, fine-tuning MLLMs on downstream tasks remains a standard practice to tailor models for specific domains or improve instruction-following performance (Dai et al., 20… view at source ↗
Figure 2
Figure 2. Overview of AWARe. Given a multimodal base model, AWARe identifies important neurons or weight groups using activation-based saliency on calibration samples, then freezes high-saliency weights and updates only low-saliency weights during fine-tuning. 2018; Chaudhry et al., 2019) or regularization￾based methods (Kirkpatrick et al., 2017), often incur high computational overhead or struggle to scale to the billions of… view at source ↗
Figure 3
Figure 3. Overview of AWARe. For each target lin￾ear layer, we profile the activation matrix produced by calibration samples, normalize the activation statistics across sequence and feature dimensions, and obtain a saliency score for each output channel. High-saliency channels are selected and frozen to preserve pretrained knowledge, while the remaining channels stay trainable for downstream adaptation. 3.2 Activation-Based S… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Layer-wise distribution of high-activation neurons across different projection types. The heatmaps [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: , targeting both the self-attention projec￾tions and the multimodal (mm_projector) yields the most favorable balance between plasticity and stability. Specifically, the mm_projector is essen￾tial for learning new downstream concepts; Lock￾ing it limits adaptation and c…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

153 extracted references · 42 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

Show all 153 references
  1. [9]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025 a . Qwen3-vl tech...

    2025 arXiv

  2. [10]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025 b . https://arxiv.org/abs/2502...

    2025 arXiv

  3. [11]

    Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, and 1 others. 2024. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673

    2024 arXiv

  4. [12]

    James Burgess, Jeffrey J Nirschl, Laura Bravo-S \'a nchez, Alejandro Lozano, Sanket Rajan Gupte, Jesus G Galaz-Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, and 1 others. 2025. Microvqa: A multimodal reasoning benchmark for microscopy-based scientific researc...

    2025

  5. [13]

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. 2019. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486

    2019 arXiv

  6. [14]

    Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Jingkuan Song, and Lianli Gao. 2024 a . Coin: A benchmark of continual instruction tuning for multimodel large language models. Advances in Neural Information Processing Systems, 37:57817--57840

    2024

  7. [15]

    Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ip, and Sam Kwong. 2025. Sefe: Superficial and essential forgetting eliminator for multimodal continual instruction tuning. In ICML

    2025

  8. [16]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185--24198

    2024

  9. [17]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  10. [18]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS

    2023

  11. [19]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, and 1 others. 2024. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In ACL, pages 1932--1945

    2024

  12. [20]

    Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR

    2019

  13. [21]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language mod...

    2024

  14. [22]

    Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. 2025 a . Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In Proceedings of the 63rd Annual Meeting of the Association for Compu...

    2025

  15. [23]

    Haiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu, Da-Han Wang, Jian Xu, Xu-Yao Zhang, and Cheng-Lin Liu. 2025 b . Federated continual instruction tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325--1335

    2025

  16. [24]

    Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

    2015 arXiv

  17. [25]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML, pages 2790--2799. PMLR

    2019

  18. [26]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR

    2022

  19. [27]

    Jiajun Hu, Jian Zhang, Lei Qi, Yinghuan Shi, and Yang Gao. 2024. Learn to preserve and diversify: Parameter-efficient group with orthogonal regularization for domain generalization. In European Conference on Computer Vision, pages 198--216. Springer

    2024

  20. [28]

    Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. 2025. Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering. In Proceedings of the computer vision and pattern recogniti...

    2025

  21. [29]

    Wenke Huang, Jian Liang, Zekun Shi, Didi Zhu, Guancheng Wan, He Li, Bo Du, Dacheng Tao, and Mang Ye. 2025. Learn from downstream and be yourself in multimodal large language models fine-tuning. In ICML

    2025

  22. [30]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700--6709

    2019

  23. [31]

    Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, and Fahad Shahbaz Khan. 2024. Effectiveness assessment of recent large vision-language models. Visual Intelligence, 2(1):17

    2024

  24. [32]

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In European conference on computer vision, pages 235--251. Springer

    2016

  25. [33]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and 1 others. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national aca...

    2017

  26. [34]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  27. [35]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  28. [36]

    Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. 2024 b . Revisiting catastrophic forgetting in large language model tuning. arXiv preprint arXiv:2406.04836

    2024 arXiv

  29. [37]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730--19742. PMLR

    2023

  30. [38]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888--12900. PMLR

    2022

  31. [39]

    Jian Liang, Wenke Huang, Guancheng Wan, Qu Yang, and Mang Ye. 2025. Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26170--26180

    2025

  32. [40]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87--100

    2024

  33. [41]

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2023. Vila: On pre-training for visual language models. In CVPR

    2023

  34. [42]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740--755

    2014

  35. [43]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved baselines with visual instruction tuning. In CVPR

    2023

  36. [44]

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In ICML

    2024

  37. [45]

    Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. 2018. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31

    2018

  38. [46]

    David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. NeurIPS, 30

    2017

  39. [47]

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurIPS

    2021

  40. [48]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747

    2023 arXiv

  41. [49]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing

    2025

  42. [50]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR

    2019

  43. [51]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209

    2021

  44. [52]

    James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419

    1995

  45. [53]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  46. [54]

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947--952. IEEE

    2019

  47. [55]

    Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang. 2023. Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models. arXiv preprint arXiv:2311.08011

    2023 arXiv

  48. [56]

    Amin Heyrani Nobari, Kaveh Alimohammadi, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, and Navid Azizan. 2025. Activation-informed merging of large language models

    2025

  49. [57]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In ICML, pages 8748--8763

    2021

  50. [58]

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2018. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910

    2018 arXiv

  51. [59]

    Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. 2024. Multimodal instruction tuning with conditional mixture of lora. arXiv preprint arXiv:2402.15896

    2024 arXiv

  52. [60]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In CVPR, pages 8317--8326

    2019

  53. [61]

    Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. 2024. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, volume 2024, pages 4942--4964

    2024

  54. [62]

    Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. 2023. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models. arXiv preprint arXiv:2310.02998

    2023 arXiv

  55. [64]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  56. [65]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  57. [66]

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265

    2025 arXiv

  58. [67]

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. 2023. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10658--10671

    2023

  59. [68]

    Penghao Wu and Saining Xie. 2024. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084--13094

    2024

  60. [69]

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. 2024. Corda: Context-oriented decomposition adaptation of large language models. arXiv preprint arXiv:2406.05223

    2024 arXiv

  61. [70]

    Yuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li, Mingyang Liu, Xiongwei Han, Zhijiang Guo, Han Wu, and Linqi Song. 2025. Activation-guided consensus merging for large language models. NeurIPS

    2025

  62. [71]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML

    2024

  63. [72]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-...

    2024

  64. [73]

    Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. 2025. Modalprompt: Towards efficient multimodal continual instruction tuning with dual-modality guided prompt. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12...

    2025

  65. [74]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In ICCV, pages 11975--11986

    2023

  66. [75]

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2024 a . Investigating the catastrophic forgetting in multimodal large language model fine-tuning. pages 202--227. PMLR

    2024

  67. [76]

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2024 b . Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In CPAL, pages 202--227. PMLR

    2024

  68. [78]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  69. [79]

    Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. 2024. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In ICML

    2024

  70. [80]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025 a . Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479

    2025 arXiv

  71. [81]

    Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, and Derek Hoiem. 2025 b . https://arxiv.org/abs/2510.08564 How to teach large multimodal models new skills

    2025 arXiv

  72. [82]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models , year =

    Zhu, Deyao and Chen, Jun and Shen, Xiaoqian and Li, Xiang and Elhoseiny, Mohamed , journal =. Minigpt-4: Enhancing vision-language understanding with advanced large language models , year =

  73. [83]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning , year =

    Wenliang Dai and Junnan Li and Dongxu Li and Anthony Meng Huat Tiong and Junqi Zhao and Weisheng Wang and Boyang Li and Pascale Fung and Steven Hoi , booktitle =. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning , year =

  74. [84]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond , year =

    Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren , journal =. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond , year =

  75. [85]

    Improved Baselines with Visual Instruction Tuning , year =

    Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae , booktitle =. Improved Baselines with Visual Instruction Tuning , year =

  76. [86]

    VILA: On Pre-training for Visual Language Models , year =

    Ji Lin and Hongxu Yin and Wei Ping and Yao Lu and Pavlo Molchanov and Andrew Tao and Huizi Mao and Jan Kautz and Mohammad Shoeybi and Song Han , booktitle =. VILA: On Pre-training for Visual Language Models , year =

  77. [87]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , year =

    Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others , booktitle =. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , year =

  78. [88]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , year =

    Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and others , journal =. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , year =

  79. [89]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models , year =

    Zhu, Jinguo and Wang, Weiyun and Chen, Zhe and Liu, Zhaoyang and Ye, Shenglong and Gu, Lixin and Tian, Hao and Duan, Yuchen and Su, Weijie and Shao, Jie and others , journal =. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models , year =

  80. [90]

    InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency , year =

    Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others , journal =. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficie...

  81. [91]

    2025 , eprint=

    Qwen2.5-VL Technical Report , author=. 2025 , eprint=

    2025

  82. [92]

    Qwen3-VL Technical Report , year =

    Shuai Bai and Yuxuan Cai and Ruizhe Chen and Keqin Chen and Xionghui Chen and Zesen Cheng and Lianghao Deng and Wei Ding and Chang Gao and Chunjiang Ge and Wenbin Ge and Zhifang Guo and Qidong Huang and Jie Huang and Fei Huang and Binyuan Hui and Shutong Jiang and Zhaohai Li a...

  83. [93]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , year =

    Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven , booktitle =. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , year =

  84. [94]

    Catastrophic interference in connectionist networks: The sequential learning problem , volume =

    McCloskey, Michael and Cohen, Neal J , booktitle =. Catastrophic interference in connectionist networks: The sequential learning problem , volume =

  85. [95]

    Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory

    McClelland, James L and McNaughton, Bruce L and O'Reilly, Randall C , journal =. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , volume =

  86. [96]

    Overcoming catastrophic forgetting in neural networks , volume =

    Kirkpatrick, James and Pascanu, Razvan and Rabinowitz, Neil and Veness, Joel and Desjardins, Guillaume and Rusu, Andrei A and Milan, Kieran and Quan, John and Ramalho, Tiago and Grabska-Barwinska, Agnieszka and others , journal =. Overcoming catastrophic forgetting in neural n...

  87. [97]

    Revisiting catastrophic forgetting in large language model tuning , year =

    Li, Hongyu and Ding, Liang and Fang, Meng and Tao, Dacheng , journal =. Revisiting catastrophic forgetting in large language model tuning , year =

  88. [98]

    Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models , year =

    Sung, Yi-Lin and Yoon, Jaehong and Bansal, Mohit , journal =. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models , year =

  89. [99]

    IEEE Transactions on Audio, Speech and Language Processing , year=

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning , author=. IEEE Transactions on Audio, Speech and Language Processing , year=

  90. [100]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning , year =

    Luo, Yun and Yang, Zhen and Meng, Fandong and Li, Yafu and Zhou, Jie and Zhang, Yue , journal =. An empirical study of catastrophic forgetting in large language models during continual fine-tuning , year =

  91. [101]

    Investigating the catastrophic forgetting in multimodal large language model fine-tuning , year =

    Zhai, Yuexiang and Tong, Shengbang and Li, Xiao and Cai, Mu and Qu, Qing and Lee, Yong Jae and Ma, Yi , booktitle =. Investigating the catastrophic forgetting in multimodal large language model fine-tuning , year =

  92. [102]

    Multimodal Instruction Tuning with Conditional Mixture of LoRA , year =

    Shen, Ying and Xu, Zhiyang and Wang, Qifan and Cheng, Yu and Yin, Wenpeng and Huang, Lifu , journal =. Multimodal Instruction Tuning with Conditional Mixture of LoRA , year =

  93. [103]

    Effectiveness assessment of recent large vision-language models , volume =

    Jiang, Yao and Yan, Xinyu and Ji, Ge-Peng and Fu, Keren and Sun, Meijun and Xiong, Huan and Fan, Deng-Ping and Khan, Fahad Shahbaz , journal =. Effectiveness assessment of recent large vision-language models , volume =

  94. [104]

    Investigating the Catastrophic Forgetting in Multimodal Large Language Model Fine-Tuning , year =

    Zhai, Yuexiang and Tong, Shengbang and Li, Xiao and Cai, Mu and Qu, Qing and Lee, Yong Jae and Ma, Yi , booktitle =. Investigating the Catastrophic Forgetting in Multimodal Large Language Model Fine-Tuning , year =

  95. [105]

    Learning to learn without forgetting by maximizing transfer and minimizing interference , year =

    Riemer, Matthew and Cases, Ignacio and Ajemian, Robert and Liu, Miao and Rish, Irina and Tu, Yuhai and Tesauro, Gerald , journal =. Learning to learn without forgetting by maximizing transfer and minimizing interference , year =

  96. [106]

    On tiny episodic memories in continual learning , year =

    Chaudhry, Arslan and Rohrbach, Marcus and Elhoseiny, Mohamed and Ajanthan, Thalaiyasingam and Dokania, Puneet K and Torr, Philip HS and Ranzato, Marc'Aurelio , journal =. On tiny episodic memories in continual learning , year =

  97. [107]

    Lora: Low-rank adaptation of large language models , year =

    Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =. Lora: Low-rank adaptation of large language models , year =

  98. [108]

    Lora learns less and forgets less , year =

    Biderman, Dan and Ortiz, Jose Gonzalez and Portes, Jacob and Paul, Mansheej and Greengard, Philip and Jennings, Connor and King, Daniel and Havens, Sam and Chiley, Vitaliy and Frankle, Jonathan and others , journal =. Lora learns less and forgets less , year =

  99. [109]

    Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models , year =

    Zhu, Didi and Sun, Zhongyi and Li, Zexi and Shen, Tao and Yan, Ke and Ding, Shouhong and Kuang, Kun and Wu, Chao , booktitle =. Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models , year =

  100. [110]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration , volume =

    Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song , journal =. Awq: Activation-aware weight quantization for on-device llm compression and acceleration , volume =

  101. [111]

    Activation-Informed Merging of Large Language Models , year =

    Amin Heyrani Nobari and Kaveh Alimohammadi and Ali ArjomandBigdeli and Akash Srivastava and Faez Ahmed and Navid Azizan , journal =. Activation-Informed Merging of Large Language Models , year =

  102. [112]

    Activation-Guided Consensus Merging for Large Language Models , year =

    Yao, Yuxuan and Liu, Shuqi and Liu, Zehua and Li, Qintong and Liu, Mingyang and Han, Xiongwei and Guo, Zhijiang and Wu, Han and Song, Linqi , journal =. Activation-Guided Consensus Merging for Large Language Models , year =

  103. [113]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding , year =

    Han, Song and Mao, Huizi and Dally, William J , journal =. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding , year =

  104. [114]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks , year =

    Frankle, Jonathan and Carbin, Michael , booktitle =. The lottery ticket hypothesis: Finding sparse, trainable neural networks , year =

  105. [115]

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI , year =

    Xiang Yue and Yuansheng Ni and Kai Zhang and Tianyu Zheng and Ruoqi Liu and Ge Zhang and Samuel Stevens and Dongfu Jiang and Weiming Ren and Yuxuan Sun and Cong Wei and Botao Yu and Ruibin Yuan and Renliang Sun and Ming Yin and Boyuan Zheng and Zhenzhu Yang and Yibo Liu and We...

  106. [116]

    arXiv preprint arXiv:2302.13971 , title =

    Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth. arXiv preprint arXiv:2302.13971 , title =

    arXiv

  107. [117]

    Llama 2: Open foundation and fine-tuned chat models , year =

    Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others , journal =. Llama 2: Open foundation and fine-tuned chat models , year =

  108. [118]

    and Stoica, Ion and Xing, Eric P

    Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P. , month =. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ url =

  109. [119]

    Learning transferable visual models from natural language supervision , year =

    Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and others , booktitle =. Learning transferable visual models from natural language supervisi...

  110. [120]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , year =

    Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven , booktitle =. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , year =

  111. [121]

    Gradient episodic memory for continual learning , volume =

    Lopez-Paz, David and Ranzato, Marc'Aurelio , journal =. Gradient episodic memory for continual learning , volume =

  112. [122]

    Parameter-efficient transfer learning for NLP , year =

    Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanislaw and Morrone, Bruna and De Laroussilhe, Quentin and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain , booktitle =. Parameter-efficient transfer learning for NLP , year =

  113. [123]

    The power of scale for parameter-efficient prompt tuning , year =

    Lester, Brian and Al-Rfou, Rami and Constant, Noah , journal =. The power of scale for parameter-efficient prompt tuning , year =

  114. [124]

    Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models , year =

    Ni, Shiwen and Chen, Dingwei and Li, Chengming and Hu, Xiping and Xu, Ruifeng and Yang, Min , journal =. Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models , year =

  115. [125]

    DoRA: Weight-Decomposed Low-Rank Adaptation , year =

    Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung , booktitle =. DoRA: Weight-Decomposed Low-Rank Adaptation , year =

  116. [126]

    LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin , year =

    Dou, Shihan and Zhou, Enyu and Liu, Yan and Gao, Songyang and Shen, Wei and Xiong, Limao and Zhou, Yuhao and Wang, Xiao and Xi, Zhiheng and Fan, Xiaoran and others , booktitle =. LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin , year =

  117. [127]

    Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models , year =

    Liang, Jian and Huang, Wenke and Wan, Guancheng and Yang, Qu and Ye, Mang , booktitle =. Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models , year =

  118. [128]

    Learn to preserve and diversify: Parameter-efficient group with orthogonal regularization for domain generalization , year =

    Hu, Jiajun and Zhang, Jian and Qi, Lei and Shi, Yinghuan and Gao, Yang , booktitle =. Learn to preserve and diversify: Parameter-efficient group with orthogonal regularization for domain generalization , year =

  119. [129]

    CorDA: Context-Oriented Decomposition Adaptation of Large Language Models , year =

    Yang, Yibo and Li, Xiaojie and Zhou, Zhongzhu and Song, Shuaiwen Leon and Wu, Jianlong and Nie, Liqiang and Ghanem, Bernard , journal =. CorDA: Context-Oriented Decomposition Adaptation of Large Language Models , year =

  120. [130]

    Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning , year =

    Wenke Huang and Jian Liang and Zekun Shi and Didi Zhu and Guancheng Wan and He Li and Bo Du and Dacheng Tao and Mang Ye , booktitle =. Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning , year =

  121. [131]

    Sigmoid loss for language image pre-training , year =

    Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas , booktitle =. Sigmoid loss for language image pre-training , year =

  122. [132]

    Llava-onevision: Easy visual task transfer , year =

    Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Li, Yanwei and Liu, Ziwei and Li, Chunyuan , journal =. Llava-onevision: Easy visual task transfer , year =

  123. [133]

    2025 , url =

    Zhen Zhu and Yiming Gong and Yao Xiao and Yaoyao Liu and Derek Hoiem , title =. 2025 , url =

    2025

  124. [134]

    Learning towards minimum hyperspherical energy , volume =

    Liu, Weiyang and Lin, Rongmei and Liu, Zhen and Liu, Lixin and Yu, Zhiding and Dai, Bo and Song, Le , journal =. Learning towards minimum hyperspherical energy , volume =

  125. [135]

    OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge , year =

    Kenneth Marino and Mohammad Rastegari and Ali Farhadi and Roozbeh Mottaghi , booktitle =. OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge , year =

  126. [136]

    Ocr-vqa: Visual question answering by reading text in images , year =

    Mishra, Anand and Shekhar, Shashank and Singh, Ajeet Kumar and Chakraborty, Anirban , booktitle =. Ocr-vqa: Visual question answering by reading text in images , year =

  127. [137]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering , year =

    Hudson, Drew A and Manning, Christopher D , booktitle =. Gqa: A new dataset for real-world visual reasoning and compositional question answering , year =

  128. [138]

    Towards vqa models that can read , year =

    Singh, Amanpreet and Natarajan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus , booktitle =. Towards vqa models that can read , year =

  129. [139]

    Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning , year =

    Lu, Pan and Qiu, Liang and Chen, Jiaqi and Xia, Tony and Zhao, Yizhou and Zhang, Wei and Yu, Zhou and Liang, Xiaodan and Zhu, Song-Chun , booktitle =. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning , year =

  130. [140]

    Microsoft COCO: Common Objects in Context , year =

    Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Microsoft COCO: Common Objects in Context , year =. ECCV , pages =

  131. [141]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch , year =

    Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin , booktitle =. Language models are super mario: Absorbing abilities from homologous models as a free lunch , year =

  132. [142]

    International Conference on Learning Representations , volume=

    A simple and effective pruning approach for large language models , author=. International Conference on Learning Representations , volume=

  133. [143]

    arXiv preprint arXiv:2506.05453 , year=

    Mllm-cl: Continual learning for multimodal large language models , author=. arXiv preprint arXiv:2506.05453 , year=

  134. [144]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  135. [145]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  136. [146]

    Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages=

    Docvqa: A dataset for vqa on document images , author=. Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages=

  137. [147]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    V*: Guided visual search as a core mechanism in multimodal llms , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  138. [148]

    European conference on computer vision , pages=

    A diagram is worth a dozen images , author=. European conference on computer vision , pages=. 2016 , organization=

    2016

  139. [149]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Modalprompt: Towards efficient multimodal continual instruction tuning with dual-modality guided prompt , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  140. [150]

    ICML , year=

    SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning , author=. ICML , year=

  141. [151]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  142. [152]

    Proceedings of the computer vision and pattern recognition conference , pages=

    Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering , author=. Proceedings of the computer vision and pattern recognition conference , pages=

  143. [153]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Orthogonal subspace learning for language model continual learning , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  144. [154]

    Advances in Neural Information Processing Systems , volume=

    CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  145. [155]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Federated continual instruction tuning , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.