Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Is Extending Modality The Right Path Towards Omni-Modality?

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Making an LLM multimodal by fine-tuning buys visual knowledge but degrades reasoning, instruction-following, and safety; weighted model merging preserves the original model's abilities better than omni- or modality-specific fine-tuning.

desk verdict A useful empirical study of modality fine-tuning and merging with a clear qualitative story; the omni-modality comparison (RQ3) is confounded and needs rework before that claim stands. read the letter →

arxiv 2506.01872 v1 pith:YZRHIQYI submitted 2025-06-02 cs.CL cs.CV

Tinghui Zhu , Kai Zhang , Muhao Chen , Yu Su This is my paper
classification cs.CLcs.CV
keywords modalityextensionomni-modallanguagemodelsmodelmergingweightedaverageparametershifttrade-offinstructionfollowingmultimodalfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the dominant way of building multimodal language models—taking an off-the-shelf LLM and fine-tuning it on paired modality data (text-image, text-video, text-audio)—is the right road to genuine omni-modality. The authors find a recurring trade-off: modality fine-tuning can expand what the model knows, especially with visual data, but it systematically erodes core language abilities such as reasoning, instruction following, and safety. They show that weighted model merging, where the parameter shift from the original LLM sets each candidate model's importance, preserves the base model's text abilities far better than plain averaging and better than any single fine-tuned extension, though it still trails specialized models on their own modalities. They also report that omni-modality fine-tuning, as practiced by models such as NextGPT, is less effective and less data-efficient than training separate modality-specific models. The paper's conclusion is that neither naive fine-tuning nor merging alone currently delivers robust omni-modality.

What carries the argument

The mechanism that carries the argument is the parameter shift between a base LLM and its modality-fine-tuned extension, defined as $\Delta_{\mathrm{avg}} = \mathrm{avg}\,|\theta_{\mathrm{ori}}-\theta_{\mathrm{mft}}|$ per tensor. The paper uses this shift both as a diagnostic—larger shift accompanies larger specialization and larger text degradation, with Qwen2-VL's shift about ten times that of the LLaVA variants—and as the weight signal for weighted average model merging: softmax over $\{\Delta^i_{\mathrm{avg}}\}$ gives each candidate's $\alpha_i$, and a manually set $\alpha_0$ reserves weight for the original LLM in $\theta_{\mathrm{merge}} = \alpha_0\theta_0 + (1-\alpha_0)\sum_i \alpha_i \theta_i$. The complementary mechanism is the head-masking probe on MMMU, which shows that no attention head can be removed without a large drop in multimodal accuracy, with shallow layers strongest, implying merging cannot be restricted to an identified subset. Together these make parameter-space geometry, not architecture or data, the paper's explanatory lens for why fine-tuning trades off and merging preserves.

What would settle it

Run the same base LLM (e.g., Qwen2-7B-Instruct) through image-only, video-only, audio-only, and combined omni-modality fine-tuning with matched data volume, matched training steps, and identical encoders, then evaluate on identical text benchmarks (MMLU, MATH, IFEval, HarmBench) and per-modality benchmarks. If the omni fine-tuned model matches or beats each specialized model on its own modality while degrading text abilities no more than the specialized models do, the paper's core trade-off and efficiency conclusions would be overturned; if, conversely, the omni model shows the same MMLU-Pro gain and MATH drop, the trade-off is confirmed as intrinsic to the fine-tuning procedure.

Watch

Extended reading notes

Core claim

The central claim is that extending modality—unfreezing an LLM and fine-tuning it on modality-aligned data—improves specific capabilities while damaging the base model's language faculties, and that this damage is not inevitable but must be engineered around. Concretely, the paper reports that visual extensions of Qwen2-7B-Instruct raise MMLU-Pro by up to five points while cutting MATH by up to ten points and IFEval instruction-following by roughly ten points; audio extension adds almost no knowledge; and safety declines on HarmBench across nearly all models. Weighted average merging, with softmax-normalized parameter shifts as importance weights plus a held-out weight for the original LLM, produces a merged model that matches or beats the base LLM on MMLU, IFEval, multilingual, and safety while recovering part of the reasoning loss. The head-masking experiment shows every attention head matters for multimodal performance, with shallow layers most influential, ruling out merging only a sparse subset of parameters. Finally, omni-modality fine-tuning fails the efficiency test: LLaVA-Next (image) and Vista-LLaMA (video) outperform NextGPT on their modalities with a third to half the data, and small-step fine-tuning of a merged model re-introduces the text/multimodal trade-off after about one thousand steps.

Load-bearing premise

The comparisons across modalities and training paradigms use models that differ in base LLM, architecture, data volume, and training recipe, so the paper assumes that any performance gap is caused by the modality-extension paradigm itself rather than by those confounds.

Editorial extensions

If this is right

  • Practitioners fine-tuning an LLM for a new modality should expect a three-way split: knowledge benchmarks rise (especially with large visual data), while instruction-following, math and code reasoning, and safety fall; mitigation must be part of the training plan, not an afterthought.
  • Weighted model merging with parameter-shift importance is a zero-training way to combine existing modality-specific checkpoints into an omni-modal model that keeps the base LLM's text abilities largely intact, viable whenever the candidate models share an architecture and base weights.
  • Omni-modality fine-tuning is currently the least efficient route to multi-modality: specialized image and video models achieve better per-modality scores with considerably less data, so omni-training should be judged against those specialized baselines.
  • Because every attention head shows measurable modality salience and shallow heads matter most, any merging or pruning scheme for multimodal models should weight early-layer parameters heavily rather than targeting a sparse head subset.
  • A merged omni-modal model can be improved by fine-tuning, but only for one side of the trade-off: multimodal scores rise while text scores start falling after roughly one thousand steps, so optimal step counts need to be tuned per modality or per ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the trade-off is intrinsic to gradient fine-tuning on modality data, then keeping the base LLM's weights frozen and routing modality knowledge through external adapters or memory modules is a natural next test: it would isolate whether the reasoning loss can be avoided entirely.
  • The parameter-shift signal can be used prospectively: before evaluating, compare the shift magnitude and location of candidate modality checkpoints against their shared base to predict which will add knowledge and which will erode text abilities.
  • The step-count divergence between text and multimodal benchmarks after merging suggests a concrete design lever: per-modality step budgets or a continuous text-instruction stream during multimodal fine-tuning could resolve the observed trade-off.
  • The audio result's smaller knowledge gain suggests audio extension behaves more like text extension than like vision extension, which would imply audio-specific training should maximize textual alignment; a same-base audio model could test that directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies whether extending a base LLM to additional modalities ('modality extension') is the right route toward omni-modal language models. It poses three questions: whether modality fine-tuning degrades core language abilities (RQ1), whether model merging of independently fine-tuned modality-specific models yields an effective OLM (RQ2), and whether omni-modality fine-tuning outperforms sequential/specialized extension (RQ3). Based on evaluations across text, image, video, and audio benchmarks, the authors report a trade-off: modality fine-tuning improves knowledge and some long-context abilities but harms reasoning, instruction following, and safety. They propose weighted-average merging with weights derived from per-tensor parameter shifts, and claim it best preserves the base LLM's abilities. For RQ3, they compare NextGPT and Qwen2.5-Omni against several modality-specialized models and conclude that omni-modality fine-tuning is less effective and less efficient.

Significance. If established, these findings would provide useful practical guidance for building omni-modal LLMs and could inform training strategies. The paper's strengths include a broad evaluation suite covering six textual ability families, the use of a consistent Qwen2-7B-Instruct family for most RQ1/RQ2 comparisons, a released code repository, and an interesting head-masking analysis of parameter salience. However, the central claims as stated are not established: the RQ3 conclusion rests on comparisons that violate the paper's own definition of the target paradigm, and the RQ2 'weighted merging is best' claim is directly contradicted by the reported multimodal numbers. The manuscript also reports no variance estimates anywhere, so the quantitative conclusions are not robustly supported.

major comments (5)
  1. [Section 6 and Table 6] The comparison in Table 6 cannot answer RQ3 because NextGPT is not an instance of omni-modality fine-tuning as defined in Section 3.1, which restricts the term to training with the LLM unfrozen; Section 6.1.1 states that NextGPT uses a frozen language backbone. Further, NextGPT is built on a different base LLM than the specialized models (e.g., LLaVA-Next is Vicuna-based), so the observed gaps on VizWiz and VQAv2 conflate the omni-vs-specialized contrast with backbone trainability, base model choice, and data composition. The Qwen2.5-Omni results in Table 7 are compared only against the text-only base and the merged model, not against a matched modality-specialized model, so they do not repair the confound. As written, the paper's negative answer to the title question is not supported by the experiments.
  2. [Section 4.2 and Table 3] The audio-modality analysis uses a mismatched baseline. Qwen2-Audio-7B-Instruct is reported (Table 2) as an extension of Qwen-7B-Instruct, but the evaluation in Table 3 compares it against 'Qwen-7B-Chat', which is a different base model (Qwen-7B-Chat, not Qwen2-7B-Chat). The claim that the audio model improves MMLU and MMLU-Pro by only 0.4% over its base is therefore not meaningful; the improvement must be assessed against Qwen2-7B-Chat. This undermines the conclusion about audio modality contributing limited parametric knowledge.
  3. [Section 5.5 and Table 5] The abstract and Section 5.5 state that weighted model merging 'achieves the best performance across both textual and multimodal tasks,' but the multimodal results in Table 5 contradict this: Qwen2-weighted-all scores 48.11 on MMMU, below both Qwen2-avg-all (48.78) and the original Qwen2-VL-7B-Instruct (49.44); on Video-MME it scores 61.04, below Qwen2-VL-7B-Instruct's 62.84. The textual results also do not uniformly favor weighted merging (e.g., MMLU-Pro 36.3 for weighted vs 37.4 for average in the three-model merge). The claim needs to be either corrected or supported with statistical significance testing and an explicit account of why weighted merging is preferred despite these numbers.
  4. [Throughout (Tables 3--7 and Figure 3)] No error bars, confidence intervals, or multiple-seed runs are reported anywhere. Many of the differences used to support conclusions are small (e.g., a 1--3% change on MMLU or accuracy differences of 1--2 points on multimodal benchmarks), and single-run greedy decoding cannot distinguish systematic effects from noise. Additionally, the 'alpha_0' parameter in Eq. (4) is a manually assigned weight with no sensitivity analysis, and the threshold of approximately 1,000 fine-tuning steps in Section 6.2.2 is read directly from Figure 3 without variance or a principled selection rule. The paper's quantitative claims are therefore not robustly grounded.
  5. [Section 6.2.2 and Figure 4] The conclusion that fine-tuning on different modalities 'propels the model weights in distinct directions' relies on t-SNE visualizations of weight distributions. t-SNE is a stochastic, low-dimensional embedding whose distances and directions are not quantitatively interpretable; the figure is suggestive but does not provide evidence for the claimed mechanism. The 'modality trade-off' argument would be strengthened by a quantitative metric such as cosine similarity between parameter updates or a controlled intervention test, but none is provided.
minor comments (6)
  1. [Section 2.2] The phrase 'leaving the effective training pattern of OLMs indefinitive' contains a typo; 'indefinitive' should read 'inconclusive' or 'not definitive.'
  2. [Section 5.1, Eq. (1)] Equation (1) is labeled 'average merging' but is written as a simple sum without the factor 1/n; this is mathematically incorrect and confusing, since Eq. (2) also omits normalization for the weighted case.
  3. [Section 3.2 and Table 3] In Table 3, the LLaVA-OneVision-Qwen2-7B-SI row reports Pass@1 and Pass@5 on HumanEval+ as 0.0, which is an extreme outcome for an instruction-tuned model; the authors should verify this result and, if correct, comment on why the model fails entirely on code generation.
  4. [Section 5.3, Eq. (4)] The notation in Eq. (4) is ambiguous: the index of the sum over candidate models is not specified, and it is unclear whether the softmax weights alpha_i are computed over the raw delta values or their averages; the text should define all symbols explicitly.
  5. [Figure 2] The heat map caption states that 'accuracy and probability accuracy should be as high as possible, while the KL divergence should have small absolute value,' but the color scales are not explained; it is unclear whether darker or lighter cells indicate better performance, which makes the figure hard to read.
  6. [Section 6.2.1] The fine-tuning dataset proportions are listed as text:image:video = 3:2:1, but the rationale is only that they 'approximate a balanced token exposure'; the actual token counts per dataset are not reported, and the sample size of 1,000 instances per modality is small, so the results may not be representative.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: core claims are external-benchmark evaluations of existing models and an independently evaluated merging recipe; confounds affect validity, not circularity.

full rationale

The paper's central claims are empirical comparisons of existing models and proposed merging recipes; none of the claims reduces by construction to its inputs. The modality fine-tuning analysis (Section 4) evaluates off-the-shelf model families against their published base LLMs on external textual benchmarks, so it is self-contained evidence rather than a fit. The weighted merging method (Section 5) sets weights from parameter-shift magnitudes (Eqs. 3-4) with a manually chosen alpha_0; the alpha_i are not optimized on the evaluation benchmarks, and the comparison of merged vs. average vs. base models is a genuine out-of-sample evaluation of a proposed method, not a renamed input. The omni-modality comparison (Section 6) is confounded because NextGPT uses a frozen backbone and different base LLMs than the specialized models, but confounding is a validity concern, not circularity, since the compared numbers are external measurements. The two self-citations ([43], [65]) are used for related-work context and a layer-depth intuition and do not carry the derivation. No step in the paper equates a fitted parameter with a predicted quantity or imports a uniqueness theorem from the authors' own prior work.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a handful of hand-chosen hyperparameters (alpha_0, data mixing ratio, step threshold) and on unvalidated assumptions about benchmark validity, model representativeness, and the head-masking probe. No new physical or architectural entities are postulated.

free parameters (3)
  • alpha_0 = not specified
    Manually assigned weight for the original LLM in Eq. (4); the paper never gives its value or a sensitivity sweep.
  • fine-tuning data proportion = text:image:video = 3:2:1
    Chosen by hand in Section 6.2.1 to approximate balanced token exposure; no ablation.
  • small-step fine-tuning step count = up to 2000 steps, optimal around 1000
    The step threshold separating language degradation from multimodal gain is read post hoc from Figure 3 and used to define the modality trade-off.
assumptions (4)
  • domain assumption Public benchmarks (MMLU, MMMU, Video-MME, HarmBench, etc.) measure the abilities they claim to measure.
    All conclusions about ability trade-offs are read off these benchmarks without independent validation.
  • domain assumption The selected off-the-shelf models are representative instances of the training paradigms being compared.
    Sections 4 and 6 compare a small set of public models and treat them as typifying modality fine-tuning, merging, and omni fine-tuning.
  • ad hoc to paper Masking a single attention head and measuring accuracy drop is a valid probe of modality salience.
    Section 5.2 uses this without baseline masking results or ablation of the masking procedure.
  • ad hoc to paper t-SNE projections of weight vectors expose meaningful structure about fine-tuning versus merging.
    Figure 4 is interpreted qualitatively to support the specialization-versus-interpolation story.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Extending Modality The Right Path Towards Omni-Modality?." pith.science (2026). https://pith.science/paper/YZRHIQYI

@misc{pith2026250601872,
  author       = {Pith},
  title        = {Pith review of: Is Extending Modality The Right Path Towards Omni-Modality?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZRHIQYI}},
  note         = {Machine review of arXiv:2506.01872}
}
read the original abstract

Omni-modal language models (OLMs) aim to integrate and reason over diverse input modalities--such as text, images, video, and audio--while maintaining strong language capabilities. Despite recent advancements, existing models, especially open-source ones, remain far from true omni-modality, struggling to generalize beyond the specific modality pairs they are trained on or to achieve strong performance when processing multi-modal inputs. We study the effect of extending modality, the dominant technique for training multimodal models, where an off-the-shelf language model is fine-tuned on target-domain and language data. Specifically, we investigate three key questions: (1) Does modality extension compromise core language abilities? (2) Can model merging effectively integrate independently fine-tuned modality-specific models to achieve omni-modality? (3) Does omni-modality extension lead to better knowledge sharing and generalization compared to sequential extension? Through extensive experiments, we analyze these trade-offs and provide insights into the feasibility of achieving true omni-modality using current approaches.

Figures

Figures reproduced from arXiv: 2506.01872 by the authors.

Figure 1
Figure 1. Overview of modality-specific language models and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Heat map for masking each attention head. We report accuracy, accuracy calculated by probability, and KL divergence. The [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Comparative visualizations of model weight distribu [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations

    cs.IR 2025-08 conditional novelty 5.0 of 10

    Replacing raw video and audio features with MLLM-generated natural-language captions improves hit rate and nDCG for two-tower and SASRec recommenders on MicroLens-100K.

Reference graph

Works this paper leans on

65 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Cross-modal knowl- edge transfer without task-relevant source data

    Sk Miraj Ahmed, Suhas Lohit, Kuan-Chuan Peng, Michael J Jones, and Amit K Roy-Chowdhury. Cross-modal knowl- edge transfer without task-relevant source data. InEuropean Conference on Computer Vision, pages 111–127. Springer,

    work page

  2. [2]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Vizwiz: nearly real-time answers to visual questions

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. InProceedings of the 23nd annual ACM symposium on User interface soft- ware and technology, pages 333–342, 2010. 8

    work page 2010

  4. [4]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 4

    2023

  5. [5]

    Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhi- fang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024. 4

    arXiv 2024

  6. [6]

    Command a: An enterprise-ready large language model.arXiv preprint arXiv:2504.00698, 2025

    Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alam- mar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhang- orodsky, Viraat Aryabumi, Dennis Aumiller, Rapha ¨el Ava- los, et al. Command a: An enterprise-ready large language model.arXiv preprint arXiv:2504.00698, 2025. 2, 9

    arXiv 2025

  7. [7]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  8. [8]

    Mind2web: To- wards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: To- wards a generalist agent for the web. InThirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 1

    work page 2023

Show all 65 references
  1. [9]

    Un- locking continual learning abilities in language models

    Wenyu Du, Shuang Cheng, Tongxu Luo, Zihan Qiu, Zeyu Huang, Ka Chun Cheung, Reynold Cheng, and Jie Fu. Un- locking continual learning abilities in language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6503–6522, 2024. 2

    2024

  2. [10]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024. 1, 7

    2024 arXiv

  3. [11]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 8

    2017

  4. [12]

    Mea- suring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. 3

    2009 arXiv

  5. [13]

    Mea- suring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. InInter- national Conference on Learning Representations, 2020. 3

    2020

  6. [14]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021. 3

    2021 arXiv

  7. [15]

    A survey on hal- lucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Infor- mation Systems, 43(2):1–55, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hal- lucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Infor- ma...

    2025

  8. [16]

    Vi- sual hallucinations of multi-modal large language models

    Wen Huang, Hongbin Liu, Minxin Guo, and Neil Gong. Vi- sual hallucinations of multi-modal large language models. InFindings of the Association for Computational Linguistics ACL 2024, pages 9614–9631, 2024. 2

    2024

  9. [17]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2

    2024 arXiv

  10. [18]

    Visualwebinstruct: Scaling up mul- timodal instruction data through web search.arXiv preprint arXiv:2503.10582, 2025

    Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, and Wenhu Chen. Visualwebinstruct: Scaling up mul- timodal instruction data through web search.arXiv preprint arXiv:2503.10582, 2025. 9

    2025 arXiv

  11. [19]

    Balancing continuous pre- training and instruction fine-tuning: Optimizing instruction- following in llms.arXiv preprint arXiv:2410.10739, 2024

    Ishan Jindal, Chandana Badrinath, Pranjal Bharti, Lakkidi Vinay, and Sachin Dev Sharma. Balancing continuous pre- training and instruction fine-tuning: Optimizing instruction- following in llms.arXiv preprint arXiv:2410.10739, 2024. 4

    arXiv

  12. [20]

    Designing incremental knowledge enrichment in generative pre-trained transformers

    Emilia A Kowalczyk, Mateusz Nowakowski, and Zofia Brzezi´nska. Designing incremental knowledge enrichment in generative pre-trained transformers. 2024. 2

    2024

  13. [21]

    How does vision-language adaptation impact the safety of vision lan- guage models? InThe Thirteenth International Conference on Learning Representations, 2025

    Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, and Minjoon Seo. How does vision-language adaptation impact the safety of vision lan- guage models? InThe Thirteenth International Conference on Learning Representations, 2025. 5

    2025

  14. [22]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 4

    2024 arXiv

  15. [23]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 1, 2, 3 11

    2023

  17. [25]

    Should we really edit lan- guage models? on the evaluation of edited language mod- els.Advances in Neural Information Processing Systems, 37:30850–30885, 2025

    Qi Li, Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Xinglin Pan, and Xiaowen Chu. Should we really edit lan- guage models? on the evaluation of edited language mod- els.Advances in Neural Information Processing Systems, 37:30850–30885, 2025. 3

    2025

  18. [26]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 26689–26699, 2024. 3

    2024

  19. [27]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 1, 2, 3, 4

    2023

  20. [28]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 8

    2024

  21. [29]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rig- orous evaluation of large language models for code genera- tion.Advances in Neural Information Processing Systems, 36:21558–21572, 2023. 3

    2023

  22. [30]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment.arXiv preprint arXiv:2502.04328, 2025

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment.arXiv preprint arXiv:2502.04328, 2025. 2

    2025 arXiv

  23. [31]

    Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022. 2

    2022 arXiv

  24. [32]

    Twin-merging: Dynamic integration of modular expertise in model merging.Advances in Neural Information Processing Systems, 37:78905–78935, 2025

    Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging.Advances in Neural Information Processing Systems, 37:78905–78935, 2025. 5

    2025

  25. [33]

    Vista-llama: Reliable video narrator via equal distance to visual tokens.arXiv preprint arXiv:2312.08870,

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narrator via equal distance to visual tokens.arXiv preprint arXiv:2312.08870,

    arXiv

  26. [34]

    A survey on vision-language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024

    Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024. 1

    2024 arXiv

  27. [35]

    Harmbench: a standardized evalua- tion framework for automated red teaming and robust re- fusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: a standardized evalua- tion framework for automated red teaming and robust re- fusal. InProceedings of the 41st International Conference...

    2024

  28. [36]

    Zero- shot tokenizer transfer

    Benjamin Minixhofer, Edoardo Ponti, and Ivan Vuli ´c. Zero- shot tokenizer transfer. InThe Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. 3

    2024

  29. [37]

    Landmark atten- tion: Random-access infinite context length for transformers

    Amirkeivan Mohtashami and Martin Jaggi. Landmark atten- tion: Random-access infinite context length for transformers. InWorkshop on Efficient Systems for Foundation Models@ ICML2023, 2023. 3

    2023

  30. [38]

    Gpt-4v(ision) system card, 2024

    OpenAI. Gpt-4v(ision) system card, 2024. 1, 2

    2024

  31. [39]

    Recyclable tuning for continual pre-training

    Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Recyclable tuning for continual pre-training. InFind- ings of the Association for Computational Linguistics: ACL 2023, pages 11403–11426, 2023. 3

    2023

  32. [40]

    Gpqa: A graduate-level google- proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. InFirst Conference on Language Modeling, 2024. 3

    2024

  33. [41]

    Zeroscrolls: A zero-shot benchmark for long text understanding

    Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. InFindings of the Association for Com- putational Linguistics: EMNLP 2023, pages 7977–7989,

    2023

  34. [42]

    Visualizing data using t-sne.Journal of machine learning research, 9 (11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9 (11), 2008. 10

    2008

  35. [43]

    Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization

    Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization. InThe Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024. 7

    2024

  36. [44]

    Bring your own knowl- edge: A survey of methods for llm knowledge expansion

    Mingyang Wang, Alisa Stoll, Lukas Lange, Heike Adel, Hin- rich Sch ¨utze, and Jannik Str ¨otgen. Bring your own knowl- edge: A survey of methods for llm knowledge expansion. arXiv preprint arXiv:2502.12598, 2025. 2

    2025 arXiv

  37. [45]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 4

    2024 arXiv

  38. [46]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Infor- mation P...

  39. [47]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without ...

    2022

  40. [48]

    Llama pro: Progressive llama with block expansion

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. Llama pro: Progressive llama with block expansion. InProceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 6518– 6537, 2024. 3

    2024

  41. [49]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. InForty-first International Conference on Machine Learning, 2024. 1, 2, 8

    2024

  42. [50]

    Continual learn- ing for large language models: A survey.arXiv preprint arXiv:2402.01364, 2024

    Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learn- ing for large language models: A survey.arXiv preprint arXiv:2402.01364, 2024. 2 12

    2024 arXiv

  43. [51]

    Retrieval head mechanistically explains long- context factuality

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long- context factuality. InThe Thirteenth International Confer- ence on Learning Representations, 2025. 6

    2025

  44. [52]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. InProceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017. 8

    2017

  45. [53]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025. 8

    2025 arXiv

  46. [54]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  47. [55]

    Recent advances of foundation language models-based continual learning: A survey.ACM Computing Surveys, 57(5):1–38, 2025

    Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey.ACM Computing Surveys, 57(5):1–38, 2025. 2

    2025

  48. [56]

    Editing large language models: Problems, methods, and opportunities

    Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222–...

    2023

  49. [57]

    Metamath: Bootstrap your own mathe- matical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathe- matical questions for large language models.arXiv preprint arXiv:2309.12284, 2023. 9

    2023 arXiv

  50. [58]

    Information association for lan- guage model updating by mitigating lm-logical discrepancy

    Pengfei Yu and Heng Ji. Information association for lan- guage model updating by mitigating lm-logical discrepancy. InProceedings of the 28th Conference on Computational Natural Language Learning, pages 117–129, 2024. 3

    2024

  51. [59]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. InProceedings of the IEEE/CVF Conference on ...

    2024

  52. [60]

    Mm-llms: Recent advances in multimodal large language models

    Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. InFindings of the Association for Computational Linguistics ACL 2024, pages 12401–12430, 2024. 2

    2024

  53. [61]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. InProceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 543–553, 2023. 2, 8

    2023

  54. [62]

    Cppo: Continual learning for rein- forcement learning with human feedback

    Han Zhang, Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, and Ruifeng Xu. Cppo: Continual learning for rein- forcement learning with human feedback. InThe Twelfth In- ternational Conference on Learning Representations, 2024. 3

    2024

  55. [63]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 1, 2, 4, 5, 8, 9

    2024 arXiv

  56. [64]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 3

    2023 arXiv

  57. [65]

    Unraveling cross-modality knowledge con- flicts in large vision-language models.arXiv preprint arXiv:2410.03659, 2024

    Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. Unraveling cross-modality knowledge con- flicts in large vision-language models.arXiv preprint arXiv:2410.03659, 2024. 2 13

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.