Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decoder-only multimodal LLMs carry substantial, structured redundancy in visual token processing: cutting self-attention and FFN computation on visual tokens in roughly half of layers preserves or improves benchmark performance while…

desk verdict A solid, honest empirical study showing that visual-token computation in decoder-only MLLMs is substantially redundant across about half the layers; the acceleration method is a useful byproduct, but reproducibility gaps and one overclaim need fixing. read the letter →

arxiv 2501.19036 v3 pith:GNAL62S3 submitted 2025-01-31 cs.CV

Hongliang Li , Jiaxin Zhang , Wenhui Liao , Dezhi Peng , Kai Ding , Lianwen Jin This is my paper · ORCID
classification cs.CV
keywords visualtokenredundancydecoder-onlyMLLMtraining-freeaccelerationdynamicFFNpruninghollowattentionlayerrankingFLOPsreductionmultimodalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that decoder-only multimodal large language models process visual tokens with substantial, structured, and clustered redundancy, so that much of the self-attention and FFN computation spent on those tokens can be removed without hurting accuracy. It proposes a training-free framework, RedundancyLens, that applies adjustable reductions to visual-token computation in selected layers and measures what happens. The central empirical finding is that reducing visual-token computation in roughly half of the layers keeps performance at or above the baseline on most benchmarks, while doing the same to text tokens sharply degrades performance. If this is right, it offers both a practical inference-speedup method and a design hint for future MLLM architectures: decoder-only models may not need to spend full transformer computation on visual tokens at every layer.

What carries the argument

The framework has three components. Probe-Activated Dynamic FFN reduces FFN computation for visual tokens by randomly sampling a small subset of visual tokens, computing their hidden representations, taking the mean of absolute values to score FFN hidden units, and activating only the top-K scored units for all visual tokens. Hollow Attention replaces global attention among visual tokens with local attention, letting each visual token attend only to the preceding RA visual tokens plus all text tokens, while text tokens keep full attention. Layer Ranking Algorithm is a greedy forward-search procedure that ranks layers by how much performance is preserved when reductions are applied, using a compact validation set with a penalty for performance drops, and it prioritizes later layers, which the experiments show carry more redundancy.

What would settle it

Take one trained decoder-only MLLM, rank layers for reduction on samples from a single task family such as document QA, then evaluate the same reductions on the other three task families; if performance is preserved only on the ranked family and drops on the others, the claimed structured and clustered redundancy is an artifact of the validation-set choice rather than a general property of visual-token processing.

Watch

Extended reading notes

Core claim

The paper reports that trained decoder-only MLLMs exhibit large, structured, and clustered redundancy in the self-attention and FFN operations applied to visual tokens. Using its two reduction modules, it shows that applying computational reductions to about half of the layers maintains or slightly improves performance across eight benchmarks, for both attention and FFN reductions and across multiple current decoder-only MLLMs. Applying the same reductions to text tokens causes a sharp performance collapse, which the paper takes as evidence that the redundancy is specific to visual-token processing rather than a general property of the transformer layers. The authors argue this redundancy is not due to undertraining, since the models were trained on large multimodal corpora, but arises from the inherent difference between visual and text tokens, including the fact that visual tokens have already been processed by an image encoder before entering the LLM. As an application, the framework becomes a training-free acceleration method that reduces FLOPs by roughly 30 percent and can be combined with token-compression methods for further gains.

Load-bearing premise

The layer-ranking algorithm estimates redundancy on a compact validation set of a few thousand examples and assumes that ranking transfers to the full distribution of the eight test benchmarks; if that validation mix is not representative, the observed performance-preserving layer choices may not generalise.

Editorial extensions

If this is right

  • Decoder-only MLLMs can be accelerated at inference time by about 30 percent FLOPs without any training, simply by reducing visual-token FFN and attention computation in roughly half the layers.
  • The per-token computation reduction is orthogonal to visual-token compression: combining it with a token-pruning baseline reaches about 50 percent FLOPs reduction while scoring better on most benchmarks than that baseline alone at the same budget.
  • Text tokens cannot be treated the same way, which implies the redundancy is not a general transformer property but is tied to the visual modality.
  • Redundancy is concentrated in later layers, suggesting future decoder-only architectures can allocate heavy visual-token computation mostly to earlier layers and use lighter processing later.
  • The best acceleration strategy is content-dependent: text-dense images favor reducing computation per visual token, while low-information images favor reducing the number of visual tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same redundancy could be exploited at training time: a decoder-only MLLM trained from scratch with hollow attention or dynamic FFN in late layers might keep most of its accuracy while being inherently faster, rather than patched at inference.
  • The layer-ranking search could potentially be replaced by an unsupervised signal such as attention statistics or hidden-state similarity, removing the need for a validation set and making the method applicable to arbitrary new checkpoints at zero evaluation cost.
  • Because visual tokens come from a rich image encoder while text tokens are linearly mapped, the redundancy may extend to other long token types that enter the LLM after heavy encoding, such as audio frames, video patches, or multi-image inputs.
  • The paper's comparison hints at a hybrid controller that picks, per input or per benchmark, how many layers to reduce and how many tokens to drop, since text-dense and low-information images favor opposite acceleration strategies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training-free framework, RedundancyLens, to analyze and exploit redundancy in how decoder-only MLLMs process visual tokens. The two reduction mechanisms are Probe-Activated Dynamic FFN (Section 3.1.1, Eqs. (3)-(10)), which activates only a subset of FFN parameters for visual tokens, and Hollow Attention (Section 3.1.2), which restricts visual-to-visual attention to a local window while preserving other attention. A Layer Ranking Algorithm (Algorithm 1, Section 3.2) selects which layers to reduce using a compact validation set. Experiments on InternVL2-8B, Qwen2-VL-7B, MiniCPM-V 2.6, and LLaVA-OneVision across eight benchmarks show that reducing visual-token computation in roughly half the layers preserves or even slightly improves performance, while the same reductions applied to text tokens sharply degrade performance. The method is also positioned as a training-free inference accelerator, reporting about 30% FLOPs reduction and showing compatibility with token-pruning methods FastV and VTW.

Significance. If the central claim is correct, the paper provides a useful architectural insight: decoder-only MLLMs carry substantial, layer-clustered redundancy in visual-token self-attention and FFN processing, and this redundancy can be exploited at inference time without retraining. The experiments are broad, covering four recent MLLMs and eight diverse benchmarks, and the paper explicitly compares against FastV and VTW at matched FLOPs ratios. The proposed reduction mechanisms are clearly specified and the combined use with token pruning is a sensible orthogonal contribution. The main limitations are that the layer-ranking validation set overlaps in distribution with three of the eight evaluation benchmarks, no wall-clock latency is measured, and the claim that the redundancy is unique to decoder-only MLLMs is not supported by experiments on cross-attention architectures.

major comments (4)
  1. [Section 4.1 / Algorithm 1] The layer ranking that determines LRA and LRF in Table 1 is selected using a validation set that includes 200 samples each from the DocVQA, InfoVQA, and ChartQA validation splits, and these three datasets are also among the eight evaluation benchmarks. Because the ranker is allowed to pick the layers that best preserve performance on this aggregate, the maintained performance on DocVQA, InfoVQA, and ChartQA in Figure 3 and Table 1 is partly a selection outcome rather than an independent prediction. The held-out benchmarks (MME, MMStar, HallusionBench, TextVQA) mitigate this concern, but the central redundancy claim would be substantially strengthened by a stability analysis, e.g., ranking on one split and evaluating on another, or ranking on only the non-overlapping validation subsets and reporting all eight test curves.
  2. [Section 4.5 / Table 1] The inference-acceleration claim is supported only by FLOPs ratios, not by measured end-to-end latency. The proposed sparse and dynamic operations, including random token sampling, top-K selection, and local attention masking, may carry implementation overhead that prevents the FLOPs reduction from translating into wall-clock speedup on actual hardware. Please report per-benchmark or aggregate latency (e.g., seconds per sample or tokens per second) for the baseline, Ours, and the combined Ours+FastV settings on the same GPU, or explicitly state that only FLOPs are being compared and temper the acceleration language accordingly.
  3. [Abstract / Section 4.4] The paper repeatedly claims that the observed redundancy is 'unique to decoder-only MLLMs,' but all four evaluated models are decoder-only; no cross-attention or hybrid MLLM is tested with the same reduction framework. The comparison to cross-attention architectures in Section 2 and Figure 1 is architectural and conceptual, not experimental. Either soften the claim to 'observed in decoder-only MLLMs' or add an analogous evaluation on at least one cross-attention model to support the uniqueness assertion.
  4. [Section 3.1.2] The definition of Hollow Attention states that 'each visual token attends to the preceding RA visual tokens and all text tokens.' In a standard causal decoder-only MLLM, visual tokens precede text tokens in the sequence and therefore cannot attend to later text tokens under causal masking. If the intended implementation instead preserves text-to-visual attention (as suggested by Figure 5, which says 'the attention from text tokens to all visual tokens needs to be preserved'), the text should be corrected to describe the exact mask pattern, and the attention matrix in Figure 2 should be drawn consistently with that pattern.
minor comments (5)
  1. [Section 3.2 / Section 4.3] The number of pre-ranked layers Lp is never specified, although Algorithm 1 and the hybrid ranking strategy in Figure 7 depend on it. Please state the value of Lp used for each model or explain how it was chosen.
  2. [Section 4.3] During inference, the Probe-Activated Dynamic FFN samples only 10% of visual tokens to estimate the hidden-activation magnitudes, so the selected FFN parameters can vary across forward passes for the same input. Please report the variance or use fixed seeds, and state how many runs were used for the numbers in Table 1.
  3. [Figure 3] The redundancy curves are shown only for InternVL2-8B and Qwen2-VL-7B, while MiniCPM-V 2.6 and LLaVA-OneVision are only reported in Table 1. Adding the corresponding curves for the other two models, or stating that the same qualitative pattern was observed, would make the cross-model claim easier to verify.
  4. [Table 1] Some cells appear to have missing spacing, e.g., '79390.6' and 'FFNAttentionReduction' in Figure 5. Please check the table and figure rendering so that each benchmark column is unambiguous.
  5. [Section 1] The term 'training-free' is used both for the redundancy analysis and for the acceleration method, but the Layer Ranking Algorithm itself requires a construction-validation loop with hundreds of evaluations. Please clarify that no model weights are updated and that the ranking cost is a one-time analysis cost, not a per-inference cost.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the redundancy claim rests on measured performance of frozen models after interventions, not on a fitted equation or self-citation chain.

full rationale

The paper's derivation chain is empirical and self-contained against external benchmarks. The proposed reductions (Probe-Activated Dynamic FFN and Hollow Attention) are defined as computational substitutions, and the Layer Ranking Algorithm selects layers using a validation set; the central curves in Figure 3 and Table 1 are then measured on eight benchmarks. The redundancy conclusion is not equivalent to the ranking procedure: the ranking chooses the least harmful layers, but it cannot force a model to preserve performance if no layer is actually dispensable. The text-token comparison (Figure 4) provides an independent control: the same reductions applied to text tokens degrade sharply, which is a measured contrast rather than a consequence of the definition. The validation set does include subsets of DocVQA, InfoVQA, and ChartQA, and the authors acknowledge in the Limitations that the limited-scale validation set and greedy search 'may fail to identify the optimal combination of layers for reduction'; this is a selection-bias/generalization concern, not a circular derivation. Self-citations (OCRBench, DocKylin) appear as benchmark/related-work references and are not load-bearing for the redundancy argument. There is no imported uniqueness theorem and no ansatz smuggled in via citation. Score 1 reflects only minor self-citation and mild validation-set overlap; no reduction step is circular.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the usual empirical assumptions of frozen-model intervention studies plus a set of hand-selected reduction hyperparameters. No new physical entities are introduced; the reductions are algorithmic interventions on existing modules.

free parameters (6)
  • RA (Hollow Attention range) = 256
    Chosen by hand as the typical number of tokens in one sub-image. Figure 6a shows sensitivity to this value.
  • K (activated FFN parameter fraction) = 20% of original parameter count
    Chosen by hand to balance efficiency and accuracy; Figure 6b shows sensitivity.
  • M (probe sampling ratio) = 10% during inference, 100% during layer ranking
    The probe samples a small fraction of visual tokens to estimate neuron importance; the ranking phase uses all tokens to reduce fluctuation.
  • alpha (penalty coefficient in Layer Ranking) = 2
    Hand-chosen to penalize negative performance differences during layer ranking.
  • Lp (number of last layers pre-ranked by position) = unreported
    The paper states the last Lp layers are ranked by position but never gives the value, preventing exact replication.
  • LRA and LRF (numbers of layers reduced for attention and FFN) = InternVL2-8B: LRA=16, LRF=17; Qwen2-VL-7B: LRA=13, LRF=14; LLaVA-OneVision-7B: LRA=13, LRF=14; MiniCPM-V 2.6: LRA=13…
    Selected per model to match the FLOPs ratios of the baseline methods being compared, so the reported performance depends on these hand-chosen targets.
assumptions (5)
  • domain assumption Neuron importance estimated from a random 10% sample of visual tokens transfers to all visual tokens at a given layer.
    Probe-Activated Dynamic FFN selects top-K units using only M sampled tokens (Eq. 3-8); this transferability is assumed rather than proven.
  • domain assumption Text tokens attending to all visual tokens is sufficient to preserve cross-modal information even when visual tokens attend only locally.
    Hollow Attention retains text-to-visual attention and assumes this channel carries the information needed for downstream reasoning.
  • domain assumption Frozen pretrained MLLMs can be surgically modified for visual tokens without retraining, and benchmark scores on the modified models remain meaningful.
    The entire framework is training-free; the paper assumes no distribution shift from the intervention invalidates the evaluation.
  • domain assumption The compact validation set used for layer ranking is representative of the eight test benchmarks.
    The authors state in the Limitations that the limited-scale validation set may fail to capture true model behavior.
  • domain assumption The attention reduction and FFN reduction can be composed additively without harmful interactions.
    Table 1 reports simultaneous attention and FFN reduction (LRA, LRF), implicitly assuming the two interventions are independent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs." pith.science (2026). https://pith.science/paper/GNAL62S3

@misc{pith2026250119036,
  author       = {Pith},
  title        = {Pith review of: RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GNAL62S3}},
  note         = {Machine review of arXiv:2501.19036}
}
read the original abstract

Current Multimodal Large Language Model (MLLM) architectures face a critical tradeoff between performance and efficiency: decoder-only architectures achieve higher performance but lower efficiency, while cross-attention-based architectures offer greater efficiency but lower performance. The key distinction lies in how visual tokens are processed. Decoder-only architectures apply self-attention and FFN operations on visual tokens, while cross-attention architectures skip these computations. To investigate whether redundancy exists in this computationally expensive process, we propose a training-free framework for analyzing trained MLLMs. It consists of Probe-Activated Dynamic FFN and Hollow Attention, which enable adjustable reductions in computations for visual tokens, as well as a Layer Ranking Algorithm that prioritizes layers for these reductions. Extensive experiments demonstrate substantial, structured, and clustered redundancy unique to decoder-only MLLMs, offering valuable insights for future MLLM architecture design. Furthermore, by leveraging our reduction framework as a training-free inference acceleration approach, we achieve performance comparable to or better than state-of-the-art methods while remaining compatible with them. Code will be publicly available at https://github.com/L-Hugh/RedundancyLens.

Figures

Figures reproduced from arXiv: 2501.19036 by the authors.

Figure 1
Figure 1. Comparison between decoder-only and cross [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed computational reductions for visual tokens: (a) Probe-Activated Dynamic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Impact of applying self-attention or FFN reductions across various layer proportions. The x-axis represents [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison of the reductions [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The reduction pattern for Qwen2-VL-7B. No [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies on (a) the attention range in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison of different layer [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

  2. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

Reference graph

Works this paper leans on

57 extracted references · 22 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko aj Bi\' n...

    work page 2022

  4. [4]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL : A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  6. [6]

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. Honeybee: Locality-enhanced projector for multimodal LLM . arXiv preprint arXiv:2312.06742

    arXiv 2024

  7. [7]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. MiniGPT-v2 : large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

    arXiv 2023

  8. [8]

    Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, Changyi Liu, Dewen Fan, Huihui Xiao, Jiahong Wu, Fan Yang, Size Li, and Di Zhang. 2024 a . EVLM : An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177

    arXiv 2024

Show all 57 references
  1. [9]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2025. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, pages 19--35

    2025

  2. [10]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024 b . Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330

    2024 arXiv

  3. [11]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang,...

    2024 arXiv

  4. [12]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024 d . InternVL : Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In...

    2024

  5. [13]

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. NVLM : Open frontier-class multimodal LLMs . arXiv preprint arXiv:2409.11402

    2024 arXiv

  6. [14]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP : towards general-purpose vision-language models with instruction tuning. In NeurIPS

    2023

  7. [15]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, a...

    2024 arXiv

  8. [16]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. 2024. VLMEvalKit : An open-source toolkit for evaluating large multi-modality models. In ACM MM, page 11198–11201

    2024

  9. [17]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. MME : A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394

    2024 arXiv

  10. [18]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904--6913

    2017

  11. [19]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. HallusionBench : An advanced diagnostic suite for entangled language hallucination and visual illusion in large...

    2024

  12. [20]

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. 2025. LLaVA-UHD : An LMM perceiving any aspect ratio and high-resolution images. In ECCV, pages 390--406

    2025

  13. [21]

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. 2024. ZipVL : Efficient large vision-language models with dynamic token sparsification and KV cache compression. arXiv preprint arXiv:2410.08584

    2024 arXiv

  14. [22]

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. CogVLM2 : Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500

    2024 arXiv

  15. [23]

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. 2024. mPLUG-DocOwl 1.5 : Unified structure learning for OCR -free document understanding. arXiv preprint arXiv:2403.12895

    2024 arXiv

  16. [24]

    Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. 2024. Mini-Monkey : Alleviating the semantic sawtooth effect for lightweight MLLMs via complementary image pyramid. arXiv preprint arXiv:2408.02034

    2024 arXiv

  17. [25]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . LLaVA-OneVision : Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  18. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP -2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, volume 202, pages 19730--19742

    2023

  19. [27]

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024 b . Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, pages 26763--26773

    2024

  20. [28]

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2024. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. arXiv preprint arXiv:2405.05803

    2024 arXiv

  21. [29]

    Haogeng Liu, Quanzeng You, Xiaotian Han, Yongfei Liu, Huaibo Huang, Ran He, and Hongxia Yang. 2024 a . Visual anchors are strong information aggregators for multimodal large language model. arXiv preprint arXiv:2405.17815

    2024 arXiv

  22. [30]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS, volume 36, pages 34892--34916

    2023

  23. [31]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 b . MMBench : Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281

    2024 arXiv

  24. [32]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. 2024 c . OCRBench : On the hidden mystery of OCR in large multimodal models. arXiv preprint arXiv:2305.07895

    2024 arXiv

  25. [33]

    Llama-Team. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  26. [34]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. DeepSeek-VL : Towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525

    2024 arXiv

  27. [35]

    Feipeng Ma, Yizhou Zhou, Hebei Li, Zilong He, Siying Wu, Fengyun Rao, Yueyi Zhang, and Xiaoyan Sun. 2024. EE-MLLM : A data-efficient and compute-efficient multimodal large language model. arXiv preprint arXiv:2408.11795

    2024 arXiv

  28. [36]

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. C hart QA : A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, pages 2263--2279

    2022

  29. [37]

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. 2022. InfographicVQA . In WACV, pages 2582--2591

    2022

  30. [38]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. DocVQA : A dataset for VQA on document images. In WACV, pages 2199--2208

    2021

  31. [39]

    OpenAI. 2024. GPT-4 technical report. arXiv preprint arXiv:2303.08774

    2024 arXiv

  32. [40]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. LLaVA-PruMerge : Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388

    2024

  33. [41]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2016. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR

    2016

  34. [42]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In CVPR, pages 8309--8318

    2019

  35. [43]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA : Open and efficient foundation l...

    2023 arXiv

  36. [44]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, volume 30

    2017

  37. [45]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . Qwen2-VL : Enhancing vision-language ...

    2024 arXiv

  38. [46]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2024 b . CogVLM : Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079

    2024 arXiv

  39. [47]

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2025. Vary: Scaling up the vision vocabulary for large vision-language model. In ECCV, pages 408--424

    2025

  40. [48]

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. 2024. PyramidDrop : Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247

    2024 arXiv

  41. [49]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2...

    2024 arXiv

  42. [50]

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, and Fei Huang. 2023. UR eader: Universal OCR -free visually-situated language understanding with multimodal large language model. In Find...

    2023

  43. [51]

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024 a . mPLUG-Owl3 : Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840

    2024 arXiv

  44. [52]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024 b . mPLUG-Owl : Modularization empowers large languag...

    2024 arXiv

  45. [53]

    Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng. 2024. TextHawk : Exploring efficient fine-grained perception of multimodal large language models. arXiv preprint arXiv:2404.09204

    2024 arXiv

  46. [54]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird : Transformers for longer sequences. In NeurIPS, volume 33, pages 17283--17297

    2020

  47. [55]

    Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin. 2024. DocKylin : A large multimodal model for visual document understanding with efficient visual slimming. arXiv preprint arXiv:2406.19101

    2024 arXiv

  48. [56]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  49. [57]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024. Mini GPT -4: Enhancing vision-language understanding with advanced large language models. In ICLR

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.