Pith. sign in

REVIEW 5 major objections 6 minor 20 cited by

Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that text-visual attention in large vision-language models is a misleading pruning signal and that a training-free method using the visual encoder's own attention plus token diversity keeps 91.5% of LLaVA-1.5-7B's average…

desk verdict VisPruner is a genuinely useful, training-free pruning method backed by a solid analysis of why text-attention pruning fails; the weaknesses are missing robustness details and benchmark coverage, not the core idea. read the letter →

arxiv 2412.01818 v2 pith:Z2QMCJLC submitted 2024-12-02 cs.CV cs.AI

Qizhe Zhang , Aosong Cheng , Ming Lu , Renrui Zhang , Zhiyong Zhuo , Jiajun Cao , Shaobo Guo , Qi She
show 1 more author
Shanghang Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords tokenpruningvision-languagemodelsvisualattentionCLStraining-freeinferenceaccelerationshiftdiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the attention scores text tokens pay to image tokens inside a vision-language model are the wrong signal to use when pruning visual tokens. It shows two problems with that signal: a positional "shift" that biases selection toward lower image patches, and a "dispersion" that spreads attention too evenly to separate important tokens from redundant ones. The proposed alternative, VisPruner, keeps a small set of high-attention tokens from the image encoder and adds back a similarity-diverse set of the remaining tokens, all before the language model runs. The reported payoff is that LLaVA-1.5-7B retains 91.5% of its average score across ten benchmarks with only 32 of 576 visual tokens, and the method beats text-attention and token-merging baselines at every tested token budget. A reader should care because visual tokens dominate inference cost, and this suggests that a cheap visual-saliency signal plus diversity can outperform question-sensitive attention for pruning.

What carries the argument

The machinery is a two-step, training-free selection performed before the language model. First, the [CLS] attention row of the visual encoder, defined as the attention from the global class token to each image patch, is averaged over heads and thresholded to pick a small set of important tokens that capture concentrated foreground saliency. Second, on the remaining tokens the method iteratively removes tokens most similar to others under cosine similarity, leaving a diverse set that covers background areas; the important and diverse sets are then re-sorted into original image order and projected into the language model. This design lets pruning happen before the LLM, which keeps the method compatible with attention optimization techniques and avoids the positional bias found in in-LLM attention.

What would settle it

Run the identical 32-token pruning comparison on a benchmark built around small background details, such as a detailed captioning or spatial-relation benchmark: if VisPruner's accuracy falls toward random or drops below FastV at the same token budget, the claim that visual-encoder attention plus diversity preserves needed information is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that text-visual attention inside the language model is not a trustworthy basis for visual token pruning, and that a better signal comes from the visual encoder itself. It documents an attention shift caused by rotary position embeddings, where later text tokens preferentially attend to lower image patches from the first layer onward, and an attention dispersion that leaves text-visual attention too uniform to identify important tokens. VisPruner replaces that signal with the [CLS] attention of the visual encoder, which is concentrated on foreground content, and then adds a similarity-based diverse set of the remaining tokens so background information survives. On LLaVA-1.5-7B, retaining 32 of 576 visual tokens is reported to preserve 91.5% of average performance over ten benchmarks, and the method is reported to outperform FastV, SparseVLM, and VisionZip at 128, 64, and 32 retained tokens.

Load-bearing premise

The load-bearing premise is that the image encoder's [CLS] attention marks the tokens the language model needs and that similarity-based diversity covers what it misses; if important information lives in small, background, or question-specific tokens, high-ratio pruning will collapse.

Editorial extensions

If this is right

  • At a 94.4% visual-token reduction, LLaVA-1.5-7B retains 91.5% of its average score across the ten evaluated benchmarks without any fine-tuning.
  • Pruning before the language model cuts FLOPs and latency more than in-LLM pruning and works with FlashAttention, giving LLaVA-1.5-7B a reported 91% FLOP reduction and 75% latency reduction.
  • The method transfers beyond LLaVA-1.5, preserving more relative performance than FastV on LLaVA-NeXT, Video-LLaVA, Qwen-VL, InternVL, and CogVLM at matching token budgets.
  • At high reduction ratios, question-independent visual saliency plus diversity outperforms text-visual attention and token merging, suggesting that merging similar tokens can inject noise relative to simply dropping them.
  • The same visual-cue approach extends to longer visual sequences, including high-resolution images and video inputs, where redundancy is higher and relative gains are larger.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmarks are mostly question-answering and hallucination tasks, the paper does not establish that the method survives detailed captioning or tasks where answers depend on small background objects; testing those would settle the generality.
  • The diversity term is question-blind, so a natural extension would condition the retained background tokens on the text prompt, a variant the paper does not explore.
  • If visual-encoder attention is indeed the better pruning signal, then improving or aligning that attention with the language model should directly improve pruning quality, pointing toward co-design of encoders and pruners.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies token pruning in vision-language models and argues that text-visual attention inside the language model is a poor pruning signal, because of positional bias (attention shift) and flat attention concentration (attention dispersion). The authors propose VisPruner, a training-free, plug-and-play method that selects a small set of important visual tokens using the visual encoder's [CLS] attention and then supplements them with diverse tokens chosen by a cosine-similarity based duplicate-removal procedure. The method is evaluated on LLaVA-1.5, LLaVA-NeXT, Video-LLaVA, Qwen-VL, InternVL, and CogVLM across image VQA, hallucination, and video QA benchmarks, and compared with ToMe, FastV, SparseVLM, LLaVA-PruMerge, and VisionZip. The main empirical claim is that at 32 retained tokens on LLaVA-1.5-7B, VisPruner retains 91.5% of full-model average performance while pruning 94.4% of visual tokens, consistently outperforming text-visual attention baselines and the visual merging baseline VisionZip. The paper also reports FLOPs and latency reductions.

Significance. If the reported results are firm, the paper makes a useful and practical contribution: it challenges the dominant text-visual attention paradigm for VLM token pruning, provides a simple visual-cue based alternative that requires no training, and demonstrates the approach across several model families. The analysis of attention shift and dispersion is a concrete, testable observation that other researchers can build on. The paper also gives credit to concurrent work sharing similar conclusions and provides code, which supports reproducibility. However, the paper's broader claim that visual cues are generally more reliable than text-visual attention is only tested on a specific set of benchmarks, and several load-bearing numerical claims need correction or additional statistical support before the conclusions can be taken at face value.

major comments (5)
  1. [Abstract, Section 1, Section 5.7, Supplementary Table 6] The abstract and introduction state that VisPruner 'can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%' and that pruning 94.4% of visual tokens in LLaVA-1.5-7B reduces inference FLOPs by 'more than 95%'. Supplementary Table 6 for LLaVA-1.5-7B at 95% token reduction reports FLOPs of 0.76 T vs 8.02 T (a 90.5% reduction) and CUDA time of 77.15 ms vs 107.26 ms (a 28% reduction). The 91% FLOP and 75% latency figures match LLaVA-NeXT-7B in Table 8, not LLaVA-1.5-7B. The efficiency claims must be corrected and attributed to the appropriate model.
  2. [Section 5.2, Table 1] No standard deviations or repeated-run statistics are reported, and several of the claimed margins over VisionZip are small. For example, at 128 retained tokens, SQA-IMG differs by 0.2 points (69.1 vs 68.9), TextVQA by 0.2 points (57.0 vs 56.8), MMBench-CN by 0.6 points (57.3 vs 56.7), and MM-Vet by 1.1 points (33.7 vs 32.6). Without variance estimates, the statement that VisPruner consistently outperforms VisionZip is not statistically supported.
  3. [Section 4.1, Section 4.2, Section 5.6, Fig. 5] The hyperparameter r in Eq. (4) determines the split between attention-based important tokens and similarity-based diverse tokens, but its value at each operating point is never reported and no sensitivity analysis is provided. The ablation in Fig. 5 only compares the 'Important' variant (which appears to use the entire budget for attention-selected tokens) with the full VisPruner; it does not isolate the effect of r on the final performance. Since r is a free parameter of the method, the main results are not fully reproducible without reporting its value and stability.
  4. [Section 5.1, Section 5.2, Section 4.2] The ten image benchmarks are dominated by visual question answering and object-presence/hallucination tasks. None of them requires detailed captioning or fine-grained spatial layout reasoning. VisPruner's diverse-token component is question-independent and based on cosine similarity in CLIP feature space, so the claim that visual-cue pruning is generally superior to text-visual attention is not established for tasks in which small-detail or background tokens are task-critical. The authors should either add such benchmarks (e.g., NoCaps or Flickr30k captioning, spatial reasoning subsets) or explicitly scope the claims to the tested task distribution.
  5. [Algorithm 1, Section 4.2] The pseudocode is difficult to reconcile with the text and with Eq. (4). The variable r is used both as the important-token ratio in Eq. (4) and as the per-iteration removal count in Algorithm 1. The input 'normalized' is not defined, the update r = min(r, remaining_idx.shape[0] - n) is not explained, and the termination condition is implicit. The algorithm should be rewritten with distinct variable names, a clear initialization and termination condition, and a stated guarantee that the output size exactly equals R(1-r) diverse tokens.
minor comments (6)
  1. [Abstract] The sentence 'Based on the analysis, We propose VisPruner' has an incorrect capitalization; 'We' should be 'we'.
  2. [Supplementary Table 6] The upper bound row for LLaVA-1.5-13B contains extra duplicated values ('85.99 107.26 85.88') that do not match the column layout; the table should be reformatted.
  3. [Fig. 5 caption] The caption does not define what 'Important' means in terms of r or the token budget; please specify the hyperparameter setting used for the ablation.
  4. [Section 5.5] For Qwen-VL, InternVL, and CogVLM, VisPruner is only compared with FastV; the absence of SparseVLM and VisionZip on these architectures should be acknowledged, since the main tables compare only with FastV.
  5. [Supplementary Section 8.1] There is a typo: 'adpot' should be 'adopt'.
  6. [References] The page list for reference [9] is out of order ('1, 3, 5, 6, 7, 8, 2'); please clean up the reference metadata.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VisPruner's pruning signal is an external visual-encoder attention map and its claims are tested on standard benchmarks, not derived from the benchmarks.

full rationale

The paper's derivation uses CLIP [CLS] attention (Eqs. 2, 4, 5) and similarity-based diversity (Algorithm 1) to select visual tokens before the language model; these selection rules depend only on the input image, the computation budget R, and the hyperparameter r. The evaluation in Tables 1-4 compares the resulting token set against standard benchmarks and external baselines (FastV, SparseVLM, VisionZip, etc.), so the reported relative accuracies are not determined by construction. The Section 3 analysis of attention shift and dispersion is an empirical study, and Fig. 3c's comparison of pruning signals is an experiment rather than a definition of the method. No fitted parameter is renamed as a prediction, no load-bearing self-citation appears, and no uniqueness theorem is imported from the authors' prior work. The small hyperparameter r is a design choice and is not fitted to the ten-benchmark average in a way that would force the headline result. The paper is therefore self-contained against external benchmarks and shows no circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method is training-free and uses already-available attention signals, so it introduces no new physical entities and no fitted model weights. The only real free parameter is the budget-split r, and the paper's analysis relies on two domain assumptions about what [CLS] attention and cosine-similarity diversity each capture.

free parameters (1)
  • r = not specified numerically, described as a hyperparameter
    In Eq. (4), tau is the (1-R*r) percentile of a_v, so r decides how the retained budget is split between attention-important tokens and diversity tokens. The paper does not report the value or a sensitivity study, so the working value is a free parameter.
assumptions (3)
  • domain assumption Rotary position embedding long-term decay is the cause of the observed text-visual attention shift.
    Section 3.2 attributes the text-visual attention concentration on later visual token indices to the long-term decay property cited from prior work [47]. The paper's own ablation shows that removing the decay in attention (equi last attention) reduces but does not eliminate the dispersion, so this attribution is partially causal and has some independent support in the literature.
  • domain assumption The average of the first row of the visual encoder's attention matrix is a meaningful importance signal for the language model's visual needs.
    Section 4.1 and Fig. 3 show that [CLS] attention has a concentrated distribution, and the paper assumes this concentration corresponds to question-relevant content. This is treated as an empirical premise, not proven by the paper.
  • domain assumption Question-independent diversity sampling covers the visual information that high-attention tokens miss.
    Section 4.2 assumes that removing tokens with high cosine similarity preserves coverage of the input image. If background information is important for a specific question, this heuristic may fail. It is a plausible domain assumption, but it is not justified by a dedicated analysis beyond the ablation in Fig. 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs." pith.science (2026). https://pith.science/paper/Z2QMCJLC

@misc{pith2026241201818,
  author       = {Pith},
  title        = {Pith review of: Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2QMCJLC}},
  note         = {Machine review of arXiv:2412.01818}
}
read the original abstract

Large vision-language models (LVLMs) generally contain significantly more visual tokens than their textual counterparts, resulting in a considerable computational burden. Recent efforts have been made to tackle this issue by pruning visual tokens early within the language model. Most existing works use attention scores between text and visual tokens to assess the importance of visual tokens. However, in this study, we first analyze the text-visual attention in the language model and find that this score is not an ideal indicator for token pruning. Based on the analysis, We propose VisPruner, a plug-and-play method that utilizes visual cues for more effective token pruning in LVLMs. Specifically, we first use visual attention to select a limited number of significant tokens. Then, we remove duplicate tokens from the remaining ones based on their similarity. By retaining diverse tokens alongside the initially selected important tokens, we maximally preserve the visual information of the input image. Experimental results demonstrate that our VisPruner sustains strong performance across various VLM architectures and reduction ratios, significantly outperforming existing methods based on text-visual attention. Notably, without any training, VisPruner can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%, while maintaining comparable performance. Our code is available at https://github.com/Theia-4869/VisPruner.

Figures

Figures reproduced from arXiv: 2412.01818 by the authors.

Figure 1
Figure 1. The illustration of different pruning methods. Correct answer parts are shown in blue, while hallucinations due to prun￾ing are shown in red. Text-visual attention methods like FastV often preserve the lower parts of input images, which can lead to the loss of crucial visual information during early pruning (e.g. the iron in the man’s hand). Random pruning removes positional bias but still fails to preserve importan… view at source ↗
Figure 2
Figure 2. Analysis of text-visual attention shift. (a) Selection frequency and received attention of visual tokens. There is a clear positive correlation between selection frequency and the attention received, which is also accompanied by a positional bias. (b) Proportion of visual tokens with top 25% text-visual attention across each quartile position. In the shallower layers of the language model, attention is notably conce… view at source ↗
Figure 3
Figure 3. a. It is observable that [CLS] attention is more con￾centrated, with the top 20% visual tokens absorbing about 80% attention. In contrast, text-visual attention, whether or not position embedding is removed, is spread across more visual tokens. We also illustrate the density distribution of different attentions in Fig. 3b. Different from the extreme peak in [CLS] attention, the density of last attention both be￾fore… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of VisPruner. We begin by selecting a small portion of important tokens with rich information, based on the [CLS] attention from the visual encoder. For the remaining tokens, we progressively remove duplicates based on similarity, ultimately retaining anot…
Figure 5
Figure 5. Figure 5: Ablation study of the core components. Random de￾notes randomly selecting tokens, Important refers to selecting only tokens with high attention scores, and VisPruner represents the fi￾nal version of our method, which also includes a diverse set of tokens with low redun…
Figure 6
Figure 6. Figure 6: Distribution of visual attention over token positions in CLIP. From left to right: attention distribution of the [ [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Distribution of visual-text attention over visual token positions in LLaMA. The top rows display the average attention distribution [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Distribution of visual-text attention over visual token positions in LLaMA. The top rows display the attention distribution from [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Visualizations of attention maps from the [CLS] token in the visual encoder and the last token in the language model. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    SFPruner performs redundancy-aware visual token pruning in a single non-iterative pass using semantics-guided ridge leverage scores and directional masking, cutting selection latency from 112.4 ms to 2.5 ms at 512 tok...

  2. EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

    cs.AR 2026-08 conditional novelty 6.0 of 10

    A co-designed edge accelerator that avoids loading redundant expert weights when speculative decoding meets mixture-of-experts, cutting latency by up to 56.3% and energy by up to 44.1% in simulation.

  3. When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A break-even study of Qwen2.5-VL on two GPUs finds that a post-vision pruner and a pre-vision resolution rule both cut end-to-end latency, but pre-vision routing wins by skipping vision encoding, not by removing the m...

  4. Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

    cs.AI 2026-08 conditional novelty 6.0 of 10

    MAP predicts sample-specific middle-layer attention with a lightweight model and prunes visual tokens before the LLM, keeping 97.5% of performance at 5.56% of tokens.

  5. Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CaRe retains about 96% of full-token VLM benchmark performance while pruning about 94% of visual tokens, by calibrating retained anchors with confidence-gated signals from pruned tokens.

  6. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

  7. Training-free Token Reduction for Vision Mamba

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MTR uses Mamba's timescale parameter Δ as a token importance score to merge unimportant tokens, giving training-free inference speedups with small accuracy loss.

  8. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  9. Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Pruning visual tokens degrades visual grounding because position IDs become misaligned; preserving the original position IDs recovers most of the lost accuracy with no extra cost.

  10. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  11. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  12. AdaFV: Rethinking of Visual-Language alignment for VLM acceleration

    cs.CV 2025-01 conditional novelty 6.0 of 10

    AdaFV prunes visual tokens before the LLM by adaptively mixing visual saliency and text-to-image similarity, achieving state-of-the-art training-free VLM acceleration at high reduction rates.

  13. Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Tracking cross-layer attention momentum lets training-free pruning reactivate late-rising visual tokens, cutting final-layer tokens to ~23 at 77.8% prune while keeping near-baseline multimodal accuracy.

  14. SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Using the separator token as a position-free query, SepPrune prunes 80% of vision tokens while retaining 96.3% of Qwen2.5-VL-7B's original accuracy.

  15. LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression

    cs.CV 2025-08 conditional novelty 5.0 of 10

    LightVLM accelerates vision-language model inference with pyramid token merging and KV cache compression, preserving about 98% accuracy with only 3% of image tokens.

  16. GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A training-free token pruning method that combines cosine-similarity saliency with greedy redundancy removal to preserve accuracy at high compression ratios.

  17. Do Concept Replacement Techniques Really Erase Unacceptable Concepts?

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Weight-editing concept erasure methods fail to prevent reconstruction of unacceptable images in an image-to-image pipeline; a face-targeted editing method, AntiMirror, replaces celebrity likenesses with better fidelit...

  18. FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.

  19. AdaTP: Attention-Debiased Token Pruning for Video Large Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    AdaTP prunes visual tokens in video LLMs by debiasing attention scores, reducing FLOPs to about a quarter of the vanilla model at matching benchmark accuracy.

  20. A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    The paper claims 92.6% dynamic visual-token pruning with retained (or 110% of) baseline VQA performance, but the full text supplied is a different paper, so the claim is unverifiable here.

Reference graph

Works this paper leans on

70 extracted references · 22 canonical work pages · cited by 20 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1, 2

    arXiv 2023

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 2, 6, 7

    arXiv 2023

  4. [4]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 6, 7

    arXiv 2022

  5. [5]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430, 2024. 3

    arXiv 2024

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 5

    2021

  7. [7]

    Collecting highly paral- lel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. InProceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies , pages 190–200, 2011. 2

    work page 2011

  8. [8]

    Llavolta: Efficient multi-modal models via stage-wise visual context compression

    Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Llavolta: Efficient multi-modal models via stage-wise visual context compression. arXiv preprint arXiv:2406.20092, 2024. 3

    arXiv 2024

Show all 70 references
  1. [9]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 1, 3, 5, 6, 7, 8, 2

    2024 arXiv

  2. [10]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF conference on computer...

    2024

  3. [11]

    Funnel-transformer: Filtering out sequential redundancy for efficient language processing

    Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. Advances in neural informa- tion processing systems, 33:4271–4282, 2020. 2

    2020

  4. [12]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. 2, 8

    2022

  5. [13]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Pi- otr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. 5

    2023 arXiv

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    2010 arXiv

  7. [15]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 1, 3, 6

    2024

  8. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  9. [17]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  10. [18]

    On the token distance modeling ability of higher rope attention dimension

    Xiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. On the token distance modeling ability of higher rope attention dimension. arXiv preprint arXiv:2410.08703, 2024. 2, 3

    2024 arXiv

  11. [19]

    Matryoshka query trans- former for large vision-language models

    Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query trans- former for large vision-language models. arXiv preprint arXiv:2405.19315, 2024. 3

    2024 arXiv

  12. [20]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  13. [21]

    Pyramid-bert: Reducing complexity via successive core-set based token selection

    Xin Huang, Ashish Khetan, Rene Bidart, and Zohar Karnin. Pyramid-bert: Reducing complexity via successive core-set based token selection. arXiv preprint arXiv:2203.14380 ,

    arXiv

  14. [22]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 1, 3, 6

    2019

  15. [23]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2758–2766, 2017. 6, 2

    2017

  16. [24]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 1, 2

    arXiv

  17. [25]

    Openim- ages: A public dataset for large-scale multi-label and multi-class image classification

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper 9 Uijlings, Stefan Popov, Andreas Veit, et al. Openim- ages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available fr...

    2017

  18. [26]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  19. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  20. [28]

    Tokenpacker: Effi- cient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Effi- cient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024. 3

    2024 arXiv

  21. [29]

    Tgif: A new dataset and benchmark on animated gif description

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016. 2

    2016

  22. [30]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 3, 6, 1

    2023 arXiv

  23. [31]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2024. 3

    2024

  24. [32]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 2, 6, 7

    2023 arXiv

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  26. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 3, 6

    2024

  27. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1, 2, 6

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2

    2024

  29. [37]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 1, 6

    2025

  30. [38]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  31. [39]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 2, 6

    2024 arXiv

  32. [40]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 7, 2

    2023 arXiv

  33. [41]

    Efficient transformers with dynamic to- ken pooling

    Piotr Nawrot, Jan Chorowski, Adrian Ła ´ncucki, and Edoardo M Ponti. Efficient transformers with dynamic to- ken pooling. arXiv preprint arXiv:2211.09761, 2022. 2

    2022 arXiv

  34. [42]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems, 35:...

    2022

  35. [43]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  36. [44]

    Compressive transform- ers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayaku- mar, and Timothy P Lillicrap. Compressive transform- ers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019. 2

    1911 arXiv

  37. [45]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  38. [46]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 1, 3, 6

    2019

  39. [47]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  40. [48]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 2

    2023 arXiv

  41. [49]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 2 10

    2024

  42. [50]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  43. [51]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1, 3

    2023 arXiv

  44. [52]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  45. [53]

    [cls] token tells everything needed for training-free efficient mllms

    Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [cls] token tells everything needed for training-free efficient mllms. arXiv preprint arXiv:2412.05819, 2024. 3, 8

    2024 arXiv

  46. [54]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiX- uan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2025. 1, 6, 7

    2025

  47. [55]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy re- duction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy re- duction. arXiv preprint arXiv:2410.17247, 2024. 2, 3, 5

    arXiv

  48. [56]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017. 6, 2

    2017

  49. [57]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 2

    2016

  50. [58]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 2, 6

    2024 arXiv

  51. [59]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024. 3, 6, 7, 8

    2024

  52. [60]

    Fit and prune: Fast and training-free visual token pruning for multi-modal large language models

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. arXiv preprint arXiv:2409.10197, 2024. 2, 3, 5

    2024 arXiv

  53. [61]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 1, 6, 2

    2023 arXiv

  54. [62]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 6

    2019

  55. [63]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 5

    2023

  56. [64]

    Llava-mini: Efficient image and video large mul- timodal models with one vision token

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large mul- timodal models with one vision token. arXiv preprint arXiv:2501.03895, 2025. 3

    2025 arXiv

  57. [65]

    Sparsevlm: Vi- sual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Vi- sual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024. 2, 3, 5, 6, 7

    arXiv

  58. [66]

    A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yib- ing Song, Kai Wang, Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms. arXiv preprint arXiv:2412.03324 ,

    arXiv

  59. [68]

    Datasets We evaluate our method on a total of 13 widely used bench- marks, including 10 image benchmarks and 3 video bench- marks

    Details of experimental setup 7.1. Datasets We evaluate our method on a total of 13 widely used bench- marks, including 10 image benchmarks and 3 video bench- marks. Each task is described as follows. 7.1.1. Image benchmarks We conduct experiments on 10 image benchmarks used i...

  60. [69]

    Attention distribution We first present the distribution of visual attention in CLIP

    Detailed Analysis of Attention in VLMs 8.1. Attention distribution We first present the distribution of visual attention in CLIP. As shown in Fig. 6, the left two subplots show the visual attention in the penultimate layer of CLIP. The visual to- kens used in LLaV A-1.5 also c...

  61. [70]

    Efficiency Analysis with FlashAttention In Tab. 6, Tab. 7 and Tab. 8, we compare the computational efficiency between FastV and our VisPruner under LLaV A- 1.5-7B, LLaV A-1.5-13B, and LLaV A-NeXT-7B, respec- tively. Unlike FastV , which prune visual token within the LLM, VisPr...

  62. [2024]

    3 11 Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs Supplementary Material

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.