Pith. sign in

REVIEW 4 major objections 7 minor 61 references

Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When token pruning discards most of an image's visual tokens, a question-guided caption generated from the discarded tokens recovers most of the accuracy the pruned model loses.

desk verdict A plausible caption-compensation mechanism for pruned LVLMs, undermined by a missing control on whether discarded tokens are actually the source of the gains. read the letter →

arxiv 2508.01236 v1 pith:HS7KRTGQ submitted 2025-08-02 cs.CV

Mingyu Fu , Wei Suo , Ji Ma , Lin Yuanbo Wu , Peng Wang , Yanning Zhang This is my paper
classification cs.CV
keywords LargeVisionLanguageModelsVisualTokenPruningInformationLossAdaptiveContentCompensationQuestion-guidedCaptioningDirectPreferenceOptimizationSelf-supervisedLearningImage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual token pruning makes large vision-language models cheaper by dropping redundant image tokens, but past roughly 90% pruning the remaining tokens carry too little information and accuracy degrades sharply. This paper argues that the information is not truly gone: the discarded tokens still contain question-relevant content, and a lightweight caption model can convert them into a short caption. ACCM does this by generating several candidate captions from the discarded tokens together with the user question, selecting the most contextually appropriate one, and giving it to the frozen LVLM along with the retained tokens. On seven benchmarks the authors report that this compensation recovers most of the accuracy lost to pruning, beating the previous best pruning method by 20.6% while using 6.5% fewer FLOPs at around 97% pruning.

What carries the argument

The machinery has two moving parts. The first is the discarded visual token set $V_l$ plus a lightweight caption model $\mathcal{C}$ that shares the LVLM's CLIP image encoder, so the tokens produced during pruning are reused rather than re-encoded; $\mathcal{C}$ converts $V_l$ and the question into a set $M$ of beam-searched captions. The second is a selector $S$ that encodes each question-caption pair with a LongCLIP text encoder, classifies the pair embeddings with a small transformer, and returns the best caption via an argmax. Because the selection step is non-differentiable, both modules are trained as a preference optimization problem with DPO, where KL divergence to the unpruned LVLM's output scores each candidate caption and creates positive and negative examples.

What would settle it

Run ACCM on images where the correct answer refers only to objects in the retained tokens; if performance still rises when the caption is generated from the discarded tokens, the gain comes from the caption model's prior knowledge rather than recovered visual content, refuting the mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that visual information removed by aggressive token pruning can be re-supplied as text. After pruning divides the visual sequence into retained tokens $V_r$ and discarded tokens $V_l$, ACCM generates a caption set $M=\mathcal{C}(\mathrm{concat}[Q;V_l])$ from the discarded tokens under the guidance of the question $Q$, and a selector $S$ picks the contextually appropriate caption $m_s$, so the LVLM generates its answer from $(V_r, m_s, Q)$. The caption model and the selector are trained jointly with Direct Preference Optimization, using preference pairs built by comparing KL divergence between the original unpruned output and outputs obtained with each candidate caption; this requires no human or automated labels. The reported result is consistent improvement over FastV, VTW, LLaVolta, SparseVLM, and PyramidDrop across MME, MMBench, POPE, MMVP, SEED-Bench, GQA, and Flickr30k at both 93.5% and around 97% pruning, with lower FLOPs in most settings.

Load-bearing premise

The discarded image tokens still contain enough question-relevant information that a lightweight caption model can summarize them into a caption restoring what the kept tokens lost; if the pruning rule already throws away uninformative tokens, there is nothing left to recover.

Editorial extensions

If this is right

  • At around 97% pruning, ACCM reports average absolute gains of 7.8 to 15.4 points over PyramidDrop depending on the backbone, making extreme pruning rates usable instead of collapse-prone.
  • Because the LVLM weights stay frozen, the trained caption model and selector transfer between LLaVA-1.5-7B and LLaVA-NeXT-7B with roughly equal results, so the compensation modules behave like plug-in components for other LVLMs.
  • Question guidance matters: adding the user question to the caption model improves MME by 30.0 points and POPE by 2.9 points over a generic caption, and the selector adds further gains, confirming the paper's two-stage design.
  • Swapping the caption model for UniversalCap gives nearly the same scores as ClipCap, which indicates the benefit comes from the compensation pipeline and its training signal, not from one specific captioning architecture.
  • Scaling the self-supervised training set from 20k to 60k samples raises MME, MMBench, and GQA scores steadily, so the method continues to improve with more unlabeled data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's premise would corrupt $V_l$ before captioning; if captions from shuffled discarded tokens perform as well as genuine ones, the caption model is relying on language priors rather than on visual content.
  • The KL-divergence preference signal rewards captions that make the pruned model imitate the unpruned model, which could entrench the unpruned model's hallucination or reasoning errors instead of correcting them; that risk is not examined in the paper.
  • The same compensation idea could apply to video or high-resolution LVLMs, where pruning removes a much larger share of tokens; a caption model summarizing discarded spatiotemporal tokens would be a natural extension.
  • TFLOPs are lower, but the caption model and selector add extra decoding steps; a wall-clock latency study on edge hardware would show whether the FLOPs advantage survives in practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Adaptive Content Compensation Method (ACCM), a plug-and-play module for large vision-language models under aggressive visual token pruning. ACCM uses a lightweight, question-conditioned caption model to convert discarded visual tokens V_l into one or more captions, and a selector chooses the most contextually appropriate caption from several candidates; the caption is then concatenated with the retained visual tokens and fed to the frozen LVLM. The caption model and selector are trained jointly with DPO, using the unpruned LVLM's output as a preference signal, without additional human labeling. Experiments on LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-NeXT-7B across seven benchmarks report substantial gains over prior pruning methods at high pruning rates (e.g., 20.6% average improvement over PyramidDrop with 6.5% fewer FLOPs).

Significance. If the central mechanism is substantiated, ACCM would be a practically useful and low-cost complement to existing token-pruning methods: it freezes the LVLM, trains only two small auxiliary modules, and appears to transfer across LVLMs and caption models. The DPO-based self-distillation formulation is a clever way to avoid manual preference labels, and the cross-model transfer experiment in Table 5 and caption-model substitution in Table 4 are useful robustness checks. However, the headline quantitative claims currently rest on incomparable operating points (Tables 1 and 2), an unstated normalization in the 'Average' column, and an unverified assumption that discarded visual tokens V_l, rather than a question-conditioned language prior, are the source of the recovered information. These issues must be addressed before the paper's central claims can be accepted.

major comments (4)
  1. [Section 4.3, Tables 1 and 2] The comparisons are not at matched pruning rates. In Table 1, the header states around 93.5% pruning, but ACCM retains 36 of 576 tokens (6.3%) whereas the baselines retain 66-72 tokens (11.5%-12.5%), i.e., the baselines are pruned at roughly 87.5%-88.5% rather than 93.5%. Table 2 has the same problem: ACCM retains 18/576 tokens (3.1%) while FastV, VTW, LLaVolta, SparseVLM, and PDrop retain 8.0%-9.5%. The claimed average gains therefore compare different operating points. The authors should either match retained-token counts across methods or report performance as a function of retained tokens (as in Figure 4) for all methods at the same counts.
  2. [Section 3.2.1, Eq. (2); Table 3] The central claim that discarded visual tokens V_l are the source of recovered information is not isolated. The caption model C takes concat[Q; V_l] as input and is trained by DPO to make the pruned LVLM imitate the unpruned output O0 (Section 3.3.1), so C could in principle learn a question-conditioned language prior from the 40k instruction samples that produces benchmark-relevant text without using any visual content from V_l. Table 3 ablates question guidance and the selector but never removes or permutes V_l. I request a control in which V_l is replaced by (a) no visual input (C(concat[Q]) only) and (b) visual features from a different image or shuffled features. If gains persist under (a) or (b), the method is text-prior compensation rather than visual content compensation, and the abstract's claim of mitigating visual information loss is not supported.
  3. [Tables 1, 2, and 5] The 'Average' column cannot be reproduced from the reported per-benchmark scores because the normalization is not stated. For example, in the LLaVA-1.5-7B row of Table 1, the arithmetic mean of the seven listed scores is about 319, not 62.2. Since the paper's headline improvements (e.g., '20.6% over PyramidDrop') are drawn from this column, the normalization must be specified (e.g., each score divided by the unpruned model's score, then averaged) and any weighting justified.
  4. [Section 4.2, Tables 1 and 2] The reported TFLOPs do not state whether they include the caption model and selector. ACCM at inference must run the caption model with beam search to produce B=3 captions (Eq. 2) and the LongCLIP-based selector (Eqs. 3-4); these computations are additional to the pruned LVLM forward pass. If the TFLOPs in Tables 1-2 count only the LVLM, then the '6.5% fewer FLOPs' claim is incomplete and may be reversed once caption generation is included. The authors should report end-to-end FLOPs or explicitly state the scope of the efficiency numbers.
minor comments (7)
  1. [Section 3.2.1, footnote 2] The definition of pruning rate as 'the proportion of retained visual tokens to the total original visual tokens' is inconsistent with the usage in Tables 1-2, where 6.3% retained tokens is called 93.5% pruning; the definition should refer to discarded tokens or the text should say 'retention rate'.
  2. [Section 4.4] 'Tabel 3' should be 'Table 3'.
  3. [Introduction, second paragraph] 'degrades seriouly' should be 'seriously'.
  4. [Tables 1, 2, and 5] The note 'We normalize the test score of MME for illustrative purposes' appears only in the Figure 3 caption; the normalization used to compute the 'Average' columns in the tables is not described anywhere.
  5. [Section 2.1] 'LLaVA [19]' incorrectly points to the BLIP-2 reference; the LLaVA citation should be [23] or [25].
  6. [Section 3.2.1, Eq. (2)] Eq. (2) writes concat[Q; V_l] as the input to the caption model, but a ClipCap-style model requires projected visual features; specify whether V_l are the original CLIP patch embeddings, the projected features, or the pruned token embeddings and how they are concatenated with text.
  7. [Abstract and Section 3.3.1] The phrase 'without any human or automated labeling' is slightly overstated, because the preference dataset is built from 40k samples of the LLaVA-665k instruction dataset and from outputs of the unpruned LVLM; clarify that no new labels are collected but existing instruction data and model outputs are used.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: ACCM's gains are an empirical, externally benchmarked result; the sole self-citation is non-load-bearing.

full rationale

The central claim—that question-guided captions generated from discarded tokens improve pruned-LVLM benchmark performance—is not derived by definition. Eq. (2) defines M = C(concat[Q; V_l]) and Eq. (5) conditions generation on (V_r, m_s, Q), but no equation defines the target output Y in terms of itself, and no fitted quantity is later renamed as a prediction. The DPO training in Sec. 3.3.1 uses the unpruned LVLM's output O0 to construct preference labels; this is a self-distillation supervision signal, not a circular definition of the evaluated claim, and the paper validates against external benchmarks (MME, MMBench, POPE, MMVP, SEED, GQA, Flickr30k) with the LVLM weights frozen. The only self-citation, Ref. [40] (same research group), appears in a list of prior token-pruning methods and is not used to justify ACCM's mechanism or its uniqueness, so it is not load-bearing. The absence of a feature-shuffling control for V_l is a legitimate attribution/correctness concern, but it does not make the derivation circular; it concerns whether the observed gain comes from visual content versus a question-conditioned language prior.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

ACCM's central claim rests on the empirical assumption that pruned-away tokens can be converted into a useful text summary, and that the unpruned model itself provides trustworthy labels. The method has no closed-form derivation; its contribution is the training procedure and architecture, so the ledger is dominated by domain assumptions and a few hand-chosen hyperparameters rather than fitted constants.

free parameters (4)
  • beta (DPO temperature) = 1
    Set to 1 following [14,31] and used in Eqs. 6-7; it controls the strength of preference optimization but is not tuned or ablated in the paper.
  • number of caption candidates B at inference = 3
    Inference uses three beam-search captions (Section 4.2); no sensitivity analysis is reported for B.
  • training data size = 40k
    Randomly sampled from LLaVA-665k (Section 4.2); Figure 3 shows scores still rise from 20k to 60k, so the choice is an operating point, not a saturation-motivated value.
  • positive/negative preference split rule = not specified
    Section 3.3.1 divides captions into A+ and A- by comparing KL divergence to O0, but neither the threshold nor the pairing strategy is stated; this choice determines the DPO labels.
assumptions (4)
  • domain assumption Discarded visual tokens V_l preserve enough question-relevant visual information to generate a useful caption.
    Section 3.2.1 (Eq. 2) conditions caption generation solely on V_l and Q; the entire compensation mechanism fails if high pruning rates leave V_l information-free.
  • domain assumption The unpruned LVLM's output O0 is a reliable supervisor for preference construction.
    Section 3.3.1 uses KL divergence to O0 to label captions as positive or negative; if O0 is wrong or hallucinated, the trained modules learn to mimic incorrect answers.
  • domain assumption Reference-free DPO, following [14,31], is a valid optimization for both the caption generator and the selector.
    Eqs. 6-7 drop the reference model from vanilla DPO [36]; the paper cites [14,31] for this but provides no analysis of how this affects caption quality.
  • domain assumption Sharing the CLIP vision encoder between the LVLM and the caption model is sufficient for the caption model to interpret V_l.
    Section 3.2.1 reuses the LVLM's CLIP encoder in the caption model for efficiency; no experiments test whether a different encoder would change compensation quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models." pith.science (2026). https://pith.science/paper/HS7KRTGQ

@misc{pith2026250801236,
  author       = {Pith},
  title        = {Pith review of: Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HS7KRTGQ}},
  note         = {Machine review of arXiv:2508.01236}
}
read the original abstract

Despite the great success of Large Vision Language Models (LVLMs), their high computational cost severely limits their broad applications. The computational cost of LVLMs mainly stems from the visual sequence of the input, which consists of hundreds or even thousands of tokens. Although existing methods have made progress by removing redundant tokens, they suffer from severe performance degradation with high pruning rates due to the loss of visual information. In this paper, we propose an Adaptive Content Compensation Method (ACCM), which can effectively mitigate the visual information loss via an image caption. Specifically, ACCM comprises two key components: a lightweight caption model and a selector. Firstly the caption model generates question-related descriptions under the guidance of the user instruction. Then the selector further identifies a contextually appropriate caption from multiple candidates. Leveraging self-supervised learning, our modules could be learned efficiently without any human or automated labeling. We conduct extensive experiments across seven benchmarks and the results show that ACCM significantly outperforms existing methods with lower FLOPs (e.g., surpassing SOTA by 20.6% with 6.5% fewer FLOPs).

Figures

Figures reproduced from arXiv: 2508.01236 by the authors.

Figure 1
Figure 1. We apply FastV with 97% pruning rate on LLaVA [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our approach. Our ACCM consists of two components: a lightweight caption model and a selector. To be [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The impact of training data size in self-supervised [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance comparison of our approach and exist [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of token pruning results and supple [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 20 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  3. [3]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022)

    arXiv 2022

  4. [4]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichten- hofer, and Judy Hoffman. 2022. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461 (2022)

    arXiv 2022

  5. [5]

    Azzedine Boukerche, Lining Zheng, and Omar Alfandi. 2020. Outlier detection: Methods, models, and classification. ACM Computing Surveys (CSUR) 53, 3 (2020), 1–37

    work page 2020

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  7. [7]

    Yunhang Shen Yulei Qin Mengdan Zhang Xu Lin Jinrui Yang Xiawu Zheng Ke Li Xing Sun Yunsheng Wu Rongrong Ji Chaoyou Fu, Peixian Chen. 2021. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2021)

    arXiv 2021

  8. [8]

    Jieneng Chen, Luoxin Ye, Ju He, Zhaoyang Wang, Daniel Khashabi, and Alan L Yuille. 2024. Efficient large multi-modal models via visual context compression. Advances in Neural Information Processing Systems 37 (2024), 73986–74007

    work page 2024

Show all 61 references
  1. [9]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision. Springer, 19–35

    2024

  2. [10]

    Yi Chen, Jian Xu, Xu-Yao Zhang, Wen-Zhuo Liu, Yang-Yang Liu, and Cheng- Lin Liu. 2024. Recoverable compression: A multimodal vision token recovery mechanism guided by text information. arXiv preprint arXiv:2409.01179 (2024)

    2024 arXiv

  3. [11]

    Marcella Cornia, Lorenzo Baraldi, Giuseppe Fiameni, and Rita Cucchiara. 2021. Universal captioner: Long-tail vision-and-language model training through content-style separation. arXiv preprint arXiv:2111.12727 1, 2 (2021), 4

    2021 arXiv

  4. [12]

    Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. Advances in neural information processing systems 33 (2020), 4271–4282

    2020

  5. [13]

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakar- avarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Confer- ence on Machine Learning . PMLR, 3690–3699

    2020

  6. [14]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv e-prints (2024), arXiv–2403

    2024

  7. [15]

    Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. 2024. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876 (2024)

    2024 arXiv

  8. [16]

    Xin Huang, Ashish Khetan, Rene Bidart, and Zohar Karnin. 2022. Pyramid-BERT: Reducing complexity via successive core-set based token selection.arXiv preprint arXiv:2203.14380 (2022)

    2022 arXiv

  9. [17]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6700–6709

    2019

  10. [18]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023)

    2023 arXiv

  11. [19]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  12. [20]

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2024. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392 (2024)

    2024 arXiv

  13. [21]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen

  14. [22]

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2024. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. arXiv preprint arXiv:2405.05803 (2024)

    2024 arXiv

  15. [23]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

    2024

  16. [24]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llavanext: Improved reasoning, ocr, and world knowledge

    2024

  17. [25]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  18. [26]

    Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang

  19. [27]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2024. Mmbench: Is your multi-modal model an all-around player?. In European conference on computer vision. Springer, 216–233

    2024

  20. [28]

    Bruce T Lowerre. 1976. The harpy speech recognition system. Carnegie Mellon University

    1976

  21. [29]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024. Deepseek-vl: towards real- world vision-language understanding. arXiv preprint arXiv:2403.05525 (2024)

    2024 arXiv

  22. [30]

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. 2024. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural Information Processing Systems 37 (2024), 23464–23487

    2024

  23. [31]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37 (2024), 124198–124235

    2024

  24. [32]

    Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 (2021)

    2021 arXiv

  25. [33]

    Shun Qian, Bingquan Liu, Chengjie Sun, Zhen Xu, and Baoxun Wang. 2024. Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation. arXiv preprint arXiv:2410.10319 (2024)

    2024 arXiv

  26. [34]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  27. [35]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

    2018

  28. [36]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2023), 53728–53741

    2023

  29. [37]

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava- prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 (2024)

    2024

  30. [38]

    Shihui Ying Liejun Wang Xibin Zhao Shuyi Ji, Zizhao Zhang and Yue Gao. 2020. Kullback–leibler divergence metric learning. IEEE transactions on cybernetics 52(4) (2020), 2047–2058

    2020

  31. [39]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525 (2023)

    2023 arXiv

  32. [40]

    Wei Suo, Ji Ma, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang

  33. [41]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  34. [42]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9568–9578

    2024

  35. [43]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  36. [44]

    arXiv preprint arXiv:2412.06458 (2024)

    Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models. arXiv preprint arXiv:2412.06458 (2024)

    2024 arXiv

  37. [45]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition . 4566–4575

    2015

  38. [46]

    Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. 2024. Zero-TPrune: Zero- shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16070–16079

    2024

  39. [47]

    Xinjian Wu, Fanhu Zeng, Xiudong Wang, and Xinghao Chen. 2023. Ppt: To- ken pruning and pooling for efficient vision transformers. arXiv preprint arXiv:2310.01812 (2023)

    2023 arXiv

  40. [48]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  41. [49]

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mplug-owl3: Towards long image- sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840 (2024)

    2024 arXiv

  42. [50]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the association for computational linguistics 2 (2014), 67–78

    2014

  43. [51]

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision. Springer, 310–325

    2024

  44. [52]

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al . 2024. Pyramiddrop: MM ’25, October 27–31, 2025, Dublin, Ireland Mingyu Fu et al. Accelerating your large vision-language models via pyramid visual redu...

    2024 arXiv

  45. [53]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

    2022 arXiv

  46. [54]

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al

  47. [55]

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. 2024. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487 (2024)

    2024 arXiv

  48. [56]

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. 2025. LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token. arXiv preprint arXiv:2501.03895 (2025)

    2025 arXiv

  49. [59]

    arXiv preprint arXiv:2410.04417 (2024)

    Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 (2024)

    2024 arXiv

  50. [61]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  51. [2022]

    Advances in neural information processing systems 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    2022

  52. [2023]

    arXiv preprint arXiv:2305.10355 (2023)

    Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    2023 arXiv

  53. [2024]

    arXiv preprint arXiv:2411.10803 (2024)

    Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model. arXiv preprint arXiv:2411.10803 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.