Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning

Chieh-Kai Lin; Hung-Jen Chen; Min Sun; Ruei-Chi Lai; Shiang-Feng Tsai; Tzu-Chun Chien

REVIEW 3 major objections 5 minor 1 cited by

Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper identifies misaligned position IDs, not lost visual information, as the primary cause of pruning-induced performance collapse in visual grounding, and shows that preserving the original IDs recovers about 90% of the accuracy at…

desk verdict A genuinely useful zero-cost fix for pruning-induced grounding collapse, packaged with a causal story that is stronger than the evidence; the fix deserves refereeing even if the primary cause claim should be softened. read the letter →

arxiv 2506.21873 v1 pith:CMCU2KGT submitted 2025-06-27 cs.CV cs.AI

Tzu-Chun Chien , Chieh-Kai Lin , Shiang-Feng Tsai , Ruei-Chi Lai , Hung-Jen Chen , Min Sun This is my paper

classification cs.CVcs.AI

keywords visualgroundingtokenpruningpositionIDsmultimodallargelanguagemodelsreferringexpressioncomprehensionrotaryembeddingsspatialreasoningefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why token pruning, a standard way to speed up multimodal large language models, destroys their ability to localize objects from natural-language descriptions. It argues that the main culprit is not lost visual information but misaligned position IDs: after pruning, surviving visual tokens are renumbered, and the model loses the spatial layout it was trained on. The proposed method, GAP, keeps the original position IDs after pruning, recovering about 90% of the unpruned accuracy on referring expression comprehension (REC) across five models and six pruning strategies. This matters because it suggests a zero-cost, retraining-free fix that makes existing pruning methods safe for spatially demanding tasks.

What carries the argument

The central mechanism is the position ID sequence assigned to visual tokens before they enter the LLM, combined with the rotary embedding that encodes these IDs. Standard pruning re-assigns IDs after selection, scrambling the grid; GAP instead retains the original IDs, so each surviving token keeps its pre-pruning grid coordinate. This adjustment is a drop-in correction to any pruning method and carries no computational overhead.

What would settle it

Train a nonlinear classifier (e.g., a small MLP) on the final vision-encoder layer's features to predict each token's two-dimensional grid position; if it reaches high accuracy, the claim that spatial information is nearly lost by the LLM input is falsified. Alternatively, run an ablation that randomly permutes the preserved original position IDs while keeping the same selected tokens: if grounding accuracy stays near GAP's level, then the specific ID values, not merely their consistency, would not be doing the claimed work.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that pruning-induced performance collapse in visual grounding is caused primarily by misalignment between visual tokens and their position IDs, not by the loss of visual content. Two distinct misalignments are identified: permuted misalignment, where importance-based reordering assigns tokens to sequential IDs that no longer match their original grid positions, and shifted misalignment, where removed tokens cause survivors to be renumbered starting from the first ID. The paper supports the causal role of these misalignments by showing that inducing either misalignment without pruning any tokens reproduces a large performance drop, and by a linear probe showing that spatial information is nearly absent from ViT features by the time they reach the LLM. Consequently, the LLM must rely on its constructed position IDs as the main spatial channel, and GAP, which preserves the original position ID sequence while discarding pruned tokens, recovers about 90% of the unpruned REC accuracy with no added training, memory, or computation.

Load-bearing premise

The argument assumes that by the time visual features reach the language model, spatial layout is essentially unavailable in the features themselves, a conclusion drawn from a linear probe; if the language model could decode positions nonlinearly, the causal story would be weaker.

Editorial extensions

If this is right

GAP can be added to any token-pruning method that selects a subset of visual tokens, provided the original coordinates of survivors are known, and it improves accuracy under CLS-visual, text-visual, random, and spatial pruning.
Because GAP requires no retraining, it can be applied post hoc to already-deployed pruning pipelines without touching the model weights.
Applying GAP does not hurt VQA-style tasks; on GQA, VizWiz, and OKVQA it is neutral or slightly positive, so it can be enabled unconditionally.
At aggressive pruning ratios the recovered accuracy can exceed the unpruned baseline, consistent with earlier reports that moderate token reduction sometimes helps performance.
The finding implies that pruning evaluations that only report VQA-style accuracy understate the risk of pruning for tasks that require spatial layout.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism is right, then token-merging schemes that create new tokens rather than selecting survivors should also suffer grounding collapse unless they preserve position information, suggesting a testable variant of GAP for merge-based pruning.
The linear-probe result predicts that injecting a lightweight spatial readout at the LLM input layer would reduce the model's dependence on position IDs and thereby lower pruning sensitivity, an experiment the paper does not run.
The near-zero cost of the fix opens the possibility of pushing token pruning far more aggressively for latency-critical deployment, since the main obstacle identified here is positional rather than informational.
One could construct an adversarial test that scrambles the original position IDs while keeping the same selected tokens; if grounding accuracy stays high under such scrambling, the specific ID values would not be the causal channel the paper claims.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The practical result is solid and easy to state: when token pruning wrecks visual grounding accuracy, simply keeping the original position IDs for the surviving tokens recovers most of the lost performance. The paper demonstrates this across five MLLMs and four pruning methods, with enough consistency to make the effect believable. That is a real, minimal, and novel fix. Prior pruning work focused on which tokens to drop, not on the positional identities of the survivors. Identifying that failure mode is a legitimate contribution, and the zero-overhead property (Table 6) makes it directly usable.

The main results are broad and artifact-free as far as I can see: no fitted parameters, held-out benchmarks, several models and pruning strategies. The recovery is not uniform - Shikra only gets back to about 54% of baseline while LLaVA gets 92% - so the abstract's 90% claim is model-specific, but the pattern is unmistakable.

The soft spots are in the causal diagnosis, not the fix. Section 4.2.1 describes controlled misalignment experiments that isolate the effect of misaligned position IDs, but it never reports the numbers. That is the direct evidence for the causal claim, and it is missing. Section 4.2.2 uses a linear probe on ViT layers to argue spatial information is nearly lost and therefore position IDs are the primary source. As the stress-test note says, a linear probe does not rule out nonlinear decoding, and probing ViT layers rather than the post-projection LLM input weakens the inference. The empirical fix can be excellent even without the causal claim being settled, but primary cause is stronger than what current evidence supports.

Minor issues: no code, no error bars, vague description of the probe setup. These are addressable and do not undermine the core result. This paper is for people working on efficient MLLM inference or robustness in visual grounding. I would send it to peer review, asking the authors to tone down the causal wording and provide the missing numbers from Section 4.2.1. The fix itself is worth publishing; the diagnosis can be softened.

Referee Report

3 major / 5 minor

Summary. The paper studies the drastic accuracy drop observed in visual grounding (REC) when vision tokens are pruned from multimodal LLMs. The authors identify misaligned position IDs—both shifted and permuted after pruning—as the primary cause, and propose Grounding-Aware Token Pruning (GAP), a training-free modification that retains the original pre-pruning position IDs while discarding pruned tokens. Experiments on five MLLMs (LLaVA-v1.5, LLaVA-v1.6, LLaVA-NeXT, MiniGPTv2, Shikra) and six pruning methods show consistent recovery: e.g., LLaVA-v1.5 RefCOCO val rises from 15.34% to 51.42%, about 92% of the unpruned baseline, with no added TTFT or memory cost (Table 6). The authors also report that GAP does not hurt VQA performance.

Significance. GAP is a simple, parameter-free, and architecture-agnostic add-on. Its empirical benefit is substantial and well demonstrated across models and pruning methods, and the efficiency measurement in Table 6 is a genuine strength. The controlled experiments in §4.2.1 and the linear probe in §4.2.2 are intended to establish the causal mechanism, but they are currently under-specified: the former is presented without quantitative results and the latter measures only linear decodability of pre-projector features. If the causal claim is softened to 'a major and practically addressable cause,' the paper's contribution stands; if the authors add quantitative misalignment experiments and/or a nonlinear-probe intervention, the causal claim would be much stronger. No code is released, which limits reproducibility of the exact numbers; nevertheless, the main empirical pattern is likely robust.

major comments (3)

[4.2.1] The controlled misalignment experiments in §4.2.1 are the direct evidence that position-ID misalignment, rather than loss of visual content, causes the REC collapse, but they are reported only qualitatively ('the presence of both types of misalignment alone results in performance degradation') and Fig. 2b appears to contain the only numbers, with no axis labels or error bars in the manuscript text. Please report the exact RefCOCO accuracies for the aligned, permuted, and shifted conditions, describe how the conditions were constructed (which tokens were kept, how position IDs were manipulated), and state how many runs were averaged. This is load-bearing for the 'primary cause' claim.
[4.2.2, Table 2] The linear probe in Table 2 is trained on ViT features before the projector and measures linear decodability, not the information available to a Transformer. A nonlinear readout by the LLM or a reorganization by the MLP projector could in principle recover spatial information, so the conclusion that the LLM 'must rely heavily on its own constructed position IDs' is not entailed by the 2.61% top-1 accuracy at layer 23. I recommend adding a nonlinear probe on the post-projector input to the LLM, or an intervention that shuffles only the LLM position IDs while keeping visual tokens and their order fixed; either test would directly support or weaken the causal diagnosis.
[Table 1] The 99%-tokens result for MiniGPTv2 (87.35 to 18.58) is remarkable and motivates the paper, but the manuscript does not explain how removing roughly 1% of tokens is sufficient to trigger the position-ID mechanism, nor does it report the corresponding '+ GAP' accuracy at the same ratio. Please provide the number of removed tokens, the effect of the resulting position-ID shift on the remaining sequence, and the GAP result at 99% tokens; without these, the motivating observation and its link to the mechanism remain under-specified.

minor comments (5)

[4.2.1] The example in §4.2.1 uses inconsistent notation: given S = [4,2,1,5,3], the text says the tokens are rearranged as {v1, v4, v3, v0, v2}, which does not follow from Eq. (2) and mixes 0-based v0 with 1-based v1..v5. Please rewrite this example with the actual indexing convention used by topk.
[Eq. (7)] The spatial pruning formula as written, i = n·N/r, produces indices larger than N for typical reduction ratios (e.g., r=0.5), so it does not define a valid pruning set. Please correct the formula to sample the intended positions.
[Sec. 5.2] The text states that 'grounding task performance consistently dropped by more than 40% across all models under pruning,' but Table 5 shows LLaVA-v1.6 dropping from 87.58 to 56.35, a relative drop of about 36%. Please correct the statement or define the metric used for the comparison.
[References] Reference [1] (CLIP) has a corrupted author list ('Chris Hallacy Aditya Ramesh Gabriel Goh...') and should be corrected.
[Table 2] The capitalization of layer labels in Table 2 ('Layer 1' vs 'layer 23') is inconsistent; please unify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GAP is a fixed, parameter-free intervention validated on held-out benchmarks, and the causal diagnosis is supported by independent controlled experiments.

full rationale

The paper's central claim is that pruning-induced misalignment of position IDs is the primary cause of grounding performance drops, and that GAP, which restores original position IDs, recovers most of the lost accuracy. This claim is not circular: GAP is a fixed rule (retain the original position IDs) with no fitted parameters, so the reported RefCOCO/RefCOCO+/RefCOCOg improvements in Tables 5 and 7 are genuine held-out benchmark evaluations rather than quantities reconstructed from the method's own definition. The controlled experiments in Sec. 4.2.1 degrade grounding performance with misalignment alone while removing no visual tokens, providing independent evidence for the causal role of position IDs. The linear probe in Sec. 4.2.2 supporting the 'spatial information is nearly lost' claim is a possible overclaim because linear decodability is not the same as information availability to a nonlinear Transformer, but this is a correctness/evidential-strength concern, not circularity. No load-bearing self-citations were found; the cited prior works are external pruning methods and model papers. The method's validity does not depend on accepting the strongest causal interpretation, since the empirical recovery is documented directly. Overall, the derivation chain is self-contained and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two domain assumptions: that the LLM relies on position IDs for spatial reasoning (supported by the linear probe) and that re-indexing position IDs does not alter other inference behavior. No free parameters are fitted; the reduction ratio is an experimental variable, not a tuned constant. No new entities are postulated.

assumptions (3)

domain assumption The LLM component of the MLLM must derive spatial layout from its own position IDs because the ViT output at layer 23 retains almost no linearly decodable position information.
Sec. 4.2.2 trains a linear probe on each ViT layer; accuracy drops from 98.37% to 2.61%. This assumes linear decodability is a faithful proxy for the information available to the downstream LLM.
domain assumption Retaining the original position IDs after pruning leaves all other parts of the forward pass (attention masking, norm layers, KV cache) unchanged, so the only difference between standard pruning and GAP is the position ID values.
This underlies the claim of no additional computational or memory overhead (Sec. 5.3, Table 6). In practice, implementations must adapt rotary embeddings and generation loops, but the conceptual assumption is that no other component changes.
ad hoc to paper The degradation observed after pruning is wholly attributable to position-ID misalignment, not to the loss of visual information from removing tokens.
The paper attempts to support this with the no-token-loss misalignment experiments (Sec. 4.2.1) but does not provide their quantitative results in the text, so this remains an assumption in the chain of reasoning.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning." pith.science (2026). https://pith.science/paper/CMCU2KGT

@misc{pith2026250621873,
  author       = {Pith},
  title        = {Pith review of: Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMCU2KGT}},
  note         = {Machine review of arXiv:2506.21873}
}

read the original abstract

Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual grounding, establishing themselves as a general interface for various vision-language applications. This progress has driven the development of token pruning methods to mitigate the high computational costs associated with processing numerous visual tokens. However, we observe that pruning significantly weakens the model's grounding ability, leading to incorrect predictions and drastic performance degradation. In Referring Expression Comprehension (REC), for instance, pruning causes the accuracy of LLaVA on the RefCOCO validation set to drop from 56.14% to 15.34%. Our analysis identifies misaligned position IDs after pruning as the primary cause of this degradation, as both the order and value of these IDs are crucial for maintaining performance in grounding tasks. To address this issue, we propose Grounding-Aware Token Pruning (GAP), a simple yet effective adjustment to position IDs that recovers REC accuracy back to 51.42%, which is 90% of the original performance in the without pruning setting, all while requiring no additional training, memory, or computational overhead. Applied to models such as Shikra, MiniGPTv2, and the LLaVA series, our method consistently improves performance across various token pruning strategies.

Figures

Figures reproduced from arXiv: 2506.21873 by the authors.

**Figure 1.** Catastrophic performance drop. Comparison of the performance of LLaVA on the visual grounding dataset RefCOCO. Performance drops drastically after pruning, but is restored after applying our approach, GAP, which preserves the grounding ability of MLLMs. [17, 20], computational costs and inefficiencies increase. High memory usage, latency, and higher FLOPs hinder scalability. To address these issues, various token … view at source ↗

**Figure 2.** Visual Token and Position ID Misalignment. We analyze two types of misalignment caused by pruning and conduct experiments on LLaVA as described in Sec. 4.2.1. The results show that while these misalignments have minimal impact on the vision question answering task (GQA), they lead to significant performance degradation in the grounding task (RefCOCO), highlighting a major limitation that must be resolved. fraction … view at source ↗

**Figure 3.** Overview of GAP. (a) The MLLM concatenates visual and text tokens, assigning ordered position IDs. (b) Standard pruning reorders position IDs based on the pruned tokens. (c) GAP discards pruned tokens without incurring additional overhead, while preserving the original position IDs. modified position IDs, and ensure these updates are applied appropriately during the generation process. GAP prevents the misalignment … view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Cross-ratio generalization capabilities. The gray line represents the w/o pruning result with 100% tokens. This figure shows the accuracy trend across different pruning ratios using the CLS-visual pruning method Eq. (4), with and without GAP. See Sec. 5.4 for more det…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference
cs.CV 2026-07 conditional novelty 7.0 of 10

Same-accuracy pruned vision-language prefixes can retain very different amounts of traceable OCR support (PosECR 0.620 vs 0.270 for Target vs Random at 30% budget), so pruning evaluation should report spatial provenan...

Reference graph

Works this paper leans on

33 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

Learning transferable visual models from natu- ral language supervision

Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agar- wal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever Alec Radford, Jong Wook Kim. Learning transferable visual models from natu- ral language supervision. arXiv:2103.00020, 2021. 2

arXiv 2021
[2]

Lmms-eval: Accelerating the development of large multimoal models, 2024

Kaichen Zhang* Fanyi Pu* Xinrun Du Yuhao Dong Hao- tian Liu Yuanhan Zhang Ge Zhang Chunyuan Li Bo Li*, Peiyuan Zhang* and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024. 6

work page 2024
[3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

2020
[4]

PuMer: Pruning and merging tokens for efficient vision language models

Qingqing Cao, Bhargavi Paranjape, and Hannaneh Ha- jishirzi. PuMer: Pruning and merging tokens for efficient vision language models. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 12890–12903, Toronto, Canada, 2023. Association for Computational Linguistics. 1

work page 2023
[5]

Shikra: Unleashing multi- modal llm’s referential dialogue magic

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 1, 2, 6

arXiv 2023
[6]

Chasing sparsity in vision transform- ers: An end-to-end exploration

Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transform- ers: An end-to-end exploration. Advances in Neural Infor- mation Processing Systems, 34:19974–19988, 2021. 7

work page 2021
[7]

Recoverable compression: A mul- timodal vision token recovery mechanism guided by text in- formation

Yi Chen, Jian Xu, Xu-Yao Zhang, Wen-Zhuo Liu, Yangyang Liu, and Cheng-Lin Liu. Recoverable compression: A mul- timodal vision token recovery mechanism guided by text in- formation. ArXiv, abs/2409.01179, 2024. 1, 6, 7

arXiv 2024
[8]

Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. 2

work page 2019

Show all 33 references

[9]

Minigpt-4: Enhancing vision-language understanding with advanced large language models

Xiaoqian Shen Xiang Li Mohamed Elhoseiny Deyao Zhu, Jun Chen. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592, 2023. 2

2023 arXiv
[10]

Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 2

2024
[11]

Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3608–3617, 2018. 1

2018
[12]

Ferret: Refer and ground anything anywhere at any granularity

Zhe Gan Xianzhi Du Bowen Zhang Zirui Wang Lian- gliang Cao Shih-Fu Chang Yinfei Yang Haoxuan You, Hao- tian Zhang. Ferret: Refer and ground anything anywhere at any granularity. arXiv:2310.07704, 2023. 2

2023 arXiv
[13]

Hudson and Christopher D

Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and composi- tional question answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693–6702, 2019. 1, 3

2019
[14]

Minigpt- v2: Large language model as a unified interface for vision- language multi-task learning

Xiaoqian Shen Xiang Li Zechun Liu Pengchuan Zhang Raghuraman Krishnamoorthi Vikas Chandra Yunyang Xiong Jun Chen, Deyao Zhu and Mohamed Elhoseiny. Minigpt- v2: Large language model as a unified interface for vision- language multi-task learning. arXiv:2310.09478, 2023. 1, 2, 6

arXiv
[15]

Mdetr– modulated detection for end-to-end multi-modal understand- ing

Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr– modulated detection for end-to-end multi-modal understand- ing. arXiv preprint arXiv:2104.12763, 2021. 2

2021 arXiv
[16]

ReferItGame: Referring to objects in pho- tographs of natural scenes

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 787–798, Doha, Qatar, 2014. Assoc...

2014
[17]

Llava-onevision: Easy visual task transfer

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1

2024 arXiv
[18]

Improved baselines with visual instruction tuning, 2023

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 2

2023
[19]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 2, 6

2023
[20]

Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1, 2, 6

2024
[21]

Ok-vqa: A visual question answer- ing benchmark requiring external knowledge

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answer- ing benchmark requiring external knowledge. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3190–3199, 2019. 1

2019
[22]

Llava-prumerge: Adaptive token reduction for efficient large multimodal models

Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,
[23]

Grounding dino: Marrying dino with grounded pre-training for open-set object detection

Tianhe Ren Feng Li Hao Zhang Jie Yang Qing Jiang Chun- yuan Li Jianwei Yang Hang Su Jun Zhu Lei Zhang Shi- long Liu, Zhaoyang Zeng. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv:2303.05499, 2023. 2

2023 arXiv
[24]

Less is more: A sim- ple yet effective token reduction method for efficient multi- modal llms

Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A sim- ple yet effective token reduction method for efficient multi- modal llms. arXiv preprint arXiv:2409.10994, 2024. 1, 2, 3, 6, 7

2024 arXiv
[25]

Smarttrim: Adaptive tokens and attention pruning for efficient vision-language models.arXiv preprint arXiv:2305.15033, 2023

Zekun Wang, Jingchang Chen, Wangchunshu Zhou, Haichao Zhu, Jiafeng Liang, Liping Shan, Ming Liu, Dongliang Xu, Qing Yang, and Bing Qin. Smarttrim: Adaptive tokens and attention pruning for efficient vision-language models.arXiv preprint arXiv:2305.15033, 2023. 7

2023 arXiv
[26]

Cogvlm: Visual expert for pretrained language models

Wenmeng Yu Wenyi Hong Ji Qi Yan Wang Junhui Ji Zhuoyi Yang Lei Zhao Xixuan Song Jiazheng Xu Bin Xu Juanzi Li Yuxiao Dong Ming Ding Jie Tang Weihan Wang, Qing- song Lv. Cogvlm: Visual expert for pretrained language models. arXiv:2311.03079, 2023. 2

2023 arXiv
[27]

Sigmoid loss for language image pre- training

Alexander Kolesnikov Lucas Beyer Xiaohua Zhai, Basil Mustafa. Sigmoid loss for language image pre- training. arXiv:2303.15343, 2023. 2

2023 arXiv
[28]

Referring expression comprehension: A survey of methods and datasets

Qi Wu Yanyuan Qiao, Chaorui Deng. Referring expression comprehension: A survey of methods and datasets. IEEE TMM, 2020. arXiv:2007.09554. 2

2020 arXiv
[29]

Balancing performance and efficiency: A multimodal large language model pruning method based image text interaction.ArXiv, abs/2409.01162,

Gaotong Yu, Yi Chen, and Jian Xu. Balancing performance and efficiency: A multimodal large language model pruning method based image text interaction.ArXiv, abs/2409.01162,

arXiv
[30]

Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023. 2

2023
[31]

Eva: Exploring the limits of masked visual representation learning at scale

Binhui Xie Quan Sun Ledell Wu Xinggang Wang Tiejun Huang Xinlong Wang Yue Cao Yuxin Fang, Wen Wang. Eva: Exploring the limits of masked visual representation learning at scale. arXiv:2211.07636, 2022. 2

2022 arXiv
[32]

[cls] attention is all you need for training- free visual token pruning: Make vlm inference faster

Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shang- hang Zhang. [cls] attention is all you need for training- free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818, 2024. 7

2024 arXiv
[33]

Sparsevlm: Visual token sparsification for efficient vision- language model inference

Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparsevlm: Visual token sparsification for efficient vision- language model inference. 2024. 1, 2, 5

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Learning transferable visual models from natu- ral language supervision

Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agar- wal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever Alec Radford, Jong Wook Kim. Learning transferable visual models from natu- ral language supervision. arXiv:2103.00020, 2021. 2

arXiv 2021

[2] [2]

Lmms-eval: Accelerating the development of large multimoal models, 2024

Kaichen Zhang* Fanyi Pu* Xinrun Du Yuhao Dong Hao- tian Liu Yuanhan Zhang Ge Zhang Chunyuan Li Bo Li*, Peiyuan Zhang* and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024. 6

work page 2024

[3] [3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

2020

[4] [4]

PuMer: Pruning and merging tokens for efficient vision language models

Qingqing Cao, Bhargavi Paranjape, and Hannaneh Ha- jishirzi. PuMer: Pruning and merging tokens for efficient vision language models. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 12890–12903, Toronto, Canada, 2023. Association for Computational Linguistics. 1

work page 2023

[5] [5]

Shikra: Unleashing multi- modal llm’s referential dialogue magic

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 1, 2, 6

arXiv 2023

[6] [6]

Chasing sparsity in vision transform- ers: An end-to-end exploration

Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transform- ers: An end-to-end exploration. Advances in Neural Infor- mation Processing Systems, 34:19974–19988, 2021. 7

work page 2021

[7] [7]

Recoverable compression: A mul- timodal vision token recovery mechanism guided by text in- formation

Yi Chen, Jian Xu, Xu-Yao Zhang, Wen-Zhuo Liu, Yangyang Liu, and Cheng-Lin Liu. Recoverable compression: A mul- timodal vision token recovery mechanism guided by text in- formation. ArXiv, abs/2409.01179, 2024. 1, 6, 7

arXiv 2024

[8] [8]

Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. 2

work page 2019