Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Spectral energy of visual features can set per-image token budgets for VLMs, improving accuracy at matched compute.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:40 UTC pith:6SJIEY35

load-bearing objection Simple adaptive-budget trick with real but modest gains; the abstract overclaims (Qwen2.5 missing) and the energy-preservation story is unverified. the 4 major comments →

arxiv 2603.05950 v2 pith:6SJIEY35 submitted 2026-03-06 cs.CV cs.AI

Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models

Jialuo He , Huangxun Chen This is my paper
classification cs.CV cs.AI
keywords vision-language modelstoken pruningadaptive token budgetsingular value decompositionspectral energyinference accelerationtraining-free
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

E-AdaPrune proposes that the right number of visual tokens for a vision-language model is an intrinsic property of the image, measurable from the singular value spectrum of the visual feature matrix. It computes an image-specific token budget by keeping only enough singular components to retain a set fraction (τ) of the total squared spectral energy, then feeds that budget to any existing top-k token selector. Under matched average token budgets across nine benchmarks and three model families, this adaptive budgeting improves average accuracy by up to 0.6%, with the largest relative gain (+5.1%) on MMVet reasoning. The method is training-free and adds roughly 8 ms per image when budget estimation uses randomized SVD.

Core claim

The paper's central claim is that visual token budgets should be adaptive, and that the correct budget per image is encoded in the spectral energy distribution of the visual feature matrix. By computing the smallest rank k* whose cumulative squared singular values reach a threshold τ of total energy, E-AdaPrune allocates more tokens to information-dense scenes (flat spectra) and fewer to redundant scenes (steep spectral decay). This k* is then used as the cut-off for existing selectors such as attention-based top-k or merging methods, decoupling budget determination from token selection. The paper reports consistent average gains up to 0.6% over static baselines at matched mean token budgets

What carries the argument

The central object is the singular value decomposition (SVD) of the visual feature matrix Z_V ∈ R^{n_v × d_v}. The method defines 'image energy' as the sum of squared singular values and selects the minimum number of components k* that capture a fraction τ of that energy (Eq. 6), clamped to a practical range [k_min, k_max]. This energy-based rank acts as a content-aware token budget that is orthogonal to—and can be combined with—any existing token-scoring heuristic (FastV, PyramidDrop, VisionZip). Randomized SVD (rSVD) is used to estimate the spectrum cheaply, adding about 8 ms per image.

Load-bearing premise

The number of singular-value components needed to reach a τ fraction of the total squared spectral energy of the whole visual feature matrix is a valid proxy for the number of visual tokens the VLM actually needs, and this energy is actually preserved after the downstream token selector picks its top tokens—two assumptions the paper does not directly verify.

What would settle it

Compute, on a benchmark set, the squared spectral energy contained in the actually selected top-k tokens (according to the integrated selector) for each image and compare it to the τ threshold that determined k*. If the retained energy falls far below τ for many images, or if an image with a flat spectrum (high k*) answers correctly with far fewer tokens, the proxy is not doing the causal work the paper claims.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the spectral-energy budget is a valid proxy for what a VLM needs, static top-k pruning is suboptimal: simple images can be pruned more aggressively and the saved compute can be redirected to dense images.
  • Because E-AdaPrune is training-free and model-agnostic, it can be dropped into existing acceleration pipelines without altering their scoring mechanisms, potentially combining with many future selectors.
  • The energy-threshold τ acts as a global information-preservation guarantee across heterogeneous datasets, which may make adaptive pruning more predictable across benchmarks.
  • The measured +5.1% relative MMVet gain suggests that reasoning-heavy, information-dense scenes are precisely where static budgets discard critical tokens, so adaptive budgeting may help high-level VQA tasks specifically.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract and full text disagree on scope: the abstract reports a fourth backbone (Qwen2.5-VL-3B) and a specific SQA^I result (35.8% fewer tokens, 52.5% more tokens on hard cases) that do not appear in the body; readers should treat the body as authoritative for what was actually run.
  • The validity of the energy proxy itself is not directly verified: the paper assumes the top-k tokens selected by the downstream selector retain the spectral energy that justified k*, but never measures the energy contained in the retained token set.
  • A natural extension is to turn τ into a tunable per-query or per-task knob rather than a fixed global hyperparameter, since the paper's results show gains shrink at lower τ (99.0% vs 99.8%).
  • The spectral-energy idea may transfer to other sequence-compression settings (e.g., video frames or long documents), but the claim of being text-agnostic has not been stress-tested beyond the VLM benchmarks reported.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. E-AdaPrune proposes a training-free, plug-and-play module that replaces a static visual token budget in VLM token-pruning pipelines with an image-specific budget k*, computed as the minimum rank at which the cumulative squared singular values of the visual feature matrix reach a threshold τ. This budget is then passed to existing selectors (FastV, PyramidDrop, VisionZip). The paper claims that, under matched average token budgets, this adaptive allocation improves average accuracy by up to 0.6%, with a +5.1% relative gain on MMVet, while adding only 8 ms per image via randomized SVD. Experiments are reported on three LLaVA backbones and nine benchmarks, although the abstract additionally claims Qwen2.5-VL-3B results that do not appear in the body.

Significance. If the claims hold, the contribution is a simple and attractive addition to the visual-token-pruning toolbox: it is training-free, introduces no learnable parameters, and can be dropped into existing pruning pipelines by replacing a static budget with a per-image budget. The matched-average-budget evaluation protocol is a strength, and the rSVD latency analysis is a useful engineering result. The central weakness is that the paper's stated justification—that the selected token set preserves a fixed fraction of spectral energy—is never verified; the selectors do not retain the singular components on which k* is based. In addition, the abstract overstates the experimental scope by promising Qwen2.5-VL-3B results that are absent. With the requested revisions, the core idea could be publishable as an adaptive budget heuristic rather than as a principled energy-preserving method. I agree with the stress-test assessment: the circularity concern does not land, because k* is defined from the feature matrix alone, but the energy-preservation mechanism is unsupported.

major comments (4)
  1. [§3.2, Eq. (6); §4.1] The central claim that k* 'preserves spectral energy' is not established. k* is the minimal rank of the full visual feature matrix Z_V whose cumulative squared singular values reach τ, but the selectors used here do not retain the top singular components: FastV chooses attention-top-k tokens after layer 2, PDrop drops layer-wise by importance scores, and VisionZip merges tokens by similarity. The retained token subset can therefore have an energy fraction substantially different from τ, and the paper never measures this quantity. A simple check is to report ||Z_retained||_F^2 / ||Z_V||_F^2 for the actually selected tokens. Table 4 also provides internal evidence against the energy link: with q=0 the average budget changes from 130 to 156 tokens but MMB is essentially unchanged (62.8 vs 62.5). Please either verify retained-energy preservation or reframe k* as a heuristic budget, ideally w
  2. [Abstract; §4.1; §4.3] The abstract states that E-AdaPrune is evaluated 'across four VLM backbones' and gives concrete Qwen2.5-VL-3B numbers for SQA^I (35.8% fewer tokens, 0.52% relative decrease, 1.94% improvement). The experiments in the body are limited to LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-NeXT-8B; no Qwen experiment appears. This is a factual overstatement of the reported scope and should be corrected by adding the Qwen experiments or removing the claims.
  3. [§4.2; Tables 1-2] Many of the core improvements are within the range of a few tenths of a percentage point, and no significance testing, confidence intervals, or repeated runs are reported. At τ=99.0% the paper itself reports average improvements of only 0.3%, 0.1%, and 0.1% for the three baselines. The phrase 'consistently yields' is stronger than the evidence supports. The tables also mix raw scores and relative-to-base percentages without a clear mapping, making it difficult to verify the stated averages. Please add statistical support and clarify the reporting scheme.
  4. [§4.1; Algorithm 1] The method depends on τ, kmin/kmax, and the PDrop ρ clamp intervals ([0.4,0.6] at τ=99.8%, [0.6,0.8] at τ=99.0%). None of these are justified by sensitivity analysis, and kmin/kmax are not reported. Because the static baseline budget is the mean of the adaptive budgets, these choices directly determine the comparison operating point. A sensitivity table over τ and the clamp ranges is needed to show that the reported gains are not artifacts of calibration.
minor comments (5)
  1. [§4.1] The text says 'nine benchmarks' but enumerates only eight; Table 1 actually contains nine columns (MMB and MMBCN are counted separately). Please align the text and the table.
  2. [Figures 2 and 3] Figures 2 and 3 appear to contain the same diagram with different captions. If this is not a layout artifact, one of the figures should be replaced.
  3. [Appendix, Algorithm 2] Algorithm 2 introduces p, k_sub, and a target dimension without clearly connecting them to the t and q used in §4.4. The notation should be made consistent for reproducibility.
  4. [§2.3; §4] The paper positions E-AdaPrune against learned adaptive methods (ATP-LLaVA, VCM, VisionThink) but provides no experimental comparison with any of them. Even a single reference comparison would help clarify the trade-offs.
  5. [Abstract; Conclusion] The abstract's 'up to 0.6%' and '+5.1% relative gain' mix absolute and relative performance changes. The paper should state clearly which quantities are relative to Base and which are relative to the static baseline.

Circularity Check

0 steps flagged

No circular derivation: k* is a self-contained spectral-energy rank; gains are measured externally against budget-calibrated baselines.

full rationale

The budget k* is computed from Eq. (6) as the minimal rank whose cumulative squared singular values reach τ, using only the visual feature matrix Z_V and the hyperparameters τ, k_min, k_max; no labels, benchmark scores, or downstream accuracy are used to derive it. Thus the central 'adaptive budget' is not a fitted parameter renamed as a prediction. The static baselines are calibrated to the dataset mean of the adaptive budgets for a matched-average-cost comparison; this is an experimental control, not a derivation loop. The energy-preservation language ('preserving a certain proportion of spectral energy') is a stated target for the rank k*, not a verified property of the actually retained attention-selected token subset; if the selected tokens do not retain the promised energy fraction, that is an unvalidated mechanism or correctness concern, not circularity. The paper's self-citations (e.g., refs [7] and [14], which share authors with the present paper) appear only in related-work survey sentences about token merging and are not load-bearing for the proposed method. No uniqueness theorem, ansatz, or definition is imported from the authors' prior work to force the result. The empirical improvements are measured on external benchmarks and are not equivalent by construction to the budget definition.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central method introduces no new physical or architectural entities. The load-bearing free parameters are the energy threshold τ, the k min/k max clamps, and the PyramdDrop ρ-clamps; all are hand-set. The key domain assumption is that spectral energy of the full feature matrix corresponds to the information content actually needed after attention-based token selection.

free parameters (4)
  • Energy threshold τ = 99.0% / 99.8%
    Chosen by hand as the preservation target; controls the average token budget and the magnitude of reported gains.
  • kmin/kmax rank bounds = not specified numerically
    Clamp k_raw to stability bounds; values are not reported, so the effective adaptivity range is under-specified.
  • PDrop dropping-ratio clamp = [0.4,0.6] (τ=99.8%), [0.6,0.8] (τ=99.0%)
    Hand-set per compression level; severely restricts the adaptive ratio for PyramidDrop, weakening the 'adaptive' claim for that selector.
  • rSVD target rank t and power iterations q = t=300, q=2 in reported run; q=0/1 explored
    These affect the estimated singular value spectrum and hence the budget; q=0 changes average token count from 130 to 156 at t=300.
axioms (5)
  • domain assumption The squared singular values of the visual feature matrix Z_V measure task-relevant information content of the image.
    Section 3.2; the entire budget criterion rests on this equivalence between spectral energy and semantic informativeness.
  • domain assumption Preserving τ fraction of total spectral energy is sufficient for the VLM to maintain accuracy on the evaluated benchmarks.
    Section 3.2; no PAC-style or independent validation that the energy threshold corresponds to an accuracy floor.
  • domain assumption The top-k token selector (attention/feature norm) will retain the tokens that realize the preserved energy.
    Section 3.2 and experiments; the paper never shows that attention-top-k tokens carry the same spectral energy as the principal components.
  • standard math Randomized SVD with power iteration yields singular values close enough to exact SVD for the rank-selection step.
    Section 4.4 and Appendix Algorithm 2, based on Halko et al.; empirically validated only at t=300, q=2.
  • domain assumption The nine benchmarks and the three LLaVA backbones provide a representative evaluation of VLM performance.
    Section 4.1; standard benchmarks, but the claim about general applicability is inferred from these models.

pith-pipeline@v1.3.0-alltime-deepseek · 13405 in / 11571 out tokens · 103011 ms · 2026-08-02T18:40:20.013611+00:00 · methodology

0 comments
read the original abstract

Visual token reduction is critical for accelerating Vision-Language Models (VLMs), since visual inputs are represented as token sequences that introduce substantial computational overhead in the LLM backbone. However, most pruning pipelines treat efficiency primarily as a token selection problem and retain a fixed visual token budget across inputs, overlooking the substantial variation in image information density. We propose E-AdaPrune, an energy driven adaptive pruning framework that determines an image specific token budget from the singular value spectrum of the visual feature matrix and passes this budget to existing token selectors. By preserving a certain proportion of spectral energy, our method allocates more tokens to information dense scenes while assigning fewer tokens to redundant scenes, without introducing additional learnable parameters. We evaluate E-AdaPrune across four VLM backbones, three token selectors, and nine benchmarks under matched average token budgets. Results show that E-AdaPrune removes a substantial amount of redundant computation from simple cases and converts the saved budget into larger gains on information rich cases. Notably, on SQA$^\mathrm{I}$ with Qwen2.5-VL-3B, E-AdaPrune uses 35.8\% fewer tokens for simple cases with only a 0.52\% relative performance decrease. The saved budget is redirected to hard cases, which receive 52.5\% more tokens and achieve a 1.94\% relative performance improvement.

Figures

Figures reproduced from arXiv: 2603.05950 by Huangxun Chen, Jialuo He.

Figure 1
Figure 1. Figure 1: Different images contain different amounts of visual information, indicating that a static token budget may either discard critical details or retain unnecessary redundancy. All examples are from TextVQA [33]. strategy, utilizing a fixed top-k budget or a predefined pruning ratio across all in￾put instances. Such static approaches fail to account for the significant variability in information density betwe… view at source ↗
Figure 2
Figure 2. Figure 2: VLM forward process architecture. Circled numbers indicate viable locations for visual token pruning: ① the vision-LLM interface and ② intermediate LLM layers. vision encoder EV maps an input image X V into a visual feature space: \mathbf {Z}^V = \mathcal {E}_V(\mathcal {X}^V) \in \mathbb {R}^{n_v \times d_v}, (1) where nv and dv denote the number of output visual tokens and their respective feature dimens… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of static and adaptive pruning. (a) FastV uses a fixed top-k budget regardless of image content. (b) E-AdaPrune determines a content-aware budget k ∗ via an image-specific energy criterion, optimizing token retention for varying information densities. V and R denote token importance scores and rankings. existing research suggests that specific tasks like document understanding or high-precision … view at source ↗
Figure 4
Figure 4. Figure 4: Singular value spectra comparison. Simple image (a) shows sharp decay with k ∗ = 95. Hard image (b) has a flat spectrum requiring k ∗ = 259. Red dashed lines indicate the adaptive rank at τ = 99%. threshold τ : k_{raw}=\min \left \{k\in \{1,\dots ,n\}\mid \frac {\sum _{i=1}^{k}\sigma _{i}^{2}}{\sum _{i=1}^{n}\sigma _{i}^{2}}\ge \tau \right \} (6) While kraw represents the ideal information-centric budget, … view at source ↗
Figure 5
Figure 5. Figure 5: TextVQA visualization (τ = 99.0%). Compared to static FastV (bottom), FastV+E (middle) adaptively allocates tokens—retaining more for dense scenes to en￾sure accuracy and fewer for simple scenes to improve efficiency. Green and red indicate correct and incorrect responses, respectively. all, these results demonstrate that energy-driven adaptive budgeting effectively handles multi-scale visual representatio… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding

    cs.CV 2026-05 unverdicted novelty 6.0

    PARCEL is a new visual tokenization architecture combining pool-anchored resampling with conditioned elastic queries to enhance performance-efficiency tradeoffs in LVLMs over prior matryoshka methods.

  2. FreqCache: Accelerating Embodied VLN Models with Adaptive Frequency-Guided Token Caching

    cs.RO 2026-04 unverdicted novelty 6.0

    FreqCache uses frequency domain properties to adaptively select, refresh, and budget token caches in VLN models, delivering 1.59x speedup with negligible overhead.

  3. Toward Native Multimodal Modeling: A Roadmap

    cs.CV 2026-05 unverdicted novelty 3.0

    A roadmap that defines architectural nativity for multimodal models and categorizes them into Multi-to-Text, Multi-to-Target, and Multi-to-Multi types while outlining an industrial pipeline toward unified transformer-...

Reference graph

Works this paper leans on

50 extracted references · 8 linked inside Pith · cited by 3 Pith papers

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Advances in neural information processing systems35, 23716– 23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022)

    2022

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    In: The Eleventh International Conference on Learning Representations (2023)

    Bolya,D.,Fu,C.Y.,Dai,X.,Zhang,P.,Feichtenhofer,C.,Hoffman,J.:Tokenmerg- ing: Your vit but faster. In: The Eleventh International Conference on Learning Representations (2023)

    2023

  5. [5]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Cai, M., Yang, J., Gao, J., Lee, Y.J.: Matryoshka multimodal models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  6. [6]

    In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition

    Cao, J., Ye, P., Li, S., Yu, C., Tang, Y., Lu, J., Chen, T.: Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language trans- former. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 15710–15719 (2024)

    2024

  7. [7]

    Advances in neural information pro- cessing systems37, 73986–74007 (2024)

    Chen, J., Ye, L., He, J., Wang, Z.Y., Khashabi, D., Yuille, A.: Efficient large multi- modal models via visual context compression. Advances in neural information pro- cessing systems37, 73986–74007 (2024)

    2024

  8. [8]

    In: European Conference on Computer Vision

    Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., Chang, B.: An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In: European Conference on Computer Vision. pp. 19–35. Springer (2024)

    2024

  9. [9]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Advances in neural information processing systems36, 49250–49267 (2023)

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems36, 49250–49267 (2023)

    2023

  11. [11]

    In: Proceedings of the 60th Annual Meeting ofthe Associationfor Computational Linguistics (Volume 1: Long Papers)

    Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., Tang, J.: Glm: General language model pretraining with autoregressive blank infilling. In: Proceedings of the 60th Annual Meeting ofthe Associationfor Computational Linguistics (Volume 1: Long Papers). pp. 320–335 (2022)

    2022

  12. [12]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025)

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R., Shan, C., He, R.: MME: A comprehensive evaluation benchmark for multimodal large language models. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025)

    2025

  13. [13]

    SIAM review53(2), 217–288 (2011)

    Halko, N., Martinsson, P.G., Tropp, J.A.: Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review53(2), 217–288 (2011)

    2011

  14. [14]

    He et al

    Han, Y., Liu, X., Zhang, Z., Ding, P., Wang, D., Chen, H., Yan, Q., Huang, S.: Filter, correlate, compress: Training-free token reduction for mllm acceleration (2025) 16 J. He et al

    2025

  15. [15]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  16. [16]

    In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Khaki, S., Guo, J., Tang, J., Yang, S., Chen, Y., Plataniotis, K.N., Lu, Y., Han, S., Liu, Z.: Sparsevila: Decoupling visual sparsity for efficient vlm inference. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 23784–23794 (October 2025)

    2025

  17. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y.: Seed- bench: Benchmarking multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13299– 13308 (2024)

    2024

  18. [18]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    2023

  19. [19]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Li, K., Goyal, S., Semedo, J.D., Kolter, J.Z.: Inference optimal VLMs need fewer visual tokens and more parameters. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  20. [20]

    In: Proceedings of the 2023 conference on empirical methods in natural language processing

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. pp. 292–305 (2023)

    2023

  21. [21]

    arXiv preprint arXiv:2501.021891, 1 (2025)

    Li, Z., Wu, X., Du, H., Nghiem, H., Shi, G.: Benchmark evaluations, applica- tions, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.021891, 1 (2025)

    Pith/arXiv arXiv 2025

  22. [22]

    In: Proceedings of the AAAI Confer- ence on Artificial Intelligence

    Lin, Z., Lin, M., Lin, L., Ji, R.: Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In: Proceedings of the AAAI Confer- ence on Artificial Intelligence. vol. 39, pp. 5334–5342 (2025)

    2025

  23. [23]

    In: The Thirteenth International Con- ference on Learning Representations (2025)

    Liu, H., Yan, W., Zaharia, M., Abbeel, P.: World model on million-length video and language with blockwise ringattention. In: The Thirteenth International Con- ference on Learning Representations (2025)

    2025

  24. [24]

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2024)

    2024

  25. [25]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llava-next: Improved reasoning, ocr, and world knowledge (January 2024)

    2024

  26. [26]

    In: Thirty-seventh Conference on Neural Information Processing Systems (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)

    2023

  27. [27]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    2024

  28. [28]

    Advances in neural information processing systems35, 2507– 2521 (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems35, 2507– 2521 (2022)

    2022

  29. [29]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Sys- tems (2025)

    Luo, R., Shan, R., Chen, L., Liu, Z., Wang, L., Yang, M., Xia, X.: VCM: Vision con- cept modeling with adaptive vision token compression via instruction fine-tuning. In: The Thirty-ninth Annual Conference on Neural Information Processing Sys- tems (2025)

    2025

  30. [30]

    Foundations and Trends®in Machine Learning3(2), 123–224 (2011)

    Mahoney, M.W.: Randomized algorithms for matrices and data. Foundations and Trends®in Machine Learning3(2), 123–224 (2011)

    2011

  31. [31]

    OpenAI: Gpt-4v(ision) system card (2023) E-AdaPrune 17

    2023

  32. [32]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Shang, Y., Cai, M., Xu, B., Lee, Y.J., Yan, Y.: Llava-prumerge: Adaptive token reduction for efficient large multimodal models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22857–22867 (2025)

    2025

  33. [33]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317–8326 (2019)

    2019

  34. [34]

    arXiv preprint arXiv:2403.05530 (2024)

    Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vin- cent,D.,Pan,Z.,Wang,S.,etal.:Gemini1.5:Unlockingmultimodalunderstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    Pith/arXiv arXiv 2024

  35. [35]

    Advances in neural information pro- cessing systems30(2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

    2017

  36. [36]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  37. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., et al.: Conical visual concentration for efficient large vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14593–14603 (2025)

    2025

  38. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19792– 19802 (2025)

    2025

  39. [39]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    Yang, S., Li, J., Lai, X., Wu, J., Li, W., MA, Z., Yu, B., Zhao, H., Jia, J.: Vision- think: Smart and efficient vision language model via reinforcement learning. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  40. [40]

    In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision

    Yang, S., Xu, R., Cui, C., Wang, T., Lin, D., Pang, J.: Vflowopt: A token pruning framework for lmms with visual information flow-guided optimization. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision. pp. 23924– 23934 (2025)

    2025

  41. [41]

    Authorea Preprints (2025)

    Yao, L., Xing, L., Shi, Y., Li, S., Liu, Y., Dong, Y., Zhang, Y.F., Li, L., Dong, Q., Dong, X., et al.: Towards efficient multimodal large language models: A survey on token compression. Authorea Preprints (2025)

    2025

  42. [42]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ye, X., Gan, Y., Ge, Y., Zhang, X.P., Tang, Y.: Atp-llava: Adaptive token pruning for large vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24972–24982 (2025)

    2025

  43. [43]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Ye, X., Gan, Y., Huang, X., Ge, Y., Tang, Y.: Voco-llama: Towards vision com- pression with large language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29836–29846 (2025)

    2025

  44. [44]

    arXiv preprint arXiv:2308.02490 (2023)

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: Mm- vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023)

    Pith/arXiv arXiv 2023

  45. [45]

    IEEE transactions on pattern analysis and machine intelligence46(8), 5625–5644 (2024)

    Zhang, J., Huang, J., Jin, S., Lu, S.: Vision-language models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence46(8), 5625–5644 (2024)

    2024

  46. [46]

    arXiv preprint arXiv:2309.15112 (2023) 18 J

    Zhang, P., Dong, X., Wang, B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., Duan, H., Zhang, S., Ding, S., et al.: Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 (2023) 18 J. He et al

    Pith/arXiv arXiv 2023

  47. [47]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, Q., Cheng, A., Lu, M., Zhang, R., Zhuo, Z., Cao, J., Guo, S., She, Q., Zhang, S.: Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20857–20867 (2025)

    2025

  48. [48]

    In: Forty-second International Conference on Machine Learning (2025)

    Zhang, Y., Fan, C.K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D.A., Okuno, T., Nakata, Y., Keutzer, K., Zhang, S.: SparseVLM: Visual token sparsifi- cation for efficient vision-language model inference. In: Forty-second International Conference on Machine Learning (2025)

    2025

  49. [49]

    In: The Twelfth International Conference on Learning Representations (2024)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision- language understanding with advanced large language models. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  50. [50]

    highlighted tokens

    Zou, X., Lu, D., Wang, Y., Yan, Y., Lyu, Y., Zheng, X., Zhang, L., Hu, X.: Don’t just chase “highlighted tokens” in MLLMs: Revisiting visual holistic context reten- tion. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025) E-AdaPrune 19 Appendix Algorithm 2:Efficient Energy-based Rank Selection via Randomized SVD Input:V...

    2025