Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free adapter that merges similar visual tokens before the LLM and progressively prunes attention-unimportant ones inside LLM layers can cut computation by roughly 7x with near-unchanged accuracy, and re-spending the savings on…

desk verdict A genuinely useful efficiency method with strong empirical results, but the PageRank equation as printed is degenerate — likely a typo — and the headline numbers carry selection and simulator caveats. read the letter →

arxiv 2412.03248 v2 pith:V5T7MQS4 submitted 2024-12-04 cs.CV cs.AIcs.CL

Yiwu Zhong , Zhuoming Liu , Yin Li , Liwei Wang This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords adaptiveinferencetokenmergingpruningmulti-modalLLMvisualredundancyPageRankimportancelongvideounderstandingtraining-freeacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a training-free adapter can make pre-trained multimodal LLMs much cheaper to run by removing redundant visual tokens in two steps: merging visually similar tokens before they enter the LLM, then progressively dropping the visual tokens that attention-based PageRank scores mark as least important at each LLM layer. On LLaVA-OneVision-7B this reduces FLOPs by a factor of 6.8 and prefill time by 8.0 while keeping video-benchmark accuracy nearly unchanged, and on LLaVA-1.5-7B it reduces FLOPs 3.7x with only a manageable image-benchmark loss. The same budget can then be re-spent: at matched compute, sampling 192 frames instead of 32 raises long-video accuracy by +4.6 on MLVU. If the paper is right, efficiency gains come from exploiting redundancy already present in the frozen model, no retraining is needed, and the saved budget can buy denser temporal coverage.

What carries the argument

The load-bearing object is the PageRank importance score of Eq. (1): each token's score is a sum of its incoming attention weights weighted by the scores of the tokens that attend to it, making the attention matrix an adjacency graph over all LLM tokens. The other two components are iterative token merging by cosine similarity before the LLM, and the piecewise scheduler of Eq. (2) with parameters $\ell^1$ and $\ell^2$ that decides which LLM layers start and stop visual-token pruning. Together they turn a frozen multimodal LLM into a configurable accuracy-efficiency device whose operating point is set by a retention ratio and two layer indices.

What would settle it

Run the same merging and pruning schedule on the same base model but choose the visual tokens to drop uniformly at random within each layer; if the random version matches the reported VideoMME score, the attention-based importance ranking is not doing the work, and the paper's mechanism claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that most visual tokens in a multimodal LLM are redundant at inference time, so a carefully scheduled combination of pre-LLM merging and in-LLM pruning can cut computation by an order of magnitude without retraining. The method merges adjacent visual tokens by cosine similarity before the LLM, then at each layer computes a PageRank-style importance score from the softmax attention matrix and prunes only visual tokens, leaving text tokens untouched. A piecewise scheduler keeps all visual tokens in early layers, prunes them gradually in middle layers, and removes them in later layers, reflecting the paper's finding that early layers perform cross-modal fusion while later layers focus on text reasoning. The authors support this with video and image benchmarks showing near-base accuracy at large compute reductions, and with an adaptive table showing a 40-fold FLOPs range at under 13% VideoMME drop.

Load-bearing premise

The load-bearing premise is that the PageRank scores computed from each layer's softmax attention reliably rank which visual tokens can be removed at that layer, so that dropping most visual tokens in later layers costs almost no accuracy.

Editorial extensions

If this is right

  • Frozen multimodal LLMs can be accelerated at deployment time by plugging in merging and pruning, with no fine-tuning or weight changes.
  • One checkpoint can serve many hardware budgets: adjusting the merging ratio and l1, l2 spans roughly a 40x FLOPs range at under 13% VideoMME accuracy loss.
  • The FLOPs saved by token reduction can be reinvested in denser video sampling; at equal compute, 192-frame inputs beat the 32-frame base model by +4.6 on MLVU.
  • Visual token redundancy is large: keeping only 25% of video tokens preserves close-to-base accuracy, and later LLM layers tolerate aggressive visual pruning, suggesting architectural guidance for future efficient multimodal LLMs.
  • Pruning text tokens during inference is costly (VideoMME drops from 58.2 to 45.7), so efficient designs should keep text tokens intact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the PageRank-based token choice with random visual-token pruning under the same merging and layer schedule; if random pruning matches accuracy, the attention ranking is not the source of the gains.
  • Because the scheduler and merging are modality-agnostic, the same adapter may transfer to audio or other non-text token streams in multimodal LLMs, though the paper only evaluates vision.
  • The incompatibility with FlashAttention noted by the paper suggests a natural next step: computing approximate token importance without materializing the full attention matrix, which would combine token reduction with efficient attention kernels.
  • The frame-budget result implies that for long-video tasks the optimal operating point may involve many more frames than current 32-frame training and evaluation protocols use; searching the frame-count versus token-retention frontier per video length is a direct extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AIM, a training-free adaptive inference method for multimodal LLMs. Visual tokens are first merged by cosine similarity before entering the LLM, then progressively pruned inside each LLM layer according to a PageRank-style importance score derived from attention weights, with a piecewise-linear retention scheduler. The method is evaluated on LLaVA-OneVision-7B for video and LLaVA-1.5-7B for image benchmarks, reporting up to 6.8x/8.0x FLOPs/prefill-time reductions with minimal accuracy loss, and a +4.6 MLVU improvement when the saved compute is re-spent on more sampled frames.

Significance. If the implementation matches a corrected version of the scoring formula, the work is practically relevant: it is training-free, applies to both video and image LLMs, and includes a broad comparison against FastV, VTW, PDrop, and LLaVA-Prumerge. Strengths include public code, detailed ablations on merging ratios and pruning schedulers (Tables 4, 5, A, B), and the clear demonstration that re-spending FLOPs on additional frames improves long-video understanding (Table 2). The main limitation is that the core scoring equation, as printed, is mathematically degenerate, so the method cannot be reconstructed from the paper until that equation is corrected and the selection of default hyperparameters is addressed.

major comments (3)
  1. [§3.3, Eq. (1)] Equation (1) cannot produce a token ranking as written. Since A^l is softmax-normalized, each row sums to 1, and the equation is s = (1/(N^l+M^l)) A s. A nonzero solution would be an eigenvector of the row-stochastic matrix A^l with eigenvalue N^l+M^l >= 2, contradicting the Perron-Frobenius bound that the spectral radius is 1. Starting from the uniform initialization stated in the text, one power iteration leaves every score at 1/(N^l+M^l), and further iterations decay to zero. Thus Eq. (1) either is not the rule used in the experiments (in which case the paper mis-specifies its core mechanism) or the pruning is effectively random (in which case the reported accuracy-efficiency curves are not reproducible). Please rewrite Eq. (1) with the actual PageRank variant, e.g., a damped iteration with a teleport term or a transpose with a damping factor, and state the initialization and iteration count explicitly.
  2. [§4, Implementation Details and Table 5] The default configuration (merging retention 25%, l1=14, l2=22 for video) was selected using the VideoMME ablations in Table 5, and the same benchmark is then used for the headline "no performance drop" result in Table 1. This makes the VideoMME row of Table 1 an in-sample selection result. Please either hold out a validation split, report hyperparameter selection on a development set separate from the reported benchmark, or explicitly frame VideoMME as the tuning benchmark and base the main claims on the other benchmarks (MVBench, MLVU, EgoSchema, NextQA, PerceptionTest), which are out-of-sample and do show small losses.
  3. [§4, Implementation Details and Tables 1–3] Prefill time is reported as computed with the LLM-Viewer library rather than measured on hardware. Since the "8.0x prefill time" reduction is a headline claim, please provide wall-clock prefill measurements for at least the base model and the default configuration on a reference GPU, or clearly label the reported values as roofline estimates and validate them on a subset. The same applies to the FLOPs numbers, which can be estimated accurately but are not actual runtime measurements.
minor comments (5)
  1. [§3.3, text after Eq. (2)] The sentence "the number of visual tokens input to the next layer is N^1 × rl" should presumably read N^l × rl; please correct the superscript.
  2. [Appendix A] The heading "Addtional Ablation Study" should be "Additional Ablation Study".
  3. [Table 3, caption] The caption states that "with less computation cost, our method outperforms baselines on most benchmarks," but there are two operating points for Ours with different FLOPs; please specify which operating point is being compared.
  4. [Table 5] Please clarify that Exp. 1 (l1=28, l2=29) is equivalent to no pruning under the scheduler in Eq. (2), since r_l = 1 for all layers l <= 28; naming this condition explicitly would help readers interpret the ablation.
  5. [Appendix B] The statement that the method is "not compatible" with FlashAttention is clearer if expanded: token pruning is incompatible with FlashAttention because attention probabilities are not materialized, while token merging alone is compatible, as Table D shows.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's efficiency and accuracy claims are empirical evaluations of a training-free token-merging and pruning adapter, not derivations from fitted inputs or self-cited uniqueness theorems.

full rationale

The paper contains no derivation step in which a predicted quantity is equivalent by construction to a fitted input. Token merging (cosine-similarity averaging) and token pruning (attention-based PageRank scores with the scheduler in Eq. 2) are operational heuristics applied to a frozen LLaVA model; the reported FLOPs/prefill reductions and benchmark accuracies are measured, not derived from the method's own equations. Hyperparameters (25% retention, l1=14, l2=22) are chosen from ablations on VideoMME and then reported on the same benchmark in Table 1; this is test-set selection rather than circular reasoning, because the accuracy values are not forced to a target by the parameter choice. The few self-citations ([81], [83], [99]) appear only in related-work context and do not supply a load-bearing premise or an imported uniqueness theorem. One internal issue is worth noting as a correctness/reproducibility concern rather than circularity: Eq. 1 as written, with row-stochastic softmax attention A and normalization 1/(N^l+M^l), cannot generate a non-uniform importance ranking (power iteration from a uniform initialization stays uniform and decays to zero), so the implementation must differ from the printed equation or the described pruning would be effectively random. This affects whether the mechanism is faithfully specified, but it does not make any benchmark result equivalent to the method's inputs, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two empirically motivated design choices: attention-based PageRank importance ranking for pruning, and per-frame spatial merging. Neither is derived from first principles; both are supported by ablations. The efficiency numbers also depend on LLM-Viewer estimates with fixed assumed text token counts. No invented entities are introduced.

free parameters (6)
  • Video token merging retention ratio = 25% default; 50%, 12.5%, 6.3%, 3.1%, 1.6% in adaptive sweep
    Selected to hit a target FLOPs regime; VideoMME ablations in Table 4 show 25% keeps performance while cutting tokens.
  • Video pruning scheduler l1 = 14
    Layer where pruning starts, chosen via VideoMME ablations in Table 5; starting earlier reduces accuracy.
  • Video pruning scheduler l2 = 22
    Layer where visual tokens are fully removed, chosen via VideoMME ablations; removing tokens by layer 15 or 8 hurts accuracy.
  • Image token merging retention ratio = 12.5%
    Selected for image experiments with LLaVA-1.5-7B to balance efficiency and accuracy.
  • Image pruning scheduler l1 and l2 = l1=13, l2=21
    Chosen for the image benchmark configuration in Table 3.
  • Assumed text token count for FLOPs estimation = 100 for video, 40 for image
    Used with LLM-Viewer to compute FLOPs and prefill time; different prompt lengths would change the reported ratios.
assumptions (5)
  • domain assumption Spatial token merging within each video frame preserves temporal information.
    Section 3.2 states that merging across frames is harmful; Appendix Table B supports this empirically, but it remains a design assumption.
  • domain assumption Text tokens are more important than visual tokens in later LLM layers and are never pruned.
    Section 3.3 and Table 6 show pruning text tokens drops VideoMME from 58.2 to 45.7.
  • domain assumption PageRank computed from softmax attention weights ranks visual token usefulness at each layer.
    Equation 1 defines importance; the paper gives empirical ablations but no guarantee that attention graph ordering is stable or sufficient.
  • domain assumption LLM-Viewer FLOPs and prefill time estimates with fixed text token counts approximate actual inference cost.
    Used for all efficiency tables; prefill times are simulator estimates rather than wall-clock measurements.
  • ad hoc to paper The piecewise linear scheduler with l1 and l2 spans the useful retention policies.
    Equation 2 is a heuristic chosen by the authors; ablations show it works, but the functional form is not derived from a principle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning." pith.science (2026). https://pith.science/paper/V5T7MQS4

@misc{pith2026241203248,
  author       = {Pith},
  title        = {Pith review of: AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5T7MQS4}},
  note         = {Machine review of arXiv:2412.03248}
}
abstract

Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos. However, these models usually rely on extensive visual tokens from visual encoders, leading to high computational demands, which limits their applicability in resource-constrained environments and for long-context tasks. In this work, we propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements with a minimum performance drop. Our method consists of a) iterative token merging based on embedding similarity before LLMs, and b) progressive token pruning within LLM layers based on multi-modal importance. With a minimalist design, our method can be applied to both video and image LLMs. Extensive experiments on diverse video and image benchmarks demonstrate that our method substantially reduces computation load (e.g., a $\textbf{7-fold}$ reduction in FLOPs) while preserving the performance of video and image LLMs. Further, at a similar computational cost, our method outperforms the state-of-the-art methods in long video understanding (e.g., $\textbf{+4.6}$ on MLVU). Additionally, our in-depth analysis provides insights into token redundancy and LLM layer behaviors, offering guidance for future research in designing efficient multi-modal LLMs. Our code is available at https://github.com/LaVi-Lab/AIM.

Figures

Figures reproduced from arXiv: 2412.03248 by the authors.

Figure 1
Figure 1. Key idea and finding. Our training-free method enables adaptive inference of pre-trained multi-modal LLMs, supporting a wide range of computational conditions. In comparison to the base pre-trained model, our method substantially reduces FLOPs, without or with a manageable performance drop. real-time processing on mobile devices). Further, as the number of video frames increases, the total number of to￾kens also gro… view at source ↗
Figure 2
Figure 2. Overview of our training-free method for pre-trained multi-modal LLMs, optimized for accuracy-efficiency trade-off. During inference, we first merge the input visual tokens of LLM based on the cosine similarity between token embeddings, reducing their redun￾dancy. The retained tokens are then fed into the LLM. Token pruning is applied within the LLM layers using the Page Rank algorithm, with a scheduler controlling … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    Across five video LLMs, 12.5-25.5% of multiple-choice items are correct at a lower visual budget but wrong at a higher one, even when aggregate scaling curves look safe.

  2. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

  3. Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ProxyV introduces proxy vision tokens that take over expensive attention and feed-forward computation in later layers of decoder-only multimodal models, cutting FLOPs by 25-46% while retaining or improving accuracy on...

  4. FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    FrameFusion merges similar adjacent-frame visual tokens before importance-based pruning, reducing video LVLM tokens by 70% with under 3% average performance loss.

  5. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

  6. AdaTP: Attention-Debiased Token Pruning for Video Large Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    AdaTP prunes visual tokens in video LLMs by debiasing attention scores, reducing FLOPs to about a quarter of the vanilla model at matching benchmark accuracy.

  7. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Reference graph

Works this paper leans on

102 extracted references · 52 canonical work pages · cited by 7 Pith papers

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 ,

    arXiv

  2. [2]

    Conditional computation in neural networks for faster models

    Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015. 3

    arXiv 2015

  3. [3]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023. 2, 4

    2023

  4. [4]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 1

    arXiv 2023

  5. [5]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference accelera- tion for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference accelera- tion for large vision-language models. InEuropean Confer- ence on Computer Vision, pages 19–35. Springer, 2025. 2, 5, 6, 14

    2025

  6. [6]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 1, 5

    2023

  7. [7]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 14

    arXiv 1904

  8. [8]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 1, 2

    arXiv 2023

Show all 102 references
  1. [9]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. 14

    2023 arXiv

  2. [10]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information pro- cessing systems, 35:30318–30332, 2022. 14

    2022

  3. [11]

    HeatViT: Hardware-Efficient Adap- tive Token Pruning for Vision Transformers

    Peiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie, Kenneth Liu, Zhenglun Kong, Xin Meng, Zhengang Li, Xue Lin, Zhenman Fang, et al. HeatViT: Hardware-Efficient Adap- tive Token Pruning for Vision Transformers. InProceedings of the IEEE International Symposium on High-Performance C...

    2023

  4. [12]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In In- ternational Conference on Machine Learning, pages 5547–

  5. [13]

    Hsu, and Shang-Hong Lai

    Gueter Josmy Faure, Jia-Fong Yeh, Min-Hung Chen, Hung- Ting Su, Winston H. Hsu, and Shang-Hong Lai. Hermes: temporal-coherent long-form understanding with episodes and semantics, 2024. 2

    2024

  6. [14]

    Adaptive Token Sampling for Efficient Vision Transformers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and J¨urgen Gall. Adaptive Token Sampling for Efficient Vision Transformers. In Proceedings of the European Conference on Computer Vi...

    2022

  7. [15]

    Spatially adaptive computation time for residual networks

    Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1039– 1048, 2017. 3

    2017

  8. [16]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. 5

    2024

  9. [17]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 5

    2024 arXiv

  10. [18]

    PoWER-BERT: Accelerating BERT Inference via Progressive Word-Vector Elimination

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. PoWER-BERT: Accelerating BERT Inference via Progressive Word-Vector Elimination. In Proceedings of the International Conference on Machine Learning , pages 3690–3...

    2020

  11. [19]

    Making the v in vqa matter: El- evating the role of image understanding in visual question answering, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: El- evating the role of image understanding in visual question answering, 2017. 5

    2017

  12. [20]

    Speedboost: Anytime pre- diction with uniform near-optimality

    Alex Grubb and Drew Bagnell. Speedboost: Anytime pre- diction with uniform near-optimality. In Artificial Intelli- gence and Statistics, pages 458–466. PMLR, 2012. 3

    2012

  13. [21]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. 2

    2024

  14. [22]

    Dynamic neural networks: A sur- vey

    Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A sur- vey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456, 2021. 1, 3

    2021

  15. [23]

    Learning anytime predictions in neural net- works via adaptive loss balancing

    Hanzhang Hu, Debadeepta Dey, Martial Hebert, and J An- drew Bagnell. Learning anytime predictions in neural net- works via adaptive loss balancing. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3812– 3821, 2019. 3

    2019

  16. [24]

    Longrecipe: Recipe for efficient long context generalization in large language models, 2024

    Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, and Bryan Hooi. Longrecipe: Recipe for efficient long context generalization in large language models, 2024. 2

    2024

  17. [25]

    Language is not all you need: Aligning perception with language mod- els

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language mod- els. Advances in Neural Information Processing Systems , 36, 2024. 1

    2024

  18. [26]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 5

    2019

  19. [27]

    Anytime recognition with routing convolutional networks

    Zequn Jie, Peng Sun, Xin Li, Jiashi Feng, and Wei Liu. Anytime recognition with routing convolutional networks. IEEE transactions on pattern analysis and machine intelli- gence, 43(6):1875–1886, 2019. 3

    2019

  20. [28]

    Anytime recognition of objects and scenes

    Sergey Karayev, Mario Fritz, and Trevor Darrell. Anytime recognition of objects and scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 572–579, 2014. 3

    2014

  21. [29]

    Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search

    Gyuwan Kim and Kyunghyun Cho. Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search. arXiv preprint arXiv:2010.07003, 2020. 2

    2010 arXiv

  22. [30]

    Learned Token Pruning for Transformers

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned Token Pruning for Transformers. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784–794, 2022. 2

    2022

  23. [31]

    SPViT: Enabling Faster Vision Transform- ers Latency-Aware Soft Token Pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. SPViT: Enabling Faster Vision Transform- ers Latency-Aware Soft Token Pruning. In Proceedings of the European Conference on Computer Vision, pages 620–

  24. [32]

    Text-conditioned resampler for long form video understanding, 2024

    Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zis- serman, and Federico Tombari. Text-conditioned resampler for long form video understanding, 2024. 2

    2024

  25. [33]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 3, 5, 15

    2024 arXiv

  26. [34]

    2d or not 2d? adaptive 3d convolution selec- tion for efficient video recognition

    Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry S Davis. 2d or not 2d? adaptive 3d convolution selec- tion for efficient video recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6155–6164, 2021. 3

    2021

  27. [35]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 2

    2023 arXiv

  28. [36]

    Mvbench: A comprehensive multi- modal video understanding benchmark, 2023

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark, 2023. 1

    2023

  29. [37]

    Mvbench: A comprehensive multi- modal video understanding benchmark, 2024

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark, 2024. 5

    2024

  30. [38]

    Evaluating object hallucination in large vision-language models, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023. 5

    2023

  31. [39]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 2

    2023 arXiv

  32. [40]

    Boosting multimodal large language models with visual to- kens withdrawal for rapid inference

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual to- kens withdrawal for rapid inference. Proceedings of the AAAI Conference on Artificial Intelligence, 2025. 2, 5, 6

    2025

  33. [41]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 1, 2, 3, 5, 6

    2023

  34. [42]

    Adap- tive Sparse ViT: Towards Learnable Adaptive Token Prun- ing by Fully Exploiting Self-Attention

    Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adap- tive Sparse ViT: Towards Learnable Adaptive Token Prun- ing by Fully Exploiting Self-Attention. arXiv preprint arXiv:2209.13802, 2022. 2

    2022 arXiv

  35. [43]

    Mm- bench: Is your multi-modal model an all-around player?,

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mm- bench: Is your multi-modal model an all-around player?,

  36. [44]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. 5

    2022

  37. [45]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (ACL 2024) , 2024. 1, 2

    2024

  38. [46]

    Egoschema: A diagnostic benchmark for very long- form video language understanding, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding, 2023. 5

    2023

  39. [47]

    Llama3.2

    Meat. Llama3.2. https://ai.meta.com/blog/ llama - 3 - 2 - connect - 2024 - vision - edge - mobile-devices/, 2024. 2

    2024

  40. [48]

    Adavit: Adaptive vision transformers for efficient image recogni- tion

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recogni- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12309–12318,

  41. [49]

    Ar-net: Adaptive frame resolution for effi- cient action recognition

    Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for effi- cient action recognition. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,...

    2020

  42. [50]

    Compositional chain of thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain of thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  43. [51]

    Encod- ing and controlling global semantics for long-form video question answering, 2024

    Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong- Duy T Nguyen, See-Kiong Ng, and Anh Tuan Luu. Encod- ing and controlling global semantics for long-form video question answering, 2024. 2

    2024

  44. [52]

    OpenAI. ChatGPT. https://openai.com/blog/ chatgpt/, 2023. 2

    2023

  45. [53]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 2

    2023

  46. [54]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35...

    2022

  47. [55]

    The pagerank citation ranking: Bringing order to the web

    Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, 1999. 4

    1999

  48. [56]

    Ia-red2: Interpretability-aware redundancy reduction for vision transformers

    Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-red2: Interpretability-aware redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898–24911, 2021. 3

    2021

  49. [57]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica P ˘atr˘aucean, Lucas Smaira, Ankush Gupta, Adri`a Recasens Continente, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Joseph Heyward, Mateusz Mali- nowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Rapha...

    2023

  50. [58]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervi- sion. In ICML, 2021. 1

    2021

  51. [59]

    DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. Advances in Neural Information Processing Systems , 34: 13937–13949, 2021. 2, 3

    2021

  52. [60]

    Finding the sweet spot: Analysis and improve- ment of adaptive inference in low resource settings

    Daniel Rotem, Michael Hassid, Jonathan Mamou, and Roy Schwartz. Finding the sweet spot: Analysis and improve- ment of adaptive inference in low resource settings. InPro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2023

  53. [61]

    Interpolating video-llms: Toward longer- sequence lmms in a training-free manner, 2024

    Yuzhang Shang, Bingxin Xu, Weitai Kang, Mu Cai, Yuheng Li, Zehao Wen, Zhen Dong, Kurt Keutzer, Yong Jae Lee, and Yan Yan. Interpolating video-llms: Toward longer- sequence lmms in a training-free manner, 2024. 1

    2024

  54. [62]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. In ICCV, 2025. 2, 5, 6, 14, 15

    2025

  55. [63]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. 5

    2019

  56. [64]

    Dynamic Token Pruning in Plain Vision Trans- formers for Semantic Segmentation

    Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, and Yi- fan Liu. Dynamic Token Pruning in Plain Vision Trans- formers for Semantic Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 777–786, 2023. 2

    2023

  57. [65]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the Com- puter Vision and Pattern Recognition Conference , pages 18992–19001, 2025. 2

    2025

  58. [66]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  59. [67]

    SpAtten: Ef- ficient Sparse Attention Architecture with Cascade Token and Head Pruning

    Hanrui Wang, Zhekai Zhang, and Song Han. SpAtten: Ef- ficient Sparse Attention Architecture with Cascade Token and Head Pruning. In Proceedings of the IEEE Interna- tional Symposium on High-Performance Computer Archi- tecture, pages 97–110, 2021. 2

    2021

  60. [68]

    Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers

    Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. Zero- tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16070–16079, 2024. 2, 4

    2024

  61. [69]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  62. [70]

    Skipnet: Learning dynamic routing in convolutional networks

    Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the Euro- pean conference on computer vision (ECCV) , pages 409– 424, 2018. 3

    2018

  63. [71]

    Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture, 2024

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture, 2024. 2

    2024

  64. [72]

    Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition

    Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. In Advances in Neural Information Processing Systems, 2021. 3

    2021

  65. [73]

    Videollamb: Long-context video understanding with recur- rent memory bridges, 2024

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video understanding with recur- rent memory bridges, 2024. 2

    2024

  66. [74]

    Visual context win- dow extension: A new perspective for long video under- standing, 2024

    Hongchen Wei and Zhenzhong Chen. Visual context win- dow extension: A new perspective for long video under- standing, 2024. 2

    2024

  67. [75]

    Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers

    Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2092–2101, 2023. 2

    2023

  68. [76]

    Longvlm: Efficient long video under- standing via large language models, 2024

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video under- standing via large language models, 2024. 2

    2024

  69. [77]

    Blockdrop: Dynamic inference paths in residual net- works

    Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8817–8826, 2018. 3

    2018

  70. [78]

    Next-qa:next phase of question-answering to explaining temporal actions, 2021

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question-answering to explaining temporal actions, 2021. 5

    2021

  71. [79]

    Conical visual concen- tration for efficient large vision-language models

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. Conical visual concen- tration for efficient large vision-language models. In Pro- ceedings of the Computer Vision and Pattern Recognition Confe...

    2025

  72. [80]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1

    2024 arXiv

  73. [81]

    Smartadapt: Multi-branch object detection framework for videos on mo- biles

    Ran Xu, Fangzhou Mu, Jayoung Lee, Preeti Mukherjee, Somali Chaterji, Saurabh Bagchi, and Yin Li. Smartadapt: Multi-branch object detection framework for videos on mo- biles. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 2528–2538,

  74. [82]

    The greedy miser: learning under test-time budgets

    Zhixiang Xu, Kilian Q Weinberger, and Olivier Chapelle. The greedy miser: learning under test-time budgets. InPro- ceedings of the 29th International Coference on Interna- tional Conference on Machine Learning, pages 1299–1306,

  75. [83]

    Learning to inference adaptively for multimodal large language models

    Zhuoyan Xu, Khoi Duc Nguyen, Preeti Mukherjee, Saurabh Bagchi, Somali Chaterji, Yingyu Liang, and Yin Li. Learning to inference adaptively for multimodal large language models. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, 2025. 3

    2025

  76. [84]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 2

    2024 arXiv

  77. [85]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 1, 5

    2024 arXiv

  78. [86]

    Tr-BERT: Dynamic Token Reduction for Accelerating BERT Inference

    Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. Tr-BERT: Dynamic Token Reduction for Accelerating BERT Inference. arXiv preprint arXiv:2105.11618, 2021. 2

    2021 arXiv

  79. [87]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024. 2

    2024

  80. [88]

    Fit and prune: Fast and training-free visual token pruning for multi-modal large language models

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 22128– 22136, 2025. 2

    2025

  81. [89]

    Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov

    Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive To- kens for Efficient Vision Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809–10818, 2022. 2

    2022

  82. [90]

    Llm inference unveiled: Survey and roofline model in- sights, 2024

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model in- sights, 2024. 5

    2024

  83. [91]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1

    2023

  84. [92]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 1, 2, 5, 15

    2024 arXiv

  85. [93]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 1

    2022 arXiv

  86. [94]

    Llava-next: A strong zero-shot video understanding model,

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model,

  87. [95]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 2

    2024

  88. [96]

    Sparsevlm: Vi- sual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Vi- sual token sparsification for efficient vision-language model inference. In International Conference on Machine Learn-...

    2025

  89. [97]

    Multimodal chain-of- thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. 2

    2023 arXiv

  90. [98]

    Treat visual tokens as text? but your mllm only needs fewer efforts to see

    Zeliang Zhang, Phu Pham, Wentian Zhao, Kun Wan, Yu- Jhe Li, Jianing Zhou, Daniel Miranda, Ajinkya Kale, and Chenliang Xu. Treat visual tokens as text? but your mllm only needs fewer efforts to see. arXiv preprint arXiv:2410.06169, 2024. 2

    arXiv

  91. [99]

    Beyond embeddings: The promise of visual table in vi- sual reasoning

    Yiwu Zhong, Zi-Yuan Hu, Michael Lyu, and Liwei Wang. Beyond embeddings: The promise of visual table in vi- sual reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 6876–6911. Association for Computational Linguis- tics, 2024. 2

    2024

  92. [100]

    Mlvu: A comprehensive benchmark for multi- task long video understanding, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi- task long video understanding, 2024. 1, 5

    2024

  93. [101]

    Languagebind: Extending video- language pretraining to n-modality by language-based se- mantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video- language pretraining to n-modality by language-based se- mantic alignment. arXiv preprint arXiv:2310.01852, 2023. 2

    arXiv

  94. [102]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2 Appendices In the appendices, we provide more detailed results in addi- ti...

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.