Pith. sign in

REVIEW 17 cited by

An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.06764 v3 pith:HXHSEMVR submitted 2024-03-11 cs.CV cs.AIcs.CL

Liang Chen , Haozhe Zhao , Tianyu Liu , Shuai Bai , Junyang Lin , Chang Zhou , Baobao Chang This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords fastvmodelsattentioncomputationallvlmsperformancetokensb-parameter
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this study, we identify the inefficient attention phenomena in Large Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5, QwenVL-Chat and Video-LLaVA. We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs, suggesting a need for a sparser approach compared to textual data handling. To this end, we introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent ones. Our evaluations demonstrate FastV's ability to dramatically reduce computational costs (e.g., a 45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a wide range of image and video understanding tasks. The computational efficiency and performance trade-off of FastV are highly customizable and pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve a lower budget than that of a 7B-parameter model, while still maintaining superior performance. We believe FastV has practical values for deployment of LVLMs in edge devices and commercial models. Code is released at https://github.com/pkunlp-icler/FastV.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Visual-token attention filtering improves structured pruning of vision-language models, keeping 94% of average benchmark accuracy after removing 20% of parameters.

  2. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

  3. ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ORCA compresses 3D CT tokens into organ-guided connected regions with sinusoidal centroid encoding, outperforming grid average and other compressors at matched budgets.

  4. LayoutLite: Token-Level Implicit Layout Analysis for Efficient Document OCR

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Token-level importance scoring trained by RL plus a layout-detection teacher prunes half the visual tokens in VLM document OCR while holding OmniDocBench scores within about 1.3-2.2 points and cutting prefill cost by ...

  5. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  6. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  7. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  8. SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A training-free framework jointly quantizes a VLA model to 4 bits and prunes visual tokens, recovering or exceeding full-precision success rates at 1.93x speedup.

  9. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  10. AdaFV: Rethinking of Visual-Language alignment for VLM acceleration

    cs.CV 2025-01 conditional novelty 6.0 of 10

    AdaFV prunes visual tokens before the LLM by adaptively mixing visual saliency and text-to-image similarity, achieving state-of-the-art training-free VLM acceleration at high reduction rates.

  11. LongViTU: Instruction Tuning for Long-Form Video Understanding

    cs.CV 2025-01 conditional novelty 6.0 of 10

    LongViTU, an automatically generated 121k-pair video QA dataset with 4.6-minute average certificate length, provides modest SFT gains on long-video benchmarks.

  12. ST$^3$: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming

    cs.CV 2024-12 conditional novelty 6.0 of 10

    ST3 progressively removes inattentive visual tokens across network layers and across generation steps, achieving about 2x faster MLLM inference with roughly 30 to 50 percent of the original KV cache memory.

  13. AVA-VLM: Adaptive Visual Attention-Vision Language Model for In-the-Wild Construction Site Monitoring

    cs.CV 2026-07 conditional novelty 5.0 of 10

    AVA-VLM reduces visual-token usage by 69% while improving PPE-violation F1 by 13 points over direct-QA baselines by training a VLM to adaptively crop high-resolution local regions from a downsampled global image.

  14. GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A training-free token pruning method that combines cosine-similarity saliency with greedy redundancy removal to preserve accuracy at high compression ratios.

  15. FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.

  16. What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph

    cs.CV 2025-01 conditional novelty 5.0 of 10

    G-Prune prunes visual tokens for MLLMs via graph-based information propagation, cutting LLaVA-NeXT FLOPs by about 63% with small accuracy loss.

  17. Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends

    cs.CL 2025-01 conditional novelty 3.0 of 10

    A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.

Pith tools