Pith. sign in

REVIEW 10 cited by

VoCo-LLaMA: Towards Vision Compression with Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12275 v2 pith:3ZI6UG36 submitted 2024-06-18 cs.CV

Xubing Ye , Yukang Gan , Xiaoke Huang , Yixiao Ge , Yansong Tang This is my paper
classification cs.CV
keywords visioncompressiontokensllmsvoco-llamaapproachcompresscompressed
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Vision-Language Models (VLMs) have achieved remarkable success in various multi-modal tasks, but they are often bottlenecked by the limited context window and high computational cost of processing high-resolution image inputs and videos. Vision compression can alleviate this problem by reducing the vision token count. Previous approaches compress vision tokens with external modules and force LLMs to understand the compressed ones, leading to visual information loss. However, the LLMs' understanding paradigm of vision tokens is not fully utilised in the compression learning process. We propose VoCo-LLaMA, the first approach to compress vision tokens using LLMs. By introducing Vision Compression tokens during the vision instruction tuning phase and leveraging attention distillation, our method distill how LLMs comprehend vision tokens into their processing of VoCo tokens. VoCo-LLaMA facilitates effective vision compression and improves the computational efficiency during the inference stage. Specifically, our method achieves minimal performance loss with a compression ratio of 576$\times$, resulting in up to 94.8$\%$ fewer FLOPs and 69.6$\%$ acceleration in inference time. Furthermore, through continuous training using time-series compressed token sequences of video frames, VoCo-LLaMA demonstrates the ability to understand temporal correlations, outperforming previous methods on popular video question-answering benchmarks. Our approach presents a promising way to unlock the full potential of VLMs' contextual window, enabling more scalable multi-modal applications. The project page, along with the associated code, can be accessed via https://yxxxb.github.io/VoCo-LLaMA-page/.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    CoreMatching couples token pruning and neuron pruning in vision-language models by selecting tokens that activate the most core neurons, achieving large inference speedups with minor accuracy loss.

  2. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  3. PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

    cs.MM 2025-07 conditional novelty 6.0 of 10

    Keeping only the first 12 layers of Qwen2-VL plus self-distillation and a modality-aware contrastive loss yields a 3B unified multimodal retriever within 1.8 points of the 7B model on M-BEIR.

  4. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  5. GEM: Empowering LLM for both Embedding Generation and Language Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GEM fine-tunes decoder-only LLMs with bottleneck special tokens and a mixed next-token prediction and contrastive objective, giving them text-embedding ability with only a modest MMLU drop.

  6. VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Fine-tuning Qwen2.5-VL on VisReason, a 489K-example multi-round visual chain-of-thought dataset (165K with pseudo-depth), modestly improves LLM-judged visual reasoning scores, with caveats about self-referential 3D ev...

  7. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

  8. Task-Aware KV Compression For Cost-Effective Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.

  9. Towards General Continuous Memory for Vision-Language Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A vision-language model can act as its own continuous memory encoder, compressing external multimodal knowledge into eight embeddings that improve reasoning when prepended to the frozen model.

  10. UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    UniVG-R1 uses CoT supervised fine-tuning plus GRPO with difficulty-aware reweighting to make Qwen2-VL substantially better at multi-image, reasoning-based visual grounding.

Pith tools