EgoPolice introduces a 185-hour annotated police body-worn camera benchmark showing state-of-the-art video models fail on high-stakes actions due to motion, occlusion, and low inter-class visual separability.
hub
Llava-mini: Efficient image and video large mul- timodal models with one vision token
15 Pith papers cite this work, alongside 1 external citations. Polarity classification is still indexing.
15
Pith papers citing it
1
external citations · Pith
abstract
The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision tokens (continuous representations) and integrate them and textual instructions into the context of large language models (LLMs), where large-scale parameters and numerous context tokens (predominantly vision tokens) result in substantial computational overhead. Previous efforts towards efficient LMMs always focus on replacing the LLM backbone with smaller models, while neglecting the crucial issue of token quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal vision tokens. To achieve a high compression ratio of vision tokens while preserving visual information, we first analyze how LMMs understand vision tokens and find that most vision tokens only play a crucial role in the early layers of LLM backbone, where they mainly fuse visual information into text tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to fuse visual information into text tokens in advance, thereby facilitating the extreme compression of vision tokens fed to LLM backbone into one token. LLaVA-Mini is a unified large multimodal model that can support the understanding of images, high-resolution images, and videos in an efficient manner. Experiments across 11 image-based and 7 video-based benchmarks demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by 77%, deliver low-latency responses within 40 milliseconds, and process over 10,000 frames of video on the GPU hardware with 24GB of memory.
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
LookWhen factorizes video recognition into learning when, where, and what to compute via uniqueness-based token selection and dual-teacher distillation, achieving better accuracy-FLOPs trade-offs than baselines on multiple datasets.
MS-Resampler deploys multiple scope-specific resamplers with explicit spatial priors and adaptive fusion to outperform single-scope global cross-attention in MLLMs on ten benchmarks with minimal added cost.
CIVIC is a path-consistent compact visual inference framework that reduces KV-cache memory to approximately one-third and end-to-end latency in VLMs while preserving accuracy via text-aligned KL distillation and adaptive spatial retention.
SPpruner reduces visual tokens in VLMs via focus identification followed by context-aware scanning, retaining 22.2% tokens for 2.53x speedup on Qwen2.5-VL with negligible accuracy loss.
OProver-32B achieves top Pass@32 scores on MiniF2F, ProverBench, and PutnamBench by combining continued pretraining with iterative agentic proving, retrieval, SFT on repairs, and RL on unresolved cases using a 6.86M-proof dataset.
VisMMoE exploits visual-expert affinity via token pruning to achieve up to 2.68x faster VL-MoE inference on memory-constrained hardware while keeping accuracy competitive.
Geo3DPruner uses geometry-aware global attention and two-stage voxel pruning to remove 90% of visual tokens from spatial videos while keeping over 90% of original performance on 3D scene benchmarks.
POINTS-Long is a dual-mode multimodal large language model that uses dynamic visual token scaling to retain 97.7-99.7% accuracy on long-form tasks with 1/40 to 1/10th the tokens and supports streaming via detachable KV-cache.
Training-free SVD leverage-score pruning of vision tokens in VLMs outperforms attention-based methods at extreme budgets of 16–32 tokens.
Fourier Compressor uses FFT to remove frequency-domain redundancy from visual tokens in VLMs, retaining over 96% accuracy with up to 83.8% FLOP reduction.
ELVA uses rule-based RL rewards to rank negatives by similarity, reducing grain blindness in universal multimodal retrieval and reporting a 13.1% gain on a new multi-grain benchmark.
Visual tokens enter VLMs as raw signals and are reshaped differently by in-context versus layer-injection paradigms, each capturing distinct frequency characteristics that drive task performance.
VEN-VL introduces an enrich-then-compact visual ensemble MoE approach claiming superior performance-efficiency trade-off in multimodal tasks using fewer condensed visual tokens.
citing papers explorer
-
EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footage
EgoPolice introduces a 185-hour annotated police body-worn camera benchmark showing state-of-the-art video models fail on high-stakes actions due to motion, occlusion, and low inter-class visual separability.
-
LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute
LookWhen factorizes video recognition into learning when, where, and what to compute via uniqueness-based token selection and dual-teacher distillation, achieving better accuracy-FLOPs trade-offs than baselines on multiple datasets.
-
MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs
MS-Resampler deploys multiple scope-specific resamplers with explicit spatial priors and adaptive fusion to outperform single-scope global cross-attention in MLLMs on ten benchmarks with minimal added cost.
-
CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Models
CIVIC is a path-consistent compact visual inference framework that reduces KV-cache memory to approximately one-third and end-to-end latency in VLMs while preserving accuracy via text-aligned KL distillation and adaptive spatial retention.
-
Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
SPpruner reduces visual tokens in VLMs via focus identification followed by context-aware scanning, retaining 22.2% tokens for 2.53x speedup on Qwen2.5-VL with negligible accuracy loss.
-
OProver: A Unified Framework for Agentic Formal Theorem Proving
OProver-32B achieves top Pass@32 scores on MiniF2F, ProverBench, and PutnamBench by combining continued pretraining with iterative agentic proving, retrieval, SFT on repairs, and RL on unresolved cases using a 6.86M-proof dataset.
-
VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
VisMMoE exploits visual-expert affinity via token pruning to achieve up to 2.68x faster VL-MoE inference on memory-constrained hardware while keeping accuracy competitive.
-
Geometry-Guided 3D Visual Token Pruning for Video-Language Models
Geo3DPruner uses geometry-aware global attention and two-stage voxel pruning to remove 90% of visual tokens from spatial videos while keeping over 90% of original performance on 3D scene benchmarks.
-
POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs
POINTS-Long is a dual-mode multimodal large language model that uses dynamic visual token scaling to retain 97.7-99.7% accuracy on long-form tasks with 1/40 to 1/10th the tokens and supports streaming via detachable KV-cache.
-
Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models
Training-free SVD leverage-score pruning of vision tokens in VLMs outperforms attention-based methods at extreme budgets of 16–32 tokens.
-
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
Fourier Compressor uses FFT to remove frequency-domain redundancy from visual tokens in VLMs, retaining over 96% accuracy with up to 83.8% FLOP reduction.
-
ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval
ELVA uses rule-based RL rewards to rank negatives by similarity, reducing grain blindness in universal multimodal retrieval and reporting a 13.1% gain on a new multi-grain benchmark.
-
The Hidden Evolution of Disguised Visual Context inside the VLM
Visual tokens enter VLMs as raw signals and are reshaped differently by in-context versus layer-injection paradigms, each capturing distinct frequency characteristics that drive task performance.
-
VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding
VEN-VL introduces an enrich-then-compact visual ensemble MoE approach claiming superior performance-efficiency trade-off in multimodal tasks using fewer condensed visual tokens.
- Synthetic Homes: A Multimodal Generative AI Pipeline for Residential Building Data Generation under Data Scarcity