Pith. sign in

REVIEW 18 cited by

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11703 v1 pith:WDES6ZZL submitted 2024-03-18 cs.CV cs.AI

Ruyi Xu , Yuan Yao , Zonghao Guo , Junbo Cui , Zanlin Ni , Chunjiang Ge , Tat-Seng Chua , Zhiyuan Liu
show 2 more authors
Maosong Sun Gao Huang
This is my paper
classification cs.CVcs.AI
keywords imagesllava-uhdvisualencodingllava-1lmmsmodelaspect
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Visual encoding constitutes the basis of large multimodal models (LMMs) in understanding the visual world. Conventional LMMs process images in fixed sizes and limited resolutions, while recent explorations in this direction are limited in adaptivity, efficiency, and even correctness. In this work, we first take GPT-4V and LLaVA-1.5 as representative examples and expose systematic flaws rooted in their visual encoding strategy. To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution. LLaVA-UHD includes three key components: (1) An image modularization strategy that divides native-resolution images into smaller variable-sized slices for efficient and extensible encoding, (2) a compression module that further condenses image tokens from visual encoders, and (3) a spatial schema to organize slice tokens for LLMs. Comprehensive experiments show that LLaVA-UHD outperforms established LMMs trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088) resolution images using only 94% inference computation, and achieves 6.4 accuracy improvement on TextVQA. Moreover, the model can be efficiently trained in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of LLaVA-1.5). We make the data and code publicly available at https://github.com/thunlp/LLaVA-UHD.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

    cs.CV 2024-08 conditional novelty 8.0 of 10

    MME-RealWorld is the largest manually annotated high-resolution benchmark for MLLMs, where even the best models achieve less than 60% accuracy on challenging real-world tasks.

  2. PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

    cs.CV 2024-10 accept novelty 7.0 of 10

    PyramidDrop accelerates LVLMs by staged, similarity-based dropping of visual tokens that become redundant in deeper layers, delivering 40% faster training and 55% lower inference cost with comparable accuracy.

  3. VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Counterfactual present/removed teacher views attribute visually supported corrections and reconstruct student-anchored distillation targets that beat source-mixed multimodal OPD.

  4. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  5. When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    STS is a two-stage pruning framework that decouples structural diversity via repulsion sampling from semantic filtering via cross-attention to reduce redundancy in visual tokens for VLMs.

  6. Self-Prophetic Decoding to Unlock Visual Search in LVLMs

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    SeProD is a plug-and-play self-prophetic decoding framework that combines pre- and post-training LVLM capabilities via probability-based sampling to improve coherent visual search and multi-step reasoning.

  7. How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    F^3A is a training-free visual token pruning router that treats pruning as task-conditioned evidence search and allocates a fixed vision token budget using question cues and frozen sparse heads without extra LLM passes.

  8. E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes

    cs.CV 2026-04 conditional novelty 6.0 of 10

    E-VLA integrates event streams directly into VLA models via lightweight fusion, raising Pick-Place success from 0% to 60-90% at 20 lux and from 0% to 20-25% under severe motion blur.

  9. Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models

    cs.CV 2025-12 unverdicted novelty 6.0 of 10

    Visual Funnel resolves contextual blindness in MLLMs by constructing an entropy-scaled portfolio of hierarchically structured image crops that preserves both local detail and global context.

  10. Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs

    cs.CV 2025-05 unverdicted novelty 6.0 of 10

    Chain-of-Focus enables VLMs to adaptively search and zoom on important image areas via a two-stage SFT and RL pipeline on a custom 3K-sample dataset, yielding 5% gains on the V* benchmark across resolutions from 224 to 4K.

  11. Less Detail, Better Answers: Degradation-Driven Prompting for VQA

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    Degradation-Driven Prompting improves VQA by intentionally reducing image detail and using masks, lines, and examples to guide models toward essential structures.

  12. E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes

    cs.CV 2026-04 conditional novelty 5.0 of 10

    Degradation-Driven Prompting improves VLM accuracy on deceptive visual tasks by downsampling inputs and routing them through classification, external visual tools, and a critic.

  13. VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Fine-tuning Qwen2.5-VL on VisReason, a 489K-example multi-round visual chain-of-thought dataset (165K with pseudo-depth), modestly improves LLM-judged visual reasoning scores, with caveats about self-referential 3D ev...

  14. MiniCPM-V: A GPT-4V Level MLLM on Your Phone

    cs.CV 2024-08 conditional novelty 5.0 of 10

    MiniCPM-Llama3-V 2.5 delivers GPT-4V-level multimodal performance on phones through architecture, pretraining, and alignment optimizations.

  15. InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

    cs.CV 2024-07 conditional novelty 5.0 of 10

    InternLM-XComposer-2.5 is a 7B vision-language model supporting up to 96K context that reaches GPT-4V-level performance on image, video, and multi-turn tasks and adds LoRA-driven text-image composition capabilities.

  16. MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

    cs.CV 2026-07 reject novelty 4.0 of 10

    A multi-scale gated fusion plus question-conditioned token router is specified to cut VLM visual tokens ~80% while improving accuracy, but all reported gains are simulated placeholders.

  17. How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

    cs.CV 2024-04 unverdicted novelty 4.0 of 10

    InternVL 1.5 narrows the performance gap to proprietary multimodal models via a stronger transferable vision encoder, dynamic high-resolution tiling, and curated English-Chinese training data.

  18. Toward Native Multimodal Modeling: A Roadmap

    cs.CV 2026-05 unverdicted novelty 3.0 of 10

    A roadmap that defines architectural nativity for multimodal models and categorizes them into Multi-to-Text, Multi-to-Target, and Multi-to-Multi types while outlining an industrial pipeline toward unified transformer-...

Pith tools