Pith. sign in

REVIEW 37 cited by

Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.15998 v2 pith:KBR4GOJO submitted 2024-08-28 cs.CV cs.AIcs.LGcs.RO

Min Shi , Fuxiao Liu , Shihao Wang , Shijia Liao , Subhashree Radhakrishnan , Yilin Zhao , De-An Huang , Hongxu Yin
show 8 more authors
Karan Sapra Yaser Yacoob Humphrey Shi Bryan Catanzaro Andrew Tao Jan Kautz Zhiding Yu Guilin Liu
This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.RO
keywords encodersmllmsvisiondesignmixturevisualcomplexeagle
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 37 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Exam for Active Observers

    cs.CV 2026-07 conditional novelty 7.0 of 10

    On a new 17-task benchmark of active visual observation, the best frontier multimodal model solves 10.6% of items and humans solve 96.1%.

  2. LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LLaVA-SP adds six spatial tokens produced by multi-scale cropping or pooling and cross-attention to MLLMs, improving 10/11 benchmarks over LLaVA-1.5 with nearly unchanged latency.

  3. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  4. Hidden in plain sight: VLMs overlook their visual representations

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VLMs perform far worse than their own visual encoders on vision-centric tasks because the language model fails to use accessible visual information and instead follows its language priors.

  5. Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.

  6. Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    F3 adds attention-guided noise to adversarial images so that large vision-language models produce answers that are much closer to their clean-image answers.

  7. AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An agent-driven framework adaptively selects a small subset of benchmark questions for MLLMs, preserving over 90% ranking accuracy with roughly 4-5% of the data.

  8. AIDE: Agentically Improve Visual Language Model with Domain Experts

    cs.CV 2025-02 conditional novelty 6.0 of 10

    AIDE, a four-stage agentic pipeline, uses the VLM itself to select and enrich training examples with expert tool outputs, yielding modest single-digit benchmark gains on Eagle-8B.

  9. QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    QLIP trains a quantized image autoencoder with both reconstruction and text-alignment losses, yielding a tokenizer that supports multimodal understanding and text-to-image generation in one model.

  10. The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering

    cs.CV 2025-02 conditional novelty 6.0 of 10

    VISTA reduces hallucination in vision-language models by adding a per-image visual steering vector to hidden states and blending in early-layer logits, cutting CHAIR object hallucination by about 40%.

  11. Diffusion Instruction Tuning

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Lavender fine-tunes vision-language models by aligning their attention maps with Stable Diffusion's attention targets, improving accuracy on 20 benchmarks with as few as 0.13 million training examples.

  12. Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Using large-scale adversarially pretrained vision encoders in LLaVA yields 2x and 1.5x robustness gains on captioning and VQA, and cuts jailbreak success rates by over 10% relative to CLIP fine-tuning baselines.

  13. LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Co-training an open-vocabulary detector with long, detailed image captions generated by a large vision-language model improves zero-shot detection, especially for rare classes.

  14. Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Eagle2-9B matches or outperforms much larger vision-language models on many benchmarks through a carefully constructed post-training data strategy.

  15. Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Multimodal LLMs show systematic weaknesses in instance-level visual correspondence, and CoLVA, trained with a fine-grained vision expert and object-level contrastive learning, reaches 49.8% accuracy on the new MMVM be...

  16. MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

    cs.CV 2025-01 conditional novelty 6.0 of 10

    MoVE-KD distills multiple visual encoders into one student encoder using mixture-of-LoRA-experts and CLS-attention-guided distillation, improving LLaVA-family VLMs at single-encoder cost.

  17. A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A hybrid human/GPT-4o pipeline plus two new filtering scores produces a 424k-pair text-rich image instruction dataset that improves fine-tuned multimodal models on OCR and document understanding benchmarks.

  18. LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A hierarchical window transformer that injects image details into upsampled CLIP features and compresses them with cross-scale window attention improves MLLM fine-grained perception by 3.7% on average over LLaVA-UHD.

  19. FastVLM: Efficient Vision Encoding for Vision Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A five-stage hybrid encoder that downsamples images 64x produces 4x fewer tokens than prior hybrid designs and 16x fewer than a ViT, giving about a 3x faster first-token time at similar VLM accuracy in Apple's benchmarks.

  20. Apollo: An Exploration of Video Understanding in Large Multimodal Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Design decisions for video-LMMs can be made on 2-4B models and datasets and transfer to larger models, yielding efficient Apollo models, though some SOTA claims are contradicted by the paper's own table.

  21. Olympus: A Universal Task Router for Computer Vision Tasks

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Olympus is a trained MLLM router that delegates 20 vision tasks to specialist models and supports chain-of-action execution of up to five tasks per instruction.

  22. Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Using Florence-2's multi-depth, multi-prompt visual features in an MLLM yields consistent benchmark gains over CLIP-based counterparts, especially on text-heavy tasks.

  23. Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A TD-trained vision value model guides sentence-level inference-time search in VLMs, cutting hallucination and improving caption quality, with self-training gains on nine benchmarks.

  24. Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Concatenating visual tokens from multiple MLLMs in the same family and linearly merging their language model deltas yields a training-free fusion that improves multimodal benchmark scores over each source model.

  25. MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.

  26. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  27. FaceLLM: A Multimodal Large Language Model for Face Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fine-tuning InternVL3 on ChatGPT-generated face QA pairs yields a face-specialized MLLM with the highest reported accuracy among MLLMs on FaceXBench.

  28. Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Manager aggregates multi-layer unimodal representations and improves both two-tower VLMs (ManagerTower) and MLLMs (LLaVA-OV-Manager) on 24 downstream tasks.

  29. Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Splitting the vision encoder of an MLLM into domain-specific experts with a lightweight router yields small benchmark improvements at near-zero extra inference cost.

  30. Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models

    cs.AI 2025-05 reject novelty 5.0 of 10

    A causal intervention architecture with confounder dictionaries is applied to LLaVA, producing modest hallucination reductions on POPE and CHAIR but with methodological caveats.

  31. FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression

    cs.CV 2024-12 conditional novelty 5.0 of 10

    FlashSloth compresses visual input to 90 tokens with attention pooling and an embedded cross-attention query module, achieving 2-5x faster response than tiny MLLM baselines with competitive benchmark scores.

  32. Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings

    cs.CV 2024-11 conditional novelty 5.0 of 10

    DyVTE trains a lightweight hyper-network to detect when text tokens have absorbed enough image information, then removes all visual tokens at that layer, cutting MLLM FLOPs by 20-56% with small accuracy changes.

  33. ChatRex: Taming Multimodal LLM for Joint Perception and Understanding

    cs.CV 2024-11 conditional novelty 5.0 of 10

    ChatRex couples a universal proposal network with an LLM that retrieves box indices, reaching 48.2 mAP on COCO and strong referring and region-level results.

  34. Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Valley2, a 7B-scale open-source multimodal model, reports second-best OpenCompass average (67.4) among sub-10B models and the highest score (79.66) on its own in-house Ecom-VQA benchmark.

  35. YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls

    cs.SD 2024-12 reject novelty 4.0 of 10

    A video-guided sound effects model with a learnable audio-visual aggregator and multi-modal chain-of-thought module reports strong VGGSound benchmark scores, but its few-shot claim rests on three qualitative samples.

  36. Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A survey maps the field of MLLM explainability and interpretability into data, model, and training and inference perspectives.

  37. Do Language Models Understand Time?

    cs.CV 2024-12 conditional novelty 3.0 of 10

    A survey arguing that video-LLMs rely on pretrained encoders and short-biased datasets, leaving them weak at long-term temporal reasoning such as causality and event progression.

Pith tools