REVIEW 21 cited by
From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Multi-modal Large Language Models (MLLMs) have made significant strides in expanding the capabilities of Large Language Models (LLMs) through the incorporation of visual perception interfaces. Despite the emergence of exciting applications and the availability of diverse instruction tuning data, existing approaches often rely on CLIP or its variants as the visual branch, and merely extract features from the deep layers. However, these methods lack a comprehensive analysis of the visual encoders in MLLMs. In this paper, we conduct an extensive investigation into the effectiveness of different vision encoders within MLLMs. Our findings reveal that the shallow layer features of CLIP offer particular advantages for fine-grained tasks such as grounding and region understanding. Surprisingly, the vision-only model DINO, which is not pretrained with text-image alignment, demonstrates promising performance as a visual branch within MLLMs. By simply equipping it with an MLP layer for alignment, DINO surpasses CLIP in fine-grained related perception tasks. Building upon these observations, we propose a simple yet effective feature merging strategy, named COMM, that integrates CLIP and DINO with Multi-level features Merging, to enhance the visual capabilities of MLLMs. We evaluate COMM through comprehensive experiments on a wide range of benchmarks, including image captioning, visual question answering, visual grounding, and object hallucination. Experimental results demonstrate the superior performance of COMM compared to existing methods, showcasing its enhanced visual capabilities within MLLMs.
Forward citations
Cited by 21 Pith papers
-
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
SSVAL uses visual anchor prompts plus spatial and frequency-domain alignment losses to reduce visual information loss in MLLMs and improve fine-grained perception benchmarks.
-
InCoM: Intent-Driven Perception and Structured Coordination for Mobile Manipulation
InCoM reports 23–28 percentage-point success-rate gains in mobile manipulation benchmarks by dynamically reweighting multi-scale perception via inferred motion intent and decoupling base-arm action generation with flo...
-
Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).
-
Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference
Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.
-
Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models
Language models accept misleading endorsements more readily from higher-expertise persona sources, and the effect can be partially steered away by subtracting a residual-stream vector.
-
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
A 1,680-question video benchmark shows leading multimodal models lag humans by ~15 points on visual knowledge, and a See-Think-Answer RL-trained model narrows the gap.
-
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
A new hypothesis test for binary LLM responses treats semantically equivalent query perturbations as an unknown null set and gives asymptotic validity and consistency guarantees under a uniformity assumption.
-
TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bank
TMUAD detects both structural and logical anomalies by comparing a query image's textual object descriptions, object crops, and image patches against three normal memory banks, reaching state-of-the-art AUROC on seven...
-
Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection
A decoding pipeline reuses cached tokens and anneals sampling temperature to accelerate self-consistency hallucination detection by up to 3x without meaningful AUROC loss.
-
CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
A method that uses a handful of 'semantic retrieval heads' instead of all attention heads to decide which key-value cache entries can be dropped, plus layer-wise cache budgeting, reportedly beats prior KV compression ...
-
METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.
-
Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor
Stable Diffusion features, especially when conditioned on the question, improve vision-centric multimodal question answering when fused with CLIP.
-
Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.
-
Diffusion Instruction Tuning
Lavender fine-tunes vision-language models by aligning their attention maps with Stable Diffusion's attention targets, improving accuracy on 20 benchmarks with as few as 0.13 million training examples.
-
CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology
A single 15B pathology model unifies patch-level and whole-slide tasks and reports state-of-the-art results on 39 of 42 benchmark datasets.
-
Libra: Leveraging Temporal Images for Biomedical Radiology Analysis
Libra introduces a Temporal Alignment Connector for multimodal LLMs that fuses current and prior chest X-ray features and reports improved radiology report generation on MIMIC-CXR.
-
Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
A three-stage training method that aligns object texts, coordinates, and cropped images, plus a GPT-4V-based data synthesis pipeline, lets 1.5B and 3B models match or exceed larger MLLMs on grounding and VQA benchmarks.
-
Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models
I-FAS reformulates face anti-spoofing as interpretable VQA with spoof-aware captions, a lopsided language-model loss, and a globally aware connector, achieving state-of-the-art cross-domain results.
-
ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
A new eight-task benchmark with in-domain metrics KGI and KPI reveals that existing multimodal editing methods degrade on related samples, and the proposed HICE method achieves a better balance.
-
VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding
VidHalluc is a 5,002-video paired benchmark for action, temporal sequence, and scene transition hallucinations in video MLLMs, and DINO-HEAL is a training-free saliency reweighting method that improves hallucination s...
-
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
HEAL evaluates preference optimization by measuring ranking accuracy and strength correlation between model likelihoods and proxy reward scores over multi-response hypothesis spaces.
Discussion (0). Continue with ORCID to comment.