Pith. sign in

REVIEW 7 cited by

DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.15232 v4 pith:SY2BVIZO submitted 2024-05-24 cs.CV cs.CL

Run Luo , Yunshui Li , Longze Chen , Wanwei He , Ting-En Lin , Ziqiang Liu , Lei Zhang , Zikai Song
show 4 more authors
Xiaobo Xia Tongliang Liu Min Yang Binyuan Hui
This is my paper · ORCID
classification cs.CVcs.CL
keywords modelsdeemimagelargediffusionperceptionhigherlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model's resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's visual perception performance to a large extent (e.g., 4% higher on RobustVQA, 6.5% higher on MMVP and 12.8 % higher on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

  2. VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning

    cs.CL 2025-04 conditional novelty 6.0 of 10

    A self-supervised training method that selects and merges instruction-relevant vision tokens, cutting LVLM compute while claiming stable VQA accuracy and better dense perception.

  3. Towards Modality Generalization: A Benchmark and Prospective Analysis

    cs.CV 2024-12 conditional novelty 6.0 of 10

    The paper introduces a benchmark showing that current multimodal and domain-generalization methods perform poorly on unseen modalities, with domain-generalization methods winning in the weak setting but not in the str...

  4. OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

    cs.CL 2025-05 conditional novelty 5.0 of 10

    OmniCharacter is a speech-language role-playing agent that generates character-specific voice responses with low latency, trained on a new 10K-dialogue, 135K-audio dataset of 20 game characters.

  5. Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PRIOR reweights the next-token prediction loss in vision-language pretraining by 1 minus the probability assigned by a text-only reference LLM, and reports consistent benchmark improvements over standard NTP.

  6. LaVin-DiT: Large Vision Diffusion Transformer

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A single diffusion transformer with a spatial-temporal VAE and in-context example pairs unifies over 20 vision tasks, with strong scores on several image benchmarks and uneven evidence on video tasks.

  7. Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.

Pith tools