Pith. sign in

REVIEW 24 cited by

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11839 v2 pith:UPQ23KJT submitted 2024-06-17 cs.CV cs.AIcs.CLcs.LG

Fei Wang , Wenxuan Zhou , James Y. Huang , Nan Xu , Sheng Zhang , Hoifung Poon , Muhao Chen This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords preferencemultimodaloptimizationproblemmdpomodelimagelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. Through a comparative experiment, we identify the unconditional preference problem in multimodal preference optimization, where the model overlooks the image condition. To address this problem, we propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. Moreover, we introduce a reward anchor that forces the reward to be positive for chosen responses, thereby avoiding the decrease in their likelihood -- an intrinsic problem of relative preference optimization. Experiments on two multimodal LLMs of different sizes and three widely used benchmarks demonstrate that mDPO effectively addresses the unconditional preference problem in multimodal preference optimization and significantly improves model performance, particularly in reducing hallucination.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. LPOI: Listwise Preference Optimization for Vision Language Models

    cs.CV 2025-05 conditional novelty 7.0 of 10

    LPOI reduces VLM hallucination by training the model to prefer the original image over progressively masked versions of the same image, using a listwise ranking loss built from pairwise preference data.

  3. Probing Visual Language Priors in VLMs

    cs.CV 2024-12 conditional novelty 7.0 of 10

    ViLP shows that vision-language models often answer from text priors instead of image content, and an image-corruption DPO method partially fixes this.

  4. Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SPIRAL aligns a model's rendered-image input behavior to its native-text input behavior using self-generated text-path supervision, nearly closing the vision-text compression gap on VTCBench.

  5. Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Groc-PO applies preference optimization at three grounded stages — object grounding, contextual grounding, grounded reasoning — and outperforms final-answer-only DPO on hallucination and complex-reasoning benchmarks.

  6. Adaptive Perturbation Selection for Contrastive Audio Decoding

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    A learned per-example router over a 105-perturbation audio library improves contrastive decoding for audio-LLM hallucination, with task-dependent best distortions (e.g., reverse audio for temporal order).

  7. Controlling Multimodal LLMs via Reward-guided Decoding

    cs.CV 2025-08 conditional novelty 6.0 of 10

    MRGD guides MLLM decoding with a learned hallucination reward and a detector-based recall reward, allowing users to trade off object precision, recall, and test-time compute while reducing object hallucinations on CHA...

  8. Explicit Preference Optimization: No Need for an Implicit Reward Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.

  9. ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.

  10. AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization

    cs.CV 2025-04 conditional novelty 6.0 of 10

    AdaViP reduces object hallucination in multimodal LLMs by training on image-edited preference pairs and adaptively weighting vision and language preferences.

  11. Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilization

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Mol-LLM combines SELFIES text, 2D molecular graphs, and preference optimization to build a generalist chemistry LLM that outperforms prior generalist models on most property, reaction, and translation benchmarks.

  12. CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

    cs.CL 2025-01 conditional novelty 6.0 of 10

    CHiP adds image-level and phrase/token-level preference signals to DPO for multimodal LLMs, and the authors report large hallucination-rate reductions on Object HalBench, AMBER, MMHal, and HallusionBench.

  13. Enhancing Audiovisual Speech Recognition through Bifocal Preference Optimization

    eess.AS 2024-12 conditional novelty 6.0 of 10

    A preference optimization objective with input-side and output-side preference pairs improves audiovisual ASR word error rates on How2, VisSpeech, and Ego4D beyond prior state-of-the-art.

  14. VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    VisDoMRAG runs separate text and image retrieval and reasoning streams, fuses them with an LLM consistency check, and the authors release a multi-document benchmark with visually rich content.

  15. MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization

    cs.CV 2024-12 conditional novelty 6.0 of 10

    MMedPO weights preference-optimization training samples by clinical relevance scores, combining hallucinated text answers and locally noised lesion images, and reports improved medical VQA and report generation metrics.

  16. T-REG: Preference Optimization with Token-Level Reward Regularization

    cs.CL 2024-12 conditional novelty 6.0 of 10

    T-REG adds contrastive-prompt self-generated token-level rewards as a weighted regularizer to DPO and SimPO, improving Alpaca Eval 2 length-controlled win rate by up to 3.8% and Arena-Hard win rate by up to 4.4% over ...

  17. SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

    cs.CV 2024-11 conditional novelty 6.0 of 10

    SymDPO replaces answer text in multimodal demonstrations with meaningless symbols during preference training, forcing models to use image context and improving in-context learning performance on five benchmarks.

  18. MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Multi-domain RLVR data mixing, guided by a quadratic surrogate fitted to 11 pilot runs, improves a Qwen2-VL-2B model's out-of-distribution accuracy by about 5 points over uniform mixing.

  19. PerPO: Perceptual Preference Optimization via Discriminative Rewarding

    cs.AI 2025-02 conditional novelty 5.0 of 10

    PerPO trains multimodal LLMs by ranking their candidate answers with deterministic visual rewards (IoU, edit distance) and using the reward differences as margins in listwise preference optimization.

  20. DAMA: Data- and Model-aware Alignment of Multi-modal LLMs

    cs.CV 2025-02 conditional novelty 5.0 of 10

    DAMA scales the DPO penalty per example using CLIP-derived data hardness and current model reward gaps, and is reported to improve hallucination and general benchmarks for LLaVA-1.5 7B and 13B.

  21. MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 4.0 of 10

    MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.

  22. DPO Learning with LLMs-Judge Signal for Computer Use Agents

    cs.AI 2025-06 conditional novelty 4.0 of 10

    An LLM-as-Judge pipeline that scores synthetic GUI interaction trajectories and fine-tunes a 2B model with DPO yields a local computer-use agent that beats its base model on 15-step OSWorld tasks.

  23. Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

    cs.CL 2025-05 conditional novelty 3.0 of 10

    A single-author survey that taxonomizes reward-model design and learning strategies across training, inference, and post-inference for LLMs.

  24. ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning

    cs.CL 2025-05 reject novelty 2.0 of 10

    ASPO's adaptive sentence-level loss, by the paper's own definitions, reduces exactly to the standard DPO loss, leaving no difference in the optimization objective.

Pith tools