Pith. sign in

REVIEW 11 cited by

InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.12368 v2 pith:UJ3YAYQK submitted 2025-01-21 cs.CV cs.CL

Yuhang Zang , Xiaoyi Dong , Pan Zhang , Yuhang Cao , Ziyu Liu , Shengyuan Ding , Shenxi Wu , Yubo Ma
show 5 more authors
Haodong Duan Wenwei Zhang Kai Chen Dahua Lin Jiaqi Wang
This is my paper
classification cs.CVcs.CL
keywords rewardixc-2multi-modalmodelinstructionlvlmsmodelstraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DRM: Diffusion-based Reward Model With Step-wise Guidance

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    DRM turns a pre-trained diffusion model into a step-wise reward model and uses it for dense RL training (Step-wise GRPO) and guided sampling to improve final image quality.

  2. Visual-ERM: Reward Modeling for Visual Equivalence

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Visual-ERM is a new multimodal reward model that supplies fine-grained visual feedback for training vision-language models on chart-to-code, table, and SVG tasks, yielding measurable gains over prior rewards.

  3. Unified Reward Model for Multimodal Understanding and Generation

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    UnifiedReward is the first unified reward model that jointly assesses multimodal understanding and generation to provide better preference signals for aligning vision models via DPO.

  4. REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    REVA-PO stabilizes GRPO-style RL for CXR report generation via response-level adaptive KL weights and validation-anchored policy resets, reporting new SOTA BLEU and clinical F1 scores.

  5. AdsQA: Towards Advertisement Video Understanding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    AdsQA adds an ad-video question-answering benchmark and ReAd-R, a GRPO-trained model that beats 7B baselines but not larger closed models.

  6. LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

    cs.CV 2025-08 conditional novelty 6.0 of 10

    RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...

  7. VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model

    cs.CV 2025-04 unverdicted novelty 6.0 of 10

    VLM-R1 applies R1-style RL using rule-based rewards on visual tasks with clear ground truth to achieve competitive performance and superior generalization over SFT in vision-language models.

  8. Visual-RFT: Visual Reinforcement Fine-Tuning

    cs.CV 2025-03 conditional novelty 6.0 of 10

    Visual-RFT applies reinforcement learning with verifiable perception rewards to improve large vision-language models on fine-grained classification, few-shot detection, and grounding tasks.

  9. LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

    cs.CV 2024-11 unverdicted novelty 6.0 of 10

    LLaVA-CoT adds autonomous multistage reasoning to vision-language models, delivering 9.4% gains over its base model and outperforming larger models like Gemini-1.5-pro on reasoning benchmarks via a 100k annotated data...

  10. DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Creates DriveReward dataset with counterfactual annotations and a 1B VLM reward model that outperforms larger VLMs on driving tasks and matches rule-based rewards in RL and trajectory scoring.

  11. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools