Pith. sign in

REVIEW 4 major objections 6 minor 39 references

This paper proposes that vision-language agents train better when reinforcement learning is augmented with a contrastive loss applied only to the model's own perception tokens, identified by entropy shifts under image perturbation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:02 UTC pith:VHTMKXTR

load-bearing objection CPPO is a solid incremental method that deserves a real review; the token-detection proof is circular and one gating control is missing, but the headline gains over GRPO and PAPO look credible. the 4 major comments →

arxiv 2601.00501 v2 pith:VHTMKXTR submitted 2026-01-01 cs.CV

CPPO: Contrastive Perception Policy Optimization for VLM Agents

Ahmad Rezaei , Mohsen Gholami , Saeed Ranjbar Alvar , Kevin Cannons , Mohammad Asiful Hossain , Zhou Weimin , Yong Zhang , Mohammad Akbari This is my paper
classification cs.CV
keywords reinforcement learningvision-language modelscontrastive learningperception tokensentropy-based detectionvisual groundingmultimodal reasoningpolicy optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CPPO's claim is that vision-language models trained with reinforcement learning benefit from a dedicated, self-supervised signal for visual perception, applied only to the tokens that actually depend on the image. The method detects those tokens by measuring how much the model's output entropy rises when the image is perturbed to destroy information, then applies a contrastive loss that keeps the token distribution stable under harmless image changes and pushes it away from the distribution under information-destroying changes. The loss is gated to rollouts with positive advantage, so only successful reasoning trajectories receive the perception feedback. If the claim holds, VLM agents can improve visual grounding without extra LLM judges, ground-truth reasoning annotations, or forced separation of perception from reasoning, which is what the experiments report.

Core claim

The paper's central discovery is that perception tokens can be identified within a VLM's natural generation using the model's own output distributions, and that a token-level contrastive loss on those tokens improves both perception and reasoning. For each token, the entropy of the predictive distribution is computed under the original image and under an information-removing perturbation; tokens whose entropy increases most are treated as perception tokens. An InfoNCE contrastive loss is then applied to those tokens, with the distribution under the original image as anchor, an information-preserving perturbation as positive sample, and an information-removing perturbation as negative sample.

What carries the argument

The load-bearing mechanism is the entropy-shift perception-token detector combined with the Contrastive Perception Loss (CPL). The detector computes, for each generated token, the entropy difference ΔH between the model's predictive distribution given an information-removing image and given the original image; the top-k tokens by ΔH form the perception set. CPL is an InfoNCE loss over the token's predictive distribution, with the original image's distribution as anchor, an information-preserving perturbation's distribution as the positive sample, and the information-removing perturbation's distribution as the negative sample, averaged over the perception tokens and added to the RL objective

Load-bearing premise

The whole method rests on the assumption that removing image information raises the model's entropy more for perception tokens than for reasoning tokens; if an information-removing perturbation also scrambles reasoning tokens, or fails to disrupt genuinely visual tokens, the contrastive loss lands on the wrong tokens and the reported gains can vanish.

What would settle it

On a controlled dataset with known perception tokens (for example, a visible equation with a distracting background), compute the entropy-shift detector's top-k set and compare against ground truth; if the set excludes clearly image-grounded tokens or includes tokens that are answerable without the image, the criterion fails. Alternatively, apply CPPO with an information-preserving perturbation that actually changes the correct answer and check whether the contrastive loss pushes the token distribution the wrong way.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A perception training signal can be produced at token level with no extra models, no ground-truth chain-of-thought, and no forced output tagging, reducing the cost and complexity of perception-aware RL.
  • Selective application to detected perception tokens avoids over-regularizing reasoning tokens, while advantage gating prevents reinforcing incorrect perception on failed rollouts; ablations confirm each component adds accuracy.
  • On seven math and visual-reasoning benchmarks, CPPO-trained 3B and 7B models average 40.0% and 48.2% accuracy, outperforming the base RL objective (37.8%/46.7%) and a prior perception-loss method (38.1%/46.8%) under the same data and steps.
  • The gain is not just extra compute: training with the contrastive loss for two epochs (39% more time than base RL) beats training the base RL for four epochs (100% more time).
  • The entropy-based token detection tracks human-identified perception tokens in qualitative samples, with ROUGE-1 F1 improving as the top-k threshold grows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The entropy-shift criterion is a general principle: any perturbation that removes task-relevant information could expose which output tokens depend on that information, so the same contrastive framing may transfer to audio, video, or other input modalities.
  • Because CPL is a local, distribution-level constraint, it could be dropped into other policy-gradient or even off-policy objectives, not just the group-relative objective used here; the advantage-gating idea is method-agnostic.
  • A testable extension: vary the perturbation family (e.g., masking specific regions versus global blur) and see whether detection accuracy and downstream accuracy move together; if they decouple, the entropy proxy may be selecting salient tokens rather than genuinely perception-dependent ones.
  • At larger model scales, the entropy-shift signal may weaken as models grow more robust to occlusion, so the top-k fraction may need to become adaptive per rollout rather than fixed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CPPO, an RL finetuning method for VLMs that augments GRPO with a Contrastive Perception Loss (CPL). It selects 'perception tokens' by the entropy increase of the policy's next-token distribution when the input image is corrupted, and applies an InfoNCE loss that pulls the distribution under the original image toward an information-preserving view and away from an information-removing view, gated to trajectories with positive group-relative advantage (Eq. 10). Training on ViRL39K with Qwen2.5-VL-3B/7B, the authors report averaged accuracies of 40.0% and 48.2% on seven benchmarks, above GRPO (37.8/46.7) and PAPO (38.1/46.8), plus ablations on Geometry3K and a compute comparison showing CPPO's 39% extra time beats a 100% GRPO budget.

Significance. If the causal mechanism is as claimed, CPPO is a useful contribution to perception-aware RL for VLMs: it is unsupervised, does not require LLM judges, forced <perception> tags, or CoT annotations, and the token-level contrastive signal is more targeted than PAPO's uniform KL. The compute-parity experiment in Table 7 is a genuine strength—it directly addresses the 'just train longer' alternative. Hyperparameters λ and k were fixed for the held-out benchmark evaluation, so the main comparison is not circular. However, the paper's central causal claim—that the gains come specifically from contrastive learning on detected perception tokens—is not yet established: the supporting ablation is confounded, the theoretical justification is an assumption, and no uncertainty quantification or code is provided. With the missing controls and clarifications, the significance would be solid; in its current form it is conditional.

major comments (4)
  1. [Supplementary Sec. 6, Eq. (13)] Proposition 1 is not proved. Eq. (12) reduces ΔH to MI(o_i,t; I | q,o_<t) − MI(o_i,t; I− | q,o_<t). The proof then asserts Eq. (13), which states that this difference is nonnegative for perception tokens. That inequality is exactly the property the proposition claims to establish; no independent argument or measurement supports it. Since the top-k selection in Eq. (5) and therefore all CPL gradients depend on this criterion, the method's theoretical grounding is currently an assumption. Please either provide a non-circular justification or explicitly frame Eq. (13) as an assumption and validate it directly, e.g., by comparing detected tokens with human/LLM perception annotations as in Sec. 9, not only by ROUGE overlap.
  2. [§4.4, Table 2] The last ablation row is confounded. Moving from '+ CPL on Top-k Perc. Tokens' to '+ Advantage Gating' changes two things at once: the auxiliary loss objective is now multiplied by 1[A_i > 0] and the subset of rollouts that receive it changes. The reported 2.0-point gain could therefore be due to the advantage mask alone, independent of CPL or perception-token selection. A control with the same gate applied to a non-perception auxiliary loss (e.g., identical InfoNCE on randomly selected tokens, or on reasoning tokens) is required. If that control matches 38.6, the statement in §4.2 that CPPO's advantage over PAPO is 'directly' due to the contrastive loss on perception tokens would be unsupported.
  3. [§4.2 and Table 1 caption] The fairness of the main comparison is unclear. The caption says prior methods' released checkpoints were used, but §4.2 states that both CPPO and PAPO were trained under identical conditions (same ViRL39K data and same number of steps). These statements are inconsistent unless PAPO's released checkpoint was produced under exactly the authors' training recipe. If released checkpoints were used, training-data and compute differences (not only the CPL term) could explain part of the gap. Please clarify whether the GRPO/PAPO numbers are from retrained models under identical settings or from public checkpoints, and if the latter, temper the 'identical conditions' claim.
  4. [Table 1] No uncertainty information is reported. Differences between CPPO and the closest baselines are as small as 0.3–0.5 points on individual benchmarks (e.g., 3B MMMU-Pv, 7B MVisionm), and all results are averages over 8 responses at temperature 1.0. Without standard errors over at least a few independent runs, or bootstrap CIs over the 8 samples, the claim of consistent improvement cannot be separated from sampling noise. Please add error bars or significance tests to Table 1 (and ideally to Tables 2–4).
minor comments (6)
  1. [Eq. (5)] Typo: 'to which that receive' should be 'to which the CPL loss is applied.'
  2. [§3.2.2 vs. Algorithm 1] The text says CPL is applied only to perception tokens from 'correct' rollouts, but Eq. (10) and Algorithm 1 use A_i > 0, which is a relative group-advantage condition, not correctness. Please align the terminology.
  3. [Supplementary Sec. 8] Heading says 'vs. GPRO' but the method is GRPO; fix the typo.
  4. [Table 1] Several 7B rows have formatting errors (e.g., '55.845.427.6', '72.2 56.944.8'), making columns hard to read.
  5. [Supplementary Sec. 9] The claim that ROUGE-1 improves 'up to 100%' is confusing because 100% is defined as all tokens with positive ΔH, not all output tokens. Clarify the x-axis definition.
  6. [Availability] The code link is given only as '/githubCode'. For a method paper, a working link to code and checkpoints is essential for replication.

Circularity Check

1 steps flagged

Entropy-shift proof assumes the mutual-information gap it claims to establish; the token-detection justification is circular, though the main benchmark comparison is externally tested.

specific steps
  1. self definitional [Supplementary Sec. 6 (Proof of Proposition 1), Eq. (13)-(14); also Sec. 3.2.1 Eq. (3)]
    "Our main assumption is that the conditional mutual information between perception tokens in o_i and I should be greater than their conditional mutual information with the perturbed image I^-. Formally, if o_i,t is a perception token, we assume the following inequality holds for its conditional mutual information: MI(o_i,t;I|q,o_i,<t) - MI(o_i,t;I^-|q,o_i,<t) >= 0. (13) Substituting this inequality into (12) yields H_i,t(I^-)-H_i,t(I) >= 0. (14). Thus, an increase in predictive entropy, ΔH_i,t, serves as a principled proxy for identifying vision-dependent tokens."

    Equation (12) in the same proof rewrites ΔH exactly as MI(o_i,t;I|q,o_i,<t) - MI(o_i,t;I^-|q,o_i,<t). Proposition 1's conclusion is that this ΔH is >=0 for perception tokens. Eq. (13) assumes exactly that inequality for perception tokens. So Eq. (14) is the assumption restated through an identity, not a derivation. The proof never independently establishes that entropy shift identifies perception tokens; it only names the assumed MI gap as the defining property of a perception token.

full rationale

The clearest circular step is the proof of Proposition 1: the claimed proxy is shown to follow only after assuming the very MI inequality that constitutes the claim. This is a genuine self-definitional reduction in the theoretical justification for perception-token detection. However, the central empirical claim—CPPO's benchmark gains over GRPO and PAPO—is evaluated on external test benchmarks (Table 1) that were not used to fit λ, k, or the perturbation choices; the ablations (Tables 2-4) are on Geometry3K and the main results are on seven disjoint benchmarks. Hyperparameter selection on a validation set is standard and not circular. The self-citations ([3], [9]) are used only as general evidence that VLMs underperform in multimodal reasoning; they are not load-bearing. The missing control for the advantage-gating confound (e.g., a non-perception CPL applied only on positive-advantage rollouts) is a genuine experimental attribution risk, but it is a correctness/ablation concern rather than a circularity of the derivation. Similarly, the use of GPT5-mini in Sec. 9 is external, though acknowledged as imperfect. Overall, one central theoretical component reduces by construction, but the headline benchmark result retains independent empirical content, so the circularity burden is moderate.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The central claim is largely empirical: it rests on GRPO training assumptions, on the perturbation taxonomy, and on the entropy-shift proxy. The proxy is the most fragile load-bearing assumption because its proof assumes the target inequality.

free parameters (5)
  • λ (CPL weight) = 0.02
    Tuned on Geometry3K (Table 4); best at 0.02, performance degrades at 0.01, 0.03, 0.04.
  • k (top-k token ratio) = 50%
    Tuned on Geometry3K (Table 3); 50% beats 5/25/75/100%.
  • τ (InfoNCE temperature) = 0.1
    Set in Table 8; no sensitivity study reported.
  • β (KL penalty weight) = 0.01
    Standard GRPO coefficient; fixed across experiments.
  • perturbation severity = 80% patch mask / 30% retain crop / color jitter 0.2-1.3 etc.
    Perturbation families chosen by hand; validated on four benchmarks in Table 6, not tuned per-task.
axioms (3)
  • ad hoc to paper Perception tokens satisfy MI(o_t; I | q, o_<t) ≥ MI(o_t; I− | q, o_<t)
    Supplementary Eq. (13); this inequality is exactly what Proposition 1 needs to prove and is asserted without independent evidence.
  • domain assumption Information-preserving perturbations retain query-relevant content and information-removing perturbations remove it
    Table 6 supports this empirically on average (≈1% drop vs >14% drop), but it is an assumption about the augmentation suite for all inputs.
  • domain assumption Group-relative advantage A_i > 0 identifies rollouts worth regularizing
    Advantage gating in Eq. (10) assumes positive-advantage rollouts contain correct perception tokens; no direct measure of perception correctness is used.

pith-pipeline@v1.3.0-alltime-deepseek · 15306 in / 12632 out tokens · 122196 ms · 2026-08-03T13:02:41.829865+00:00 · methodology

0 comments
read the original abstract

We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision--language models (VLMs). Reliable perception is a core requirement for VLM-based agents that must reason and act in open-ended environments: faulty visual grounding cascades directly into faulty actions, hallucinated tool calls, and unsafe decisions. While reinforcement learning (RL) has significantly improved reasoning in language models, extending these advances to multimodal agents requires improving both perception and reasoning. Prior works address this challenge mainly through explicit perception rewards, which often require extra LLM judges, ground-truth annotations, or forced separation of perception from reasoning. CPPO addresses this limitation in a self-supervised manner by extending the RL objective with a Contrastive Perception Loss (CPL) that provides a direct learning signal for visual grounding. The contrastive objective encourages the model to become more sensitive to input visual information. To apply this signal effectively, CPPO identifies perception tokens using an entropy-shift mechanism in the model's output distributions under perturbed images and applies the contrastive loss selectively to those tokens during training. Experiments show that CPPO surpasses prior methods while avoiding extra models, making training more efficient and scalable, and yielding policies that are better suited to perception-critical agentic tasks.

Figures

Figures reproduced from arXiv: 2601.00501 by Ahmad Rezaei, Kevin Cannons, Mohammad Akbari, Mohammad Asiful Hossain, Mohsen Gholami, Saeed Ranjbar Alvar, Yong Zhang, Zhou Weimin.

Figure 1
Figure 1. Figure 1: CPPO vs. prior perception-rewarding methods. Prior work follows three strategies: (1) Visionary-R1 and Vision-SR1 force the policy to generate separated perception from reasoning, followed by an LLM perception reward, (2) Perception-R1 uses ground-truth CoT and an LLM as a judge to provide perception reward, and (3) PAPO applies a perception loss to all rollout tokens. In contrast, CPPO uses entropy of the… view at source ↗
Figure 2
Figure 2. Figure 2: An overview of CPPO. For each rollout oi, perception tokens are identified and their probability distributions are com￾puted under three conditions: the original image I (anchor sample: πθ(oi,t)), an information-preserving perturbation I + (positive sample: π + θ (oi,t)), and an information-removing perturbation I − (negative sample: π − θ (oi,t)). Similarities sim πθ(oi,t), π + θ (oi,t)  and sim πθ(oi,t)… view at source ↗
Figure 3
Figure 3. Figure 3: Sample outputs generated with CPPO with top 40% detected perception tokens. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: CPPO vs. GRPO (avg@8) on Qwen2.5-VL-3B across in-domain and out-of-domain scenarios. The X-axis represents RL training [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sample information-removing perturbations and information-preserving perturbations. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Quantitative evaluation of perception token detection. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sample generated responses by CPPO and GRPO. GRPO exhibits a perception error that is corrected in the CPPO response. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sample generated responses by CPPO and GRPO. GRPO exhibits a perception error that is corrected in the CPPO response. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Sample generated responses by CPPO and GRPO. GRPO exhibits a perception error that is corrected in the CPPO response. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 8 linked inside Pith

  1. [1]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    Pith/arXiv arXiv 2025

  2. [2]

    Why reasoning matters? a survey of ad- vancements in multimodal reasoning (v1).arXiv preprint arXiv:2504.03151, 2025

    Jing Bi, Susan Liang, Xiaofei Zhou, Pinxin Liu, Junjia Guo, Yunlong Tang, Luchuan Song, Chao Huang, Guangyu Sun, Jinxi He, et al. Why reasoning matters? a survey of ad- vancements in multimodal reasoning (v1).arXiv preprint arXiv:2504.03151, 2025. 1

    arXiv 2025

  3. [3]

    From segments to scenes: Temporal understand- ing in autonomous driving via vision-language model.arXiv preprint arXiv:2512.05277, 2025

    Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Hei- darikhazaei, Zhou Weimin, Yong Zhang, and Mohammad Akbari. From segments to scenes: Temporal understand- ing in autonomous driving via vision-language model.arXiv preprint arXiv:2512.05277, 2025. 1

    Pith/arXiv arXiv 2025

  4. [4]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PmLR, 2020. 2, 4, 5

    2020

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 13

    2024

  6. [6]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning, 2025. 1, 2, 4

    2025

  7. [7]

    Openvlthinker: Complex vision- language reasoning via iterative sft-rl cycles, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: Complex vision- language reasoning via iterative sft-rl cycles, 2025. 3

    2025

  8. [8]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 11198–11201, 2024. 6

    2024

  9. [9]

    Spatial reasoning with vision-language models in ego-centric multi-view scenes.arXiv preprint arXiv:2509.06266, 2025

    Mohsen Gholami, Ahmad Rezaei, Zhou Weimin, Sitong Mao, Shunbo Zhou, Yong Zhang, and Mohammad Akbari. Spatial reasoning with vision-language models in ego-centric multi-view scenes.arXiv preprint arXiv:2509.06266, 2025. 1

    arXiv 2025

  10. [10]

    Perceptual decoupling for scalable multi-modal reasoning via reward- optimized captioning.arXiv preprint arXiv:2506.04559,

    Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong, Xin Jin, Zhenguo Li, James T Kwok, and Yu Zhang. Perceptual decoupling for scalable multi-modal reasoning via reward- optimized captioning.arXiv preprint arXiv:2506.04559,

    arXiv

  11. [11]

    Decoupled visual interpretation and linguistic reasoning for math problem solving.arXiv preprint arXiv:2505.17609, 2025

    Zixian Guo, Ming Liu, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Decoupled visual interpretation and linguistic reasoning for math problem solving.arXiv preprint arXiv:2505.17609, 2025. 3

    Pith/arXiv arXiv 2025

  12. [12]

    Generalization in re- inforcement learning by soft data augmentation

    Nicklas Hansen and Xiaolong Wang. Generalization in re- inforcement learning by soft data augmentation. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13611–13617. IEEE, 2021. 3

    2021

  13. [13]

    Bradley Knox, and Dorsa Sadigh

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Con- trastive preference learning: Learning from human feedback without reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024. 3

    2024

  14. [14]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025. 3

    2025

  15. [15]

    CURL: Contrastive unsupervised representations for rein- forcement learning

    Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representations for rein- forcement learning. InProceedings of the 37th International Conference on Machine Learning, pages 5639–5650. PMLR,

  16. [16]

    Vision matters: Sim- ple visual perturbations can boost multimodal math reason- ing.arXiv preprint arXiv:2506.09736, 2025

    Yuting Li, Lai Wei, Kaipeng Zheng, Jingyuan Huang, Linghe Kong, Lichao Sun, and Weiran Huang. Vision matters: Sim- ple visual perturbations can boost multimodal math reason- ing.arXiv preprint arXiv:2506.09736, 2025. 1, 3

    arXiv 2025

  17. [17]

    Self-rewarding vision- language model via reasoning decomposition, 2025

    Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhen- wen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd- Graber, Haitao Mi, and Dong Yu. Self-rewarding vision- language model via reasoning decomposition, 2025. 2, 3

    2025

  18. [18]

    Noisyrollout: Reinforcing visual reasoning with data aug- mentation.arXiv preprint arXiv:2504.13055, 2025

    Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data aug- mentation.arXiv preprint arXiv:2504.13055, 2025. 1, 2

    arXiv 2025

  19. [19]

    Inter-GPS: Inter- pretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-GPS: Inter- pretable geometry problem solving with formal language and symbolic reasoning. InProceedings of the 59th An- nual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Natural Language Processing...

    2021

  20. [20]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. InIn- ternational Conference on Learning Representations (ICLR),

  21. [21]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning?, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Hong- gang Zhang. We-math: Does your large multimodal model achieve human-like mathematical reasoning?, 2024. 6, 10

    2024

  22. [22]

    Data-efficient reinforcement learning with self-predictive representations

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. InInternational Conference on Learning Representations,

  23. [23]

    Semi-off-policy reinforcement 17 learning for vision-language slow-thinking reasoning, 2025

    Junhao Shen, Haiteng Zhao, Yuzhe Gu, Songyang Gao, Kuikun Liu, Haian Huang, Jianfei Gao, Dahua Lin, Wen- wei Zhang, and Kai Chen. Semi-off-policy reinforcement 17 learning for vision-language slow-thinking reasoning, 2025. 3

    2025

  24. [24]

    Hybridflow: A flexible and efficient rlhf frame- work.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf frame- work.arXiv preprint arXiv: 2409.19256, 2024. 12

    Pith/arXiv arXiv 2024

  25. [25]

    Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning.arXiv preprint arXiv:2504.08837, 2025

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning.arXiv preprint arXiv:2504.08837, 2025. 3, 6, 11

    Pith/arXiv arXiv 2025

  26. [26]

    Mea- suring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset. InThe Thirty-eight Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track,

  27. [27]

    Perception-aware pol- icy optimization for multimodal reasoning.arXiv preprint arXiv:2507.06448, 2025

    Zhenhailong Wang, Xuehang Guo, Sofia Stoica, Haiyang Xu, Hongru Wang, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, et al. Perception-aware pol- icy optimization for multimodal reasoning.arXiv preprint arXiv:2507.06448, 2025. 1, 2, 3

    Pith/arXiv arXiv 2025

  28. [28]

    Visionary-r1: Mitigating shortcuts in visual reasoning with reinforcement learning, 2025

    Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-r1: Mitigating shortcuts in visual reasoning with reinforcement learning, 2025. 1, 2, 3

    2025

  29. [29]

    Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward.arXiv preprint arXiv:2506.07218, 2025

    Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward.arXiv preprint arXiv:2506.07218, 2025. 2, 3

    arXiv 2025

  30. [30]

    Log- icvista: Multimodal llm logical reasoning benchmark in vi- sual contexts, 2024

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Log- icvista: Multimodal llm logical reasoning benchmark in vi- sual contexts, 2024. 6, 10

    2024

  31. [31]

    Contrastive preference optimization: Push- ing the boundaries of LLM performance in machine transla- tion

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Push- ing the boundaries of LLM performance in machine transla- tion. InProceedings of the 41st International Conference on Machine Learning, pages 55204–55224. PMLR, 2024. 3

    2024

  32. [32]

    Visionthink: Smart and efficient vision lan- guage model via reinforcement learning.arXiv preprint arXiv:2507.13348, 2025

    Senqiao Yang, Junyi Li, Xin Lai, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Visionthink: Smart and efficient vision lan- guage model via reinforcement learning.arXiv preprint arXiv:2507.13348, 2025. 3

    Pith/arXiv arXiv 2025

  33. [33]

    Look-back: Implicit visual re-focusing in mllm reasoning, 2025

    Shuo Yang, Yuwei Niu, Yuyang Liu, Yang Ye, Bin Lin, and Li Yuan. Look-back: Implicit visual re-focusing in mllm reasoning, 2025. 3

    2025

  34. [34]

    R1-onevision: Advancing gen- eralized multimodal reasoning through cross-modal formal- ization.arXiv preprint arXiv:2503.10615, 2025

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing gen- eralized multimodal reasoning through cross-modal formal- ization.arXiv preprint arXiv:2503.10615, 2025. 1

    Pith/arXiv arXiv 2025

  35. [35]

    MMMU-pro: A more robust multi-discipline multi- modal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neu- big. MMMU-pro: A more robust multi-discipline multi- modal understanding benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pa...

    2025

  36. [36]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math prob- lems? InarXiv, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math prob- lems? InarXiv, 2024. 6

    2024

  37. [37]

    Taco:temporal latent action-driven contrastive loss for visual reinforcement learning

    Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daum ´e III, and Furong Huang. Taco:temporal latent action-driven contrastive loss for visual reinforcement learning. InAdvances in Neural Information Processing Systems, pages 48203–48225. Curran Associates, Inc., 2023. 3

    2023

  38. [38]

    Shuffle-r1: Efficient rl framework for multimodal large language models via data-centric dynamic shuffle, 2025

    Linghao Zhu, Yiran Guan, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Bin Qin, Jian Luan, Yuliang Liu, and Xiang Bai. Shuffle-r1: Efficient rl framework for multimodal large language models via data-centric dynamic shuffle, 2025. 3

    2025

  39. [39]

    Dynamath: A dynamic visual bench- mark for evaluating mathematical reasoning robustness of vi- sion language models, 2025

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual bench- mark for evaluating mathematical reasoning robustness of vi- sion language models, 2025. 6 18

    2025