Pith. sign in

REVIEW 4 major objections 4 minor 50 references

Perturbing images with entropy-guided adversarial noise during RL sampling improves visual reasoning in VLMs by around 2% on math benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:07 UTC pith:SRURYCFQ

load-bearing objection Entropy-guided adversarial sampling is a new and plausible way to improve GRPO for visual reasoning, but the paper never demonstrates the attack actually raises response entropy. the 4 major comments →

arxiv 2512.10414 v2 pith:SRURYCFQ submitted 2025-12-11 cs.AI

Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention

Yang Yu , Zhuangzhuang Chen , Lanqing Li , Xiaomeng Li This is my paper
classification cs.AI
keywords reinforcement learningvision-language modelsentropy interventionadversarial attackGRPOpolicy explorationvisual reasoningtoken selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the standard reinforcement-learning fine-tuning method GRPO for vision-language models suffers from collapsing policy entropy, which limits exploration. Instead of intervening during policy optimization as prior work does, the authors intervene during RL sampling by adding small adversarial perturbations to the input image, chosen to increase the entropy of the model's sampled responses. They show that this entropy-guided adversarial sampling boosts accuracy on both in-domain math datasets and out-of-domain visual reasoning benchmarks. A careful reader would care because it suggests a lightweight, sampling-side alternative to entropy-controlling objectives that leaves the policy loss unchanged.

Core claim

The paper introduces SaEI (Selective-adversarial Entropy Intervention), a method that maintains exploratory behavior in GRPO by attacking the visual input with the gradient of the average token-wise entropy of sampled responses. Clean-image responses and adversarial-image responses are mixed into one group for advantage estimation and policy update. A token-selective variant recomputes the entropy only on tokens with moderate entropy, avoiding both low-entropy factual tokens and high-entropy already-explorative tokens. Experiments report SaEI outperforms vanilla GRPO by 2.00% on MM-Eureka (64.45 vs 62.45) and 2.16% on Geometry3K (56.18 vs 54.02), with the best out-of-domain average across th

What carries the argument

EgAS (Entropy-Guided Adversarial Sampling) formulates the entropy of sampled responses as an adversarial objective and uses projected gradient descent to perturb the image, thereby increasing policy entropy during sampling. TsEC (Token-Selective Entropy Computation) filters tokens by entropy rank, keeping only the middle third, so the adversarial attack targets moderately uncertain tokens that blend continuation and direction. Together they intervene in entropy at the sampling stage rather than the policy-update stage, which is the paper's claimed novelty.

Load-bearing premise

The objective in Equation (6) evaluates the current policy's likelihood on clean images for both clean- and adversarial-image responses, while the old policy's denominator uses the actual sampling input; this ratio is not a true importance weight, and the paper does not formally justify that the surrogate is valid for optimizing on the mixed sampling distribution.

What would settle it

Train two GRPO policies on the same data—one with SaEI and one with a control that adds random Gaussian noise matched in L2 norm to the adversarial perturbation. If the random-noise control achieves the same or greater improvement in response diversity and accuracy, then SamEI's gains do not come from entropy-guided exploration but merely from perturbation, undermining the claimed mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If SaEI is correct, entropy collapse in GRPO can be mitigated without altering the policy optimization objective, decoupling exploration-promotion from the loss.
  • A single-step adversarial attack (T=1) suffices for the gains, making the method computationally affordable, while two steps already cause instability.
  • The token-selective principle implies that not all tokens should be equally targeted for entropy intervention; moderate-entropy tokens carry the useful exploratory signal.
  • Better out-of-domain generalization suggests that sampling-time input perturbation can transfer reasoning improvements beyond the training distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A broader implication is that any policy-gradient method could benefit from entropy-guided input perturbation, not just GRPO, since the entropy objective is defined on responses and does not depend on the advantage estimation scheme.
  • The paper's logic suggests an easy testable extension: applying the same adversarial entropy attack to other modalities (e.g., audio or structured inputs) and measuring whether exploration increases there too.
  • If the off-policy ratio issue is real, a corrected formulation that treats adversarial responses as off-policy data could reveal whether the gains come from entropy-driven exploration or from implicit behavior cloning of diverse rollouts.
  • The reported training-stability advantage over KL-Cov hints that sampling-side intervention may avoid the variance spikes seen with token-update restriction, but this claim needs direct variance analysis beyond the reported standard deviations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SaEI, a method that intervenes in policy entropy during the RL sampling phase of GRPO rather than during policy optimization. SaEI first computes the entropy of responses sampled from the current policy on clean visual inputs, then uses a PGD-style attack to perturb the image so as to increase this entropy, and finally mixes responses sampled from the clean and adversarial images to form the GRPO group. A token-selective entropy computation (TsEC) restricts the entropy objective to tokens of moderate entropy, motivated by prior work on the roles of high- and low-entropy tokens. Experiments on Geometry3K and MM-Eureka report consistent in-domain gains over vanilla GRPO and several baselines, together with improved out-of-domain averages, and ablations examine the contribution of each component and compare with random noise.

Significance. If the causal mechanism claimed by the paper were verified, the contribution would be an interesting and simple sampling-side intervention that complements existing optimization-side entropy-control methods. The empirical package is in many ways solid: three seeds with standard deviations, ablations of both components, a random-noise control, out-of-domain evaluation on four benchmarks, and a check at a smaller group size. The reported gains are modest but consistent. However, the central claim — that adversarial entropy maximization is what drives the gains — is not directly supported by the measurements in the paper. The reported evidence is downstream accuracy; no experiment shows that the adversarial image actually increases the entropy of the rollout distribution. Moreover, the gradient computation in Eq. (4) is underspecified, and the objective in Eq. (6) is an off-policy mismatch that is neither derived nor tested separately. The paper is therefore best read at present as an empirical data-augmentation heuristic whose mechanism remains unsubstantiated.

major comments (4)
  1. [§4.2, Eq. (4), Fig. 2]
  2. [§4.2, Eq. (3)–(4)]
  3. [§4.2, Eq. (6)]
  4. [§5.3, Fig. 4]
minor comments (4)
  1. [§5.3]
  2. [§5.5, Fig. 6]
  3. [Eq. (2)]
  4. [§5.4]

Circularity Check

0 steps flagged

No significant circularity: SaEI's accuracy gains are measured on external benchmarks and its entropy intervention is an empirically tested mechanism, not a fitted input renamed as a prediction.

full rationale

The paper's derivation chain is not circular. SaEI defines policy entropy in Eq. (3), uses its gradient to generate adversarial images in Eq. (4), and trains with the mixed-sample objective in Eq. (6). The reported target is benchmark accuracy, which is external to the method and compared against vanilla GRPO, NoisyRollout, and KL-Cov. No fitted parameter is folded into a predicted quantity: hyperparameters such as α, T, and group sizes are fixed before evaluation. The entropy curves in Fig. 2 are an observed effect of the intervention on training dynamics, not a restatement of the attack objective, since the attack targets the old policy's sampling distribution while the entropy curves track the training policy. The main risks highlighted by the skeptic—that Eq. (4) may not be an unbiased gradient of true response entropy, and that the causal link between adversarial entropy and accuracy is not directly measured—are correctness/evidence concerns, not circularity. The missing appendix for prompts is a completeness issue, not a circular step. The citation to Wang et al. [36] for token-entropy patterns is external and not load-bearing through self-citation. There is a mild self-referential flavor in using the entropy quantity both as an intervention target and as an explanatory metric, but because accuracy is measured externally this does not constitute a definitional reduction. Thus no circular step is exhibited, and the score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or architectural entities. The free parameters are the adversarial step size, the one-step attack choice, and the entropy trisection thresholds. The most important hidden premise is the off-policy ratio in Eq. (6); if that premise fails, the method is not a pure GRPO variant. The entropy-improves-exploration assumption is inherited from the literature and is the conceptual engine of the paper.

free parameters (3)
  • Adversarial step size alpha = -2/255 (MM-Eureka), -3/255 (Geometry3K)
    Chosen separately per dataset in Sec. 5.3; no search procedure or sensitivity analysis reported beyond the random-noise comparison.
  • PGD attack iteration count T = 1 (T=2 discarded)
    Set to 1 because T=2 caused training instability after 30 steps (Sec. 5.5, Fig. 8). This is a hand-chosen hyperparameter.
  • TsEC entropy-rank split thresholds = bottom 1/3 and top 1/3 discarded
    The trisection of tokens by entropy in Eq. 7 is chosen ad hoc in Sec. 4.3; only the three-way grouping is ablated, not the exact thresholds.
axioms (4)
  • domain assumption Higher policy entropy during GRPO sampling improves exploration and downstream reasoning performance.
    Invoked from prior entropy-collapse works [7, 44] in Sec. 1 and 4.1. It is not proven in this paper and is the causal backbone of the method.
  • domain assumption The gradient of sampled-response entropy with respect to the visual input is a valid and effective steering signal for exploration.
    Core mechanism of EgAS in Sec. 4.2, Eq. (4). Requires differentiability through stochastic sampling and the vision encoder, and assumes one small PGD step changes exploration without destroying useful image content.
  • ad hoc to paper Tokens can be trisected by entropy into low, moderate, and high groups with distinct functional roles, and using only the moderate group is optimal.
    Based on Wang et al. [36] and operationalized as equal thirds in Sec. 4.3, Eq. (7). The exact split and the claim that high-entropy tokens need no intervention are not independently derived here.
  • ad hoc to paper The off-policy ratio in Eq. (6), with current-policy likelihood on clean images and old-policy denominator on clean or adversarial images, is a valid objective for the mixed sample.
    Stated but not derived in Sec. 4.2. The numerator and denominator condition on different inputs for adversarial responses, so this is not the standard GRPO importance weight.

pith-pipeline@v1.3.0-alltime-deepseek · 13877 in / 9713 out tokens · 95733 ms · 2026-08-03T17:07:52.669880+00:00 · methodology

0 comments
read the original abstract

Recently, reinforcement learning (RL) has become a common choice in enhancing the reasoning capabilities of vision-language models (VLMs). Considering existing RL-based finetuning methods, entropy intervention turns out to be an effective way to benefit exploratory ability, thereby improving policy performance. Notably, most existing studies intervene in entropy by simply controlling the update of specific tokens during policy optimization of RL. They ignore the entropy intervention during the RL sampling that can boost the performance of GRPO by improving the diversity of responses. In this paper, we propose Selective-adversarial Entropy Intervention, namely SaEI, which enhances policy entropy by distorting the visual input with the token-selective adversarial objective coming from the entropy of sampled responses. Specifically, we first propose entropy-guided adversarial sampling (EgAS) that formulates the entropy of sampled responses as an adversarial objective. Then, the corresponding adversarial gradient can be used to attack the visual input for producing adversarial samples, allowing the policy model to explore a larger answer space during RL sampling. Then, we propose token-selective entropy computation (TsEC) to maximize the effectiveness of adversarial attack in EgAS without distorting factual knowledge within VLMs. Extensive experiments on both in-domain and out-of-domain datasets show that our proposed method can greatly improve policy exploration via entropy intervention, to boost reasoning capabilities. Code will be released once the paper is accepted.

Figures

Figures reproduced from arXiv: 2512.10414 by Lanqing Li, Xiaomeng Li, Yang Yu, ZhuangZhuang Chen.

Figure 1
Figure 1. Figure 1: (a). Existing methods intervene in policy entropy from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of our SaEI and vanilla GRPO in terms [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our proposed Selective adversarial Entropy Intervention (SaEI). It consists of entropy-guided adversarial sampling [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of an original image, its adversarial ver [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison with random noise. We add multiple-step [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: An example of a visual question-answer pair. The sample comes from HallusionBench, an OOD benchmark. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablation on iteration number of adversarial attack. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 30 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    Generat- ing natural language adversarial examples.arXiv preprint arXiv:1804.07998, 2018

    Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo- Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generat- ing natural language adversarial examples.arXiv preprint arXiv:1804.07998, 2018. 4

    Pith/arXiv arXiv 2018

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 2

    Pith/arXiv arXiv 2025

  4. [4]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gersten- berger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI confer- ence on artificial intelligence, pages 17682–17690, 2024. 2

    2024

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024. 2

    Pith/arXiv arXiv 2024

  6. [6]

    Sft memorizes, rl generalizes: A compara- tive study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A compara- tive study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025. 2

    Pith/arXiv arXiv 2025

  7. [7]

    The entropy mechanism of reinforce- ment learning for reasoning language models.arXiv preprint arXiv:2505.22617, 2025

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforce- ment learning for reasoning language models.arXiv preprint arXiv:2505.22617, 2025. 1, 3, 5, 6

    Pith/arXiv arXiv 2025

  8. [8]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

  9. [9]

    Murphy: Reflective multi-turn reinforcement learn- ing for self-correcting code generation in large language models

    Chanakya Ekbote, Vijay Lingam, Behrooz Omidvar-Tehrani, Luke Huan, Sujay Sanghavi, Anoop Deoras, and Stefano Soatto. Murphy: Reflective multi-turn reinforcement learn- ing for self-correcting code generation in large language models. 2025. 1

    2025

  10. [10]

    Posterior-grpo: Rewarding reasoning processes in code gen- eration.arXiv preprint arXiv:2508.05170, 2025

    Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. Posterior-grpo: Rewarding reasoning processes in code gen- eration.arXiv preprint arXiv:2508.05170, 2025. 1

    Pith/arXiv arXiv 2025

  11. [11]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776,

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776,

    Pith/arXiv arXiv

  12. [12]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2024

  13. [13]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633– 638, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633– 638, 2025. 1, 2, 5, 6

    2025

  14. [14]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025. 2

    Pith/arXiv arXiv 2025

  15. [15]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    Pith/arXiv arXiv

  16. [16]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 1, 2

    Pith/arXiv arXiv 2024

  17. [17]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 2

    2024

  18. [18]

    Noisyrollout: Reinforcing visual reasoning with data aug- mentation.arXiv preprint arXiv:2504.13055, 2025

    Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data aug- mentation.arXiv preprint arXiv:2504.13055, 2025. 3, 5, 6, 7

    arXiv 2025

  19. [19]

    Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025. 3

    Pith/arXiv arXiv 2025

  20. [20]

    Visionreasoner: Unified visual perception and reasoning via reinforcement learning.arXiv preprint arXiv:2505.12081, 2025

    Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiaya Jia. Visionreasoner: Unified visual perception and reasoning via reinforcement learning.arXiv preprint arXiv:2505.12081, 2025. 3

    arXiv 2025

  21. [21]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025. 2, 3

    Pith/arXiv arXiv 2025

  22. [22]

    Visual- rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual- rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025. 3

    Pith/arXiv arXiv 2025

  23. [23]

    Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning.arXiv preprint arXiv:2105.04165, 2021

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning.arXiv preprint arXiv:2105.04165, 2021. 5

    Pith/arXiv arXiv 2021

  24. [24]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 5 9

    Pith/arXiv arXiv 2023

  25. [25]

    Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017. 4

    Pith/arXiv arXiv 2017

  26. [26]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learn- ing.CoRR, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Jun- jun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learn- ing.CoRR, 2025. 3, 5

    2025

  27. [27]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand `es, and Tatsunori B Hashimoto. s1: Simple test-time scaling. InProceedings of the 2025 Confer- ence on Empirical Methods in Natural Language Processing, pages 20286–20332, 2025. 2

    2025

  28. [28]

    Visual adversarial examples jailbreak aligned large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Hen- derson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. InPro- ceedings of the AAAI conference on artificial intelligence, pages 21527–21536, 2024. 4

    2024

  29. [29]

    Proximal policy optimization algo- rithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms.arXiv preprint arXiv:1707.06347, 2017. 3

    Pith/arXiv arXiv 2017

  30. [30]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing su- perb reasoning models with reinforcement learning.arXiv preprint arXiv:2504.13914, 2025. 1, 2

    arXiv 2025

  31. [31]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025. 2

    Pith/arXiv arXiv 2025

  32. [32]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2(3), 2024

    Qwen Team et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2(3), 2024. 2

    Pith/arXiv arXiv 2024

  33. [33]

    Llamav-o1: Rethinking step-by-step vi- sual reasoning in llms

    Omkar Thawakar, Dinura Dissanayake, Ketan Pravin More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Il- muz Zaman Mohammed Zumri, Jean Lahoud, Rao Muham- mad Anwer, et al. Llamav-o1: Rethinking step-by-step vi- sual reasoning in llms. InFindings of the Association for Computational Linguistics: ACL 2025, pages 24290–24315,

    2025

  34. [34]

    Mea- suring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Sys- tems, 37:95095–95169, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Sys- tems, 37:95095–95169, 2024. 5

    2024

  35. [35]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2

    Pith/arXiv arXiv 2024

  36. [36]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939, 2025

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shix- uan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939, 2025. 2, 5

    Pith/arXiv arXiv 2025

  37. [37]

    Chain-of-thought reasoning without prompting.Advances in Neural Information Pro- cessing Systems, 37:66383–66409, 2024

    Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting.Advances in Neural Information Pro- cessing Systems, 37:66383–66409, 2024. 2

    2024

  38. [38]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024. 2

    2024

  39. [39]

    Grounded chain-of-thought for multimodal large language models

    Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Rongrong Ji. Grounded chain-of-thought for multimodal large language models. arXiv preprint arXiv:2503.12799, 2025. 2

    Pith/arXiv arXiv 2025

  40. [40]

    Monte carlo tree search boosts reasoning via iterative prefer- ence learning.arXiv preprint arXiv:2405.00451, 2024

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative prefer- ence learning.arXiv preprint arXiv:2405.00451, 2024. 2

    Pith/arXiv arXiv 2024

  41. [41]

    Llava-cot: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2087– 2098, 2025. 2

    2087

  42. [42]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024. 2

    Pith/arXiv arXiv 2024

  43. [43]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023. 2

    2023

  44. [44]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xi- aochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gao- hong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025. 1, 2, 3

    Pith/arXiv arXiv 2025

  45. [45]

    What’s behind ppo’s collapse in long-cot? value opti- mization holds the secret.arXiv preprint arXiv:2503.01491,

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value opti- mization holds the secret.arXiv preprint arXiv:2503.01491,

    Pith/arXiv arXiv

  46. [46]

    Vapo: Efficient and reliable re- inforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable re- inforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025. 2

    Pith/arXiv arXiv 2025

  47. [47]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024. 5

    2024

  48. [48]

    Gen- erating natural adversarial examples.arXiv preprint arXiv:1710.11342, 2017

    Zhengli Zhao, Dheeru Dua, and Sameer Singh. Gen- erating natural adversarial examples.arXiv preprint arXiv:1710.11342, 2017. 4

    Pith/arXiv arXiv 2017

  49. [49]

    Easyr1: An efficient, scalable, multi-modality rl training framework, 2025

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework, 2025. 5 10

    2025

  50. [50]

    Reinforced mllm: A survey on rl-based reasoning in multimodal large language models

    Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang, Zheming Yang, Jian Xu, and Minghui Qiu. Reinforced mllm: A survey on rl-based reasoning in multimodal large language models. arXiv preprint arXiv:2504.21277, 2025. 2 11

    Pith/arXiv arXiv 2025