Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Improving Generalization Robustness of Multimodal RLVR

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PIRL is a training recipe that keeps RLVR-trained multimodal models accurate when prompts are paraphrased or reformatted.

desk verdict A useful, honest robustness method whose novel components are not cleanly isolated; the effect over GRPO-MT is small but the formal framing and candid limitations make it worth review. read the letter →

arxiv 2608.08802 v1 pith:UZ2J6ZWG submitted 2026-08-09 cs.AI

Pengfei Zhou , Zhiwei Tang , Xiaopeng Peng , Chenrui Zhou , Lama Moukheiber , Yixing Ma , Bin Xu , Jiajun Song
show 6 more authors
Zhenglin Wan Wangbo Zhao Jiasheng Tang Bohan Zhuang Fan Wang Yang You
This is my paper · ORCID
classification cs.AI
keywords promptrobustnessRLVRmultimodallargelanguagemodelsrewarddecompositionsemanticequivalenceclassadversarialconsistencyvisualquestionansweringtemplatestress
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that RLVR-trained multimodal models are brittle to prompt rephrasing because the standard objective gives no way to distinguish a wrong answer from a badly formatted one, and because it trains only on a thin slice of the prompts the model will meet. It argues that both problems can be addressed by optimizing over the semantic equivalence class of a prompt, and it proposes Prompt-Invariant RLVR (PIRL), which decomposes the reward into a format-aware trinary term and an entailment-based semantic term, and regularizes the policy to be consistent under an embedding-space adversary that perturbs the instruction. On exam, medical, and legal visual question answering, template-stress accuracy of PIRL-trained policies drops by at most about 1 percent on average, compared with roughly 3 percent for standard GRPO, while in-distribution accuracy is largely preserved. The method's theoretical results identify why this works: reward decomposition restores useful variation within format buckets, and invariance control with small $\varepsilon$ excludes the format-fixed shortcut policies that the training objective cannot distinguish.

What carries the argument

The load-bearing objects are: the semantic equivalence class $\llbracket\boldsymbol{x}\rrbracket$, the set of transformations that preserve the task answer; the Dynamic Trinary Reward $R_{\mathrm{DTR}} = R_t + \alpha_{\mathrm{rew}} R_s$, where $R_t$ distinguishes correct, wrong-but-formatted, and format-failure and $R_s$ is a bounded entailment score; and the adversarial consistency objective, which perturbs instruction-token embeddings within an $\ell_2$ ball to drive down DTR reward and adds a token-level KL penalty between clean and perturbed policy distributions. The theory supplies two population-level conditions, (C1) Reward Decomposition and (C2) Invariance Control; Proposition B.10 shows that under small $\varepsilon$ the format-fixed shortcut branch is excluded while the correct-answer branch is feasible, and Proposition B.11 gives a coverage-transfer bound. The per-token KL is an acknowledged tractable surrogate, not a certificate of answer-level invariance.

What would settle it

Take a PIRL-trained policy and a hand-built set of literal paraphrases and template swaps that lie outside the $\ell_2$ ball used in training; if answer-level accuracy under those rewordings drops as much as GRPO's while clean accuracy stays high, the per-token KL surrogate did not transfer to answer-level invariance. A cheaper diagnostic: track $\Phi_{wf}$ during GRPO training, and if it does not rise toward 1 as format compliance increases, the variance-entanglement mechanism is not the operative cause of the robustness gap.

Watch

Extended reading notes

Core claim

The central claim is that prompt-template brittleness in RLVR-trained multimodal LLMs is not an inevitable side effect of reinforcement learning but can be addressed by two design conditions: reward decomposition and invariance control. The paper formalizes a worst-case objective over the semantic equivalence class of a prompt and shows that binary verifiers entangle format with semantics (Theorem B.1), and that the training objective cannot identify the prompt-invariant policy when unseen formats carry test mass (Proposition B.5). PIRL instantiates the two conditions with a Dynamic Trinary Reward, which assigns $+1$, $-1$, and $0$ to correct, format-failure, and wrong-but-formatted cases alongside an NLI-based semantic score, and with a projected-gradient adversary on the instruction embedding plus a per-token KL consistency penalty. The paper's empirical claim is that under template-stress evaluation PIRL's mean accuracy drops by at most about 1 percentage point versus roughly 3 percentage points for GRPO, and that PIRL has the smallest mean drop among trained methods under dynamic evaluation.

Load-bearing premise

The load-bearing premise is that small bounded perturbations of the instruction-token embedding are a workable stand-in for worst-case semantics-preserving rewordings, and that optimizing a per-token KL consistency penalty transfers to answer-level invariance on literal rephrased prompts.

Editorial extensions

If this is right

  • On seven public VQA benchmarks and two base models of 7–8B parameters, template-stress accuracy of PIRL-trained policies drops by about 1 percentage point on average while GRPO drops about 3 percentage points (Table 1).
  • In-distribution accuracy is not sacrificed for robustness: PIRL usually matches GRPO on standard-format benchmarks and on some benchmarks improves both standard and stress accuracy.
  • The robustness gain concentrates where the non-identifiability theory says it should: as more unseen templates are introduced, GRPO's in-distribution accuracy falls while PIRL stays flat (Figure 3).
  • Under dynamic evaluation PIRL has the smallest mean performance drop among trained methods, but on task-altering mutations (multiple-choice converted to free-form, document layout changes) every method degrades substantially, so the claim is scoped to semantic rewording rather than all input changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the surrogate-transfer assumption holds, the same consistency-regularizer design could be applied to image-embedding perturbations; the paper's own modality ablation already suggests that image robustness requires image-side adversarial training, a direct extension of the same mechanism.
  • The theory predicts a diagnostic that could be measured in any existing GRPO run: the within-format reward-variance share $\Phi_{wf}$ should climb toward 1 as format compliance rises, while PIRL's floor stays bounded by $4\alpha_{\mathrm{rew}}^2 m/(\alpha_{\mathrm{rew}}+2)^2$; the paper's instrumentation in Appendix C.3 checks exactly this without retraining.
  • Because the coverage-transfer bound transfers a policy's own reward, a natural follow-up is to choose the training transformation pool adaptively from the deployment prompt distribution rather than fixing five templates, making the small stress-drop number a design target rather than an evaluation artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies why RLVR-trained multimodal LLMs are brittle to prompt paraphrases and template changes, and proposes Prompt-Invariant RLVR (PIRL) as a remedy. The authors identify two causes: binary verifiers entangle format compliance with semantic correctness, and training over a narrow prompt distribution cannot identify behavior on unseen formats. They formalize these with a reward-variance share theorem and a non-identifiability proposition, and derive two design conditions, C1 (reward decomposition) and C2 (invariance control). PIRL instantiates C1 with a Dynamic Trinary Reward (DTR) plus decoupled advantage normalization, and C2 with an embedding-space adversarial prompt sampler plus a per-token KL consistency regularizer. Experiments on exam, medical, and legal VQA benchmarks with Qwen2.5-VL-7B and Qwen3-VL-8B compare PIRL with GRPO and multi-template GRPO-MT under template stress and dynamic evaluation. The main empirical claim is that PIRL has the smallest mean degradation among trained methods, with abstract-level summary that PIRL drops only ≤1% on average while GRPO drops about 3%. The paper explicitly concedes in Section 4.5 that the ablation does not isolate DTR or the adversary, and in Appendix A that the per-token surrogate's effect on answer-level divergence remains open.

Significance. If the aggregate empirical claims hold, PIRL is a useful contribution: it targets a real deployment risk for high-stakes multimodal VQA, and the paper ships a sizable evaluation suite with two backbones and seven benchmarks. The theoretical propositions (Theorem B.1, Propositions B.5, B.10, B.11) are derived from stated definitions and are not fitted to the experiments, and the manuscript is unusually transparent about the limits of its adversarial surrogate and its component ablation. These strengths are genuine. The significance is tempered by two facts: the advantage of full PIRL over GRPO-MT is, on one backbone, within the paper's own noise band, and the specific mechanisms claimed as contributions—DTR and the embedding adversary—are not cleanly isolated by the reported ablation. As a result, the paper presently supports the full recipe as a heuristic combination, but not the attribution of its gains to the proposed components.

major comments (3)
  1. [Section 4.5, Table 3] The component ablation does not support the claim that DTR or the embedding adversary contributes to the leading-order robustness gain. On Exam VQA Avg, full PIRL has ΔT=+0.6±0.5, MT+DTR has 0.0±0.5, and ADV+DTR without MT has -0.9±0.5; the paper's own inconclusive threshold is ±1.5pp, so the novel components are not separated from multi-template sampling. Since the abstract and introduction present DTR and the adversary as the method's two measures, the central attribution needs evidence: add a MT+ADV (no DTR) row, run paired comparisons across the per-benchmark T-Stress gaps, and ideally include a control with random embedding noise to show the adversary's adversarial selection, not mere perturbation, is the active ingredient.
  2. [Appendix A, Eq. (8)] The mechanism connecting the per-token KL consistency penalty to the answer-level C2 objective is explicitly left open: the paper states that whether optimizing the per-token surrogate decreases answer-level divergence remains an open empirical question. This is load-bearing because the consistency regularizer is one of the two named components of PIRL, and without evidence that training reduces answer-level divergence between clean and adversarial prompts, the regularizer could be acting as a generic smoothing term. I recommend measuring the answer-level KL or total-variation distance on held-out prompt pairs before and after PIRL training, or otherwise providing a direct test of the surrogate transfer.
  3. [Table 1, Qwen3-VL-8B] The claimed advantage over GRPO-MT is not consistently outside the noise band. Computing mean template-stress gaps from Table 1 gives about -0.1pp for PIRL versus -1.1pp for GRPO-MT on Qwen2.5-VL-7B, but only about -0.5pp versus -0.9pp on Qwen3-VL-8B, a difference of roughly 0.4pp against per-benchmark uncertainties of ±0.4 to ±0.7pp. The abstract's 'GRPO drops ~3%' matches only the Qwen2.5 backbone, not the Qwen3 result (~1.3%). To support the headline claim, the authors should report a statistical comparison across benchmarks and seeds (e.g., a paired test or bootstrap over the seven benchmarks), rather than relying on mean gaps that are partly within the inconclusive threshold.
minor comments (5)
  1. [Section 4.6 and Figure 5] The text says PIRL restores template-stress robustness to base level; this is accurate for Qwen2.5-VL-7B (mean ΔT ≈ -0.1 vs. base ≈ -0.1) but less so for Qwen3-VL-8B, where PIRL's mean is -0.5 vs. the base's +0.3.
  2. [Table 5, Eq. (7)] The hyperparameter table does not list the advantage-mixing coefficient α_adv used in DAN, which is needed for exact reproduction; either add it to Table 5 or state explicitly that it equals 1.0.
  3. [Section 4.1] The description 'sample one from n∈{1,...,15} templates' is ambiguous: n could be the number of sampled templates or an index over templates; Figure 3 makes the intended meaning clearer, and the text would benefit from a definition such as 'evaluate with a pool of n templates.'
  4. [Appendix C, Algorithm 2] The notation τ̄_φ^{x0}(x0) is hard to parse; define the frozen adversarial prompt with a lighter notation, e.g., x̃0, and use it consistently in the policy update step.
  5. [Section 2.2] Theorem B.1 is stated under an independence assumption that the main text later relaxes in Appendix B.2; a forward pointer to the relaxation when the formula is first introduced would reduce the risk of misreading the independence assumption as essential to the argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's formal statements follow from explicit definitions, and its empirical claims are tested on external benchmarks with the surrogate and ablation limitations openly stated.

full rationale

The theoretical results are self-contained and not circular. Theorem B.1 computes the within-format reward-variance share from the stated multiplicative verifier definition V = Vf·Vc with Vf independent of Vc; the paper explicitly treats Phi_wf as a "reward-side proxy" for gradient signal, not as a fitted prediction of any experimental curve. Proposition B.5 constructs two policies that coincide on the training support by Assumption B.4 and differ only on a separated unseen format tau_B; the conclusion that P_train-optimal policies can differ under P_test is exactly the content of that construction, which is a valid non-identifiability counterexample rather than a circular inference. Theorem B.8 bounds the DTR variance share under the stated within-bucket dispersion assumption on the semantic score Rs, and Proposition B.10 derives consequences directly from the formal definitions of C1 and C2; these are logical derivations from stated assumptions, not restatements of the experiments. The empirical evaluation is also not circular: it uses external benchmarks (MMK12, MathVista, Olym-Phys, VQA-RAD, PathVQA, GMAI-MMBench, DocVQA) and external base models, and the paper explicitly says Proposition B.5 "is not a fitted prediction of this curve" (Section 4.3). The ablation concedes that DTR and the adversary are not individually isolated beyond multi-template training: "this ablation does not cleanly isolate DTR's or the adversary's individual contributions beyond MT" (Section 4.5). That is a stated limitation of empirical attribution, which is the opposite of presenting a fitted input as a prediction. Appendix A also states that the empirical question of whether the per-token surrogate "decreases the answer-level divergence, remains open," so the surrogate-transfer assumption is declared rather than disguised as a theorem. No load-bearing self-citation or imported uniqueness theorem from the authors is used: the cited work provides baselines, benchmarks, and standard objectives such as GRPO, PPO, and group DRO, rather than supplying the paper's conclusions. The remaining concerns in the manuscript are about empirical attribution strength and surrogate-transfer validity, not about circular derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or ontological entities. Its postulates are modeling assumptions about reward structure, extractor behavior, semantic-score variance, and the fidelity of embedding-space perturbations. The key hyperparameters were set by hand and are free parameters in the empirical claim, not derived quantities.

free parameters (4)
  • alpha_rew (semantic reward weight) = 0.01
    Chosen by manual sweep on the Exam validation split; balances the trinary reward and the NLI semantic score. It is not derived from theory.
  • lambda (consistency penalty weight) = 1.0
    Chosen by manual sweep; sets the strength of the adversarial-to-clean KL penalty. The invariance transfer depends on this value.
  • epsilon_emb (adversary trust-region radius) = 10% of median instruction-embedding norm
    Chosen by hand; defines the size of the perturbation neighborhood that stands in for semantics-preserving rewordings in embedding space.
  • alpha_adv (advantage-mixing coefficient) = not reported in text
    Coefficient in the decoupled advantage normalization of Eq. 7; no explicit value is given, so its effect on the updates is not fully specified.
assumptions (5)
  • domain assumption V_f and V_c are assumed independent in the main reward-variance theorem (Theorem B.1).
    The monotonicity formula for the within-format variance share relies on independence; the appendix partially relaxes this, but the main qualitative claim uses it.
  • domain assumption Format-exact extraction: A_tau(y) is non-empty if and only if V_f = 1.
    Theorem B.8 requires the extractor to be exact on format, which may fail in practice when models use delimiters inconsistently.
  • domain assumption R_s has non-degenerate within-bucket dispersion: Var(R_s | V_c^ext=v, V_f=v') >= m in every positive-probability bucket.
    The DTR variance-floor theorem depends on this non-degeneracy assumption for the NLI semantic score, which is not verified empirically.
  • domain assumption There exists a separated unseen format tau_B whose verifier constraint is disjoint from all training formats.
    Proposition B.5 constructs non-identifiability from the existence of such a format; this is a coverage assumption about deployment.
  • ad hoc to paper Embedding-space l2 perturbations on the instruction block are a valid surrogate for semantics-preserving prompt transformations.
    The paper explicitly says perturbations need not decode to literal prompts and that the adversary is not a certificate of on-manifold transformations; the whole invariance method rests on this surrogate assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Generalization Robustness of Multimodal RLVR." pith.science (2026). https://pith.science/paper/UZ2J6ZWG

@misc{pith2026260808802,
  author       = {Pith},
  title        = {Pith review of: Improving Generalization Robustness of Multimodal RLVR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZ2J6ZWG}},
  note         = {Machine review of arXiv:2608.08802}
}
abstract

Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only $\le 1\%$, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 17 canonical work pages

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    arXiv 2025

  2. [2]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

    arXiv 2025

  3. [3]

    e-SNLI:Naturallanguage inference with natural language explanations.Advances in Neural Information Processing Systems, 31, 2018

    Oana-MariaCamburu,TimRocktäschel,ThomasLukasiewicz,andPhilBlunsom. e-SNLI:Naturallanguage inference with natural language explanations.Advances in Neural Information Processing Systems, 31, 2018

    work page 2018

  4. [4]

    Seibel, Yu Qiao, and Junjun He

    Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J. Seibel, Yu Qiao, and Junjun He. GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical ai.Advances in Neural Information Processing ...

    work page 2024

  5. [5]

    When llm meets drl: Advancing jailbreaking efficiency via drl-guided search.Advances in Neural Information Processing Systems, 37:26814–26845, 2024

    Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. When llm meets drl: Advancing jailbreaking efficiency via drl-guided search.Advances in Neural Information Processing Systems, 37:26814–26845, 2024

    2024

  6. [6]

    DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 11 Improving Generalization Robustness of Multimodal RLVR

    arXiv 2025

  7. [7]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. InProceedings of the 62nd Annual Meeting of the Association for Co...

    work page 2024

  8. [8]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    arXiv 2003

Show all 35 references
  1. [9]

    Reinforce++: Stabilizing critic-free policy opti- mization with global advantage normalization.arXiv preprint arXiv:2501.03262, 2025

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: Stabilizing critic-free policy opti- mization with global advantage normalization.arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  2. [10]

    Toward evaluative thinking: Meta policy optimization with evolving reward models.arXiv preprint arXiv:2504.20157, 2025

    Zae Myung Kim, Chanwoo Park, Vipul Raheja, Suin Kim, and Dongyeop Kang. Toward evaluative thinking: Meta policy optimization with evolving reward models.arXiv preprint arXiv:2504.20157, 2025

    2025 arXiv

  3. [11]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. Lingshu: A generalist foundation model...

    2025 arXiv

  4. [12]

    Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman

    Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific Data, 5(1):180251, 2018

    2018

  5. [13]

    Vision matters: Simple visual perturbations can boost multimodal math reasoning.arXiv preprint arXiv:2506.09736, 2025

    Yuting Li, Lai Wei, Kaipeng Zheng, Jingyuan Huang, Linghe Kong, Lichao Sun, and Weiran Huang. Vision matters: Simple visual perturbations can boost multimodal math reasoning.arXiv preprint arXiv:2506.09736, 2025

    2025

  6. [14]

    R1-fuzz: Specializing language models for textual fuzzing via reinforcement learning.arXiv preprint arXiv:2509.20384, 2025

    Jiayi Lin, Liangcai Su, Junzhe Li, and Chenxiong Qian. R1-fuzz: Specializing language models for textual fuzzing via reinforcement learning.arXiv preprint arXiv:2509.20384, 2025

    2025

  7. [15]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  8. [16]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InThe Twelfth International Conference on Learning ...

    2024

  9. [17]

    R-horizon: How far can your large reasoning model really go in breadth and depth?arXiv preprint arXiv:2510.08189, 2025

    Yi Lu, Jianing Wang, Linsen Guo, Wei He, Hongyin Tang, Tao Gui, Xuanjing Huang, Xuezhi Cao, Wei Wang, and Xunliang Cai. R-horizon: How far can your large reasoning model really go in breadth and depth?arXiv preprint arXiv:2510.08189, 2025

  10. [18]

    Information-theoretic reward decomposition for generalizable rlhf.arXiv preprint arXiv:2504.06020, 2025

    Liyuan Mao, Haoran Xu, Amy Zhang, Weinan Zhang, and Chenjia Bai. Information-theoretic reward decomposition for generalizable rlhf.arXiv preprint arXiv:2504.06020, 2025

    2025

  11. [19]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2199–2208, 2021

    2021

  12. [20]

    MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based rei...

    2025 arXiv

  13. [21]

    Beyond benchmarks: Dynamic, automatic and systematic red-teaming agents for trustworthy medical language models.arXiv preprint arXiv:2508.00923, 2025

    Jiazhen Pan, Bailiang Jian, Paul Hager, Yundi Zhang, Che Liu, Friedrike Jungmann, Hongwei Bran Li, Chenyu You, Junde Wu, Jiayuan Zhu, Fenglin Liu, Yuyuan Liu, Niklas Bubeck, Christian Wachinger, Chen Chen, Zhenyu Gong, Cheng Ouyang, Georgios Kaissis, Benedikt Wiestler, and Dan...

    2025 arXiv

  14. [22]

    Hashimoto, and Percy Liang

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neu- ral networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020

    2020

  15. [23]

    Proximal policy opti- mization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  16. [24]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  17. [25]

    On the value of out-of-distribution testing: An example of Goodhart’s law

    Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel. On the value of out-of-distribution testing: An example of Goodhart’s law. InAdvances in Neural Information Processing Systems, pages 407–417, 2020

    2020

  18. [26]

    Blaschko, Sien Moens, and Tomasz Stanisławek

    Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Paweł Józiak, Rafał Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Ackaert, Ernest Valveny, Matthew B. Blaschko, Sien Moens, and Tomasz Stanisławek. Document understanding dataset and evaluation (dud...

    2023

  19. [27]

    Rlhfpoison: Reward poisoningattackforreinforcementlearningwithhumanfeedbackinlargelanguagemodels

    Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, and Chaowei Xiao. Rlhfpoison: Reward poisoningattackforreinforcementlearningwithhumanfeedbackinlargelanguagemodels. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2024

  20. [28]

    Causally- enhanced reinforcement policy optimization.arXiv preprint arXiv:2509.23095, 2025

    Xiangqi Wang, Yue Huang, Yujun Zhou, Xiaonan Luo, Kehan Guo, and Xiangliang Zhang. Causally- enhanced reinforcement policy optimization.arXiv preprint arXiv:2509.23095, 2025

    2025

  21. [29]

    Adversarial preference learning for robust llm alignment.arXiv preprint arXiv:2505.24369, 2025

    Yuanfu Wang, Pengyu Wang, Chenyang Xi, Bo Tang, Junyi Zhu, Wenqiang Wei, Chen Chen, Chao Yang, Jingfeng Zhang, Chaochao Lu, Yijun Niu, Keming Mao, Zhiyu Li, Feiyu Xiong, Jie Hu, and Mingchuan Yang. Adversarial preference learning for robust llm alignment.arXiv preprint arXiv:2...

    2025 arXiv

  22. [30]

    Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping

    Yue Yang, Shuibai Zhang, Wenqi Shao, Kaipeng Zhang, Yi Bin, Yu Wang, and Ping Luo. Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping. InInternational Conference on Learning Representations, 2025

    2025

  23. [31]

    Reward-guided prompt evolving in reinforcement learning for llms

    Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V Le, Qijun Tan, and Yuan Liu. Reward-guided prompt evolving in reinforcement learning for llms. InForty-second International Conference on Machine Learning, 2025

    2025

  24. [32]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?Advances in Neural Information Processing Systems, 38:57654–57689, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?Advances in Neural Information Processing Systems, 38:57654–57689, 2025

    2025

  25. [33]

    A survey of reinforcement learning for large reasoning models.arXiv preprint arXiv:2509.08827, 2025

    Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua,...

    2025 arXiv

  26. [34]

    Improving reward model generalization from adversarial process enhanced preferences

    Zhilong Zhang, Tian Xu, Xinghao Du, Xingchen Cao, Yihao Sun, and Yang Yu. Improving reward model generalization from adversarial process enhanced preferences. InForty-second International Conference on Machine Learning, 2025

    2025

  27. [35]

    Clean” applies none and “Both

    Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang, Zheming Yang, Jian Xu, and Minghui Qiu. Rein- forced mllm: A survey on rl-based reasoning in multimodal large language models.arXiv preprint arXiv:2504.21277, 2025. 13 Improving Generalization Robustness of Multimodal RLVR A. Form...

    2025 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.