Pith. sign in

REVIEW 3 major objections 5 minor 36 references

R2PO claims that separating the policy that explores during training from the policy that answers at inference — via a lightweight residual head — improves reasoning accuracy and training stability at zero inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

R²PO trains LLM reasoning policies with a separate residual rollout head, reporting 3.4% higher MATH-500 and 1.3% higher APPS accuracy than GRPO.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Plausible decoupling idea for GRPO, but the Stage-2 objective is ambiguous enough that the reported gains could be an artifact; worth a serious referee, not a desk reject. the 3 major comments →

arxiv 2601.11960 v3 pith:YMV7HAO3 submitted 2026-01-17 cs.LG cs.AIcs.CL

R²PO: Decoupling Rollout and Inference Policies for LLM Reasoning

Jingchu Wang , Bingbing Xu , Yige Yuan , Dan Zhang , Bin Xie , Xiaoqian Sun , Huawei Shen This is my paper
classification cs.LG cs.AIcs.CL
keywords LLM reasoningreinforcement learningGRPOexploration-exploitationresidual rollout headGroup Inverse-Frequency rewardtraining stabilityverifiable rewards
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

R2PO makes a structural claim: in reinforcement learning for LLM reasoning, the policy used to generate training trajectories should not be the same as the policy that produces final answers. The paper argues that forcing both roles into one output head entangles exploration and exploitation gradients, suppresses diversity, and invites late-training collapse. It proposes a small Residual Rollout-Head that adds a logit offset to the frozen backbone, is trained with a Group Inverse-Frequency reward to produce diverse rollouts, and is detached at inference. Experiments report average gains over standard GRPO of 3.4% on MATH-500 and 1.3% on APPS, plus fewer formatting errors, lower length bias, and immunity to a deliberately injected noise trap. If correct, this gives a simple, inference-free architectural lever for improving reasoning RL.

Core claim

The central discovery is that the "one policy for both training and inference" assumption in GRPO-style LLM reasoning RL is a harmful inductive bias. R2PO attaches a two-layer Residual Rollout-Head to the transformer's hidden states; its logit output is added to the base LM head, defining a separate exploration policy. Stage 1 freezes the backbone and optimizes the head with a Group Inverse-Frequency reward, rewarding trajectories that are rare within their reward group. Stage 2 freezes the head and uses it purely as a behavioral sampler while the base policy is optimized on task rewards. On two open base models at 3B and 8B scale, the paper reports consistent improvements over vanilla GRPO,

What carries the argument

Residual Rollout-Head: a two-layer MLP over the transformer's hidden states, zero-initialized so exploration begins identical to the base policy. It produces a logit-level offset that defines the exploration policy πφ = Softmax(f_RO(H) + f_LM(H)). The head absorbs exploration-gradient noise, leaving the base LM head stable; the two-stage alternating GRPO scheme first trains the head on a Group Inverse-Frequency reward, then uses the frozen head as the trajectory sampler for updating the base policy.

Load-bearing premise

The method depends on the Stage-2 update being a valid policy-gradient step: it samples trajectories from the frozen exploration head but writes the GRPO importance ratio using the base policy's own old distribution, without redefining the old policy as the exploration policy or adding an off-policy correction. If those two distributions diverge, the gradient is not an unbiased estimate of the intended objective.

What would settle it

The clearest check: rerun Stage 2 with importance ratios computed as πθ(y)/πφ(y) instead of πθ(y)/πθ_old(y), or explicitly set πθ_old equal to πφ. If the MATH-500 gain over vanilla GRPO disappears, the reported improvement is an artifact of a biased estimator rather than of architectural decoupling. A complementary test is to freeze the rollout head and sample Stage-2 trajectories from the base policy instead; if gains persist, decoupling itself is not the mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If R2PO is correct, reasoning RL does not need to sacrifice inference quality to encourage exploration: a separate head can absorb the noise.
  • Gains concentrate on harder, out-of-distribution benchmarks, suggesting the decoupling expands discovered reasoning paths rather than just fitting the training distribution.
  • The perturbation experiment implies the decoupled head shields the main policy from reward misspecification and late-training formatting collapse.
  • Length-bias reduction indicates the model stops using verbosity as a buffer against negative gradients, giving more concise responses.
  • Because the Residual Rollout-Head is discarded at inference, the method adds zero deployment latency and no inference-time parameter overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves implicit: replace the learned Residual Rollout-Head with fixed random logit noise of matched magnitude. If gains persist, trajectory diversity alone, not learned exploration, explains the result.
  • The same two-policy separation could be dropped into other RLVR algorithms, not just GRPO, since most single-policy reasoning RL methods share the conflation the paper targets.
  • The ablation showing a main-reward rollout head slightly beats the Group Inverse-Frequency variant suggests reward design may be secondary; a simpler temperature- or noise-based rollout policy might capture much of the benefit.
  • At larger scales, the 7.8–10.2% parameter overhead could be reduced with a low-rank residual head, but the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that standard RL fine-tuning of LLM reasoners conflates two distinct distributions: the trajectory distribution used for optimization (which should be diverse and exploratory) and the response distribution used at inference (which should be accurate and stable). To separate them, the authors propose R2PO, which attaches a lightweight two-layer MLP 'Rollout-Head' to the LM head. The head produces a residual logit offset, yielding an exploration policy pi_phi, while the original LM head defines the inference policy pi_theta. Training alternates between Stage 1, which optimizes the Rollout-Head using a group inverse-frequency (GIF) reward to encourage diverse rollouts, and Stage 2, which freezes the Rollout-Head and updates the main policy using GRPO on trajectories sampled from pi_phi. Experiments on Qwen2.5-3B and Qwen3-8B across GSM8K, MATH-500, MBPP, HumanEval, and APPS report average gains over a GRPO baseline, reduced length bias, and robustness to formatting-reward misspecification. Code is stated to be available.

Significance. If the core claim holds, R2PO would be a simple, low-overhead way to improve exploration in LLM RL without changing the inference-time policy: the auxiliary head adds about 8–10% training parameters and no inference latency, and the reported gains on MATH-500 and APPS are non-trivial. The paper also provides a useful diagnostic angle by showing that a decoupled exploration head can absorb reward-misspecification noise (Section 4.3.2). However, the manuscript currently has a load-bearing gap in the Stage-2 objective: the off-policy update is not correctly specified, and because the Rollout-Head shares the backbone, the claimed 'frozen behavioral sampler' is not in fact stationary during Stage 2. The empirical claims also rest on single runs with no error bars or significance testing, which makes the 'consistently outperforms' statement unsubstantiated. The availability of code is a strength, but the manuscript itself must be self-contained on the training objective.

major comments (3)
  1. [Section 3.4.2, Eqs. (4)–(6)] The Stage-2 objective is not a valid estimator of J_GRPO as written. Eq. (4) defines J_GRPO with an importance ratio pi_theta/pi_theta_old and sampling y ~ pi_theta_old. Eq. (6) states that Stage 2 optimizes J_GRPO(theta) with y ~ pi_phi, but the paper never redefines pi_theta_old to equal pi_phi nor derives an off-policy correction. If pi_theta_old is the previous main policy, the ratio pi_theta/pi_theta_old is incorrect for trajectories sampled from pi_phi. If pi_theta_old is intended to be pi_phi, then because pi_phi = Softmax(f_RO(H) + f_LM(H)) shares the backbone H that is being updated in Stage 2 (Eq. (1)), pi_phi changes as theta updates, so it is not a fixed behavior policy. The clipping in Eq. (4) does not remove the resulting bias. The authors must either (a) sample and freeze all Stage-2 trajectories at the start of the stage using a truly stationary behavior policy and use th
  2. [Table 1 and Section 4.2] All quantitative results are from single runs with no seeds, no confidence intervals, and no significance tests. The abstract and Section 4.2 claim R2PO 'consistently outperforms' GRPO, but Table 1 shows a regression on Qwen3-8B GSM8K (GRPO 88.55 vs. R2PO 88.48) and mixed differences on other cells (e.g., Qwen2.5-3B HumanEval 68.90 vs. 68.90? Actually 68.90 vs. 68.90? The table shows 68.90 and 68.90? It shows 68.90 for GRPO and 68.90 for R2PO? The table lists 68.90 and 68.90? In the table, Qwen2.5-3B GRPO HumanEval is 68.90 and R2PO is 68.90, so no gain. Wait the table in the prompt: Qwen2.5-3B + GRPO 58.60 68.90 8.22; R2PO 59.00 68.90 9.52; so HumanEval equal. Qwen3-8B GSM8K is 88.55 vs 88.48, a regression. The 'average accuracy increment' depends on averaging across datasets with different scales. The claim of consistent improvement is not supported by the evidence as presented. At min
  3. [Section 4.1, Evaluation Protocol] The paper notes that for MATH-500 the OpenCompass rule-based metric is not fully compatible with the training output format, so the authors use their own 'tailored metric.' This is a potential comparability threat: if the tailored metric differs from the standard protocol used in prior GRPO comparisons, the absolute numbers may not be comparable across papers. The authors should provide the exact criteria of their tailored metric, ideally the evaluation script, and confirm that the same metric is applied to baseline and R2PO at the same evaluation points. Without this, the headline MATH-500 gains (3.4% and 2.8%) cannot be independently reproduced.
minor comments (5)
  1. [Section 3.3, Eq. (3)] The 'Z-score Standardization' applied to GIF scores is mentioned but never defined. Specify whether standardization is across the group or across training steps, and give the exact formula.
  2. [Section 4.1 and References] MATH-500 is cited to Zhang et al. (2025), but that reference is about image generation and does not describe MATH-500. Please cite the original source of MATH-500 (e.g., the MATH dataset and its 500-problem subset).
  3. [Section 4.3.2, Table 2 and Table 3] The perturbation experiment measures only 'the first 100 samples' and reports very small error rates (0.125% to 0.875%). The conclusions about 'perturbation immunity' would be stronger if the measurement set and the perturbation-injection protocol were described in more detail, including how many runs and whether the 10-step injection is fixed across runs.
  4. [Section 4.3.3, Table 4] Length comparison is based on a single early-phase GRPO model at 100 steps. The choice of reference point is not justified beyond 'sufficiently standardized'; report sensitivity to this choice.
  5. [General] There are minor typos, e.g., 'efficieancy' in Section 4.5, and inconsistent use of 'R2PO' vs. 'R$^2$PO' in the abstract and body. Also, the sentence in Section 4.2 about 'destructive gradient noise' is plausible but not directly evidenced; consider softening.

Circularity Check

0 steps flagged

No significant circularity: the reported gains are measured on independent held-out benchmarks, no load-bearing self-citation is used, and the main concern (Stage-2 off-policy ambiguity) is a correctness/validity issue, not a by-construction reduction.

full rationale

The paper's central empirical claim is that R2PO outperforms GRPO on MATH-500, APPS, GSM8K, MBPP, and HumanEval. These are external, held-out benchmarks evaluated with the OpenCompass framework plus a rule-based reward/parser; none of these benchmark numbers are used to fit the rollout head or the main-policy objective. The GIF reward is a count-based, per-batch inverse-frequency signal; while it explicitly encourages diversity, the primary result (accuracy gains) is not the same as the optimized reward, and the ablation using the standard correctness/formatting reward exceeds the GIF variant, showing the reported benefit is not simply a restatement of the exploration reward. The paper contains no identifiable chain of self-citations that carries the load: the references are to external prior work and the authors do not invoke a personal uniqueness theorem or an unverified prior result as a premise. The clearest weakness is in Section 3.4.2 / Eq. (6): the objective is written as J_GRPO(θ) with trajectories sampled from πφ, while Eq. (4) defines the importance ratio using πθ_old and describes πθ_old as the previous-iteration policy. The text does not explicitly state that πθ_old is redefined as πφ, nor does it derive an off-policy correction. This is a potentially load-bearing ambiguity about whether the Stage-2 gradient is an unbiased estimator of the intended objective, and it may be a correctness or reproducibility risk; however, it is not circular in the sense of the target result being equivalent to its inputs by construction. Similarly, the authors' disclosure that MATH-500 is evaluated with their own tailored metric because the OpenCompass rule is incompatible with the training output format is a measurement-protocol risk, not a fitted-input-called-prediction or renaming pattern. The paper's own limitation section restricts claims to 3B/8B and single-turn reasoning, further indicating the contribution is empirical rather than an assumed-first-principles derivation. On the circularity scale, the derivation and evaluation are self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The paper's method rests on a shared backbone with a small auxiliary head, plus several unreported or hand-chosen hyperparameters. The most serious assumption is that the Stage-2 GRPO update is valid when sampling from πφ; this is not clearly derived and is not supported by an independent off-policy argument.

free parameters (6)
  • R_acc / R_fmt reward weights
    Defined in Eq. 2 but numerical values are never reported; the central reward signal depends on them.
  • GRPO group size G = 8
    Set in A.3; standard but chosen by hand.
  • KL coefficient beta = 0.04
    Set in A.3; chosen by hand.
  • PPO clip range epsilon = 0.2
    Set in A.3; chosen by hand.
  • Learning rate schedule = 1e-6
    A.3 says 'constant learning rate' and 'cosine decay schedule'; the actual schedule is contradictory and unspecified.
  • Alternating stage schedule
    The proportion of Stage-1 vs Stage-2 steps is never specified, yet it controls the decoupling dynamics.
axioms (4)
  • domain assumption The GRPO objective in Eq. 4 is a valid update for θ when trajectories are sampled from the frozen exploration policy πφ.
    Section 3.4.2 states y∼πφ but does not specify whether πold is redefined as πφ or an off-policy correction is applied; without this, the gradient estimator is biased.
  • domain assumption Inference-optimal responses and optimization-optimal trajectories can be usefully separated by a residual logit perturbation on a shared backbone.
    Sections 1 and 3.1 frame this as the motivating hypothesis; it is not derived.
  • domain assumption Zero-initialization of the Rollout-Head keeps the initial exploration policy equal to the base policy and stabilizes cold-start.
    Section 3.2; assumed, not proven.
  • standard math Standard GRPO/PPO clipping and KL regularization remain valid under alternating freeze-update schedules.
    Used in Eq. 4; assumed from Shao et al. 2024.
invented entities (1)
  • Residual Rollout-Head (two-layer MLP) no independent evidence
    purpose: Produces a logit-level offset f_RO(H) added to f_LM(H), defining the exploration policy πφ for training rollouts.
    No external falsifiable prediction; its value is only demonstrated through the paper's own ablations and benchmark results.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning." pith.science (2026). https://pith.science/paper/YMV7HAO3

@misc{pith2026260111960,
  author       = {Pith},
  title        = {Pith review of: R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMV7HAO3}},
  note         = {Machine review of arXiv:2601.11960}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R$^2$PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R$^2$PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.

Figures

Figures reproduced from arXiv: 2601.11960 by Bingbing Xu, Bin Xie, Dan Zhang, Huawei Shen, Jingchu Wang, Xiaoqian Sun, Yige Yuan.

Figure 1
Figure 1. Figure 1: Decoupling Ideal Responses and Optimization Trajectories. Left: The mismatch between ideal inference responses and ideal training trajectories. Right: Single-head policy optimization versus R 2PO with a decoupled Rollout-Head for stable exploration. sults underscore that R 2PO significantly outper￾forms the standard GRPO baseline, achieving an average accuracy increment of 3.4% on MATH￾500 and 1.3% on APPS… view at source ↗
Figure 2
Figure 2. Figure 2: illustrates the overall framework and the interaction between the two stages. 3.1 Motivation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The mean and variance of rewards during the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 14 linked inside Pith

  1. [1]

    Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and R \' e mi Munos

    Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and R \' e mi Munos. 2016. https://proceedings.neurips.cc/paper/2016/hash/afda332245e2af431fb7b672a68b659d-Abstract.html Unifying count-based exploration and intrinsic motivation . In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Informatio...

    2016

  2. [2]

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. 2025. https://doi.org/10.48550/ARXIV.2506.14758 Reasoning with exploration: An entropy perspective . CoRR, abs/2506.14758

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.14758 2025

  3. [3]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html Deep reinforcement learning from human preferences . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing System...

    2017

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . CoRR, abs/2110.14168

    Pith/arXiv arXiv 2021

  5. [5]

    OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

  6. [6]

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. 2025. https://doi.org/10.48550/ARXIV.2505.22617 The entropy mechanism of reinforcement learning for reasoning language models . CoRR, abs/2505.22617

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.22617 2025

  7. [7]

    DeepSeek - AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. https://doi.org/10.48550/ARXIV.2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinfor...

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  8. [8]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    Pith/arXiv arXiv 2025

  9. [9]

    Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. 2025. https://openreview.net/forum?id=PzSG5nKe1q RLEF: grounding code llms in execution feedback with reinforcement learning . In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 . OpenReview.net

    2025

  10. [10]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. http://proceedings.mlr.press/v80/haarnoja18b.html Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor . In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15,...

    2018

  11. [11]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html Measuring coding challenge competence with APPS . In Proceedings of the...

    2021

  12. [12]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, and 4 others. 2024. https://doi.org/10.48550/ARXIV.2411.15124 T \" u...

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.15124 2024

  13. [13]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, and 4 others. 2025. https://arxiv.org/abs/2411.15124 Tulu 3: Pushing...

    Pith/arXiv arXiv 2025

  14. [14]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu - Hong Hoi. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/8636419dea1aa9fbd25fc4248e702da4-Abstract-Conference.html Coderl: Mastering code generation through pretrained models and deep reinforcement learning . In Advances in Neural Information Processing Systems 35: Ann...

    arXiv 2022

  15. [15]

    Joel Lehman and Kenneth O. Stanley. 2011. https://doi.org/10.1162/EVCO\_A\_00025 Abandoning objectives: Evolution through the search for novelty alone . Evol. Comput., 19(2):189--223

    doi:10.1162/evco 2011

  16. [16]

    Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/81166fbd9cc5adf14031cdb69d3fd6a8-Abstract-Conference.html No representation, no trust: Connecting representation, collapse, and trust issues in PPO . In Advances in Neural Information Processing Systems 38: Annual Conf...

    2024

  17. [17]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730--27744

    2022

  18. [18]

    Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest K

    Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest K. Ryu. 2025. https://doi.org/10.48550/ARXIV.2509.26114 Clip-low increases entropy and clip-high decreases entropy in reinforcement learning of large language models . CoRR, abs/2509.26114

    doi:10.48550/arxiv.2509.26114 2025

  19. [19]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  20. [20]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . CoRR, abs/1707.06347

    Pith/arXiv arXiv 2017

  21. [21]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://doi.org/10.48550/ARXIV.2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . CoRR, abs/2402.03300

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  22. [22]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html Reflexion: language agents with verbal reinforcement learning . In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informatio...

    2023

  23. [23]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html Learning to summarize with human feedback . In Advances in Neural Information Processing Systems 33: Annual Conference o...

    2020

  24. [24]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. 2018. http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction , 2 edition. MIT Press, Cambridge, MA

    2018

  25. [25]

    Ning Tao, Anthony Ventresque, Vivek Nallur, and Takfarinas Saber. 2024. https://doi.org/10.3390/A17070287 Enhancing program synthesis with large language models using many-objective grammar-guided genetic programming . Algorithms, 17(7):287

    doi:10.3390/a17070287 2024

  26. [26]

    Ziyu Wan, Yunxiang Li, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, and Ying Wen. 2025. https://doi.org/10.48550/ARXIV.2503.09501 Rema: Learning to meta-think for llms with multi-agent reinforcement learning . CoRR, abs/2503.09501

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.09501 2025

  27. [27]

    Muning Wen, Junwei Liao, Cheng Deng, Jun Wang, Weinan Zhang, and Ying Wen. 2024. https://arxiv.org/abs/2402.06700 Entropy-regularized token-level policy optimization for language agent reinforcement . Preprint, arXiv:2402.06700

    Pith/arXiv arXiv 2024

  28. [28]

    Huazhi Xu, Xiaoyan Luo, and Wencong Xiao. 2024 a . https://doi.org/10.1007/S11760-023-02981-6 Multi-residual unit fusion and wasserstein distance-based deep transfer learning for mill load recognition . Signal Image Video Process., 18(4):3187--3196

    doi:10.1007/s11760-023-02981-6 2024

  29. [29]

    Wujiang Xu, Wentian Zhao, Zhenting Wang, Yu - Jhe Li, Can Jin, Mingyu Jin, Kai Mei, Kun Wang, and Dimitris N. Metaxas. 2025. https://doi.org/10.48550/ARXIV.2509.22576 EPO: entropy-regularized policy optimization for LLM agents reinforcement learning . CoRR, abs/2509.22576

    doi:10.48550/arxiv.2509.22576 2025

  30. [30]

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. 2024 b . https://openreview.net/forum?id=WvFoJccpo8 Qa-lora: Quantization-aware low-rank adaptation of large language models . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . O...

    2024

  31. [31]

    Devansh Yadav and Shouvick Mondal. 2025. https://doi.org/10.1016/J.JSS.2025.112543 Evaluating pre-trained large language models on zero shot prompts for parallelization of source code . J. Syst. Softw., 230:112543

    arXiv 2025

  32. [32]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  33. [33]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, and 16 others. 2025. https://doi.org/10.48550/ARXIV.2503.14476 DAPO: an open-source LLM reinforcement learning system at scale . ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.14476 2025

  34. [34]

    Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Ziyu Guo, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Peng Gao, and Hongsheng Li. 2025. https://doi.org/10.1109/CVPR52734.2025.02669 Let's verify and reinforce image generation step by step . In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025 , page...

    arXiv 2025

  35. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  36. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.