Pith. sign in

REVIEW 4 major objections 4 minor 25 references

Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A supervised cross-entropy loss recovers the policy gradient for RLVR

desk verdict Clear writing and correct algebra, but the actual update is underspecified, the key hyperparameters are missing, and the abstract mislabels a benchmark; the empirical results are interesting enough to take through review. read the letter →

arxiv 2509.02522 v3 pith:3MQXZU63 submitted 2025-09-02 cs.CL cs.LG

Jiaming Li , Longze Chen , Ze Gong , Yukun Chen , Lu Wang , Wanwei He , Run Luo , Min Yang This is my paper · ORCID
classification cs.CLcs.LG
keywords reinforcementlearningwithverifiablerewardsimplicitactor-criticsupervisedreformulationcross-entropylosslog-probabilityratioRLOOadvantageLLMmathematicalreasoningpolicygradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PACS recasts reinforcement learning with verifiable rewards (RLVR) as a supervised classification task: the outcome reward is treated as a label, and the policy itself is trained, with a cross-entropy loss, to predict whether each of its own sampled responses is correct. The paper's central claim is that this supervised objective is not an approximation but an exact reformulation of policy optimization: the gradient decomposes into a standard policy-gradient term plus a reward-prediction correction, so one network acts as both actor and critic in a single update. If the claim holds, RLVR post-training can avoid separate value models and high-variance Monte Carlo advantages, and can borrow the stability of supervised learning. On AIME-level math benchmarks with a 7B model, PACS reports pass@256 gains of roughly 11 to 15 points over the PPO and GRPO baselines while maintaining higher policy entropy during training.

What carries the argument

The load-bearing object is the RLOO-style advantage score $\psi$ defined in Eq. 7, with a policy log-probability-ratio proxy $\hat r = \beta\log(\pi_\theta/\pi_{\rm ref})$ from Eq. 8. The load-bearing identity is the gradient decomposition of the cross-entropy loss into an actor term $l\nabla_\theta\log\pi_\theta$ and a critic term $(R-\sigma(\psi))\nabla_\theta\psi$. This identity is what turns a supervised classifier into an implicit actor-critic method. Stability relies on two supporting mechanisms: periodically hard-resetting $\pi_{\rm ref}$ to a recent snapshot of $\pi_\theta$ so the log-ratio proxy does not drift, and class-imbalance weights that rebalance correct versus incorrect samples.

What would settle it

Train PACS without the periodic reset of the reference policy, or monitor the sigmoid-transformed score against true reward on a held-out set: if gradients diverge or the score stops tracking actual correctness, the claimed actor-critic coupling is not the source of stability.

Watch

Extended reading notes

Core claim

The discovery is that a binary cross-entropy loss, whose labels are the verifiable outcome rewards and whose logits come from a policy-parameterized advantage-like score, carries the same gradient as classic policy-gradient RLVR while adding an implicit critic term. With score function $\psi(q,o_i;\pi_\theta) = \hat r(q,o_i;\pi_\theta) - \frac{1}{G-1}\sum_{j\ne i}\hat r(q,o_j;\pi_\theta)$ and reward proxy $\hat r = \beta\log(\pi_\theta/\pi_{\rm ref})$, the loss gradient is $$-\mathbb E\left[\, l(q,o;\pi_\$\theta$)\nabla_\$\theta$\log\pi_\$\theta$(o|q) + \big(R(q,o)-\$\sigma$(\psi)\big)\nabla_\$\theta$\psi(q,o;\pi_\$\theta$)\right],$$ where $l$ is the per-sample cross-entropy. The first bracket is the actor: a policy-gradient update weighted by how well the current score predicts the reward. The second bracket is the critic: a prediction-error residual that adjusts the score toward the true reward. Because both use the same parameters, the paper argues the actor and critic are coupled implicitly, eliminating the temporal mismatch of separate value networks. The paper further reports that the resulting method consistently outperforms PPO and GRPO on MATH 500, AMC23, AIME 2024, and AIME 2025 with Qwen2.5-3B and Qwen2.5-7B.

Load-bearing premise

The method's stability and performance rest on the assumption that the log-probability-ratio score keeps meaningfully measuring response quality throughout training, which the paper handles by periodically resetting the reference policy rather than proving it will hold.

Editorial extensions

If this is right

  • RLVR training can be implemented as a supervised classification loop over the policy's own rollouts, with no separate value network, no generalized advantage estimation, and no clipping ratio.
  • The same gradient structure means PACS retains the credit-assignment behavior of a policy gradient while replacing Monte Carlo advantage noise with a prediction-error signal.
  • Training dynamics reported in the paper show higher sustained gradient norms and non-collapsing entropy, so the method keeps exploring longer than PPO or GRPO.
  • On AIME 2024 and AIME 2025, the reported pass@256 gains of roughly 11 to 15 points over PPO and GRPO on a 7B model would make PACS a stronger drop-in trainer for math reasoning at moderate scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the supervised formulation is the right lens, the supervised-learning toolbox (class-imbalance weighting, curriculum ordering, label smoothing, and early stopping on validation accuracy) should transfer directly to RLVR without altering the policy-gradient structure.
  • The paper treats the reset schedule of $\pi_{\rm ref}$ as a practical choice; a testable extension is to trigger the reset adaptively from the drift of $\log(\pi_\theta/\pi_{\rm ref})$, removing the schedule hyperparameter while preserving the claimed stability.
  • The ablation results suggest the advantage estimator matters most on hard tasks: on AIME, the RLOO variant beats Dr. GRPO by several points, while on MATH 500 they nearly tie; a sparser-reward domain such as program synthesis should show a larger RLOO-versus-Dr.-GRPO gap if that pattern is real.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PACS, a method for RLVR that defines a policy-parameterized score function ψ(q,o;πθ) as an RLOO advantage over log-probability-ratio reward proxies and optimizes a binary cross-entropy loss against the verifiable binary reward. The authors derive a gradient decomposition (Eq. 6) into a policy-gradient term and a reward-estimation term and claim that this constitutes an implicit actor-critic coupling that inherently recovers the classical policy gradient update. Experiments on Qwen2.5-3B and Qwen2.5-7B across MATH-500, AMC23, AIME 2024, and AIME 2025 compare PACS with PPO and GRPO, reporting consistent improvements, particularly on AIME at high pass@k. An ablation of a weighting mechanism and a β sensitivity analysis are included, and code is released.

Significance. If the claims hold, PACS would be a simple and effective RLVR trainer with meaningful gains over PPO and GRPO on mathematical reasoning benchmarks, and the open-source release is a practical strength. However, the conceptual significance is limited: the gradient decomposition in Eq. (6) is a chain-rule identity for any on-policy differentiable objective, and the 'critic' is not an independent value estimate but a reparameterization of the same policy. The empirical results are interesting, but the theoretical interpretation, the unspecified class-imbalance weighting, and the unspecified reference-policy reset schedule currently prevent the method from being fully evaluated or reproduced.

major comments (4)
  1. [Section 3.3, Eq. (6); Section 3.4.1] Equations (3) and (6) are correct for the full expectation gradient, but they include the score-function term l(q,o;πθ)∇θ log πθ(o|q). The implementation discussion in Section 3.4.1 instead analyzes only the direct derivative through ψ: it says that minimizing the loss for a correct sample maximizes ψ and hence increases πθ(oi|q), which corresponds to the −(R−σ)∇θψ term alone. The manuscript never states whether the implementation backpropagates through ψ for fixed rollouts or computes the full expectation gradient. If the former, the implemented loss is a reward-weighted classification loss, and the central claim that PACS 'inherently recovers the classical policy gradient update' is unsupported; if the latter, the intuitive description in Section 3.4.1 is incomplete because the l∇θ log πθ term can oppose the direct term, e.g., for a correct sample with small σ the score-function term has a negative weight. This distinction is load-bearing for the paper's main claim and must be resolved explicitly.
  2. [Section 3.4.2] The class-imbalance weighting mechanism is load-bearing: the ablation in Table 2 shows large drops without it (e.g., AIME 2024 pass@32 falls from 43.90 to 32.58 for Qwen2.5-7B), yet Section 3.4.2 gives no formula for the weights, no statement of how they enter Equation (9), and no hyperparameter values. Citing King and Zeng (2001) is not sufficient to specify the method. Please provide the exact weighted loss and the weight-setting rule.
  3. [Section 3.4, Eqs. (7)–(8)] The score function depends on the log-probability ratio β log(πθ/πref), and the paper states that this ratio may grow over time and that πref is periodically hard-reset to a recent snapshot with optimizer-state reinitialization. No reset frequency, snapshot-selection rule, or optimizer-state reset procedure is given, and β is chosen empirically (β=1) via the sweep in Figure 3 rather than derived. Because ψ is the entire learning signal, the stability and behavior of PACS hinge on these unspecified choices, so the method is not yet fully defined.
  4. [Section 3.2, Eq. (1); Abstract] Calling Equation (1) a supervised learning task is misleading: the expectation is over o∼πθ(·|q), so the objective is an on-policy RL objective whose gradient decomposition in Equation (6) is a chain-rule identity for any differentiable on-policy loss. The 'critic' role is not an independent value estimate but a reparameterization of the same policy's log-probabilities, so the 'implicit actor-critic coupling' claim amounts to a naming choice rather than a new algorithmic mechanism. The abstract and Section 3.3 should be revised to avoid overclaiming.
minor comments (4)
  1. [Table 2] At pass@256, the base model outperforms PACS on AMC23 (Base 99.18 vs PACS 98.51) and AIME 2024 (Base 62.55 vs PACS 59.78); this is acknowledged in Section C.2 but should be addressed in the main results to qualify the claim of superior performance at large sampling budgets.
  2. [References] The DeepScaleR reference (Luo et al., 2025) is listed as a Notion blog without a URL or access date; please provide a complete citation.
  3. [Figure 3] The 3D heatmaps are difficult to read for the exact β and k values; a tabular version or clearer axis labels would improve interpretability.
  4. [General] There are typographical artifacts such as 'im Plicit' and 'aSupervised' in the abstract and Section 3.2; please proofread the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

Core derivation is a chain-rule identity; experiments are external, so no circularity found.

full rationale

The central claim is that optimizing the supervised cross-entropy loss in Eq. (1) inherently recovers a policy-gradient update through the actor/critic decomposition in Eq. (6). This derivation is a standard REINFORCE identity: because the loss is an expectation over outputs sampled from pi_theta, its gradient must contain the score-function term l * grad_log pi_theta, plus the direct derivative of the loss with respect to theta. The chain-rule steps in Eqs. (3)-(5) are mathematically correct for any differentiable score function psi, and the conclusion follows from the definition of the expectation rather than from assuming the target result. The score function psi in Eqs. (7)-(8) is an RLOO advantage built from beta log(pi_theta / pi_ref), with beta treated as a hyperparameter; the gradient identity does not depend on fitting psi to force the policy-gradient term. Calling the two terms in Eq. (6) 'actor' and 'critic' is a labeling choice, and the critic is not an independently fitted value function, but that is a framing issue rather than a circular prediction. The experimental claims are grounded in external benchmarks (MATH 500, AMC23, AIME 2024, AIME 2025) against PPO and GRPO baselines, with reported pass@k numbers, so the performance results are not a renamed version of the training objective. One internal inconsistency exists: Section 3.4.1 describes the loss dynamics as if only the direct derivative through psi is used, while Eq. (6) includes the additional score-function gradient term; the paper does not state which update is implemented. This is a correctness and implementation ambiguity, not a circularity, and it does not make the derivation self-fulfilling. No load-bearing self-citations were found; the cited prior work on RLOO, DPO, GRPO, and Dr. GRPO is external, and no uniqueness theorem from the present authors is invoked to force the choice of objective.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The 'implicit actor-critic coupling' is an interpretative label for the gradient decomposition, not a separate mechanism.

free parameters (3)
  • β (reward proxy scaling) = 1.0
    Selected by sweeping {0.1, 0.5, 1, 2, 10} in Figure 3; performance is sensitive to β on AIME.
  • Class-imbalance weights = not reported
    Section 3.4.2 says differential weights are assigned to correct and incorrect samples per King and Zeng (2001), but no formula or numeric values are given; the ablation shows this component is critical on AIME.
  • Reference policy reset schedule = not reported
    Section 3.4 resets πref to a recent policy snapshot and reinitializes optimizer states, but the frequency is not specified; the method's stability depends on it.
assumptions (4)
  • standard math REINFORCE gradient identity: for f(θ,o), ∇θ E_{o∼πθ}[f] = E[f ∇θ log πθ + ∇θ f].
    Used in Eq. 3 to decompose the gradient; standard calculus and probability.
  • domain assumption The verifiable reward R(q,o) is a predictable function of (q,o), so a score function can be trained to predict it.
    Central to treating rewards as cross-entropy labels; correctness is assumed learnable from text without hidden factors.
  • standard math RLOO leave-one-out baseline provides an unbiased relative advantage for i.i.d. samples within a group.
    Eq. 7 relies on this estimator; standard result from Kool et al. (2019).
  • ad hoc to paper Periodically resetting πref to a snapshot of πθ keeps the log-ratio reward proxy bounded and meaningful.
    Section 3.4 invokes this without derivation; if the ratio grows, training becomes unstable. It is a practical design choice inherited from Liu et al. (2025a).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR." pith.science (2026). https://pith.science/paper/3MQXZU63

@misc{pith2026250902522,
  author       = {Pith},
  title        = {Pith review of: Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3MQXZU63}},
  note         = {Machine review of arXiv:2509.02522}
}
read the original abstract

Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming, however existing RLVR methods often suffer from sparse reward signals and unstable policy gradient updates inherent to RL-based approaches. To address the challenges, we propose PACS, a novel RLVR framework that achieves imPlicit Actor Critic coupling via a Supervised learning framework. By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. A detailed gradient analysis shows that this supervised formulation inherently recovers the classical policy gradient update while providing more stable and efficient training. Extensive experiments demonstrate that PACS significantly outperforms strong open-source models and RLVR baselines, yielding substantial average gains of +8.26% (4B) and +9.57% (8B) over base models offering a promising avenue for LLMs post-training with verifiable rewards. Our code and data are available as open source at https://github.com/ritzz-ai/PACS.

Figures

Figures reproduced from arXiv: 2509.02522 by the authors.

Figure 1
Figure 1. Comparison between RLVR and the supervised learning reformulation, where the query [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the PACS framework. The framework consists of three main com￾ponents: (1) Reward Proxy Computation, which calculates a reward proxy rˆ based on the log￾probability ratio. (2) Group Computation, which computes RLOO-based advantage scores ψ from the reward proxies. (3) Cross-Entropy Loss, which converts the RLVR problem into a supervised learning task, optimizing a scoring function parameterized by … view at source ↗
Figure 3
Figure 3. Performance analysis of PACS with varying [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training dynamics of Qwen2.5-7B on the DeepScaleR dataset across different optimiza [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 4 canonical work pages

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall ´e, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet ¨Ust¨un, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprint arXiv:2402.14740,

    arXiv

  2. [4]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456,

    arXiv

  3. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  4. [6]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models

    Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262,

    arXiv

  5. [7]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

    arXiv

  6. [11]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864, 2025a. 10 Preprint Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero...

    arXiv

  7. [12]

    Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li

    Notion Blog. Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S 2r: Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502.12853,

    arXiv

  8. [13]

    Qwen-Team

    URL https://arxiv.org/abs/2412.15115. Qwen-Team. Qwq-32b: Embracing the power of reinforcement learning,

    arXiv

Show all 25 references
  1. [14]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  2. [16]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,

    arXiv

  3. [17]

    org/abs/2501.12599,

    URL https://arxiv. org/abs/2501.12599,

    arXiv

  4. [18]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

    arXiv

  5. [19]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118,

    arXiv

  6. [20]

    Learning to reason without external rewards

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590,

    arXiv

  7. [21]

    Ttrl: Test-time reinforcement learning

    Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084,

    arXiv

  8. [22]

    enumerate

    is employed. Through vllm, high-throughout and low-latency text generation is achieved. Consistent inference configurations are adopted across all experiments to ensure fairness and comparability of evaluation results. Hyperparameters Configuration Enable prefix caching True G...

    2024

  9. [24]

    Underlined numbers indicate the second best

    Bold numbers indicate the best performance. Underlined numbers indicate the second best. AIME 2025(pass@k) Model k = 1 2 4 8 16 32 64 128 256 PACS 13.33 18.19 23.73 29.65 36.00 42.88 49.45 54.48 58.22 - GRPO 8.39 13.01 17.69 21.78 25.77 29.86 34.34 39.26 43.73 - Dr. GRPO 13.93...

    2025

  10. [25]

    Underlined numbers indicate the second best

    Bold numbers indicate the best performance. Underlined numbers indicate the second best. The superiority of RLOO is most pronounced on the AIME 24 and AIME 25 (Tables 13 and 14). Here, the nuanced algorithmic difference between RLOO and Dr. GRPO is amplified into a clear perfo...

    2025

  11. [500]

    Underlined numbers indicate the second best

    Bold numbers indicate the best performance. Underlined numbers indicate the second best. AMC23(pass@k) Model k = 1 2 4 8 16 32 64 128 256 PACS 57.83 65.57 71.94 78.17 83.83 88.22 91.95 95.51 98.51 - GRPO 48.66 57.43 63.96 69.68 75.71 81.63 86.94 90.95 94.06 - Dr. GRPO 58.29 66...

    2024

  12. [2001]

    Wouter Kool, Herke van Hoof, and Max Welling

    doi: 10.1093/oxfordjournals.pan.a004868. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! drlStructPred@ICLR,

    doi

  13. [2017]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  14. [2021]

    Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang

    URL https://arxiv.org/abs/2107.03374. Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546,

    arXiv

  15. [2023]

    Tulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124,

    arXiv

  16. [2024]

    Vineppo: Refining credit assignment in rl training of llms

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in rl training of llms. arXiv preprint arXiv:2410.01679,

    arXiv

  17. [2025]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- b...

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.