Pith. sign in

REVIEW 4 major objections 5 minor 48 references

SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Temperature-scaled softmax group advantages keep prompt weighting bounded and outperform GRPO under identical rewards.

desk verdict Read this for the theory, which is clean and genuinely new; the empirical claims overreach because all headline numbers come from the clipped PPO surrogate the paper itself excludes from the theorems. read the letter →

arxiv 2608.09271 v1 pith:NWOPEFVB submitted 2026-08-10 cs.LG cs.AI

Jefferson Hernandez , Jaywon Koo , Zilin Xiao , Chen Wei , Vicente Ordonez This is my paper · ORCID
classification cs.LGcs.AI
keywords SoftmaxGRPOgroup-basedreinforcementlearningpromptdifficultyweightingbinaryrewardslogmoment-generatingfunctionMaxRLreasoningpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SoftmaxGRPO claims that swapping GRPO's z-score group normalization for a temperature-scaled softmax advantage fixes a known pathology: under binary rewards, GRPO weights easy prompts so strongly that its gradient signal diverges as pass rate approaches 1, while SoftmaxGRPO's weights stay bounded. The paper proves an exact finite-group population objective for binary rewards, identifies MaxRL as the zero-temperature limit, and shows that for bounded scalar rewards the large-group update exactly optimizes the log moment-generating function of reward. It also proves that no universal finite-group scalar objective exists once rewards take three or more levels, so binary rewards are special. Empirically, SoftmaxGRPO reallocates measured gradient budget toward moderate-difficulty prompts and improves over GRPO under identical rewards on verifiable and non-verifiable reasoning tasks.

What carries the argument

The central object is the temperature-scaled softmax group advantage, $w_i\propto e^{R_i/\tau}$ with centered advantage $A_i=M w_i-1$, applied to a group of $M$ rollouts per prompt. Under binary rewards the machinery reduces to a one-dimensional prompt-weighting function $\omega_{M,\tau}(p)$: conditioning on the number of other successes $S\sim\mathrm{Binomial}(M-1,p)$ gives per-sample weight gaps $\Delta_S^{(\tau)}$, whose binomial expectation is a Bernstein-polynomial weight that integrates to the exact objective $h_{M,\tau}(p)$. This is what carries the argument: the same softmax construction yields the finite-$M$ binary theorem, the MaxRL low-temperature limit, and, through concentration of the softmax denominator, the large-group log-moment-generating objective for bounded scalar rewards.

What would settle it

Simulate $M$ binary rollouts at known pass probability $p$ for a policy whose score $\nabla_\theta \log \pi_\theta$ is tractable, average the unclipped SoftmaxGRPO update over many groups, and compare with $\omega_{M,\tau}(p)\nabla_\theta p$; a mismatch would falsify Eq. (4).

Watch

Extended reading notes

Core claim

For binary rewards in the unclipped on-policy regime, SoftmaxGRPO's expected group update on a prompt with pass probability $p$ equals $\omega_{M,\tau}(p)\nabla p$, where $\omega_{M,\tau}(p)=\mathbb{E}_{S\sim\mathrm{Binomial}(M-1,p)}[\Delta_S^{(\tau)}]$ is a Bernstein polynomial with positive coefficients and is finite at $p=1$. The paper proves this weight is the derivative of an exact scalar objective $h_{M,\tau}(p)$, recovers MaxRL's truncated weighting $\omega_{M,0}(p)=(1-(1-p)^{M-1})/p$ as $\tau\to 0$, and approaches maximum-likelihood $1/p$ weighting in the joint low-temperature, large-group limit. For bounded scalar rewards, the large-group update is exactly the gradient of $\log Z_\tau(\theta;x)=\log\mathbb{E}_{\pi_\theta}[e^{R/\tau}]$. With three or more reward levels, however, the finite-group update is generically non-conservative, so no analogous scalar potential exists without extra assumptions; this is the paper's sharp boundary for clean objective-based analysis.

Load-bearing premise

The benchmark results assume that the PPO-clipped, KL-regularized update used in practice behaves like the unclipped on-policy objective whose exact properties are proven.

Editorial extensions

If this is right

  • GRPO's easy-prompt gradient blow-up is avoidable without changing the reward signal: a single temperature parameter keeps prompt weights finite at every pass rate.
  • Adjusting $\tau$ interpolates between REINFORCE-like uniform weighting and MaxRL-like hard-prompt emphasis, with maximum-likelihood weighting approached only in the joint low-temperature, large-group limit.
  • For bounded scalar rewards and large groups, SoftmaxGRPO training is exactly optimizing $\log\mathbb{E}[e^{R/\tau}]$, so weak overlap rewards acquire a well-defined exponential-utility objective.
  • The finite-group scalar objective exists only when the reward geometry is one-dimensional (binary); with three or more reward levels, no universal potential exists and per-distribution analysis is required.
  • Measured gradient allocation shifts from near-solved prompts ($p\ge 0.9$) toward moderate difficulty, which is the concrete mechanism behind the reported gains over GRPO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bounded-weighting mechanism is the source of the gains, SoftmaxGRPO should help most on tasks with many near-solved prompts, and should also reduce update variance there; that prediction is testable on any dataset by binning pass rates.
  • The appendix's effective-sample-size rule suggests an adaptive per-prompt or per-batch temperature schedule, which the paper leaves for future work; such a schedule could remove the need to hand-tune $\tau$ to reward scale.
  • The non-conservativeness result implies that with fine-grained scalar rewards, comparing training methods by their 'objective value' may be meaningless, since no scalar potential exists; a testable extension would detect cyclic gradient accumulation around closed loops in reward space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SoftmaxGRPO, a group-based policy-gradient variant that replaces GRPO's z-score normalized advantages with temperature-scaled softmax weights w_i ∝ exp(R_i/τ) and centered advantages A_i = M w_i − 1. For binary rewards the authors derive an exact finite-M population objective h_M,τ(p) whose derivative is ω_M,τ(p) (Eq. (4)), show that its low-temperature limit is the MaxRL weighting with T = M−1 (Eq. (5)), and prove that for bounded scalar rewards the large-M update optimizes the log moment-generating function of reward (Eq. (7)). They also give a finite-M counterexample showing that a universal scalar potential does not exist for rewards with three or more levels (Eq. (8)). Empirically the paper reports gains over GRPO on verifiable tasks (GSM8K, Countdown, DeepMath), non-verifiable tasks (Poetry, MeetingBank), and a gradient-allocation analysis (Table 3), with all practical runs using a clipped, reference-KL-regularized PPO surrogate (Eqs. (10)–(11)).

Significance. The theoretical core is a genuine strength: the finite-M binary-reward analysis is clean, self-contained, and explicitly derives the Bernstein-polynomial form of ω_M,τ, the closed form of h_M,τ, and the exact MaxRL limit; the large-group scalar objective and the non-conservative counterexample are also clearly proved. If the empirical bridge were solid, this would be a valuable drop-in objective family with temperature-controlled prompt-difficulty weighting, avoiding GRPO's easy-prompt blow-up. The main reservations concern the empirical support: the headline configurations are selected from an ablation sweep, and all benchmark results are obtained through a clipped/KL-regularized surrogate that the paper itself excludes from its exact theorems. These issues are fixable and do not undermine the derivations, but they need to be addressed before the practical claims can be taken at face value.

major comments (4)
  1. [§6.2, Appendix B.1, Table 5] The main GSM8K and Countdown configurations are selected as the best-performing cells of a factorial sweep: Appendix B.1 states that (M, τ) = (8, 0.1) are the best-performing cells of the sweep in Table 5. GRPO, by contrast, is reported under a single matched configuration with no equivalent sweep, even though one could sweep group size, PPO clip, and KL coefficient for GRPO. The headline differences in Table 2 (75.8 vs 73.5 on GSM8K and 58.1 vs 57.7 on Countdown) may therefore reflect favorable hyperparameter selection for SoftmaxGRPO rather than the advantage computation itself. Please report the full sweep for all methods or use a pre-selected (non-best-cell) configuration for the main comparison.
  2. [§4, Eqs. (10)–(11), Tables 2–4] The exact results of Sections 3.2–3.4 apply to the unclipped, on-policy objective in Eq. (2), whereas every benchmark result in Tables 2–4 and every gradient-allocation number in Table 3 is produced by the clipped, reference-KL-regularized surrogate of Eqs. (10)–(11). The paper acknowledges that clipping is inactive only to first order at θ = θ_old and that away from that local regime the surrogate trades objective fidelity for stability, but it never quantifies how active clipping and the KL penalty are during training. Without diagnostics such as the fraction of clipped tokens or the average KL divergence over training, the claimed reallocation and the reported gains cannot be definitively attributed to the bounded weighting ω_M,τ(p) rather than to an interaction with the trust-region machinery. Please add such diagnostics, or run a small-scale unclipped experiment that directly targets Eq. (2).
  3. [§6.3, Table 3] The gradient-allocation measurement is not reproducible as reported. The table gives fractions of token-level gradient budget by pass-rate bin, but the text does not specify how the pass rate p is estimated, from which checkpoint or training phase the gradients are taken, whether the gradients are computed under Eq. (10) or Eq. (2), or how the token-level budget is aggregated across prompts and steps. Since Table 3 is the central empirical evidence for the predicted reallocation away from near-solved prompts, this methodology needs to be specified precisely, ideally with a description of the measurement protocol in an appendix and with multiple checkpoints or seeds to show stability.
  4. [§6.5, Table 5] The Countdown ablation shows several (τ, M) configurations with collapsed (<30 tokens) or inflated (>100 tokens) response lengths, and the Pass@1 values are non-monotonic and noisy across nearby configurations (e.g., 29.8% at τ = 0.5, M = 4 vs 57.8% at τ = 0.3, M = 4). This instability suggests a strong interaction between the softmax temperature and the PPO trust region. The paper treats this as an optimization issue, but the absence of multiple seeds and variance estimates makes it difficult to assess whether the reported improvements are robust or whether they reflect particular runs of an unstable training procedure. Please report standard deviations or at least a second seed for the main configurations.
minor comments (5)
  1. [Abstract] The abstract contains a formatting defect where several words are run together: 'universalfinite-groupscalarobjectivecannotexistwithoutadditionalassumptions'. This should be corrected.
  2. [Appendix B.4] The human-judge validation uses a single human rater, so the reported human–judge correlations of 0.74 (Poetry) and 0.89 (MeetingBank) measure agreement with one rater and do not establish inter-rater reliability. This should be stated more prominently, or additional raters should be recruited.
  3. [Appendix B.2] The claim that the GRPO regression from 1.5B to 3B on Countdown is 'consistent with verifier-RL scaling behavior reported independently by Cai and Provilkov [3]' is a post-hoc citation rather than a quantitative comparison; please either report the cited numbers or soften the claim.
  4. [Appendix A.3] The adaptive-τ effective-sample-size rule is a closed-form contribution, but its behavior at the boundaries k = 0 and k = M is only mentioned in passing. A sentence explaining that the rule is undefined exactly at these boundaries and that those cases require no adaptation would be helpful.
  5. [§4] The sentence 'At θ = θ_old, clipping is inactive to first order' is stated twice, once in Section 4 and once in the 'Scope of claims' paragraph. One statement would suffice, with the second reserved for a pointer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the finite-M binary theorem and large-group scalar theorem are closed-form consequences of the method's definition and standard score identities; the PPO approximation and hyperparameter selection are explicitly disclosed rather than disguised as derivations.

full rationale

The core derivation chain is a genuine first-principles computation, not a reduction to its own inputs. Given the defining weights w_i = exp(R_i/tau)/sum_j exp(R_j/tau), the centered advantages A_i = M w_i - 1, and the unclipped objective J^uc in Eq. (2), the paper computes E[grad J^uc] for i.i.d. groups. The proof conditions on S_i ~ Binomial(M-1, p), applies the score identities E[R_i grad log pi] = grad p and E[(1-R_i) grad log pi] = -grad p, and obtains omega_{M,tau}(p) = E[Delta_S^(tau)] in Eq. (4). This is a closed-form evaluation of the definition using standard identities; the scalar potential h_{M,tau} is then constructed by integrating omega, which is the unique one-dimensional potential and not an additional assumption. The low-temperature limit in Eq. (5) is a limit of the same expression, and identifying it with MaxRL compares the derived weight to an external weighting function, not a result forced by a self-citation. The bounded-scalar-reward theorem in Eq. (7) follows from the law of large numbers and the score identity: lim E[G_M] = E[X g]/E[X] - E[g] = grad log Z_tau, an independent mathematical derivation. The non-existence counterexample in Eq. (8) is computed explicitly from cross-partials on the simplex and is therefore an independent negative result. The only places where the exact theorem does not cover the experiments are explicitly disclosed: Section 4 states that the PPO layer 'should be viewed as a trust-region approximation to the unclipped on-policy objective, not as part of the exact theorem,' and the 'Scope of claims' paragraph repeats this. That is an external-validity caveat, not circularity, because the paper does not claim the clipped surrogate is derived from omega_{M,tau}. The selected temperatures (M=8, tau=0.1 for GSM8K/Countdown) are disclosed in Appendix B.1 as the best-performing cells of the sweep in Table 5; this is transparent hyperparameter selection on the reported benchmark rather than a fitted parameter disguised as a prediction, and Table 5 itself shows a robust performance plateau at low tau. Self-citations [15], [40], and [41] appear only in the related-work survey of multimodal reasoning and are not load-bearing for the objective derivation, the limit theorems, or any uniqueness claim. No equation reduces to its own input, and no reported 'prediction' is forced by construction, so the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central mathematical claims rest only on standard score identities, i.i.d. sampling assumptions, and bounded-reward and moment conditions, all stated in the paper. The empirical claims additionally depend on hand-set hyperparameters such as tau, M, beta, and clip range, and on the PPO trust-region approximation that is not covered by the theorems. No new physical or model entities are introduced.

free parameters (5)
  • Softmax temperature tau = 0.1 for GSM8K and Countdown, 0.3 for DeepMath and non-verifiable tasks
    Controls the entire weighting family in Eq.(1); selected per task by ablation or experience, with no automatic rule used in the main runs.
  • Rollout group size M = 8 (GSM8K, Countdown, non-verifiable), 16 (DeepMath)
    The finite-M objective depends on M; chosen per task rather than derived from the theory.
  • KL coefficient beta = 1e-3
    Trust-region weight in Eq.(11); required for stable multi-step PPO and not part of the exact theorem.
  • PPO clip epsilon = 0.20 to 0.28
    Clipping range in Eq.(10); tuned jointly with temperature; not covered by the unclipped population analysis.
  • Reward blend coefficients = 0.6 F1SQuAD + 0.4 ROUGE-L; format blend 0.35/0.65
    Hand-set reward composition for weak and non-verifiable tasks; affects all comparisons in Tables 2 and 4.
assumptions (5)
  • domain assumption Rollouts within a group are i.i.d. samples from the current policy pi_theta
    Used to condition on S_i ~ Binomial(M-1,p) and prove Eq.(4); stated as the unclipped on-policy regime in Section 3.2.
  • standard math Score identity E[grad_theta log pi_theta] = 0 and S_i independent of z_i
    Core identities used in the Appendix A proof of Eq.(4).
  • domain assumption Rewards are bounded and policy scores have finite (1+eta)-moment
    Needed for the law-of-large-numbers step in Eq.(7); stated in Appendix A.1.
  • domain assumption The PPO clipped loss with reference KL approximates the unclipped on-policy objective
    Section 4 explicitly calls this a trust-region approximation, not part of the theorem; all experimental results rely on this approximation.
  • domain assumption Measured pass rates in Table 3 accurately reflect prompt difficulty
    Gradient allocation is binned by measured pass rate p, but the estimation procedure for p is not described in the main text or appendices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation." pith.science (2026). https://pith.science/paper/NWOPEFVB

@misc{pith2026260809271,
  author       = {Pith},
  title        = {Pith review of: SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWOPEFVB}},
  note         = {Machine review of arXiv:2608.09271}
}
read the original abstract

Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using only lightweight text-similarity rewards.

Figures

Figures reproduced from arXiv: 2608.09271 by the authors.

Figure 1
Figure 1. SoftmaxGRPO defines a smooth objective family over prompt difficulty. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of training dynamics in ImageNet classification under exact maximum likelihood, REINFORCE, GRPO, and SoftmaxGRPO. With sufficient rollouts, SoftmaxGRPO closely matches cross-entropy training, while REINFORCE fails to make progress from low initial success rates even at large rollout counts. Countdown is a controlled combinatorial task requiring four integers to be combined into a target value via basic ar… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 22 canonical work pages

  1. [1]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. InInternational Conference on Learning Representations, 2018. URLhttps://openreview. net/forum?id=S1ANxQW0b

    work page 2018

  2. [2]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InThe twelfth international conference on learning representations, 2024

    2024

  3. [3]

    Escapingtheverifier: Learningtoreasonviademonstrations.arXivpreprintarXiv:2511.21667, 2025

    LockeCaiandIvanProvilkov. Escapingtheverifier: Learningtoreasonviademonstrations.arXivpreprintarXiv:2511.21667, 2025

    arXiv 2025

  4. [4]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025

    arXiv 2025

  5. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [6]

    What is the objective of reasoning with reinforcement learning?arXiv preprint arXiv:2510.13651, 2025

    Damek Davis and Benjamin Recht. What is the objective of reasoning with reinforcement learning?arXiv preprint arXiv:2510.13651, 2025

    arXiv 2025

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  8. [8]

    Openvlthinker: Complex vision-language reasoning via iterative sft-rl cycles

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: Complex vision-language reasoning via iterative sft-rl cycles. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural Information Processing Systems, volume 38, pages 123817–123846. Curran Associates, Inc., 2025

    work page 2025

Show all 48 references
  1. [9]

    Cold-startreinforcementlearningwithsoftmaxpolicygradient.AdvancesinNeuralInformation Processing Systems, 30, 2017

    NanDingandRaduSoricut. Cold-startreinforcementlearningwithsoftmaxpolicygradient.AdvancesinNeuralInformation Processing Systems, 30, 2017

    2017

  2. [10]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  3. [11]

    Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    2025 arXiv

  4. [12]

    Learning to reason for long-form story generation

    Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation. InSecond Conference on Language Modeling, 2025. URLhttps://openreview.net/forum?id=dr3eg5ehR2

    2025

  5. [13]

    RLP: Reinforcement as a pretraining objective

    Ali Hatamizadeh, Syeda Nahida Akter, Shrimai Prabhumoye, Jan Kautz, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Yejin Choi. RLP: Reinforcement as a pretraining objective. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openre...

    2026

  6. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  7. [15]

    Shah, Qihua Dong, Zilin Xiao, Jaywon Koo, and Vicente Ordonez

    Ruozhen He, Nisarg A. Shah, Qihua Dong, Zilin Xiao, Jaywon Koo, and Vicente Ordonez. Beyond referring expressions: Scenario comprehension visual grounding, 2026. URLhttps://arxiv.org/abs/2604.02323

    2026

  8. [16]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical datas...

    2026

  9. [17]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. URLhttps: //openreview.net/forum?id=d7KBjmI3GmQ. 11 SoftmaxG...

    2021

  10. [18]

    MeetingBank: A benchmark dataset for meeting summarization

    Yebowen Hu, Timothy Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. MeetingBank: A benchmark dataset for meeting summarization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedingsofthe61stAnnualMeetingoftheAssociationforCo...

    2023 doi

  11. [19]

    Answer-consistent chain-of-thought reinforcement learning for multi-modal large langauge models.arXiv preprint arXiv:2510.10104, 2025

    Minbin Huang, Runhui Huang, Chuanyang Zheng, Jingyao Li, Guoxuan Chen, Han Shi, and Hong Cheng. Answer-consistent chain-of-thought reinforcement learning for multi-modal large langauge models.arXiv preprint arXiv:2510.10104, 2025

  12. [20]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  13. [21]

    Understanding r1- zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1- zero-like training: A critical perspective. InSecond Conference on Language Modeling, 2025. URLhttps://openreview. net/forum?id=5PAF7PAY2Y

    2025

  14. [22]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  15. [23]

    On-policy distillation.Thinking Machines Lab: Connectionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation

    2025 doi

  16. [24]

    General-reasoner: Advancing llm reasoning across all domains.arXiv preprint arXiv:2505.14652, 2025

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing llm reasoning across all domains.arXiv preprint arXiv:2505.14652, 2025

    2025 arXiv

  17. [25]

    Reward augmented maximum likelihood for neural structured prediction

    Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. InAdvances in Neural Information Processing Systems, volume 29, 2016

    2016

  18. [26]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. Iterative reasoning preference optimization. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=4XIKfvNYvx

    2024

  19. [27]

    Rethinking the trust region in llm reinforcement learning.arXiv preprint arXiv:2602.04879, 2026

    Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in llm reinforcement learning.arXiv preprint arXiv:2602.04879, 2026

    2026 arXiv

  20. [28]

    A Yang Qwen, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengpeng Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint, 2024

    2024

  21. [29]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openrev...

    2023

  22. [30]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling,

  23. [31]

    Optimal completion distillation for sequence learning

    Sara Sabour, William Chan, and Mohammad Norouzi. Optimal completion distillation for sequence learning. In International Conference on Learning Representations, 2019. URLhttps://openreview.net/forum?id=rkMW1hRqKX

    2019

  24. [32]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  25. [33]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [34]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  27. [35]

    Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025. 12 SoftmaxGRPO: Learning to Reason using Softmax Advantage Gr...

    2025 arXiv

  28. [36]

    Maximum likelihood reinforcement learning.arXiv preprint arXiv:2602.02710, 2026

    Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette. Maximum likelihood reinforcement learning.arXiv preprint arXiv:2602.02710, 2026

    2026

  29. [37]

    Vl-rethinker: Incentivizing self- reflection of vision-language models with reinforcement learning

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self- reflection of vision-language models with reinforcement learning. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances i...

    2025

  30. [38]

    Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz,...

    2025

  31. [39]

    Thoughts are all over the place: On the underthinking of long reasoning models

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of long reasoning models. InThe Thirty-ninth Annual Conf...

    2025

  32. [40]

    Sportr: A benchmark for multimodal large language model reasoning in sports, 2026

    Haotian Xia, Haonan Ge, Junbo Zou, Hyun Woo Choi, Xuebin Zhang, Danny Suradja, Botao Rui, Ethan Tran, Wendy Jin, Zhen Ye, Xiyang Lin, Christopher Lai, Shengjie Zhang, Junwen Miao, Shichao Chen, Rhys Tracy, Vicente Ordonez, Weining Shen, and Hanjie Chen. Sportr: A benchmark for...

    2026

  33. [41]

    Proxythinker: Test-time guidance through small visual reasoners

    Zilin Xiao, Jaywon Koo, Siru Ouyang, Jefferson Hernandez, Yu Meng, and Vicente Ordonez. Proxythinker: Test-time guidance through small visual reasoners. InThe Fourteenth International Conference on Learning Representations (ICLR), 2026

    2026

  34. [42]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  35. [43]

    DAPO: An open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaz...

    2025

  36. [44]

    STar: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022. URLhttps://openreview.net/forum?id=_3ELRdg2sgI

    2022

  37. [45]

    Kat-v1: Kwai-autothink technical report.arXiv preprint arXiv:2507.08297, 2025

    Zizheng Zhan, Ken Deng, Huaixi Tang, Wen Xiang, Kun Wu, Weihao Li, Wenqiang Zhu, Jingxuan Xu, Lecheng Huang, Zongxian Feng, et al. Kat-v1: Kwai-autothink technical report.arXiv preprint arXiv:2507.08297, 2025

    2025 arXiv

  38. [46]

    Reinforcing general reasoning without verifiers

    Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. InThe Fourteenth International Conference on Learning Representations,

  39. [2024]

    URLhttps://openreview.net/forum?id=Ti67584b98

  40. [2026]

    Please reason step by step, and put your final answer within \boxed{}

    URLhttps://openreview.net/forum?id=nnwvwge40d. A. Exact SoftmaxGRPO Derivation Thisappendixprovestheexactfinite- 𝑀 binary-rewardstatementusedinEq. (4)andgivesthepromisedclosed-form transform ℎ𝑀,𝜏. As in the main text, gradients act only on the log-likelihood terms in Eq.(2); r...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.