Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Group Relative Policy Optimization can silently stop increasing—or even lower—the likelihood of correct answers; the paper calls this Lazy Likelihood Displacement and fixes it with token-selective penalization.

desk verdict A solid extension of likelihood displacement from DPO to GRPO, with a plausible mechanism and a useful fix; the performance gains are suggestive but not yet shown to be more than noise. read the letter →

arxiv 2505.18830 v1 pith:LKDC5EVB submitted 2025-05-24 cs.LG cs.CL

Wenlong Deng , Yi Ren , Muchen Li , Danica J. Sutherland , Xiaoxiao Li , Christos Thrampoulidis This is my paper
classification cs.LGcs.CL
keywords GroupRelativePolicyOptimizationLazyLikelihoodDisplacementnegativegradienttoken-levelselectivepenalizationNTHRmathreasoningreinforcementlearningforLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Group Relative Policy Optimization (GRPO), a widely used reinforcement-learning rule for reasoning language models, can fail to increase—or even decrease—the likelihood of correct responses during training. The paper names this Lazy Likelihood Displacement (LLD), and traces it to the algorithm's uniform penalization of every token in an incorrect response: tokens that resemble correct reasoning steps carry a negative gradient that also pushes down the correct response's likelihood. Because partially correct and wrong-format responses are the worst affected, penalizing them wholesale is counterproductive. The authors propose NTHR, a token-level selective penalization that computes each negative token's influence on correct-response likelihood and reduces the penalty on the most influential tokens. Across math benchmarks and models from 0.5B to 3B parameters, GRPO+NTHR mitigates LLD and improves average accuracy.

What carries the argument

The central object is the Group Weighted Hidden Embedding Score (GWHES), a weighted sum of inner products between hidden embeddings of tokens in correct and incorrect responses, with weights given by token-level prediction-error similarity (α^- and α^+). It appears in the paper's Theorem 4.4 as the quantity that suppresses d/dt log π(y+|x). The companion object is NTHR, a per-token score s^-_{j,<k'} that measures how much a single token in an incorrect response lowers the likelihood of the group of correct responses; tokens above a threshold τ (set from the mutual influence of correct responses) get their advantage scaled down by η. Together they turn the group-level GRPO update into a token-selective penalization rule.

What would settle it

Train GRPO on a controlled set of math problems where each incorrect response is constructed to share no tokens or hidden representations with the correct response; if LLD still appears, the claimed mechanism—shared embeddings transmitting the negative penalty—is not the cause.

Watch

Extended reading notes

Core claim

The paper claims that GRPO exhibits Lazy Likelihood Displacement: after training, correct responses often show only marginal likelihood gains or outright declines. The mechanism is identified theoretically: when rewards are binary, GRPO reduces to a weighted preference optimization between correct and incorrect response groups, and the negative gradient on incorrect responses transfers to correct responses through shared token embeddings and token-level prediction-error similarity. The paper proves (Theorem 4.4) that the likelihood change of a correct response is reduced by a term—the Group Weighted Hidden Embedding Score—that grows with the inner products between hidden embeddings of positive and negative tokens. This explains why nearly-correct responses suffer most, and motivates NTHR, which downweights penalties on negative tokens whose influence score is high. Empirically, NTHR consistently outperforms vanilla GRPO on math reasoning benchmarks across model sizes.

Load-bearing premise

The load-bearing premise is that the model can be treated as an unconstrained-features softmax and that GRPO's clipping, KL term, and near-one likelihood ratios can be neglected in the online setting, with the additional assumption that all sampled responses start with distinct first tokens.

Editorial extensions

If this is right

  • GRPO and its variants, not only DPO, are vulnerable to likelihood displacement; practitioners using group-relative advantage updates should expect correct-response likelihoods to stagnate or drop.
  • LLD is concentrated in questions whose incorrect responses are nearly correct or correct in the wrong format, so whole-response penalties are a suboptimal use of negative gradients.
  • Selectively reducing penalties on influential negative tokens (NTHR) mitigates LLD and improves average math-reasoning accuracy by 0.8–2.4 percentage points across 0.5B–3B models.
  • GWHES can rank questions by LLD severity, giving a diagnostic signal during training.
  • NTHR adds little computational overhead and can be combined with dynamic sampling, making it practical for online GRPO training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mechanism suggests that any token-uniform negative pressure in advantage-based RL, including variants like DAPO and Dr.GRPO, could be corrected with the same token-selection idea; the paper only interprets these variants, it does not test that claim.
  • NTHR's threshold and scaling depend on binary rewards; extending the influence score to graded or learned rewards could broaden the method to non-verifiable tasks.
  • If embedding similarity is the carrier of LLD, then increasing diversity among sampled responses or adding explicit contrastive objectives between correct and incorrect tokens might reduce LLD without per-token scoring.
  • A natural test is to apply NTHR to code or open-ended reasoning benchmarks, where 'nearly correct' partial solutions are common; the paper restricts evaluation to mathematics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper studies GRPO and identifies a phenomenon it calls Lazy Likelihood Displacement (LLD): during GRPO training, the likelihood of correct responses can increase only marginally or even decrease. The authors connect LLD to negative-gradient effects previously studied in DPO and analyze the GRPO update under an online, unconstrained-features simplification. They derive a quantity called GWHES (Theorem 4.4), which they identify as the source of LLD, and propose a token-level method, NTHR, that downweights penalties on tokens in incorrect responses with high GWHES. Experiments on five math benchmarks across 0.5B to 3B models report average gains for GRPO+NTHR over GRPO, and an additional DeepSeek-1.5B experiment is reported as unstable due to context-window limitations. The paper also includes ablations on the threshold and penalty scale, a complexity analysis, and qualitative examples of high-NTHR tokens.

Significance. The paper addresses a timely and practically relevant issue: GRPO is widely used for LLM reasoning, and the possibility that its negative gradients inadvertently suppress correct-response likelihood is worth understanding. The extension of the DPO likelihood-displacement analysis to group-based RL is a useful conceptual contribution, and the GWHES score together with the NTHR selection algorithm are concrete and inexpensive enough to be of practical interest. The falsifiable prediction that high-GWHES samples coincide with low-likelihood-gain samples is tested in Table 1, and the comparison against random token dropping in Figure 4 is a reasonable control. The main weakness is evidentiary: Theorem 4.4 is stated more strongly than the proof supports, and the headline performance claim rests on single-run experiments with small, sometimes benchmark-negative differences and hyperparameters selected on the evaluation benchmarks. If the theoretical statement is sharpened and the empirical claims are backed by variance estimates or a validation-based selection procedure, this would be a solid contribution.

major comments (5)
  1. [§4.1 and Appendix 7.2] The proof of Theorem 4.4 derives d/dt ln π(y+|x) = (I) − (II) + (III) + (IV) in Eq. (17), but the theorem statement uses only the hidden-embedding terms (I) and (II) and claims that as Eq. (5) increases, the likelihood change has smaller magnitude. This does not follow from the proof: the unembedding terms (III) and (IV) are not shown to be constant or negligible, and a decrease in the derivative does not imply a decrease in its magnitude once the derivative crosses zero. The theorem should be restated as a claim about the hidden-embedding contribution to the derivative, or the missing terms must be bounded. This is the main theoretical basis for attributing LLD to uniform token penalization, so the statement and proof need to match.
  2. [§5.4, Table 2] The abstract and the contribution list claim "consistent performance gains" across 0.5B–3B models, but Table 2 reports a single run per condition with no standard errors, confidence intervals, or multiple seeds. Average gains are 0.8–2.4 points and some deltas are one question on 30-problem AIME24 sets (e.g., Qwen2.5-Math-1.5B 13.3→16.7; Qwen2.5-3B 6.7→10.0). In every model row at least one individual benchmark is worse under NTHR (e.g., Qwen2.5-0.5B Minerva 9.2→8.1; Qwen2.5-Math-1.5B DeepScaler row Minerva 32.4→29.4 and Olympiad 31.9→31.4), so "consistently outperforms GRPO on math benchmarks" is not supported by the table as reported. Without replication or variance estimates, the current experiments do not exclude noise as an explanation for the average gains.
  3. [Appendix 7.6, Tables 4 and 5] The hyperparameters β and η are selected using ablations on the same evaluation benchmarks that produce the reported gains, and the chosen η=2|0.5−p| is not the best variant in Table 5 for either model shown (η=1−p gives 42.66 vs 41.94 for Qwen2.5-Math-1.5B; η=p gives 36.54 vs 36.30 for Qwen2.5-3B). This is a form of test-set selection, and it weakens the generalization claim. The authors should fix hyperparameters on a validation split or show that the conclusion is insensitive to the choice of β and η across multiple seeds.
  4. [§5.3, Algorithm 1 and Fig. 4] The LLD-mitigation experiment is partly built by construction: NTHR identifies tokens with the largest GWHES and reduces their penalty, and the GWHES is exactly the term Theorem 4.4 identifies as lowering the likelihood of correct responses. Therefore Fig. 4 primarily confirms that downweighting a derivative term that is known to be negative makes the derivative larger; it does not independently validate the causal mechanism or the token-selection rule. The held-out benchmark results in Table 2 could provide that independent evidence, but they are subject to the issues in the previous two comments. The paper should either add a control that downweights a different set of tokens or explicitly limit the claim to "NTHR improves likelihood by construction; its value is shown by the benchmark comparison."
  5. [§2.2, §4, Appendix 7.2] The theoretical result depends on several strong simplifications: clipping and the KL penalty are ignored, γ≈1, the population approximation p≈P(r=1) is used in Lemma 4.2, and all responses are assumed to have distinct first tokens. The paper does not report whether these conditions hold in the actual training runs (e.g., the fraction of clipped tokens or the KL magnitude), and the distinct-first-token assumption is implausible for natural-language math reasoning where responses often start with the same tokens. The theorem's scope should be stated more carefully, and the experiments should at least check the clipping and KL magnitudes to show that the analyzed regime is the one being trained.
minor comments (6)
  1. [§2.1, Eq. (1)] The normalization factor is displayed as "1PG i=1 |yi|" without parentheses; the intended denominator should be typeset as 1/(Σ_i |y_i|) to avoid ambiguity.
  2. [§3, Fig. 1 caption and Table 1] Model names are inconsistent (Table 1 uses "Qwen-1.5B-deepseek" while Fig. 1 uses "Deepseek-1.5B"); please align naming throughout.
  3. [§5.1, text near Eq. (7)] The phrase "can negatively affect the likelihood of a group of a group of correct responses" contains a duplicated phrase; please revise.
  4. [Table 1] The Top-K overlap statistics are based on only 47 or 57 questions; reporting confidence intervals or a permutation-test p-value would make the validation of GWHES more convincing.
  5. [Appendix 7.7, Table 6] The fact that the DeepSeek-1.5B numbers are averages of the last two checkpoints because of training instability should appear in the table caption or in the main text, not only in the prose of Appendix 7.7.
  6. [§5.4, first paragraph] The text describing "an average improvement of 0.8%" should state that the numbers are percentage points, not relative percentages, to avoid ambiguity.

Circularity Check

2 steps flagged · score 5.0 of 10

Mechanism-supporting evidence is partly built into NTHR's definition; the benchmark gains remain independent.

  1. self definitional [Sec. 5.2–5.3, Eq. (7)–(8), Algorithm 1, Fig. 4]
    "As shown in Corollary 4.5, tokens with high positive NTHR values tend to cause significant reductions in the likelihood of generated correct responses. To mitigate this adverse effect, we introduce a selective penalization strategy ... that attenuates the penalty on tokens from negative responses whose NTHR scores exceed a threshold τ. ... The results in Fig. 4 show that GRPO +NTHR consistently improves likelihood change across all samples, as evidenced by the green bars surpassing the blue bars."

    Eq. (5) decomposes d/dt lnπθ(y+|x) and identifies term (II) = p− Σ α−⟨h+, h−⟩ as the negative-gradient contribution that increases laziness. Eq. (7) defines s−_j,<k′ = Σ_i Σ_k α−_k,k′⟨h_x,y+_{i,<k}, h_x,y−_{j,<k′}⟩, which is exactly the per-token summand of term (II). Algorithm 1 selects tokens with s− > τ and scales their advantage by η < 1 (η = 0 in Sec. 5.3). Because the simplified GRPO gradient in Sec. 2.2 is linear in the token advantages, attenuating these selected advantages removes precisely the terms Eq. (5) says suppress the log-likelihood of y+. Fig. 4 therefore largely restates the construction of NTHR: the method is defined by downweighting the same tokens its theory labels as LLD-causing, so the observed increase in ∆(x) is partly built in. Other terms in Eq.

  2. self definitional [Sec. 4.2, Tab. 1, Eq. (2) and Eq. (5)]
    "Theorem 4.4 suggests using ∆GWHES as a metric to identify samples that cause LLD. Here, we verify the validity of this metric experimentally. ... The results in Tab. 1 indicate that the overlap achieved using GWHES is significantly greater than that of the random ranking baseline."

    GWHES is not an independently measured covariate; it is term (II) of the very expansion of d/dt lnπθ(y+|x) given in Eq. (5)/(17). The validation ranking compares the descending order of GWHES with the ascending order of ∆(x) from Eq. (2), which is an integrated version of the same derivative. Both rankings are computed on the same questions and model checkpoints (first 100 AIME questions and the MATH subset), so a high overlap in Tab. 1 is largely a self-consistency check of the algebra used to define GWHES rather than an independent empirical confirmation that GWHES causes LLD. The test can still fail if other terms dominate, so it is not entirely vacuous, but it does not provide the kind of external support that the held-out benchmark results provide.

full rationale

The paper's central practical claim, NTHR's performance gains in Sec. 5.4 and Tab. 2, is independent of the GWHES/NTHR construction: the held-out math benchmark results are external to the embedding-based score used to select tokens, so those gains stand on their own as empirical evidence. The circularity is confined to the mechanism-supporting demonstrations. Tab. 1 validates GWHES by comparing it with the same log-likelihood derivative whose expansion defines GWHES, and Fig. 4 shows that a method defined by downweighting the GWHES-identified tokens improves that same derivative; both are largely self-consistency checks rather than independent discoveries. No load-bearing self-citation appears: the framework borrowed from [18] is corroborated by external work [17] and is not invoked as a uniqueness or external-support theorem, and the unconstrained-features assumption cites both standard and self-authored sources but is explicitly stated as an assumption. The Sec. 2.2 online simplification relies on external works [1, 7]. The evaluation concerns raised by the skeptic, such as small unreplicated deltas, hyperparameters tuned on evaluation benchmarks, and unstable DeepSeek runs, are correctness and robustness risks rather than circularity under the rules of this pass. Overall score 5 reflects partial circularity in the in-sample mechanism evidence, with independent benchmark support remaining.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the unconstrained-features model, the online simplification, and the population approximation for rewards, in addition to the method's tuned hyperparameters beta and eta. No new physical entities are introduced; LLD is a label for an empirical phenomenon and GWHES/NTHR are derived quantities.

free parameters (2)
  • beta (threshold scale) = 1 (main); ablated over {-infinity, 0, 0.1, 1}
    Scales the threshold tau in Eq (8); chosen via ablation on the same math benchmarks (Sec 7.6, Tab 4).
  • eta (penalty scale) = 2*|0.5-p|
    Downweights the advantage of selected negative tokens; chosen via ablation (Sec 7.6, Tab 5) and depends on the per-question success rate p.
assumptions (4)
  • domain assumption Assumption 4.3: Expressive neural networks can produce unconstrained embeddings h_x in R^d independent of architecture specifics.
    Used to derive Theorem 4.4; if real LLM representations are constrained, the GWHES characterization may not hold.
  • domain assumption Online training simplification: gamma approximately 1 and clipping/KL can be neglected (Sec 2.2).
    Justified by near-online sampling and prior work [1, 7], but it removes the clipping and KL terms from the loss before analysis.
  • ad hoc to paper All responses' first tokens are distinct (Appendix 7.2).
    Mild assumption introduced to simplify the gradient expansion in the proof of Theorem 4.4.
  • domain assumption Binary rewards (1 correct, 0 incorrect) and population approximation p approximately P(r=1) (Lemma 4.2).
    Used to reduce GRPO to group preference optimization; exact group estimates are replaced by population values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization." pith.science (2026). https://pith.science/paper/LKDC5EVB

@misc{pith2026250518830,
  author       = {Pith},
  title        = {Pith review of: On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LKDC5EVB}},
  note         = {Machine review of arXiv:2505.18830}
}
read the original abstract

Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide a theoretical analysis of GRPO's learning dynamic, identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.

Figures

Figures reproduced from arXiv: 2505.18830 by the authors.

Figure 1
Figure 1. We show that negative gradients can lead to small or reduced likelihood change of positive [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Inspecting negative (incorrect) samples of questions with small average likelihood change [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Key insight: Tokens of negative samples (incorrect responses) can be logically or step [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: GRPO+NTHR consistently improves likelihood change of correct responses, as indicated by the green bars exceeding the blue bars. While GRPO+Random offers only modest improvements, GRPO+NTHR consistently outperforms it, highlighting the effectiveness of NTHR in identifyi…
Figure 5
Figure 5. Figure 5: Performance across training iterations for various models, [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond the Sampled Token: Preserving Candidate Support in RLVR

    cs.AI 2025-10 conditional novelty 6.0 of 10

    SimKO, an asymmetric top-K token-level regularizer for RLVR, improves pass@K on math/logic benchmarks across K=1..256 without lowering pass@1 relative to GRPO.

Reference graph

Works this paper leans on

35 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpg: A simple and strong reinforcement learning baseline for model reasoning.arXiv preprint arXiv:2504.02546, 2025

    Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning.arXiv preprint arXiv:2504.02546, 2025

    arXiv 2025

  2. [2]

    DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

    Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thram- poulidis. Dare the extreme: Revisiting delta-parameter pruning for fine-tuned models.arXiv preprint arXiv:2410.09344, 2024

    work page Pith review arXiv 2024

  3. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  4. [4]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008, 2024

    arXiv 2024

  5. [5]

    The local elasticity of neural networks

    Hangfeng He and Weijie Su. The local elasticity of neural networks. InInternational Conference on Learning Representations, 2020

    work page 2020

  6. [6]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  7. [7]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    arXiv 2025

  8. [8]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

Show all 35 references
  1. [9]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  2. [10]

    Med-r1: Reinforce- ment learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-r1: Reinforce- ment learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    2025

  3. [11]

    Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:3843–3857, 2022

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:...

    2022

  4. [12]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025. 10

    2025 arXiv

  5. [13]

    Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog

    2025

  6. [14]

    Neural collapse with unconstrained features

    Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022

    2022

  7. [15]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive.arXiv preprint arXiv:2402.13228, 2024

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive.arXiv preprint arXiv:2402.13228, 2024

    2024 arXiv

  8. [16]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  9. [17]

    Unintentional unalignment: Likelihood displacement in direct preference optimization.arXiv preprint arXiv:2410.08847, 2024

    Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization.arXiv preprint arXiv:2410.08847, 2024

    2024 arXiv

  10. [18]

    Learning dynamics of llm finetuning.arXiv preprint arXiv:2407.10490, 2024

    Yi Ren and Danica J Sutherland. Learning dynamics of llm finetuning.arXiv preprint arXiv:2407.10490, 2024

    2024 arXiv

  11. [19]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  12. [20]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  13. [21]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  14. [22]

    Aime problem set 1983-2024, 2023

    Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL https://www.kaggle.com/ datasets/hemishveeraboina/aime-problem-set-1983-2024

    1983

  15. [23]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  16. [24]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  17. [25]

    Breaking the softmax bottleneck: A high-rank rnn language model.arXiv preprint arXiv:1711.03953, 2017

    Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank rnn language model.arXiv preprint arXiv:1711.03953, 2017

    2017 arXiv

  18. [26]

    Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  19. [27]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  20. [28]

    Advancing llm reasoning generalists with preference trees.arXiv preprint arXiv:2404.02078, 2024

    Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees.arXiv preprint arXiv:2404.02078, 2024

    2024 arXiv

  21. [29]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  22. [30]

    Implicit geometry of next-token prediction: From language sparsity patterns to model representations.arXiv preprint arXiv:2408.15417, 2024

    Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations.arXiv preprint arXiv:2408.15417, 2024. 11 7 Appendix Notation.For any time t≥0 , we use W(t) , wz(t), and hz(...

    2024 arXiv

  23. [31]

    one commercially available ten-button lock may be opened by pressing – in any order – the correct five buttons

    found small base model may not follow the format prompt well. We also use Qwen2.5-Math- 1.5B [24] to show the influence of math knowledge. For 3B model, we use Qwen-2.5-3B [23] base model. For all models, we conduct reinforcement fine-tuning using GRPO and NTHR using the same ...

  24. [32]

    From the given graph, we can observe the following: - The roots off(x)are atx= 1 andx= 3

    Graph off(x): The graph off(x)is a quadratic function. From the given graph, we can observe the following: - The roots off(x)are atx= 1 andx= 3. - The vertex of the parabola is atx= 2

  25. [33]

    Therefore: - The roots of g(x)are also atx= 1 andx= 3

    Graph ofg(x) =−f(x):The graph ofg(x)is the reflection of the graph off(x)across the x-axis. Therefore: - The roots of g(x)are also atx= 1 andx= 3. - The vertex of the parabola forg(x)is atx= 2. 3.Graph ofh(x) =f(−x):** The graph ofh(x)is the reflec- tion of the graph off(x)acr...

  26. [34]

    This simplifies to: f(x) = 0 The roots off(x)are atx= 1 andx= 3

    Intersection off(x)andg(x): Since g(x) =−f(x), the graphs off(x)andg(x)intersect at the points where f(x) =−f(x). This simplifies to: f(x) = 0 The roots off(x)are atx= 1 andx= 3. Therefore, the graphs off(x)andg(x)intersect at two points: a= 2

  27. [35]

    This implies thatf(x)is an even function, and its graph is symmetric about the y-axis

    Intersection off(x)andh(x): Since h(x) =f(−x), the graphs off(x)andh(x)intersect at the points where f(x) =f(−x). This implies thatf(x)is an even function, and its graph is symmetric about the y-axis. The roots off(x) are atx= 1 andx= 3, but these are not symmetric about the y...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.