Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that GRPO shifts a model's output toward answers it can already produce, while supervised fine-tuning rewrites the model and erodes old knowledge.

desk verdict Honest, well-scoped comparison undermined by a 50x LR confound; the checkpoint analysis is a useful start but the amplify-vs-replace claim outruns the evidence. read the letter →

arxiv 2507.10616 v2 pith:MM6TYJA6 submitted 2025-07-13 cs.LG cs.AIcs.CL

Neel Rajani , Aryo Pradipta Gema , Seraphina Goldfarb-Tarrant , Ivan Titov This is my paper
classification cs.LGcs.AIcs.CL
keywords reinforcementlearningsupervisedfine-tuningGRPOchain-of-thoughtparameter-levelanalysisknowledgedegradationcapabilityamplificationpost-trainingdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training a large language model to reason on maths problems can be done by reinforcement learning through Group Relative Policy Optimization (GRPO), which scores a group of the model's own sampled answers relative to one another, or by supervised fine-tuning (SFT), which teaches the model to imitate a stronger model's worked solutions. Using the same base model, the same maths questions, and hyperparameters matched as closely as the algorithms allow, the paper finds that GRPO produces modest maths gains with small losses on knowledge-heavy benchmarks, while SFT produces larger maths gains but noticeably larger losses on MMLU, MMLU-Pro, and IFEval. The proposed reason is that GRPO mostly reinforces outputs the base model already produced, whereas SFT forces the model to adopt a new output distribution and in doing so overwrites internal knowledge. These observations give a preliminary mechanism for why RL amplifies existing capabilities while SFT replaces old skills with new ones.

What carries the argument

The central instrument is the checkpoint trace: at 20 training checkpoints the authors measure the per-token KL divergence from the base model on held-out MATH-500, and the normalized Frobenius norm of the difference for each of seven weight matrices per layer (query, key, value, output, gate, up, and down projections). This trace localises the difference between the algorithms: both concentrate change in query and key matrices, but SFT's changes are much larger and reach mid-layer MLPs. A causal-tracing result on the base model, which locates factual associations in mid-layer MLPs, supplies the link between that locality and knowledge loss. Freezing the MLP layers with high average indirect effect, namely layers 20-26, is the intervention derived from this mechanism, and it is the part of the paper that returns inconclusive results.

What would settle it

Train the same base model on the same maths problems with GRPO and SFT at equal learning rates and equal maximum completion lengths; if SFT no longer shows larger mid-layer MLP updates and larger drops on MMLU-class benchmarks, the amplification-versus-replacement claim is not supported.

Watch

Extended reading notes

Core claim

At the claim's center is a controlled comparison of two training runs on an easier K-12 maths subset, each with 20 saved checkpoints: GRPO at a learning rate of 1e-6 and SFT at 5e-5. On benchmarks, SFT after three epochs improves MATH-500 more than GRPO but drops GSM8k, MMLU-Pro, and IFEval further; per-token KL divergence from the base model rises early and plateaus high under SFT, while GRPO's divergence grows gradually and stays far lower. Parameter-level norms show that both methods change query and key weight matrices the most, but SFT's updates are much larger and extend into mid-layer MLPs, the region previously associated with storing factual associations. The paper hypothesizes that these larger mid-layer MLP updates corrupt memorised knowledge, and it tests this by freezing those MLPs during SFT, finding mixed results: GPQA:Diamond improves while other benchmarks degrade. The conclusion is explicitly preliminary: GRPO sharpens skills the base model already has, while SFT buys larger targeted gains by replacing old behaviours.

Load-bearing premise

The comparison assumes that the observed differences come from the choice of algorithm, even though GRPO was trained at a learning rate 50 times smaller than SFT and with a shorter maximum completion length.

Editorial extensions

If this is right

  • Practitioners who care about preserving general knowledge should prefer GRPO for reasoning post-training because it does less in-domain good but also less out-of-domain harm.
  • For applications where a specific maths or code skill matters more than preserving old behaviour, SFT is the higher-gain choice, and training for three epochs pushes the gain-loss trade-off further in the same direction.
  • The early, steep KL rise under SFT means the model's output identity changes quickly, so post-training safety evaluations of an SFT model may not reflect the base model's remaining capabilities.
  • The consistent pattern of largest changes to query and key matrices suggests reasoning training works largely by redirecting what tokens the model attends to, making attention-focused interventions more promising than freezing large blocks of weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the learning-rate gap is the true driver of the differences, then 'replacement' is a property of aggressive optimization rather than of SFT as a family; the paper itself flags this confound, and a matched-learning-rate experiment would settle it.
  • A testable consequence of the amplification view is that GRPO's maths gains should shrink or vanish on problems the base model cannot solve even with many samples; the paper's difficulty-filtering observations point that way, but the direct experiment on hard, long-horizon problems is not run here.
  • Because SFT's knowledge loss is attributed to mid-layer MLP updates, a sharper intervention would freeze only the specific MLP layers identified by causal tracing while allowing all other layers to train, rather than freezing the whole block, and evaluate on a broader knowledge suite.
  • The checkpoint-trace methodology could be applied to blended RL-plus-SFT schedules, which related work suggests mitigate both failure modes, to see whether the blended model's parameter signature looks more like GRPO's or SFT's.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper reports a comparative study of GRPO and SFT for math reasoning on OLMo-2-1124-7B-Instruct using the CN-K12 subset of OpenR1-Math-220k. It evaluates benchmark performance, KL divergence from the base model, and per-matrix Frobenius norm differences across 20 checkpoints; it then tests freezing MLPs or training only the query and key matrices in SFT. The authors conclude that GRPO amplifies existing capabilities while SFT replaces old knowledge with new reasoning behavior.

Significance. The question is timely and the experimental scaffolding is thoughtful: the same base model, same data, same prompt template, 20 checkpoints, and open hyperparameter details provide a useful starting point. The paper also contributes a candid account of RLVR instability and reward-function design. However, the headline mechanism claim is not supported by the evidence as presented. The GRPO and SFT runs differ in learning rate by a factor of 50 and in maximum completion length, and the parameter-level and KL results can be explained by those differences. The freezing experiments, run only for SFT, are explicitly inconclusive. The manuscript is therefore a useful preliminary exploration rather than a demonstration of the 'amplify versus replace' distinction.

major comments (5)
  1. [Section 3.2 / Figure 3] The central comparison is confounded by the 50x learning-rate gap (GRPO 1e-6, SFT 5e-5), which the authors themselves call 'clearly an important confound.' Since Adam-style updates scale parameter displacement approximately with learning rate, the larger Frobenius norms in Figure 6 and the larger KL divergence in Figure 4 are the expected mechanical consequences of a higher learning rate, not evidence of a qualitatively different update mechanism. The benchmark differences are also affected: a fair test of 'amplification versus replacement' requires either matched learning rates (for example, with stronger regularization for GRPO, or a lower-LR SFT arm) or a clear argument that the learning-rate difference is intrinsic to the algorithms. As it stands, the headline claim cannot be attributed to the algorithm identity.
  2. [Appendix A / Section 4] The GRPO runs cap completions at 1500 tokens while SFT uses up to 4096 tokens, a mismatch the authors acknowledge as 'clearly a confound.' Masked tokens do not contribute to the GRPO backward pass, so GRPO updates are computed on substantially fewer tokens; this alone can shrink the KL and norm differences, and it also teaches the policy to produce shorter outputs. The checkpoint-level analyses in Section 4 therefore do not separate algorithm from token budget. An additional run with GRPO at 4096 tokens, or with SFT truncated to 1500 tokens, is needed before the parameter-level comparison can support the mechanistic conclusion.
  3. [Sections 4.1 and 4.2] The claim that GRPO 'reinforces existing capabilities' while SFT 'rewrites' the model is a mechanistic interpretation of descriptive statistics, not a measurement of mechanism. The normalized Frobenius norms in Figures 5 and 6 differ by roughly 40-70x, matching the order of the learning-rate ratio, and no correction or sensitivity analysis is provided. The KL results in Figure 4 are likewise presented without error bars or repeated seeds, and the text notes that a stronger KL is expected from a higher learning rate. At most, the data support that the particular GRPO run changed the model less than the particular SFT run; that is a narrower statement than the title's dichotomy.
  4. [Section 5] The freezing experiments are run only on SFT and produce mixed results: freezing MLPs improves GPQA:Diamond but degrades other benchmarks, and training only query and key matrices lowers performance broadly. The authors themselves state the results are inconclusive. These experiments therefore cannot independently validate the mid-layer MLP overwriting hypothesis, and the causal-tracing motivation in Appendix D is transferred from prior work rather than directly linked to training outcomes. The conclusion should be softened accordingly, or the hypothesis should be tested with additional interventions, such as freezing other parameter subsets and checking whether GRPO behavior changes under analogous intervention.
  5. [Section 3.3 / Figure 3] All benchmark and KL comparisons come from single runs with no error bars, confidence intervals, or multiple seeds; this is acknowledged only in the Limitations section. Several reported differences are small (for example, MMLU 0.62 versus 0.59 for GRPO, and GSM8k 0.80 versus 0.78), and the abstract's 'less degradation' claim could easily be within run-to-run noise. Moreover, the headline trade-off is drawn by comparing GRPO at 1 epoch to SFT at 3 epochs; at 1 epoch, SFT does not show a greater in-domain benefit on MATH-500 (0.32 versus 0.35 baseline, while GRPO reaches 0.40). The authors should either report uncertainty and matched-epoch comparisons or explicitly restrict the conclusion to the 3-epoch SFT setting.
minor comments (4)
  1. [Section 3.1 vs Appendix B.2] The text in Section 3.1 states that format rewards are scaled by 0.2, while Appendix B.2 says 0.1; please reconcile the two values.
  2. [Section 4.2] The list of excluded matrices mentions 'query and key norm matrices,' but it would be clearer to state q_norm and k_norm explicitly, and to specify how the normalized Frobenius norm is computed (for example, relative to the norm of the corresponding base matrix).
  3. [Figures 3 and 7] The figure legends show several bars per benchmark, but the numerical values are not tabulated; adding a small table with the exact accuracies would improve readability and make the no-error-bars caveat easier for readers to assess.
  4. [Abstract and Introduction] The phrase 'similar hyperparameters' in the abstract understates the 50x learning-rate difference and the 4096-versus-1500 token-length difference; consider phrasing such as 'same data and pipeline with algorithm-specific hyperparameters' to avoid overstating the degree of alignment.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the GRPO/SFT contrast is an empirical comparison with acknowledged confounds, not a fit or self-citation renamed as a prediction.

full rationale

The paper's claims are descriptive and comparative: GRPO and SFT are run on the same questions, evaluated on external benchmarks (AIME24, MATH-500, GSM8k, GPQA:Diamond, MMLU, MMLU-Pro, IFEval), and then analyzed via directly measured KL divergences and parameter-norm differences. None of these quantities is defined in terms of the conclusion. The 'amplify versus replace' framing is explicitly presented as a hypothesis ('preliminary indication', 'we hypothesise'), not as a quantity that is its own output. The causal-tracing motivation for freezing is supported by an independent replication in Appendix D using the external ROME dataset and by prior work (Geva et al., 2021; Meng et al., 2023), and the freezing experiments are reported as inconclusive, so they do not function as a circular validation. The acknowledged learning-rate and maximum-completion-length mismatches are threats to internal validity, but a confounded comparison is not a circular one: the paper does not fit a parameter and then report that parameter as a prediction, nor does it derive the conclusion from a self-citation. The only self-citation (Huang et al., 2025, which includes author Titov) appears in related work as supporting context and is not load-bearing for the paper's own measurements. Therefore no circular step can be exhibited with the required specificity, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical, not derived, so there are no fitted parameters in the mathematical sense. The numbers listed are experimental choices that the central comparison depends on. The assumptions about mid-layer MLPs and causal tracing come from cited prior work and are imported without re-verification in this paper.

free parameters (3)
  • GRPO learning rate = 1e-6
    Chosen by tuning for training stability; differs from SFT learning rate, creating a confound the paper acknowledges.
  • SFT learning rate = 5e-5
    Chosen because lower rates gave poor SFT gains; 50x higher than GRPO, making parameter-norm comparisons confounded.
  • GRPO max completion length = 1500 tokens
    Forced by memory limits; SFT used 4096 tokens, a second confound the paper acknowledges in Appendix A.
assumptions (3)
  • domain assumption Mid-layer MLPs are key for factual associations (Geva et al. 2021; Meng et al. 2023), so larger mid-layer updates degrade knowledge benchmarks.
    Invoked in Section 4.2.2 and used to motivate the freezing experiments.
  • domain assumption KL divergence computed on MATH-500 logits is a meaningful measure of distribution shift from the base model.
    Used in Section 4.1 to compare GRPO and SFT training trajectories.
  • domain assumption The base model OLMo-2-1124-7B-Instruct, already trained with RLVR, is representative for studying reasoning post-training.
    Acknowledged in the Limitations section as a confound because the model already received RLVR training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them." pith.science (2026). https://pith.science/paper/MM6TYJA6

@misc{pith2026250710616,
  author       = {Pith},
  title        = {Pith review of: Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MM6TYJA6}},
  note         = {Machine review of arXiv:2507.10616}
}
read the original abstract

Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforcement learning (RL) and supervised fine-tuning (SFT), but their training dynamics are poorly understood. We present a comparative analysis of RL and SFT on the same maths problems with the same model and similar hyperparameters. We find that RL yields minor in-domain gains on maths and slight degradation on knowledge-intensive benchmarks like MMLU, while both trends are more pronounced in SFT. We also analyse model parameters across checkpoints, observing that both algorithms modify query and key weights the most. Meanwhile, SFT exhibits greater updates and also affects mid-layer MLPs more, leading us to hypothesise that this may have caused the out-of-domain degradation. We therefore investigate whether freezing parts of the model during training can mitigate the reduced performance on knowledge-intensive benchmarks. However, our results are inconclusive, with benefits on GPQA:Diamond and degradation on other benchmarks. Taken together, our observations provide a preliminary indication for why RL amplifies existing capabilities, while SFT replaces old skills with new ones.

Figures

Figures reproduced from arXiv: 2507.10616 by the authors.

Figure 1
Figure 1. Accuracy and format rewards of OLMo-2-1124-Instruct during GRPO on questions sampled from CN-K12-91k subset. Format rewards increase quickly as the model follows instruc￾tions correctly, while accuracy rewards increase more slowly and plateau. and recipes that worked poorly are provided in Appendix C. 3.2. Reasoning training with SFT [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The gradient norm and loss curves of OLMo-2-1124- Instruct during SFT on questions and completions from the CN￾K12-91k subset. Following our experiments with GRPO, we adopt the work￾ing configuration from GRPO to SFT. We make a best effort attempt at keeping as many hyperparameters constant as possible, including the learning rate scheduling, the total number of questions seen by the model, the batch sizes, and the … view at source ↗
Figure 3
Figure 3. Evaluation results on a wide range of benchmarks. We produce all results with extractive match, and reuse sampling parameters from training (same system prompt, temperature=0.7 and top p=0.95). Unless otherwise indicated, all benchmarks are zero-shot CoT. results, as shown in our final training run in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The KL divergence between OLMo-2-1124-7B-Instruct before training and during training on the withheld MATH-500 dataset. We find SFT to cause divergence from the base model much more quickly and to a more pronounced degree than GRPO. In line with our other findings, we …
Figure 5
Figure 5. Figure 5: The parameter-level impact of GRPO, as measured by the normalized Frobenius norm of the difference between every matrix of OLMo-2-1124-7B-Instruct before and after GRPO. An animated GIF across training is available at https://gifyu. com/image/bs3yP. R1’s CoTs, which is…
Figure 7
Figure 7. Figure 7: Evaluation results on the same benchmarks as in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Accuracy rewards of Qwen2.5-7B-Instruct during GRPO on questions sampled from OpenR1-Math-220k and the CN-K12 subset. In the former, the model continuously struggles on many of the questions, while in the latter, the model gradually exhibits much stronger performance. …
Figure 9
Figure 9. Figure 9: Accuracy and format rewards of OLMo-2-1124-Instruct during GRPO on questions sampled from CN-K12-91k subset. Format rewards increase quickly as the model follows instruc￾tions correctly, while accuracy rewards increase more slowly and plateau. We find another crucial f…
Figure 10
Figure 10. Figure 10: Grad norm of updates, format rewards and completion lengths of Qwen-2.5-7B-Instruct during GRPO on questions from CN-K12-91k subset. After successfully learning the format, the model starts spamming XML tags until reaching the maximum completion length of 2048, with e…
Figure 11
Figure 11. Figure 11: Accuracy and format rewards when using low learning rates and clipping (baseline), or high learning rates and clipping. found that only setting the “system prompt” parameter in the open-r1 YAML file did not actually modify the system prompt of the model saved locally …
Figure 12
Figure 12. Figure 12: The average indirect effect (AIE) of the MLPs of OLMo￾2-1124-7B-Instruct. We use the same exact hyperparameters and dataset as Meng et al. (2023), though not all of the ∼1200 prompts were answered correctly by OLMo-2-1124-7B-Instruct, and so only 794 are used for comp…
Figure 13
Figure 13. Figure 13: Grad norm of updates and KL divergence between the current policy and reference policy (ie. Qwen-2.5-7B-Instruct before and during during GRPO) on questions from CN-K12-91k subset. Note the log scale on the y-axis. Before finding hyperparameters with greater training …
Figure 14
Figure 14. Figure 14: Screenshot of the CLI during the unstable training run shown in [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Our training parameters. SFT parameters were identical, except for a learning rate of 5e-5 and the GRPO parameters removed. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Value Drifts: Tracing Value Alignment During LLM Post-Training

    cs.CL 2025-10 conditional novelty 6.0 of 10

    Value alignment in LLMs is set largely during supervised fine-tuning; standard preference-optimization datasets carry too little stance contrast to re-align it, but with engineered contrast algorithms differ (DPO ampl...

  2. EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

    cs.AI 2026-02 conditional novelty 5.0 of 10

    EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.

Reference graph

Works this paper leans on

48 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [3]

    org/abs/2404.14082

    URL https://arxiv. org/abs/2404.14082. Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling,

    arXiv

  2. [6]

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J

    URL https://arxiv.org/ abs/2406.02128. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  3. [8]

    Fourrier, C., Habib, N., Kydl ´ıˇcek, H., Wolf, T., and Tunstall, L

    URL https://arxiv.org/abs/2504.04022. Fourrier, C., Habib, N., Kydl ´ıˇcek, H., Wolf, T., and Tunstall, L. Lighteval: A lightweight framework for llm evaluation,

    arXiv

  4. [9]

    GeminiTeam

    URL https://arxiv.org/abs/ 2503.01307. GeminiTeam. Gemini 2.5: Our most in- telligent ai model. https://blog. google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ , 03

    arXiv 2025

  5. [10]

    Geva, M., Schuster, R., Berant, J., and Levy, O

    Accessed: 2025-04-27. Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories,

    work page 2025

  6. [11]

    URL https://arxiv.org/abs/2012.14913. GrokAI. Grok 3 beta — the age of reasoning agents. https://x.ai/news/grok-3, 02

    arXiv 2012

  7. [12]

    Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B

    Accessed: 2025-04-27. Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Ac- celerate: Training and inference at scale made sim- ple, efficient and adaptable. https://github.com/ huggingface/accelerate,

    work page 2025

  8. [13]

    org/abs/2503.22828

    URL https://arxiv. org/abs/2503.22828. Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in lan- guage models,

    arXiv

Show all 48 references
  1. [14]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J

    URL https://arxiv.org/ abs/2301.04213. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding,

    arXiv

  2. [15]

    Hochlehnert, A., Bhatnagar, H., Udandarao, V ., Albanie, S., Prabhu, A., and Bethge, M

    URL https: //arxiv.org/abs/2009.03300. Hochlehnert, A., Bhatnagar, H., Udandarao, V ., Albanie, S., Prabhu, A., and Bethge, M. A sober look at progress in language model reasoning: Pitfalls and paths to repro- ducibility,

    2009 arXiv

  3. [16]

    Huang, Z., Cheng, T., Qiu, Z., Wang, Z., Xu, Y ., Ponti, E

    URL https://arxiv.org/abs/ 2504.07086. Huang, Z., Cheng, T., Qiu, Z., Wang, Z., Xu, Y ., Ponti, E. M., and Titov, I. Blending supervised and reinforcement fine-tuning with prefix sampling,

  4. [17]

    HuggingFace

    URL https: //arxiv.org/abs/2507.01679. HuggingFace. Open r1: A fully open reproduction of deepseek-r1, January

    arXiv

  5. [18]

    LI, J., Beeching, E., Tunstall, L., Lipkin, B., So- letskyi, R., Huang, S

    URL https://arxiv.org/abs/2411.15124. LI, J., Beeching, E., Tunstall, L., Lipkin, B., So- letskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y ., Lample, G., and Polu, S. Numina- math. [https://huggingface.co/AI-MO/ Numina...

    arXiv

  6. [19]

    Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W

    URL https: //arxiv.org/abs/2305.20050. Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like training: A critical perspective, 2025a. URL https://arxiv. org/abs/2503.20783. Liu, Z., Chen, Y ., Shoeybi, M., Catanzaro, B., and Pi...

    arXiv

  7. [20]

    Ma, W., He, J., Snell, C., Griggs, T., Min, S., and Zaharia, M

    URL https://arxiv.org/abs/2407.21783. Ma, W., He, J., Snell, C., Griggs, T., Min, S., and Zaharia, M. Reasoning models can be effective without think- ing,

    arXiv

  8. [21]

    Mukherjee, S., Yuan, L., Hakkani-Tur, D., and Peng, H

    URL https://arxiv.org/abs/2202.05262. Mukherjee, S., Yuan, L., Hakkani-Tur, D., and Peng, H. Reinforcement learning finetunes small subnetworks in large language models,

    arXiv

  9. [22]

    org/abs/2505.11711

    URL https://arxiv. org/abs/2505.11711. 10 Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y ., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfi...

  10. [24]

    URL https: //arxiv.org/abs/2412.16720. OpenAI. Openai o3 and o4-mini system card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/ o3-and-o4-mini-system-card.pdf , 04

    arXiv

  11. [25]

    Prakash, N., Shaham, T

    Accessed: 2025-04-27. Prakash, N., Shaham, T. R., Haklay, T., Belinkov, Y ., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking,

    2025

  12. [26]

    Qwen-Team

    URL https:// arxiv.org/abs/2402.14811. Qwen-Team. Qwen2.5-math: The world’s leading open- sourced mathematical llms. Qwen Blog,

    arXiv

  13. [27]

    Rein, D., Hou, B

    URL https://arxiv.org/abs/ 1910.02054. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark,

    1910 arXiv

  14. [28]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y

    URL https://arxiv.org/abs/2311.12022. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models,

    arXiv

  15. [29]

    URL https://arxiv.org/abs/2402.03300. Sun, K. and Dredze, M. Amuro and char: Analyzing the relationship between pre-training and fine-tuning of large language models,

    arXiv

  16. [30]

    Tang, Y ., Zheng, K., Synnaeve, G., and Munos, R

    URL https://arxiv.org/ abs/2408.06663. Tang, Y ., Zheng, K., Synnaeve, G., and Munos, R. Optimiz- ing language models for inference time objectives using reinforcement learning,

    arXiv

  17. [31]

    org/abs/2503.19595

    URL https://arxiv. org/abs/2503.19595. Team-OLMo. 2 olmo 2 furious,

    arXiv

  18. [32]

    Tigges, C., Hanna, M., Yu, Q., and Biderman, S

    URL https:// arxiv.org/abs/2501.00656. Tigges, C., Hanna, M., Yu, Q., and Biderman, S. Llm circuit analyses are consistent across training and scale,

    arXiv

  19. [33]

    von Werra, L., Belkada, Y ., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallou´edec, Q

    URL https://arxiv.org/abs/2407.10827. von Werra, L., Belkada, Y ., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallou´edec, Q. Trl: Transformer reinforcement learn- ing. https://github.com/huggingface/trl,

    arXiv

  20. [34]

    Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W

    URL https://arxiv.org/abs/2211.00593. Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi- task language underst...

    arXiv

  21. [35]

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D

    URL https://arxiv.org/abs/2406.01574. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models,

    arXiv

  22. [36]

    Yan, J., Li, Y ., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y ., and Zhang, Y

    URL https://arxiv.org/abs/ 2201.11903. Yan, J., Li, Y ., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y ., and Zhang, Y . Learning to reason under off-policy guidance,

    arXiv

  23. [37]

    URL https://arxiv.org/abs/ 2504.14945. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Fan, T., Liu, G., Liu, L., Liu, X., Lin, H., Lin, Z., Ma, B., Sheng, G., Tong, Y ., Zhang, C., Zhang, M., Zhang, W., Zhu, H., Zhu, J., Chen, J., Chen, J., Wang, C., Yu, H., Dai, W....

    arXiv

  24. [38]

    Yue, Y ., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y ., Song, S., and Huang, G

    URL https://arxiv.org/ abs/2503.14476. Yue, Y ., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y ., Song, S., and Huang, G. Does reinforcement learn- ing really incentivize reasoning capacity in llms beyond the base model?,

    arXiv

  25. [39]

    Zhao, R., Meterez, A., Kakade, S., Pehlevan, C., Jelassi, S., and Malach, E

    URL https://arxiv.org/ abs/2504.13837. Zhao, R., Meterez, A., Kakade, S., Pehlevan, C., Jelassi, S., and Malach, E. Echo chamber: Rl post-training amplifies behaviors learned in pretraining,

    arXiv

  26. [40]

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y ., Zhou, D., and Hou, L

    URL https: //arxiv.org/abs/2504.07912. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y ., Zhou, D., and Hou, L. Instruction-following evalu- ation for large language models,

    arXiv

  27. [41]

    nvcr.io/nvidia/cuda:12.0.0-cudnn8-devel- ubuntu22.04

    URL https: //arxiv.org/abs/2311.07911. 11 Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them A. Reproducibility The experiments for this investigation were run on a ma- chine with 8 H100 GPUs via the Edinburgh International Data Facility (EIDF) and the...

    arXiv

  28. [42]

    for both algorithms. This allows us to keep large parts of the pipeline identical: intricate details like dataset pre-processing or data- parallel model loading and training are all constant between both algorithms. Matching the effective global batch size helps us ensure that...

    2025

  29. [43]

    In the former, the model continuously struggles on many of the questions, while in the latter, the model gradually exhibits much stronger performance

    Accuracy rewards of Qwen2.5-7B-Instruct during GRPO on questions sampled from OpenR1-Math-220k and the CN-K12 subset. In the former, the model continuously struggles on many of the questions, while in the latter, the model gradually exhibits much stronger performance. Note tha...

    2025

  30. [44]

    Note that Shao et al

    For the full set of hyperparameters used for training, refer to Appendix A. Note that Shao et al. (2024) actually perform multiple gradient updates per global step (this corresponds to the hyperparameter µ in their original GRPO algorithm, and 6See https://github.com/huggingfa...

    2024

  31. [45]

    Grad norm of updates, format rewards and completion lengths of Qwen-2.5-7B-Instruct during GRPO on questions from CN-K12-91k subset. After successfully learning the format, the model starts spamming XML tags until reaching the maximum completion length of 2048, with explosions...

  32. [46]

    system prompt

    Accuracy and format rewards when using low learning rates and clipping (baseline), or high learning rates and clipping. found that only setting the “system prompt” parameter in the open-r1 Y AML file did not actually modify the system prompt of the model saved locally or pushe...

    2025

  33. [47]

    We find OLMo-2-1124-7B-Instruct to exhibit similar phe- nomena as the models examined by Meng et al. (2023). There seems to be an ’early site’ of MLPs that are causally implicated on the last subject token, and a ’late site’ of MLPs in upper layers with high average indirect e...

    2023

  34. [48]

    We use the same exact hyperparameters and dataset as Meng et al

    The average indirect effect (AIE) of the MLPs of OLMo- 2-1124-7B-Instruct. We use the same exact hyperparameters and dataset as Meng et al. (2023), though not all of the ∼1200 prompts were answered correctly by OLMo-2-1124-7B-Instruct, and so only 794 are used for computing AI...

    2023

  35. [2020]

    Cabannes, V ., Arnal, C., Bouaziz, W., Yang, A., Charton, F., and Kempe, J

    URL https:// arxiv.org/abs/2005.14165. Cabannes, V ., Arnal, C., Bouaziz, W., Yang, A., Charton, F., and Kempe, J. Iteration head: A mechanistic study of chain-of-thought,

    2005 arXiv

  36. [2021]

    EssentialAI, :, Shah, D

    https://transformer- circuits.pub/2021/framework/index.html. EssentialAI, :, Shah, D. J., Rushton, P., Singla, S., Parmar, M., Smith, K., Vanjani, Y ., Vaswani, A., Chaluvaraju, A., Hojel, A., Ma, A., Thomas, A., Polloreno, A., Tanwer, A., Sibai, B. D., Mansingka, D. S., Shiva...

    2021

  37. [2022]

    https://transformer-circuits.pub/2022/in-context- learning-and-induction-heads/index.html. OpenAI. Openai o1 system card,

    2022

  38. [2023]

    Brown, T

    URL https://arxiv.org/abs/2304.01373. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter,...

    arXiv

  39. [2024]

    Allal, L

    Accessed: 2025-04-27. Allal, L. B., Lozhkov, A., Bakouch, E., Bl ´azquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydl ´ıˇcek, H., Lajar´ın, A. P., Srivastav, V ., Lochner, J., Fahlgren, C., Nguyen, X.-S., Fourrier, C., Burtenshaw, B., Larcher, H., Zhao, H., Zakka, C.,...

    2025 arXiv

  40. [2025]

    Bereska, L

    Accessed: 2025-04-27. Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety – a review,

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.