Pith. sign in

REVIEW 3 major objections 4 minor 22 references

This paper claims that privileged likelihood shifts under opposing verdicts mostly move in the same direction, so they measure counterfactual sensitivity rather than answer-relevant learning value, and that reallocating credit away from suc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:34 UTC pith:CZ2YV4EM

load-bearing objection A real diagnostic and consistent gains, but the headline claim overreaches: CPC cannot see outcome information coded in same-sign magnitude differences, and the paper never tests decodability. the 3 major comments →

arxiv 2607.27888 v1 pith:CZ2YV4EM submitted 2026-07-30 cs.AI

Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning

Qiangqiang He , Zhongheng Wu , ZiJian Wang This is my paper
classification cs.AI
keywords counterfactual sensitivitytoken-level credit assignmentlong-CoT reasoningRLVRon-policy self-distillationGRPOcredit reallocationprivileged likelihood shifts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to test a quietly assumed premise in long-CoT RL training: that when a model is given extra 'privileged' context telling it the answer was right or wrong, the resulting changes in its next-token probabilities carry trustworthy information about which tokens mattered. By re-scoring fixed trajectories under deliberately opposite verdicts, it finds the opposite: most tokens move the same way under both verdicts, about 58% of the update signal overlaps, and the tokens that move most are substitutable connectors like 'Wait' and 'Therefore', while the digits and operators doing the actual math barely budge. The paper concludes that privileged likelihood shifts are a measure of counterfactual sensitivity—how much a token flinches at extra context—not of learning value—and builds CSCR, a minimal GRPO extension that discounts highly sensitive tokens while preserving the verifier's sign and total credit. A sympathetic reader cares because if true, a whole family of self-distillation training schemes has been optimizing the wrong quantity, and a one-line fix reallocating credit yields consistent gains across five math benchmarks on two model sizes.

Core claim

The central claim is that privileged likelihood shifts 'fail to provide reliable answer-aligned directions, while their magnitudes primarily reflect counterfactual sensitivity rather than token-level learning value.' Concretely: keeping each trajectory and prefix fixed, scoring the same token under a positive verdict and under a negative verdict produces same-sign shifts for 91–96% of the tokens that change at all; the full-vocabulary divergence signals overlap by a mean CPC of 0.583, against a CPC of exactly 0 for the opposite-sign advantages GRPO would assign. The large-shift vocabulary is dominated by surface-form tokens such as The, But, Wait, Therefore—the paper verifies this across fou

What carries the argument

Counterfactual sensitivity score s = max(|z^pos|, |z^neg|), the maximum log-likelihood shift of a sampled token under positive and negative privileged re-scoring. The paper maps s through a soft gate w = 1 for s<λ, and 1−γ(1−exp(−α(s−λ))) above, then normalizes w within each response and multiplies the GRPO advantage, so total credit per trajectory and the sign set by the verifier are preserved. The machinery is doing the work of converting the diagnosis—that magnitude tracks sensitivity, not usefulness—into a training rule without adding a critic or changing the reward direction.

Load-bearing premise

The whole diagnosis leans on treating 'sign reversal under opposite verifier conditions' as the required signature of answer-aligned token credit; if a useful privileged signal could be encoded in shift magnitude alone without flipping sign, the paper's central conclusion would not follow.

What would settle it

Take the same 400 fixed trajectories and fit a logistic probe to the full-vocabulary shift vectors z_pos and z_neg to predict which of the two conditions produced them; if the probe achieves high accuracy on held-out tokens, the shifts do carry outcome-specific information, contradicting the paper's claim that they only reflect shared sensitivity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If privileged likelihood shifts mostly lack answer-aligned polarity, then OPSD-style self-distillation objectives inherit that defect; the paper's SD-GRPO experiments show that even flipping a few shift signs collapses training into repetition.
  • CSCR's fixed credit budget means it never strengthens the reward beyond what the verifier assigned; all gains come from where credit sits, not from how much exists.
  • The same-sign dominance persists across polarized, answer-level, outcome-only, and reference-solution prompt constructions, so the diagnosis is not an artifact of one phrasing.
  • Since CSCR outperforms GRPO with the same number of policy updates, the improvement is a credit-placement effect rather than a compute or sampling effect.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's diagnosis is falsifiable by a probe: train a classifier on the full shift vectors to predict the outcome condition; if one-token shifts are highly classifiable, then the conclusion that shifts 'cannot provide answer-aligned credit' would be too strong. (The paper does not run this test—it only checks sign composition and overlap—so this is my editorial inference.)
  • A natural extension the paper leaves implicit: use the sensitivity score as a general-purpose robustness signal—e.g., reweighting training data or detecting tokens likely to be influenced by prompt manipulations, not just verifier-conditioned prompts.
  • If the 'same-sign overlap' reflects generic style shifts, the same diagnostic could be run in code-generation RLVR (with unit tests as the verifier) and should predict the same pattern; that is a testable prediction about where CSCR-style credit reallocation would help.
  • The sign-flip criterion is itself a choice: GRPO's opposite advantages are only one reference. An informative teacher might encode outcome in magnitude or in higher-order correlations; CSCR succeeds precisely by treating magnitude as noise, which is consistent with the paper's view but the paper does not test the alternative.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper challenges the premise underlying on-policy self-distillation (OPSD) for long-CoT RLVR: that dense distributional supervision from a privileged self-teacher provides answer-aligned token-level credit. Fixing on-policy trajectories and re-scoring them under positive/negative counterfactual prompts, the authors report that most affected tokens shift in the same direction under both conditions, that the induced forward-KL optimization signals overlap by ~58% (CPC), and that large shifts concentrate on surface-form tokens. They interpret this as evidence that privileged shifts measure counterfactual sensitivity rather than learning value, and propose CSCR, which attenuates credit for sensitive tokens via a thresholded exponential weight and renormalizes token advantages. CSCR is evaluated on Qwen3-1.7B/4B on five math benchmarks, outperforming GRPO and five OPSD-style baselines. Ablations show SD-GRPO (trusting shift signs) collapses, and moderate downweighting is best.

Significance. If the diagnosis is correct, the paper makes a valuable contribution: it provides a falsifiable test (CPC and token-type analysis) and a simple, effective fix (CSCR) that consistently improves over GRPO at equal update counts. The Appendix A derivation of CPC_GRPO=0 is technically correct, the same-sign pattern is replicated across four counterfactual prompt families, and the SD-GRPO collapse is a striking demonstration that sign information from privileged shifts is not reliably usable. The paper also ships a controlled comparison with five self-distillation baselines under identical update counts. The main weakness is that the diagnostic statistics do not by themselves rule out answer information encoded in magnitudes, and the hyperparameter selection and single-seed results leave the strength of the central claim partially open. With a decodability test and additional robustness evidence, this would be a solid contribution.

major comments (3)
  1. [§2.2, Eqs. (6)–(9), Table 2] The paper interprets CPC≈0.58 as evidence that privileged shifts 'fail to provide reliable answer-aligned directions,' but CPC measures only coordinate-wise sign agreement. An informative signal could be encoded in magnitude differences among same-sign coordinates. The manuscript does not test whether the full shift vectors Δ_pos_i and Δ_neg_i (Eq. 6) are linearly separable by outcome condition. The reference to CPC_GRPO=0 in Appendix A is not the appropriate null: OPSD optimizes a full conditional distribution, and an informative privileged teacher need not reverse signs. I recommend adding a decodability check (e.g., logistic regression on Δ vectors or a per-token classifier) and reporting the resulting accuracy/separation. This is load-bearing because the abstract and §5 conclusions rest on this inference.
  2. [§4.3, Fig. 5 and §4.1] The default CSCR parameters λ=0.05, α=10, γ=0.2 are chosen from the same training/validation curves used to support the claim that 'moderate downweighting is most effective.' This makes the ablation a tuning exercise rather than independent corroboration. Moreover, Fig. 5 shows upweighting with γ=0.2 also improves over GRPO, which is not fully reconciled with the diagnosis that sensitivity is noise. Please provide a principled selection rule, evaluate CSCR on a holdout setting or a separate data split, and report sensitivity to λ, α, and γ.
  3. [Table 4 and Fig. 3] The headline claim 'consistently outperforms GRPO' is based on a single run per method per model; no error bars, multiple seeds, or significance tests are reported. Some gaps are small (e.g., Qwen3-4B AMC23: CSCR 97.8 vs GRPO 96.9; Qwen3-1.7B AMC23: CSCR 88.8 vs base 84.1 but no variance). Since Mean@32 involves sampling, multiple seeds are needed to support the claim. Also, Fig. 1 caption says 'across model scales,' but §2.2 describes diagnostics only on Qwen3-1.7B; clarify whether Qwen3-4B is included.
minor comments (4)
  1. [Abstract and §5] The phrasing 'privileged shifts fail to provide reliable answer-aligned directions' is stronger than the evidence supports. Suggest softening to 'do not provide evidence of reliable answer-aligned directions' or 'appear unreliable' until the decodability test is added.
  2. [Table 3 / Appendix B] The main-text Table 3 and Appendix B.1 differ in whether whitespace tokens are excluded; Appendix C tables explicitly say 'pure-whitespace tokens are excluded' but the main table does not. Please state this consistently in all tables.
  3. [§2.2, token-type analysis] The classification of tokens into 'surface-form' vs 'problem-specific reasoning content' is qualitative and based on top-10/top-50 lists. A quantitative measure (e.g., part-of-speech statistics or overlap with vocabulary of equations) would strengthen the claim.
  4. [§4.1, Eq. (14)] The notation s_i,t = sg[max(|z_pos|,|z_neg|)] is clear, but it would help to state explicitly that z_pos and z_neg are computed with the counterfactual prompts of Table 1 and that no ground-truth answer is used during training for CSCR; this is an important point of comparison with the OPSD baselines.

Circularity Check

0 steps flagged

No significant circularity; diagnosis and method are tested against held-out benchmarks

full rationale

The paper's core claims are: (i) privileged likelihood shifts under opposing conditions show high same-sign overlap, and (ii) attenuating counterfactually sensitive tokens improves GRPO. Claim (i) is an empirical measurement: Eq. 1 defines z_pos and z_neg, Eq. 6 defines Delta_pos and Delta_neg, and Eqs. 8-9 define CPC. The reported CPC ~0.583 comes from 400 fixed trajectories and is replicated under three alternative counterfactual prompt constructions in Appendix C. Claim (ii) is evaluated on five held-out benchmarks in Table 4, providing external evidence independent of the diagnostic measure. The method and diagnosis share the counterfactual sensitivity measure, but this is not circular: the method's benefit could have failed, and the benchmarks are not derived from the diagnostic equations. Hyperparameters lambda=0.05, alpha=10, gamma=0.2 are chosen from ablations, not presented as derived predictions. Appendix A's CPC_GRPO=0 is a legitimate mathematical reference point, not a load-bearing self-citation. The paper contains no load-bearing self-citations by the present authors. The reader's concern that answer-aligned supervision could be encoded in magnitude differences rather than sign reversal describes an inferential gap in the diagnostic premise, not a step that reduces by construction to its own inputs. That is a correctness or validity concern, not circularity. The derivation chain is self-contained and externally benchmarked, so no circularity is identified.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. Its load-bearing assumptions are the sign-reversal criterion for useful token-level supervision and the representativeness of the diagnostic sample. The main free parameters are the three CSCR hyperparameters, all chosen by hand or from validation dynamics.

free parameters (3)
  • λ = 0.05
    Used both as the diagnostic significance threshold and as the CSCR sensitivity onset. Set by convention/default; diagnostic statistics are shown to be stable across thresholds, but the method threshold is not derived.
  • α = 10
    Decay rate in the CSCR weight mapping (Eq. 14). Chosen by hand/default; no theoretical justification or separate tuning protocol is reported.
  • γ = 0.2
    Maximum attenuation strength in Eq. 14. The ablation in Fig. 5 shows γ=0.2 is best, so it is effectively selected by validation/training dynamics rather than derived.
axioms (3)
  • domain assumption A useful answer-aligned token-level signal should reverse token probability directions between positive and negative outcome conditions.
    The CPC comparison in §2.2 and Table 2 treats overlap between opposing conditions as evidence of unreliability. This assumes the GRPO-antiparallel reference in Appendix A is the right benchmark for a useful privileged signal, which is not independently established.
  • domain assumption 400 on-policy trajectories from 100 DAPO-17K questions on Qwen3-1.7B are representative of long-CoT reasoning at scale.
    The main diagnostic uses this dataset/model only. Some robustness is provided by alternative prompt constructions, but not by a diverse set of domains or trajectories.
  • standard math Standard softmax derivative and local clipping-inactivity for PPO/GRPO policy gradients.
    Used in Appendix A to derive antiparallel GRPO signals and zero CPC at the on-policy reference point.

pith-pipeline@v1.3.0-daily-deepseek · 24192 in / 12667 out tokens · 137456 ms · 2026-07-31T23:34:34.933362+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) is central to improving long-CoT reasoning in large language models. Critic-free methods such as GRPO convert response-level rewards into advantages and uniformly broadcast them across tokens, overlooking their unequal contributions to the final outcome. On-policy self-distillation (OPSD) instead provides dense distributional supervision by minimizing the forward KL divergence between an unprivileged policy and a privileged self-teacher, implicitly assuming that the resulting likelihood shifts encode reliable answer-aligned information. We test this premise by fixing each sampled trajectory and re-scoring it under two opposing outcome conditions, one asserting correctness and the other incorrectness. Most affected tokens shift in the same direction under both conditions, with few sign reversals and substantial overlap in the induced optimization signals. Large shifts also concentrate on highly substitutable surface-form tokens, whereas tokens carrying problem-specific reasoning content are less sensitive. These findings show that privileged shifts fail to provide reliable answer-aligned directions, while their magnitudes primarily reflect counterfactual sensitivity rather than token-level learning value. Based on these observations, we propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages to preserve both the original credit budget and verifier-determined direction. On long-CoT mathematical reasoning benchmarks, CSCR consistently outperforms GRPO baseline with the same number of policy updates. Targeted ablations further corroborate our diagnosis: privilege-induced directions are unreliable, moderate downweighting is most effective, and stronger modulation destabilizes optimization.

Figures

Figures reproduced from arXiv: 2607.27888 by Qiangqiang He, Zhongheng Wu, Zijian Wang.

Figure 1
Figure 1. Figure 1: Counterfactual likelihood-shift statistics across model scales, computed from 400 on-policy trajectories generated for 100 questions sampled from DAPO-17K, with four trajectories per question, using 𝜆 = 0.05. (a) Proportions of tokens significant under both, exactly one, or neither privileged condition. (b) Proportions of jointly significant tokens with same-sign or opposite-sign shifts. also depends stron… view at source ↗
Figure 2
Figure 2. Figure 2: Composition of full-vocabulary entries with significant shifts under the positive privileged condition, categorized by their corresponding shifts under the negative condition across thresholds in Qwen3-1.7B. find that opposing conditions induce predominantly same￾sign shifts and substantially overlapping optimization signals, while large shifts concentrate on highly substitutable surface￾form tokens rather… view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of GRPO and CSCR on Qwen3-1.7B over 300 steps. (a) Average reward. (b) Average response length. controlled comparison, all self-distillation baselines use the ground-truth answer as privileged information, whereas CSCR uses the counterfactual conditions in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of SD-GRPO. (a) Average reward. (b) Policy entropy. (c) Average response length. (d) Flipped-token fraction [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effects of upweighting and downweighting counterfactually sensitive tokens under different modulation strengths 𝛾. (a) Average reward. (b) Policy entropy. (c) Response length. (d) Sensitive-token fraction (𝑠𝑖,𝑡 ≥ 𝜆). (e) Validation Mean@8. References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language mod… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 16 linked inside Pith

  1. [1]

    InInternational Conference on Learning Representations, volume 2024, pages 21246– 21263,

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem.On-policydistillationoflanguagemodels:Learning from self-generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246– 21263,

    2024

  2. [4]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  3. [5]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Bau- mann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    Pith/arXiv arXiv

  4. [6]

    arXiv preprint arXiv:2603.07079,

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe,YiZhou,DennisWei,NathalieBaracaldo,andKimin Lee.Entropy-awareon-policydistillationoflanguagemodels. arXiv preprint arXiv:2603.07079,

    Pith/arXiv arXiv

  5. [7]

    Vineppo: Refining credit assignment in rl training of llms.arXiv preprint arXiv:2410.01679,

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, AlessandroSordoni,SivaReddy,AaronCourville,andNico- las Le Roux. Vineppo: Refining credit assignment in rl training of llms.arXiv preprint arXiv:2410.01679,

    Pith/arXiv arXiv

  6. [8]

    Distillm: Towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898,

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898,

    Pith/arXiv arXiv

  7. [9]

    Tulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124,

    Pith/arXiv arXiv

  8. [10]

    Unifying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026a

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026a. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai ...

    arXiv 2024

  9. [11]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Pith/arXiv arXiv

  10. [13]

    MAA and Math-AI

    URL https://arxiv.org/abs/2406.06592. MAA and Math-AI. AMC23: 2023 american mathemat- ics competition benchmark. Hugging Face dataset,

    Pith/arXiv arXiv 2023

  11. [15]

    Grpo-𝑙𝑎𝑚𝑏𝑑𝑎 : Credit assignment improves llm reasoning.arXiv preprint arXiv:2510.00194,

    PrasannaParthasarathi,MathieuReymond,BoxingChen,Yufei Cui, and Sarath Chandar. Grpo-𝑙𝑎𝑚𝑏𝑑𝑎 : Credit assignment improves llm reasoning.arXiv preprint arXiv:2510.00194,

    arXiv

  12. [16]

    Privileged information distillation for language models

    Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Cac- cia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942,

    Pith/arXiv arXiv

  13. [17]

    Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  14. [18]

    Gtpo and grpo-s: Token and sequence- level reward shaping with policy entropy.arXiv preprint arXiv:2508.04349,

    Hongze Tan, Zihan Wang, Jianfei Pan, Jinghao Lin, Hao Wang, Yifan Wu, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. Gtpo and grpo-s: Token and sequence- level reward shaping with policy entropy.arXiv preprint arXiv:2508.04349,

    arXiv

  15. [19]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245,

    Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245,

    Pith/arXiv arXiv

  16. [20]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  17. [21]

    Self-distilled rlvr.arXiv preprint arXiv:2604.03128, 2026a

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Ji- aqi Wang, and Nan Duan. Self-distilled rlvr.arXiv preprint arXiv:2604.03128, 2026a. YuxiaoYang,XiaoyunWang,andWeitongZhang. Ogls-sd:On- policy self-distillation with outcome-guided logit steering for llm reasoning.arXiv preprint arXiv:2605.12400, 2026b. ...

    Pith/arXiv arXiv

  18. [22]

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang,FeiyuChen,andAdityaGrover

    URL https://huggingface.co/datasets/math-ai/aime26. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang,FeiyuChen,andAdityaGrover. Self-distilledreasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026a. Ziqi Zhao, Xinyu Ma, Liu Yang, Yujie Feng, Daiting Shi, Jingzhou He, Xin Xin, Zhaochun Ren, and Xiao...

    Pith/arXiv arXiv

  19. [23]

    We consider only the clipped policy-gradient term and omit the KL regularizer

    9 A Proof of Zero Counterfactual Pertur- bation Consistency for GRPO We establish that, at the on-policy reference point, assigning advantages of opposite signs to the same trajectory produces antiparallel GRPO optimization signals in logit space and therefore yields zero Counterfactual Perturbation Consistency (CPC). We consider only the clipped policy-g...

    2012

  20. [2023]

    A 40- problem benchmark derived from the 2023 AMC 12A and AMC 12B competitions

    URL https://huggingface.co/datasets/math-ai/amc23. A 40- problem benchmark derived from the 2023 AMC 12A and AMC 12B competitions. Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlcsd: Reinforcement learning with contrastive on-policy self-distillation.arXiv preprint arXiv:2606.11709,

    Pith/arXiv arXiv 2023

  21. [2024]

    Beyond benchmarks: MathArena as an evalua- tion platform for mathematics with LLMs.arXiv preprint arXiv:2605.00674,

    Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvalddson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: MathArena as an evalua- tion platform for mathematics with LLMs.arXiv preprint arXiv:2605.00674,

    Pith/arXiv arXiv

  22. [2026]

    URL https://arxiv.org/abs/2605. 00674. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInter- national Conference on Learning Representations, volume 2024, pages 32694–32717,

    2024