Pith. sign in

REVIEW 3 major objections 5 minor 38 references

On-Policy Self-Distillation without Any Supervision

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A language model can self-distill without any supervision by using its own majority-vote answer as the teacher's reference, and the paper reports this label-free variant outperforms supervised on-policy self-distillation on the evaluated…

desk verdict Genuinely new label-free self-distillation recipe with a thorough experimental package, but its headline claim of beating supervised OPSD rests on a token-budget mismatch that should be fixed before trusting the comparison. read the letter →

arxiv 2608.06296 v1 pith:ZJYZJGWD submitted 2026-08-06 cs.LG

Yijiang Li , Bingyang Wang , Yijun Liang , Yunjie Tian , Di Fu , Nuno Vasconcelos This is my paper
classification cs.LG
keywords unsupervisedself-distillationon-policydistillationself-consistencymajorityvotingpseudo-labelingtoken-levelLLMpost-trainingmathematicalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that the ground-truth solution in on-policy self-distillation is unnecessary: a model can construct its own privileged context from internal consistency. The proposed method, U-OPSD, samples multiple rollouts per prompt, takes the plurality answer as a pseudo-solution when it clears a self-consistency threshold, conditions a teacher distribution on the shortest agreeing rollout, and distills that distribution into the student along prefixes of disagreeing rollouts. If this is right, post-training for reasoning no longer requires annotated solutions, environmental rewards, or a stronger teacher in the regime where the base model is competent but imperfect. The paper reports that the label-free variant matches or beats supervised SFT, GRPO, and OPSD with ground-truth labels on five competition-math benchmarks across six Qwen3 configurations.

What carries the argument

The central object is the consensus-conditioned teacher: a stop-gradient copy of the student policy conditioned on a pseudo-solution $y^+$ that is the shortest rollout agreeing with the plurality vote over $G=8$ self-sampled rollouts, gated by a self-consistency threshold $\tau=1/2$. The mechanism turns agreement into dense token-level supervision by evaluating $D_\beta$, in the $\beta\to0$ limit the forward KL divergence, at every prefix of the model's own disagreeing rollouts; this tells the student what its solution-aware self would do at exactly the positions where its trajectories diverge from consensus. That is how the model corrects itself without any external labels.

What would settle it

Take a held-out set with known gold answers, compute the accuracy of the model's plurality-vote pseudo-labels on it, and run U-OPSD on problems whose pseudo-label is wrong: the method predicts that distilling the teacher conditioned on those wrong pseudo-labels lowers accuracy below base, so measuring that drop (or its absence) settles whether consensus is a reliable correctness signal.

Watch

Extended reading notes

Core claim

The discovery, stated on the paper's own terms, is that the privileged context of on-policy self-distillation can be endogenous. Equation (5) replaces the gold solution with the shortest agreeing rollout $y^+$ selected from the model's own rollouts by plurality vote under threshold $\tau=1/2$, and applies a token-level divergence $D_\beta$ between the solution-conditioned teacher $\bar{\pi}(\cdot|x,y^+,y^-_{<n})$ and the student $\pi_\theta(\cdot|x,y^-_{<n})$ along prefixes of disagreeing rollouts. Prompts whose vote does not clear the threshold, or whose rollouts all agree, contribute no gradient, so training concentrates on the model's competence frontier. With this substitution, U-OPSD reports gains of 8.5 and 10.7 points over base on Qwen3-4B and Qwen3-8B in non-thinking mode, and results that match or exceed supervised OPSD and GRPO while using no ground-truth solutions.

Load-bearing premise

The load-bearing premise is that a model's own majority-vote answer is usually correct enough to serve as the teacher reference, because a wrong plurality answer would be actively distilled into the student along the disagreeing rollouts.

Editorial extensions

If this is right

  • A reasoning model can be post-trained on unlabeled problem statements alone, so annotation cost is no longer a bottleneck in this regime.
  • U-OPSD improves Qwen3-4B and Qwen3-8B by 8.5 and 10.7 points over base on the five-benchmark average in non-thinking mode without any gold labels.
  • It exceeds supervised OPSD by 3.2 and 2.3 points in non-thinking mode and matches it in thinking mode, while surpassing GRPO by up to 1.1 points.
  • Training automatically focuses on prompts with a confident consensus but disagreeing rollouts, eliminating the need for external difficulty labels or curriculum scheduling.
  • The full-vocabulary forward-KL objective is load-bearing: reducing it to sampled-token policy-gradient supervision costs 13.7 points, and JSD or reverse KL stagnates or diverges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because majority-vote pseudo-labels are bounded by the base model's own answer accuracy, U-OPSD amplifies existing competence rather than creating new capability, and systematic base-model biases would be reinforced rather than corrected.
  • Editorial inference: the same consensus-as-teacher mechanism should transfer to open-ended generation if exact-match voting is replaced by semantic or soft agreement, which the paper itself identifies as the key extension.
  • Editorial inference: the ablations suggest the default operating point is conservative, since $G=12$ rollouts, an EMA-updated teacher, and a lower threshold all improve the reported numbers, so the headline gains likely understate what the mechanism can do with more generation budget.
  • Editorial inference: the collapse under reverse KL and stagnation under JSD indicate that the divergence choice interacts with pseudo-label noise, so testing robust divergences under deliberately corrupted votes is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Unsupervised On-Policy Self-Distillation (U-OPSD), a training method that removes the need for ground-truth solutions, external teachers, or environmental feedback in on-policy self-distillation. For each unlabeled prompt, the method samples several rollouts from a stop-gradient copy of the policy, takes a majority vote over parsed answers, and uses the shortest agreeing rollout as a pseudo-solution. It then conditions a teacher distribution on this pseudo-solution and distills it into the student along prefixes of disagreeing rollouts, using a token-level divergence. Experiments on five mathematical reasoning benchmarks with Qwen3 models at 4B and 8B scales, in both non-thinking and thinking modes, and on two instruction-tuned checkpoints, report consistent gains over the base model and claim that U-OPSD matches or exceeds supervised baselines including SFT, GRPO, and OPSD, all without external labels.

Significance. If the central claim holds, the paper makes a useful conceptual contribution: it suggests that a model's own consensus, rather than ground-truth solutions, can provide the privileged context needed for on-policy self-distillation. The paper is honest about its scope, includes an OPSD reproduction check, and provides a broad set of ablations on the teacher reference, distillation target, rollouts per prompt, confidence threshold, divergence family, and teacher update rule. The main comparative claim, however, is currently supported by a resource-unmatched comparison and by best-checkpoint numbers without seed-replicated error bars. The idea is interesting and the empirical design is substantial, but the evidence does not yet establish that consensus substitutes for ground truth in the way the conclusion states.

major comments (3)
  1. [Section 4.1, Tables 1–2] The headline comparison between U-OPSD and supervised OPSD is not resource-matched. U-OPSD uses G=8 rollouts per prompt and a 4,096-token completion budget, while OPSD uses one rollout per prompt and a 1,024-token budget; with the same number of prompts per step, this is a 32-fold difference in generated tokens, reflected in the 9-hour versus 3.5-hour runtimes reported in Appendix D. The statement in Section 4.3 that 'G=8 ... makes the comparison in Tables 1 and 2 cost-matched' is contradicted by the setup text, and the cited Zhao et al. [2026] result concerns supervised OPSD, not U-OPSD's voting and distillation pipeline. Since Table 4 shows that the teacher trace content is decisive (full trace 57.10 vs label-only 43.40) and Section 4.3 reports a 96.3% parse rate at 4,096 tokens, the token budget is not a nuisance parameter: the reported margins over OPSD could be a compute and sequence-length effect rather than evidence that consensus replaces ground truth. I request matched-budget runs, such as U-OPSD at 1,024 tokens or OPSD at 4,096 tokens with equal per-prompt rollout counts, or an explicit argument that the comparison is invariant to these choices.
  2. [Section 4.1, Section 5] All main results are reported as best checkpoints over steps 25–150 without seed-replicated error bars. The paper's own limitation section states that 'seed-replicated error bars for every training setup are pending and will be added in a revision.' On the 30-problem benchmarks evaluated with 12 samples, checkpoint-to-checkpoint variation can exceed the margins that distinguish methods; for example, Table 2's 8B thinking comparison is 77.99 versus 77.97, and Table 3's pass@1 differences are within the reported population standard deviation. Before the central comparative claim can be evaluated, the authors should provide seed-replicated means and error bars for at least the headline U-OPSD/OPSD comparisons, or fixed-step results rather than best-of-six checkpoint selection.
  3. [Section 4.3, Eq. (5)] The method's correctness rests on the assumption that majority-vote pseudo-labels are correct often enough to serve as teacher context, but the only direct evidence is a 64-prompt probe with 86.7% gold-match. When the pseudo-answer is wrong, Eq. (5) distills the wrong answer along every disagreeing rollout, so the 13.3% error rate is not inert noise. The paper acknowledges in Section 5 that consensus is 'bounded by that answer's accuracy,' but it does not quantify how pseudo-label errors affect training or show robustness to corrupted votes, for example by weighting vote margin. I do not regard this as fatal, but the claim that an unlabeled pipeline outperforms the supervised one needs either a larger pseudo-label-quality study or a sensitivity analysis with deliberately corrupted pseudo-labels.
minor comments (5)
  1. [Abstract and Algorithm 1] The abstract says the method 'distills it into prefixes of the model's longest incorrect completion,' but Algorithm 1's default selects 'one uniform disagreeing' rollout, with longest-1 appearing only as a variant in Table 5; please align the abstract with the actual default.
  2. [Section 4.3] The phrase 'makes the comparison in Tables 1 and 2 cost-matched' is inaccurate because Section 4.1 states that OPSD uses 32 prompts with a single rollout while U-OPSD uses G=8 rollouts per prompt; please rephrase or provide the actual per-step generation budgets.
  3. [Table 3 caption] The 'mean ± population standard deviation' in Table 3 is computed over three single-sample evaluation runs, not over independent training seeds; please state this explicitly so readers do not mistake it for seed variance.
  4. [Figure 4] The right panel legend appears garbled in the caption ('fixed EMA 0.99 EMA 0.995 EMA 0.999'); please clarify the labels for the teacher-update ablation.
  5. [Equation (5)] The notation W_k in Eq. (5) is not defined in the text; please replace it with the notation used for the selected disagreeing rollout set, such as B^-_x, or define it explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the endogenous teacher signal is the method's stated design, and the reported gains are validated against external benchmarks.

full rationale

The central mechanism is self-referential by construction: Eq. (5) is exactly the supervised OPSD objective of Eq. (4) with the ground-truth solution y* replaced by the pseudo-solution y+, which is the model's own shortest agreeing rollout. However, the paper does not derive benchmark improvement from this substitution; it treats consensus-correctness as an empirical assumption, checks it on 64 prompts (86.7% pseudo-label accuracy in Section 4.3), and measures final performance on held-out AIME/MATH/AMC benchmarks. The Section 5 limitation that majority-vote supervision is 'bounded by that answer's accuracy' confirms that the authors do not claim the endogenous signal is a priori equivalent to ground truth. There is one minor self-citation (Wu et al. 2025 includes co-author Y. Tian) in a related-work enumeration, but it is not load-bearing. The U-OPSD versus OPSD comparison is potentially confounded by the 4,096-token vs 1,024-token completion budget and by best-checkpoint selection, yet those are experimental-design concerns rather than circularity.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The method's central contribution is a training objective with no fitted parameters in the mathematical sense, but several hand-set hyperparameters (G, tau, token budget, selection rules, divergence choice) are tuned on the evaluation benchmarks, and the core correctness signal is an assumption about self-consistency. No invented physical entities are introduced.

free parameters (8)
  • G (rollouts per prompt) = 8 (default); 12 is 4.7 points better
    Controls vote resolution and the pool of disagreeing rollouts; chosen to match OPSD's budget, not by principle. Figure 4 middle and Table 11.
  • tau (self-consistency threshold) = 0.5 (default); 0 is best in ablation
    Filters which prompts get a pseudo-label. The paper's own ablation (Figure 4 left, Table 10) shows tau=0 outperforms the default, so the threshold is hand-set and not justified by the data.
  • Max completion length = 4096 tokens vs OPSD's 1024
    Needed for rollouts to reach a boxed answer; drives runtime to about 9 hours versus 3.5 hours, making the comparison not compute-matched (Appendix D).
  • Teacher reference selection = shortest agreeing rollout (default)
    Algorithm 1 defaults to shortest; Table 4 shows longest agreeing rollout is better (59.00 vs 57.10). The abstract also says 'shortest pseudo-solution'.
  • Distillation target selection = one uniform disagreeing rollout (disagree-1)
    Algorithm 1 default; abstract says 'longest incorrect completion' but longest-1 is a separate variant. Table 5 shows disagree-all is worst.
  • Divergence D_beta = forward KL, beta=0, full vocabulary
    Follows OPSD; Table 6 shows top-k=100 truncation reaches 59.01 versus full-vocabulary 57.10, so the default is not the best configuration found.
  • Teacher update rule = fixed/frozen teacher (default)
    OPSD setting; Figure 4 right and Table 12 show EMA 0.995 gives 2.4 points more at best checkpoint. The paper uses the frozen teacher in main tables.
  • LoRA / optimizer hyperparameters = rank 64, alpha 128, lr 5e-6, temperature 1.1, top-p 0.95, top-k 20
    Standard hand-set training hyperparameters from the OPSD recipe, reported in Section 4.1; no sensitivity analysis except learning rate in Table 13.
assumptions (6)
  • domain assumption Agreement among independently sampled rollouts is a proxy for correctness.
    Section 3.2: 'agreement among multiple independently sampled rollouts provides an endogenous confidence signal.' Measured 86.7% pseudo-label accuracy on 64 prompts; the method's benefit depends on this.
  • domain assumption The teacher conditioned on a pseudo-solution y+ provides a useful target distribution for the student on disagreeing rollouts.
    Eq. (5) assumes the divergence term is corrective; if y+ is wrong, this actively trains away from the disagreeing, potentially correct answer. Section 5 limits: 'bounded by that answer's accuracy.'
  • domain assumption Answer extraction and canonicalization recover the intended final answer from rollouts.
    Appendix C describes parsing boxed answers; 96.3% extraction rate at the 4096-token budget. Invalid rollouts are excluded, so extraction failures change the vote distribution.
  • standard math Standard probability and divergence machinery (KL, JSD, stop-gradient, importance ratios) behaves as expected.
    Used throughout Section 3; background from prior OPD/OPSD literature.
  • domain assumption The 30k OpenThoughts subset contains no leakage of the evaluation benchmarks into the unlabeled problem statements.
    Training prompts come from OpenThoughts; evaluation is on AIME, MATH, AMC, and HMMT. The paper does not report contamination checks between training set and benchmarks.
  • domain assumption The base model is competent enough that majority vote is informative and weak enough that disagreement remains to correct.
    Section 6: 'a base model competent enough to vote well and fallible enough to have headroom'; this is not proven, only inferred from results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-Policy Self-Distillation without Any Supervision." pith.science (2026). https://pith.science/paper/ZJYZJGWD

@misc{pith2026260806296,
  author       = {Pith},
  title        = {Pith review of: On-Policy Self-Distillation without Any Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZJYZJGWD}},
  note         = {Machine review of arXiv:2608.06296}
}
read the original abstract

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose Unsupervised On-Policy Self-Distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo-solution by majority vote under a self-consistency threshold. It then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO. On AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at the 4B and 8B scales, respectively, and outperforms OPSD by an average of 3.2% and 2.3%. In thinking mode, U-OPSD remains on par with OPSD, outperforming it by 0.9% at 4B and matching it at 8B, while surpassing GRPO by 0.7% and 1.1%, respectively.

Figures

Figures reproduced from arXiv: 2608.06296 by the authors.

Figure 1
Figure 1. Comparison between OPSD / SDFT with ground-truth solution or ICLs (left), SDPO with [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of Unsupervised On-policy Self-Distillation ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training curves of the Qwen3-4B runs quoted in Tables 1 and 2, on AIME24, AIME25 and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Configuration ablations, on Qwen3-8B non-thinking with longest-1. The vertical axis [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 5 canonical work pages

  1. [2026]

    URL https://arxiv.org/abs/ 2603.25562

    doi: 10.48550/arXiv.2603.25562. URL https://arxiv.org/abs/ 2603.25562. Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. InProceedings of the 35th International Conference on Machine Learning, pages 1607–1616,

    arXiv doi:10.48550/arxiv.2603.25562

  2. [3]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    arXiv

  3. [5]

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al

    URLhttps://arxiv.org/abs/2306.08543. Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. OpenThoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178,

    arXiv

  4. [6]

    URL https://arxiv.org/abs/2506.04178

    doi: 10.48550/arXiv.2506.04178. URL https://arxiv.org/abs/2506.04178. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (ReST) for language modeling. arXi...

    arXiv doi:10.48550/arxiv.2506.04178

  5. [8]

    Large language models can self-improve.arXiv preprint arXiv:2210.11610,

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve.arXiv preprint arXiv:2210.11610,

    arXiv

  6. [10]

    Aditya Prakash, Josiah Hester, Jindong Wang, and Srijan Kumar

    Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Aditya Prakash, Josiah Hester, Jindong Wang, and Srijan Kumar. UniSD: Towards a unified self-distillation framework for large language models.arXiv preprint arXiv:2605.06597,

    arXiv

  7. [11]

    Respecting self-uncertainty in on-policy self-distillation for efficient LLM reasoning.arXiv preprint arXiv:2605.13255,

    Junlong Ke, Zichen Wen, Weijia Li, Conghui He, and Linfeng Zhang. Respecting self-uncertainty in on-policy self-distillation for efficient LLM reasoning.arXiv preprint arXiv:2605.13255,

    arXiv

  8. [12]

    press/v267/ko25a.html

    URL https://proceedings.mlr. press/v267/ko25a.html. 13 Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models.arXiv preprint arXiv:2506.06395,

    arXiv

Show all 38 references
  1. [13]

    Self-evolving visual questioner.arXiv preprint arXiv:2606.13929,

    Yijun Liang, Hengguang Zhou, Ming Li, Lichen Li, Cho-Jui Hsieh, and Tianyi Zhou. Self-evolving visual questioner.arXiv preprint arXiv:2606.13929,

  2. [14]

    Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning.arXiv preprint arXiv:2506.24119,

    Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning.arXiv preprint arXiv:2506.24119,

  3. [15]

    HERO: Hindsight- enhanced reflection from environment observations for agentic self-distillation.arXiv preprint arXiv:2606.11559, 2026a

    Haoran Liu, Yuwei Zhang, Xiyao Li, Bohan Lyu, and Jingbo Shang. HERO: Hindsight- enhanced reflection from environment observations for agentic self-distillation.arXiv preprint arXiv:2606.11559, 2026a. Xiaogeng Liu, Xinyan Wang, Yingzi Ma, Yechao Zhang, and Chaowei Xiao. When a...

    arXiv

  4. [16]

    URL https://thinkingmachines.ai/ blog/on-policy-distillation

    doi: 10.64434/tml.20251026. URL https://thinkingmachines.ai/ blog/on-policy-distillation. MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhan...

    doi

  5. [17]

    Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia

    URL https://arxiv.org/abs/2506.13585. Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,

    arXiv

  6. [18]

    Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    Mihir Prabhudesai et al. Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    arXiv

  7. [19]

    Self-consistency preference optimization.arXiv preprint arXiv:2411.04109,

    Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference optimization.arXiv preprint arXiv:2411.04109,

    arXiv

  8. [20]

    CRISP: Compressed reasoning via iterative self-policy distillation.arXiv preprint arXiv:2603.05433, 2026a

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. CRISP: Compressed reasoning via iterative self-policy distillation.arXiv preprint arXiv:2603.05433, 2026a. 14 Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. On-policy sel...

    arXiv

  9. [21]

    Deepseek- math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, et al. Deepseek- math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  10. [22]

    Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026a

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026a. Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL’s razor: Why online reinforcement learning forgets less. InThe F ourteenth...

    arXiv

  11. [23]

    Rui Wang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Wenhao Yu, and Kam-Fai Wong

    URLhttps://arxiv.org/abs/2508.06471. Rui Wang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Wenhao Yu, and Kam-Fai Wong. Demystifying on-policy distillation: Roles, pathologies, and regulations.arXiv preprint arXiv:2607.13399,

    arXiv

  12. [24]

    URL https://arxiv.org/abs/ 2607.13399

    doi: 10.48550/arXiv.2607.13399. URL https://arxiv.org/abs/ 2607.13399. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Confe...

    doi arXiv

  13. [25]

    ISBN 979-8-89176-288-6

    Association for Computational Linguistics. ISBN 979-8-89176-288-6. doi: 10.18653/v1/2025.acl-industry.24. URLhttps://aclanthology.org/2025.acl-industry.24/. 15 Guande Wu, Huan Song, Yawei Wang, Qiaojing Yan, Yijun Tian, Lin Lee Cheong, and Panpan Xu. SDRT: Enhance vision-langu...

    2025 arXiv

  14. [26]

    Lightning OPD: Efficient post-training for large reasoning models with offline on-policy distillation.arXiv preprint arXiv:2604.13010,

    Yecheng Wu, Song Han, and Hai Cai. Lightning OPD: Efficient post-training for large reasoning models with offline on-policy distillation.arXiv preprint arXiv:2604.13010,

    arXiv

  15. [27]

    URLhttps://arxiv.org/abs/2604.13010

    doi: 10.48550/ arXiv.2604.13010. URLhttps://arxiv.org/abs/2604.13010. Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613,

    doi arXiv

  16. [28]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    arXiv

  17. [29]

    Chenglin Yang, Lingxi Xie, Chi Su, and Alan L. Yuille. Snapshot distillation: Teacher–student optimization in one generation.arXiv preprint arXiv:1812.00123,

    arXiv

  18. [31]

    On-policy context distillation for language models.arXiv preprint arXiv:2602.12275,

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models.arXiv preprint arXiv:2602.12275,

    arXiv

  19. [32]

    net/forum?id=T2TZ0RY4Zk

    URL https://openreview. net/forum?id=T2TZ0RY4Zk. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.arXiv preprint arXiv:2401.10020, 2024a. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho,...

    arXiv

  20. [34]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning.arXiv preprint arXiv:2203.14465,

    arXiv

  21. [36]

    Absolute zero: Reinforced self-play reasoning with zero data.arXiv preprint arXiv:2505.03335, 2025a

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, et al. Absolute zero: Reinforced self-play reasoning with zero data.arXiv preprint arXiv:2505.03335, 2025a. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner...

    arXiv

  22. [37]

    Learning to reason without external rewards.arXiv preprint arXiv:2505.19590, 2025b

    Xuandong Zhao et al. Learning to reason without external rewards.arXiv preprint arXiv:2505.19590, 2025b. Yuxin Zuo et al. Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084,

    arXiv

  23. [38]

    pub.” denotes the numbers published in the official OPSD repository; “ours

    16 A Reproduction of OPSD Reproduction check.Before any unsupervised experiment we reproduced OPSD itself with the released code and hyperparameters. Across the 30 (checkpoint, benchmark) points published in the OPSD repository, our reproduction matches to within±4 points avg@...

  24. [2018]

    Self-distilled RLVR.arXiv preprint arXiv:2604.03128,

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled RLVR.arXiv preprint arXiv:2604.03128,

    arXiv

  25. [2019]

    Right question is already half the answer: Fully unsupervised llm reasoning incentivization.arXiv preprint arXiv:2504.05812,

    Qingyang Zhang, Haitao Wu, Changqing Zhang, et al. Right question is already half the answer: Fully unsupervised llm reasoning incentivization.arXiv preprint arXiv:2504.05812,

    arXiv

  26. [2021]

    R-zero: Self-evolving reasoning llm from zero data.arXiv preprint arXiv:2508.05004,

    Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, et al. R-zero: Self-evolving reasoning llm from zero data.arXiv preprint arXiv:2508.05004,

    arXiv

  27. [2022]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    arXiv

  28. [2023]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, et al. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    arXiv

  29. [2024]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    12 DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  30. [2025]

    Serl: Self-play reinforcement learning for large language models with limited data.arXiv preprint arXiv:2505.20347,

    Wenkai Fang, Shunyu Liu, Yang Zhou, et al. Serl: Self-play reinforcement learning for large language models with limited data.arXiv preprint arXiv:2505.20347,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.