Pith. sign in

REVIEW 3 major objections 4 minor 50 references

DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DASH reweights token-level distillation signals by the realized divergence history along each rollout, and the authors report it improves over their matched OPSD reruns on every benchmark at all three model scales without any additional…

desk verdict A genuine, honestly reported incremental method with a strong ablation suite; test-set checkpoint selection makes the headline gains look optimistic, but the effect survives the obvious controls and the paper deserves a serious referee. read the letter →

arxiv 2608.06243 v2 pith:AB5ZJOP4 submitted 2026-08-06 cs.AI

ZhiYan Hou , Xinyu Tang , Hongyan An , Jianjin Zhang , Weizhen Wang , Yunyun Han , Gengsheng Li , Xiangzhao Hao
show 4 more authors
Haiyun Guo Wenbin Hu Jinqiao Wang Yafeng Deng
This is my paper · ORCID
classification cs.AI
keywords on-policyself-distillationtemporalcreditassignmenttoken-levelsupervisionweightingadaptivepropagationgatessequence-relativedivergencegapsmathematicalreasoningbenchmarksreinforcementlearningwithverifiablerewardsQwen3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that on-policy self-distillation (OPSD), which trains reasoning models with dense token-level teacher–student supervision, leaves the temporal structure of each rollout unused because it averages local divergences with a uniform coefficient. The same local divergence value $d_t$ can follow very different discrepancy histories, and standard OPSD cannot tell those contexts apart. The proposed method, DASH, converts each token's divergence gap from the sequence mean into an adaptive propagation gate and aggregates backward, so that supervision weights depend on how divergences evolved before each token. The authors report that on three mathematical reasoning benchmarks and three Qwen3 model scales, DASH improves over their matched OPSD reruns on every setting, raising four-seed macro-averages by 1.4 to 3.2 points with no additional teacher or student forward pass. If the result holds, dense distillation can be made markedly more sample-efficient by reweighting signals that are already being computed.

What carries the argument

The carrying mechanism is the adaptive propagation gate $\lambda_t$ together with backward multi-step aggregation. At each position, the local forward-KL divergence $d_t = D_{\mathrm{KL}}(\pi_t^T \| \pi_t^S)$ between the privileged teacher and the student is capped per vocabulary item at $\tau = 0.05$ to form the local signal $r_t$; DASH centers this signal by the sequence mean $\bar{r}$ and converts the gap into a detached gate $\lambda_t = \operatorname{sg}[\sigma(-\kappa(r_t - \bar{r}))]$ with $\kappa = 5$. The backward recursion $A_T = r_T$, $A_t = r_t + \lambda_t A_{t+1}$ makes later signals decay multiplicatively through the intervening gates, so each position's effective coefficient follows $c_1 = 1$, $c_k = 1 + \lambda_{k-1} c_{k-1}$ and is a function of the realized divergence path that precedes it. Because the gates are detached, gradients flow only through the local signals, and the added work is a scalar backward scan under 1% of step time: the method reweights the direct distillation gradient rather than introducing a new learning signal.

What would settle it

Rerun the matched OPSD-versus-DASH comparison on Qwen3-1.7B selecting checkpoints on a validation set disjoint from AIME 2024, AIME 2025, and HMMT February 2025, or at a fixed step budget, and compare four-seed macro-averages; the central claim fails if DASH's advantage over the matched OPSD rerun falls below roughly one point or inverts. A supporting diagnostic is to compare the per-seed variance and the distribution of selected checkpoint step indices: if DASH's selected checkpoints are systematically later or more variable across seeds than OPSD's, the best-within-200-step rule is the likely source of the reported gain.

Watch

Extended reading notes

Core claim

The central claim is that the uniform coefficient profile of vanilla OPSD is a genuine bottleneck and that a sequence-conditioned profile improves reasoning accuracy on every benchmark and scale tested. The paper first establishes a motivating fact about on-policy autoregressive generation: at a reasoning fork, two rollouts can share the same local divergence and then diverge sharply in future divergence and in terminal outcome, so the local scalar alone cannot determine how much supervision that token should carry. DASH's response is to clip the local forward-KL contributions into a signal $r_t$, compute the gap $g_t = r_t - \bar{r}$ to the sequence mean, and map it through a detached sigmoid gate $\lambda_t = \sigma(-\kappa g_t)$ that stays more open after below-average divergence and more closed after above-average divergence, before running the backward recursion $A_t = r_t + \lambda_t A_{t+1}$. The component ablations attribute the gain primarily to this discrepancy-conditioned allocation: fixed multi-step aggregation, the reversed gate direction, and uniform coefficient scaling each recover only part of the improvement. The authors state that DASH reuses the teacher and student distributions OPSD already computes, so the reported gains require no additional teacher or student forward pass.

Load-bearing premise

The reported uniform improvements rely on a checkpoint-selection protocol under which, for each training seed, the checkpoint that scores best on the three test benchmarks themselves is chosen; if that in-sample selection helps DASH more than OPSD, the gains may not hold under a held-out selection protocol at the same training budget.

Editorial extensions

If this is right

  • On Qwen3-1.7B, DASH raises the four-seed OPSD macro-average from 41.87 to 45.07, with gains of 1.40 points at 4B and 1.60 points at 8B, and it beats the matched OPSD rerun on every individual benchmark at every scale.
  • The gain comes with no extra forward passes: DASH reuses the teacher and student distributions OPSD already computes, adding only a scalar backward aggregation that costs under 1% of step time.
  • The improvement is not explained by fixed multi-step aggregation, reversed gate direction, or uniform coefficient scaling, so the discrepancy-conditioned allocation itself is the operative ingredient.
  • Forward KL is the effective local signal for the gates: symmetric JSD and reverse KL score 6.84 and 3.60 macro-average points lower on Qwen3-1.7B, and compressing the vocabulary to the top-100 teacher tokens plus a tail loses only 0.70 points while a top-1 approximation loses 10.24 points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gate mechanism only requires a per-token scalar trajectory, so the same sequence-relative-gap reweighting could be transplanted onto other dense signals, such as process-reward scores, per-token advantages, or entropy-based weights, without re-deriving the method.
  • Because the headline numbers use best-within-200-step reporting on the test benchmarks, the gain available under a strict held-out protocol is probably smaller than 3.20 points; quantifying that gap would separate the value of the reweighting policy from the value of favorable checkpoint selection.
  • The paper's motivating observation, identical local divergence with divergent futures, implies that the local KL is not a sufficient statistic for downstream mismatch; a gate that conditions on a short divergence history instead of just the deviation from the sequence mean is a natural testable next step.
  • The appendix's GRPO hybrid shows the aggregation idea transfers to outcome-level RL with scale-dependent results (67.30 vs. 66.40 at 8B), suggesting DASH composes with existing reward-weighting schemes rather than replacing them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DASH (Divergence-Adaptive Supervision Horizons), a modification of on-policy self-distillation (OPSD) for training reasoning language models. Standard OPSD averages token-level forward-KL distillation losses with uniform coefficients; DASH replaces these with sequence-conditioned coefficients. After clipping each local divergence, DASH computes a sigmoid gate from the gap between the local clipped signal and its sequence-level mean, then applies a backward recursion A_t = r_t + λ_t A_{t+1} to produce a weighted loss whose token coefficients depend on the realized discrepancy history. The authors provide a fixed-horizon gradient decomposition (Prop. 1) as structural motivation, clarifying that DASH does not estimate the score-function term in that decomposition. Experiments on Qwen3-1.7B/4B/8B on AIME 2024, AIME 2025, and HMMT February 2025 report that DASH improves over the authors' matched OPSD reruns on every benchmark at every scale, with ablations separating the benefit of adaptive gates from fixed multi-step aggregation, gate direction, coefficient scale, divergence choice, sensitivity κ, and vocabulary support. The paper is transparent about its experimental protocol: checkpoints are selected within the 200-step budget by maximizing the unweighted average on the three test benchmarks, which the authors call 'best-within-200-step reporting rather than held-out validation selection.'

Significance. If the reported improvements are robust, DASH is a valuable, low-cost contribution: it requires no additional teacher or student forward pass, is simple to implement on top of OPSD, and its ablations (Inverse-gap, fixed λ, scale-matched controls, normalized DASH) provide credible evidence that the gains come from the sequence-conditional coefficient allocation rather than from trivial scale effects. The formal gradient decomposition is correct and clearly scoped, and the release of code plus four-seed matched OPSD reruns is a strength. However, the central quantitative claim of uniform improvement over OPSD rests on a checkpoint-selection rule that uses the test benchmarks as the selection set, and the main hyperparameter κ is also chosen on the same test benchmarks. Because several per-benchmark gains are small (0.6–1.4 points) relative to the four-seed standard deviations (0.3–1.3 points), the in-sample selection can materially bias the comparison. The paper is transparent about this protocol, but transparency does not remove the need for a fixed-checkpoint or held-out-validation check.

major comments (3)
  1. [Section 5.1 and Appendix B.3] The reporting protocol selects, for each seed, the checkpoint in {20, 40, ..., 200} that maximizes the unweighted average on the three final benchmarks (Eqs. 38–40). Because these benchmarks are also the evaluation set, every reported score is a maximum over ten candidates. The abstract's claim that DASH 'improves over our matched vanilla OPSD reruns on every benchmark at all three scales' therefore depends on this argmax selection not systematically favoring DASH. That is not guaranteed: if DASH has higher checkpoint-to-checkpoint variance or a later performance peak than OPSD, the selection bias can differ between methods. With per-benchmark gains as small as 0.6 points (Qwen3-4B HMMT 2025, Table 1), a fixed checkpoint rule (e.g., step 200) or a held-out-validation rule could plausibly flip the direction on individual benchmarks. Please report the OPSD-vs-DASH comparison at a predetermined checkpoint and under a validation-selected checkpoint, with seeded paired differences.
  2. [Section 5.4 (Figure 4a)] The propagation sensitivity κ is selected as the value (κ=5) achieving the highest macro-average on the three test benchmarks. Because the same benchmarks are used for final reporting, this is in-sample hyperparameter tuning on the test set. The claim that every tested κ outperforms OPSD is robust to this issue, but the headline configuration's advantage over OPSD is inflated by the fact that the hyperparameter was chosen to maximize the reported target. The paper should either fix κ a priori, select it on a held-out validation split, or show the OPSD comparison for each κ value (Table 8 partially does this) and state explicitly that κ was not tuned on the test set.
  3. [Appendix C] The four-seed results are reported as per-benchmark means and standard deviations, but no paired seed-level OPSD-versus-DASH differences are given. The central 'every benchmark at every scale' claim relies on uniformity across nine benchmark-scale settings, and the standard deviations (0.30–1.30) are comparable to several per-benchmark gains (e.g., HMMT 2025 on 4B: +0.6 points; AIME 2025 on 8B: +0.6 points). Without a seed-level paired table or a paired summary (e.g., the number of seeds for which DASH beats OPSD on each benchmark, or a sign test), the reader cannot assess whether the uniform improvement is consistent or driven by a subset of seeds. Please add the per-seed paired deltas and a simple paired analysis.
minor comments (4)
  1. [Section 3 / Section 5.1] The metric Avg@12 is first used in Section 5.1 without a definition; define it in the preliminaries or before its first use in the experimental setup.
  2. [Section 4.2, Eq. (9)] The clipped signal r_t is used in Eq. (9) and described in Algorithm 1, but it would improve readability to define r_t explicitly before Eq. (9), since the main text introduces d_t in Eq. (2) and the clipped variant only appears later.
  3. [Appendix A.4] The comment that individual forward-KL summands can be negative and hence r_t is not guaranteed non-negative is important; consider highlighting this in the main text where clipping is introduced, because readers may assume r_t is a standard non-negative KL divergence.
  4. [Table 1 caption] The caption states that the top three distinct results are highlighted with 1st/2nd/3rd superscripts, but these superscripts are not visible in the text rendering of the table; please ensure they appear in the published artifact or replace them with explicit bold/color formatting.

Circularity Check

2 steps flagged · score 4.0 of 10

In-sample checkpoint and κ selection on the test benchmarks, not the DASH objective itself, is the load-bearing circular element.

  1. fitted input called prediction [Section 5.1 (Experimental Setup) and Appendix B.3 (Training Budget and Checkpoint Selection), Eqs. (37)-(40)]
    "we save checkpoints every 20 steps and, for each method, model scale, and training seed, select the checkpoint within 200 steps that maximizes the unweighted average across the three benchmarks; all three benchmark scores are taken from that single checkpoint. We refer to this as best-within-200-step reporting rather than held-out validation selection."

    The headline claim that DASH improves over matched OPSD on every benchmark is evaluated from checkpoints chosen by maximizing the unweighted average of the three final test benchmarks among ten candidates per seed. Thus the reported per-benchmark scores are the argmax of the evaluation metric over the training trajectory, not an independent fixed-budget evaluation. This in-sample selection can systematically favor whichever method has larger checkpoint variance or a later peak, so the reported advantage is partly an artifact of the selection rule rather than a fixed-training-budget comparison. The paper is transparent about the protocol, but the reported numbers are maxima on the evaluation set, not independent predictions.

  2. fitted input called prediction [Section 5.4 (Design Choices and Sensitivity), 'Propagation sensitivity κ'; main results in Table 1]
    "We evaluate κ∈{1,2,5,10,20}. As shown in Figure 4a, every tested value outperforms the standard OPSD macro-average of 41.87 ... Among the tested settings, κ=5 achieves the highest macro-average of 45.07 ... We therefore use κ=5 in the main experiments."

    The main DASH configuration used for the headline claim is chosen by maximizing the same macro-average over the three test benchmarks that is later reported as DASH's result. The 1.7B DASH numbers in Table 1 are therefore not an evaluation of a pre-specified method; they are the best point on the κ grid selected on the evaluation set. This is a mild circular fit: the claim that DASH improves over OPSD is made with a configuration whose sensitivity parameter was selected to maximize the reported metric. The paper discloses the sweep but does not provide a held-out κ selection, so part of the reported gain is a selected maximum rather than a prediction.

full rationale

The DASH method itself is not circular: L_DASH is defined as a backward aggregation of clipped local KL terms with detached sigmoid gates, and the inverse-gap control performs much worse, so the gate direction is an empirical finding rather than a definitional identity. The gradient decomposition is explicitly offered only as structural motivation: the paper states 'This fixed-horizon decomposition provides only structural motivation' and DASH does not estimate the trajectory term or introduce score-function gradients. There is no load-bearing self-citation: the OPSD baseline is attributed to Zhao et al., not to the present authors, and no uniqueness theorem is imported from the authors' own prior work. The circularity is confined to the evaluation protocol. Both the checkpoint index (Eq. 39) and the sensitivity κ are selected on the same three test benchmarks (AIME 2024, AIME 2025, HMMT February 2025) used to report the headline improvements, so the reported gains are maxima over the selection axes rather than independent fixed-budget predictions. The paper is transparent about both choices, but transparency does not remove the in-sample selection. On the other hand, the claimed advantage is not forced by construction: matched OPSD reruns, inverse-gap, fixed-coefficient, and scale-matched controls could plausibly have matched or beaten DASH, and the ablations show that several alternatives do not. Accordingly, the appropriate score is 4: partial circularity from fitted selection on the test set, while the core contribution retains independent empirical content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

DASH introduces an algorithmic component, the adaptive propagation gate, but no new physical or ontological entity. The main free parameters are the propagation sensitivity κ, selected on the test set, and the clipping threshold τ, chosen without ablation. The axioms are largely inherited from OPSD plus a design hypothesis about gate direction that is tested, not derived.

free parameters (2)
  • Propagation sensitivity κ = 5
    Selected as the best value among {1,2,5,10,20} on the test benchmarks (Figure 4a, Section 5.4). This is a post-hoc fit to the evaluation set and is used in all main experiments.
  • Clipping threshold τ = 0.05
    Applied to each vocabulary-level divergence contribution (Algorithm 1); no ablation is reported, so it is a hand-chosen constant that shapes the local signal used for both loss and gates.
assumptions (4)
  • domain assumption The privileged teacher's reference-conditioned distribution π̄θ(·|s_t, z) is a useful dense supervision target at every student-visited prefix.
    This is the core premise of OPSD (Sec. 3, Eq. 1), adopted without independent verification in this paper.
  • domain assumption The forward KL divergence over the full vocabulary is an appropriate local distillation signal.
    Section 5.4 tests alternatives JSD and reverse KL and finds them worse, but the main method fixes forward KL; the choice is empirical, not derived.
  • ad hoc to paper The sign/direction of the gap-to-gate mapping (below-average divergence opens the gate, above-average closes it) is correct.
    This is the central design hypothesis of DASH (Eq. 12, Sec. 5.3). The inverse-gap control is worse, but the mapping is not derived from principles; it is tested on the same benchmarks.
  • standard math The log-derivative identity and zero-conditional-mean of score functions hold for the student policy with detected trajectories.
    Standard results from policy gradient theory (Williams 1992; Sutton et al. 2000), invoked in the proof of Proposition 1 in Sec. 4.1 and App. A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models." pith.science (2026). https://pith.science/paper/AB5ZJOP4

@misc{pith2026260806243,
  author       = {Pith},
  title        = {Pith review of: DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AB5ZJOP4}},
  note         = {Machine review of arXiv:2608.06243}
}
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: https://github.com/DBtxy/DASH-OPSD

Figures

Figures reproduced from arXiv: 2608.06243 by the authors.

Figure 1
Figure 1. Motivation and mechanism of DASH. (a) The current local divergence Dt is weakly associated with the mean future divergence D¯>t within student-generated trajectories. The plot visualizes 2,500 of the 30k analyzed tokens. (b) Two rollouts matched by local divergence at a reasoning fork subsequently exhibit sharply different divergence profiles and terminal outcomes. (c) Vanilla OPSD aggregates the matched local diver… view at source ↗
Figure 2
Figure 2. Overview of DASH. A privileged teacher evaluates a student-generated rollout to produce local distillation signals. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Component ablations of DASH on Qwen3-1.7B. (a) Fixed values of the propagation coefficient [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Parameter ablations of DASH on Qwen3-1.7B. (a) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 12 linked inside Pith

  1. [1]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , journal =. 2024 , eprint =

    2024

  2. [2]

    2025 , doi =

    Guo, Daya and Yang, Dejian and Zhang, Haowei and others , journal =. 2025 , doi =

    2025

  3. [3]

    International Conference on Learning Representations , year =

    Let's Verify Step by Step , author =. International Conference on Learning Representations , year =

  4. [4]

    2025 , eprint =

    Parthasarathi, Prasanna and Reymond, Mathieu and Chen, Boxing and Cui, Yufei and Chandar, Sarath , journal =. 2025 , eprint =

    2025

  5. [5]

    Gu, Yuxian and Dong, Li and Wei, Furu and Huang, Minlie , booktitle =

  6. [6]

    International Conference on Learning Representations , year =

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author =. International Conference on Learning Representations , year =. 2306.13649 , archivePrefix =

    arXiv

  7. [7]

    arXiv preprint arXiv:2601.18734 , year =

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author =. arXiv preprint arXiv:2601.18734 , year =. 2601.18734 , archivePrefix =

    arXiv

  8. [8]

    arXiv preprint arXiv:2603.07079 , year =

    Entropy-Aware On-Policy Distillation of Language Models , author =. arXiv preprint arXiv:2603.07079 , year =. 2603.07079 , archivePrefix =

    arXiv

Show all 50 references
  1. [9]

    arXiv preprint arXiv:2605.21606 , year =

    When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning , author =. arXiv preprint arXiv:2605.21606 , year =. 2605.21606 , archivePrefix =

    arXiv

  2. [10]

    arXiv preprint arXiv:2606.22600 , year =

    On the Position Bias of On-Policy Distillation , author =. arXiv preprint arXiv:2606.22600 , year =. 2606.22600 , archivePrefix =

    arXiv

  3. [11]

    arXiv preprint arXiv:2605.30833 , year =

    Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation , author =. arXiv preprint arXiv:2605.30833 , year =. 2605.30833 , archivePrefix =

    arXiv

  4. [12]

    2025 , eprint =

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and others , journal =. 2025 , eprint =

    2025

  5. [13]

    2025 , eprint =

    Guha, Etash and Marten, Ryan and Keh, Sedrick and Raoof, Negin and Smyrnis, Georgios and Bansal, Hritik and Nezhurina, Marianna and Mercat, Jean and Vu, Trung and Sprague, Zayne and others , journal =. 2025 , eprint =

    2025

  6. [14]

    2026 , eprint =

    Lin, Chen and Chen, Kedi and Zhang, Wei , journal =. 2026 , eprint =

    2026

  7. [15]

    Purified

    Shen, Zhanming and Tong, Jintao and Yan, Shaotian and Shen, Chen and Chen, Hao and Ye, Wentao and Hu, Xiaomeng and Miao, Rui and Wang, Haobo and Zhao, Junbo and Chen, Gang and Ye, Jieping , journal =. Purified. 2026 , eprint =

    2026

  8. [16]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =. 2022 , eprint =

    2022

  9. [17]

    2026 , eprint =

    Li, Yuhan and Zhang, Mingxu and Shen, Dazhong and Sun, Ying , journal =. 2026 , eprint =

    2026

  10. [18]

    2024 , howpublished =

    2024

  11. [19]

    2025 , howpublished =

    2025

  12. [20]

    and Liu, Alisa and Dziri, Nouha and Lyu, Shane and Gu, Yuling and Malik, Saumya and Graf, Victoria and Hwang, Jena D

    Lambert, Nathan and Morrison, Jacob and Pyatkin, Valentina and Huang, Shengyi and Ivison, Hamish and Brahman, Faeze and Miranda, Lester James V. and Liu, Alisa and Dziri, Nouha and Lyu, Shane and Gu, Yuling and Malik, Saumya and Graf, Victoria and Hwang, Jena D. and Yang, Jian...

    2024

  13. [21]

    2025 , eprint =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Fan, Tiantian and Liu, Gaohong and Liu, Lingjun and Liu, Xin and Lin, Haibin and Lin, Zhiqi and Ma, Bole and Sheng, Guangming and Tong, Yuxuan and Zhang, Chi and Zhang, Mofan and Zha...

    2025

  14. [22]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in

    Yue, Yang and Chen, Zhiqi and Lu, Rui and Zhao, Andrew and Wang, Zhaokai and Yue, Yang and Song, Shiji and Huang, Gao , journal =. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2025 , eprint =

    2025

  15. [23]

    arXiv preprint arXiv:1707.06347 , year =

    Proximal Policy Optimization Algorithms , author =. arXiv preprint arXiv:1707.06347 , year =. 1707.06347 , archivePrefix =

    arXiv

  16. [24]

    Advances in Neural Information Processing Systems , year =

    Training Language Models to Follow Instructions with Human Feedback , author =. Advances in Neural Information Processing Systems , year =

  17. [25]

    Advances in Neural Information Processing Systems , year =

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author =. Advances in Neural Information Processing Systems , year =

  18. [26]

    arXiv preprint arXiv:2110.14168 , year =

    Training Verifiers to Solve Math Word Problems , author =. arXiv preprint arXiv:2110.14168 , year =. 2110.14168 , archivePrefix =

    arXiv

  19. [27]

    Measuring Mathematical Problem Solving with the

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , booktitle =. Measuring Mathematical Problem Solving with the

  20. [28]

    arXiv preprint arXiv:2107.03374 , year =

    Evaluating Large Language Models Trained on Code , author =. arXiv preprint arXiv:2107.03374 , year =. 2107.03374 , archivePrefix =

    arXiv

  21. [29]

    Advances in Neural Information Processing Systems , year =

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , year =

  22. [30]

    , booktitle =

    Zelikman, Eric and Wu, Yuhuai and Mu, Jesse and Goodman, Noah D. , booktitle =

  23. [31]

    arXiv preprint arXiv:2211.14275 , year =

    Solving Math Word Problems with Process- and Outcome-Based Feedback , author =. arXiv preprint arXiv:2211.14275 , year =. 2211.14275 , archivePrefix =

    arXiv

  24. [32]

    Wang, Peiyi and Li, Lei and Shao, Zhihong and Xu, Runxin and Dai, Damai and Li, Yifei and Chen, Deli and Wu, Yu and Sui, Zhifang , booktitle =

  25. [33]

    Rewarding Progress: Scaling Automated Process Verifiers for

    Setlur, Amrith and Nagpal, Chirag and Fisch, Adam and Geng, Xinyang and Eisenstein, Jacob and Agarwal, Rishabh and Agarwal, Alekh and Berant, Jonathan and Kumar, Aviral , booktitle =. Rewarding Progress: Scaling Automated Process Verifiers for. 2025 , eprint =

    2025

  26. [34]

    2024 , eprint =

    Kazemnejad, Amirhossein and Aghajohari, Milad and Portelance, Eva and Sordoni, Alessandro and Reddy, Siva and Courville, Aaron and Le Roux, Nicolas , journal =. 2024 , eprint =

    2024

  27. [35]

    Machine Learning , volume =

    Learning to Predict by the Methods of Temporal Differences , author =. Machine Learning , volume =

  28. [36]

    International Conference on Learning Representations , year =

    High-Dimensional Continuous Control Using Generalized Advantage Estimation , author =. International Conference on Learning Representations , year =

  29. [37]

    Machine Learning , volume =

    Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , author =. Machine Learning , volume =

  30. [38]

    Advances in Neural Information Processing Systems , year =

    Policy Gradient Methods for Reinforcement Learning with Function Approximation , author =. Advances in Neural Information Processing Systems , year =

  31. [39]

    arXiv preprint arXiv:1503.02531 , year =

    Distilling the Knowledge in a Neural Network , author =. arXiv preprint arXiv:1503.02531 , year =. 1503.02531 , archivePrefix =

    arXiv

  32. [40]

    Proceedings of the Conference on Empirical Methods in Natural Language Processing , year =

    Sequence-Level Knowledge Distillation , author =. Proceedings of the Conference on Empirical Methods in Natural Language Processing , year =

  33. [41]

    Ko, Jongwoo and Kim, Sungnyun and Chen, Tianyi and Yun, Se-Young , booktitle =

  34. [42]

    Proceedings of the International Conference on Artificial Intelligence and Statistics , year =

    A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , author =. Proceedings of the International Conference on Artificial Intelligence and Statistics , year =

  35. [43]

    Advances in Neural Information Processing Systems , year =

    Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks , author =. Advances in Neural Information Processing Systems , year =

  36. [44]

    Neural Networks , volume =

    A New Learning Paradigm: Learning Using Privileged Information , author =. Neural Networks , volume =

  37. [45]

    International Conference on Learning Representations , year =

    Unifying Distillation and Privileged Information , author =. International Conference on Learning Representations , year =

  38. [46]

    Robotics: Science and Systems , year =

    Asymmetric Actor Critic for Image-Based Robot Learning , author =. Robotics: Science and Systems , year =

  39. [47]

    Advances in Neural Information Processing Systems , year =

    Bridging the Imitation Gap by Adaptive Insubordination , author =. Advances in Neural Information Processing Systems , year =

  40. [48]

    Neural Computation , volume =

    Long Short-Term Memory , author =. Neural Computation , volume =

  41. [49]

    Learning Phrase Representations Using

    Cho, Kyunghyun and van Merri. Learning Phrase Representations Using. Proceedings of the Conference on Empirical Methods in Natural Language Processing , year =

  42. [50]

    arXiv preprint arXiv:2605.20643 , year=

    AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals , author=. arXiv preprint arXiv:2605.20643 , year=

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.