Pith. sign in

REVIEW 2 major objections 5 minor 23 references

REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single shared adaptive controller for on-policy distillation matches or beats per-domain tuned extrapolation coefficients across math, code, and multi-teacher settings.

desk verdict Clean, well-specified method proposal with a strong ablation, but the headline empirical claims are not yet supported by single-run, no-error-bar numbers. read the letter →

arxiv 2608.11698 v2 pith:FE77OSEY submitted 2026-08-12 cs.LG cs.AI

Yang Sun , Lichao Ma , Houyuan Qin , Yuxin Liu , Hanyang Lu , Yao Zhu , Pinlong Cai , Guohang Yan This is my paper · ORCID
classification cs.LGcs.AI
keywords on-policydistillationrewardextrapolationtoken-levelcompatibilityadaptivebudgetlargelanguagemodelsteacheralignmentmulti-teacherreliability-adaptivecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles a specific instability in on-policy distillation: when a student is trained to push beyond its teacher by scaling the teacher–reference log-ratio with a single global constant $\lambda$, a few extreme tokens can dominate the update, and the preferred $\lambda$ changes across domains. REOPD separates teacher alignment from extrapolation, using a token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ that keeps the standard alignment term intact and gates only the beyond-teacher residual through a compatibility weight $q_t$ and a shared micro-batch budget $\gamma_b$. The paper claims this makes per-domain coefficient sweeps unnecessary: a single adaptive controller outperforms G-OPD on single-teacher mathematics and on both multi-teacher domains, matches it on single-teacher code, and requires no verifier, reward model, value model, or extra rollout. A reader should care because the method, if right, replaces a brittle global hyperparameter with an online control signal built entirely from log-probabilities that OPD already computes.

What carries the argument

The load-bearing object is the token-wise effective coefficient $\lambda_{b,t}=1+\gamma_b q_t$, inserted into the distillation cost as $C_{b,t}=a_{b,t}-\gamma_b q_t r_{b,t}$. Here $a_t$ is the student–teacher alignment cost, $r_t$ is the teacher–reference log-ratio that defines the implicit reward, $q_t$ is a compatibility weight built from a stop-gradient student–teacher discrepancy proxy $\hat{\delta}_t$ (with $q_t=\exp(-\hat{\delta}_t/\tau)$, so agreement keeps $q_t$ near $1$ and disagreement attenuates the residual), and $\gamma_b$ is a shared micro-batch budget computed from the compatibility-weighted residual proportion $\rho_b$ and the reliable residual RMS $s_b$ after EMA smoothing, clipped to $[0,\gamma_{\max}]$. This object does the work of separating teacher alignment from extrapolation: it reduces to standard OPD at $\gamma=0$, to fixed-$\lambda$ ExOPD when $q=1$ and $\gamma=\lambda-1$, and it guarantees $1\le \lambda_{b,t}\le 1+\gamma_{\max}$ so no single token can receive unbounded extrapolation.

What would settle it

Take a training set where teacher outputs are systematically corrupted at the token level (for example, math solutions with deliberately inserted arithmetic slips), keep the student initialization and prompts identical, and compare REOPD against OPD and a verifier-gated baseline. If the tokens receiving the largest amplified residual are the corrupted tokens and accuracy drops relative to OPD, then the compatibility gate is not a sufficient safety signal.

Watch

Extended reading notes

Core claim

The central claim is that reward extrapolation in on-policy distillation fails not because extrapolation is dangerous, but because a single global coefficient applies the same gain to every token. REOPD establishes that the token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ resolves this by decomposing ExOPD's uniform residual multiplier into two adaptive parts: a token gate $q_t$ that preserves the residual when the student and teacher agree locally and suppresses it when they disagree, and a shared, bounded budget $\gamma_b$ that reflects how much compatible residual the current batch can support. The paper states that this construction preserves teacher alignment at all times, bounds the effective coefficient to $[1, 1+\gamma_{\max}]$, and reuses only the student, teacher, and reference log-probabilities already needed by G-OPD. Empirically, REOPD reports better or matching accuracy relative to G-OPD across single-teacher math, single-teacher code, and shared multi-teacher distillation, and its ablation shows that removing the token gate costs 4.27 points while removing the budget bound costs 0.50 points, identifying token-level filtering as the principal component.

Load-bearing premise

The load-bearing premise is that local student–teacher agreement marks the teacher's residual signal as safe to amplify, so if the student and teacher both agree on tokens that are nevertheless wrong, REOPD extrapolates confidently in wrong directions.

Editorial extensions

If this is right

  • Fixed-coefficient ExOPD's per-domain $\lambda$ sweep can be replaced by online adaptation using only quantities already computed in OPD, eliminating a costly full-training tuning loop.
  • Token-level compatibility, not the batch budget, is the main source of improvement, so residual filtering per token is the intervention that stabilizes reward extrapolation.
  • A single shared adaptive controller works across routed domain teachers, so multi-teacher distillation does not need separate per-teacher coefficients.
  • The added safety comes at no extra inference cost: no verifier, reward model, value model, or rollout beyond standard OPD is required.
  • Because the effective coefficient is bounded by $1+\gamma_{\max}$, extreme teacher–reference log-ratio peaks cannot dominate the policy update beyond the explicit cap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: since $q_t$ measures compatibility rather than correctness, the method's safety depends on the teacher being mostly right; injecting a sparse, cheap correctness signal, such as verifier labels only where $q_t$ is high, could suppress confident joint errors without giving up the adaptive budget.
  • Editorial inference: the logged dynamics show $\gamma_b$ saturating at its upper bound late in training, suggesting the controller's main work happens early; a simpler schedule that starts small and anneals to $\gamma_{\max}$ may reproduce much of REOPD's benefit in single-teacher settings.
  • Editorial inference: the discrepancy proxy is computed per token on the sampled response, so a token with low $q_t$ because of an earlier student mistake is merely down-weighted, not corrected; combining REOPD with rollout truncation or rollback could produce alignment gains the current method leaves on the table.
  • Editorial inference: the controller is tested only at one model scale and teacher family, so the strongest testable extension is running REOPD on larger student models and heterogeneous teacher families to see whether the shared budget remains stable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes REOPD, a reliability-adaptive extension of ExOPD for on-policy distillation. REOPD replaces the global extrapolation coefficient with a token-level compatibility weight q_t and a micro-batch-level adaptive budget γ_b, yielding an effective token-wise coefficient λ_{b,t}=1+γ_b q_t that preserves the standard teacher-alignment term while gating only the beyond-teacher residual. The method requires no verifier, reward model, value model, or extra rollout beyond standard OPD. The paper evaluates REOPD on single-teacher mathematics, single-teacher code, and mixed-domain multi-teacher distillation, reporting that REOPD outperforms OPD and the fixed λ=1.25 ExOPD baseline on mathematics and on both multi-teacher domains, while performing comparably on single-teacher code. A component ablation in the mathematics setting shows that removing the token-level gate lowers accuracy by 4.27 points, identifying token compatibility as the main active component.

Significance. If the empirical claims hold, the contribution is useful: the decomposition of teacher alignment from extrapolation control is conceptually clean, the special cases (γ_b=0 recovers OPD, q_t=1 and fixed γ_b recovers ExOPD) are well stated, and the ablation cleanly attributes the main effect to the token gate rather than to the batch controller. The method is not circular: q_t and γ_b are stop-gradient functions of already-computed log-probabilities, and B0 is calibrated on alignment RMS, not on evaluation accuracy. The main weakness is that the headline comparisons rest on small differences (0.19 to 1.38 percentage points) reported without repeated seeds, confidence intervals, or cluster-aware uncertainty. Because the evaluation pools many completions per problem, the effective sample size is much smaller than the reported number of completions, so the central 'outperforms/matching' claims are currently underdetermined. The paper acknowledges that q_t is a compatibility measure rather than a correctness estimator; that limitation is real but secondary relative to the missing uncertainty quantification.

major comments (2)
  1. [Secs. 4.1, 4.2, Table 1] The central empirical claim is not yet supported by the reported statistics. All main results come from a single step-50 checkpoint, no repeated seeds are reported for the main table, and the only seed mentioned (42) is used for ablations. The text says 'across repeated experiments' (Secs. 4.1 and 5), but no repeated experiments are shown. Pooling 3,840 mathematics completions from 120 problems, or 2,868 code completions from 717 tasks, treats responses to the same problem as independent, while the effective sampling unit is the problem. The reported differences of 0.19 to 1.38 percentage points can therefore easily be within cluster noise. Please provide variance-aware evidence: multiple seeds with means and standard deviations, or cluster bootstrap confidence intervals at the problem level, and state explicitly whether the REOPD advantages over OPD and ExOPD are statistically significant. Without this, the abstract's 'outperforms' and 'matching' statements are not supported.
  2. [Sec. 4.3] The comparison against the best fixed coefficient chosen from the same evaluation sweep is an oracle baseline, and the paper is transparent that this is not a post-hoc favorable baseline for REOPD. However, the four margins reported here are +0.19, −0.15, +0.03, and +0.42 percentage points, which fall within the uncertainty range described in the previous comment. The claim that REOPD 'reaches comparable or better performance than the task-specific best fixed coefficient' needs error bars or confidence intervals for both the REOPD runs and the swept ExOPD curves. Without such intervals, the reader cannot distinguish a genuine adaptation advantage from sampling noise.
minor comments (5)
  1. [Sec. 3.3, Eq. (6)] The phrase 'low-variance k 3 discrepancy proxy' is unclear: the symbol 'k' or 'k3' is never defined, and the equation as written only introduces x and δ. Please either define the term or remove it.
  2. [Sec. 4.1] The auto-calibration of B0 is described only verbally as a 'scaled moving average of the alignment RMS, κ RMS_b(a)' during the first K0 controller calls. Please provide the exact formula and initialization details, since B0 directly affects the magnitude of the extrapolation budget.
  3. [Table 1 and Sec. 4.2] The terminology is inconsistent: the abstract and Sec. 4.2 refer to 'G-OPD' as the comparison method, while Table 1 labels the corresponding rows 'OPD'. Please clarify whether G-OPD denotes the general framework and 'OPD' denotes the λ=1 instantiation, and use the terminology consistently throughout.
  4. [Figure 3] Panel (b) reports 'mean compatibility q_t and effective coefficient λ_{b,i,t}'; because the effective coefficient is token-wise and varies within a batch, please clarify over which tokens the mean is taken and whether the plotted quantity is the micro-batch average of λ or the average of γ_b times the average q_t.
  5. [Sec. 3.3 and Conclusion] The abstract and title use 'reliability' to describe the gating mechanism, but Sec. 3.3 correctly states that q_t measures local compatibility, not correctness. Since the paper explicitly acknowledges that the gate cannot detect cases where both student and teacher are wrong, please qualify the abstract's phrase 'reliable teacher-reference directions' to avoid overstating the semantic content of the gate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: REOPD's core coefficient is constructed from already-available log-probabilities, and the reported benchmark comparisons are measured against external evaluation sets rather than fitted to the claims.

full rationale

The derivation chain is self-contained. The token-level compatibility weight q_{b,i,t} is defined directly from the student, teacher, and reference log-probabilities via Eqs. (1)-(7), all of which are quantities already computed in the standard G-OPD pipeline, and both q and the micro-batch budget gamma_b are explicitly detached from the computation graph as stop-gradient control signals. The budget scale B0 is calibrated in automatic mode from a scaled moving average of alignment RMS, not from evaluation accuracy, so no benchmark target is fitted or renamed as a prediction. The final objective in Eq. (5) is a construction, not a derived empirical claim, and the paper's stated contributions are empirical comparisons on AIME, HMMT, HumanEval+, MBPP+, and LiveCodeBench against external baselines OPD and fixed-coefficient ExOPD. The acknowledged limitation that q measures local compatibility rather than correctness is a scope caveat, not a circular step, because the method never claims q is an independent correctness oracle. The skeptical concern that single-run, pooled-completion statistics may not support the reported margins is a statistical-evidence issue, not a reduction of the result to its inputs, and therefore does not raise the circularity score.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is carried by two adapted quantities: q_t, a stop-gradient function of the student-teacher log-ratio, and gamma_b, a smoothed function of batch residual statistics. These are not fit to evaluation accuracy, but they are heuristic controller components with several hand-set hyperparameters. No new physical or architectural entities are introduced.

free parameters (6)
  • tau = 0.007
    Temperature in q_t = exp(-delta_t / tau); controls how sharply large student-teacher discrepancies suppress extrapolation. Hand-chosen, shared across domains, no sensitivity study reported.
  • gamma_max = 1
    Upper bound on the micro-batch budget gamma_b, so the effective coefficient is at most 2. Hand-chosen; the no_bound ablation shows removing it costs 0.50 points.
  • B0 (budget scale) = auto-calibrated with kappa=0.5 over the first 10 controller calls
    Initial scale in e_gamma = clip(B0 * rho_bar / s_bar, 0, gamma_max); calibrated from a scaled moving average of alignment RMS during early training, then fixed. This is a fitted controller constant, though not fit to evaluation accuracy.
  • beta (EMA smoothing) = 0.95
    EMA coefficient for rho and s sufficient statistics; chosen hyperparameter.
  • beta_gamma (budget smoothing) = 0.9
    Smoothing coefficient for the gamma target; chosen hyperparameter.
  • warm-up schedule = math: five controller calls at gamma=0.25; code and multi-teacher: none
    Ad hoc per-domain warm-up; means the controller is not fully hyperparameter-free across settings despite avoiding a lambda sweep.
assumptions (5)
  • domain assumption The teacher-reference log-ratio r_t defines a valid dense implicit reward for on-policy distillation, as in G-OPD.
    REOPD inherits the G-OPD reward framing from [4]; if r_t is not a good training signal, finely gating it does not repair the reward.
  • ad hoc to paper The student-teacher discrepancy proxy delta_t is a valid gate for extrapolation reliability: large disagreement should suppress the extra residual.
    Eqs. (6)-(7) define q_t from delta_t; no theory or external correctness signal links local log-ratio agreement to safe extrapolation. Section 5 admits the proxy cannot detect cases where both policies are wrong.
  • ad hoc to paper Batch statistics rho_b and s_b, after EMA smoothing, correctly indicate how much extrapolation a micro-batch can support.
    Eqs. (8)-(12) set the budget gamma_b; the ratio B0*rho/s and clipping are heuristics with no optimality argument.
  • standard math exp(x) >= x + 1 for real x, which ensures delta_t >= 0 and q_t in (0,1].
    Used in Sec. 3.3 to guarantee the compatibility weight range in exact arithmetic.
  • domain assumption The PPO-style surrogate with negative token cost as advantage is a valid policy update for this distillation objective.
    Standard PPO machinery from [15] is reused; REOPD inherits its approximation and clipping behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation." pith.science (2026). https://pith.science/paper/FE77OSEY

@misc{pith2026260811698,
  author       = {Pith},
  title        = {Pith review of: REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FE77OSEY}},
  note         = {Machine review of arXiv:2608.11698}
}
abstract

On-policy distillation (OPD) trains a student on its own trajectories under dense token-level supervision from a teacher. Reward-extrapolation methods such as ExOPD amplify the teacher-reference log-likelihood ratio to move beyond direct imitation, but apply a single global coefficient $\lambda$ to every token. This can drive the student to fit extreme peaks in the implicit reward, causing reward hacking and unstable training, and the optimal $\lambda$ varies across domains, requiring costly sweeps. We propose REOPD, a reliability-adaptive reward extrapolation framework for OPD. REOPD combines a token-level compatibility weight with a batch-level adaptive budget, yielding a token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions. It requires no verifier, reward model, value model, or extra rollout beyond standard OPD. REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code, demonstrating effective fine-grained reliability adaptation across domains and teacher configurations.

Figures

Figures reproduced from arXiv: 2608.11698 by the authors.

Figure 1
Figure 1. Fixed-coefficient ExOPD applies the same residual gain λ − 1 to every token and may amplify teacher–reference log-ratio peaks. REOPD preserves teacher alignment, gates the beyond-teacher residual with compatibility qb,i,t , and derives a bounded micro-batch budget γb from synchronized statistics. Our key observation is that teacher alignment and extrap￾olation beyond the teacher should be controlled separately. A to… view at source ↗
Figure 2
Figure 2. Aggregate performance across fixed ExOPD coe [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Raw controller trajectories. Panel (a) reports the micro-batch budget [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 4 canonical work pages

  1. [1]

    On-policy distillation of language mod- els: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Pi- otr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language mod- els: Learning from self-generated mistakes. InThe Twelfth International Conference on Learning Representa- tions, 2024. URL https://openreview.net/forum? id=3zKtaqxLhW

    work page 2024

  2. [2]

    Rethink- ing on-policy distillation of large language models: Phe- nomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethink- ing on-policy distillation of large language models: Phe- nomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026. URL https://arxiv.org/ abs/2604.13016

    arXiv 2026

  3. [4]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapo- lation.arXiv preprint arXiv:2602.12125, 2026

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiy- ong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapo- lation.arXiv preprint arXiv:2602.12125, 2026. URL https://arxiv.org/abs/2602.12125

    arXiv 2026

  4. [5]

    Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and JeffDean. Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/ abs/1503.02531

    arXiv 2015

  5. [6]

    Yoon Kim and Alexander M. Rush. Sequence-level knowl- edge distillation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URLhttps://aclanthology.org/D16-1139/

    doi:10.18653/v1/d16-1139 2016

  6. [7]

    MiniLLM: Knowledge distillation of large language mod- els

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language mod- els. InThe Twelfth International Conference on Learn- ing Representations, 2024. URL https://openreview. net/forum?id=5h0qf7IBZZ

    work page 2024

  7. [8]

    Model extrapolation expedites alignment

    Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Model extrapolation expedites alignment. InProceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 1025–1041, Vienna, Austria, 2025. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/ 2025.acl-long.51. URL https://a...

    doi:10.18653/v1/ 2025

  8. [9]

    LLM-oriented token-adaptive knowledge distillation.Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 40(40):34070–34078, 2026

    Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. LLM-oriented token-adaptive knowledge distillation.Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 40(40):34070–34078, 2026. doi: 10.1609/aaai. v40i40.40701. URL https://ojs.aaai.org/index. php/AAAI/article/view/40701

    doi:10.1609/aaai 2026

Show all 23 references
  1. [10]

    ASKD: Reinforcement learning-style knowledge distillation with quality-adaptive skewness.Proceedings of the AAAI Conference on Ar- tificial Intelligence, 40(41):34781–34789, 2026

    Mingjie Zhang, Xiaoling Zhou, Yuxiao Luo, Yiyu Liu, Shikun Zhang, and Wei Ye. ASKD: Reinforcement learning-style knowledge distillation with quality-adaptive skewness.Proceedings of the AAAI Conference on Ar- tificial Intelligence, 40(41):34781–34789, 2026. doi: 10.1609/aaai.v...

    2026 doi

  2. [11]

    Prune-OPD: Efficient and reliable on-policy dis- tillation for long-horizon reasoning.arXiv preprint arXiv:2605.07804, 2026

    Zhicheng Yang, Zhijiang Guo, Yifan Song, Minrui Xu, Yongxin Wang, Yiwei Wang, Xiaodan Liang, and Jing Tang. Prune-OPD: Efficient and reliable on-policy dis- tillation for long-horizon reasoning.arXiv preprint arXiv:2605.07804, 2026. URL https://arxiv.org/ abs/2605.07804

    2026 arXiv

  3. [12]

    SCOPE: Signal-calibrated on-policy dis- tillation enhancement with dual-path adaptive weighting

    Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, and Xunliang Cai. SCOPE: Signal-calibrated on-policy dis- tillation enhancement with dual-path adaptive weighting. arXiv preprint arXiv:2604.10688, 2026. URL https: //arxiv.org...

    2026 arXiv

  4. [13]

    SG-OPD: Sign-gated on- policy distillation via sign-consistency gating and phased teacher sampling.arXiv preprint arXiv:2606.09304, 2026

    Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, and Xiaosong Yuan. SG-OPD: Sign-gated on- policy distillation via sign-consistency gating and phased teacher sampling.arXiv preprint arXiv:2606.09304, 2026. URLhttps://arxiv.org/abs/2606.09304

    2026 arXiv

  5. [14]

    Reward-gated on-policy distillation

    Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, and Alek- sandar Bojchevski. Reward-gated on-policy distillation. arXiv preprint arXiv:2607.04037, 2026. URL https: //arxiv.org/abs/2607.04037

    2026 arXiv

  6. [15]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  7. [16]

    Qwen3 Technical Re- port.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen3 Technical Re- port.arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388

    2025 arXiv

  8. [17]

    DeepMath-103K: A large- scale, challenging, decontaminated, and verifiable math- ematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. DeepMath-103K: A large- scale, challenging, decontaminated, and verifiable math- ematical da...

    2025 arXiv

  9. [18]

    Advancing LLM reasoning generalists with preference trees.arXiv preprint Preprint– REOPD: Reliability-AdaptiveRew ardExtrapolation forOn-PolicyDistillation 9 arXiv:2404.02078, 2024

    Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruob- ing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees.arXiv preprint Preprint– REOPD:...

    2024 arXiv

  10. [19]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. InInternational Conference on Learn- ing Representations, 2019. URL https://openreview. net/forum?id=Bkg6RiCqY7

    2019

  11. [20]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv. org/abs/2107.03374

    2021 arXiv

  12. [21]

    Le, and Charles Sutton

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021. URL https://arxiv. org/abs/2108.07732

    2021 arXiv

  13. [22]

    Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation.arXiv preprint arXiv:2305.01210, 2023

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Ling- ming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation.arXiv preprint arXiv:2305.01210, 2023. URLhttps://arxiv.org/abs/2305.01210

    2023 arXiv

  14. [23]

    LiveCodeBench: Holistic and contamination free evaluation of large language mod- els for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language mod- els for code.arXiv preprint arXiv:2403.07974, 2024. URL https:/...

    2024 arXiv

  15. [2026]

    URLhttps://arxiv.org/abs/2604.14084

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.