Pith. sign in

REVIEW 3 major objections 5 minor 51 references

The paper proposes a progress-aware rollout-horizon scheduler that claims higher average math benchmark scores with 49–71% less training time than fixed schedules in on-policy distillation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:50 UTC pith:YCCGYIHY

load-bearing objection Adaptive FastOPD is a solid, well-described efficiency method for OPD that deserves a serious referee; the 'highest average' claim needs error bars, and the boundary-region proxy is a real but not fatal limitation. the 3 major comments →

arxiv 2607.29494 v1 pith:YCCGYIHY submitted 2026-07-31 cs.LG

Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

Qian Tan , Huaifei Liang , Xuanyu Zhu , Lei Jiang , Yuqiang Li This is my paper
classification cs.LG
keywords on-policy distillationrollout horizonprogress-aware schedulingplateau detectionteacher-student signalstraining efficiencymathematical reasoninglarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tackles the cost of on-policy distillation: student-generated rollouts need teacher supervision, and long responses in a batch delay training. Instead of expanding the maximum rollout length on a fixed schedule or against an absolute agreement threshold, Adaptive FastOPD expands only when learning in the current boundary region has plateaued and enough rollouts actually reach that region. This keeps the rollout budget aligned with the student's stage-specific progress. Across two teacher–student pairs and one math dataset, the paper reports the best average score among compared methods while reducing training time by 49.1–71.2% relative to the longest fixed-horizon baseline, and shows smaller performance swings than fixed schedules across tested hyperparameter settings.

Core claim

On its own terms, the central discovery is that the right moment to lengthen student rollouts can be read from four normalized teacher–student signals measured relative to their values when the current horizon was entered. When these signals stop improving in the final chunk of the current horizon, and a sufficient fraction of rollouts reaches that chunk, the horizon is expanded by one fixed increment. This stage-relative design removes the need to pick a global expansion interval or an absolute agreement threshold, which the paper shows to be brittle: a fixed schedule with the wrong interval can collapse average scores by roughly 10 points on either model pair. With the adaptive rule, both

What carries the argument

Four boundary-region signals—top-k candidate overlap, shared probability mass, a shared-greedy penalty, and an unshared-token penalty—are computed only for positions in the final chunk of the current horizon. Each is converted to a normalized badness score relative to its baseline at horizon entry, the maximum across signals is EMA-smoothed, and a plateau counter triggers expansion. A utilization gate, satisfied when the smoothed horizon-hit rate or boundary-reach rate crosses a threshold, prevents a small tail of long responses from forcing a costlier rollout.

Load-bearing premise

The load-bearing premise is that teacher–student agreement in the current boundary chunk, checked against that chunk's own baseline, is the right proxy for whether the student is ready to benefit from a longer rollout; if plateauing at the boundary does not track the value of longer rollouts, the schedule is driven by a proxy that can mislead.

What would settle it

Run a per-pair tuned fixed expansion schedule with the same total training time as Adaptive FastOPD; if it matches or beats the reported average scores, the progress-aware rule is not the source of the gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Replacing a fixed expansion interval with a stage-relative plateau rule yields higher average benchmark scores and shorter training time than the best fixed schedule in both test pairs.
  • The method's performance stays within a narrow band across the tested hyperparameter settings, whereas fixed schedules vary widely (e.g., 9.1–9.8 point swings between best and worst intervals).
  • Removing the utilization gate hurts: on the Qwen3 pair it lowers accuracy and increases training time, because expansions expose horizons reached by only a small tail of rollouts.
  • The dominant signal varies across horizons and stages, so aggregating multiple complementary signals is needed for the reported performance; the overlap-only variant is fastest but drops average score by 1.9 points.
  • The ablations suggest an accuracy–efficiency frontier: dropping two of the four signals produces close-to-best accuracy in about half the training time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The stage-relative design may transfer to other on-policy or RL settings where budget allocation should follow learning dynamics, not just OPD: any setting with a length/cost axis and a progress signal could use the same plateau-plus-utilization rule.
  • Inference: A testable extension is to replace the four fixed signals with a single learned or automatically selected progress indicator (e.g., boundary-region loss) and check whether stability across hyperparameters survives.
  • Inference: The plateau logic assumes that the boundary region is the only place where longer rollouts add value; if harder tasks show mid-prefix supervision matters more, the rule would need a second monitor and the reported savings would narrow.
  • Inference: The time savings depend on long-tail stragglers dominating batch time; on batching hardware that hides latency, the utilization gate should contribute less, which is directly measurable per horizon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Adaptive FastOPD replaces fixed rollout-horizon schedules in on-policy distillation with a progress-aware expansion rule. Starting from a short horizon H, the method monitors four teacher–student agreement/badness signals in the boundary region [H−ΔH, H), normalizes each signal against its baseline at horizon entry, aggregates them by taking the maximum normalized badness, smooths with an EMA, and triggers an expansion to H+ΔH only after K_pat consecutive non-improving observations and a length-utilization condition based on horizon-hit/boundary-reach rates. Experiments on DAPO-Math-17K with two teacher–student pairs compare against OPD with fixed horizons and fixed-schedule FastOPD, reporting higher macro-average accuracy with 49.1–71.2% less training time than OPD 15K. Ablations vary the expansion interval, hyperparameters, and the set of progress signals.

Significance. If the reported effects are reliable, the idea of stage-relative plateau detection is a useful and well-specified contribution to efficient OPD. The method is described in enough detail to reproduce, the four-signal design is motivated, and the ablations in Tables 2–4 show that fixed schedules are indeed sensitive to the expansion interval and that the multi-signal aggregation matters. However, the empirical base is narrow: two teacher–student pairs, one dataset, and no uncertainty estimates. The headline advantage over fixed schedules is small relative to likely benchmark noise. The central 'progress-aware' claim additionally rests on an unvalidated proxy—plateau in the boundary region—and the paper does not compare against the adaptive alternatives it cites in Related Work.

major comments (3)
  1. [§Method, Eq. (4) and Eq. (16)] The expansion rule assumes that a plateau in normalized badness in B_H=[H−ΔH, H) means the student is ready for longer horizons. Since rollouts are truncated at H, teacher supervision at the final chunk is conditioned on artificial cutoffs; a plateau there may indicate that the student has matched the teacher at the truncation positions, not that positions ≥ H have become learnable. No experiment separates these interpretations. Please validate the proxy—e.g., by probing with occasional longer rollouts before expansion, or by comparing boundary-plateau expansion with a schedule driven by plateau over the full prefix.
  2. [Table 1 / Experiments] The headline 'highest average performance' rests on macro-average differences of 0.4 points (DeepSeek: 56.1 vs 55.7 vs 55.6) and 1.0 point (Qwen3: 20.1 vs 19.1). With no repeated seeds, standard errors, or per-benchmark variance, these differences are within plausible noise for 16-sample exact-match scores. Table 3's robustness range of 55.2–56.1 further suggests sensitivity to training variability. Report multiple seeds with error bars and a paired test, or soften the claim.
  3. [Related Work / Experimental Setup] Several adaptive alternatives are described (Prune-OPD, ESR, PG-OPD, ADWIN) but none is evaluated. Without a comparison on the same data, the paper does not demonstrate that progress-aware expansion outperforms or is more robust than existing adaptive mechanisms; the claim that prior methods are hyperparameter-sensitive is supported only by the fixed-schedule FastOPD ablation. Add at least one adaptive baseline, or explicitly restrict the paper's scope to 'fixed-schedule vs adaptive expansion.'
minor comments (5)
  1. [Eq. (2)] The floor is missing in the expansion formula; it should read floor((n−1)/F) or state that division is integer.
  2. [Eq. (16)] Typo: 'K_pat is the the required number.'
  3. [Tables 2 and 3] Some numerical entries run together (e.g., '4.05.4' should be '4.0 5.4'). Please fix the table formatting.
  4. [§Why Multiple Signals] The ordering 'OGPU' for dominance duration is not defined; spell out which signals dominate in what order.
  5. [Figure 2] The figure is referenced but not visible in the submitted text; ensure the final version includes the three panels and readable axes.

Circularity Check

0 steps flagged

No circularity: the expansion rule is a heuristic control policy and the reported gains are measured on independent benchmarks and runtimes.

full rationale

The paper's derivation chain is empirical rather than definitional. Adaptive FastOPD's horizon update (Eq. 3) is gated by a plateau detector built from normalized teacher–student signals (Eqs. 5–10, 13–16) and by utilization rates (Eqs. 17–19). The claimed outcome—higher average accuracy with less wall-clock time (Tables 1–3)—is evaluated on external benchmarks (AIME, AMC, MATH-500, Minerva, OlympiadBench) and on measured training time, not on the same quantities that trigger expansion. No equation maps the plateau signal into a benchmark score, so the central result is not equivalent to its inputs by construction. The stage-specific baseline in Eq. (12) is a normalization device, not a fitted parameter later reported as a prediction. The 'robustness' claim is supported by a hyperparameter sweep (Table 3) rather than by an identity. The only load-bearing premise that is imported rather than derived is the boundary-region assumption: the Method cites prior work (Li et al. 2026; Ziheng et al. 2026; Zhang et al. 2026a) for the claim that teacher supervision becomes less reliable at later positions and therefore monitors only B_H = [H−ΔH, H) (Eq. 4). That is a substantive assumption and a possible correctness risk—the skeptical concern that a plateau at forced truncation points may reflect truncation artifacts rather than readiness for longer horizons—but it is not circular, because the final accuracy numbers are not computed from the progress signals. No self-citation chain, ansatz smuggled via citation, or renaming of a known result is load-bearing in the reported comparison. Score 0.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

No genuinely new entities are introduced. The paper's 'new' content is a control rule built from seven tunable hyperparameters (N_base, β, K_pat, τ_hit, τ_reach, H max, ΔH, plus the signal set and max-aggregation choice). The rule is evaluated against external benchmarks, so the axiom ledger is dominated by domain assumptions about what the boundary signals mean, not by fitted quantities.

free parameters (7)
  • N_base = 3
    Number of valid boundary observations used to set the stage-relative baseline (Eq. 12). Chosen by the authors; ablation shows 2 vs 3 gives similar accuracy.
  • β = 0.8
    EMA coefficient for smoothed badness (Eq. 15) and utilization rates. Ablated (0.6, 0.8); no principle given for the value.
  • K_pat = 5
    Plateau patience: number of consecutive non-improving smoothed observations before the plateau condition fires (Eq. 16). Ablated (5, 7, 10); no data-driven selection.
  • τ_hit = 0.1
    Minimum smoothed horizon-hit rate for the utilization gate (Eq. 19). Ablated only by setting both thresholds to 0.
  • τ_reach = 0.3
    Minimum smoothed boundary-reach rate for the utilization gate (Eq. 19). Ablated only by setting both thresholds to 0.
  • H_0, ΔH, H_max = 1024, 1024, 15360
    Initial horizon, expansion chunk size, and maximum horizon. All inherited from the FastOPD setup rather than derived; ΔH also defines the boundary-region width (Eq. 4).
  • k (top-k candidate set) = 16
    Candidate set size for OPD and for the overlap/shared-mass signals; ablated only for the standard OPD objective in prior work.
axioms (4)
  • domain assumption Teacher supervision is more reliable on shorter prefixes and becomes less corrective as prefix drift accumulates; hence monitoring only the boundary region is sufficient.
    Adopted from cited prior work (Li et al. 2026; Ziheng et al. 2026; Zhang et al. 2026a) and invoked at the start of the boundary-region paragraph to justify monitoring only positions [H−ΔH, H). The whole plateau signal depends on this proxy being valid.
  • domain assumption The four signals measured on the boundary region—overlap, shared mass, shared-greedy penalty, unshared-token penalty—are informative about future learning at longer horizons.
    The paper motivates this via ablation (Table 4) but provides no mechanism or theory showing that plateauing of these signals predicts the value of expanding by ΔH. The P-signal even produces a 'skip' when no shared candidates exist, silently dropping observations.
  • ad hoc to paper Taking the maximum normalized badness across tied metrics is the right aggregation (Eq. 14).
    The paper justifies max with 'governed by the least aligned metric', but no alternative aggregations (mean, min, weighted) are compared; the choice determines which signal triggers expansion and is part of the tuning surface.
  • domain assumption The environment/generation statistics are stationary enough that a single non-shuffled epoch of 279 steps of DAPO-Math-17K supports comparisons of final downstream accuracy.
    All runs use one epoch without shuffling; benchmark variance under this protocol is not measured, so the significance of small average differences (e.g., 56.1 vs 55.6) is unknown.

pith-pipeline@v1.3.0-daily-deepseek · 11801 in / 9671 out tokens · 74913 ms · 2026-08-03T05:50:38.472984+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher--student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher--student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher--student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1--71.2\% relative to OPD 15K, and remains robust across a range of hyperparameter settings.

Figures

Figures reproduced from arXiv: 2607.29494 by Huaifei Liang, Lei Jiang, Qian Tan, Xuanyu Zhu, Yuqiang Li.

Figure 1
Figure 1. Figure 1: Overview of Adaptive FastOPD. The student first generates responses under the current horizon [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Horizon-adaptation diagnostics for DeepSeek-R1-Distill-Qwen-1.5B with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 16 linked inside Pith

  1. [1]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  2. [7]

    Beat the long tail: Distribution-Aware Speculative Decoding for RL Training

    Shao, Zelei and Srivatsa, Vikranth and Srivastava, Sanjana and Wu, Qingyang and Ariyak, Alpay and Wu, Xiaoxia and Patel, Ameen and Wang, Jue and Liang, Percy and Dao, Tri and Zhang, Ce and Zhang, Yiying and Athiwaratkun, Ben and Xu, Chenfeng and Wang, Junxiong. Beat the long tail: Distribution-Aware Speculative Decoding for RL Training. Proceedings of the...

    2026

  3. [8]

    International Conference on Learning Representations , volume=

    On-policy distillation of language models: Learning from self-generated mistakes , author=. International Conference on Learning Representations , volume=

  4. [9]

    Gu, Yuxian and Dong, Li and Wei, Furu and Huang, Minlie , booktitle=

  5. [10]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  6. [11]

    arXiv preprint arXiv:2602.15260 , year=

    Fast and Effective On-Policy Distillation from Reasoning Prefixes , author=. arXiv preprint arXiv:2602.15260 , year=

    arXiv

  7. [12]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Fast and effective on-policy distillation from reasoning prefixes , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  8. [14]

    ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning , year=

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe , author=. ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning , year=

    2026

  9. [15]

    16th USENIX symposium on operating systems design and implementation (OSDI 22) , pages=

    Orca: A distributed serving system for \ Transformer-Based \ generative models , author=. 16th USENIX symposium on operating systems design and implementation (OSDI 22) , pages=

  10. [16]

    and Zhang, Hao and Stoica, Ion , booktitle=

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle=. Efficient Memory Management for Large Language Model Serving with

  11. [17]

    HybridFlow: A Flexible and Efficient RLHF Framework , author=

  12. [18]

    He, Bingxiang and Qu, Zekai and Liu, Zeyuan and Chen, Yinghao and Zuo, Yuxin and Qian, Cheng and Zhang, Kaiyan and Chen, Weize and Xiao, Chaojun and Cui, Ganqu and others , journal=

  13. [19]

    arXiv preprint arXiv:2501.12948 , year=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  14. [20]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=

    2025

  15. [22]

    arXiv preprint arXiv:2503.14476 , year=

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , author=. arXiv preprint arXiv:2503.14476 , year=

    Pith/arXiv arXiv

  16. [23]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  17. [24]

    Measuring Mathematical Problem Solving With the MATH Dataset , url =

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , booktitle =. Measuring Mathematical Problem Solving With the MATH Dataset , url =

  18. [25]

    Advances in Neural Information Processing Systems , year=

    Solving Quantitative Reasoning Problems with Language Models , author=. Advances in Neural Information Processing Systems , year=

  19. [26]

    Advances in neural information processing systems , volume=

    Solving quantitative reasoning problems with language models , author=. Advances in neural information processing systems , volume=

  20. [27]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  21. [28]

    Yang, An and Zhang, Beichen and Hui, Binyuan and Gao, Bofei and Yu, Bowen and Li, Chengpeng and Liu, Dayiheng and Tu, Jianhong and Zhou, Jingren and Lin, Junyang and others , journal=

  22. [30]

    arXiv preprint arXiv:2505.23281 , year=

    MathArena: Evaluating LLMs on Uncontaminated Math Competitions , author=. arXiv preprint arXiv:2505.23281 , year=

    Pith/arXiv arXiv

  23. [31]

    Advances in Neural Information Processing Systems , volume=

    Matharena: Evaluating llms on uncontaminated math competitions , author=. Advances in Neural Information Processing Systems , volume=

  24. [33]

    arXiv preprint arXiv:2605.13643 , year=

    Prefix teach, suffix fade: Local teachability collapse in strong-to-weak on-policy distillation , author=. arXiv preprint arXiv:2605.13643 , year=

    Pith/arXiv arXiv

  25. [36]

    Wu, Yecheng and Han, Song and Cai, Hai , journal=

  26. [39]

    Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Ramos Garea, S.; Geist, M.; and Bachem, O. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, 21246--21263

    2024

  27. [40]

    Balunovic, M.; Dekoninck, J.; Petrov, I.; Jovanovi \'c , N.; and Vechev, M. 2026. Matharena: Evaluating llms on uncontaminated math competitions. Advances in Neural Information Processing Systems, 38

    2026

  28. [41]

    Gu, Y.; Dong, L.; Wei, F.; and Huang, M. 2024. Minillm : Knowledge distillation of large language models. In The twelfth international conference on learning representations

    2024

  29. [42]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081): 633--638

    2025

  30. [43]

    He, B.; Qu, Z.; Liu, Z.; Chen, Y.; Zuo, Y.; Qian, C.; Zhang, K.; Chen, W.; Xiao, C.; Cui, G.; et al. 2025. Justrl : Scaling a 1.5 b llm with a simple rl recipe. arXiv preprint arXiv:2512.16649

    arXiv 2025

  31. [44]

    He, C.; Luo, R.; Bai, Y.; Hu, S.; Thai, Z.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3828--3850

    2024

  32. [45]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1

    2021

  33. [46]

    A.; Ahmed, A.; Fayyaz, Z.; Di, S.; Hong, M.; and Anwar, A

    Khan, A. A.; Ahmed, A.; Fayyaz, Z.; Di, S.; Hong, M.; and Anwar, A. 2026. Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing. arXiv preprint arXiv:2606.02218

    Pith/arXiv arXiv 2026

  34. [47]

    H.; Gonzalez, J.; Zhang, H.; and Stoica, I

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J.; Zhang, H.; and Stoica, I. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, 611--626

    2023

  35. [48]

    Lewkowycz, A.; Andreassen, A.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; et al. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35: 3843--3857

    2022

  36. [49]

    Li, Y.; Zuo, Y.; He, B.; Zhang, J.; Xiao, C.; Qian, C.; Yu, T.; Huan-ang Gao ; Yang, W.; Liu, Z.; and Ding, N. 2026. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning

    2026

  37. [50]

    Liang, K.; Tang, C.; Bai, C.; Liu, W.; Yang, S.; and Wu, Y. 2026. ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation. arXiv preprint arXiv:2605.28396

    Pith/arXiv arXiv 2026

  38. [51]

    Luo, F.; Chuang, Y.-N.; Wang, G.; Xu, Z.; Han, X.; Zhang, T.; and Braverman, V. 2026. Demystifying opd: Length inflation and stabilization strategies for large language models. arXiv preprint arXiv:2604.08527

    Pith/arXiv arXiv 2026

  39. [52]

    Shao, Z.; Srivatsa, V.; Srivastava, S.; Wu, Q.; Ariyak, A.; Wu, X.; Patel, A.; Wang, J.; Liang, P.; Dao, T.; Zhang, C.; Zhang, Y.; Athiwaratkun, B.; Xu, C.; and Wang, J. 2026. Beat the long tail: Distribution-Aware Speculative Decoding for RL Training. In Proceedings of the 9th Annual Conference on Machine Learning and Systems (MLSys)

    2026

  40. [53]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. HybridFlow: A Flexible and Efficient RLHF Framework

    2025

  41. [54]

    Wu, Y.; Han, S.; and Cai, H. 2026. Lightning opd: Efficient post-training for large reasoning models with offline on-policy distillation. arXiv preprint arXiv:2604.13010

    Pith/arXiv arXiv 2026

  42. [55]

    Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  43. [56]

    Yang, A.; Zhang, B.; Hui, B.; Gao, B.; Yu, B.; Li, C.; Liu, D.; Tu, J.; Zhou, J.; Lin, J.; et al. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122

    Pith/arXiv arXiv 2024

  44. [57]

    Yang, Z.; Guo, Z.; Song, Y.; Xu, M.; Wang, Y.; Wang, Y.; Liang, X.; and Tang, J. 2026. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning. arXiv preprint arXiv:2605.07804

    Pith/arXiv arXiv 2026

  45. [58]

    S.; Kim, G.-W.; Kim, S.; and Chun, B.-G

    Yu, G.-I.; Jeong, J. S.; Kim, G.-W.; Kim, S.; and Chun, B.-G. 2022. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), 521--538

    2022

  46. [59]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38: 113222--113244

    2026

  47. [60]

    D.; Batra, S

    Zhang, D.; Yang, Z.; Janghorbani, S.; Han, J.; Ressler II, A.; Qian, Q.; Lyng, G. D.; Batra, S. S.; and Tillman, R. E. 2026 a . Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, 25553--25569

    2026

  48. [61]

    Zhang, Q.; Yuan, Q.; Lin, H.; Lu, Y.; Han, X.; Sun, L.; Li, X.; Xu, M.; Li, J.; and Zhao, X. 2026 b . ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation. arXiv preprint arXiv:2607.13124

    Pith/arXiv arXiv 2026

  49. [62]

    Zhang, Y.; Chai, J.; Fu, Y.; Tu, S.; Wang, X.; Lin, W.; Yin, G.; Zhang, Q.; Zhu, Y.; and Zhao, D. 2026 c . Are Full Rollouts Necessary for On-Policy Distillation? arXiv preprint arXiv:2605.31490

    Pith/arXiv arXiv 2026

  50. [63]

    Zhao, Q.; Song, H.; Tian, S.; Shao, J.; and Li, X. 2026. Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts. arXiv preprint arXiv:2606.21994

    Pith/arXiv arXiv 2026

  51. [64]

    N.; and Terzopoulos, D

    Ziheng, Z.; Li, J.; Tang, H.; Wu, Y. N.; and Terzopoulos, D. 2026. Less is more: Early stopping rollout for on-policy distillation. arXiv preprint arXiv:2605.27028

    Pith/arXiv arXiv 2026