Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Relative update signals from a weak proxy can improve a stronger LLM without the strong model exploring rewards itself.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 05:07 UTC pith:QDW5TLAS

load-bearing objection Solid modular packaging of relative-log-ratio transfer that works on Qwen3 math/code; the modularity claim is real but still λ-tuned and family-bound. the 3 major comments →

arxiv 2607.11505 v1 pith:QDW5TLAS submitted 2026-07-13 cs.LG cs.AI

Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

Daocheng Fu , Rong Wu , Yu Yang , Xuemeng Yang , Jianbiao Mei , Licheng Wen , Pinlong Cai , Yong Liu
show 2 more authors
Botian Shi Yu Qiao
This is my paper
classification cs.LG cs.AI
keywords post-trainingproxy explorationupdate signal transferweak-to-strongdistribution matchingreward optimizationLLM alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Post-training usually forces the model you care about to do its own expensive trial-and-error search for high-reward answers, which makes exploration hard to cache, reuse, or share across models. This paper claims that search can be offloaded to a cheap proxy: run ordinary reward optimization on the proxy, then extract only the relative log-ratio change between the proxy before and after that training. That directional signal is transferred to the primary model through a calibrated alignment objective so the primary never has to sample rewards itself. Because the transferred object is a relative improvement rather than the proxy’s final absolute policy, weaker proxies can still raise stronger primaries, and the same cached signal can be applied to several targets. On Qwen3 models in math and code, signals from 1.7B and 4B proxies raise an 8B primary, with transfer strength controlled by a single scalar. The practical stake is modular, asynchronous post-training: explore once on a small model, reuse many times on larger ones.

Core claim

The paper establishes that token-level relative log-ratio improvements extracted from a proxy’s reward optimization can be transferred, after simple anchor calibration, to guide a different primary model’s policy alignment, so that update signals from substantially weaker proxies can improve stronger primaries on math and code without the primary performing its own reward exploration.

What carries the argument

Proxy-guided Update Signal Transfer (PUST): extract Δϕ = log(π+ϕ / πϕ) from the proxy pair, form the calibrated utility rλ = Δϕ − λ Δθ against the primary’s frozen anchor, and optimize the primary by maximizing expected utility (equivalently a KL to a dynamically induced target).

Load-bearing premise

A single scalar calibration is enough for the proxy’s relative token-level improvement to remain a useful directional guide for a differently sized primary model that never explores rewards itself.

What would settle it

Transfer the same proxy-extracted signal to a larger primary on held-out math or code tasks and check whether performance fails to rise, or rises only when λ is tuned so aggressively that the primary collapses, relative to direct reward optimization of the primary under matched data and compute.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Proxy-guided Update Signal Transfer (PUST), a post-training framework that decouples policy exploration from alignment. A lightweight proxy is optimized with GRPO to discover high-reward behaviors; the relative token-level log-ratio improvement Δϕ = log(π+ϕ/πϕ) (Eq. 1) is extracted and transferred to a primary model via an anchor-calibrated utility rλ = Δϕ − λΔθ (Eqs. 3–5), which is shown to be equivalent to KL matching to a dynamically constructed target. Experiments on Qwen3-1.7B/4B/8B across math (DeepMath-103K) and code (Eurus-RL-Code) claim that signals from weaker proxies improve stronger primaries (Tables 1–2), can be reused across primaries with few steps (Table 3), exhibit partial transitivity (Table 4), and are adjustable via λ (Fig. 4, Appendix Tables 8–9).

Significance. If the central claim holds, PUST would convert post-training from a monolithic on-policy process into a modular pipeline of asynchronous proxy exploration, cacheable relative signals, and calibrated transfer, supporting weak-to-strong improvement and cross-model reuse without primary-side reward rollouts. The formulation (Eqs. 1–5) is clean and standard, the empirical gains on held-out math/code benchmarks are clear within the Qwen3 family, and the paper correctly distinguishes relative-signal transfer from absolute distillation (related to Direct-OPD). Strengths include explicit weak-to-strong results, signal reusability demonstrations, and open code/model links. These would be practically valuable for multi-domain and multi-scale LLM post-training if robustness beyond a single family and fixed λ is established.

major comments (3)
  1. [§3.3–3.4, Eqs. (1)–(5), Fig. 4] §3.3–3.4 and Eqs. (1)–(5): The load-bearing premise is that the proxy’s token-level relative log-ratio Δϕ remains a meaningful directional improvement for a different primary after only a scalar anchor calibration λ. Fig. 4 and Appendix Tables 8–9 show high sensitivity: optimal λ shifts with proxy–primary gap (≈1.08 for 4B→8B vs ≈1.57 for 1.7B→8B) and too-small λ collapses accuracy. All positive results use post-hoc fixed λ chosen on the same transfer setting. This does not yet establish that Δϕ is robustly reusable once the primary leaves the Qwen3 logit geometry or when λ cannot be re-tuned on the target; adaptive or per-state calibration (flagged as future work in §6) is needed to support the modularity claim.
  2. [Tables 1–4, §4.1–4.2] Tables 1–4 and §4.1–4.2: Main results report only point estimates (Mean@16) with no seed variance, confidence intervals, or multiple independent runs. Given that λ is a free parameter and transfer is sensitive to it, the absence of error bars makes it impossible to judge whether the reported gains (e.g., 4B→8B average +30.2 on math) are statistically reliable or whether reuse/transitivity claims hold under re-sampling. At minimum, multi-seed means and stds (or bootstrap CIs) on the primary transfer runs are required for the central empirical claims.
  3. [§4, Appendix B] §4 and Appendix B: All experiments stay inside the Qwen3 family (1.7B/4B/8B, non-thinking). Cross-family transfer (different tokenizer, pretraining mixture, or architecture) is not tested, yet the abstract and conclusion claim “seamless cross-model transfer.” A single out-of-family primary (or at least a different base of comparable scale) would substantially strengthen or falsify the reusability claim; without it the modularity argument remains family-specific.
minor comments (5)
  1. [§2, Fig. 2] Fig. 2 caption and §2: OPD variants are extended to 500 steps “based on converged-region statistics”; clarify exactly how mean/std of the plateau are used and whether early stopping was applied to GRPO for fairness.
  2. [Table 2] Table 2 footnote 1 notes an inherent LCB inversion between Qwen3-4B and 8B base; a short discussion of why the base ranking is non-monotonic would help readers interpret the +8.3 LCB gain under PUST.
  3. [§3.4.2, Eq. (5)] §3.4.2 Eq. (5): The equivalence to KL against a dynamic target is stated but not written out; a one-line derivation of the induced target π* ∝ π_ref · (π+ϕ/πϕ)^{1/λ} (or similar) would make the distribution-matching view fully explicit.
  4. [Appendix B.1] Appendix B.1: Hyper-parameter tables list KL coefficient 0.0 for both GRPO and transfer; confirm whether any other regularizer (entropy bonus, etc.) is present, as the entropy plots in Figs. 11/15 suggest non-trivial entropy dynamics.
  5. [§A.3] Related work §A.3 correctly cites Direct-OPD; a short explicit comparison table (absolute vs relative signal, cacheability, λ calibration) would help position the novelty more sharply.

Circularity Check

0 steps flagged

No significant circularity: PUST is an empirical transfer method whose relative-signal definitions and primary-model gains are measured, not forced by construction or self-citation.

full rationale

The paper proposes a three-stage pipeline (proxy GRPO exploration, extraction of token-level relative log-ratio Δϕ = log(π⁺_ϕ/π_ϕ) in Eq. 1, and calibrated transfer via r_λ = Δϕ − λ Δ_θ in Eqs. 3–5). These equations simply define the training objective; they do not claim a first-principles derivation of an independent quantity that is then “predicted.” Performance claims (Tables 1–4, Figs. 2, 4, 5) are ordinary empirical measurements of Mean@16 / Pass@k gains of the primary model on held-out math and code benchmarks after the transfer stage. The scalar λ is an openly swept hyper-parameter (Fig. 4, Appendix Tables 8–9); reporting results under tuned λ is standard practice, not a fitted input renamed as an independent prediction. Related-work citations (OPD, Direct-OPD, weak-to-strong) supply context and are not load-bearing uniqueness theorems authored by the present team. No equation reduces by construction to its own inputs, no self-citation chain forces the central claim, and the method is falsifiable on external benchmarks. Hence circularity score 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 1 invented entities

The central claim rests on standard RL/distillation math plus one free scalar (λ) and the modeling choice that relative proxy log-ratios are transferable directional signals. No new physical entities; the ‘update signal’ is an operational construct defined by Eq. 1. Domain assumptions are ordinary for LLM post-training (verifiable rewards, shared vocabulary, on-policy sampling).

free parameters (1)
  • calibration coefficient λ = typically 1.0–1.5; fitted optima ~1.08 and ~1.57
    Controls how strongly the primary is penalized for already absorbing the proxy direction (Eq. 3). Chosen per proxy–primary pair (e.g. 1.0 vs 1.5) and further optimized by polynomial fit (λ*≈1.08 for 4B→8B, ≈1.57 for 1.7B→8B). Performance is sensitive to this choice (Fig. 4, Tables 8–9).
axioms (4)
  • ad hoc to paper Token-level relative log-ratio Δϕ = log(π+ϕ/πϕ) is a transferable directional improvement signal independent of the proxy’s absolute capability.
    Core modeling choice of §3.3; enables weak-to-strong claims. Supported empirically but not derived from first principles.
  • domain assumption Anchor-calibrated utility rλ = Δϕ − λ Δθ yields stable policy improvement when maximized under the primary’s own distribution (Eqs. 4–5).
    Standard KL-regularized matching form; assumes a fixed scalar λ suffices across states.
  • domain assumption Rule-based verifiable rewards (correct/incorrect) and GRPO group advantages provide reliable exploration targets for math/code proxies.
    Standard in recent math/code RL; stated in §B.1.
  • domain assumption Shared vocabulary and comparable tokenization across Qwen3 sizes allow direct transfer of token-level log-ratios.
    Implicit in setup §3.1; required for Δϕ to be well-defined on the primary.
invented entities (1)
  • Proxy-guided Update Signal (relative improvement Δϕ cached as a transferable object) no independent evidence
    purpose: Decouple exploration from primary alignment so signals can be stored, reused, and applied across models.
    Defined operationally by Eq. 1 and the three-stage pipeline; not an independent physical object. Independent evidence is only the empirical transfer experiments in this paper.

pith-pipeline@v1.1.0-grok45 · 21751 in / 3156 out tokens · 38170 ms · 2026-07-14T05:07:58.055858+00:00 · methodology

0 comments
read the original abstract

Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.

Figures

Figures reproduced from arXiv: 2607.11505 by Botian Shi, Daocheng Fu, Jianbiao Mei, Licheng Wen, Pinlong Cai, Rong Wu, Xuemeng Yang, Yong Liu, Yu Qiao, Yu Yang.

Figure 1
Figure 1. Figure 1: Comparison of post-training pipelines. (a) Serial: Sequential domain exploration, risking catastrophic forgetting. (b) Parallel: Parallel domain exploration followed by unified policy alignment. (c) Proxy Asynchronous (Ours): A proxy model conducts asynchronous exploration; extracted signals are subsequently transferred to various primary models. Decoupling these update signals from the base model enables … view at source ↗
Figure 2
Figure 2. Figure 2: Mean@16 trajectories on AIME2024 and AIME2025. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the PUST mechanism. This pipeline decouples exploration from alignment via three sequential stages. First, a relative update signal is extracted from the explored proxy model pair. Next, to prevent over-updating, the signal is dynamically calibrated against the primary model’s anchor. Finally, the calibrated signal is transferred to guide the primary model’s alignment, ensuring stable and f… view at source ↗
Figure 4
Figure 4. Figure 4: Sensitivity analysis on different proxy models and calibration coefficients. Each curve is fitted to the experimental results using a cubic polynomial. A value of R2 closer to 1 indicates a better fit. Building on the previous experiments, we show that proxy-explored update signals can improve the primary model and be transferred or reused. To study the transfer behavior, we conduct a sen￾sitivity analysis… view at source ↗
Figure 6
Figure 6. Figure 6: ∆ Pass@K (pass@k - pass@1) over the number of samples k. As the number of model samples increases, accuracy steadily improves, indicating that more useful information is being extracted. 4.5 Comparison of Exploration Quality Between Proxy and Primary Models The update signals explored by a proxy model improve the primary model, but an important question remains: how large is the gap between signals explore… view at source ↗
Figure 7
Figure 7. Figure 7: Revisiting the relationship between data/environment and models. By decoupling the exploration process [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Response length during GRPO training of the proxy expert models [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Mean@16 of Math Test under differ￾ent calibration coefficient λ (4B → 8B). 1 10 20 30 40 50 60 70 Training Steps 0.25 0.30 0.35 0.40 0.45 Entropy λ=0.5 λ=0.75 λ=0.95 λ=1.0 λ=1.1 λ=1.5 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 13
Figure 13. Figure 13: Training loss under different calibra￾tion coefficient λ (4B → 8B). 1.7B→8B transfer. Figures 14–17 and [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Conditioned entropy of primary model under different λ (1.7B → 8B). 1 10 20 30 40 50 60 70 Training Steps 0.00 1.00 2.00 3.00 4.00 Grad Norm λ=0.5 λ=1.0 λ=1.25 λ=1.5 λ=1.75 [PITH_FULL_IMAGE:figures/full_fig_p017_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Training loss under different calibra￾tion coefficient λ (1.7B → 8B). 17 [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 22 linked inside Pith

  1. [1]

    A survey of post-training scaling in large language models

    Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, et al. A survey of post-training scaling in large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 2771–2791, 2025

    2025

  2. [2]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  3. [3]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024

    2024

  5. [5]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Tuning language models by proxy.arXiv preprint arXiv:2401.08565, 2024

    Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy.arXiv preprint arXiv:2401.08565, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Weak-to-strong generalization via direct on-policy distillation.arXiv preprint arXiv:2607.05394, 2026

    Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, and Hao Zhou. Weak-to-strong generalization via direct on-policy distillation.arXiv preprint arXiv:2607.05394, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13(9):9, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13(9):9, 2024

    2024

  11. [11]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: //github.com/open-compass/opencompass, 2023

    2023

  12. [12]

    Matharena: Evaluating llms on uncontaminated math competitions.Advances in Neural Information Processing Systems, 38, 2026

    Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovi´c, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions.Advances in Neural Information Processing Systems, 38, 2026

    2026

  13. [13]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  14. [14]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  15. [15]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, volume 2025, pages 58791–58831, 2025

    2025

  17. [17]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025. 10

    Pith/arXiv arXiv 2025

  19. [19]

    Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxi- ang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Pith/arXiv arXiv 1909

  21. [21]

    Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704, 2024

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601, 2024

    2024

  24. [24]

    Genprm: Scaling test-time compute of process reward models via generative reasoning.arXiv preprint arXiv:2504.00891, 2025

    Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. Genprm: Scaling test-time compute of process reward models via generative reasoning.arXiv preprint arXiv:2504.00891, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling.arXiv preprint arXiv:2502.06703, 2025

    Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling.arXiv preprint arXiv:2502.06703, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Tldr: Token-level detective reward model for large vision language models

    Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. Tldr: Token-level detective reward model for large vision language models. InInternational Conference on Learning Representations, volume 2025, pages 45205–45236, 2025

    2025

  27. [27]

    Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. InFindings of the Association for Computational Linguistics: ACL 2024, pages 14969–14981, 2024

    2024

  28. [28]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023

    2023

  29. [29]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  30. [30]

    How abilities in large language models are affected by supervised fine-tuning data composition

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computat...

    2024

  31. [31]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  32. [32]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819, 2021

    2021

  33. [33]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016

    2016

  34. [34]

    Gkd: A general knowledge distillation framework for large-scale pre-trained language model

    Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Shu Zhao, Peng Zhang, and Jie Tang. Gkd: A general knowledge distillation framework for large-scale pre-trained language model. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 5: Industry Track), pages 134–148, 2023

    2023

  35. [35]

    Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025. 11

    Pith/arXiv arXiv 2025

  36. [36]

    Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  37. [37]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Fast and effective on-policy distillation from reasoning prefixes

    Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25553–25569, 2026

    2026

  39. [39]

    Black-box on-policy distillation of large language models.arXiv preprint arXiv:2511.10643, 2025

    Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-box on-policy distillation of large language models.arXiv preprint arXiv:2511.10643, 2025

    arXiv 2025

  40. [40]

    Improving weak-to-strong generalization with scalable oversight and ensemble learning.arXiv preprint arXiv:2402.00667, 2024

    Jitao Sang, Yuhang Wang, Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye, Shuyu Wei, and Jinlin Xiao. Improving weak-to-strong generalization with scalable oversight and ensemble learning.arXiv preprint arXiv:2402.00667, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Incentivizing strong reasoning from weak supervision

    Yige Yuan, Teng Xiao, Shuchang Tao, Xue Wang, Jinyang Gao, Bolin Ding, and Bingbing Xu. Incentivizing strong reasoning from weak supervision. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (V olume 1: Long Papers), pages 7138–7156, 2026

    2026

  42. [42]

    Weak-to-strong preference optimiza- tion: Stealing reward from weak aligned model

    Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong preference optimiza- tion: Stealing reward from weak aligned model. InInternational Conference on Learning Representations, volume 2025, pages 37116–37144, 2025. 12 Appendix Contents A Related Works 13 A.1 Reward Optimization . . . . . . . . . . . . . . . . . . . . . . . . . ...

    arXiv 2025