Pith. sign in

REVIEW 3 major objections 6 minor 64 references

Online Knowledge Distillation with Reward Guidance

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes a reward-guided imitation learning framework for knowledge distillation in which a student policy and a reward model play a min-max game, and argues it outperforms prior KD baselines on ten LLM benchmarks while…

desk verdict Genuinely new min-max framework for preference-based KD with strong empirical results, but the online regret guarantee does not cover the implemented algorithm. read the letter →

arxiv 2505.18952 v1 pith:YXNXTX5X submitted 2025-05-25 cs.LG

Chen Jia This is my paper
classification cs.LG
keywords knowledgedistillationpreferenceoptimizationrewardmodelimitationlearningmin-maxlargelanguagemodelsregretboundsuboptimality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation for large language models is usually done by making the student copy the teacher's outputs. This paper instead frames distillation as reward-guided imitation learning: a student policy and a reward model are trained in a min-max game, where the reward model, kept within a confidence set built from preference data, tries to maximize the performance gap between teacher and student while the student minimizes the worst-case gap. The framework is instantiated in offline, online, and white-box moment-matching variants. The paper claims that the offline student attains suboptimality $O(\sqrt{\log(N/\delta)/N})$ and the online student attains regret $O(\sqrt{T\log T\log(T/\delta)})$, and that all variants outperform prior KD baselines on five black-box and five white-box LLM benchmarks.

What carries the argument

The central object is the confidence set $\mathcal{R}(D^{\mathrm{pref}}) = \{ r\in\mathcal{G}_r : L_r(D^{\mathrm{pref}}) \ge \max_{r\in\mathcal{G}_r} L_r(D^{\mathrm{pref}}) - \zeta \}$ of reward functions consistent with preference data, together with the min-max objective $\hat\pi = \arg\min_{\pi\in\Pi}\max_{r\in\mathcal{R}(D^{\mathrm{pref}})} J(\pi_E,r) - J(\pi,r)$. The inner maximization selects the least favorable reward model in the confidence set, making the student robust to reward uncertainty rather than tied to a single MLE reward estimate. In the white-box variant, the performance difference lemma rewrites this same objective as a moment-matching objective between teacher Q-values and the student's realized Q-values, which admits on-policy RL optimization.

What would settle it

Take a teacher that is weaker than the student on a subset of prompts, run online PbKD with the always-teacher label and with a judge-sampled label, and track per-iteration improvement; if the always-teacher variant plateaus or degrades while the judge-sampled variant keeps improving, the 'teacher always preferred' assumption is the active ingredient.

Watch

Extended reading notes

Core claim

The paper's central claim is that preference-based knowledge distillation can be formulated as a min-max optimization between a student policy and a reward model, where the reward model is restricted to a maximum-likelihood confidence set defined by preference data. In this formulation the reward model is the adversary: it selects the reward function within the confidence set that makes the student look worst relative to the teacher, and the student optimizes against this worst case. The paper argues this distributionally robust objective is more appropriate than direct teacher-cloning, especially when the teacher's outputs are suboptimal for a downstream task or when the capacity gap is large. The claim is supported by suboptimality and regret bounds and by experiments on ten LLM benchmarks, where the online variant improves steadily with iterations.

Load-bearing premise

The online algorithm assumes the teacher's responses are always preferred over the student's and sets every preference label accordingly, whereas the regret theorem assumes preferences are sampled from a stochastic Bradley–Terry model; if teachers are not always better, the collected labels mislead the reward model.

Editorial extensions

If this is right

  • The student learns to close the gap to the teacher under the worst plausible reward function, so it is not tied to a single estimated reward model.
  • In the online variant, each iteration augments the preference set with comparisons between the current student and the teacher, yielding a self-improving distillation loop.
  • The white-box variant turns preference-based KD into a Q-function matching problem, making on-policy policy-gradient algorithms applicable to distillation.
  • The bounds imply that, up to problem-dependent constants, $O(\log(1/\delta)/\epsilon^2)$ preference samples or iterations suffice for an $\epsilon$-optimal student in both settings.
  • If the teacher is suboptimal for a downstream task, the min-max objective can in principle avoid amplifying the teacher's mistakes, unlike direct behavior cloning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A mismatch worth probing: Algorithm 2 labels every online sample with $o_{t-1}=1$ (teacher always preferred), while Theorem 2's regret bound assumes preferences sampled from a Bradley–Terry model with a realizable, bounded reward; deterministic labels cannot be represented by a bounded reward difference without saturating the sigmoid, so the theory does not cover the implemented algorithm as state
  • One testable extension is to replace the always-teacher label with an adaptive preference signal (e.g., a judge or a learned reward) in the online loop and compare regret; the paper's reported gains may be driven partly by the deterministic labels rather than by the min-max machinery.
  • The min-max confidence-set formulation is essentially distributionally robust RLHF restricted to a distillation gap, so its regret analysis could transfer to general preference-based alignment, not only teacher–student distillation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes a preference-based knowledge distillation (PbKD) framework for LLMs, framed as a min-max game between a student policy and a reward model constrained to a MLE-based confidence set. The authors present an offline variant (Algorithm 1), an online variant with teacher/student preference collection (Algorithm 2), and a white-box moment-matching extension (Section 6). The main theoretical claims are an offline suboptimality bound of O(sqrt(log(N/δ)/N)) (Theorem 1) and an online regret bound of O(sqrt(T log T log(T/δ))) (Theorem 2). Experiments on five black-box and five white-box benchmarks show consistent gains over several KD baselines.

Significance. If the theoretical results were valid, this would be a meaningful step: it provides a principled confidence-set formulation for preference-guided distillation, with explicit sample-complexity and regret guarantees, and the empirical results are extensive and reproducible in their setup. The paper also ships a detailed appendix with proofs and ablations. However, as written, the online regret bound does not cover the algorithm that is actually implemented and evaluated, and the white-box extension lacks its own analysis. These gaps materially reduce the strength of the central claims, though they do not invalidate the empirical findings.

major comments (3)
  1. [Section 5.1, Eq. (1), Theorem 2, Appendix A.3] The regret bound assumes preferences are drawn from a BTL model with a realizable reward r* in Gr, but Algorithm 2 sets o_{t-1}=1 deterministically for every pair. Under Assumption 2, the reward difference is bounded, so sigma(r(tau0)-r(tau1)) < 1 for all r in Gr; hence no realizable reward induces the observed labels. This breaks Assumption 1 and the conditional-expectation step in Lemma 7: with o_s = 1, E[exp(xi_s)|F_{s-1}] = exp(ell_r(1;·)), so inequality (30) no longer controls the BTL-expected loss needed by Lemma 3. The bound of Theorem 2 is therefore unproven for Algorithm 2.
  2. [Proof of Theorem 2, Step 2] The elliptical potential bound is invoked for a policy pi_t that maximizes ||phi(x,pi_E)-phi(x,pi)||^2_{Sigma_t^{-1}}, but Algorithm 2 defines pi_t as the solution of the min-max problem in Eq. (10), and Algorithm 3 only adds an approximate uncertainty gradient. No statement in the paper ensures that pi_t is the optimistic maximizer, so the cumulative norm bound does not follow for the presented method.
  3. [Section 6] The moment-matching (MM PbKD) variant is presented as part of the framework, but Theorems 1 and 2 are derived for the reward-guided objective in Eq. (3), not for the Q-function reformulation in Eq. (12). The paper does not provide any formal guarantee for the white-box variant, which is a stated contribution of the paper.
minor comments (6)
  1. [Section 4.2, Remark 1] The claim of a convergence rate O(sqrt(log(N/δ)/N)) omits the dependence on the concentrability coefficient C_r and the constant kappa, which can be large or unbounded; the remark should state the bound with these dependencies.
  2. [Algorithm 2] In the loop, 'tau_{1,t-1} ~ pi_{t-1} | x_t' should read '| x_{t-1}'.
  3. [Section 5.1] The sentence 'we collect a the preference-labeled sample' contains a grammatical error.
  4. [Appendix C, Step 2] The word 'Definding' should be 'Defining'.
  5. [Limitations] The list does not mention the mismatch between the theoretical assumptions and the deterministic online labeling scheme; this should be added.
  6. [Tables 1 and 2] The formatting of '⌞ Best-of-N' is inconsistent; consider aligning it with other baselines.

Circularity Check

1 steps flagged · score 2.0 of 10

No circular derivation: the offline and online bounds follow from stated realizability/concentration assumptions; the only circularity-adjacent element is a mild empirical loop in which GPT-4 serves as teacher, preference labeler, and supplementary judge.

  1. other [Section 7.1 Experimental Setup; Appendix E.2 Evaluation metrics]
    "Offline preference data is pre-collected by fine-tuning several LLMs on 10,000 samples to generate candidate outputs, ranked by GPT-4 feedback. ... Besides, we include the GPT-4 feedback scores [60] as a supplementary evaluation metric, by asking GPT-4 to compare model-generated responses with the ground truth answers."

    GPT-4 is simultaneously (a) the black-box teacher whose outputs are distilled, (b) the labeler that ranks preference pairs used to train the reward model, and (c) the judge for the supplementary GPT-4 feedback score. Optimizing the student to be preferred by the GPT-4-based RM therefore directly inflates the GPT-4-judged metric, so part of the reported improvement on that column is a self-consistency artifact rather than an independent measure of task quality. The reduction is empirical rather than algebraic, but the GPT-4 feedback column is partly circular. The primary ROUGE-L results and the theoretical bounds are independent.

full rationale

The derivation chain itself is not circular. Theorem 1 bounds suboptimality through the MLE confidence set and a concentrability coefficient, with standard bracketing and concentration arguments; the bound is not definitionally equal to the training objective. Theorem 2's proof is also a standard martingale plus elliptical-potential argument, although its stated policy-selection assumption (optimistic argmax of uncertainty) does not match Algorithm 2's min-max update and the always-teacher labels in Algorithm 2 violate BTL realizability; those are correctness gaps, not circularity. The white-box moment-matching section cites the performance difference lemma [44,26] and builds on AMMD [26] and Swamy et al. [44] rather than smuggling in an unverified self-citation. The only circularity-adjacent element is the use of GPT-4 for both preference labeling and a supplementary evaluation metric, which inflates that specific column by construction but does not affect the main suboptimality/regret claims or the ROUGE-L comparisons. Score 2 reflects that mild empirical loop, with the central claims intact.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central bounds rest on realizability and linearity of the reward, plus a BTL preference model. The online algorithm breaks BTL by using deterministic labels, and the hyperparameters beta, zeta, alpha are tuned by hand. No new entities are postulated.

free parameters (4)
  • Beta (Lagrange multiplier)
    Controls the trade-off between the performance gap and the MLE regularization in Eq. (5), Eq. (9), and Algorithm 3; no value or tuning procedure is reported.
  • Zeta (confidence radius)
    Slack in the confidence set definition (Eq. (4), Eq. (11), Eq. (13)); theoretical role, no practical value given.
  • Alpha (uncertainty bonus weight)
    Weight for the uncertainty-based exploration term in Algorithm 3; no value reported.
  • Number of online iterations T = 5 (black-box), 3 (white-box)
    Chosen by hand; results improve with more iterations, so it is a tuned hyperparameter rather than a derived quantity.
assumptions (6)
  • domain assumption The ground-truth reward r* lies in the reward function class G_r (Assumption 1)
    Realizability is standard in preference-based RL theory but unlikely to hold exactly for a linear reward layer on fixed LLM features.
  • domain assumption Reward is linearly parameterized as r_theta(x,tau) = theta^T phi(x,tau) with bounded features and parameters (Assumption 2)
    Used to bound the bracketing number and simplify concentrability; a strong restriction for LLM outputs.
  • domain assumption Preference labels are generated by a Bradley-Terry-Luce model with a realizable reward (Eq. (1))
    Underlies the MLE loss and the confidence set; contradicted by the deterministic always-teacher labels in Algorithm 2.
  • ad hoc to paper Online preference labels are always 'teacher wins' (Section 5.1)
    A simplifying assumption for data collection, but inconsistent with the stochastic BTL model used in Theorem 2.
  • domain assumption Deterministic state transitions in the episodic MDP
    Language generation is modeled as deterministic prefix transitions; standard in sequence-level KD formulations.
  • domain assumption Bounded inverse-sigmoid slope: ||1/(sigma(1-sigma))||_inf <= kappa
    Technical assumption used in Lemma 1 to translate TV distance into reward difference; untested for neural reward features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Knowledge Distillation with Reward Guidance." pith.science (2026). https://pith.science/paper/YXNXTX5X

@misc{pith2026250518952,
  author       = {Pith},
  title        = {Pith review of: Online Knowledge Distillation with Reward Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXNXTX5X}},
  note         = {Machine review of arXiv:2505.18952}
}
abstract

This work studies knowledge distillation (KD) for large language models (LLMs) through preference optimization. We propose a reward-guided imitation learning framework for sequential KD, formulating a min-max optimization problem between the policy and reward model (RM) to minimize the performance gap between the student and teacher policies. Specifically, the reward optimization is constrained to achieve near-optimality within a confidence set for preference alignment. For preference data construction, we explore both offline and online preference-based KD. Additionally, we reformulate the RM using the $Q$-value function and extend the framework to white-box KD, where the teacher policy's predicted probabilities are accessible. Theoretical analysis and empirical results demonstrate the effectiveness of the proposed framework.

Figures

Figures reproduced from arXiv: 2505.18952 by the authors.

Figure 1
Figure 1. We formulate reward-guided imitation learning as the optimization of the performance gap [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The prompt template for GPT-4 feedback. Evaluation. Evaluation metrics include ROUGE-L [33]. Besides, we include the GPT-4 feedback scores [60] as a supplementary evaluation metric, by asking GPT-4 to compare model-generated responses with the ground truth answers and raise 1-10 scores for both responses. We the prompt is largely followed [29] and illustrated in [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗
Figure 3
Figure 3. Impact of reward model (Q-Function) size on performance over online iterations [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 32 canonical work pages

  1. [1]

    Improved algorithms for linear stochastic bandits

    Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011

    2011

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv, 2023

    2023

  3. [3]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024

    2024

  4. [4]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  5. [5]

    Claude 3 family

    Anthropic. Claude 3 family. https://www.anthropic.com/index/claude-3, 2024. Ac- cessed: 2024-06-04

    work page 2024

  6. [7]

    Gpt-4 is openai’s most advanced system, producing safer and more useful responses, 2024

    Andre Brown. Gpt-4 is openai’s most advanced system, producing safer and more useful responses, 2024

    work page 2024

  7. [8]

    Value-incentivized preference optimization: A unified approach to online and offline rlhf

    Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320, 2024

    arXiv 2024

  8. [9]

    Knowledge distillation of black-box large language models, 2024

    Hongzhan Chen, Ruijun Chen, Yuqi Yi, Xiaojun Quan, Chenliang Li, Ming Yan, and Ji Zhang. Knowledge distillation of black-box large language models, 2024

    work page 2024

Show all 64 references
  1. [10]

    Information-theoretic considerations in batch reinforcement learning

    Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International conference on machine learning, pages 1042–1051. PMLR, 2019

    2019

  2. [11]

    Distilling knowledge learned in bert for text generation

    Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling knowledge learned in bert for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7893–7905, 2020

    2020

  3. [12]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  4. [13]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  5. [14]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  6. [15]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [16]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    2023

  8. [17]

    Ultrafeedback: Boosting language models with scaled ai feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pages 9722–9744. PMLR, 2024. 10

    2024

  9. [18]

    Stochastic linear optimization under bandit feedback

    Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. In 21st Annual Conference on Learning Theory, number 101, pages 355–366, 2008

    2008

  10. [19]

    Openllama: An open reproduction of llama

    Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama. URL: https://github. com/openlm-research/open_llama, 2023

    2023

  11. [20]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  12. [21]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  13. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  14. [23]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  15. [24]

    Unnatural instructions: Tuning language models with (almost) no human labor

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, 2023

    2023

  16. [25]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Associat...

    2023

  17. [26]

    Adversarial moment-matching distillation of large language models

    Chen Jia. Adversarial moment-matching distillation of large language models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  18. [27]

    Association for Computational Linguistics

  19. [28]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2016

    2016

  20. [29]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, 2020

    2020

  21. [30]

    Direct preference knowledge distillation for large language models

    Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, and Furu Wei. Direct preference knowledge distillation for large language models. arXiv preprint arXiv:2406.19774, 2024

    2024 arXiv

  22. [31]

    Distillm: Towards streamlined distillation for large language models

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  23. [32]

    Autoregressive knowledge distillation through imitation learning

    Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6121–6133, 2020

    2020

  24. [33]

    Openorca: An open dataset of gpt augmented flan reasoning traces, 2023

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and “Teknium”. Openorca: An open dataset of gpt augmented flan reasoning traces, 2023

    2023

  25. [34]

    TinyGSM: Achieving >80% on GSM8K with Small Language Models

    Bingbin Liu, Sébastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. TinyGSM: Achieving >80% on GSM8K with Small Language Models. arXiv preprint, abs/2312.09241, 2023. 11

    2023 arXiv

  26. [35]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, pages 74–81, 2004

    2004

  27. [36]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 35:27730–27744, 2022

    2022

  28. [37]

    Orca: Progressive learning from complex explanation traces of gpt-4

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Hassan Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023

    2023 arXiv

  29. [38]

    Linearly parameterized bandits

    Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395–411, 2010

    2010

  30. [39]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024

    2024 arXiv

  31. [40]

    Hybrid rl: Using both offline and online data can make rl efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  32. [41]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  33. [42]

    Patient knowledge distillation for bert model compression

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNL...

    2019

  34. [43]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, 33:3008–3021, 2020

    2020

  35. [44]

    Of moments and match- ing: A game-theoretic framework for closing the imitation gap

    Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and match- ing: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022–10032. PMLR, 2021

    2021

  36. [45]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Compu- tational Lin...

    2023

  37. [46]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  38. [47]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  39. [48]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  40. [49]

    Selective knowledge distillation for neural machine translation

    Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. Selective knowledge distillation for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr...

    2021

  41. [50]

    Self-instruct: Aligning language models with self-generated in- structions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistic...

    2023

  42. [51]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisti...

    2023

  43. [52]

    f-divergence minimization for sequence-level knowledge distillation

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10817–10834, 2023

    2023

  44. [53]

    Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Procee...

    2022

  45. [54]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  46. [55]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024

    2024

  47. [56]

    Provable offline preference-based reinforcement learning

    Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  48. [57]

    Online iterative reinforcement learning from human feedback with general preference model

    Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. Online iterative reinforcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems, 37:81773–81807, 2024

    2024

  49. [58]

    Plad: Preference-based large language model distillation with pseudo-preference pairs

    Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, and Chao Zhang. Plad: Preference-based large language model distillation with pseudo-preference pairs. In Findings of the Association for Computational Li...

    2024

  50. [59]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024

    2024 arXiv

  51. [60]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  52. [61]

    Mathematical analysis of machine learning algorithms

    Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023

    2023

  53. [62]

    Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023

    Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023

    2023

  54. [63]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 2299–2314, 2024

    2024

  55. [65]

    s Pr(o | x, τ0, τ1) Pr∗ (o | x, τ0, τ1) # ≤ −2 logE x∼d0 ,τ0 ∼π0 |x,τ1 ∼π1 |x o∼Pr∗ (o|x,τ0 ,τ1 )

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 13 Appendix Table of Contents A Supporting Lemmas . . . . . ....

    1909 arXiv

  56. [2023]

    arXiv preprint arXiv:2312.00754

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.