Pith. sign in

REVIEW 3 major objections 6 minor 44 references

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A DPO-style loss that also anchors the student to the teacher's output distribution better restores pruned and small chatbots' conversational ability.

desk verdict A clean but incremental DPO extension for white-box distillation; the math holds up, the empirics are suggestive but not yet rigorous. read the letter →

arxiv 2506.15717 v1 pith:WVS5VYPW submitted 2025-06-03 cs.LG cs.AIcs.CL

Zhengze Zhang , Shiqi Wang , Yiqun Shen , Simin Guo , Dahua Lin , Xiaoliang Wang , Nguyen Cam-Tu , Fei Tan This is my paper
classification cs.LGcs.AIcs.CL
keywords distribution-awaredirectpreferenceoptimizationknowledgedistillationwhite-boxconversationalabilityprunedlanguagemodelssmallKLdivergenceLLMalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Smaller and pruned language models lose conversational ability quickly, and the usual fix—distilling from a large teacher by turning teacher and student responses into winning and losing pairs for Direct Preference Optimization (dDPO)—uses only the teacher's sampled text. This paper proposes daDPO, a DPO-style loss that adds a second KL term anchoring the student to the teacher's full output distribution. The paper derives the corresponding optimal policy as a weighted geometric mixture of reference and teacher policy boosted by reward, and shows empirically that daDPO outperforms dDPO on MT-Bench, AlpacaEval, and in-domain win rates for pruned Vicuna models and Qwen2.5 small models. The central claim is that preference optimization and white-box distribution distillation can be unified in one objective rather than stacked as separate losses.

What carries the argument

The load-bearing object is the daDPO loss, a DPO loss with two log-ratio terms: $\beta_1$ times the usual student-vs-reference log ratio and $\beta_2$ times the student-vs-teacher log ratio, computed at the sentence level rather than per token. Its role in the argument is to make the teacher's output distribution a second KL anchor around the student policy; Theorem 1 gives the closed-form optimal policy as a geometric mean of reference and teacher weighted by $\beta_1$ and $\beta_2$, and the gradient shows the teacher log-ratio term $\beta_2\delta_{\mathrm{te}}$ up-weights updates when the teacher strongly prefers the winning response. Because the KL is sentence-level, the loss is usable even when teacher and student tokenizers differ. Relative to dDPO, it costs only teacher log-probabilities of the two responses.

What would settle it

Train the same student with a deliberately unreliable teacher—for example, a teacher whose greedy responses are correct but whose output distribution is flattened or flipped on the losing responses—and compare dDPO versus daDPO; if daDPO's win rate drops below dDPO's when only the teacher-distribution term is corrupted, the claim that the teacher distribution is a helpful anchor is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper discovers that replacing the reference policy with the teacher distribution alone (rDPO) is unstable, but keeping both anchors yields a loss that is both stable and more informative. Starting from the RL objective $\max \mathbb{E}[r(x,y)] - \beta_1 D_{\mathrm{KL}}[\pi_\theta \| \pi_{\mathrm{ref}}] - \beta_2 D_{\mathrm{KL}}[\pi_\theta \| \pi_{\mathrm{te}}]$, Theorem 1 shows the optimal policy is $\pi^*_\theta(y|x) \propto \pi_{\mathrm{ref}}(y|x)^{\beta_1/(\beta_1+\beta_2)} \, \pi_{\mathrm{te}}(y|x)^{\beta_2/(\beta_1+\beta_2)} \exp(r(x,y)/(\beta_1+\beta_2))$, a geometric interpolation between the student reference and the teacher, so the teacher distribution enters the preference model itself. The resulting daDPO loss is $-\mathbb{E}\,\log\sigma\left[\beta_1\!\left(\log\frac{\pi_\theta(y_t|x)}{\pi_{\mathrm{ref}}(y_t|x)}-\log\frac{\pi_\theta(y_s|x)}{\pi_{\mathrm{ref}}(y_s|x)}\right)+\beta_2\!\left(\log\frac{\pi_\theta(y_t|x)}{\pi_{\mathrm{te}}(y_t|x)}-\log\frac{\pi_\theta(y_s|x)}{\pi_{\mathrm{te}}(y_s|x)}\right)\right]$. Setting $\beta_2=0$ recovers DPO and $\beta_1=0$ recovers rDPO. Empirically, on Vicuna pruned models and the Qwen2.5 family, daDPO consistently beats dDPO: the 20% pruned Vicuna's in-domain preference rate improves from $-31\%$ to $-7.3\%$, and Qwen2.5-1.5B reaches $+14.0\%$ against its 7B teacher. The paper interprets this as distribution information teaching instruction following that sampled responses alone miss.

Load-bearing premise

The teacher model is assumed to be a reliable preference oracle and a good target distribution: its greedy responses are called winners and its output probabilities are treated as desirable, so a miscalibrated, biased, or too-distant teacher would feed bad signal into both KL anchors.

Editorial extensions

If this is right

  • A 20% pruned Vicuna-7B reaches an in-domain preference rate of $-7.3\%$, close to the unpruned teacher and much better than dDPO's $-31\%$; the 50% pruned model improves on all three reported benchmarks.
  • Qwen2.5-1.5B distilled from Qwen2.5-7B with daDPO reaches $+14.0\%$ in-domain win rate against the teacher, occasionally beating it, and Qwen2.5-0.5B's AlpacaEval score rises to $45.09\%$ from $42.27\%$ with dDPO.
  • The method distills across model families: with Qwen2.5-7B as teacher and Llama3.2-1B as student, daDPO beats dDPO while rDPO fails, because sentence-level log-probabilities do not require shared tokenizers.
  • The balance of anchors matters: increasing $\beta_2$ up to 1 improves over $\beta_2=0$, but an unduly large $\beta_2$ weakens the reference constraint and harms stability, matching the gradient interpretation.
  • The alignment tax stays small: on MMLU, ARC-C, GSM8K, and HellaSwag, daDPO roughly preserves or slightly improves the base student's scores relative to dDPO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the teacher term acts as a preference weighting, one could approximate daDPO in black-box settings by sampling multiple teacher responses and forming preference pairs, testing how much of the gain comes from distribution access versus extra ranking signal.
  • Beyond the paper: the positive $14\%$ win rate for the 1.5B student suggests the student can exploit rank information the teacher does not put into any single generation; a testable extension is to see whether the win rate grows with preference-pair coverage rather than with teacher size.
  • Beyond the paper: the geometric-mixture optimal policy implies daDPO implicitly defines a composite reference distribution proportional to $\pi_{\mathrm{ref}}^{\beta_1/(\beta_1+\beta_2)}\,\pi_{\mathrm{te}}^{\beta_2/(\beta_1+\beta_2)}$; checking whether this mixture is closer to the student's pre-training distribution could predict when distillation succeeds.
  • Beyond the paper: cross-family use assumes teacher log-probabilities computed on student text are informative across tokenizers; a direct test is to compare daDPO when teacher and student share a tokenizer versus when they do not, holding everything else fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces daDPO, a modification of DPO-based distillation that adds a second KL penalty anchoring the student to the teacher's output distribution, alongside the standard reference-model KL term. The authors derive the resulting RL objective's optimal policy (Theorem 1), convert it into a DPO-style loss (Eq. 10, Algorithm 1), and show that it reduces to DPO when β2=0 and to rDPO when β1=0. Empirically, they evaluate daDPO on pruned Vicuna-1.5-7B models and on Qwen2.5 1.5B/0.5B students distilled from Qwen2.5-7B, reporting improvements over dDPO on MT-Bench, AlpacaEval, and an in-domain GPT-4o judged win rate. They also include cross-family experiments (Qwen2.5-7B teacher to Llama3.2-1B student), ablation studies on β1/β2, an alignment-tax check, and a case study.

Significance. If the empirical claims hold, daDPO is a simple and practically useful extension of DPO that injects white-box teacher distribution information into preference-based distillation while preserving DPO's closed-form derivation and implementation simplicity. The theoretical derivation is standard but clean, and the degradation to DPO/rDPO at boundary hyperparameter values is a nice sanity check. The method's main novelty is modest—adding a second KL anchor—but it is internally coherent and could be adopted in resource-constrained LLM deployment. The significance is currently limited by the absence of statistical rigor, missing final hyperparameter values, and the paper's own cross-family results, which weaken the 'consistent superiority' claim.

major comments (3)
  1. [§5.2, Tables 1–4] The central empirical claim that daDPO 'consistently outperforms' dDPO is supported only by single runs with no significance tests, no error bars across seeds, and no released code. Several reported margins are small relative to the reported AlpacaEval standard deviations: e.g., Table 1 shows 54.04±1.76 versus 50.94±1.77 for the 20% pruned Vicuna model, and Table 4 shows 65.71±1.67 versus 65.34±1.67 for the cross-family Llama3.2-1B student. These differences could easily be within run-to-run noise. Please report at least three seeds with means and confidence intervals (or error bars), run a paired significance test for the key daDPO-versus-dDPO comparisons, and release code and configuration files to make the experiments reproducible.
  2. [§5.2, §5.3.5] The final values of β1 and β2 are never reported for any experiment, even though the method's behavior depends critically on the β1:β2 ratio. The ablation in Figure 5 shows that performance varies substantially with these hyperparameters, and the paper only states the ranges from which they were tuned. Without per-experiment final values, readers cannot reproduce the results or judge whether the reported improvements are artifacts of favorable tuning. Please include the selected β1 and β2 for every table and figure, or provide a tuning curve with the chosen operating point marked.
  3. [§5.3.4–5.3.6, Table 4, Figure 4] The paper's own evidence qualifies the 'consistently outperforms' claim. In the cross-family experiment (Table 4), the in-domain win rate is -36.0 for daDPO versus -36.3 for dDPO, which is effectively a tie, and Figure 4 shows that the daDPO advantage over dDPO shrinks when the teacher is the 1.5B model rather than 7B. This suggests the benefit is concentrated in same-family settings with a sufficiently capable teacher. Please either temper the unqualified claim in the abstract and conclusion to state this regime boundary, or provide additional evidence that daDPO also helps in the weak-teacher and cross-family regimes.
minor comments (6)
  1. [§3.2, Eq. (3)] The typesetting of the log-ratio terms is incorrect: 'β log πθ(yt|x) / log πref(yt|x)' should read 'β log(πθ(yt|x) / πref(yt|x))', and the same issue appears in Eq. (10) and in the gradient formula in §4.2.
  2. [Appendix A, Eq. (14)] The formula for r(x,y) has a misplaced division and should be written as (β1+β2)(log π*θ(y|x) − log(πref(y|x)^{β1/(β1+β2)} πte(y|x)^{β2/(β1+β2)})) + (β1+β2) log Z(x).
  3. [§5.1, Tables 1–4] The AlpacaEval column mixes the point estimate and standard deviation into one token (e.g., '50.94%1.77'); use '50.94 ± 1.77%' or '50.94% (±1.77)'.
  4. [§5.3.6] The heading 'Dose daDPO work' should be 'Does daDPO work'.
  5. [Appendix E, Tables 7–9] The column header 'Wine' appears in place of 'Win' in all three tables.
  6. [§2, References] Some references are incomplete or inconsistently formatted, e.g., 'Gu et al.' appears without a year in the introduction, and the reference list contains entries with missing venue information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: daDPO is derived from a two-KL RL objective and its empirical claims are benchmarked with standard hyperparameter tuning, not fitted to the evaluation metrics.

full rationale

The central derivation is self-contained: Eq. (7) defines an RL objective with a reward term and two KL penalties, Theorem 1 solves for the optimal policy (Eq. 8), and Eq. (10) follows by substituting the reward expression into the Bradley-Terry preference model. The loss reduces to DPO when beta2 = 0 and to rDPO when beta1 = 0, showing the construction is a nontrivial interpolation rather than a restatement of the inputs. The preferred and dispreferred labels come from teacher greedy samples versus student samples, and the teacher distribution enters only through the beta2 KL anchor; no parameter is fitted to MT-Bench, AlpacaEval, or in-domain win rates, and beta1/beta2 are standard hyperparameters. The author self-citations (e.g., Wang et al. 2024b, Li et al. 2023a, Zhang et al. 2024) appear in related-work, evaluation-prompt, and loss-formulation contexts and are not load-bearing for the theorem or the main empirical result. The paper also explicitly states limitations in Section 7, and its reliance on the teacher being a white-box model is an assumption, not a circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on β1 and β2 chosen by grid search and on standard DPO/RL assumptions. No new physical or model entities are introduced.

free parameters (2)
  • β1 = tuned over {0.01, 0.1, 1.0}, final per-setting values not reported
    Controls the KL weight toward the reference policy in Eq. 7; selected by performance, not derived.
  • β2 = tuned over {0.001, 0.01, 0.1, 1.0}, final per-setting values not reported
    Controls the KL weight toward the teacher distribution in Eq. 7; this is the key new term in daDPO.
assumptions (4)
  • domain assumption Teacher greedy responses are reliable preferred responses and the teacher is an optimal policy.
    Justifies constructing the preference dataset from teacher versus student in Section 3.2; daDPO also pulls the student toward the teacher distribution, amplifying reliance on this assumption.
  • standard math Bradley-Terry model relates the policy to preference probabilities.
    Used in the derivation from Eq. 9 to Eq. 10 and in Eq. 15, following Rafailov et al. 2023.
  • domain assumption Sentence-level MDP formulation is equivalent to token-level DPO.
    Assumed in Section 4.2 and cited to Rafailov et al. 2024 and Zeng et al. 2024, allowing comparison with token-level KL methods.
  • domain assumption Teacher log-probabilities of student-generated sentences are well-defined and informative even when tokenizers differ.
    The cross-family experiments in Table 4 rely on scoring student text with the teacher tokenizer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of daDPO: Distribution-Aware DPO for Distilling Conversational Abilities." pith.science (2026). https://pith.science/paper/WVS5VYPW

@misc{pith2026250615717,
  author       = {Pith},
  title        = {Pith review of: daDPO: Distribution-Aware DPO for Distilling Conversational Abilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WVS5VYPW}},
  note         = {Machine review of arXiv:2506.15717}
}
read the original abstract

Large language models (LLMs) have demonstrated exceptional performance across various applications, but their conversational abilities decline sharply as model size decreases, presenting a barrier to their deployment in resource-constrained environments. Knowledge distillation with Direct Preference Optimization (dDPO) has emerged as a promising approach to enhancing the conversational abilities of smaller models using a larger teacher model. However, current methods primarily focus on 'black-box' KD, which only uses the teacher's responses, overlooking the output distribution offered by the teacher. This paper addresses this gap by introducing daDPO (Distribution-Aware DPO), a unified method for preference optimization and distribution-based distillation. We provide rigorous theoretical analysis and empirical validation, showing that daDPO outperforms existing methods in restoring performance for pruned models and enhancing smaller LLM models. Notably, in in-domain evaluation, our method enables a 20% pruned Vicuna1.5-7B to achieve near-teacher performance (-7.3% preference rate compared to that of dDPO's -31%), and allows Qwen2.5-1.5B to occasionally outperform its 7B teacher model (14.0% win rate).

Figures

Figures reproduced from arXiv: 2506.15717 by the authors.

Figure 1
Figure 1. Conversation ability (AlpacaEval scores) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of in-domain performance of 100 test samples between DPO with different models (teacher v.s. student) as reference policy. Here x is different β values and y is in-domain score ence policy (i.e., using πte as πref ) can improve performance in some cases, discarding the refer￾ence policy constraint leads to instability, espe￾cially as the model difference increases (as in the heavily pruned case). This sug… view at source ↗
Figure 3
Figure 3. Baselines with distribution information on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Ablation study on the impact of β1 (for ref￾erence model KL) and β2(for teacher model KL) in daDPO. Here y-axis shows the AlpacaEval scores and x-axis shows the value of β. of the AlpacaEval scores for the 0.5B and 1.5B versions of Qwen2.5. The results from [PITH_FULL…
Figure 6
Figure 6. Figure 6: Evaluation prompt for GPT-4o-2024-08-06. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations

    work page 2024

  5. [5]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  6. [6]

    Claude. 2024. Conversation with claude 3.5 sonnet. Anthropic. Anthropic AI assistant, version 3.5 Sonnet, knowledge cutoff April 2024

    work page 2024

  7. [7]

    Daya Guo DeepSeek-AI, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  8. [8]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    arXiv 2024

Show all 44 references
  1. [9]

    CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models

    Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, and Fei Tan. CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    2024

  2. [10]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations

    2024

  3. [11]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  4. [12]

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. 2024. Aligner: Efficient alignment by learning to correct. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  5. [13]

    Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, and Jungwook Choi. 2024. Improving conversational abilities of quantized large language models via direct preference alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024

  6. [14]

    Dedong Li, Rui Zhao, and Fei Tan. 2023 a . Cwseg: An efficient and general approach to chinese word segmentation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 1--10

    2023

  7. [15]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 b . Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  8. [16]

    Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, and Furu Wei. 2024. Direct preference knowledge distillation for large language models. arXiv preprint arXiv:2406.19774

    2024 arXiv

  9. [17]

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. 2024. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580--606

    2024

  10. [18]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024 a . Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434

    2024 arXiv

  11. [19]

    Yixin Liu, Pengfei Liu, and Arman Cohan. 2024 b . Understanding reference policies in direct preference optimization. arXiv preprint arXiv:2407.13709

    2024 arXiv

  12. [20]

    Jinghui Lu, Dongsheng Zhu, Weidong Han, Rui Zhao, Brian Mac Namee, and Fei Tan. 2022. What makes pre-trained language models better zero-shot learners? Association for Computational Linguistics ACL 2023

    2022

  13. [21]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

    2023

  14. [22]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems

    2024

  15. [23]

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. In The Thirty-eighth Annual ...

    2024

  16. [24]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems

    2022

  17. [25]

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024. From r to q: Your language model is secretly a q-function. First Conference on Language Modeling

    2024

  18. [26]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2023

  19. [27]

    Sahithya Ravi, Patrick Huber, Akshat Shrivastava, Aditya Sagar, Ahmed Aly, Vered Shwartz, and Arash Einolghozati. 2024. Small but funny: A feedback-driven approach to humor distillation. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics

    2024

  20. [28]

    V Sanh. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of Thirty-third Conference on Neural Information Processing Systems (NIPS2019)

    2019

  21. [29]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  22. [30]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems

    2020

  23. [31]

    Hai-Long Sun, Zhun Sun, Houwen Peng, and Han-Jia Ye. 2025 a . Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In ACL

    2025

  24. [32]

    Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, et al. 2025 b . Parrot: Multilingual visual instruction tuning. In ICML

    2025

  25. [33]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  26. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  27. [35]

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. Conference on Language Modeling

    2023

  28. [36]

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024 a . Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations

    2024

  29. [37]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. In the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2023

  30. [38]

    Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, and Nguyen Cam-Tu. 2024 b . Reward difference optimization for sample reweighting in offline RLHF . In Findings of the Association for Computational Linguistics: EMNLP 2024

    2024

  31. [39]

    Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6268--6278

    2023

  32. [40]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  33. [41]

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. Proceedings of the 41st International Conference on Machine Learning

    2024

  34. [42]

    Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. In Findings of the Association for Computational Linguistics ACL 2024, pages...

    2024

  35. [43]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. Advances in Neural Information Processing Systems

    2023

  36. [44]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.