Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

Rethinking DPO: The Role of Rejected Responses in Preference Misalignment

T0 review · 3 major / 2 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct Preference Optimization is dominated by rejected responses; Bounded-DPO rebalances the loss so chosen responses win, and it tops IFEval and GSM8K.

desk verdict A clean one-line DPO fix with an honest but under-powered empirical section; the theory mostly restates the design, and the 'outperforms' claim needs error bars and a held-out lambda. read the letter →

arxiv 2506.12725 v1 pith:YHYIC63G submitted 2025-06-15 cs.AI cs.CL

Jay Hyeon Cho , JunHyeok Oh , Myunsoo Kim , Byung-Jun Lee This is my paper
classification cs.AIcs.CL
keywords DirectPreferenceOptimizationBounded-DPOmisalignmentrejectedresponsesreferencemodelalignmentlossgeometryinstructionfollowingmathematicalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct Preference Optimization (DPO) is meant to raise the probability of chosen responses and lower that of rejected ones, but the paper shows the loss can be driven to its minimum by suppressing only the rejected response, with the chosen response barely moving or even decreasing. The proposed fix, Bounded-DPO (BDPO), replaces the model's rejected-response probability inside the DPO loss with a mixture of that probability and the reference model's probability, so the rejected term can no longer dominate the gradient. In a single preference-pair setting, BDPO provably has minimizers with $\pi_\theta(y_w\mid x)=1$ and $\pi_\theta(y_l\mid x)=0$, and it keeps the chosen-response probability above $(1-\lambda)\pi_{\mathrm{ref}}(y_w\mid x)$ under monotone optimization. In experiments on instruction-following and math benchmarks at two model scales, BDPO outperforms DPO, DPOP, and DPO+NLL while adding no training-cost overhead. The paper's proposal is thus a low-cost loss-level correction that makes DPO actually serve its stated objectives.

What carries the argument

The load-bearing object is the mixture distribution $\pi_{\mathrm{mix}}(y\mid x)=\lambda\pi_\theta(y\mid x)+(1-\lambda)\pi_{\mathrm{ref}}(y\mid x)$, placed where DPO has $\pi_\theta(y_l\mid x)$ in the denominator of its log-ratio. Because $\pi_{\mathrm{mix}}(y_l\mid x)\ge(1-\lambda)\pi_{\mathrm{ref}}(y_l\mid x)>0$, the loss cannot be minimized by collapsing the rejected response to zero, and the gradient factor $\lambda/\pi_{\mathrm{mix}}(y_l\mid x)$ stays finite even when $\pi_\theta(y_l\mid x)\to0$, whereas DPO, DPO+NLL, and DPOP all retain the unbounded factor $1/\pi_\theta(y_l\mid x)$. The mixing weight $\lambda$ sets how much of the denominator follows the trained model versus the reference model, and the ablation finds $\lambda=0.5$ gives the best results while BDPO beats DPO at every tested value.

What would settle it

Train BDPO on a dataset with several preference pairs per prompt while tracking $\pi_\theta(y_w\mid x)$ and the in-distribution mass $\pi_\theta(y_w\mid x)+\pi_\theta(y_l\mid x)$; if, in a run with monotonically decreasing BDPO loss, any prompt's chosen-response probability falls below $(1-\lambda)\pi_{\mathrm{ref}}(y_w\mid x)$, or if the in-distribution mass drifts downward as with DPO, then the single-pair guarantees do not transfer to the multi-pair setting.

Watch

Extended reading notes

Core claim

The paper's central claim is that DPO's poor chosen-response behavior is a structural consequence of the term $1/\pi_\theta(y_l\mid x)$ in its gradient: as the rejected-response probability approaches zero this term drives rejected-dominated, unbounded updates, so the objective can be nearly satisfied without ever promoting $\pi_\theta(y_w\mid x)$. BDPO removes that failure mode by defining the loss $L_{\mathrm{BDPO}}=-\mathbb{E}_D\log\sigma\bigl(\beta\log\frac{\pi_\theta(y_w\mid x)}{\lambda\pi_\theta(y_l\mid x)+(1-\lambda)\pi_{\mathrm{ref}}(y_l\mid x)}-\hat{s}_{\mathrm{ref}}(w;l)\bigr)$, whose mixture denominator is bounded below by $(1-\lambda)\pi_{\mathrm{ref}}(y_l\mid x)$ and whose gradient with respect to the rejected probability is $\beta\sigma(-\Delta_{\mathrm{BDPO}})\cdot\lambda/\pi_{\mathrm{mix}}(y_l\mid x)$ rather than the unbounded $1/\pi_\theta(y_l\mid x)$. For one preference pair per prompt, the paper proves the BDPO optimum is $\pi_\theta(y_w\mid x)=1$ and $\pi_\theta(y_l\mid x)=0$, so every BDPO minimizer also minimizes DPO but not conversely, and it proves that monotone loss decrease implies $\pi_\theta(y_w\mid x)\ge(1-\lambda)\pi_{\mathrm{ref}}(y_w\mid x)$. Empirically, BDPO raises chosen log-probabilities, lowers rejected ones, keeps KL divergence to the reference model small, and records the best IFEval and GSM8K scores among the compared methods, with $\lambda=0.5$ recommended.

Load-bearing premise

The load-bearing premise is that the clean single-pair result—where assigning the chosen response probability 1 and the rejected response probability 0 is optimal—still controls real LLM training, which uses many preference pairs, finite batches, and optimization steps that do not steadily lower the loss at every step.

Editorial extensions

If this is right

  • DPO's failure to increase chosen-response probability is a property of its loss geometry, not a tuning artifact; any variant that keeps the unbounded $1/\pi_\theta(y_l\mid x)$ factor, including DPOP and DPO+NLL, inherits the same rejected-response dominance.
  • BDPO satisfies DPO's two stated objectives at once: the chosen-response probability has a reference-anchored lower bound under monotone optimization, and the rejected-response probability is driven to zero at the optimum.
  • Because BDPO changes only the loss and adds no parameters or sampling, it can replace the DPO loss in existing training pipelines with no additional wall-clock time.
  • The balanced optimization translates into higher task scores than DPO, DPOP, and DPO+NLL on both instruction-following (IFEval) and mathematical reasoning (GSM8K), at both tested model scales.
  • BDPO reduces negative log-likelihood on chosen responses without an explicit NLL term, suggesting its reference mixture supplies the chosen-response pressure that DPO+NLL adds by hand.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: in multi-pair settings, BDPO should behave like a KL-regularized reward maximizer, so its learned policy should resemble policies trained by reward-based RLHF more than DPO-trained policies do; comparing output distributions would test this.
  • The lower bound $(1-\lambda)\pi_{\mathrm{ref}}(y_w\mid x)$ can be used as a monitoring signal: in a run whose BDPO loss is monotonically decreasing, any measured chosen-response probability below that bound would indicate the single-pair theory has not transferred to the training distribution.
  • An annealed $\lambda$ schedule, starting near 1 for early exploration and decreasing toward 0.5 to anchor to the reference, could combine DPO's flexibility with BDPO's stability; the paper only tests fixed values.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper argues that DPO's loss is dominated by the rejected response, causing the probability of the chosen response to stagnate or fall. It proposes BDPO, which replaces πθ(yl|x) in the DPO denominator with λπθ(yl|x)+(1−λ)πref(yl|x), so that the rejected-response contribution cannot vanish and the gradient remains bounded. The paper provides a single-pair theoretical analysis (Theorems 1 and 2), loss visualizations, toy experiments, training-dynamics plots, and evaluations on IFEval and GSM8K with Qwen2.5-0.5B and 7B models, reporting that BDPO outperforms DPO, DPOP, DPO+NLL, and several other baselines.

Significance. BDPO is a simple, computationally free modification with a clear motivation: it directly addresses a known failure mode of DPO while preserving DPO's overall optimization structure. The gradient computation in Section 4.1 is transparent and correctly shows that the problematic 1/πθ(yl) factor is replaced by λ/πmix(yl), which is bounded below. The paper also ships code and includes an unusually explicit limitations section. However, the headline claim that BDPO 'outperforms existing algorithms' is not established by the current experiments, which are single runs with test-set hyperparameter selection and small margins. The theoretical guarantees are conditional on a single-pair setting and an unrealistic monotonicity assumption, and one ablation figure appears to contradict the associated text. With these issues addressed, the contribution would be a useful incremental advance.

major comments (3)
  1. [Section 5.3 / Appendix B] The paper states in Appendix B that all experiments were run once per configuration, and Figure 5 selects λ=0.5 on the same IFEval and GSM8K benchmarks that Tables 1-2 then report. In addition, Appendix B.3 says the final evaluation used the best-performing learning rate and epoch count for each algorithm. The reported advantages over the strongest baselines are small (0.58-0.60 IFEval points on Qwen2.5-0.5B, 0.75 points on Qwen2.5-7B, and 0.91 GSM8K points), and no variance estimates, repeated seeds, or paired significance tests are given. As written, the claim that BDPO outperforms existing algorithms is not statistically distinguishable from run-to-run noise and may reflect selection favoritism.
  2. [Section 5.4 / Figure 5] The text in Section 5.4 says BDPO 'consistently outperforms DPO across all tested λ values,' but the plotted values do not support this: on IFEval, BDPO at λ=0.1 is 24.7, below the DPO total score of 25.25 in Table 1; on GSM8K, the plotted y-range (28-30) excludes the DPO score of 27.45 in Table 2, making the comparison shown in the figure unclear. This internal inconsistency must be resolved before the robustness claim can be evaluated.
  3. [Section 4.1 / Appendix I.3] Theorem 2's lower bound (1−λ)πref(yw|x) ≤ πθ(yw|x) is proved under the assumption that every optimization step decreases the BDPO loss monotonically. Stochastic gradient descent does not satisfy this assumption; the loss can increase between steps. The main text's statement that 'BDPO provides a lower bound for the chosen probability' therefore overstates the guarantee. In addition, Theorems 1 and 2 are stated only for a single preference pair per prompt, while the experiments train on datasets with many pairs per prompt. The Limitations section acknowledges the single-pair scope, but the abstract and Section 4.1 should qualify the 'guarantees' accordingly.
minor comments (2)
  1. [Section 3.1] The sentence 'setting πθ(yl|x) = 0 leads to an unbounded loss' is inaccurate: as πθ(yl|x)→0, the DPO loss −log σ(rθ(yw)−rθ(yl)) tends to 0 because rθ(yl)→−∞. What is unbounded is the gradient with respect to πθ(yl), specifically the factor 1/πθ(yl). The later gradient analysis in Section 4.1 states this correctly, and the earlier sentence should be revised for consistency.
  2. [Section 4.2.1 / Appendix B.1] The main text says 'one pair of preference data is randomly selected, and the remaining two responses are implicitly treated as OOD,' while Appendix B.1 says 'two chosen and two rejected responses were selected at random' for each prompt. These descriptions are inconsistent and make the toy experiment irreproducible; please clarify whether one pair or two pairs per prompt were used.

Circularity Check

1 steps flagged · score 4.0 of 10

Empirical 'outperforming' claim is partially self-confirming because λ=0.5 and per-algorithm training settings are selected on the same IFEval/GSM8K benchmarks used for the reported tables; the theoretical derivation is independent, not circular.

  1. fitted input called prediction [Section 5.4 (Figure 5), Section 5.3 (Tables 1-2), Appendix B.3]
    "The final evaluation was conducted using the best-performing combination of learning rate and epoch count for each algorithm. ... BDPO (λ=0.5) achieves the highest total score, outperforming all baselines under both loose and strict accuracy criteria."

    λ=0.5 is not chosen a priori: Figure 5 selects it as the value yielding the best IFEval and GSM8K scores, and Section 5.3 then reports BDPO (λ=0.5) on those same benchmarks as the headline comparison. The reported 'outperforming' result is thus the maximum of the selection objective, not an independent prediction on data not used for selection. Appendix B.3's rule that each algorithm uses its best learning rate and epoch count on the final evaluation makes the cross-algorithm gap partly a test-set tuning artifact rather than a test of the loss design. With all experiments run once per configuration, the selected margin is also not statistically separated from noise.

full rationale

BDPO's derivation is not circular in the prohibited sense. The loss is defined by replacing π_θ(y_l) with π_mix; Theorem 1's optimum (π(y_w)=1, π(y_l)=0), Corollary 1, Theorem 2, and the bounded gradient 1/π_mix all follow from that definition and from the stated single-pair and monotonic-decrease assumptions. Those assumptions are restrictive (the paper itself limits the theory to a single preference pair per prompt), but they are not hidden versions of the conclusions; they are stated premises. There is no load-bearing self-citation chain and no imported uniqueness argument. The one genuine circular element is in the empirical evaluation: Figure 5 selects λ=0.5 because it maximizes IFEval and GSM8K scores, and Tables 1-2 then present BDPO (λ=0.5) on those same benchmarks as evidence that BDPO outperforms baselines. Appendix B.3 adds the same test-set selection loop for learning rate and epoch count per algorithm. The reported margins are therefore partly the value of the selection criterion, not an independent prediction on held-out benchmarks. I do not count Theorem 1 as circular: although it restates the design intent, the theorem is a mathematical consequence of the objective rather than an input to it.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three substantive assumptions: a single-pair setting for the theorems, a positive reference probability for the rejected response, and monotone loss decrease in Theorem 2. The only fitted hyperparameter is the mixture weight lambda, chosen by ablation on the evaluation benchmarks. No new entities are introduced.

free parameters (1)
  • lambda (mixture weight) = 0.5 (chosen via ablation on IFEval and GSM8K)
    Controls the balance between the learned and reference policies in the BDPO denominator; selected because it gave the best scores in Section 5.4, not derived from first principles.
assumptions (4)
  • domain assumption Single preference pair per prompt (one chosen and one rejected response per prompt).
    Used in Section 4.1 and Theorem 1; the paper acknowledges in the Limitations that multi-pair settings are left to future work.
  • domain assumption Reference probability of the rejected response is strictly positive, pi_ref(y_l|x) > 0.
    Required in Theorem 1; natural for language models but not verified exhaustively for all prompts in the experiments.
  • ad hoc to paper Every optimization step reduces the BDPO loss monotonically.
    Assumed in Theorem 2; not guaranteed by SGD, and no empirical verification is reported.
  • standard math Strict monotonicity of log and sigmoid functions.
    Used in the proof of Theorem 1 and Corollary 1 to convert loss minimization to ratio maximization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking DPO: The Role of Rejected Responses in Preference Misalignment." pith.science (2026). https://pith.science/paper/YHYIC63G

@misc{pith2026250612725,
  author       = {Pith},
  title        = {Pith review of: Rethinking DPO: The Role of Rejected Responses in Preference Misalignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHYIC63G}},
  note         = {Machine review of arXiv:2506.12725}
}
read the original abstract

Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.

Figures

Figures reproduced from arXiv: 2506.12725 by the authors.

Figure 1
Figure 1. Contour maps of the loss with reference model [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Contour maps for various values of α ∈ {0.01, 0.1, 1, 10}, focusing on the region of interest where πθ(yl |x) ∈ [0, 0.25]. The results show that, depending on the value of α, DPO+NLL may either fail to address the limitations of DPO (low α), or cause significant deviation from the reference model (high α). to significantly deviate from the reference model πref(yw|x) = 0.4 and πref(yl |x) = 0.1. This high￾lights the … view at source ↗
Figure 3
Figure 3. Training behavior of DPO, DPOP, DPO+NLL, and BDPO on a toy task with four prompts and four [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Learning dynamics of four algorithms—DPO, DPOP, DPO+NLL, and BDPO—are shown. All algorithms, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Ablation study on the choice of λ in BDPO. The left plot shows the IFEval total score on QWEN 0.5B; the right plot shows exact match accuracy on GSM8K. BDPO outperforms DPO across all λ values, with λ = 0.5 yielding the best overall results. performance using prompt-le…
Figure 6
Figure 6. Figure 6: Log of the total in-distribution probability [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Evolution of chosen and rejected probabilities across training steps. DPO and DPOP show only the [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Probability trends over training steps. DPO shows a simultaneous decrease in both chosen and rejected [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

    cs.AI 2026-07 accept novelty 6.0 of 10

    Offline DPO on expert states with one-step student negatives and schema-preserving augmentation improves LLM agents beyond SFT and can match online GRPO without environment interaction.

Reference graph

Works this paper leans on

36 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, and 1 others. 2024. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704

    arXiv 2024

  2. [2]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  3. [3]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  4. [4]

    Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. 2024 a . Preference learning algorithms do not learn preference rankings. arXiv preprint arXiv:2405.19534

    arXiv 2024

  5. [5]

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024 b . Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335

    arXiv 2024

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  7. [7]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, and 1 others. 2024. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning

    work page 2024

  8. [8]

    Michael Han Daniel Han and Unsloth team. 2023. http://github.com/unslothai/unsloth Unsloth

    2023

Show all 36 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

  3. [11]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  4. [12]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170--11189

    2024

  5. [13]

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. 2024. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. arXiv preprint arXiv:2406.09279

    2024 arXiv

  6. [14]

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. 2024. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436

    2024 arXiv

  7. [15]

    I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  8. [16]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

    2022

  9. [17]

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228

    2024 arXiv

  10. [18]

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733

    2024 arXiv

  11. [19]

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024 a . From r to q^ * : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358

    2024 arXiv

  12. [20]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024 b . Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  13. [21]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p...

    2020

  14. [22]

    Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. 2024. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.08847

    2024 arXiv

  15. [23]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

    2020

  16. [24]

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. 2024. Preference fine-tuning of llms should leverage suboptimal, on-policy data. In Forty-first International Conference on Machine Learning

    2024

  17. [25]

    Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, R \'e mi Munos, Bernardo \'A vila Pires, Michal Valko, Yong Cheng, and 1 others. 2024. Understanding the performance gap between online and offline alignment algorithms. arXiv preprin...

    2024 arXiv

  18. [26]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  19. [27]

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675

    2024 arXiv

  20. [28]

    Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu, and Yingfan Hu. 2024. Minor dpo reject penalty to increase training robustness. arXiv preprint arXiv:2408.09834

    2024 arXiv

  21. [29]

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024. Is dpo superior to ppo for llm alignment? a comprehensive study. In Forty-first International Conference on Machine Learning

    2024

  22. [30]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  23. [31]

    Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, and 1 others. 2025. Advancing llm reasoning generalists with preference trees. In The Thirteenth International Conference on Learning Representations

    2025

  24. [32]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning

    2024

  25. [33]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. CoRR

    2023

  26. [34]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.