Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single squared calibration term added to DPO keeps the chosen response's likelihood from falling during alignment and, according to the paper, makes the learned implicit rewards match ground-truth reward scale, yielding consistent gains…

desk verdict A simple anchor regularizer that helps in practice, wrapped in a broken calibration story; the theorem is false and the calibration claim doesn't match the implemented loss. read the letter →

arxiv 2412.14516 v1 pith:Y2FOB6OC submitted 2024-12-19 cs.LG cs.CL

Teng Xiao , Yige Yuan , Huaisheng Zhu , Mingxiao Li , Vasant G Honavar This is my paper · ORCID
classification cs.LGcs.CL
keywords directpreferenceoptimizationrewardcalibrationLLMalignmentcontrastivelearningimplicitmode-seekingbehaviorreverseKLdivergenceRLHF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cal-DPO claims that contrastive preference optimization methods such as DPO fail to fully align language models because their objectives depend only on the relative difference between the implicit rewards of the chosen and rejected responses, not on the absolute values of those rewards. During training, the likelihood of the chosen response can therefore fall even while the preference margin grows, which is especially harmful for reasoning and mathematics. The paper's fix is a squared calibration term, added to the standard pairwise preference loss, that anchors the implicit reward of the chosen response to $+1/(2\beta)$ and that of the rejected response to $-1/(2\beta)$; this is a one-line modification of DPO with no new hyperparameters. The paper argues that this makes the learned implicit rewards comparable in scale to ground-truth rewards, and proves that, when calibrated against the true reward, Cal-DPO minimizes an upper bound on the KL-regularized RLHF objective and inherits its mode-seeking behavior. Across standard benchmarks, the paper reports consistent gains over DPO and several of its variants.

What carries the argument

The central object is the implicit reward $\hat{r}_\theta(x,y)=\log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$, the log-likelihood ratio that DPO treats as a reward in place of a learned reward model. The machinery of Cal-DPO is the squared calibration term $(\hat{r}_\theta(x,y)-r(x,y)/\beta)^2$ added to the pairwise preference loss; in the implemented loss, $r(x,y_w)=1/2$ and $r(x,y_l)=-1/2$, so the chosen log-ratio is anchored at $1/(2\beta)$ and the rejected log-ratio at $-1/(2\beta)$. The anchor sets the level, not just the margin, of the implicit rewards, which is what prevents the chosen response's reward from sliding downward while the preference margin grows. The theoretical engine is a convex-analysis inequality relating forward and reverse KL divergences: the reverse-KL RLHF objective is bounded by the forward-KL term plus the squared calibration error, yielding the claimed mode-seeking, upper-bound guarantee.

What would settle it

With a dataset whose true reward is known (e.g., the IMDb sentiment log-odds used in the paper), train Cal-DPO and compare the learned chosen-response implicit reward to $r(x,y_w)/\beta$; the calibration claim predicts they match, so observing the implicit reward pinned near $1/(2\beta)$ while the true reward has a different scale, or finding the Theorem 2 inequality violated when evaluated with the true reward, would refute the central claim.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that the failure of contrastive preference methods is a scale-calibration failure, not a ranking failure. DPO-style losses keep increasing the margin between the chosen and rejected log-likelihood ratios, but nothing pins down the level of those ratios; the paper reports that DPO's chosen-response implicit reward drifts below zero during training while Cal-DPO's stays positive. Cal-DPO's objective is the DPO loss plus two squared anchors: $\log(\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x))$ pulled toward $1/(2\beta)$ and the rejected log-ratio pulled toward $-1/(2\beta)$. The paper defines calibration as equality with the true reward $r(x,y)/\beta$, proves that the first term of its population objective has a negative-gradient property that raises high-reward responses and lowers low-reward ones, and proves that minimizing Cal-DPO minimizes an upper bound on the reverse-KL RLHF objective, making it mode-seeking like RLHF. On preference datasets the anchors are implemented with $r=+1/2$ and $r=-1/2$, and the experiments report consistent improvements over DPO, IPO, SLiC, CPO, and DPO variants on reasoning, summarization, dialogue, and sentiment tasks.

Load-bearing premise

The load-bearing premise is that the fixed targets $+1/2$ and $-1/2$, used in the practical loss, stand in for the unknown true reward scale, because the paper's calibration and upper-bound promises are proved against the true reward $r(x,y)/\beta$, not against these fixed numbers.

Editorial extensions

If this is right

  • If the paper is right, a practitioner can add one line to a DPO training loop and stop the chosen-response likelihood from decreasing, at no additional hyperparameter cost.
  • The theoretical equivalence means that Cal-DPO, DPO, and RLHF converge to the same optimal policy in the large-data, large-capacity limit, and Cal-DPO does so through a reverse-KL upper bound.
  • The calibration recipe transfers to other pairwise losses; Cal-IPO and Cal-SLiC inherit the same anchor-based protection, with reported gains on the Anthropic-HH dataset.
  • The largest reported gains are in instruction following and mathematics, where the chosen response is often the single correct answer and preserving its likelihood matters most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to estimate the reward scale from the preference data rather than fixing the targets at $\pm 1/2$, since the practical targets are a stand-in for an unknown oracle scale.
  • Bridging the theory to the implemented fixed targets, for example by treating $\pm 1/2$ as a normalized reward scale, would sharpen the interpretation of the method as calibration rather than generic regularization.
  • The same anchor idea should transfer to on-policy iterative preference learning, which the paper names as a limitation, and in that setting it could prevent repeated training rounds from eroding the likelihood of good responses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Cal-DPO, a modification of direct preference optimization (DPO) that adds a squared-error calibration term pushing the implicit reward of the chosen response toward +1/(2β) and the implicit reward of the rejected response toward −1/(2β). The authors claim this calibrates the learned implicit rewards to the scale of ground-truth rewards, preserves a "negative gradient" property, and provably minimizes an upper bound on the KL-regularized RLHF objective (Theorem 2). The paper reports experiments on reasoning, summarization, dialogue, and controlled sentiment tasks, showing consistent improvements over DPO and several baselines, together with pseudocode and an open-source implementation.

Significance. If the theoretical claims were correct, the paper would offer a simple, one-line modification to DPO with substantial benchmark gains and a principled explanation of mode-seeking behavior. The empirical results are interesting and the method is easy to reproduce from the provided pseudocode. However, the paper's central theoretical contribution is not supported: the proof of Theorem 2 contains an invalid replacement of a sum over responses by an expectation under the reference policy, and a concrete counterexample falsifies the claimed inequality. In addition, the theory is stated for calibration to an oracle reward, while the implemented loss uses hand-set constants ±1/2, so the theoretical guarantees do not transfer to the algorithm actually evaluated. The empirical gains provide some value, but they do not compensate for the advertised theoretical advantages that are invalid as stated.

major comments (3)
  1. [Appendix A.4, Eq. (47)] The proof of Theorem 2 is invalid. The Bregman inequality in Eq. (46) yields a term (1/4) * Σ_y (s_y − r_y)^2, a sum over the response space, but the next step replaces this sum with E_{y∼πref}[(log πθ(y|x)/πref(y|x) − r(x,y)/β)^2], which is not an upper bound and can be much smaller. Concretely, take Y={1,2}, πref=(0.99,0.01), r(x,y)=0, πθ=(0.5,0.5), and β=1. Then p=π*=πref, q=πθ, and the claimed inequality in Eq. (47) reads D_KL(q||p)=1.614 ≤ D_KL(p||q)+E[...] = 0.637+0.614 = 1.251, which is false. Consequently, the advertised upper bound L_RL(θ) ≤ β L_CalDPO(θ) − β log Z(x) in Eq. (15) fails even under the oracle-reward setting, independent of the ±1/(2β) target substitution.
  2. [Section 4.1, Eq. (10) and Appendix A.2] The theoretical results in Section 4.2 (Definition 1, Theorems 1 and 2) are stated for calibration to the oracle reward r(x,y)/β, but the implemented loss in Eq. (10) uses r(x,y_w)=1/2 and r(x,y_l)=−1/2. The paper provides no evidence that these constants correspond to the reward scale of any dataset, so the central claim of calibrating to ground-truth rewards is not supported for the actual objective. The derivation in Appendix A.2 does not bridge this gap: it attempts a limit β→0 to obtain hard softmax weights, but under that limit the calibration targets 1/(2β) diverge, and for the finite β used in the experiments (e.g., 0.001) the two losses are not equivalent.
  3. [Section 4.2, Theorem 2] The theorem's claim that Cal-DPO "encourage[s] mode-seeking behavior by minimizing an upper bound of the reverse KL divergence" rests entirely on the invalid inequality in Eq. (47). Because that inequality fails, the paper does not establish that Cal-DPO minimizes an upper bound of the RLHF objective, nor that it has the claimed mode-seeking property. This is the central theoretical contribution of the paper, and the error cannot be repaired by local editing of the proof; the theorem as stated is false.
minor comments (5)
  1. [Section 4.2] The sentence "All proofs are provided in the Appendix 4.2" should refer to "Appendix A" rather than "Appendix 4.2".
  2. [Equation (13)] The first term in Eq. (13) is written with a denominator that depends on the summation variable y; the simplification used in the proof of Theorem 1 shows the denominator equals 1, but the notation in Eq. (13) is misleading and should be clarified.
  3. [Section 5 and NeurIPS Checklist item 7] The checklist claims that the experiments are accompanied by statistical significance information, but no error bars, confidence intervals, or variance estimates are reported in Tables 2–4 or Figure 2; this checklist answer is not supported by the manuscript.
  4. [Appendix A.2] The statement "setting β → 0 (β is typically small and set as 0.001)" is internally inconsistent: with β=0.001, the targets ±1/(2β) equal ±500, which is not a small-β limit of the squared-loss calibration term used in Eq. (10).
  5. [Title page] The header identifies the paper as appearing at "38th Conference on Neural Information Processing Systems (NeurIPS 2024)", but the arXiv submission is dated December 2024; if this is a journal submission, the venue line should be removed or updated.

Circularity Check

1 steps flagged · score 4.0 of 10

Calibration to the ground-truth scale is imposed by definition: Eq. (10) targets hand-set ±1/(2β) rewards, so the central calibration claim reduces to the chosen normalization rather than a derived or measured property.

  1. self definitional [Section 4.1, Definition 1 and Eq. (10); Abstract.]
    "An estimated implicit reward log πθ(y|x)/πref(y|x) for the LM policy πθ is called scale calibrated with respect to the ground truth reward if log πθ(y|x)/πref(y|x) = r(x,y)/β, ∀(x,y) ∼ D. ... In such cases, we define the reward for preference feedback as follows: r(x, yw) = 1/2 and r(x, yl) = −1/2 ... LCal−DPO ... = ... + (log πθ(yw|x)/πref(yw|x) − 1/2β)^2 + (log πθ(yl|x)/πref(yl|x) + 1/2β)^2"

    The paper's formal notion of calibration (Definition 1) is equality between the implicit reward and the oracle reward r/β. When the oracle is unavailable, the paper defines the reward to be ±1/2, and Eq. (10) then minimizes the squared distance of log(πθ/πref) to ±1/(2β). Consequently, the implemented loss enforces Definition 1 for the surrogate reward selected by the authors; it does not measure or verify the scale of any external ground-truth reward. The abstract's claim that Cal-DPO makes implicit rewards comparable in scale to ground-truth rewards therefore follows from the chosen normalization by construction, not from a derivation or an independent calibration check. The benchmark improvements are real empirical evidence, but the conceptual calibration step is self-definitional.

full rationale

No load-bearing self-citation or imported uniqueness theorem appears in the derivation chain; Theorems 1 and 2 are argued from the paper's own objective rather than from a self-citation. The one definitional circularity is the surrogate-reward substitution: the practical loss in Eq. (10) targets hand-set constants ±1/(2β), and Definition 1 calls equality to r/β calibration, so the claim of calibration to ground-truth scale is an imposed identity rather than a prediction. This warrants a moderate score rather than a high one because the method's empirical results are extensive and independent of the definitional label. Separately, the proof of Theorem 2 appears to replace a sum over responses with an expectation under πref, and the claimed inequality DKL(πθ||π*) ≤ DKL(π*||πθ) + E_{πref}[(log(πθ/πref) − r/β)^2] fails on a two-outcome counterexample; that is a flagged correctness defect, not circularity, and does not contribute to the circularity score. Similarly, the first term of Eq. (13) collapses to the standard MLE because its denominator is Σ_y πθ(y) = 1, undermining the claimed negative-gradient distinction as a proof-validity issue rather than a circular one.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central quantity the method claims to calibrate is the ground-truth reward, but the only hand-set numbers are the plus or minus one-half anchors and beta; no oracle reward is used in the practical loss, so the ledger is dominated by these chosen parameters and the ad hoc substitution.

free parameters (2)
  • beta (KL coefficient) = searched over [1e-3, 2e-3, 3e-3, 1e-2, 1e-1]
    Controls the DPO loss strength and the calibration target scale 1/(2*beta); chosen per dataset via validation search in Appendix B.1.
  • target reward values r(x,y_w)=1/2, r(x,y_l)=-1/2 = fixed by hand: +0.5, -0.5
    Substituted for the unknown ground-truth reward in Eq. (10); this sets the calibration anchor and is not derived from actual reward data (Section 4.1).
assumptions (5)
  • domain assumption Preferences follow the Bradley-Terry model with a latent reward r(x,y)
    Assumed in Section 3 Eq. (1) and used throughout the derivation of Eq. (10).
  • ad hoc to paper The oracle reward for pairwise preference feedback can be represented by r(x,y_w)=1/2 and r(x,y_l)=-1/2
    Introduced in Section 4.1 to make the calibration loss computable without an oracle; the theoretical upper bound in Theorem 2 is proven for the oracle-reward population loss, not for this substituted target.
  • domain assumption beta approaches zero and the softmax reward weights collapse to a hard argmax
    Used in Appendix A.2 to identify Eq. (10) with an empirical estimate of the population loss; the text states beta is typically small and set as 0.001.
  • standard math Bregman divergence inequality D_KL(q||p) <= D_KL(p||q) + (1/4)||s-r||^2 for the log-sum-exp potential
    Invoked in the proof of Theorem 2 (Appendix A.4) to upper-bound reverse KL; the proof is sketched rather than fully justified.
  • domain assumption With sufficiently large data and model capacity, DPO, RLHF, and Cal-DPO converge to the same optimal policy
    Stated after Theorem 2 without proof; not needed for the main claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment." pith.science (2026). https://pith.science/paper/Y2FOB6OC

@misc{pith2026241214516,
  author       = {Pith},
  title        = {Pith review of: Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2FOB6OC}},
  note         = {Machine review of arXiv:2412.14516}
}
read the original abstract

We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.

Figures

Figures reproduced from arXiv: 2412.14516 by the authors.

Figure 1
Figure 1. The implicit reward dynamics during training of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. (Left two) The training dynamics of DPO and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 2
Figure 2. AlpacaEval 2.0 evaluation results of models trained with UltraFeedback Bi￾narized dataset. The DPO and Cal-DPO are both initialized from the SFT model zephyr-7b-sft-full. We conducted experiments on the IMDB dataset to assess the generation of positive movie reviews. The task requires the model to provide positive and fluent completions of movie reviews based on given partial input texts. To per￾form a controlled ev… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The effect of the coefficient parameter β on four tasks. For Reddit TL;DR summarization and Anthropic-HH, we show the win rate over the chosen response. Big Bench Hard (BBH): BBH is a selection of 23 challenging tasks from the BigBench, focusing on areas like multistep…
Figure 5
Figure 5. Figure 5: The performance of SLiC and IPO, and their calibrated counterparts Cal-IPO and Cal-SLiC by applying [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The training dynamics of DPO and Cal-DPO on the Anthropic-HH and IMDb datasets. D.1 Case Study Warning: The following case studies contain example data that may be offensive or harmful. In this section, we present examples of comparisons between Cal-DPO and the baselin…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DPO-Shift: Shifting the Distribution of Direct Preference Optimization

    cs.CL 2025-02 conditional novelty 4.0 of 10

    DPO-Shift multiplies the rejected-response term in the DPO loss by a factor f(λ) below 1, shifting chosen-response probabilities upward at the expense of some reward margin.

Reference graph

Works this paper leans on

87 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  2. [2]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, pages 27730–27744, 2022

    2022

  3. [3]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  4. [4]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  5. [5]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    arXiv 2017

  6. [6]

    Implementation matters in deep policy gradients: A case study on ppo and trpo

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. In International Conference on Learning Representations, 2020

    2020

  7. [7]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024

    2024

Show all 87 references
  1. [9]

    Slic- hf: Sequence likelihood calibration with human feedback

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  2. [10]

    Preference fine-tuning of llms should leverage suboptimal, on-policy data

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024. 10

    2024 arXiv

  3. [11]

    Smaug: Fixing failure modes of preference optimisation with dpo-positive

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024

    2024 arXiv

  4. [12]

    Advancing llm reasoning generalists with preference trees

    Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024

    2024 arXiv

  5. [13]

    Learning word vectors for sentiment analysis

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150, 2011

    2011

  6. [14]

    Tl; dr: Mining reddit to learn automatic summarization

    Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, 2017

    2017

  7. [15]

    A framework for few-shot language model evaluation, 12 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023

  8. [16]

    Nash learning from human feedback

    Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886, 2023

    2023 arXiv

  9. [17]

    Statistical rejection sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2023

    2023

  10. [18]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024

    2024 arXiv

  11. [19]

    Gibbs sam- pling from human feedback: A provable kl-constrained framework for rlhf

    Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam- pling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456, 2023

    2023 arXiv

  12. [20]

    Direct nash optimization: Teaching language models to self-improve with general preferences

    Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024

    2024 arXiv

  13. [21]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  14. [22]

    Simper: Simple preference fine-tuning without hyperparameters by perplexity optimization

    Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: Simple preference fine-tuning without hyperparameters by perplexity optimization. arXiv, 2024

    2024

  15. [23]

    Policy optimization in rlhf: The impact of out-of-preference data

    Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584, 2023

    2023 arXiv

  16. [24]

    Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024

    2024 arXiv

  17. [25]

    Provably robust dpo: Aligning language models with noisy feedback

    Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024

    2024 arXiv

  18. [26]

    Noise contrastive alignment of language models with explicit rewards

    Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024. 11

    2024 arXiv

  19. [27]

    Copr: Continual human preference learning via optimal policy regularization

    Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, et al. Copr: Continual human preference learning via optimal policy regularization. arXiv preprint arXiv:2402.14228, 2024

    2024 arXiv

  20. [28]

    Towards efficient and exact optimization of language model alignment

    Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00856, 2024

    2024 arXiv

  21. [29]

    Noise contrastive estimation and negative sampling for condi- tional models: Consistency and statistical efficiency

    Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for condi- tional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3698–3707, 2018

    2018

  22. [30]

    Towards ana- lyzing and understanding the limitations of dpo: A theoretical perspective

    Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards ana- lyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024

    2024 arXiv

  23. [31]

    From r to q*: Your language model is secretly a q-function

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024

    2024 arXiv

  24. [32]

    Dpo meets ppo: Reinforced token optimization for rlhf

    Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922, 2024

    2024 arXiv

  25. [33]

    Token- level direct preference optimization

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. arXiv preprint arXiv:2404.11999, 2024

    2024 arXiv

  26. [34]

    A general offline reinforcement learning framework for interac- tive recommendation

    Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interac- tive recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4512–4520, 2021

    2021

  27. [35]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330, 2017

    2017

  28. [36]

    Scale calibration of deep ranking models

    Le Yan, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. Scale calibration of deep ranking models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4300–4309, 2022

    2022

  29. [37]

    Calibrated model-based deep reinforcement learning

    Ali Malik, V olodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Ermon. Calibrated model-based deep reinforcement learning. In International Conference on Machine Learning, pages 4314–4323. PMLR, 2019

    2019

  30. [38]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning

    Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  31. [39]

    On the calibration of large language models and alignment

    Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the calibration of large language models and alignment. arXiv preprint arXiv:2311.13240, 2023

    2023 arXiv

  32. [40]

    Language models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022

    2022 arXiv

  33. [41]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pages 324–345, 1952

    1952

  34. [42]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024

    2024 arXiv

  35. [43]

    How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective

    Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. arXiv preprint arXiv:2410.10093, 2024. 12

    arXiv

  36. [44]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  37. [45]

    Reinforced self-training (rest) for language modeling

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  38. [46]

    Openchat: Advancing open-source language models with mixed-quality data

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations, 2023

    2023

  39. [47]

    Fine-tuning language models with advantage-induced policy alignment

    Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231, 2023

    2023 arXiv

  40. [48]

    Machine learning: a probabilistic perspective

    Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012

    2012

  41. [49]

    Improving policy gradient by exploring under-appreciated rewards

    Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv:1611.09321, 2016

    2016 arXiv

  42. [50]

    Learning how to propagate messages in graph neural networks

    Teng Xiao, Zhengyu Chen, Donglin Wang, and Suhang Wang. Learning how to propagate messages in graph neural networks. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1894–1903, 2021

    1903

  43. [51]

    Learning to generalize from sparse and underspecified rewards

    Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In International conference on machine learning, pages 130–140. PMLR, 2019

    2019

  44. [52]

    Decoupled self-supervised learning for graphs

    Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self-supervised learning for graphs. Advances in Neural Information Processing Systems, pages 620–634, 2022

    2022

  45. [53]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023

    2023 arXiv

  46. [54]

    Zephyr: Direct distillation of lm alignment

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    arXiv

  47. [55]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  48. [56]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  49. [57]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  50. [58]

    Challenging big- bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  51. [59]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  52. [60]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 13

    arXiv

  53. [61]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  54. [62]

    Alpacaeval: An automatic evaluator of instruction-following models, 2023

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  55. [63]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Intern...

    2023

  56. [64]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, page 9, 2019

    2019

  57. [65]

    Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417, 2024

    2024 arXiv

  58. [66]

    Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2023

    2023

  59. [67]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024

    2024 arXiv

  60. [68]

    Information, divergence and risk for binary experiments

    Mark Reid, Robert Williamson, et al. Information, divergence and risk for binary experiments. JMLR, 2011

    2011

  61. [69]

    Reward augmented maximum likelihood for neural structured prediction

    Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuur- mans, et al. Reward augmented maximum likelihood for neural structured prediction. Advances In Neural Information Processing Systems, 29, 2016

    2016

  62. [70]

    Clustering with bregman divergences

    Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with bregman divergences. Journal of machine learning research, 6(10), 2005

    2005

  63. [71]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024

    2024 arXiv

  64. [72]

    chosen” response and one of the remaining three at random as the “rejected

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023. 14 A Derivations and Proofs A.1 Deriva...

    arXiv

  65. [73]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Our main claim matches our theoretical and experimental results in Section 4.2 and Section 5. Guidelines: • The answer...

  66. [74]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Please see Section 6 for the discussion of limitations. Guidelines: • The answer NA means that the paper has no limitation while the answer No means ...

  67. [75]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 27 Justification: Please refer to Section 4.2 and Appendix A for our assumptions and a complete (and correc...

  68. [76]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  69. [77]

    • Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details

    Open access to data and code 28 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code of Cal-DPO is a...

  70. [78]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide all the training a...

  71. [79]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We provide the experiment results that support the...

  72. [80]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide the computer...

  73. [81]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We make sure to preserve anonymity and conform NeurIPS Code of Ethics. Guidelin...

  74. [82]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Please see Appendix E for broader impacts. Guidelines: • The answer NA means that there is no societal im...

  75. [83]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses...

  76. [84]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: The paper does no...

  77. [85]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new assets. ...

  78. [86]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  79. [87]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.