Pith. sign in

REVIEW 3 major objections 5 minor 71 references

DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DPO's human-choice component and analytical loss are independent: any monotone loss can be paired with any monotone choice function, so convexity is not required.

desk verdict A genuinely useful normative framework for DPO, but the main theorem is false as stated and needs repair before the headline claims can stand. read the letter →

arxiv 2507.07855 v4 pith:O64Z5DXF submitted 2025-07-10 cs.LG cs.AIcs.CL

Wenxuan Zhou , Shujian Zhang , Brice Magdalou , John Lambert , Ehsan Amid , Richard Nock , Andrew Hard This is my paper
classification cs.LGcs.AIcs.CL
keywords DirectPreferenceOptimizationhumanchoicetheoryproperlossesBregmandivergenceKLST*structurenon-convexRLHF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that DPO's link to the Bradley-Terry-Luce model of human choice is a contingency, not a requirement. Its main theorem proves that any strictly increasing loss can be paired with any strictly increasing choice function through a suitable strictly proper loss, so the human-choice component of DPO is disentangled from its analytical loss. That makes convexity of the training loss optional and places DPO and many of its follow-ups as particular points in one normative design space. A reader should care because the result converts a presumed constraint into design freedom: new losses, including non-convex ones, can be used without re-justifying a choice model.

What carries the argument

The load-bearing structure is the KLST* choice model, a generalization of Bradley-Terry-Luce in which pairwise choice probabilities are expandable to binary lotteries, satisfy a local choice structure (bearability, the two zero-abstention/preference axioms) and a monotonicity condition, and allow abstention. Theorem 4.2 converts a KLST* structure into the form $p(y\succ y'\mid x)=F(u(x,y)-u(x,y'))$ with $F(z)+F(-z)\le 1$. Theorem 4.1 converts the generalized RLHF objective into a Bregman divergence, and Theorem 4.3 factors any monotone $\psi$ through any monotone $\tilde{F}$ by constructing a strictly proper loss, so the decomposition $\psi=\tilde{\ell}_0\circ\tilde{F}$ is the central identity that carries the disentanglement.

What would settle it

A direct test of the monotonicity axiom would settle the central claim: in a fixed context $x$, collect human pairwise choices without abstention and look for alternatives $A,B,C,D,E,F$ with $p(A\succ B\mid x)\ge p(D\succ E\mid x)$ and $p(B\succ C\mid x)\ge p(E\succ F\mid x)$ but $p(A\succ C\mid x)<p(D\succ F\mid x)$. A single observed violation of that inequality would falsify the KLST* representation and, with it, the universality of the pairing result.

Watch

Extended reading notes

Core claim

On the paper's own terms, DPO is not the unique normative solution but one canonical, symmetric, separable point in a large product space. The central theorem states that for any strictly increasing function $\psi:\mathbb{R}\to\mathbb{R}$ and any strictly increasing function $\tilde{F}:\mathbb{R}\to[0,1]$, there exists a strictly proper binary loss $(\tilde{\ell}_0,\tilde{\ell}_1)$ such that $\psi(z)=\tilde{\ell}_0\circ\tilde{F}(z)$, and the proof gives the loss explicitly. Combined with the KLST* representation $p(y\succ y'\mid x)=F(u(x,y)-u(x,y'))$, this means the human choice model and the training loss are disentangled: any compliant choice model works with any monotone analytical loss. DPO itself corresponds to the special canonical, symmetric, separable case with $F=\sigma$, the log-loss, and the KL regularizer.

Load-bearing premise

The framework rests on human pairwise choice probabilities satisfying the KLST* axioms — expandability, bearability, the two zero-abstention axioms, and especially monotonicity, a difference-transitivity condition that real preferences often violate — and if those axioms fail for actual evaluators, the representation $p(y\succ y'\mid x)=F(u(x,y)-u(x,y'))$ and the universal pairing result collapse.

Editorial extensions

If this is right

  • DPO-family algorithms do not need to justify their loss by a specific human choice model: any monotone $\psi$ can be paired with any KLST*-compliant $\tilde{F}$.
  • Convexity of the final loss is optional, and the RLHF KL term can be replaced by any Bregman divergence of a proper loss, so design can trade convexity for properties such as Lipschitzness.
  • Margins and home advantages fit through affine transforms of proper losses, and length normalization fits through Bregman-divergence means, so SimPO-style slack and length corrections are in-family.
  • Departing from a separable proper loss while keeping properness with more than two alternatives forces the KL divergence, so proper non-KL designs must use a non-separable construction.
  • DPO is identified as the special canonical, symmetric, separable point, and ORPO, IPO, RRHF and AlphaPO appear as canonical-connection instances in the paper's PPPO framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The disentanglement suggests transferring known proper-loss results from classification to preference optimization: any monotone composite loss has a corresponding DPO-style algorithm once a choice model is fixed.
  • The KLST* axioms are the empirical weak point; measuring abstention and difference-transitivity directly on preference data would reveal how wide the 'any human choice model' claim really is.
  • The constructive proof gives an explicit recipe for new algorithms — choose a monotone choice function and a monotone loss, then build the proper loss — so benchmarking non-convex Lipschitz losses against DPO is a direct testable extension.
  • Because abstention enters as local incomparability rather than a global outside option, the framework suggests preference-elicitation experiments where evaluators may decline to choose, a direction the paper signals as future impact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a normative generalization of Direct Preference Optimization (DPO). It replaces the Bradley-Terry-Luce model with a choice-theoretic framework called KLST*, built on expandability, a local choice structure, monotonicity, and a new abstention dimension, and it generalizes DPO's three analytical components: the RLHF regularizer, the reward-to-choice link, and the final scalar loss. The central results are Theorem 4.2, giving a utility-difference representation for KLST* choice probabilities; Theorem 4.3, claiming that any strictly increasing loss function ψ can be written as ψ = ℓ0∘F~ for a strictly proper loss (ℓ0,ℓ1) and any strictly increasing F~:R→[0,1]; and Theorem 4.4, connecting proper losses to Bregman divergences and convex conjugates. On this basis the paper argues that the human choice model is disentangled from the analytical loss, that convexity of the loss is dispensable, and that a broad family of DPO variants (SimPO, ORPO, IPO, RRHF, AlphaPO, etc.) fall under a PPPO umbrella. A toy experiment with a non-convex surrogate loss on Gemma-2-2B is reported.

Significance. If the central embedding theorem were correct, this would be a substantial unification: it would show that DPO's connection to human choice theory is one point in a much larger product space of reward functions, proper losses, and choice models, and it would open a design space of non-convex losses for preference optimization. The paper has genuine strengths: the proofs of Theorems 4.1, 4.2, and 4.4 are largely constructive and checkable; the taxonomy in Table 1 is informative; Lemma 5.2 and Lemma 5.4 are useful technical tools; and the toy experiment, though explicitly limited, provides a concrete non-convex loss that is far from the standard DPO choices. However, the significance is conditional, because Theorem 4.3 is false as stated under the paper's own definition of a real-valued proper loss, and the 'any' claims in the abstract and Section 5.1 are therefore unsupported. The paper would need a corrected theorem statement with explicit boundary and integrability hypotheses, together with a re-scoping of the main claims, before its contribution can be assessed.

major comments (3)
  1. [Theorem 4.3 / Appendix II.3] Theorem 4.3 is false as stated. The proof invokes the inverse F~^{-1}:[0,1]→R of a strictly increasing function F~:R→[0,1], but no strictly increasing function on R can attain its infimum or supremum, so Im(F~) is a proper open interval and F~^{-1} is not defined on all of [0,1]. More decisively, take ψ(z)=z and F~(z)=σ(z). The identity ψ=ℓ~0∘F~ forces ℓ~0(p)=logit(p) on (0,1). Since logit is unbounded below as p→0+, Definition 3.1's p-properness condition at p=0 cannot be satisfied by any real-valued ℓ~0(0), so no strictly proper loss in the paper's sense exists for this pair. The same example also shows that Lemma D's construction is not valid for arbitrary strictly increasing ℓ:[0,1]→R, because the integral of ℓ(t)/t² near 0 may diverge and the resulting loss may not be real-valued on the closed simplex. The theorem requires additional hypotheses (for example, matching finite limits of ψ and F~ at the boundaries, or an explicit extended-real-valued convention for losses), and the 'any' claims in the abstract and Section 5.1 must be re-scoped accordingly.
  2. [Section 3.2, Definitions 3.2-3.4] The framework's reach depends entirely on the KLST* axioms, but these axioms are introduced as new and are supported only by illustrative narratives (the Seoul indoor-activity example and the ICML keynote-speaker example), not by data or by citations to empirical choice studies. Monotonicity in Definition 3.3 is a difference-transitivity condition of the form: if p(L1≻L2)≥p(L4≻L5) and p(L2≻L3)≥p(L5≻L6), then p(L1≻L3)≥p(L4≻L6). Conditions of this kind are known to be violated in real preference data. Because Theorem 4.2 and the 'human choice model vanishes' claim in Section 5.1 apply only to KLST*-compliant choice probabilities, the paper should either provide empirical evidence for the axioms or explicitly frame them as a normative assumption and propose a concrete falsification test on paired-comparison data. Without that, the scope of the central claim is unknown.
  3. [Section 4.2, Eq. (13)-(14) and Section 4.3, Eq. (17)-(19)] The derivation moves from the KKT condition (13), which holds at the optimum of the RLHF objective (9), to the training loss (17)-(19), which is evaluated at the current, generally non-optimal policy πθ during training. This is the same type of step used in DPO, where the policy is optimal for its own implicit reward, but the paper does not state the fixed-point justification. As written, (19) appears to assume that (13) remains valid at non-optimal πθ. The authors should add an explicit paragraph explaining that the implicit reward is defined so that πθ is optimal for that reward, or restrict the normative claim to the fixed point of the loss minimization.
minor comments (5)
  1. [Abstract and Section 1] The abstract refers to 'social choice theory's textbook path' while the body consistently discusses 'human choice theory'; the terminology should be unified.
  2. [Definition 3.1 and Theorem 4.3] If the authors intend to allow extended-real-valued losses at the boundary of the simplex, Definition 3.1 should say so explicitly, and all finiteness statements in Theorem C, Lemma D, and Theorem 4.3 should be revised accordingly.
  3. [Section 4.3, Eq. (18) and Corollary 4.5] Equation (18) defines ℓ(p)=ℓ~0(1-p), but Corollary 4.5 says 'let ℓ:=ℓ~0 in (15)'; the notation is inconsistent and should be harmonized.
  4. [Appendix II.3, Lemma D] The integral ∫_p^a ℓ(t)/t² dt is not guaranteed to converge for arbitrary strictly increasing ℓ:[0,1]→R, and the case a=0 needs a convention for the orientation of the integral; the lemma should state integrability assumptions on ℓ.
  5. [Section 6, Eq. (26) and Figure 1] The loss ψ_a defined in (26) is strictly decreasing in z, whereas Theorem 4.3 concerns strictly increasing ψ. The text should state explicitly that ψ_a is the final surrogate applied to the negative margin, i.e., the relevant increasing function is z↦ψ_a(-z), to avoid a direct contradiction with Theorem 4.3.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 4.3's 'disentanglement' reduces to defining ℓ̃0 = ψ∘F̃^{-1}; the central universality claim is an identity by construction.

  1. self definitional [Theorem 4.3 (Section 4.3) and its proof in Appendix II.3]
    "F:R→[0,1] being strictly increasing, F^{-1}:[0,1]→R is also strictly increasing, so ψ∘F^{-1} is strictly increasing, and hence from Lemma D there exists a strictly proper loss (ℓ0, ℓ1) such that ℓ0 = ψ∘F^{-1}, or equivalently ψ=ℓ0∘F, as claimed."

    Eq. (16) is not derived from independent constraints; the proof sets ℓ̃0 := ψ∘F̃^{-1}, making ψ=ℓ̃0∘F̃ true by definition. The advertised universality ('any compliant ML analytical choice can be embedded with any human choice model') is then the tautology that any increasing ψ factors through any invertible increasing F̃. The hidden loss ℓ̃ is a free parameter chosen to absorb F̃, so the 'human choice model vanishes' claim is the construction itself, not a substantive prediction. Moreover the construction silently assumes F̃^{-1} is defined on all [0,1], which fails for non-surjective F̃; even where the algebra is valid, it only restates the input identity.

full rationale

The paper contains a genuine axiomatic component: Theorem 4.2 derives a Fechnerian representation from the KLST* axioms using the external Krantz et al. representation theorem, and Theorems 4.1 and 4.4 rely on standard proper-loss/Bregman theory, with self-citations serving as background facts rather than as load-bearing uniqueness claims. However, the headline disentanglement claim rests on Theorem 4.3, and its proof explicitly constructs the chief unknown as ℓ0 = ψ∘F^{-1}. The equation ψ = ℓ0∘F is therefore an identity manufactured by the proof, not a prediction with independent content. This is the central 'any ψ with any human choice model' statement, so the circularity is substantive, even though other results in the paper are independent. Separately, the proof's assertion that F^{-1} is defined on all of [0,1] fails for non-surjective F (such as the sigmoid); that is a correctness gap, not an additional circularity, and it does not change the by-construction character of the main universality claim.

Assumptions & free parameters 3 free parameters · 6 assumptions · 3 invented entities

The central claim rests on a heavy load of domain assumptions (KLST* behavioral axioms, properness premise for the RLHF objective, optimality-to-training transition) plus standard measurement theory (Krantz et al.) and proper-loss duality. The only fitted-style quantities are the toy loss parameter a and the free integration constants in Lemma D's construction; the framework itself has no parameters fitted to data.

free parameters (3)
  • Lottery mixing parameter α (MDM lotteries)
    Section 3.2: LCS axioms required for all α∈(0,1), monotonicity for a single α. Not fitted; a degree of freedom of the framework that must be validated behaviorally.
  • Toy loss parameter a in ψ_a = a ∈ {3, 6, 10}
    Section 6, Eq (26): hand-chosen values. Results depend on the value (a=10 loses to exponential loss at 44.6%, a=6 wins at 54.5%, a=3 wins at 53.0%), with no stated selection rule.
  • Integration constant K and lower limit a in Lemma D's construction
    Appendix II.3, Lemma D: any K∈R and a∈[0,1] produce a strictly proper loss, so the constructed 'hidden' loss is non-unique; the freedom is real but means the properness safeguard is partially free-form.
assumptions (6)
  • domain assumption RLHF objective with proper-loss regret: R must be a Bregman divergence (Theorem 4.1)
    Section 4.1, Eq (10): the iterative-RLHF premise that the previous optimum π* is the best initialization for the next stage 'regardless of the rewards', forcing Q to be proper. If one rejects this premise, the loss family in (22) lacks normative grounding.
  • domain assumption KLST* axioms: expandability, bearability, ZA∧P⇒ZA, P⇒ZA∧P, monotonicity (Definitions 3.2-3.4)
    Section 3.2: behavioral axioms on pairwise choice probabilities with abstention, the input to Theorem 4.2's representation p = F(u-u'). Motivated by examples, not empirically tested; monotonicity (Definition 3.3) is a difference-transitivity property that real preference data can violate.
  • standard math Krantz et al. (1989, Ch. 17, Thm 2) difference-measurement representation
    Invoked in Appendix II.2 as the external theorem yielding the additive utility representation; the paper reduces its axioms to KLST* plus expandability but does not prove the theorem itself.
  • standard math Proper loss / Bregman duality (Williamson et al. 2016, Prop 7; Gneiting & Raftery 2007)
    Used in Theorem 4.1 and Theorem 4.4; standard background results taken from the literature.
  • ad hoc to paper Optimal-policy-to-training transition: Eq (13) holds at the optimum of (9) while Eq (19) is used during training
    Section 4.2: the KKT-derived implicit reward is the reward of the optimal policy; the training loss evaluates it at π_θ. This is DPO's own approximation, inherited without discussion, and the paper presents (19) as a direct consequence.
  • ad hoc to paper Constraint F̃(z) + F̃(-z) ≥ 1 for KLST* embedding
    Section 4.3, after Eq (19): needed so that F(z) := 1 - F̃(-z) satisfies F(z) + F(-z) ≤ 1. This restricts the class of choice models covered by the 'any choice model' claim.
invented entities (3)
  • KLST* structure (local choice structure, expandability, MDM lotteries, abstention dimension)
    purpose: Axiomatic framework for pairwise choice with abstention; underpins Theorem 4.2's representation p = F(u-u') used throughout the paper.
    Invented in Section 3.2; no human-choice data are presented to test bearability, ZA∧P⇒ZA, P⇒ZA∧P, or monotonicity. The axioms are plausible but unvalidated.
  • PPPO (Proper-Proper Preference Optimization) umbrella
    purpose: Organizational device classifying DPO variants (SimPO, ORPO, IPO, RRHF, AlphaPO) in Table 1.
    Definition 5.5: a bookkeeping concept, not empirically testable; its value is organizational.
  • Toy non-convex loss ψ_a independent evidence
    purpose: Demonstrate the non-convexity freedom on gemma2 2b it.
    Eq (26), Section 6: produces a falsifiable win-rate comparison (44.6%, 54.5%, 53.0% vs exponential loss) on Alpaca Eval v2, but with two runs and no error bars.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)." pith.science (2026). https://pith.science/paper/O64Z5DXF

@misc{pith2026250707855,
  author       = {Pith},
  title        = {Pith review of: DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O64Z5DXF}},
  note         = {Machine review of arXiv:2507.07855}
}
read the original abstract

Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an explicit link with a specific normative model of human choice. Our paper elevates this connection to the full generality of DPO's normative framework. Getting there requires reworking human choice theory's textbook path for a better RLHF/ML fit. It elevates the connection to a remarkably broad viewpoint on preference optimization, considering the current panorama of DPO follow-ups. It also unveils unexpected riches for ML, chief among which the support for non-convex losses, the fact that any compliant ML analytical choice can be embedded with any human choice model, and a normative framework's umbrella wide enough to safeguard DPO's extensions (margins, length correction, ...). A toy experiment ``far away'' from the DPO crowd is given.

Figures

Figures reproduced from arXiv: 2507.07855 by the authors.

Figure 1
Figure 1. Left: ψa (26) for the three values of a considered against exponential loss.; Right: comparison of training with ψa vs expo￾nential loss. A value >50% means ψa wins (see text). stochastic error term. When these random variables are assumed to be independent and follow a specific distribution (i.e. Gumbel), the RUT framework naturally leads to BTL. Our axiomatic framework also yields to a monotonic utility function, … view at source ↗
Figure 2
Figure 2. The monotonicity property implicitly creates a real valuation of the ”niceness” of alternatives in Y α . For example, computing p(L1 ≻ L2|x) amounts to making a difference between the mappings of L1 and L2 (left, in red). The inequality in (iii), shown in the red rectangle, establishes an order between the related differences along the axis (left), and similarly for the blue rectangle. The mapping then authorizes to… view at source ↗
Figure 3
Figure 3. An illustration of assumption ZA∧P⇒ZA in graph G α x (loops indicate bearability assumption). On the left part, the thinking emoji indicates eventual abstentions in the choice L2 vs L3, which are in fact resolved if it happens that the polarity of preferences in the wedge with respect to the central node of the wedge are the same (right part of the figure). See text for details. II. Supplementary material on proofs … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Plots of ψa (46) for a = 3, 6, 10 (bottom-most to top-most thick curves) and the exponential loss (thin black curve). III. Supplementary material on toy experiment From the standpoint of the choice of a loss function, two objectives eventually competing can be followed…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 55 canonical work pages

  1. [1]

    A. Agresti. Categorical data analysis. John Wiley & Sons, 2012

    work page 2012

  2. [2]

    Alfano, S

    C. Alfano, S. Sapora, J. N. Foerster, P. Rebeschini, and Y. W. Teh. Meta-learning objectives for preference optimization, 2025. URL https://arxiv.org/abs/2411.06568

    work page arXiv 2025

  3. [3]

    Amari and H

    S. Amari and H. Nagaoka. Methods of Information Geometry. Oxford University Press, 2000

    work page 2000

  4. [4]

    S. Arimoto. Information-theoretical considerations on estimation problems. Information and control, 19: 0 181--194, 1971

    work page 1971

  5. [5]

    M. G. Azar, M. Rowland, B. Piot, Z. D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Learning Research, pages 4447--445...

    work page 2024

  6. [6]

    Bao and N

    H. Bao and N. Charoenphakdee. Being improper yet proper composite: Revisiting loss functions from perspectives of calm composite loss and property elicitation. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL https://openreview.net/forum?id=xmZaLoKHdQ

    work page 2025

  7. [7]

    J. M. Bernardo. Expected information as expected utility. The Annals of Statistics, 7: 0 686--690, 1979

    work page 1979

  8. [8]

    Blondel, A

    M. Blondel, A. F. T. Martins, and V. Niculae. Learning with Fenchel-Young losses. J. Mach. Learn. Res., 21: 0 35:1--35:69, 2020

    work page 2020

Show all 71 references
  1. [9]

    C. Chen, Z. Liu, C. Du, T. Pang, Q. Liu, A. Sinha, P. Varakantham, and M. Lin. Bootstrapping language models with DPO implicit rewards. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=dliIIodM6b

    2025

  2. [10]

    L. Chen, C. Zhu, J. Chen, D. Soselia, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro. ODIN: disentangled reward mitigates hacking in RLHF . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net...

    2024

  3. [11]

    E. Choi, A. Ahmadian, M. Geist, O. Pietquin, and M. G. Azar. Self-improving robust preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=ZSdubdbOoi

    2025

  4. [12]

    G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2024. URL https://openreview.net/forum?id=pNkOx3IVWI

    2024

  5. [13]

    A. P. Dawid. The geometry of proper scoring rules. The Annals of the ISM, 59: 0 77--93, 2007

    2007

  6. [14]

    G. Debreu. Stochastic choice and cardinal utility. Econometrica, 26: 0 440--444, 1958

    1958

  7. [15]

    Doignon and J.-C

    J.-P. Doignon and J.-C. Falmagne. Difference measurement and simple scalability with restricted solvability. Journal of Mathematical Psychology, 11 0 (4): 0 473--499, 1974

    1974

  8. [16]

    Ethayarajh, W

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  9. [17]

    Gneiting and A

    T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102 0 (477): 0 359--378, 2007

    2007

  10. [18]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024 arXiv

  11. [19]

    Y. Gu, L. Dong, F. Wei, and M. Huang. Mini LLM : Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5h0qf7IBZZ

    2024

  12. [20]

    Gupta, S

    A. Gupta, S. Tang, Q. Song, S. Zhu, J. Hong, A. Saha, V. Gupta, N. Lee, E. Kim, S. Zhu, P. Agrawal, N. Pillai, and S. S. Keerthi. Alphapo -- reward shape matters for llm alignment. In ICML'25, 2025. URL https://arxiv.org/abs/2501.03884

    2025 arXiv

  13. [21]

    Hastie, R

    T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer Series in Statistics, 2002

    2002

  14. [22]

    J. Hong, N. Lee, and J. Thorne. ORPO: monolithic preference optimization without reference model. In Y. Al - Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 1...

    2024

  15. [23]

    Huang, W

    A. Huang, W. Zhan, T. Xie, J. D. Lee, W. Sun, A. Krishnamurthy, and D. J. Foster. Correcting the mythos of KL -regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representati...

    2025

  16. [24]

    Ivison, Y

    H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023. URL https://arxiv.org/abs/2311.10702

    2023 arXiv

  17. [25]

    Kakade, A

    S. Kakade, A. T. Kalai, V. Kanade, and O. Shamir. Efficient learning of generalized linear and single index models with isotonic regression. In NIPS*24, pages 927--935, 2011

    2011

  18. [26]

    D. Kim, K. Lee, J. Shin, and J. Kim. Spread preference annotation: Direct preference judgment for efficient LLM alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=BPgK5XW1Nb

    2025

  19. [27]

    D. H. Krantz, R. D. Luce, P. Suppes, and A. Tversky. Foundations of Measurement, Volume II: Geometrical, Threshold and Probabilistic Representations. New York Academic Press, 1989

    1989

  20. [28]

    B.-K. Lee, S. Chung, C. W. Kim, B. Park, and Y. M. Ro. Phantom of latent for large language and vision models, 2025. URL https://openreview.net/forum?id=YVsiB41ifI

    2025

  21. [29]

    Y. Li, Y. Gu, L. Dong, D. Wang, Y. Cheng, and F. Wei. Direct preference knowledge distillation for large language models, 2025. URL https://arxiv.org/abs/2406.19774

    2025 arXiv

  22. [30]

    S. Liu, A. Zhang, G. Hu, H. Qian, and T.-S. Chua. Preference diffusion for recommendation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6GATHdOi1x

    2025

  23. [31]

    J. Lu, J. Li, S. An, M. Zhao, Y. He, D. Yin, and X. Sun. Eliminating biased length reliance of direct preference optimization via down-sampled KL divergence. In Y. Al - Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural...

    2024

  24. [32]

    M. J. Machina. Stochastic choice functions generated from deterministic preferences over lotteries. The Econ. Journal, 95(379): 0 575--594, 1985

    1985

  25. [33]

    M. J. Machina and M. Siniscalchi. Ambiguity and ambiguity aversion. In M. J. Machina and W. K. Viscusi, editors, Handbook of the Economics of Risk and Uncertainty, pages 729--807. Elsevier, 2014

    2014

  26. [34]

    McCarthy

    J. McCarthy. Measures of the value of information. PNAS, 42: 0 654--655, 1956

    1956

  27. [35]

    Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neura...

    2024

  28. [36]

    Mitchell

    E. Mitchell. A note on dpo with noisy preferences & relationship to ipo, 2023. URL https://ericmitchell.ai/cdpo.pdf

    2023

  29. [37]

    Nock and A

    R. Nock and A. K. Menon. Supervised learning: no loss no cry. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pages 7370--7380. PMLR , 2020. URL http://...

    2020

  30. [38]

    Nock and F

    R. Nock and F. Nielsen. On the efficient minimization of classification calibrated surrogates. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Inf...

    2008

  31. [39]

    Nock and F

    R. Nock and F. Nielsen. Bregman divergences and surrogates for learning. IEEE Trans. Pattern Anal. Mach. Intell. , 31 0 (11): 0 2048--2059, 2009. doi:10.1109/TPAMI.2008.225. URL https://doi.org/10.1109/TPAMI.2008.225

    2009 doi

  32. [40]

    R. Nock, E. Amid, and M. K. Warmuth. Boosting with tempered exponential measures. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2...

    2023

  33. [41]

    Adler, N

    Nvidia, :, B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen, S. Das, A. Dattagupta, O. Delalleau, L. Derczynski, Y. Dong, D. Egert, E. Evans, A. Ficek, D. Fridman, S. Ghosh, B. Ginsburg, I. Gitman, T. Grzegorze...

    2024 arXiv

  34. [42]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022

  35. [43]

    R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. In L. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August ...

    2024 doi

  36. [44]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing...

    2023

  37. [45]

    S. S. Ramesh, Y. Hu, I. Chaimalas, V. Mehta, P. G. Sessa, H. Bou - Ammar, and I. Bogunovic. Group robust preference optimization in reward-free RLHF . In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Informat...

    2024

  38. [46]

    M. D. Reid and R. C. Williamson. Composite binary losses. J. Mach. Learn. Res., 11: 0 2387--2422, 2010. doi:10.5555/1756006.1953012. URL https://dl.acm.org/doi/10.5555/1756006.1953012

    2010

  39. [47]

    M. D. Reid and R. C. Williamson. Information, divergence and risk for binary experiments. J. Mach. Learn. Res., 12: 0 731--817, 2011. doi:10.5555/1953048.2021029. URL https://dl.acm.org/doi/10.5555/1953048.2021029

    2011

  40. [48]

    L. J. Savage. Elicitation of personal probabilities and expectations. J. of the Am. Stat. Assoc., pages 783--801, 1971

    1971

  41. [49]

    R. Shao, B. Li, G. Liu, Y. Chen, ZhouXiang, J. Wang, X. Cai, and P. Li. Earlier tokens contribute more: Learning direct preference optimization from temporal decay perspective. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview...

    2025

  42. [50]

    Slocum, A

    S. Slocum, A. Parker-Sartori, and D. Hadfield-Menell. Diverse preference learning for capabilities and alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=pOq9vDIYev

    2025

  43. [51]

    A. Soen, I. M. Alabdulmohsin, S. Koyejo, Y. Mansour, N. Moorosi, R. Nock, K. Sun, and L. Xie. Fair wrapping for black-box predictions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Co...

    2022

  44. [52]

    S. Son, W. Bankes, S. R. Chowdhury, B. Paige, and I. Bogunovic. Right now, wrong then: Non-stationary direct preference optimization under preference drift, 2024. URL https://arxiv.org/abs/2407.18676

    2024

  45. [53]

    H. Sun, Y. Shen, and J.-F. Ton. Rethinking reward modeling in preference-based large language model alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=rfdblE10qm

    2025

  46. [54]

    Sypherd, R

    T. Sypherd, R. Nock, and L. Sankar. Being properly improper. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesv \' a ri, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings...

    2022

  47. [55]

    Tunstall, E

    L. Tunstall, E. E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. V. Werra, C. Fourrier, N. Habib, N. Sarrazin, O. Sanseviero, A. M. Rush, and T. Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https:/...

    2024

  48. [56]

    C. J. Walder and R. Nock. All your loss are belong to bayes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December ...

    2020

  49. [57]

    C. Wang, Y. Jiang, C. Yang, H. Liu, and Y. Chen. Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2cRzmWXK9N

    2024

  50. [58]

    R. C. Williamson, E. Vernet, and M. D. Reid. Composite multiclass losses. Journal of Machine Learning Research, 17 0 (222): 0 1--52, 2016. URL http://jmlr.org/papers/v17/14-294.html

    2016

  51. [59]

    T. Xiao, Y. Yuan, Z. Chen, M. Li, S. Liang, Z. Ren, and V. G. Honavar. Sim PER : A minimalist approach to preference alignment without hyperparameters. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=jfwe9qNqRi

    2025

  52. [60]

    H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. V. Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, Ju...

    2024

  53. [61]

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin. Qwen2.5-omni technical report, 2025. URL https://arxiv.org/abs/2503.20215

    2025 arXiv

  54. [62]

    S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024 b

    2024

  55. [63]

    Z. Yang, F. Wan, L. Zhong, T. Shi, and X. Quan. Weighted-reward preference optimization for implicit model fusion. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=fq24pEb8SL

    2025

  56. [64]

    H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10...

    2023

  57. [65]

    Zhang, G

    Y. Zhang, G. Zhang, Y. Wu, K. Xu, and Q. Gu. Beyond bradley-terry models: A general preference model for language model alignment, 2025. URL https://arxiv.org/abs/2410.02197

    2025 arXiv

  58. [66]

    H. Zhao, G. I. Winata, A. Das, S.-X. Zhang, D. Yao, W. Tang, and S. Sahu. Rainbow PO : A unified framework for combining improvements in preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=t...

    2025

  59. [67]

    Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305.10425, 2023. doi:10.48550/ARXIV.2305.10425. URL https://doi.org/10.48550/arXiv.2305.10425

    2023 doi arXiv

  60. [68]

    W. Zhou, R. Agrawal, S. Zhang, S. R. Indurthi, S. Zhao, K. Song, S. Xu, and C. Zhu. WPO: enhancing RLHF with weighted preference optimization. In Y. Al - Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proce...

    2024

  61. [69]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  62. [70]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  63. [71]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.