Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Explicit Preference Optimization: No Need for an Implicit Reward Model

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that DPO-style preference losses cannot preserve already-optimal behavior while improving weak regions, and introduces explicit losses—EXPO—that provably can.

desk verdict Solid, honest paper with a real formal contribution (SIC/WIC) and positive wins; the preservation theorem is narrower than the abstract admits, but the proof is correct under its stated assumptions. read the letter →

arxiv 2506.07492 v1 pith:VZRTYQTD submitted 2025-06-09 cs.LG stat.ML

Xiangkun Hu , Lemin Kong , Tong He , David Wipf This is my paper
classification cs.LGstat.ML
keywords preferenceoptimizationDPOimplicitrewardRLHFLLMalignmentregularizationinterpolationEXPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct preference optimization (DPO) and its offshoots replace the two-stage RLHF pipeline with a single loss that embeds an implicit reward. The paper claims that this reparameterization leaves a hidden structural cost: any loss in the broad quasi-convex preference optimization (QPO) family, covering DPO, IPO, GPO, and f-DPO, applies the same regularization everywhere, so improving prompts where the reference model is weak necessarily degrades prompts where it is already optimal, and as the regularization weight $\lambda\to 0$ these losses converge to a degenerate policy concentrated on the mode of the optimal policy rather than to the optimal policy itself. The paper then constructs explicit preference optimization (EXPO) losses, $\ell^c_{\mathrm{EXPO}}$ and $\ell^r_{\mathrm{EXPO}}$, that make the regularization explicit instead of implicit, and proves they preserve optimal behavior on good prompts and interpolate cleanly. A reader should care because the same structural flaw is shared by the most common alignment methods, and the proposed alternative is a drop-in objective that also wins more head-to-head comparisons in the paper's experiments.

What carries the argument

The load-bearing object is the QPO loss family, defined by a quasi-convex shape function $\psi$ and a monotone transform $\mu$ of policy-to-reference ratios; it subsumes DPO, IPO, GPO, and f-DPO, and the paper uses it to pin the structural failures on the shared implicit-reward reparameterization. The constructive machinery is EXPO's explicit objective: $\ell^c_{\mathrm{EXPO}}$ combines a supervised KL divergence between the ground-truth and induced preference distributions with an unsupervised KL to the reference policy, while $\ell^r_{\mathrm{EXPO}}$ regresses the induced preference probability onto a weighted average of reference and ground-truth preference probabilities. Both are computable from observed data without an implicit reward, and their optima recover the BT-optimal policy when the reference is already optimal.

What would settle it

Run the paper's synthetic bandit test with three responses and ground-truth preferences; if, at $\lambda\to 0$, DPO or IPO converge to the full optimal policy $\pi^*$ rather than its mode $\pi_\delta$, then Propositions 3.4 and 3.5 are wrong. Alternatively, on a dataset matching Definition D.1, find a QPO loss whose minimizer improves bad prompts while leaving $\pi^*$ untouched on good prompts, which would refute Theorem 3.1.

Watch

Extended reading notes

Core claim

The paper's central discovery is a separation between the QPO family and EXPO. For any quasi-convex preference loss of the form (10), Theorem 3.1 shows that, under a stylized dataset in which the reference policy $\pi_{\mathrm{ref}}$ already equals the BT-optimal policy $\pi^*$ on a `good` prompt partition, any minimizer that improves over $\pi_{\mathrm{ref}}$ on `bad` prompts must also move away from $\pi^*$ on good prompts. Theorem 3.6 shows that no such loss can satisfy the strong interpolation criteria, because the reparameterized losses can only reach the degenerate mode $\pi_\delta$ as $\lambda\to 0$. The proposed EXPO losses instead minimize either a KL divergence between the true and induced preference distributions plus a KL to $\pi_{\mathrm{ref}}$ (compositional), or a squared regression of the induced preference probability onto a weighted average of the reference and ground-truth preference probabilities (regression); Proposition 4.2 and Proposition 4.3 establish preservation and strong interpolation for both.

Load-bearing premise

Everything about the preservation and interpolation theorems hangs on a stylized data model: one response pair per prompt, a clean partition into good and bad prompts, an already-optimal reference on good prompts, and equal preference probabilities across the two groups; real preference data can violate any of these.

Editorial extensions

If this is right

  • If the theorems are correct, tuning $\lambda$ inside DPO, IPO, GPO, or f-DPO cannot resolve the trade-off between improving weak prompts and preserving strong prompts; a loss outside the QPO family is required.
  • If EXPO objectives behave as claimed, practitioners can train preference-aligned LLMs with a single explicit loss and unbiased SGD gradients, without reward model training or RLHF reparameterizations.
  • If the real-world results replicate, EXPO's compositional variant can incorporate unlabeled prompt data, giving offline methods access to a source of signal that was previously used mainly by online RLHF.
  • If EXPO satisfies the strong interpolation criterion, users can set $\lambda$ to control a meaningful trade-off between faithful preference matching and reference-model conservatism, instead of moving toward a degenerate mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's SIC/WIC distinction offers a cheap screening test for any new preference loss; train it on a small bandit problem with known ground truth and inspect the $\lambda\to 0$ limit before scaling up.
  • Editorial extension: because both EXPO losses operate on preference probabilities rather than log reward ratios, they could be adapted to non-Bradley-Terry labels such as ordinal or multi-graded preferences, an extension the paper does not explore.
  • Editorial extension: the preservation result predicts that DPO-family models will show measurable degradation, after alignment, on prompts where the reference model is already strong; stratifying real-data win rates by initial reference quality would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EXPO, a family of offline preference optimization losses that avoid the reparameterization trick used by DPO and related methods. The authors define a broad class of QPO losses (covering DPO, IPO, GPO, and f-DPO) and prove, under a stylized data model (Definition D.1), that any QPO loss cannot preserve an optimal policy on 'good' prompts while improving on 'bad' prompts (Theorem 3.1), and that no QPO loss satisfies the strong interpolation criterion (Theorem 3.6). They then propose two EXPO objectives (compositional and regression-based), prove that they satisfy the desired preservation and interpolation properties (Propositions 4.1–4.3), and provide synthetic and real-world experiments (Anthropic HH, IMDb, AlpacaEval 2) showing that EXPO outperforms DPO and IPO in win rates.

Significance. If the impossibility results are correct, they would establish fundamental structural limitations for a large family of popular offline preference optimization methods, beyond known issues such as lack of diversity or sensitivity to sampling. The EXPO losses are simple, computationally tractable, and do not rely on implicit reward models, making them attractive alternatives. The paper provides explicit proofs and reproducible code (stated in Appendix C), and the empirical evaluation on real datasets is a strength. However, the theoretical claims are currently proved only under restrictive assumptions that may not hold for typical preference datasets, and one of the central proofs (Theorem 3.6) is incomplete for the full QPO family as stated.

major comments (4)
  1. [Appendix D.1 (Theorem 3.1)] The proof relies crucially on Definition D.1, item 4, which requires the ground-truth preference probability p*(y1≻y2|x) to be identical for a good and a bad prompt. The proof then treats the per-prompt loss as a function of a scalar u and concludes that the minimizers for the good and bad prompts are identical. This is valid only if the policy class is unrestricted (so that u can be set independently per prompt) and if the per-prompt loss has a unique minimizer; neither condition is stated. More importantly, the theorem does not support the abstract's broad claim that 'DPO-based objectives are subject to sub-optimal regularization' for realistic datasets, where preference probabilities vary across prompts and multiple response pairs per prompt are common. Please either extend the theorem to more general data distributions or explicitly qualify the scope of the claim.
  2. [Appendix D.4 (Theorem 3.6)] The proof of Theorem 3.6 is only sketched for the case where the QPO loss depends on the log-ratio of policy and reference probabilities (i.e., μ=log). The argument introduces ρ as the log-ratio, but the general QPO loss in (10) uses an arbitrary monotonic μ, so the quantity being minimized is not necessarily a function of ρ. The step 'the only way for π̂θ to be independent of πref is if lim_{λ→0} B(λ)=±∞' is asserted without a rigorous justification, and the conclusion that only the WIC is achievable does not follow for arbitrary ψ, μ. Since Theorem 3.6 is a central impossibility result for the entire QPO family, it requires a complete proof or a restriction to a more specific subclass.
  3. [Section 3.2, Appendix E.3 (f-DPO claim)] The statement that 'minimizers of any valid f-DPO loss will be completely independent of πref for all λ∈(0,∞)' is derived from the assumption that f′ has an infinite limit at 0. Not all f-divergences used in f-DPO satisfy this property (e.g., those with bounded derivatives on their domain). The claim should be qualified to the specific divergence families for which the argument holds, or the proof should be generalized.
  4. [Section 4.3 (Proposition 4.2)] Proposition 4.2 states that the EXPO minimizer preserves π*=πref on good prompts while improving bad prompts. The proof assumes that the policy class can represent π* on good prompts independently of the updates needed for bad prompts. In a shared-parameter model with limited capacity, the preservation guarantee is only approximate and the two objectives may interfere. The paper should explicitly state this limitation, as it affects the practical relevance of the theoretical separation.
minor comments (5)
  1. [Section 4.1] The word 'Perservation' in the list of desiderata is a typo; it should be 'Preservation'.
  2. [Appendix C.1] The word 'Secion' in the sentence 'We also adopt the ℓc_EXPO loss from Secion 4.1' is a typo.
  3. [Figure 1 caption] The caption repeats 'within dbad_x' twice; the second occurrence should refer to 'dgood_x' when describing the unwanted gap.
  4. [Definition 3.2] The Strong Interpolation Criteria uses 'arg minπθ' without addressing the possibility of multiple minimizers. Please specify how ties are resolved, since the proofs (e.g., Theorem 3.1) implicitly require uniqueness.
  5. [Section 5 (real-world experiments)] The win-rate results in Figure 6 and Table 2 are reported without confidence intervals or statistical significance tests. Given the relatively small differences (e.g., DPO vs EXPO on AlpacaEval LC), error bars would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: EXPO is transparently constructed from its own stated desiderata, and the theoretical comparisons are self-contained; the sole self-citation is contextual and not load-bearing.

full rationale

The paper's central claims are the impossibility results for QPO losses (Theorems 3.1 and 3.6) and the construction of EXPO losses that avoid them (Propositions 4.1-4.3). The QPO family in (10) is defined independently of the proposed desiderata of preservation and interpolation, and the proofs of Theorems 3.1 and 3.6 proceed from the QPO loss form and Definition D.1 without importing the EXPO construction. Proposition 4.1 is a direct algebraic equivalence between (17) and (19), verified in Appendix D.6 using only the generative process from (1); no fitted parameter or prior result is used as an input. Propositions 4.2 and 4.3 follow by inspection from the definitions of the EXPO losses, which is a design verification rather than a prediction from fitted values. The only self-citation is Kong et al. (2025), cited twice in contextual remarks about how learning constraints can obscure DPO's RLHF interpretation; those remarks do not support any of the formal theorems, and the main theoretical separation is proven in this paper rather than delegated to the citation. Whether Theorem 3.1's proof is fully convincing, for example the step from equations (25)-(26) treating per-prompt minimizers as identical arguments, is a correctness or generality question, not a circularity one. The paper is self-contained against external benchmarks and its derivations do not reduce to their inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the Bradley-Terry preference model, the coverage of the QPO family, the i.i.d. sampling assumption, and a stylized prompt partition; the only fitted constant in the method is the trade-off lambda. No new physical entities are postulated.

free parameters (2)
  • lambda (EXPO trade-off) = 0.05 (compositional) and 0.2 (regression) on Anthropic HH; 0.1 and 0.01 on IMDb; 0.2 (regression) on AlpacaEval 2
    The EXPO losses require a hyperparameter lambda balancing the supervised preference term and the reference-model regularization. The paper tunes lambda per dataset and per variant, so the favorable comparisons are partly conditional on these hand-chosen values.
  • lambda (DPO/IPO baselines) = 0.1 for both on HH and IMDb; 0.05 (DPO) and 0.5 (IPO) on AlpacaEval 2
    Baseline performance depends on tuned lambda values; this is standard, but it means the empirical comparisons are not parameter-free.
assumptions (4)
  • domain assumption Human preferences follow the Bradley-Terry model (Eq 2): p*(y1≻y2|x)=σ(r*(y1,x)-r*(y2,x)) for some latent reward r*.
    All theoretical results, including the definition of a BT-optimal policy (Eq 12) and the SIC/WIC criteria, assume this generative model. If real preferences are not BT, the notion of π* used in Theorem 3.6 and Propositions 4.2/4.3 is not well-defined.
  • domain assumption The QPO family in Eq (10) with differentiable quasi-convex ψ and monotone increasing µ covers essentially all reasonable preference-optimization losses.
    The paper claims DPO, IPO, GPO, and f-DPO are special cases and that multi-modal losses are nonsensical. The impossibility theorems inherit this coverage claim; an exotic loss outside this family could potentially satisfy the SIC.
  • domain assumption Training tuples are sampled i.i.d. from Dtr as in Eq (1): z~p*, {y1,y2}~πref, x~Dx.
    The equivalence in Proposition 4.1 and the statistical interpretation of the losses rely on this sampling process. Selection bias in real preference datasets could break the equivalence.
  • ad hoc to paper The data partition in Definition D.1 (single response pair per prompt, disjoint good/bad prompt sets, identical preference probabilities across sets) is representative enough to expose the structural limitations of QPO losses.
    Theorem 3.1 and Proposition 4.2 are proved only under this stylized setup; the paper asserts it can be relaxed, but does not provide the relaxed proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explicit Preference Optimization: No Need for an Implicit Reward Model." pith.science (2026). https://pith.science/paper/VZRTYQTD

@misc{pith2026250607492,
  author       = {Pith},
  title        = {Pith review of: Explicit Preference Optimization: No Need for an Implicit Reward Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZRTYQTD}},
  note         = {Machine review of arXiv:2506.07492}
}
read the original abstract

The generated responses of large language models (LLMs) are often fine-tuned to human preferences through a process called reinforcement learning from human feedback (RLHF). As RLHF relies on a challenging training sequence, whereby a separate reward model is independently learned and then later applied to LLM policy updates, ongoing research effort has targeted more straightforward alternatives. In this regard, direct preference optimization (DPO) and its many offshoots circumvent the need for a separate reward training step. Instead, through the judicious use of a reparameterization trick that induces an \textit{implicit} reward, DPO and related methods consolidate learning to the minimization of a single loss function. And yet despite demonstrable success in some real-world settings, we prove that DPO-based objectives are nonetheless subject to sub-optimal regularization and counter-intuitive interpolation behaviors, underappreciated artifacts of the reparameterizations upon which they are based. To this end, we introduce an \textit{explicit} preference optimization framework termed EXPO that requires no analogous reparameterization to achieve an implicit reward. Quite differently, we merely posit intuitively-appealing regularization factors from scratch that transparently avoid the potential pitfalls of key DPO variants, provably satisfying regularization desiderata that prior methods do not. Empirical results serve to corroborate our analyses and showcase the efficacy of EXPO.

Figures

Figures reproduced from arXiv: 2506.07492 by the authors.

Figure 1
Figure 1. Preservation of optimal policies; proposed EXPO [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Interpolation illustration; a proposed EXPO vari [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Support for Sections 3.2 and 4.3 interpolation analysis. Dashed lines represent BT-optimal preference probabilities π ∗ , while solid lines are model learning curves for λ = 10−5 (small). Only EXPO converges to π ∗ , others converge to π δ , a degenerate solution with no generative diversity; instead all mass concentrated on just a single response at odds with the ground-truth, BT-optimal policy underlying the data … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Further support for interpolation analysis. Each plot displays the final converged probability distributions πθ(y) across varying λ (small to large) under the same conditions as in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Support for Sections 3.1 and 4.3 preservation analysis while varying λ. In the top plot prompts are drawn from d good x where πref = π ∗ , and yet as λ is reduced existing methods produce a policy that increasingly deviates from π ∗ . In contrast, in the bottom plot pr…
Figure 7
Figure 7. Figure 7: Comparison with additional baselines (outside the QPO family). Both KTO and SimPO benefit from the inclusion [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Converged probability distributions of πθ(y) for DPO, IPO, f-DPO and EXPO with large λ (here λ = 100). All methods immediately stabilize around πref, the initialization point, as expected. This figure can be viewed as the complement of [PITH_FULL_IMAGE:figures/full_fi…
Figure 9
Figure 9. Figure 9: The prompt used for evaluating the win rates of the generated responses against the chosen responses for single [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 13 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLM s

    Ahmadian, A., Cremer, C., Gall \'e , M., Fadaee, M., Kreutzer, J., \"U st \"u n, A., and Hooker, S. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLM s. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  3. [3]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  4. [4]

    Direct preference optimization with an offset

    Amini, A., Vieira, T., and Cotterell, R. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571, 2024

    arXiv 2024

  5. [5]

    G., Guo, Z

    Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    2024

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 a

    arXiv 2022

  7. [7]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 b

    arXiv 2022

  8. [8]

    G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

Show all 57 references
  1. [9]

    and Rinaldo, A

    Bong, H. and Rinaldo, A. Generalized results for the existence and consistency of the mle in the bradley-terry-luce model. In International Conference on Machine Learning, pp.\ 2160--2177. PMLR, 2022

    2022

  2. [10]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  3. [11]

    T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023

    2023

  4. [12]

    A survey on evaluation of large language models

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 0 (3): 0 1--45, 2024

    2024

  5. [13]

    Bootstrapping language models with DPO implicit rewards

    Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q., Sinha, A., Varakantham, P., and Lin, M. Bootstrapping language models with DPO implicit rewards. arXiv preprint arXiv:2406.09760, 2024

    2024 arXiv

  6. [14]

    Ultrafeedback: Boosting language models with scaled ai feedback

    Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pp.\ 9722--9744. PMLR, 2024

    2024

  7. [15]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 3029--3051, 2023

    2023

  8. [16]

    Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  9. [17]

    Kto: Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  10. [18]

    Towards analyzing and understanding the limitations of DPO : A theoretical perspective

    Feng, D., Qin, B., Huang, C., Zhang, Z., and Lei, W. Towards analyzing and understanding the limitations of DPO : A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024

    2024 arXiv

  11. [19]

    O., Rossi, R

    Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., and Ahmed, N. K. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770, 2023

    2023 arXiv

  12. [20]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  13. [21]

    Learn your reference model for real good alignment

    Gorbatovski, A., Shaposhnikov, B., Malakhov, A., Surnachev, N., Aksenov, Y., Maksimov, I., Balagansky, N., and Gavrilov, D. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656, 2024

    2024 arXiv

  14. [22]

    and Pierskalla, W

    Greenberg, H. and Pierskalla, W. A review of quasi-convex functions. Operations research, 19 0 (7): 0 1553--1570, 1971

    1971

  15. [23]

    Deepseek- R 1: I ncentivizing reasoning capability in LLM s via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek- R 1: I ncentivizing reasoning capability in LLM s via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  16. [24]

    Orpo: Monolithic preference optimization without reference model

    Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024

    2024 arXiv

  17. [25]

    and Li, Y

    Im, S. and Li, Y. Understanding the learning dynamics of alignment with human feedback. arXiv preprint arXiv:2403.18742, 2024

    2024 arXiv

  18. [26]

    https://github.com/huggingface/trl/pull/1265

    Kashif. https://github.com/huggingface/trl/pull/1265. 2024

    2024

  19. [27]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  20. [28]

    Common learning constraints alter interpretations of direct preference optimization

    Kong, L., Hu, X., He, T., and Wipf, D. Common learning constraints alter interpretations of direct preference optimization. In The 28th International Conference on Artificial Intelligence and Statistics, 2025

    2025

  21. [29]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  22. [30]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  23. [31]

    Policy optimization in rlhf: The impact of out-of-preference data

    Li, Z., Xu, T., and Yu, Y. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584v2, 2024

    2024 arXiv

  24. [32]

    On the limited generalization capability of the implicit reward model induced by direct preference optimization

    Lin, Y., Seto, S., Ter Hoeve, M., Metcalf, K., Theobald, B.-J., Wang, X., Zhang, Y., Huang, C., and Zhang, T. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409.03650, 2024

    2024 arXiv

  25. [33]

    L., Daly, R

    Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp.\ 142--150, Portland, Oregon, US...

    2011

  26. [34]

    Simpo: Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  27. [35]

    Active preference learning for large language models

    Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Active preference learning for large language models. arXiv preprint arXiv:2402.08114, 2024

    2024 arXiv

  28. [36]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  29. [37]

    Smaug: F ixing failure modes of preference optimisation with DPO -positive

    Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: F ixing failure modes of preference optimisation with DPO -positive. arXiv preprint arXiv:2402.13228, 2024

    2024 arXiv

  30. [38]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024

    2024 arXiv

  31. [39]

    B., Kumar, A., Zhang, G., and Levine, S

    Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  32. [40]

    and Schaal, S

    Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp.\ 745--750, 2007

    2007

  33. [41]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization

    Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint...

    arXiv

  35. [43]

    Rubenstein, P., Bousquet, O., Djolonga, J., Riquelme, C., and Tolstikhin, I. O. Practical and consistent estimation of f-divergences. Advances in Neural Information Processing Systems, 32, 2019

    2019

  36. [44]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  37. [45]

    Deepseek M ath: P ushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseek M ath: P ushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  38. [46]

    The importance of online data: U nderstanding preference fine-tuning via coverage

    Song, Y., Swamy, G., Singh, A., Bagnell, J., and Sun, W. The importance of online data: U nderstanding preference fine-tuning via coverage. Advances in Neural Information Processing Systems, 37: 0 12243--12270, 2024

    2024

  39. [47]

    M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2020. URL https://arxiv. org/abs, 2009

    2020

  40. [48]

    S., and Bagnell, J

    Swamy, G., Choudhury, S., Sun, W., Wu, Z. S., and Bagnell, J. A. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067, 2025

    2025

  41. [49]

    Preference fine-tuning of LLM s should leverage suboptimal, on-policy data

    Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of LLM s should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024

    2024 arXiv

  42. [50]

    D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P

    Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. \'A ., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024

    2024 arXiv

  43. [51]

    Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints

    Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints. International Conference on Learning Representations, 2024 a

    2024

  44. [52]

    Y., Xu, N., Zhang, S., Poon, H., and Chen, M

    Wang, F., Zhou, W., Huang, J. Y., Xu, N., Zhang, S., Poon, H., and Chen, M. m DPO : C onditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406.11839, 2024 b

    2024 arXiv

  45. [53]

    Is DPO superior to PPO for LLM alignment? A comprehensive study

    Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is DPO superior to PPO for LLM alignment? A comprehensive study. arXiv preprint arXiv:2404.10719, 2024

    2024 arXiv

  46. [54]

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models. arXiv preprint arXiv...

    2023 arXiv

  47. [55]

    Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. SL i C - HF : S equence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023 b

    2023 arXiv

  48. [56]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.