Pith. sign in

REVIEW 5 major objections 6 minor 27 references

Reward over-optimization in direct preference alignment stems from an unnormalized implicit reward, and a single pair-level probability-conservation penalty mitigates it for both reference-based (DPO) and reference-free (SimPO) methods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:55 UTC pith:MVPY77PA

load-bearing objection A simple, plausible pair-level probability-conserving regularizer for DPO/SimPO; worth a referee, but the empirical claims are undercut by inconsistent numbers and the mechanistic story overreaches. the 5 major comments →

arxiv 2607.16240 v1 pith:MVPY77PA submitted 2026-06-26 cs.LG cs.AI

Normalized Rewards for Preference Optimization

Shawn Im , Federico Danieli , Skyler Seto , Barry-John Theobald , Katherine Metcalf This is my paper
classification cs.LG cs.AI
keywords direct preference optimizationreward over-optimizationlikelihood displacementreward normalizationSimPOimplicit rewardoutlier tokensLLM alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that direct alignment algorithms such as DPO over-optimize because their implicit reward is not normalized: the partition function that should keep the model a proper probability distribution is dropped from the objective, so nothing stops the likelihoods of preferred and rejected responses from falling together. The proposed fix is a one-term penalty that conserves the total length-normalized probability mass of the two responses seen in each preference pair. Applied unchanged to DPO and to the reference-free SimPO, the penalty reduces likelihood displacement, keeps the model's probability assignments to unseen responses from drifting, and improves the trade-off between generation quality and general benchmark performance. The paper also reports that likelihood displacement is concentrated in a small set of outlier tokens, and that the penalty corrects those outliers by acting through low-likelihood tokens. If correct, this gives a single, method-agnostic explanation for reward over-optimization across reference-based and reference-free objectives.

Core claim

The central discovery is that likelihood displacement—the simultaneous drop in the likelihoods of both the preferred and non-preferred responses during DPO-style training—is a symptom of an unnormalized implicit reward. Because the DPO and SimPO objectives cancel the partition function Z(x), the model is free to shrink the total probability mass assigned to the observed responses, which spuriously inflates the probability of unseen responses. The paper proposes to fix this by adding the penalty R(πθ,πref,x,yw,yl)=λ[log((π̄θ(yw)+π̄θ(yl))/(π̄ref(yw)+π̄ref(yl)))]², which encourages the length-normalized probability mass of the chosen and rejected responses to stay equal to its reference value.

What carries the argument

The load-bearing object is the length-normalized pair-mass ratio penalty R(πθ,πref,x,yw,yl)=λ[log((π̄θ(yw)+π̄θ(yl))/(π̄ref(yw)+π̄ref(yl)))]². Here π̄ is the length-normalized probability of a response (the geometric mean of its token probabilities), and the penalty is minimized when the sum of the chosen and rejected response probabilities under the trained policy equals the same sum under the reference policy. The paper's argument is that this pair-sum is a usable stand-in for the intractable partition function Z(x): since the model's distribution is always normalized, conserving the mass of the two observed responses should protect the mass of all unseen responses. The penalty is added unc

Load-bearing premise

The paper assumes that conserving the total probability of the two responses seen for a prompt keeps the model's probabilities over all unseen responses stable—that the pair sum is a reliable proxy for the full, intractable partition function, a step it motivates only with a three-response toy example and does not bound.

What would settle it

Compute the actual total probability assigned to a held-out set of responses (or an estimate of the partition function) before and after N-DPO training; if the pair-sum is conserved but the held-out response probabilities still shift substantially, the proxy fails. A simpler check: in a small softmax language model, measure the gradient norm of a token's log-probability as p→0; the paper's '1e−7× dominance' claim predicts ~1/p growth, whereas a softmax parameterization gives a bounded gradient that vanishes as p→0.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • N-DPO and N-SimPO improve AlpacaEval2 win-rates relative to DPO/SimPO (e.g., over 20% relative increase in length-controlled win-rate for Llama-3.1-8B-Instruct) while reducing benchmark degradation on common-sense and reasoning tasks.
  • Implicit reward models trained with the penalty generalize better to unseen preference datasets such as HH-RLHF and HelpSteer2 in most configurations tested.
  • Because likelihood displacement is concentrated in a small set of outlier tokens, standard DPO can improve generation quality even while lowering chosen-response likelihood; the penalty removes this confounding behavior.
  • Since the same penalty applies to a reference-based and a reference-free method, it offers a unified account of over-optimization across DAA families rather than a method-specific patch.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pair-sum proxy for the partition function holds, a similar probability-conservation penalty could be adapted to online or iterative alignment methods, though the paper explicitly leaves those settings out—this is my inference, not the paper's claim.
  • The outlier-token concentration suggests a potentially simpler intervention: directly clipping or upweighting low-likelihood tokens during preference optimization might reproduce much of the benefit without introducing an extra hyperparameter.
  • The paper's mechanistic claim that low-likelihood tokens dominate the gradient (roughly as 1/p) can be tested directly in a small softmax model; if the gradient norm actually behaves like sqrt(1-p), the mechanism described in Section 4.2 would need revision even if the empirical regularizer still works.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper argues that reward over-optimization—specifically likelihood displacement—in direct alignment algorithms (DAAs) stems from a lack of normalization of the implicit reward. To address this, it introduces a regularizer R (Eq. 12) that penalizes the squared log ratio of the length-normalized total probabilities of the chosen and rejected responses under the trained and reference models. The modified objectives N-DPO (Eq. 13) and N-SimPO (Eq. 14) are evaluated on Mistral-7B, Llama-3.1-8B, and OLMo-7B trained on UltraFeedback, measuring AlpacaEval win rates, common-sense/reasoning benchmarks, and implicit reward accuracy. The paper also presents a token-level analysis (Figure 3) and a gradient derivation (Section 4.2, Appendix C) claiming that likelihood displacement is concentrated on outlier tokens and that the regularizer corrects these outliers by acting through low-likelihood tokens.

Significance. If the central claims held, this would be a useful minimal modification: a single penalty term applicable to both reference-based and reference-free DAA objectives, with evidence of improved generation-quality/benchmark trade-offs. The evaluation spans three model families and multiple benchmarks, and the gradient derivation in Appendix C is symbolic and clearly presented. The paper also contributes an interesting observational study of token-level likelihood displacement. However, the current manuscript does not cleanly isolate the effect of the regularizer for DPO, several headline numbers in the abstract are not supported by the tables, and the gradient-based mechanism contains a mathematical claim that is not correct for softmax language models. These issues are load-bearing and need to be resolved before the contribution can be fully credited.

major comments (5)
  1. [Abstract; §1; Table 1; Table 3] The abstract claims 'over a 20% relative increase in AlpacaEval2 scores' for Llama-3.1-8B-Instruct, but Table 1 shows LC 21.72→24.97 (+15.0%) and WR 24.18→27.57 (+14.0%). No reported number reaches 20%. Similarly, the abstract's 'improvements in reward modeling across datasets' is contradicted by Table 3: N-DPO lowers HelpSteer2 accuracy for Mistral (66.15→56.33) and HH-RLHF for Llama (59.55→58.73), while N-SimPO results are mixed. Please recalibrate these claims.
  2. [§5.2; Appendix A; Table 4; Table 1] The isolated effect of the regularizer is not properly tested for DPO. N-DPO differs from DPO by both length normalization and R, so the λ=0 ablation (L-DPO) is the right control, but it is reported only in Appendix A and only for AlpacaEval. Moreover, the λ=0 and N-DPO numbers in Table 4 disagree with Table 1 (e.g., Llama N-DPO LC 29.41 vs 24.97; OLMo 8.02 vs 9.88; Mistral N-DPO WR 18.16 vs 20.08). No Table-2 benchmark deltas are given for λ=0, so the headline trade-off claim cannot be attributed to R. Please provide a full matched comparison of DPO, L-DPO (λ=0), and N-DPO on all metrics, and explain the discrepancies between Table 1 and Table 4.
  3. [§4.2; Eq. (16)] The claim that ∇θπθ(y(i))/πθ(y(i)) is 'inversely proportional to the likelihood of each token' is not correct for softmax parameterization. For final-layer logits z_i, ∇_{z_i} π_i / π_i = 1 − π_i, which is bounded and tends to 1 as π_i→0, not to infinity. The '1e−7× dominance' therefore does not follow from the model's parameterization. This is a load-bearing step for the mechanistic explanation in Section 4.2 and the abstract. Please provide a correct derivation or retract the mechanism.
  4. [§3; §1; Figure 1] The paper motivates R as preserving the partition function Z(x), but the regularizer only preserves the mass of the two seen responses (yw, yl). In real preference data, these two responses are a tiny fraction of Z, so Figure 1's three-response example (where the pair covers 70% of the mass) does not transfer. Either provide a formal link—for example, a bound or assumption under which pair-mass conservation controls changes in Z—or reframe R as a heuristic whose benefits are empirical, not as a direct normalization of the partition function.
  5. [§5; Table 1; Table 2; §7] The claim of 'improved trade-offs between generation quality and general benchmark capability' is not consistently supported even for the clean comparison (SimPO vs N-SimPO). AlpacaEval2 LC decreases for Mistral (26.34→26.05) and Llama (26.74→25.54) in Table 1, and for OLMo the benchmark average degrades sharply (−0.73→−4.81) in Table 2. The introduction and discussion should be adjusted to reflect where N-SimPO actually improves, rather than stating a general improvement.
minor comments (6)
  1. [§2; Eq. (8)-(9)] SimPO is described as reference-free, but Eq. (8) defines the reward with a Z(x) and Eq. (9) defines Z(x) using π_ref. Please clarify what Z(x) denotes in the SimPO context and whether Eq. (9) is the standard SimPO formulation or a new interpretation.
  2. [Figure 3] The axis labels 'Minimum Score Distribution' and 'Token Score Distribution' are ambiguous. The text refers to 'minimum reward per sample' and 'token-wise reward'; please unify the terminology and clarify what is being plotted, including the unit.
  3. [Appendix B; Tables 20-24] The hyperparameter tables are hard to parse. For example, Table 24 lists β values as '(0.03/1.0/3.0/3.0)' without a header defining the order, and the sweep ranges overlap confusingly. Please add explicit column/row headers for the four methods.
  4. [References; Related Works] Several citations are incomplete or formatted inconsistently: 'Gupta et al., a' and 'Gupta et al., b' lack venues/years in the running text, and the bibliography entries for these references should be completed.
  5. [§5.1; Table 3] Table 3 reports reward accuracy without standard errors or a description of how the pairs were split/aggregated. At least a footnote on the evaluation protocol would help interpret the differences.
  6. [Abstract; §7] Minor wording: 'mitigates the amount of displacement within preferred responses overall, and for the outlier tokens specifically, by utilizing low-likelihood tokens' is grammatically awkward and should be rephrased for clarity.

Circularity Check

1 steps flagged

Benchmark gains are external and non-circular, but the displacement-mitigation 'finding' restates the regularizer's definition; no self-citation chain or imported uniqueness.

specific steps
  1. self definitional [Section 3, Eq. (12)-(13); Section 4.1 and Figure 2]
    "By maintaining the total probability, we mitigate offsets in likelihood that would occur given the lack of information about the full response distribution... We demonstrate that adding regularization results in less likelihood displacement, particularly for DPO where the chosen responses maintain similar likelihood throughout training."

    Eq. (12) defines R = λ [log((π̄θ(yw)+π̄θ(yl))/(π̄ref(yw)+π̄ref(yl)))]², which is minimized exactly when the length-normalized sum of the two seen responses equals the reference sum. The paper defines likelihood displacement as the decrease in total likelihood assigned to responses seen in the preference dataset, so 'the regularizer mitigates displacement on the seen pair' restates the penalty's objective. The independent evidence is the external AlpacaEval/benchmark comparison; the displacement-mitigation finding adds no information beyond the construction.

full rationale

The paper's central benchmark claims are not circular: N-DPO and N-SimPO are evaluated on AlpacaEval2 and LM Evaluation Harness, external to the regularizer's definition, and the reported gains in Tables 1-2 are genuine empirical evidence. No self-citation chain or imported uniqueness theorem appears; citations to DPO, SimPO, Razin et al., and Yoon et al. are prior external work. However, one stated finding—that the regularizer mitigates likelihood displacement on the seen (chosen, rejected) pair—is by construction: R is defined as the squared log deviation of the pair's normalized probability mass from the reference, and likelihood displacement is defined as the drop in exactly that quantity. Thus that specific 'finding' is a restatement of Eq. (12), not an independent discovery. The mechanistic outlier-token analysis is also a property of the chosen regularizer's gradient rather than an independent test. These are correctness/interpretation issues (the '1/p' gradient claim is unsupported; the λ=0 ablation lacks benchmark deltas and Table 4 disagrees with Table 1), but they are not additional circularity because the headline benchmark improvements are externally evaluated.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on three non-trivial premises the paper does not fully pay for: (1) the pair-sum proxy for the unobservable partition function (Eq. 12, §3), (2) the 1/p token-gradient dominance statement (§4.2), and (3) the per-model/method choice of λ, β, γ, LR selected on AlpacaEval1 (App. B). The method adds one new fitted strength λ (0.025–0.1) on top of standard DAA hyperparameters. No invented entities are introduced.

free parameters (4)
  • λ (regularization strength) = Mistral N-DPO 0.1; Llama N-DPO 0.025; OLMo N-DPO 0.05; Mistral N-SimPO 0.1; Llama N-SimPO 0.025; OLMo N-SimPO 0.025
    Central coefficient of the proposed penalty (Eq. 12); chosen per model/method by AlpacaEval1 win-rate (App. B, Tables 22/24). Headline gains are for these fitted values; robustness is only checked for λ∈[0.025,0.1].
  • β (DPO/SimPO temperature) = Mistral DPO 0.03, N-DPO 1.0; Llama DPO 0.01, N-DPO 3.0; OLMo DPO 0.03, N-DPO 3.0 (Table 24)
    Standard DAA hyperparameter; swept per model/method on AlpacaEval1 (Table 20).
  • γ (SimPO margin) = Mistral SimPO 1.6 / N-SimPO 2.0; Llama 0.8 / 2.0; OLMo 0.4 / 1.6 (Table 24)
    Standard SimPO hyperparameter; swept per model/method (Table 21).
  • Learning rate = 1e-7 to 1e-6 across methods/models (Table 24)
    Swept per model (Table 23); single-seed runs with no reported variance.
axioms (5)
  • domain assumption An autoregressive softmax LM is always normalized, so the implicit reward's partition function should remain 1 during training.
    Section 3, first paragraph; predicates the method on constraining Z(x), but Z(x) in Eq. (4) is a sum over an unobservable response space, not the model's softmax normalization.
  • ad hoc to paper Conserving the length-normalized probability mass of the seen pair (yw,yl) approximately conserves the global partition function over unseen responses.
    Section 3, Eq. (12) and Figure 1; asserted via a 3-response toy example; no derivation or bound connects pair-mass conservation to Z(x).
  • ad hoc to paper Token gradient contributions ∇θπθ(y(i))/πθ(y(i)) are inversely proportional to token likelihood, so a 1e-7× rarer token dominates the update.
    Section 4.2; for softmax log-probabilities the gradient norm is ≈√(1−p_t), bounded as p_t→0, so the stated inverse-proportionality is not a property of the parameterization.
  • domain assumption Length-normalized probability π̄(y)=π(y)^{1/|y|} is the correct quantity to conserve and to score rewards with.
    Used in Eq. (12) and in the reward-accuracy evaluation (Section 5.1); inherited from SimPO, but changes what 'displacement' means for the chosen metric.
  • standard math Calculus identities used in App. C (chain rule; derivative of a product of token probabilities).
    Verification: Eq. (22) correctly expands ∇R; no issue.

pith-pipeline@v1.3.0-alltime-deepseek · 19636 in / 27511 out tokens · 248103 ms · 2026-08-02T09:55:28.384271+00:00 · methodology

0 comments
read the original abstract

Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences. However, DAAs have been observed to over-optimize their implicit reward model and decrease the likelihood of preferred responses. This results in a decrease in the total likelihood assigned to responses seen in the preference dataset, potentially resulting in undesirable behavior. To counteract this undesired side-effect of DAAs, we examine the effect of using objectives that add a regularization term to maintain the total length-normalized probabilities of the chosen and rejected responses. To better understand over-optimization, we investigate how response likelihood changes are distributed over the tokens with and without regularization. We find that a significant portion of the likelihood changes are due to a small set of outlier tokens, which explains how DAAs improve generation quality despite decreasing the likelihoods of chosen responses. We apply the proposed regularization to reference-based (DPO) and reference-free (SimPO) methods and find (1) improved trade-offs between generation quality and general benchmark capability and (2) improvements in reward modeling across datasets. For example, on Llama-3.1-8B-Instruct, we see both a >20% relative increase in AlpacaEval2 scores and >9% relative performance gains on general benchmarks. Additionally, we find that the added regularization term effectively mitigates the amount of displacement within preferred responses overall, and for the outlier tokens specifically, by utilizing low-likelihood tokens.

Figures

Figures reproduced from arXiv: 2607.16240 by Barry-John Theobald, Federico Danieli, Katherine Metcalf, Shawn Im, Skyler Seto.

Figure 1
Figure 1. Figure 1: Illustrative example of likelihood displacement and how our regularization mitigates it. The reference model assigns likelihoods 0.4, 0.3, 0.3 to responses A, B, C, with matching probabilities since the partition function equals 1. We then train on the single preference A < B; the response C is never observed. The example highlights the distinction between unnormalized likelihoods and normalized probabilit… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of response likelihoods between DPO/SimPO (left) and N-DPO/N-SimPO (right) for Llama-3.1-8B-Instruct. In blue is the chosen response likelihood and in red the rejected response likelihood. 5.1. Evaluation We focus our evaluation on understanding the effect of our modifications by comparing DPO with N-DPO and SimPO with N-SimPO. We expect that due to the more strongly enforced normalization of re… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of token-wise reward distributions between DPO and N-DPO for Llama-3.1-8B-Instruct. On the left is the distribution of the minimum token reward per sample, and on the right is the distribution of all token rewards. Common Sense and Reasoning. In addition to instruction-following quality, we perform evaluation on various benchmarks from LM Evaluation Harness (Gao et al., 2024) to see how well the… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 24 linked inside Pith

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Dubois, Y ., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    Pith/arXiv arXiv

  3. [5]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

    Pith/arXiv arXiv

  4. [6]

    Robust preference optimization through reward model distillation.arXiv preprint arXiv:2405.19316,

    Fisch, A., Eisenstein, J., Zayats, V ., Agarwal, A., Beirami, A., Nagpal, C., Shaw, P., and Berant, J. Robust preference optimization through reward model distillation.arXiv preprint arXiv:2405.19316,

    Pith/arXiv arXiv

  5. [7]

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A

    URL https://zenodo.org/records/12608602. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y ., et al. Olmo: Accelerating the science of lan- guage models.arXiv preprint arXiv:2402.00838,

    arXiv

  6. [8]

    Alphapo: Reward shape matters for llm alignment.arXiv preprint arXiv:2501.03884,

    Gupta, A., Tang, S., Song, Q., Zhu, S., Hong, J., Saha, A., Gupta, V ., Lee, N., Kim, E., Zhu, S., et al. Alphapo: Reward shape matters for llm alignment.arXiv preprint arXiv:2501.03884,

    Pith/arXiv arXiv

  7. [10]

    D., Sun, W., Krish- namurthy, A., and Foster, D

    Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krish- namurthy, A., and Foster, D. J. Correcting the mythos of kl-regularization: Direct alignment without overopti- mization via chi-squared preference optimization.arXiv preprint arXiv:2407.13399,

    Pith/arXiv arXiv

  8. [12]

    Y ., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., and Choi, Y

    Lin, B. Y ., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., and Choi, Y . The unlocking spell on base llms: Rethinking alignment via in-context learning.arXiv preprint arXiv:2312.01552, 2023a. Lin, Y ., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., et al. Mitigating the alignment tax o...

    Pith/arXiv arXiv

  9. [13]

    Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive.arXiv preprint arXiv:2402.13228,

    Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive.arXiv preprint arXiv:2402.13228,

    Pith/arXiv arXiv

  10. [14]

    Safety alignment should be made more than just a few tokens deep.arXiv preprint arXiv:2406.05946,

    Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just a few tokens deep.arXiv preprint arXiv:2406.05946,

    Pith/arXiv arXiv

  11. [15]

    Unintentional unalignment: Likelihood displacement in direct preference optimization.arXiv preprint arXiv:2410.08847,

    Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization.arXiv preprint arXiv:2410.08847,

    Pith/arXiv arXiv

  12. [16]

    Preference fine-tuning of llms should leverage suboptimal, on-policy data.arXiv preprint arXiv:2404.14367,

    Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of llms should leverage suboptimal, on-policy data.arXiv preprint arXiv:2404.14367,

    Pith/arXiv arXiv

  13. [17]

    Game-theoretic regularized self-play alignment of large language models.arXiv preprint arXiv:2503.00030,

    Tang, X., Yoon, S., Son, S., Yuan, H., Gu, Q., and Bogunovic, I. Game-theoretic regularized self-play alignment of large language models.arXiv preprint arXiv:2503.00030,

    Pith/arXiv arXiv

  14. [18]

    D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P

    Tang, Y ., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. ´A., and Piot, B. Generalized preference optimization: A unified approach to offline alignment.arXiv preprint arXiv:2402.05749,

    Pith/arXiv arXiv

  15. [19]

    Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints.arXiv preprint arXiv:2309.16240,

    Wang, C., Jiang, Y ., Yang, C., Liu, H., and Chen, Y . Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints.arXiv preprint arXiv:2309.16240,

    Pith/arXiv arXiv

  16. [20]

    URL https://arxiv.org/abs/2410. 01257. Wu, Y ., Sun, Z., Yuan, H., Ji, K., Yang, Y ., and Gu, Q. Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675,

    Pith/arXiv arXiv

  17. [21]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.arXiv preprint arXiv:2312.11456,

    Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.arXiv preprint arXiv:2312.11456,

    Pith/arXiv arXiv

  18. [22]

    S., Yoon, E., Hasegawa-Johnson, M

    Yoon, H. S., Yoon, E., Hasegawa-Johnson, M. A., Kim, S., and Yoo, C. D. Confpo: Exploiting policy model confidence for critical token selection in large lan- guage model preference optimization.arXiv preprint arXiv:2506.08712,

    Pith/arXiv arXiv

  19. [23]

    I., Das, A., Zhang, S.-X., Yao, D

    Zhao, H., Winata, G. I., Das, A., Zhang, S.-X., Yao, D. D., Tang, W., and Sahu, S. Rainbowpo: A unified framework for combining improvements in preference optimization. arXiv preprint arXiv:2410.04203,

    Pith/arXiv arXiv

  20. [24]

    Zhao, Y ., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425,

    Pith/arXiv arXiv

  21. [25]

    R., Zhao, S., Song, K., Xu, S., and Zhu, C

    Zhou, W., Agrawal, R., Zhang, S., Indurthi, S. R., Zhao, S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf with weighted preference optimization.arXiv preprint arXiv:2406.11827,

    Pith/arXiv arXiv

  22. [26]

    Table 5.AlpacaEval scores (win-rate) across methods. Mistral-7B-Instruct Llama-3.1-8B-Instruct OLMo-7B-SFT AlpacaEval1 AlpacaEval1 AlpacaEval1 Reference 93.17 90.00 58.15 DPO94.6691.67 79.63 N-DPO 94.28 91.2881.24 DPO+SFT 92.1593.5266.50 We report the sensitivity of reward accuracy and reward margins to the regularization strengthλin Tables 6 and

    arXiv

  23. [27]

    Margins decrease monotonically with λ for both methods, as expected: stronger regularization limits how far the model can separate chosen from rejected responses. MISTRAL-7B-INSTRUCTLLAMA-3.1-8B-INSTRUCTOLMO-7B-SFT λN-DPO N-SIMPO N-DPO N-SIMPO N-DPO N-SIMPO 0.025 2.355 7.223 3.301 6.143 2.588 0.318 0.05 2.238 7.084 3.221 5.784 2.444 0.268 0.1 2.092 6.856 ...

    1987

  24. [2022]

    Noise contrastive alignment of language models with explicit rewards.Advances in Neural Information Pro- cessing Systems, 37:117784–117812, 2024a

    Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards.Advances in Neural Information Pro- cessing Systems, 37:117784–117812, 2024a. Chen, L., Zhu, C., Soselia, D., Chen, J., Zhou, T., Goldstein, T., Huang, H., Shoeybi, M., and Catanzaro, B. Odin: Disentangled reward mitigates hacking ...

    Pith/arXiv arXiv

  25. [2023]

    Ropo: Robust preference optimization for large language models.arXiv preprint arXiv:2404.04102,

    Liang, X., Chen, C., Qiu, S., Wang, J., Wu, Y ., Fu, Z., Shi, Z., Wu, F., and Ye, J. Ropo: Robust preference optimization for large language models.arXiv preprint arXiv:2404.04102,

    Pith/arXiv arXiv

  26. [2024]

    Training a helpful and harmless assistant with rein- forcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

  27. [2025]

    Ampo: Active multi preference optimization for self-play preference selection

    Gupta, T., Madhavan, R., Zhang, X., Bansal, C., and Rajmo- han, S. Ampo: Active multi preference optimization for self-play preference selection. InForty-second Interna- tional Conference on Machine Learning, a. Gupta, T., Madhavan, R., Zhang, X., Bansal, C., and Rajmo- han, S. Refa: Reference free alignment with fine-grained length control. InSecond Conf...

    Pith/arXiv arXiv