InfoPO: On Mutual Information Maximization for Large Language Model Alignment

Julian Katz-Samuels; Marc Versage; Qingjun Cui; Sujay Sanghavi; Teng Xiao; Tian Wang; Trishul Chilimbi; Zhen Ge

REVIEW 3 major objections 5 minor 63 references

InfoPO: On Mutual Information Maximization for Large Language Model Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InfoPO aligns LLMs by maximizing mutual information, not Bradley-Terry odds

desk verdict InfoPO's rejected-response term is constant under the paper's own sampling model, so the loss reduces to SFT on chosen responses and the reverse-KL theorem is unproven. read the letter →

arxiv 2505.08507 v1 pith:JIHRMEGI submitted 2025-05-13 cs.LG

Teng Xiao , Zhen Ge , Sujay Sanghavi , Tian Wang , Julian Katz-Samuels , Marc Versage , Qingjun Cui , Trishul Chilimbi This is my paper · ORCID

classification cs.LG

keywords InfoPOpreferenceoptimizationmutualinformationNWJestimatorBradley-TerrymodelLLMalignmentreasoningbenchmarksreverseKLdivergence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InfoPO is a new preference fine-tuning method for large language models that drops the Bradley-Terry assumption used by DPO and SimPO. Instead, it directly maximizes the conditional mutual information between response and preference label given the prompt, using the NWJ estimator. The resulting loss keeps the chosen response likely while applying a gentle penalty to rejected responses, so the model unlearns bad answers without driving down the probability of good ones. The paper argues this prevents the known failure mode where DPO-style training lowers chosen-response likelihood and hurts math and coding performance. It reports that InfoPO consistently outperforms DPO, SimPO, and related baselines on reasoning, instruction-following, chat, and summarization benchmarks.

What carries the argument

The load-bearing object is the NWJ estimator of conditional mutual information, $I(Y;C|X) \ge \mathbb{E}[f] - \mathbb{E}[e^f] + 1$, used with the critic $f_\phi(x,c) = \beta \log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$. With that critic, InfoNCE becomes exactly the DPO loss, and NWJ becomes $L_{\mathrm{InfoPO}} = -\log \pi_\theta(y_w|x) + \pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x)$. This identity is what lets the paper move from a contrastive, margin-based objective to a direct keep-chosen, gently-demote-rejected objective. Theorem 4.1 then claims that minimizing this loss is equivalent to minimizing the reverse KL divergence $D_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{chosen}})$, the mode-seeking behavior the paper argues alignment should have. The machinery simultaneously explains DPO as a special case and supplies the replacement objective.

What would settle it

Train InfoPO on a preference dataset and measure the empirical average of $\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x)$ over the rejected responses at each checkpoint. If this average stays at $1$ and the gradient contribution of that term is numerically zero while the loss trajectory matches training on $-\log \pi_\theta(y_w|x)$ alone, then the rejected-response mechanism is not doing the unlearning work the paper claims.

Watch

Extended reading notes

Core claim

The central claim is that direct preference optimization can be reframed as mutual-information maximization. DPO's loss is exactly an InfoNCE estimator of the conditional mutual information between response and preference label given the prompt when the critic is $\beta \log(\pi_\theta/\pi_{\mathrm{ref}})$. Swapping in the NWJ estimator gives the InfoPO objective $L_{\mathrm{InfoPO}} = -\log \pi_\theta(y_w|x) + \pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x)$. The first term keeps the chosen response probable; the second produces a smaller, more conservative gradient on rejected responses than DPO's reciprocal-probability gradient. The paper proves that minimizing this loss encourages mode-seeking behavior via reverse KL divergence between the model and the chosen-response distribution, and shows empirically that this preserves chosen likelihoods while improving reasoning, coding, and chat benchmarks.

Load-bearing premise

The load-bearing premise is that the rejected-response penalty in the InfoPO loss actually pushes the model during training; if rejected responses are effectively sampled from the reference distribution, the average penalty is constant, its gradient is zero, and the method reduces to supervised fine-tuning on chosen responses.

Editorial extensions

If this is right

If InfoPO is correct, preference alignment can be performed without assuming the Bradley-Terry model, so the chosen-response likelihood decline is addressed by construction rather than by an added regularization term.
The reverse-KL characterization implies InfoPO concentrates probability mass on a focused set of high-reward responses, which the paper argues is the right inductive bias for reasoning and task-specific generation.
Because the loss contains an explicit chosen-response log-likelihood term, InfoPO should require fewer manually tuned regularization hyperparameters than methods that add NLL penalties on top of a contrastive loss.
The reported gains on GSM8K, MATH, HumanEval, and MBPP suggest that preference fine-tuning with InfoPO preserves pretrained reasoning abilities while still improving instruction following and chat quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct comparison the paper leaves implicit: InfoPO trained on preference pairs should be compared against plain supervised fine-tuning on the chosen responses alone; if the rejected-response term has zero expected gradient, the two should match.
The mutual-information framing suggests a family of alignment losses indexed by MI estimator; interpolating between InfoNCE and NWJ could trade bias for variance in a controlled way, though the paper does not explore this.
If reverse-KL mode-seeking is what the loss implements, InfoPO should reduce the entropy of generated responses relative to the reference model; measuring generation diversity across training would test this prediction.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Quick take on 2505.08507: the paper's central claim does not survive contact with its own equations. InfoPO's loss in Eq. (13) is derived from NWJ, but with the stated sampling model (yl ~ pi_ref), the rejected term is constant in expectation, so the objective is just SFT on chosen responses. Theorem 4.1's proof commits the same error: Eq. (23) is forward cross-entropy plus a constant, not reverse KL.

What is actually new: the observation that DPO can be read as InfoNCE with a particular critic is a legitimate reframing, and the paper states it cleanly. The loss itself is simple enough to reimplement, and the benchmark numbers are at least internally consistent. If the theoretical problems were fixed, the empirical recipe could be worth a follow-up.

Where it falls apart: first, the NWJ derivation drops beta and the reference term; Eq. (13) only matches Eq. (12) at beta=1, and beta is never set to 1 in the experiments—Appendix B.2 tunes it in {0.5,1.0,2.0} for a loss that contains no beta. Second, under Section 4.1's sampling model, E_{yl~pi_ref}[pi_theta(yl)/pi_ref(yl)] = 1, so the rejected term has zero expected gradient. At initialization, its actual gradient increases rejected likelihood, which is the opposite of the claimed conservative unlearning. Third, the proof of Theorem 4.1 confuses forward and reverse KL; Eq. (23) is not D_KL(pi_theta || pi_chosen). Also, the critic in Eq. (9) does not actually depend on c, which makes the "preference" label in the InfoNCE bound more a formality than a modeling choice.

The experiments lack an SFT-on-chosen baseline, which is essential once the reduction to SFT is understood. The citation pattern is okay—DPO, SimPO, NCE, NWJ all cited—and the limitations section is honest, except it does not mention any of these gaps.

Who it is for: anyone working on preference optimization who wants to see a concrete example of a mutual-information estimator argument gone wrong. It is not a usable method as written. I would send it to review only because the flaw is subtle enough to deserve a referee's time; after review, the theory needs to be redone or the paper reframed as an empirical loss with an SFT baseline. I would not cite it in its current form.

Best.

Referee Report

3 major / 5 minor

Summary. The paper proposes InfoPO, a preference fine-tuning method for LLMs based on maximizing conditional mutual information I(Y;C|X) with the NWJ estimator, claiming to avoid the Bradley-Terry assumption. The method is derived as L_InfoPO = -log πθ(yw|x) + πθ(yl|x)/πref(yl|x), with a stated advantage that it prevents the chosen-response likelihood from decreasing. The authors reinterpret DPO as InfoNCE-based MI maximization, provide a gradient analysis to motivate conservative unlearning of rejected responses, prove a theorem that InfoPO minimizes reverse KL to the chosen-response distribution (Theorem 4.1), and report experiments on reasoning, chat, and summarization benchmarks where InfoPO outperforms DPO and SimPO.

Significance. If the central derivation and theorem were correct, the MI-maximization perspective on preference learning could be a useful conceptual contribution, and the empirical gains on reasoning benchmarks would be of practical interest. The paper also ships a relatively simple training objective and reports evaluations across multiple models and tasks. However, the significance is entirely conditional on the validity of the derivation of Eq. (13) and the proof of Theorem 4.1, which are the load-bearing components of the paper. As detailed in the major comments, those components fail under the paper's own stated sampling model, so the proposed method does not actually perform preference learning in expectation.

major comments (3)

[§4.1 and §4.3, Eq. (13)] Under the sampling model stated in §4.1 ('we sample x~p(x), (yw,c)~π(y,c=1|x), and (yl,c)~πref(y|x)p(c|x)'), the second term of L_InfoPO has zero expected gradient. For any normalized πθ, E_{yl~πref}[πθ(yl|x)/πref(yl|x)] = 1, so the population objective is E[-log πθ(yw|x)] plus a constant. The InfoPO objective therefore reduces to supervised fine-tuning on the chosen responses, with no preference contrast and no 'conservative unlearning' of rejected responses. Any finite-sample effect of the second term is noise, not a designed preference signal. This invalidates the paper's central claim of a BT-free preference learning method.
[§4.3, Eqs. (9), (12), (13)] The derivation of Eq. (13) from the NWJ bound (Eq. (12)) is algebraically inconsistent unless β=1 and several constant terms are dropped. Substituting the critic f = β log(πθ/πref) from Eq. (9) into the NWJ objective yields -β log(πθ(yw|x)/πref(yw|x)) + (πθ(yl|x)/πref(yl|x))^β - 1, which contains β and the reference-policy term for the chosen response. Eq. (13) contains neither β nor log πref(yw). The paper never states that β is fixed to 1, and Appendix B.2 describes searching β in [0.5, 1.0, 2.0] for 'our InfoPO' although the loss in Eq. (13) has no β parameter. The claimed objective does not follow from the stated estimator.
[Appendix A, Eq. (23) and Theorem 4.1] The proof of Theorem 4.1 is invalid. Eq. (23) is E_{π_chosen}[-log πθ] + E_{π_ref}[πθ/π_ref]. The second term equals 1 identically, so the expression is forward cross-entropy from the chosen distribution to πθ plus a constant. The reverse KL divergence D_KL(πθ || π_chosen) is E_{πθ}[log(πθ/π_chosen)], which is a different functional; the 'change of variables' from Eq. (18) to Eq. (23) is not derived and is not valid. Consequently Theorem 4.1's claim that InfoPO minimizes reverse KL is unproven, and the mode-seeking interpretation lacks support.

minor comments (5)

[§4.1, Eq. (9)] The critic fϕ(x,c) is written as independent of c; clarify whether the preference label c has any role in the critic or whether the contrast is entirely carried by the choice of yw versus yl in the expectation.
[§4.3, Eq. (12)] The constant '+1' in the NWJ estimator disappears without comment in Eq. (13). While constants do not affect the gradient, the paper should state that this is an intentional omission for clarity.
[Appendix B.2] The hyperparameter description 'For SimPO and our InfoPO, the β in SimPO was selected through a search within the range of [0.5, 1.0, 2.0]' is ambiguous. Since Eq. (13) contains no β or length-normalization parameter, it should be stated explicitly which hyperparameters were tuned for InfoPO and how the length normalization mentioned in the same paragraph enters the loss.
[Figures 1 and 2] The quantity 'Margins' is not defined in the text or captions. Specify whether it is the difference in average log-likelihoods, the reward margin, or something else, and report the standard errors or number of runs, since the curves appear to be based on a single training run.
[Table 1] On Mistral-7B, CPO achieves a GSM8K score of 33.06, which is higher than InfoPO's 32.07; the text's claim that InfoPO consistently outperforms all baselines is therefore overstated. Please qualify the claim or perform significance testing across multiple seeds.

Circularity Check

2 steps flagged · score 8.0 of 10

Under the paper's own sampling model, the rejected-response term in Eq. (13) has expectation 1 for every policy, so InfoPO reduces to SFT on chosen responses; Theorem 4.1's proof labels the same constant as reverse KL.

self definitional [Section 4.1 (sampling model, Eq. 8 context) and Section 4.3 (Eq. 13)]
"we sample x∼ p(x), (yw, c)∼ π(y, c = 1| x), and (yl, c)∼ πref (y| x)p(c| x). ... By using the preference datasets and the parameterized critic in Equation (9), we have the following InfoPO objective on preference pairs: LInfoPO(θ; x, yw, yl) = − logπθ(yw|x) +πθ(yl|x)/πref (yl|x)."

With y_l drawn from π_ref, the second term is an importance-weighting identity: E_{y_l∼π_ref}[π_θ(y_l|x)/π_ref(y_l|x)] = ∑_y π_ref(y|x) π_θ(y|x)/π_ref(y|x) = ∑_y π_θ(y|x) = 1. Hence the expected InfoPO loss is E[−log π_θ(y_w|x)] + 1, which is exactly supervised fine-tuning on chosen responses, and the gradient of the rejected term is zero in expectation. The NWJ estimator's exp term is forced to 1 by the critic choice f = β log(π_θ/π_ref) in Eq. (9); the claimed 'conservative unlearning' of rejected responses is therefore zero by construction. Any finite-sample gradient from the second term is noise, not a designed preference signal.
self definitional [Appendix A, proof of Theorem 4.1, Eq. (23)]
"Analogous to the approach used for density ratio estimation and using a change of variables, we can formalize the reverse KL objective as follows: Eπchosen [− logπθ(y|x)] + Eπref [ πθ(yl|x) πref (yl|x) ] , (23)"

The reverse KL D_KL(π_θ || π_chosen) = E_{π_θ}[log π_θ − log π_chosen], but Eq. (23) is E_{π_chosen}[−log π_θ] + E_{π_ref}[π_θ/π_ref]. The second expectation is identically 1 for any normalized π_θ, so Eq. (23) is forward cross-entropy plus a constant, not reverse KL. The theorem's conclusion is not derived; it is imported by labeling Eq. (23) 'the reverse KL objective.' Minimizing InfoPO is therefore claimed to be mode-seeking reverse-KL minimization only because the proof defines it to be so.

full rationale

The circularity is not a matter of the empirical benchmarks; the evaluations against GSM8K, MATH, AlpacaEval, etc. are external and could be valid even if the derivation failed. The circularity is in the derivation chain. Section 4.1 fixes the sampling model y_l ∼ π_ref(y|x), and Eq. (9) fixes the NWJ critic to f = β log(π_θ/π_ref). Substituting that critic into the NWJ bound (Eq. 12) makes the exp term (π_θ(y_l)/π_ref(y_l))^β. Since y_l is averaged over π_ref, this term is exactly 1 for any normalized π_θ (at β=1). Thus the expected InfoPO objective is E[−log π_θ(y_w|x)] + 1, which is SFT on chosen responses; the rejected-response gradient is zero in expectation. The paper's central claim that InfoPO 'decreases the likelihood of the rejected response' or performs 'conservative unlearning' is therefore not a property of the derived objective—it is an artifact of finite-sample noise. The same reduction appears in the proof of Theorem 4.1: Eq. (23) calls E_{π_chosen}[−log π_θ] + E_{π_ref}[π_θ/π_ref] the 'reverse KL objective,' but the second summand is identically 1 and the first is forward cross-entropy. The theorem's conclusion is assumed by labeling this expression 'reverse KL.' No parameter is fitted to a subset and then predicted (the method is parameter-free apart from hyperparameters), and no load-bearing self-citation is involved, so the score reflects the definitional reduction of the central objective rather than citation practices.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on three load-bearing choices: the critic parameterization (Eq 9) that makes the NWJ negative term a constant, the sampling assumption yl ~ pi_ref that cancels the preference signal in expectation, and the unproved change of variables in Appendix A. beta is tuned in experiments but absent from the loss, which is unexplained. No new entities are introduced.

free parameters (2)

beta (critic temperature) = search range [0.5, 1.0, 2.0]
Appears in the critic (Eq 9) and in the claimed NWJ derivation, but not in the InfoPO loss (Eq 13). Appendix B.2 tunes beta for InfoPO without a defined role, indicating the derivation is not the objective actually trained.
response likelihood normalization = average per-token log probability
Appendix B.2 normalizes likelihoods by sequence length similar to SimPO; this rescaling determines the relative weight of the NLL term and the ratio term, but the exact form for the ratio term is not specified.

assumptions (4)

domain assumption The rejected response yl is sampled from the reference policy pi_ref for the InfoPO objective
Equation (8) and Section 4.3 draw negatives from pi_ref; this makes E[pi_theta(yl)/pi_ref(yl)] = 1 and cancels the preference signal in expectation. Offline datasets sample rejected responses from other models, so the derivation does not match the experiment.
ad hoc to paper The critic f = beta log(pi_theta/pi_ref) is a valid parameterization for the NWJ estimator
This choice is what forces the NWJ negative term to a constant; the mutual information maximization framing encodes the answer rather than deriving it.
ad hoc to paper The change of variables from Equation (18) to Equation (23) is valid
Appendix A asserts this step without proof; the resulting Equation (23) is forward cross-entropy plus a constant, not the reverse KL claimed in Theorem 4.1.
standard math Z(x) = 1 in Equation (21)
Given f* = log(pi_chosen/pi_ref), the partition function is Sum_y pi_chosen(y|x) = 1; this step itself is fine, but the surrounding proof does not go through.

how reviews work

0 comments

Cite this review

Pith. "Pith review of InfoPO: On Mutual Information Maximization for Large Language Model Alignment." pith.science (2026). https://pith.science/paper/JIHRMEGI

@misc{pith2026250508507,
  author       = {Pith},
  title        = {Pith review of: InfoPO: On Mutual Information Maximization for Large Language Model Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIHRMEGI}},
  note         = {Machine review of arXiv:2505.08507}
}

read the original abstract

We study the post-training of large language models (LLMs) with human preference data. Recently, direct preference optimization and its variants have shown considerable promise in aligning language models, eliminating the need for reward models and online sampling. Despite these benefits, these methods rely on explicit assumptions about the Bradley-Terry (BT) model, which makes them prone to overfitting and results in suboptimal performance, particularly on reasoning-heavy tasks. To address these challenges, we propose a principled preference fine-tuning algorithm called InfoPO, which effectively and efficiently aligns large language models using preference data. InfoPO eliminates the reliance on the BT model and prevents the likelihood of the chosen response from decreasing. Extensive experiments confirm that InfoPO consistently outperforms established baselines on widely used open benchmarks, particularly in reasoning tasks.

Figures

Figures reproduced from arXiv: 2505.08507 by the authors.

**Figure 2.** The training dynamics of average likelihood of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** The performance comparison on coding tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 19 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. 2019. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9163--9171

work page 2019
[4]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021
[5]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

2024
[6]

Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32

work page 2019
[7]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

arXiv 2022
[8]

David Barber and Felix Agakov. 2004. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, page 201

work page 2004

Show all 63 references

[9]

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual information neural estimation. In International conference on machine learning, pages 531--540

2018
[10]

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In ...

2023
[11]

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pages 324--345

1952
[12]

Huayu Chen, Guande He, Hang Su, and Jun Zhu. 2024. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369

2024 arXiv
[13]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

2021 arXiv
[14]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597--1607. PMLR

2020
[15]

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

2017
[16]

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377

2023 arXiv
[17]

Monroe D Donsker and SR Srinivasa Varadhan. 1983. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on pure and applied mathematics, pages 183--212

1983
[18]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[19]

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation matters in deep policy gradients: A case study on ppo and trpo. In International Conference on Learning Representations

2020
[20]

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

2024 arXiv
[21]

Benjamin Eysenbach, Julian Ibarz, Abhishek Gupta, and Sergey Levine. 2019. Diversity is all you need: Learning skills without a reward function. In 7th International Conference on Learning Representations, ICLR 2019

2019
[22]

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

2023 doi
[23]

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792

2024 arXiv
[24]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[25]

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 b . Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561

2023 arXiv
[26]

Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

2014 arXiv
[27]

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. 2023. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations

2023
[28]

Martin Q Ma, Yao-Hung Hubert Tsai, Paul Pu Liang, Han Zhao, Kun Zhang, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2021. Conditional contrastive learning for improving fairness in self-supervised learning. arXiv preprint arXiv:2106.02866

2021 arXiv
[29]

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

2024 arXiv
[30]

R \'e mi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. 2023. Nash learning from human feedback. arXiv preprint arXiv:2312.00886

2023 arXiv
[31]

XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. 2010. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, pages 5847--5861

2010
[32]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

2018 arXiv
[33]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, p...

2022
[34]

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228

2024 arXiv
[35]

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733

2024 arXiv
[36]

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159

2024 arXiv
[37]

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171--5180

2019
[38]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[39]

Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715

2024 arXiv
[40]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

2017 arXiv
[41]

Jiaming Song and Stefano Ermon. 2020. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations

2020
[42]

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

2020
[43]

Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. 2024. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367

2024 arXiv
[44]

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, R \'e mi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo \'A vila Pires, and Bilal Piot. 2024. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint a...

2024 arXiv
[45]

Yao Hung Hubert Tsai, Tianqin Li, Martin Q Ma, Han Zhao, Kun Zhang, Louis Philippe Morency, and Ruslan Salakhutdinov. 2022. Conditional contrastive learning with kernel. In ICLR

2022
[46]

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. 2019. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625

2019 arXiv
[47]

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944

2023 arXiv
[48]

Michael V \"o lske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59--63

2017
[49]

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024. Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations

2024
[50]

Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant Honavar. 2024 a . How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. In Proceedings of the 2024 Conference on Empirical Methods in Natural Langu...

2024
[51]

Teng Xiao and Donglin Wang. 2021. A general offline reinforcement learning framework for interactive recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4512--4520

2021
[52]

Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. 2025. Simper: A minimalist approach to preference alignment without hyperparameters. arXiv preprint arXiv:2502.00883

2025 arXiv
[53]

Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. 2024 b . Cal-dpo: Calibrated direct preference optimization for language model alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[54]

Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. 2023. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456

2023 arXiv
[55]

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024 a . Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. ICML

2024
[56]

Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024 b . Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719

2024 arXiv
[57]

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. 2024 a . Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078

2024 arXiv
[58]

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024 b . Self-rewarding language models. arXiv preprint arXiv:2401.10020

2024 arXiv
[59]

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425

2023 arXiv
[60]

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. 2023. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964

2023 arXiv
[61]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[62]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[63]

accepted

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. 2019. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9163--9171

work page 2019

[4] [4]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021

[5] [5]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

2024

[6] [6]

Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32

work page 2019

[7] [7]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

arXiv 2022

[8] [8]

David Barber and Felix Agakov. 2004. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, page 201

work page 2004