Pith. sign in

REVIEW 6 major objections 6 minor 43 references

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Supervised fine-tuning secretly learns an implicit reward, just as DPO does; lowering its learning rate yields up to 25% relative gains before preference optimization.

desk verdict A plausible SFT-DPO unification and a promising LR heuristic, undercut by an unproven optimality-invariance assumption and single-run experiments. read the letter →

arxiv 2507.00018 v2 pith:VK6RTU7V submitted 2025-06-15 cs.LG cs.AIcs.CL

Bo Wang , Qinyuan Cheng , Runyu Peng , Rong Bao , Peiji Li , Qipeng Guo , Linyang Li , Zhiyuan Zeng
show 2 more authors
Yunhua Zhou Xipeng Qiu
This is my paper
classification cs.LGcs.AIcs.CL
keywords supervisedfine-tuningdirectpreferenceoptimizationimplicitrewardKLdivergencef-divergencepost-trainingQ-functionlearningrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the widely used Supervised Fine-Tuning (SFT) stage of LLM post-training is not merely a warm-up: it is a special case of learning an implicit reward function, the same mechanism that underlies Direct Preference Optimization (DPO). By rewriting the distribution-matching objective and choosing total variation divergence, the authors show the SFT objective reduces to maximum likelihood estimation, with the KL term to the reference model appearing only as a constant that provides no gradient. From this they conclude that conventional SFT lacks a constraint that would keep the policy close to the base model, and they show that reducing the SFT learning rate mitigates this loss and improves post-DPO instruction-following results by up to 25% relative (6% absolute win-rate increase). The paper also derives alternative SFT objectives from other f-divergences that preserve a working KL term, and it extends the logits-as-Q-function relationship from DPO to SFT. A sympathetic reader would take home a unified picture in which SFT and DPO move through the same policy-reward subspace.

What carries the argument

The load-bearing identity is the implicit-reward relation $r(x,y) = \beta \log \frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)} + V(s_0) - V(s_t)$, which connects any reward to the optimal policy under KL regularization. The paper substitutes this relation into the equivalent min-max form of distribution matching and pairs it with the total variation f-divergence, whose convex conjugate $f^*(t) = t$ makes the objective reduce to the SFT log-likelihood. This same identity is what lets the authors read LLM logits as Q-functions and derive alternative SFT targets from other f-divergences.

What would settle it

Train the same instruction-following recipe with an explicit KL penalty added to the SFT loss (so the supposedly missing term is restored) and compare to the reduced-learning-rate SFT: if the small-learning-rate gains disappear or reverse, the missing-KL diagnosis is not the operative cause. Alternatively, measure $\beta \log \frac{\pi}{\pi_{\mathrm{ref}}}$ along SFT trajectories to test whether the implicit-reward identity actually holds step-by-step.

Watch

Extended reading notes

Core claim

The central claim is that conventional SFT is a special case of implicit reward learning, identical in kind to DPO. Starting from an f-divergence between the expert's state-action distribution and the policy's, and replacing the usual entropy regularizer with KL to a reference model, the paper rewrites the objective as a min-max problem. Substituting the closed-form optimal policy and picking total variation distance (whose conjugate is the identity), the objective collapses to the standard SFT maximum-likelihood loss. Because the reference-model term in that loss is constant with respect to the policy, it drops out under differentiation; the KL constraint is absent during gradient-based SFT. The authors take this to explain why SFT can drift far from the base model, and they predict that smaller learning rates—which shrink each update step—should improve downstream preference optimization. Their experiments on instruction-following benchmarks support that prediction.

Load-bearing premise

The proof assumes the policy stays in the optimal policy-reward subspace throughout SFT optimization, so that the implicit-reward identity keeps holding at every gradient step; if ordinary gradient updates leave that subspace, the unified view and the missing-KL diagnosis lose their theoretical grounding.

Editorial extensions

If this is right

  • Sequential post-training pipelines should treat SFT learning rate as a substantive hyperparameter for downstream preference optimization, not a warm-up detail.
  • SFT objectives derived from Pearson $\chi^2$ or squared Hellinger divergences preserve a working KL term and give better post-DPO results, so the MLE objective is not the only viable SFT form.
  • If LLM logits act as Q-functions during SFT, value estimates can be read off by log-sum-exp over logits instead of Monte Carlo sampling.
  • SFT's early steps mainly move the implicit reward into a sensible range; after that, further SFT refines the reward rather than discovering it.
  • The unified view implies SFT and DPO could in principle be merged into one multi-objective loss, though this paper's interleaved and multi-objective attempts degraded performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper's claims: if SFT already learns an implicit reward, then SFT checkpoints should encode a preference ordering that can be probed directly by asking the model to rank continuations; this ordering could be used to predict DPO headroom before running DPO.
  • The small-learning-rate effect may be partly a generic regularization effect (smaller effective steps, implicit bias) rather than a specific fix for the missing KL term; comparing against weight decay or gradient clipping at the standard learning rate would separate the mechanisms.
  • The f-divergence table suggests a continuum of SFT objectives with tunable step-size modulation; one could search over the divergences' conjugate functions to find objectives that make SFT robust to learning-rate choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes a unified theoretical view in which SFT and DPO both optimize an implicit reward within an optimal policy–reward subspace. It derives, via an f-divergence min-max formulation, that the total-variation case reduces to the standard SFT/MLE objective, concludes that the KL regularizer is absent in SFT, and proposes reducing the SFT learning rate to compensate. It also derives alternative SFT objectives from Pearson chi-squared and squared Hellinger divergences, reports experiments on Llama-3-8B and Mistral-7B showing improvements on AlpacaEval2, Arena-Hard, and MT-Bench, and argues that LLM logits can be interpreted as Q-functions or value estimates during SFT.

Significance. The paper addresses a real gap: the relationship between SFT and preference optimization is usually understood empirically, and a theoretical bridge would be valuable. The empirical observation that a smaller SFT learning rate improves downstream DPO/SimPO performance is interesting and potentially practical, and the attempt to derive SFT-style losses from f-divergences is a useful direction. The paper also honestly reports failed multi-objective attempts (Appendix D) and the numerical-instability limitation for KL/JS divergences. However, the central theoretical claim is not established as written: the derivation of SFT-as-implicit-reward-learning contains a domain error in the convex conjugate, relies on an unproven invariance, and conflates sequence-level and token-level rewards. Because the 'missing KL' diagnosis and the learning-rate rationale depend on this theory, the current version cannot be accepted; with a corrected, properly scoped theory and more rigorous experiments, the empirical findings could form the basis of a solid paper.

major comments (6)
  1. [Section 3.2, Eqs. (2)-(5) and Appendix B.2] The equivalence between the TV-divergence objective and SFT ignores the domain of the convex conjugate. For f(t)=1/2|t-1|, the conjugate is f*(s)=s only on [-1/2,1/2] and +infty outside, so the reward r in Eq. (2) is bounded; substituting r=beta log(pi/pi_ref)+V(s0)-V(st) is not guaranteed to stay in that domain, and the change of variables from r to pi is not surjective. A one-step MDP example with a two-action expert already shows that the minimizer of D_TV(pi||pi_E)+beta D_KL(pi||pi_ref) differs from the MLE when the bound is active, so Eq. (5) does not follow.
  2. [Section 3.2, paragraph after Conclusion 1] The assertion that 'the relationship between the model and reward continues to satisfy Eq. (4)' during optimization is unproven. Gradient ascent on the MLE objective does not preserve the Bellman-optimality condition that characterizes pi*_r, and no invariant or projection argument is supplied; without this, SFT is not shown to search the optimal policy-reward subspace. Moreover, since Eq. (4) can be made to hold for any policy by redefining V, the conclusion that SFT 'learns an implicit reward' is close to a reparametrization unless the constrained outer minimization over r is respected.
  3. [Section 3.2, Conclusion 2] The statement that the KL term 'becomes constant with respect to the policy' is false for the original objective (1), whose term beta D_KL(pi||pi_ref) has a nonzero gradient. The actual observation is that the standard SFT loss contains no KL penalty; the claim that this follows from the distribution-matching objective depends on the invalid substitutions flagged above. The learning-rate heuristic may still be useful, but its theoretical grounding is not established.
  4. [Table 3 and Section 4.3] The empirical headline claims rest on single runs with no seeds or error bars, and Table 1's '+' entries are deltas, not uncertainty intervals. The SH-SFT row for Llama-3-Base reports Arena-Hard WR=17.3, which contradicts the text's statement that both alternative losses lead to weaker SFT checkpoints and appears to be a transcription error. Multiple seeds and corrected numbers are needed before the 25%/6% improvement claim can be assessed.
  5. [Section 3.2 and Appendix B] The derivation mixes sequence-level rewards r(x,y) (Eq. (4)) with token-level occupancy measures mu_pi and mu_E used in the f-divergence (Eq. (1)). Since the variational dual is over token-level state-action rewards, the substitution of a sequence-level implicit reward into Eqs. (25)-(27) needs an explicit measure change or telescoping argument; as written, the equality between E_muE[r(x,y)] and the token-level expectation is unjustified.
  6. [Section 4.4, Fig. 2a] The KLCC analysis does not confirm Assumption 2. Positive rank correlations between checkpoints on the same MATH-500 steps can be produced by shared prompt difficulty or trajectory length; sampling one trajectory per question gives no state distribution, and no baseline (e.g., random scores, length-based scores) is provided. The conclusion that 'logits function as a value' is therefore not supported by the presented evidence.
minor comments (6)
  1. [Abstract and Section 1] The sentence 'up to 25% relative gain and 6% absolute win rate increase in instruction following tasks' is missing a closing parenthesis; it should read '...instruction following tasks).'
  2. [Table 1] The table says models 'marked in blue,' but the table is monochrome; the '+' notation should also be explicitly defined as a delta from the baseline rather than an uncertainty interval.
  3. [Section 4.2] The text states that SimPO results show relative improvements of 20% for Llama and 25% for Mistral, but the table rows to which these percentages refer are not identified; the percentages appear to correspond to different rows (DPO vs. SimPO), so the comparison should be made explicit.
  4. [Table 4, Appendix C] Several training targets contain undefined Delta V and inconsistent signs; for example, the Reverse KL target '1 + log(log pi/pi_ref + Delta V) - V_pi(s0)' is not obviously a valid maximization objective and needs a derivation or a reference.
  5. [References] References [33] and [34] are identical (both AlpacaEval2 papers); the Arena-Hard benchmark citation appears to be missing or incorrect.
  6. [Section 5] The limitation paragraph states that KL- and JS-divergence variants were not successfully implemented; the abstract and contribution list should temper the phrase 'alternative SFT objectives from various f-divergence functions' to reflect that only Pearson chi-squared and squared Hellinger were validated.

Circularity Check

2 steps flagged · score 7.0 of 10

Central unification is definitional: implicit reward is constructed from the policy, so 'SFT searches the optimal policy-reward subspace' is true by construction; empirical results remain independent.

  1. self definitional [Section 3.2, Lemma 1 (Eq. 4), Assumption 1, Conclusion 1]
    "Lemma 1 (Relationship between Reward and Policy[17], Implicit Reward). The relationship between reward and corresponding optimal policy is : r(x, y) = β log π∗(y | x) πref(y | x) + V ∗(s0) − V ∗(st), (4) ... During the optimization process, the relationship between the model and reward continues to satisfy eq. (4), resulting in searching the optimal subspace of policy-reward."

    The 'implicit reward' in Eq. (4) is defined from the policy π (and value terms), so the assertion that the policy always satisfies Eq. (4) is not a restriction: any policy can be assigned the reward r = β log(π/πref) + V(s0) − V(st) and is then 'optimal' for that reward by construction. Substituting this definitional identity into the min-max objective (Eq. 2) and recovering the MLE term does not prove that SFT is a special case of reward discovery; it renames the policy objective as a reward objective. The same substitution is the only content of Conclusion 1, so the claimed unification is true by definition for any policy-update algorithm, not specifically SFT or DPO.

  2. self definitional [Section 3.3, Theorem 2 and Appendix B.3]
    "Theorem 2 (Intrinsic Expected Return). During the SFT process, the logits la of a language model correspond to the Q-function Q(s, a) of the learned implicit reward: la = Qˆr(s, a) + C(s) ... Equating these expressions, we have: ... li = τ β Q(s, ai) + C(s)."

    The proof equates the softmax of logits with the softmax of Q/β. Since the 'learned implicit reward' is itself defined from the policy via Eq. (4), the logits-to-Q mapping is an algebraic identity that holds for any softmax policy; it does not depend on SFT. The theoretical claim is thus a reparametrization rather than an independently derived prediction. The empirical Kendall-correlation results provide independent evidence about value-like rankings, but they do not make the definitional identification a derived result.

full rationale

The paper's circularity is not in its citation practice: Lemma 1 is attributed to [17] (Rafailov et al.), an external result, and the f-divergence machinery is standard [20]; no self-citation is load-bearing. The problem is that the paper's own Eq. (4) defines the implicit reward in terms of the policy, and the derivation of Eq. (5) and Conclusion 1 substitutes this definition into the min-max objective while asserting that the policy 'continues to satisfy eq. (4)'. Because the relation can be made to hold for any policy by choosing r = β log(π/πref) + V(s0) − V(st), the conclusion that SFT searches the optimal policy-reward subspace is a reparametrization rather than an independently derived fact. The same definitional substitution underlies the alternative f-divergence objectives in Table 4 and the logits-as-Q theorem in Section 3.3. The learning-rate experiments, alternative f-divergence experiments, and value-correlation measurements are empirical and self-contained; they do not inherit the circularity, which is why the score is high but not maximal.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The central derivation is a reparametrization: it defines the reward through the policy via the DPO-style identity, so the conclusion that SFT tracks an implicit reward is partly tautological. The load-bearing assumptions are Assumption 1 and the unproven claim that eq. (4) holds at every optimization step. The empirical LR heuristic does not depend on these assumptions and is the main independent contribution.

free parameters (3)
  • SFT learning rate (5e-6 Llama-3-8B, 1e-6 Mistral-7B) = 5e-6 / 1e-6
    Chosen by the authors for improved post-DPO results; no sweep or multiple seeds reported. Central to the empirical claim.
  • DPO beta = 0.01
    Standard hyperparameter inherited from SimPO; held fixed across experiments.
  • f-divergence selection (TV, Pearson chi^2, Squared Hellinger) = Not applicable
    Selected for numerical stability; the choice determines the derived SFT loss family and is a design decision rather than a fit.
assumptions (7)
  • standard math Convex conjugate / f-divergence duality
    Invoked in Theorem 1 and Appendix B.1 to convert f-divergence minimization into a min-max problem over reward functions.
  • standard math Softmax optimal policy form under KL regularization (Lemma 2 from [17])
    Used to derive the implicit reward identity in eq. (4), linking reward, policy, and value functions.
  • ad hoc to paper Assumption 1: initial reward simplification
    Assumes the initial reward can be treated as Vπ(s0) with Vπ(st)=0, and that the initial policy is optimal for the initial reward. Load-bearing for the substitution in eq. (5).
  • ad hoc to paper Eq. (4) holds throughout optimization
    The paper asserts that the policy stays in the optimal policy-reward subspace during gradient updates; this is unproven and essential for the MLE reduction.
  • domain assumption Terminal state value is constant (zero)
    Needed to drop Vπ(st) from the objective and conclude that only the log-policy term remains.
  • domain assumption Token-level MDP with deterministic transitions
    Defines the state-action distribution and the whole theoretical framework in Section 3.
  • ad hoc to paper Assumption 2: value-dominance assumption
    Used to justify using log-sum-exp of logits as a state-value estimate in Section 4.4; not directly verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections." pith.science (2026). https://pith.science/paper/VK6RTU7V

@misc{pith2026250700018,
  author       = {Pith},
  title        = {Pith review of: Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VK6RTU7V}},
  note         = {Machine review of arXiv:2507.00018}
}
read the original abstract

Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to \textbf{25\%} relative gain and \textbf{6\%} absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.

Figures

Figures reproduced from arXiv: 2507.00018 by the authors.

Figure 1
Figure 1. Schematic diagram: SFT and DPO are optimizing implicit rewards in the optimal policy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: We obtain rankings of different models on identical step-wise instruction￾response pairs and calculate KLCC to measure agreement between these rankings. Middle: Lower values of DPO Loss indicate better alignment between implicit reward and environment reward. And the corresponding AlpacaEval2 result. Right: Training loss for objectives derived from: total variation, Pearson χ 2 , and squared Hellinger. 2b. We … view at source ↗
Figure 3
Figure 3. The trend of V (s0) using the logits of the first response token. Performance on AlpacaEval 2 for post-DPO on SFT checkpoints at corresponding training steps. SFT in our theory is attempting to bring the implicit reward into an appropriate range for fine-grained modifications. Assump￾tion 1 assumes that at the starting point of the implicit reward search process, the model represents the optimal policy under some un… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training Metric Visualization for Multi-object Learning [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 21 canonical work pages

  1. [1]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human fee...

    work page 2022

  2. [2]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany...

    arXiv 2024

  3. [3]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei...

    arXiv 2024

  4. [4]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    arXiv 2024

  5. [5]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825,

    arXiv

  6. [6]

    LIMO: less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. LIMO: less is more for reasoning. CoRR, abs/2502.03387, 2025. 1

    arXiv 2025

  7. [7]

    LIMA: less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Pr...

    work page 2023

  8. [8]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

Show all 43 references
  1. [9]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. 1

    2024 arXiv

  2. [10]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  3. [11]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. CoRR, abs/1706.03741, 2017. 1

    2017 arXiv

  4. [12]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. CoRR, abs/2305.18290, 2023. 1, 2, 3, 7

    2023 arXiv

  5. [13]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net,

    2024

  6. [14]

    Sutherland

    Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning.CoRR, abs/2407.10490,

    arXiv

  7. [15]

    Iq-learn: Inverse soft-q learning for imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. CoRR, abs/2106.12142, 2021. 2, 3, 4, 15

    2021 arXiv

  8. [16]

    Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Bechtle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, and Martin A

    Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy H. Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Bechtle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, and Martin A. Riedmiller. Imi...

    2024

  9. [17]

    From r to q*: Your language model is secretly a q-function

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly a q-function. CoRR, abs/2404.12358, 2024. 2, 3, 4, 5, 15, 16

    2024 arXiv

  10. [18]

    Ng and Stuart Russell

    Andrew Y . Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Pat Langley, editor, Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, pages 663–670. Morg...

    2000

  11. [19]

    Entropic distribution matching in supervised fine-tuning of llms: Less overfitting and better diversity

    Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi-Quan Luo. Entropic distribution matching in supervised fine-tuning of llms: Less overfitting and better diversity. CoRR, abs/2408.16673, 2024. 3, 15

    2024 arXiv

  12. [20]

    f-gan: Training generative neural samplers using variational divergence minimization

    Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. CoRR, abs/1606.00709, 2016. 3, 5, 15

    2016 arXiv

  13. [21]

    Sequencematch: Imitation learning for autoregressive sequence modelling with backtracking

    Chris Cundy and Stefano Ermon. Sequencematch: Imitation learning for autoregressive sequence modelling with backtracking. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 3, 15

    2024

  14. [22]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. 3

    2017 arXiv

  15. [23]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International ...

    2024

  16. [24]

    Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. 3, 7

    2024

  17. [25]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Sys...

    2024

  18. [26]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand a...

    2024

  19. [27]

    Dragan, and Sergey Levine

    Joey Hong, Anca D. Dragan, and Sergey Levine. Q-SFT: q-learning for language models via supervised fine-tuning. CoRR, abs/2411.05193, 2024. 3

    2024 arXiv

  20. [28]

    Andrew Bagnell

    Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. CoRR, abs/2503.01067, 2025. 3, 4 12

    2025

  21. [29]

    Ziebart, Andrew L

    Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla P. Gomes, editors, Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July...

    2008

  22. [30]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on ...

    2023

  23. [31]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.01377, 2023. 6

    2023 arXiv

  24. [32]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework

    Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143,

    arXiv

  25. [34]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.04475, 2024. 6

    2024 arXiv

  26. [35]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Glober...

    2023

  27. [36]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  28. [37]

    RRHF: rank responses to align language models with human feedback

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: rank responses to align language models with human feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information...

    2023

  29. [38]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305.10425, 2023. 7

    2023 arXiv

  30. [39]

    Contrastive preference optimization: Pushing the bound- aries of LLM performance in machine translation

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the bound- aries of LLM performance in machine translation. In Forty-first International Conference on Machine Lear...

    2024

  31. [40]

    ORPO: monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO: monolithic preference optimization without reference model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL...

    2024

  32. [41]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment...

    2023

  33. [42]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. 8

    2023 arXiv

  34. [43]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors,Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing...

    2016

  35. [44]

    Zemel, and Shixiang Gu

    Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, and Shixiang Gu. A divergence mini- mization perspective on imitation learning methods. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, Octo...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.