Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

This paper shows self-play LLM fine-tuning is an adversarial imitation game, and proves that a chi-squared-regularized version converges with bounded rewards.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:41 UTC pith:6RVE6AIC

load-bearing objection Useful AIL reinterpretation of self-play LLM finetuning, but the advertised convergence guarantee for SPIF only holds for the idealized game, not the actual objective. the 5 major comments →

arxiv 2602.01357 v2 pith:6RVE6AIC submitted 2026-02-01 cs.LG

Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

Shangzhe Li , Xuchao Zhang , Chetan Bansal , Weitong Zhang This is my paper
classification cs.LG
keywords self-play fine-tuningadversarial imitation learninglarge language modelschi-squared divergenceduality gapmirror descentpreference alignmentreward regularization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that self-play post-training of large language models—where a model is trained against its own previous snapshots using only expert demonstrations—is a special case of adversarial imitation learning. Formulating the procedure as a min-max game between the model and a regularized implicit reward player, it proves that the self-play process converges to its equilibrium with a duality gap of O((D+B)Rmax^2/√K). The proof identifies the regularizer as the deciding factor: with a mixed chi-squared penalty the reward stays bounded within [-1/c, 1/c], whereas earlier self-play objectives allow unbounded rewards and unstable training. Guided by this, the paper proposes SPIF, a single-stage least-squares self-play objective, and reports that it outperforms the SFT baseline and SPIN across most evaluation settings.

Core claim

The central claim is that self-play fine-tuning is exactly an adversarial imitation game: at each round a reward player tries to separate expert responses from the current model's responses while the policy player tries to match the expert distribution. Theorem 4.4 states that when the reward is bounded in [-Rmax, Rmax], the policy is updated by KL-regularized mirror descent, and the reward by Bregman-regularized ascent, the average duality gap after K iterations is O((D+B)Rmax^2/√K), so the process converges to the Nash equilibrium of the game. With a mixed Pearson chi-squared regularizer, Proposition 4.1 bounds the optimal reward in [-1/c, 1/c], which tightens the bound and yields more sta

What carries the argument

The central object is the two-player game (4.1) between a policy player π (the language model) and a reward player r reparameterized through the model's own log-likelihood ratio β log(π/π_k) minus a partition function. The proof mechanism is the one-step descent lemma (Lemma A.1) combined with online mirror descent on both players, which telescopes KL divergences and produces the O(1/√K) duality gap. The mixed chi-squared regularizer ψ(r) = (c/2)E_π⋆[r^2] + (c/2)E_π[r^2] is what keeps the reward bounded and the training stable.

Load-bearing premise

The convergence guarantee rests on two unproven premises: the expert policy and true reward are realizable in the chosen function classes, and dropping the partition function from the reparameterized reward leaves the analyzed game unchanged—the second is asserted in Proposition 5.1, not demonstrated.

What would settle it

Run Algorithm 1 on a small synthetic contextual bandit with a known expert policy π*, and compare the empirical duality gap for the original reward (with β log Z(x)) versus the mapped reward (5.1) that omits it; if the two curves diverge by more than the O(1/√K) prediction, Proposition 5.1 is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Self-play fine-tuning converges to the equilibrium of the adversarial imitation game at an O(1/√K) rate, so the duality gap after K iterations is predictably small rather than ad hoc.
  • With mixed chi-squared regularization the implicit reward is bounded in [-1/c, 1/c], explaining why SPIF's gradient norms stay stable while SPIN's explode from roughly 10^4 to 10^-4.
  • The same adversarial-imitation formulation covers preference-based self-play methods: SPPO and INPO emerge as chi-squared-regularized AIL with the preference oracle as the expert (Propositions 7.1 and 7.3).
  • Because self-play is imitation, model capability is capped by the expert distribution; iterative self-play alone cannot yield unbounded improvement (Sec. 7.2).
  • An optimistic variant can strengthen the rate to O(1/K) when the preference oracle is known, matching recent no-regret preference-alignment results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the unification holds, designing a self-play algorithm reduces to choosing a statistical divergence; stability and reward-scale become predictable from the regularizer instead of requiring trial-and-error tuning.
  • Inference: The paper proves the two-stage game but the practical SPIF objective drops the partition function β log Z(x) via Proposition 5.1, which is asserted rather than demonstrated; a direct comparison between SPIF and a version retaining the partition function would test whether the surrogate changes the game.
  • Inference: The capacity-ceiling claim implies a testable prediction: on held-out benchmarks, self-play gains should plateau at roughly the expert data quality; measuring that plateau could distinguish genuine self-improvement from imitation of the SFT target.
  • Inference: The faster O(1/K) rate for optimistic variants is conditional on knowing preference probabilities; replacing the unknown oracle with an estimated one may or may not preserve the improved rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper proposes a unifying adversarial-imitation-learning (AIL) view of self-play finetuning for LLMs. The authors formulate self-play as a min-max game between a policy player π and a regularized reward player r, with a link function σ and convex regularizer ψ. They show that SPIN and linear SPIN correspond to TV/KL distance minimization, and that a mixed χ2 regularizer yields bounded rewards (Prop 4.1). They state a convergence theorem (Theorem 4.4) bounding the duality gap by O((D+B)Rmax^2/√K) for a general two-stage algorithm (Algorithm 1). For implementation, they use the closed form of the KL-regularized policy update to reparameterize the reward as Δr=β log(π/π_k) (Eq. 5.1), assert the bound remains valid (Prop 5.1), and derive a single-stage least-squares objective SPIF (Eq. 5.2). Experiments on Qwen3-4B and Mistral-7B over three self-play iterations report small but consistent improvements over SPIN and SFT, plus reward-magnitude and gradient-norm stability plots.

Significance. The paper's taxonomy (Table 1) is genuinely useful: it connects SPIN/SPPO/INPO to AIL regularizers and distance measures, and Theorem 4.4 is a standard OMD/OGA-type bound that, if properly applied, would be a meaningful theoretical contribution to LLM self-play. The proposed χ2 regularizer and the empirical stability analysis are also interesting. However, the central claimed guarantee for the actual SPIF algorithm is not established: the proof of Prop 5.1 is a placeholder, and the equivalence in Prop 5.2 is not rigorous. The paper is honest in providing implementation details and ablations, but those do not cure the theoretical gap. If the authors can either prove the equivalence under explicit assumptions or temper the theoretical claims, the contribution would be solid. As it stands, the significance is moderate and conditional.

major comments (5)
  1. [§5, Proposition 5.1] The proof of Prop 5.1 is not a proof. Theorem 4.4 is for Algorithm 1, where the reward player optimizes over a fixed, bounded reward class R with a Bregman regularizer D_f(r,r_{k-1}) on the actual reward r (Algorithm 1 line 4, Eq. 4.2). Dropping logZ(x) changes the decision variable to Δr=β log(π/π_k), whose feasible set depends on π_k and on β, and it changes both the Bregman term and the ψ regularizer. An x-only shift cancels in the bilinear payoff but does not cancel in D_f or ψ. Thus the OMD/OGA argument for a fixed reward class does not transfer. The claim that the Theorem 4.4 bound 'still holds' is asserted, not derived.
  2. [§5, Proposition 5.2 and Eq. (5.2)] The equivalence between the two-stage game and the single-stage SPIF objective is only algebraic at the level of the reward-player objective. Substituting Δr=β log(π/π_k) requires the map π→Δr to be a reparameterization of the reward class; no surjectivity or partition-function consistency is shown. In particular, the actual reward attaining the KL closed form is r=Δr+β logZ(x), with Z(x) generally different from 1, and ψ(r)≠ψ(Δr). Consequently, (5.2) is not proven equivalent to (4.2). The finite-sample implementation bL(π) is also left unanalyzed, so the practical algorithm is not connected to the theorem.
  3. [§4.1, Prop 4.1 and Remark 4.6] The bounded-reward property concerns the optimizer of the variational χ2 problem. It does not imply that |β log(π/π_k)|≤1/c during training. The magnitude of Δr is controlled by β and the policy ratio, and the least-squares targets in (5.2) encourage but do not enforce boundedness. Since Theorem 4.4's Rmax enters quadratically, the claimed tighter bound for SPIF is not established. The empirical reward magnitudes in Figure 1 are illustrative, not a guarantee.
  4. [Appendix A.2, Lemma A.2] The proof uses the inequality ⟨r⋆,π−π⋆⟩≤0 for all π∈Π without proof or a definition of r⋆. This inequality holds only if (π⋆,r⋆) is a saddle point of the payoff ⟨r,π−π⋆⟩; the theorem statement does not state this. The proof can likely be repaired by choosing r⋆=0 (which is feasible and gives zero payoff), but as written the OMD comparison argument is incomplete.
  5. [Table 3 / Appendix L] There is a direct inconsistency: Table 3 reports c=0.5, while Appendix L states 'In our main experiments, we set c=2, corresponding to rmax=0.5 and rmin=−0.5'. This affects the reported rmax/rmin and the reproducibility of Table 2. The inconsistency must be resolved.
minor comments (7)
  1. [Throughout] Typos and grammatical issues include: 'it's equilibrium' in the abstract, 'acorss' in the Table 2 caption, 'singe stage' before Prop 5.2, 'idential' in Sec 7.2, 'the r-player runs' in Appendix A.2, and 'as show in in Figure 2' in Sec 6.
  2. [Figures 1 and 2] The legend label 'SPIF ( 2)' is missing the χ symbol; it should read 'SPIF (χ2)' to match the text.
  3. [Table 1] The row 'INPO Zhang et al. [2025]' conflicts with the text, which cites INPO as Zhang et al. [2024]; ONPO is Zhang et al. [2025]. Please reconcile.
  4. [Eq. (5.2) and Prop 5.2] The Bregman regularizer in Eq. (5.2) is written as ζ/2 E_{D⋆∪Dk}[(log π/π_k)^2], whereas Prop 5.2 defines D_f(π,π_k) with an expectation over π⋆ only. Clarify the relation between these two definitions.
  5. [Table 2] No standard deviations or significance tests are reported. Given the small differences between methods, adding error bars or at least multiple-seed variances would strengthen the empirical claims.
  6. [Appendix L] The ablation text mentions c=8, while the Figure 3 caption lists c∈{0.125,0.5,2}. These values are inconsistent and should be aligned.
  7. [Algorithm 2] Line 4 says 'Update policy via (5.2)', but (5.2) uses empirical datasets D⋆ and Dk. Please specify how Dk is sampled in each iteration and how the two expectations in bL(π) are constructed (e.g., balanced batches).

Circularity Check

1 steps flagged

Minor circularity: the c-ablation confirms the SPIF loss's own reward targets; the central Theorem 4.4 derivation is otherwise self-contained.

specific steps
  1. self definitional [Eq. (5.2), Appendix L (Ablation on Hyperparameter c)]
    "with r_max = 1/c and r_min = −1/c. ... A larger value of c results in a smaller margin between r_max and r_min, whereas a smaller value of c induces a larger margin and higher reward magnitude. ... When c is reduced to 0.5, the resulting larger reward magnitude leads to degraded performance, which is consistent with the theoretical predictions in Sec 4.3."

    The reward magnitude is not an independent prediction: Eq. (5.2) defines the regression targets as r_max = 1/c and r_min = −1/c, so decreasing c makes the targets larger by construction. The ablation's observation that smaller c yields larger reward magnitude is therefore a tautology of the loss definition, not an empirical confirmation of the game-theoretic bound. It is presented as validating Sec 4.3, but the relationship is definitional.

full rationale

The central game-theoretic derivation is self-contained and not circular: Theorem 4.4 is proved from standard OMD regret bounds (Lemmas A.1, A.2) under stated assumptions (realizability, bounded reward, Bregman divergence), and Proposition 4.1 is adapted from the external LS-IQ result of Al-Hafez et al. 2023. The paper's self-citation (Li et al. 2025) appears only as related-work context and is not load-bearing. The practical bridge in Proposition 5.1 is a real proof gap: it asserts that setting logZ=0 lets the Theorem 4.4 proof 'still hold', without showing that the fixed reward-class and Bregman-divergence assumptions survive the reparameterization Δr=β log(π/π_k). I weigh that as a correctness/validity risk, not as circularity, because it is an omitted argument rather than a definitional reduction. The only genuine by-construction circularity is the c-ablation: since the SPIF objective defines r_max=1/c and r_min=−1/c, the observed dependence of reward magnitude on c is a consequence of Eq. (5.2), not an independent prediction. This is an auxiliary validation issue and does not make the main theorem circular; hence the low score.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central theorem has no fitted constants; the proposed algorithm introduces four hyperparameters. The most consequential is c, because it sets the bounded reward targets, but its reported value is inconsistent across the paper. The nonstandard load-bearing step is the logZ=0 partition-function removal in Prop 5.1.

free parameters (4)
  • c = 2 (main text §L); Table 3 lists 0.5; ablation tries 0.125, 0.5, 2, 8
    Controls the reward targets rmax=1/c and rmin=-1/c in Eq. 5.2, and hence the bounded-reward magnitude. The paper chooses it by ablation, and its reported value is internally inconsistent.
  • β = 1e-3
    KL/scale coefficient in the reward reparameterization β log(π/π_k); a chosen hyperparameter that directly sets the reward scale.
  • ζ = 1e-3
    Weight on the mirror-descent/Bregman regularization term in Eq. 5.2; chosen as a hyperparameter.
  • α = 0.5
    Mix ratio between expert and previous-policy data in the mixed chi-squared regularizer; set to balanced sampling.
axioms (6)
  • domain assumption Ground-truth reward and optimal policy are realizable (r* ∈ R, π* ∈ Π)
    Assumption 3.1; without it the optimal reward/saddle point used in Theorem 4.4 and Prop 4.1 may not exist in the function classes.
  • domain assumption Rewards are bounded: r ∈ [-Rmax, Rmax]
    Theorem 4.4 assumes bounded reward; SPIN's implicit reward is unbounded, so the theorem does not directly apply to SPIN; SPIF relies on chi-squared regularization to make Rmax=1/c.
  • domain assumption LLM generation is a contextual bandit with no transition dynamics
    Section 3; this reduces AIL occupancy-measure arguments to per-prompt expectations and avoids MDP horizon issues.
  • domain assumption Exact access to expert and previous-policy expectations and exact argmax in Algorithm 1
    Algorithm 1's theoretical updates use exact expectations and exact optima; the practical SPIF algorithm uses finite data and gradient training, so the O(1/√K) guarantee is for an idealized version.
  • ad hoc to paper Partition function can be set to zero in the mapped reward
    Prop 5.1/§5: Δr = r - β log Z(x); the proof asserts the Theorem 4.4 bound still holds by re-running with logZ=0, without showing the reward class or regularizer is invariant.
  • domain assumption For the SPIN-as-KL claim, β≥1 (Lemma I.1)
    Appendix I: the contraction proof uses Jensen on t^{1/β}, requiring β≥1; SPIN's β is not constrained in practice, so the claim that non-linear SPIN minimizes KL may not hold for the hyperparameters used.

pith-pipeline@v1.3.0-alltime-deepseek · 24377 in / 18135 out tokens · 174965 ms · 2026-08-03T05:41:07.005616+00:00 · methodology

0 comments
read the original abstract

Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without preference data. However, the theoretical foundations for self-play finetuning remain underexplored. In this work, we tackle this by connecting self-play finetuning with adversarial imitation learning by formulating finetuning procedure as a min-max game between the model and a regularized implicit reward player parameterized by the model itself. This perspective unifies self-play imitation and general preference alignment within a common framework. Under this formulation, we present a game-theoretic analysis showing that the self-play finetuning will converge to it's equilibrium. Guided by this theoretical formulation, we propose a new self-play imitation finetuning algorithm based on the $\chi^2$-divergence variational objective with bounded rewards and improved stability. Experiments on various of language model finetuning tasks demonstrate consistent improvements over existing self-play methods and validate our theoretical insights.

Figures

Figures reproduced from arXiv: 2602.01357 by Chetan Bansal, Shangzhe Li, Weitong Zhang, Xuchao Zhang.

Figure 1
Figure 1. Figure 1: Reward Dynamics Analysis. We plot the reward curves (log-scaled) during training for our approach, SPIF with χ 2 regularization, and for SPIN [Chen et al., 2024]. The results show that our method produces rewards with substantially smaller magnitude, which leads to more stable learning dynamics and is consistent with our theoretical analysis predicting a tighter duality gap. approach with χ 2 regularizatio… view at source ↗
Figure 2
Figure 2. Figure 2: 0.0 0.2 0.4 0.6 0.8 1.0 Training Steps (K) 10 4 10 3 10 2 10 1 10 0 10 1 10 2 10 3 10 4 Gradient Norm (Log Scale) Gradient Norm for Qwen-3 4B SPIN SPIF ( 2) 0.0 0.2 0.4 0.6 0.8 1.0 Training Steps (K) 10 3 10 2 10 1 10 0 10 1 10 2 10 3 10 4 Gradient Norm (Log Scale) Gradient Norm for Mistral 7B SPIN SPIF ( 2) [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation on Hyperparameter c. We evaluate the impact of the hyperparameter c by setting c ∈ {0.125, 0.5, 2} and examining its effect on the self-play performance of our method. Performance is measured as the mean score across the four benchmarks used in the main experiments. We observe that both overly small and overly large values of c lead to performance degradation, highlighting the importance of an app… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. IRIS: Interpolative R\'enyi Iterative Self-play for Large Language Model Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 7.0

    IRIS unifies self-play fine-tuning under an interpolative Rényi objective with adaptive alpha scheduling and reports better benchmark scores than baselines while surpassing full supervised fine-tuning with only 13% of...

Reference graph

Works this paper leans on

23 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Substituting this inequality back into (I.2), we obtain: DKL(π⋆∥πk+1)≤(1−α)D KL(π⋆∥πk)

    Thus,Z(x)≤1, which implies logZ(x)≤0. Substituting this inequality back into (I.2), we obtain: DKL(π⋆∥πk+1)≤(1−α)D KL(π⋆∥πk). Substitutingα= 1/βyields the geometric contraction: DKL(π⋆∥πk+1)≤ 1− 1 β DKL(π⋆∥πk). Applying this inequality recursivelyktimes leads to the final convergence rate: DKL(π⋆∥πk)≤ 1− 1 β k DKL(π⋆∥πref ). This completes the proof. Cons...

    2024

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

  3. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1,

    Pith/arXiv arXiv

  4. [9]

    Measuring massive multitask language understanding.ArXiv, abs/2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding.ArXiv, abs/2009.03300,

    Pith/arXiv arXiv 2009

  5. [11]

    Near-optimal second-order guarantees for model- based adversarial imitation learning.arXiv preprint arXiv:2510.09487,

    Shangzhe Li, Dongruo Zhou, and Weitong Zhang. Near-optimal second-order guarantees for model- based adversarial imitation learning.arXiv preprint arXiv:2510.09487,

    arXiv

  6. [12]

    Provably efficient generative adversarial imitation learning for online and offline setting with linear function approximation

    Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Provably efficient generative adversarial imitation learning for online and offline setting with linear function approximation. arXiv preprint arXiv:2108.08765,

    Pith/arXiv arXiv

  7. [13]

    Hybrid inverse reinforcement learning.arXiv preprint arXiv:2402.08848,

    Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning.arXiv preprint arXiv:2402.08848,

    Pith/arXiv arXiv

  8. [14]

    Computational-statistical tradeoffs at the next-token prediction barrier: Autoregressive and im- itation learning under misspecification.arXiv preprint arXiv:2502.12465,

    Dhruv Rohatgi, Adam Block, Audrey Huang, Akshay Krishnamurthy, and Dylan J Foster. Computational-statistical tradeoffs at the next-token prediction barrier: Autoregressive and im- itation learning under misspecification.arXiv preprint arXiv:2502.12465,

    Pith/arXiv arXiv

  9. [17]

    Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675,

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment.arXiv preprint arXiv:2405.00675,

    Pith/arXiv arXiv

  10. [18]

    Qwen3 technical report.ArXiv, abs/2505.09388,

    18 An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Jingren Zhou, Junyan Lin, Kai Dang, Keqin Bao, Ke-Pei...

    Pith/arXiv arXiv

  11. [19]

    Iterative nash policy optimization: Aligning llms with general preferences via no-regret learning.arXiv preprint arXiv:2407.00617,

    Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning llms with general preferences via no-regret learning.arXiv preprint arXiv:2407.00617,

    Pith/arXiv arXiv

  12. [20]

    Improving llm general preference alignment via optimistic online mirror descent.arXiv preprint arXiv:2502.16852,

    Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, and Dong Yu. Improving llm general preference alignment via optimistic online mirror descent.arXiv preprint arXiv:2502.16852,

    Pith/arXiv arXiv

  13. [21]

    19 A Proof of Theorem 4.4 A.1 Key Lemmas We first introduce the following lemmas: Lemma A.1(One-Step Descent, Cai et al. 2020).For two policy distributionsπ ⋆ andπ, and a reward functionr:X × Y →[−Rmax, Rmax], it holds forπ ′(·|x)∝π(·|x)·exp(η·r(x,·)) that: ⟨r(x,·), π⋆(·|x)−π(·|x)⟩ ≤ ηR2 max 2 +η −1 · DKL(π⋆(·|x)∥π(·|x))−D KL(π⋆(·|x)∥π′(·|x)) Proof.For an...

    2020

  14. [22]

    Therefore, when we set logZ= 0 and re-apply the proof for Theorem 4.4, we will obtain the same result. 24 E Proof of Proposition 5.2 Proof of Proposition 5.2.Consider the reward update rule: (∆r)k = argmax∆r J(∆r) :=E ρ h σ(Eπ⋆ ∆r(x, y)−Eπk ∆r(x, y))−ψ(∆r,(∆r) k−1) i , whereψ(∆r,(∆r) k−1) =ζD f (∆r,(∆r) k−1) +cα·E π⋆[(∆r(x, y))2] +c(1−α)·E π[(∆r(x, y))2] ...

    2024

  15. [2016]

    Correcting the mythos of kl-regularization: Direct alignment without overopti- mization via chi-squared preference optimization.arXiv preprint arXiv:2407.13399,

    Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overopti- mization via chi-squared preference optimization.arXiv preprint arXiv:2407.13399,

    Pith/arXiv arXiv

  16. [2018]

    Enhancing chat language models by scaling high-quality instructional conversations.ArXiv, abs/2305.14233,

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations.ArXiv, abs/2305.14233,

    Pith/arXiv arXiv

  17. [2019]

    Lamda: Language models for dialog applications.arXiv preprint arXiv:2201.08239,

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications.arXiv preprint arXiv:2201.08239,

    Pith/arXiv arXiv

  18. [2020]

    Human alignment of large language models through online preference optimisation.arXiv preprint arXiv:2403.08635,

    Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation.arXiv preprint arXiv:2403.08635,

    Pith/arXiv arXiv

  19. [2021]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  20. [2022]

    Enhancing llm reasoning with iterative dpo: A comprehensive empirical investigation.arXiv preprint arXiv:2503.12854,

    Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xi- angyuan Lan, Dongmei Jiang, et al. Enhancing llm reasoning with iterative dpo: A comprehensive empirical investigation.arXiv preprint arXiv:2503.12854,

    Pith/arXiv arXiv

  21. [2023]

    Ls-iq: Implicit reward regularization for inverse reinforcement learning.arXiv preprint arXiv:2303.00599,

    Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning.arXiv preprint arXiv:2303.00599,

    Pith/arXiv arXiv

  22. [2024]

    Self-play fine-tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335,

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335,

    Pith/arXiv arXiv

  23. [2025]

    Contrastive preference learning: learning from human feedback without rl.arXiv preprint arXiv:2310.13639,

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: learning from human feedback without rl.arXiv preprint arXiv:2310.13639,

    Pith/arXiv arXiv