Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Game-theoretic alignment is robust for winner-based criteria, yet no smooth pairwise-preference payoff can exactly reproduce a target diversity policy.

desk verdict The consistency characterizations are solid and citable; the preference-matching impossibility is not established as written and needs a repaired proof. read the letter →

arxiv 2505.20627 v1 pith:VU5ZPJZC submitted 2025-05-27 cs.GT stat.ML

Zhekun Shi , Kaizhao Liu , Qi Long , Weijie J. Su , Jiancong Xiao This is my paper
classification cs.GTstat.ML MSC 91A1091B1291B14
keywords game-theoreticalignmentNashlearningfromhumanfeedbackCondorcetconsistencySmithpreferencematchingpairwisepreferenceszero-sumgamesLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether game-theoretic LLM alignment, in which a policy is trained as the solution of a two-player zero-sum game with payoff built from pairwise human preferences, can satisfy key alignment criteria through a clever choice of payoff function. It shows that Condorcet consistency holds for any payoff that merely preserves which response wins each pairwise comparison, and it gives exact conditions under which the Nash solution is mixed or Smith consistent. The central negative result is an impossibility theorem: no smooth payoff depending on pairwise preference ratios can guarantee a unique Nash equilibrium matching an arbitrary target policy, even when preferences follow the Bradley-Terry-Luce model. Altogether, game-theoretic alignment is robust for winner-based and diversity-in-support properties, but it cannot fully match a prescribed diverse-preference policy.

What carries the argument

The central object is the payoff matrix $\{\Psi_{ij}\}$ with $\Psi_{ij}=\Psi(P(y_i\succ y_j))$ in the max-min game (1.2). The proof of the impossibility result rests on the KKT characterization of full-support Nash equilibria (Lemma 5.1), which reduces 'the unique Nash solution is $\pi^*$' to a system of column-sum equalities. An infinitesimal variation over four distinct responses then forces the smooth function $f$ to satisfy the differential equation $f(x)+x f'(x)=C_1$, whose only solutions have the form $f(x)=C_2/x+C_3$; substituting this form back into the equalities contradicts the assumed independence of $f$ from $n$ and from $\pi^*$ while maintaining uniqueness.

What would settle it

A concrete test is to take $n=5$, set the diagonal to a constant $C$, choose a smooth $f$ not of the form $f(x)=c_2/x+c_3$, and check for many random full-support targets $\pi^*>0$ whether the column-sum equalities $\sum_{i\neq j}\pi_i^* f(\pi_i^*/\pi_j^*)+C\pi_j^*=t^*$ hold with $\sum_i\pi_i^*=1$ and whether the unique Nash solution equals $\pi^*$. Finding a single such $f$ and $\pi^*$ would refute Theorem 5.1, and finding any smooth $f$ outside the excluded family that satisfies the equalities for all $\pi^*$ would directly contradict the differential-equation step in the proof.

Watch

Extended reading notes

Core claim

Working within the framework of Nash Learning from Human Feedback, the paper replaces the raw preference $P(y\succ y')$ in the payoff of the max-min alignment game by $\Psi(P(y\succ y'))$ and characterizes which mappings $\Psi$ produce desirable alignment properties. Assuming no ties, Condorcet consistency holds if and only if $\Psi(t)\ge\Psi(1/2)$ for $t\ge 1/2$ and $\Psi(t)<\Psi(1/2)$ for $t<1/2$: the payoff need only preserve the direction of each pairwise majority. For mappings continuous at $1/2$, requiring a mixed strategy whenever no Condorcet winner exists adds the condition $\Psi(t)+\Psi(1-t)\ge 2\Psi(1/2)$, while Smith consistency is equivalent to the antisymmetry condition $\Psi(t)+\Psi(1-t)=2\Psi(1/2)$ together with the Condorcet condition; this says the game is strategically a symmetric zero-sum game. The main negative result, Theorem 5.1, states that there is no payoff matrix with constant diagonal $C$ and off-diagonal entries $f(\pi_i^*/\pi_j^*)$ for a single smooth function $f$ independent of the target policy and the response count $n$, whose unique Nash solution is an arbitrary full-support target $\pi^*$. Consequently, no smooth mapping $\Psi$ applied to pairwise preferences can achieve preference matching, even under the BTL model.

Load-bearing premise

The impossibility claim rests on Assumption 5.2, which says a learnable payoff must have a constant diagonal and off-diagonal entries given by one smooth function of the ratio $\pi_i^*/\pi_j^*$ with no dependence on the response count $n$; allow dependence on $n$, on context, or a non-smooth function, and the argument no longer goes through.

Editorial extensions

If this is right

  • If the paper is right, any payoff that preserves the winner of each pairwise comparison yields Condorcet consistency, so a learned preference model that keeps the same majority directions will not break this property.
  • The payoff $\Psi(t)=\log(t/(1-t))$ satisfies the Smith-consistency condition even when preferences do not follow the BTL model, giving a natural non-BTL generalization of RLHF that is Smith consistent.
  • Smith consistency forces the game to be strategically symmetric zero-sum, so preference models used in practice should enforce the anti-symmetry condition $P_\theta(y\succ y')+P_\theta(y'\succ y)=1$ and $P_\theta(y\succ y)=1/2$; otherwise the aligned policy can put mass outside the Smith set.
  • Exact preference matching is impossible with smooth ratio-based payoffs, meaning game-theoretic alignment can at best capture diversity qualitatively through mixed strategies rather than reproduce a prescribed diverse policy exactly.
  • Discontinuous payoff functions escape the continuous classification: a piecewise-constant $\Psi$ can be Smith consistent even when the continuous characterization does not apply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the proof is driven by smoothness, a natural conjecture the paper does not make is that a merely continuous or discontinuous payoff might achieve preference matching; the paper's own discontinuous examples already show such payoffs can have stronger consistency properties.
  • The one payoff construction the paper exhibits that does match a target policy depends explicitly on the response count $n$, so exact matching may require the alignment procedure to know the vocabulary size or to include a regularization term, a direction the paper flags as open.
  • If smoothness is the only obstruction, approximate preference matching to arbitrary accuracy should be attainable by smooth payoffs, and quantifying the approximation rate would be a concrete next test of the theorem's reach.
  • The Smith-consistency result suggests a practical diagnostic: a preference model that violates anti-symmetry should produce Nash solutions with positive mass outside the top preference class, which can be checked empirically on existing NLHF runs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies the generalized game-theoretic alignment objective (1.2), in which a mapping Ψ is applied to pairwise preferences before forming a two-player zero-sum game. It characterizes when such games are Condorcet consistent (Theorem 3.1), when they produce mixed strategies in the absence of a Condorcet winner (Theorem 3.2), and when they are Smith consistent (Theorem 4.2). It then introduces Assumption 5.2, restricting payoff matrices to have a constant diagonal and off-diagonal entries of the form f(π_i*/π_j*) for a single smooth function f, and claims Theorem 5.1, an impossibility result for exact preference matching under this assumption. Sections 3 and 4 contain careful necessity and sufficiency arguments built from explicit finite payoff tables and continuity of Ψ at 1/2. Section 5 attempts to prove the impossibility via KKT conditions and a functional equation for f, but the proof as written has serious gaps.

Significance. If the results are established, Sections 3 and 4 provide a clean and useful robustness story: Condorcet consistency and Smith consistency depend only on mild qualitative conditions on Ψ, and Smith consistency automatically forces mixed strategies. These characterizations are parameter-free and the proofs are self-contained. The claimed Theorem 5.1 would be a notable negative result about the limits of exact diversity matching in game-theoretic alignment. However, the negative result is not established as written: the KKT lemma is mis-stated for the row player, and the proof of Theorem 5.1 contains an unjustified limit step that is load-bearing for the final contradiction. The rest of the paper is sound but the central novelty in the title depends on Section 5.

major comments (2)
  1. [Section 5.1, Lemma 5.1 and Appendix A] The KKT system in Lemma 5.1 is derived in Appendix A for the epigraph formulation of min_π max_j Σ_i α_ij π_i, whose primal optimizer is the column player's strategy, not the row player's Nash solution defined in Eq. (2.1). For the row player's problem max_π min_j Σ_i α_ij π_i, all column payoffs at an optimal row strategy are at least the value, so the correct first inequality is Σ_i π*_i α_ij − t* ≥ 0, with equality on the support of the column strategy u*. The proof of Lemma 5.5 inherits the wrong sign in the definition of V2, and it also asserts without justification that a normalized element of V2 is a Nash equilibrium: the equalities in V1 only show that the quantities Σ_i π_i α_ij are equal to each other for j ∈ J0, not that they equal t*. These issues must be repaired before Lemma 5.5 can support Theorem 5.1.
  2. [Section 5.1, proof of Theorem 5.1, Eqs. (5.10)-(5.13)] The step 'Taking limit, it must hold C(π_j1,π_j2) = 0' between Eqs. (5.11) and (5.12) is not justified under Assumption 5.2, which only assumes f is smooth on (0,∞) and imposes no control of g(t)=f(t)+t f'(t) as t→0. The smooth function f(t)=ln t + d/t + c gives g(t)=ln t + const and satisfies the iterated relation g(x/π_j1)=nC+g((π_j1/π_j2)^n x/π_j2) with C=ln(π_j2/π_j1)≠0 for every n, because the logarithmic divergence cancels nC; hence Eq. (5.12) does not follow and the final contradiction in Eq. (5.13) is unsupported. A repair appears feasible by solving the functional equation (5.10) directly (its smooth solutions are g(t)=c ln t + d) and then substituting the resulting f into Eq. (5.4) to force c=0 and eliminate the n-dependent constant, but this is a substantive rewrite of the proof rather than a local correction.
minor comments (4)
  1. [Section 5.1, Eq. (5.7) to Eq. (5.8)] The passage from Eq. (5.7) to Eq. (5.8) should state that both sides are divided by δ before taking δ→0; as written, the limit of the displayed equation alone gives only 0=0.
  2. [Section 5.1, after Eq. (5.9)] The admissible range for x1 and x2 is x1+x2 < 1−π_j1−π_j2, since both are masses of two distinct remaining alternatives; the text says x1,x2 ∈ (0,1−π_j1−π_j2), which is not the correct joint condition.
  3. [Example 3.4 proof] In the display following Eq. (3.3), the summation in PΨ(δ_i′, π*) should run over the column index j, not over i; the displayed expression is otherwise clear.
  4. [Assumption 5.2 and Theorem 5.1 statement] Assumption 5.2 explicitly excludes payoffs that depend on n or on contextual information, and Remark 5.3 already notes that n-dependent payoffs can achieve preference matching; the abstract and conclusions should therefore phrase the 'fundamental limit' as holding within Assumption 5.2 rather than as an unconditional statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's new theorems are derived directly from definitions, and its self-citations supply background results rather than load-bearing premises.

full rationale

The derivation chain in this paper is self-contained with respect to its central claims. Theorems 3.1, 3.2, and 4.2 characterize the mapping Psi through direct game-theoretic constructions (Tables 2-5) and elementary KKT arguments; they do not fit parameters, rename known results, or define their conclusions into their assumptions. Theorem 5.1, the impossibility result, proceeds from an explicit structural assumption (Assumption 5.2) and derives a functional contradiction from the KKT conditions; it does not assume the impossibility it claims to prove. The paper does cite prior works by overlapping authors, notably Liu et al. (2025) for the Smith-set decomposition and Xiao et al. (2024) for the preference-matching policy. These citations provide background definitions and a combinatorial tournament fact with stated no-tie assumptions; they do not presuppose the paper's new results, and the central impossibility theorem does not reduce to them. Any concern about the validity of the limit step in the proof of Theorem 5.1 is a mathematical correctness issue, not an instance of circular reasoning, and therefore does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted. The main additional premises are the no-tie condition, continuity of Ψ at 1/2, the structural Assumption 5.2, and the prior Smith decomposition. The impossibility proof further depends on a KKT characterization that is misstated in the paper.

assumptions (5)
  • domain assumption No-Tie Assumption (Assumption 2.1): for distinct responses, P(y≻y')≠1/2.
    Used throughout to ensure strict inequalities and the unique Smith set decomposition; the authors call it minimal and practically reasonable.
  • domain assumption Continuity of Ψ at 1/2 in Theorems 3.2 and 4.2
    The necessity proofs take limits as t2→1/2; the paper shows discontinuous piecewise-constant payoffs behave differently (Examples 3.4 and 4.3).
  • domain assumption Existence of a full-support ideal policy π*>0 in Section 5
    Preference matching is defined against such a target; responses not supported by π* are excluded, and the KKT proof requires π*>0.
  • domain assumption Assumption 5.2: αii=C constant, αij=f(π_i*/π_j*) for a smooth f independent of π* and n
    This is the formal meaning of 'payoff learnable from pairwise preferences' in the impossibility proof. It is plausible under BTL but is an extra restriction on the class of payoffs.
  • domain assumption Smith set decomposition of Liu et al. (2025) (Theorem 4.1)
    The paper relies on this cited uniqueness/decomposition result to define the Smith set; this is background, not the paper's own proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching." pith.science (2026). https://pith.science/paper/VU5ZPJZC

@misc{pith2026250520627,
  author       = {Pith},
  title        = {Pith review of: Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VU5ZPJZC}},
  note         = {Machine review of arXiv:2505.20627}
}
read the original abstract

Nash Learning from Human Feedback is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game. However, using raw preference as the payoff in the game highly limits the potential of the game-theoretic LLM alignment framework. In this paper, we systematically study using what choices of payoff based on the pairwise human preferences can yield desirable alignment properties. We establish necessary and sufficient conditions for Condorcet consistency, diversity through mixed strategies, and Smith consistency. These results provide a theoretical foundation for the robustness of game-theoretic LLM alignment. Further, we show the impossibility of preference matching -- i.e., no smooth and learnable mappings of pairwise preferences can guarantee a unique Nash equilibrium that matches a target policy, even under standard assumptions like the Bradley-Terry-Luce model. This result highlights the fundamental limitation of game-theoretic LLM alignment.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

    stat.ML 2025-06 conditional novelty 5.0 of 10

    RLHF reward modeling satisfies pairwise majority and Condorcet consistency when each response pair is labeled once, because the maximum likelihood ranking then matches the Copeland rule.

Reference graph

Works this paper leans on

36 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anthropic

    A. Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024

    work page 2024

  2. [2]

    M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR, 2024

    work page 2024

  3. [3]

    M. L. Balinski and R. Laraki. Majority judgment : measuring, ranking, and electing. MIT Press, 2010

    work page 2010

  4. [4]

    B \"o rgers

    C. B \"o rgers. Mathematics of social choice: voting, compensation, and division. Society for Industrial and Applied Mathematics, 2010

    work page 2010

  5. [5]

    R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  6. [6]

    Bubeck, V

    S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  7. [7]

    Casper, X

    S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. J. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. Dragan, D. Kru...

    work page 2023

  8. [8]

    Chakraborty, J

    S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024

    arXiv 2024

Show all 36 references
  1. [9]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Au...

    2023

  2. [10]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  3. [11]

    Conitzer, R

    V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Moss\' e , E. Pacuit, S. Russell, H. Schoelkopf, E. Tewolde, and W. S. Zwicker. Position: social choice should guide ai alignment in dealing with diverse human feedback. In Forty-first Internati...

    2024

  4. [12]

    Dai and E

    J. Dai and E. Fleisig. Mapping social choice theory to RLHF . In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024

    2024

  5. [13]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J.-M. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B.-L. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D.-L. ...

    2025 arXiv

  6. [14]

    H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. RLHF workflow: From reward modeling to online RLHF . Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=a13aYUU9eU

    2024

  7. [15]

    Duersch, J

    P. Duersch, J. Oechssler, and B. C. Schipper. Pure strategy equilibria in symmetric two-player zero-sum games. International Journal of Game Theory, 41: 0 553--564, 2012. ISSN 0020-7276

    2012

  8. [16]

    Eloundou, S

    T. Eloundou, S. Manning, P. Mishkin, and D. Rock. GPTs are GPTs : L abor market impact potential of LLMs . Science, 384 0 (6702): 0 1306--1308, 2024

    2024

  9. [17]

    L. Ge, D. Halpern, E. Micha, A. D. Procaccia, I. Shapira, Y. Vorobeychik, and J. Wu. Axioms for ai alignment from human feedback. In Advances in Neural Information Processing Systems, volume 37, pages 80439--80465, 2024

    2024

  10. [18]

    W. V. Gehrlein. Condorcet’s paradox. Springer, 2006

    2006

  11. [19]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mądry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A...

    arXiv

  12. [20]

    W. Ji, W. Yuan, E. Getzen, K. Cho, M. I. Jordan, S. Mei, J. E. Weston, W. J. Su, J. Xu, and L. Zhang. An overview of large language models for statisticians. arXiv preprint arXiv:2502.17814, 2025

    2025 arXiv

  13. [21]

    Jiang, X

    D. Jiang, X. Ren, and B. Y. Lin. LLM -blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165--14178. Association for ...

    2023 doi

  14. [22]

    K. Liu, Q. Long, Z. Shi, W. J. Su, and J. Xiao. Statistical impossibility and possibility of aligning llms with human preferences: From condorcet paradox to nash equilibrium. arXiv preprint arXiv:2503.10990, 2025

    2025 arXiv

  15. [23]

    R. D. Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012

    2012

  16. [24]

    Maura-Rivero, M

    R.-R. Maura-Rivero, M. Lanctot, F. Visin, and K. Larson. Jackpot! alignment as a maximal lottery. arXiv preprint arXiv:2501.19266, 2025

    2025 arXiv

  17. [25]

    A. Mishra. Ai alignment and social choice: Fundamental limitations and policy implications. arXiv preprint arXiv:2310.16048, 2023

    2023 arXiv

  18. [26]

    Munos, M

    R. Munos, M. Valko, D. Calandriello, M. Gheshlaghi Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, A. Michi, M. Selvi, S. Girgin, N. Momchev, O. Bachem, D. J. Mankowitz, D. Precup, and B. Piot. Nash learning from human feedback. In Forty-first Internatio...

    2024

  19. [27]

    R. B. Myerson. Game theory. Harvard university press, 2013

    2013

  20. [28]

    Noothigattu, D

    R. Noothigattu, D. Peters, and A. D. Procaccia. Axioms for learning from pairwise comparisons. In Advances in Neural Information Processing Systems, volume 33, pages 17745--17754, 2020

    2020

  21. [29]

    Novikov, N

    A. Novikov, N. V\ u , M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog. Alpha E volve: A coding agent for scientific and...

    2025

  22. [30]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. Technical report, OpenAI, 2025

    2025

  23. [31]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022

  24. [32]

    Shoham and K

    Y. Shoham and K. Leyton-Brown. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, 2008

    2008

  25. [33]

    Siththaranjan, C

    A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in RLHF . In The Twelfth International Conference on Learning Representations, 2024

    2024

  26. [34]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  27. [35]

    Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu. Self-play preference optimization for language model alignment. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024. URL https://openreview.net/forum?id=Z1PDdGekgn

    2024

  28. [36]

    J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. arXiv preprint arXiv:2405.16455, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.