Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

SharedRep-RLHF claims that learning a shared low-rank representation of preferences across annotator groups makes worst-case fairness alignment work for small minorities, with provable gains over MaxMin-RLHF and O(1/Δ_min⁴) sample cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SharedRep-RLHF learns a shared preference representation across groups to improve worst-case reward estimates for minority annotators, but the theoretical guarantees are undermined by proof errors.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The empirical idea is worth a look, but the main theory is broken: Lemma 4.1's entropy equivalence is false, and the sample-complexity theorem falls with it. the 3 major comments →

arxiv 2509.03672 v1 pith:4NPX6Q66 submitted 2025-09-03 cs.LG stat.ML

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

Arpan Mukherjee , Marcello Bullo , Deniz G\"und\"uz This is my paper
classification cs.LG stat.ML
keywords RLHFdiverse preferencesMaxMin-RLHFshared representationsample complexitygroup fairnessBradley-Terry modelpessimism
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard way of handling diverse human preferences in RLHF—learning a separate reward model for each group and optimizing for the worst-off group—fails when the worst-off group is also a small minority, because each group's reward model is estimated from only its own sparse data. It proposes SharedRep-RLHF, which instead assumes all groups share a low-dimensional set of preference traits: each group's reward is a linear combination of these shared traits with group-specific weights. Because the shared trait extractor is estimated from the entire dataset, minority reward models inherit statistical strength from majority data. The paper proves that this estimator is more accurate, that the resulting policy has a provably lower regret gap against the optimal policy for every group, and that enforcing the worst-case (MaxMin) objective costs an additional O(1/Δ_min⁴) samples, where Δ_min is the smallest entropy gap in the induced Gibbs distributions. Experiments on sentiment, math, and dialogue show gains in minority score and win rate, with the largest gains at the smallest minority proportions.

Core claim

The central claim is that a shared low-rank representation of preferences across groups makes MaxMin-style fairness alignment statistically efficient when minority groups are sparsely annotated. Formally, the paper posits rewards r_u(x,y)=⟨φ(x,y), B w_u⟩ with a common feature extractor B∈R^{d×K} and group-specific weights w_u∈Δ^{K-1}; estimating B from all N samples instead of each group's N_u samples shrinks confidence radii from 1/√N_u to 1/√N. Theorem 3.4 states that for each group u, the suboptimality gap of the SharedRep pessimistic policy is at least ρ_min ξ_u smaller than MaxMin's gap, and Theorem 4.2 gives a PAC sample complexity whose MaxMin component is O(1/Δ_min⁴) in the large-gap

What carries the argument

The shared representation parameterization r_{B w_u}(x,y)=⟨φ(x,y), Bw_u⟩, with B∈R^{d×K}, K≪d, interprets columns of B as latent human traits and w_u as each group's mixture over traits. The argument's carrying mechanism is that group-specific rewards share B, so maximum-likelihood confidence sets scale with the total dataset size N rather than group size N_u; pessimism subtracts an uncertainty bonus scaled by η_SR; and worst-group identification is recast as entropy maximization of induced Gibbs distributions, which turns the MaxMin objective's sample complexity into an entropy-gap condition.

Load-bearing premise

The load-bearing premise is that the worst-off group can be found by comparing the entropies of each group's ideal response distributions; if a low-entropy group happens to have generally lower rewards, this comparison misses it, and the claimed sample-cost bound no longer tracks the difficulty of finding the true worst-off group.

What would settle it

Construct a two-group example where one group has a peaked, low-entropy Gibbs distribution but all its rewards are shifted down by a constant, while the other group has a flat, high-entropy distribution. If the low-entropy group is worst-off yet has the smaller entropy under the MaxMin policy, Lemma 4.1 is false, and the entropy-gap sample complexity cannot bound the probability of selecting the wrong group.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Minority reward modeling no longer depends only on group-specific data; any shared trait estimated from the full population lowers per-group estimation error.
  • At sufficiently large N, SharedRep's per-group suboptimality is provably smaller than MaxMin's by at least ρ_min ξ_u, so fairness gains do not vanish as the minority shrinks.
  • The MaxMin objective is learnable: O(1/Δ_min⁴) additional samples in the large-gap regime suffice to identify the worst-case group, the first such guarantee for MaxMin-RLHF.
  • If groups truly share no traits, the framework reduces to MaxMin-RLHF; the provable advantage is contingent on shared traits actually existing.
  • In practice, low minority proportions (1–10%) show the largest gains, with up to about 20 percentage-point win-rate improvements on GSM8K.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The entropy-equivalence lemma suggests a cheaper algorithm: estimate each group's Gibbs entropy and pick the max-entropy group, bypassing full pessimism; but this relies on the policy being the Gibbs distribution, so a practical implementation should verify the entropy ordering on held-out data.
  • The shared representation also offers a route to few-shot personalization: after B is learned population-wide, preferences of a new group with very few annotations could be estimated by fitting only w_new, a testable extension the paper does not pursue.
  • The GSM8K results at 15–20% minority where MaxMin slightly wins minority score suggest the shared-trait bias can hurt when group preferences diverge; an adaptive K selection or a mixture-of-experts variant would be a natural next test.
  • The Δ_min gap is defined through entropies of true Gibbs distributions; if entropy differences are small, sample complexity blows up exponentially via the Lambert-W branch, implying worst-group identification is intrinsically hard when groups are similar.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes SharedRep-RLHF, a shared low-rank representation method for reward modeling in RLHF with diverse annotator groups. The framework models each group reward as r_{B w_u} with a shared feature extractor B and group-specific simplex weights w_u, and then optimizes a KL-regularized MaxMin objective. The main theoretical claims are: (i) Lemma 3.3, an MLE concentration bound whose width scales with the total sample size N rather than the group size N_u; (ii) Theorem 3.4, a provable suboptimality gain over MaxMin-RLHF; and (iii) Theorem 4.2, a PAC sample-complexity guarantee for the MaxMin objective, with an additional O(1/Delta_min^4) term, based on Lemma 4.1 asserting that the worst-case reward group is the group with maximum Gibbs conditional entropy. Experiments on IMDb sentiment, GSM8K math reasoning, and Anthropic-HH dialogue report improvements over MaxMin-RLHF, especially at low minority proportions.

Significance. If the theoretical results were correct, the paper would make a useful contribution to fair and sample-efficient RLHF under heterogeneous preferences. The shared-representation idea is natural, the provided code and experiments are a positive feature, and the N-scaling in Lemma 3.3 is appealing. However, the central sample-complexity theorem rests on a false entropy equivalence, and the proof of the performance-comparison theorem contains an unjustified step. The empirical results are suggestive but cannot compensate for the unsound theory. The paper's main advertised guarantees are therefore not established.

major comments (3)
  1. [§4 / Appendix D, Lemma 4.1] Lemma 4.1 is false. The proof drops the log-normalizer of the Gibbs distribution. The original objective in Eq. (56) is equivalent to -beta * min_pi max_u ( KL(pi || nu_u) - log Z_u ), but Eq. (60) writes it without the -log Z_u term. Even if that term were absent, Eq. (62) incorrectly replaces E_{y~pi_r}[-log nu_u(y|x)] by H(nu_u(·|x)); the correct identity is E_{pi_r}[-log nu_u] = H(nu_u) + KL(pi_r || nu_u), which equals H(nu_u) only when pi_r = nu_u. Concrete counterexample: let Y={a,b}, pi_ref uniform, beta=1, r1(a)=1, r1(b)=0, r2(a)=-90, r2(b)=-110. Then nu_1 has entropy about 0.62 while nu_2 is sharply peaked at a with entropy near 0, yet r2 is strictly smaller than r1 under every policy, so the unique worst-case group is group 2. The lemma's conclusion fails.
  2. [§4 / Theorem 4.2, Eqs. (20)-(24)] Because Delta_min in Eq. (20) is defined through conditional entropy gaps, the misidentification bound P(hat u != u*) in Eqs. (99)-(103) depends on Lemma 4.1. The counterexample above shows that entropy ordering can be opposite to reward ordering: a uniquely worst-case group can have the smallest Gibbs entropy. Consequently, O(1/Delta_min^4) is not a sample-complexity bound for identifying the worst-case reward group, and the claimed first PAC guarantee for MaxMin-RLHF is unsupported. This is load-bearing: the second component N_MaxMin in Theorem 4.2 is exactly the price for finding u*.
  3. [§3 / Appendix C, Eq. (48)] The proof of Theorem 3.4 contains an invalid step. Eq. (47)-(48) first obtain A1 >= E[ <phi(pi^SR)-phi(pi^MM), B̃w̃> ] and then replace the right-hand side by E[ |<...>| ], claiming the absolute value can be imposed because A1 is positive. This is a non sequitur: A1 >= 0 and A1 >= g do not imply A1 >= E|g|. The positive lower bound rho_min xi_u in Eq. (55) relies precisely on this absolute-value replacement. The theorem's claimed provable advantage over MaxMin-RLHF is therefore not established by the given proof.
minor comments (3)
  1. [Throughout] Typos and infelicities: 'probabably' in the abstract, 'comaprison' in Figure 5, 'prefernces' in Section 1, 'conducte' in the conclusion, and inconsistent capitalization ('MaxMin' vs 'Maxmin'). A proofreading pass is needed.
  2. [§5, Table 2 / GSM8K] The text says SharedRep-RLHF 'consistently outperforms' MaxMin-RLHF, but at minority proportions 0.15 and 0.20 the mean minority score is lower for SharedRep (0.277 vs 0.317 and 0.270 vs 0.296). The abstract's 'gain of up to 20% in win rate' is also sensitive to the selected operating point. Please temper the empirical claims.
  3. [§2, Assumption 2.2 / Eq. (3)] The definition of xi_u uses max_x min_{y≠y'} but the proof of Theorem 3.4 in Appendix C appears to swap the min and max without comment. Even if the inequality direction happens to be acceptable, the order of quantifiers should be stated explicitly and compared with Eq. (3).

Circularity Check

0 steps flagged

No significant circularity: theoretical claims are derived from explicit concentration inequalities and standard external results; the weakness in Lemma 4.1 is a mathematical correctness error, not a circular reduction.

full rationale

The paper's derivation chain is self-contained with respect to circularity. Lemma 3.3 derives the shared-representation MLE confidence bound from a strong-convexity argument in Appendix B, while Lemma 3.1 is explicitly cited to an external source (Zhu et al., 2023). Theorem 3.4's performance comparison is a direct manipulation of pessimistic value functions using Lemma 3.3 and Assumption 2.2; no fitted parameter is relabeled as a prediction. Theorem 4.2's sample-complexity bound is derived from Lemma 4.1, a Fannes-type inequality, and Lemma 3.3, with the hardness gap Δ_min defined from ground-truth Gibbs entropies rather than from estimates. The main analytical weakness is Lemma 4.1's proof: Eqs. (59)-(60) drop the log-normalizer -ln Z_u, and Eqs. (62)-(63) replace the cross-entropy E_{π_r}[-ln ν_u] with H(ν_u), which is valid only if π_r = ν_u. This is a mathematical correctness defect (a constant-reward counterexample can violate the lemma), not a circularity: the lemma does not assume its conclusion or rename an input as an output. There are also no load-bearing self-citations; the authors' prior work is not used to justify the central premise. Hence the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The central claim rests on the linear shared-representation model, several standard RLHF assumptions, and the unproven/false Lemma 4.1. The main free parameters are the hand-chosen trait dimension K, the KL penalty beta, and the synthetic preference weights that define the experiments.

free parameters (3)
  • Inner dimension K of shared representation = K=2 for IMDb and HH, K=16 for GSM8K
    The number of shared trait dimensions is a hand-chosen hyperparameter; ablations (Appendix F, Figures 2-4) select K per task before reporting final results.
  • KL regularization coefficient beta = 0.05 in all PPO/GRPO runs
    Standard RLHF hyperparameter set by hand; listed in Appendix F Tables 5 and 8.
  • Synthetic gold-score preference weights for evaluation groups = e.g., 0.7 sentiment / 0.3 conciseness; 0.2 correctness / 0.8 brevity; 0.3 helpfulness / 0.7 harmlessness
    These hand-chosen weights define the synthetic group gold scores that generate all preference labels and drive the reported win rates and score gains, so they are effectively free parameters of the evaluation.
axioms (6)
  • domain assumption Known annotator group membership and Bradley-Terry preference model
    The paper assumes each annotator belongs to a known group and preferences follow the BT model, Eq. (1).
  • domain assumption Linear reward parameterization (Assumption 2.1)
    Rewards are assumed linear in known features: r_{B w_u}(x,y) = <φ(x,y), B w_u> with bounded columns and features.
  • domain assumption Reward gap xi_u > 0 (Assumption 2.2)
    Used in Theorem 3.4 to lower-bound the policy improvement; requires every model in the hypothesis class to be separable by some prompt.
  • domain assumption Positive prompt coverage rho_min > 0 (Assumption 2.3)
    Assumes every prompt has positive probability under rho, used for the rho_min xi_u term in Theorem 3.4.
  • domain assumption Shared representation factorization theta_u = B w_u with w_u on the simplex
    This is the core modeling premise of the paper, Eq. (2) in the introduction and used throughout.
  • ad hoc to paper Worst-case subpopulation equals maximum-entropy Gibbs group (Lemma 4.1)
    The proof (Appendix D, Eqs. 62-63) uses E_{π_r}[-ln ν_u] = H(ν_u), which is not an identity; this lemma is false in general and is load-bearing for Theorem 4.2.
invented entities (1)
  • Shared trait matrix B and group mixture weights w_u no independent evidence
    purpose: Latent structure intended to capture common preferences across groups while allowing group-specific mixtures, enabling reward estimates for sparse minorities from all data.
    B and w_u are latent parameters estimated from the same preference data used for evaluation; the paper offers no external probe (e.g., interpretable trait labels or transfer to unseen groups) that validates the latent traits independently.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences." pith.science (2026). https://pith.science/paper/4NPX6Q66

@misc{pith2026250903672,
  author       = {Pith},
  title        = {Pith review of: SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NPX6Q66}},
  note         = {Machine review of arXiv:2509.03672}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses this by learning group-specific reward models, and by optimizing for the group receiving the minimum reward, thereby promoting fairness. However, we identify that a key limitation of MaxMin-RLHF is its poor performance when the minimum-reward group is a minority. To mitigate this drawback, we introduce a novel framework, termed {\em SharedRep-RLHF}. At its core, SharedRep-RLHF learns and leverages {\em shared traits} in annotations among various groups, in contrast to learning separate reward models across groups. We first show that MaxMin-RLHF is provably suboptimal in learning shared traits, and then quantify the sample complexity of SharedRep-RLHF. Experiments across diverse natural language tasks showcase the effectiveness of SharedRep-RLHF compared to MaxMin-RLHF with a gain of up to 20% in win rate.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs

    cs.AI 2026-06 unverdicted novelty 5.0

    A survey that maps safety risks in personalized LLMs, introduces a unified taxonomy, and highlights three structural inadequacies in existing research on user-invariant safety, isolated techniques, and short-term evaluations.

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    A general language assistant as a laboratory for alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    A Sharp Fannes-type Inequality for the von Neumann Entropy

    Koenraad MR Audenaert. A sharp fannes-type inequality for the von neumann entropy. arXiv preprint quant-ph/0610146, 2006

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  5. [5]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J \'e r \'e my Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Value-incentivized preference optimization: A unified approach to online and offline rlhf

    Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Maxmin- RLHF : Alignment with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin- RLHF : Alignment with diverse human preferences. arXiv:2402.08925, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    PAL : Pluralistic alignment framework for learning from heterogeneous preferences

    Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. PAL : Pluralistic alignment framework for learning from heterogeneous preferences. arXiv:2406.08469, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    The alignment problem: How can machines learn human values? Atlantic Books, 2021

    Brian Christian. The alignment problem: How can machines learn human values? Atlantic Books, 2021

    work page 2021

  10. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Active preference optimization for sample efficient rlhf

    Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024

    2024

  12. [12]

    When personalization meets reality: A multi-faceted analysis of personalized preference learning

    Yijiang River Dong, Tiancheng Hu, Yinhong Liu, Ahmet \"U st \"u n, and Nigel Collier. When personalization meets reality: A multi-faceted analysis of personalized preference learning. arXiv preprint arXiv:2502.19158, 2025

    arXiv 2025

  13. [13]

    Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration

    Dylan J Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. arXiv preprint arXiv:2503.07453, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Personalized soups: Personalized large language model alignment via post-hoc parameter merging

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Provably feedback-efficient reinforcement learning via active reward learning

    Dingwen Kong and Lin Yang. Provably feedback-efficient reinforcement learning via active reward learning. Advances in Neural Information Processing Systems, 35: 0 11063--11078, 2022

    work page 2022

  16. [16]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv \'a ri. Bandit algorithms. Cambridge University Press, 2020

    2020

  17. [17]

    Aligning with logic: Measuring, evaluating and improving logical consistency in large language models

    Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vuli \'c , and Nigel Collier. Aligning with logic: Measuring, evaluating and improving logical consistency in large language models. arXiv preprint arXiv:2410.02205, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Learning word vectors for sentiment analysis

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142--150, 2011

    work page 2011

  19. [19]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  20. [20]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37: 0 116617--116637, 2024

    work page 2024

  21. [21]

    Personalizing reinforcement learning from human feedback with variational preference learning

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv:2408.10075, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  23. [23]

    Group robust preference optimization in reward-free rlhf

    Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems, 37: 0 37100--37137, 2024

    work page 2024

  24. [24]

    Dueling rl: Reinforcement learning with trajectory preferences

    Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with trajectory preferences. In International conference on artificial intelligence and statistics, pages 6263--6289. PMLR, 2023

    work page 2023

  25. [25]

    Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004. PMLR, 2023

    work page 2023

  26. [26]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  27. [27]

    Collective Choice and Social Welfare: An Expanded Edition

    Amartya Sen. Collective Choice and Social Welfare: An Expanded Edition. Harvard University Press, Cambridge, MA and London, England, 2017. ISBN 9780674974616. doi:10.4159/9780674974616. URL https://doi.org/10.4159/9780674974616

    doi:10.4159/9780674974616 2017

  28. [28]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Robust multi-objective controlled decoding of large language models

    Seongho Son, William Bankes, Sangwoong Yoon, Shyam Sundhar Ramesh, Xiaohang Tang, and Ilija Bogunovic. Robust multi-objective controlled decoding of large language models. arXiv preprint arXiv:2503.08796, 2025

    arXiv 2025

  30. [30]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325

    Pith/arXiv arXiv 2022

  31. [31]

    Aligning large language models with human: A survey

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL -constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL -constraint. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the ...

    work page 2024

  34. [34]

    Impact of representation learning in linear bandits

    Jiaqi Yang, Wei Hu, Jason D Lee, and Simon Shaolei Du. Impact of representation learning in linear bandits. In International Conference on Learning Representations, 2020

    work page 2020

  35. [35]

    Self-exploring language models: Active preference elicitation for online alignment

    Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Principled reinforcement learning with human feedback from pairwise or k-wise comparisons

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037--43067. PMLR, 2023

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.