Pith. sign in

REVIEW 2 major objections 6 minor 87 references

Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Preference signs, not link functions, provably train RL policies

desk verdict Sign-SZPO is a genuinely new zeroth-order estimator with a self-contained convergence proof; the main caveat is the global smoothness assumption on V(pi_theta), which the paper states plainly but whose scope is narrower than the abstract suggests. read the letter →

arxiv 2506.03066 v2 pith:THNIIJVD submitted 2025-06-03 cs.LG stat.ML

Qining Zhang , Lei Ying This is my paper
classification cs.LGstat.ML MSC 68T0590C4090C56
keywords preference-basedreinforcementlearningunknownlinkfunctionzeroth-orderoptimizationsigngradientfromhumanfeedback(RLHF)policymajorityvoteBradley-Terrymodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to settle whether a reinforcement-learning agent can provably maximize an unknown reward from human preferences without ever seeing the link function that turns return differences into preference probabilities. It claims the answer is yes, with a new algorithm, ZSPO, that reads only the sign of the value-function difference from a majority vote over batched trajectory comparisons and uses that sign to build an update direction positively correlated with the true policy gradient. If the claim holds, standard RLHF pipelines no longer need to commit to a specific preference model such as the Bradley-Terry logistic link; any monotone, anti-symmetric, smooth link function is covered by the same guarantee. The payoff is concrete: a polynomial convergence rate to a stationary policy, with the unknown-link penalty appearing only as two controllable terms — the inverse-link evaluation at $\sqrt{2/N}$ and the panelist distinguishability limit $\varepsilon_D^\star$.

What carries the argument

The mechanism that carries the argument is the sign-aligned ascent direction: a majority-vote estimate of the sign of the value-function difference combined with the random perturbation vector. The proof rests on three pieces: the near-linearization identity $V(\pi_{\theta+\mu v}) - V(\pi_\theta) \approx \mu\langle\nabla V(\theta), v\rangle$; the panelist distinguishability constant $\varepsilon_D^\star$ (the largest value gap that a panel comparing $D$-trajectory batches cannot reliably detect on average); and Khintchine-type bounds such as $\mathbb E_v |\langle \nabla V(\theta), v\rangle| = \sqrt{2/\pi}\,\|\nabla V(\theta)\|_2$ that turn sign agreement into a negative drift. Lemmas 4 and 5 bound the drift and the majority-vote sign error respectively, and together produce the telescoping sum for the theorem.

What would settle it

Take the two-step MDP in Appendix B.2.1 with a logistic link, batch size $D=1$, and a better policy whose advantage is small (e.g., $\varepsilon=0.1$); the expected panelist preference favors the worse policy, so running ZSPO with perturbation $\mu$ small enough that $\mu\langle\nabla V,v\rangle<\varepsilon_D^\star$ should stall at gradient norm $\approx\varepsilon_D^\star/\mu$. Repeating with a smooth linear-MDP control where $V$ is L-smooth should reproduce the promised $\sqrt{H/T}$ decay, isolating the distinguishability term and the smoothness assumption as the drivers of the bound.

Watch

Extended reading notes

Core claim

The central discovery is that for general MDPs with an unknown link function $\sigma$, the sign of the value function difference $V(\pi_{\theta+\mu v}) - V(\pi_\theta)$ is enough information to perform convergent policy optimization. ZSPO perturbs the policy parameter by a Gaussian vector $v_t$, collects $N$ pairs of trajectory batches of size $D$ from the original and perturbed policies, aggregates one-bit panelist preferences with a majority vote, and updates $\theta_{t+1} = \theta_t + \alpha_t\,\mathrm{sign}\big(\sum_{n=1}^N (o_{t,n}-1/2)\big) v_t$. The load-bearing approximation is that for small $\mu$, $V(\pi_{\theta+\mu v}) - V(\pi_\theta) \approx \mu \langle \nabla_\theta V(\pi_\theta), v\rangle$, so the estimated sign agrees with $\mathrm{sign}\langle \nabla_\theta V(\pi_\theta), v\rangle$ except on a controlled event; Lemma 4 converts this into a negative drift proportional to $\|\nabla_\theta V(\pi_\theta)\|_2$. The main theorem bounds the expected squared gradient norm of a randomly selected iterate by $\tilde O\big(\sqrt{Hd/T} + (1/\mu)(\varsigma^{-1}(\sqrt{2/N}) + \varepsilon_D^\star) + \mu d\big)$, and Corollary 1 tunes $\mu$ to obtain $\sqrt d\, \tilde O\big(\sqrt{H/T} + \max\{1/\sigma'(0),1\}N^{-1/4} + \sqrt H D^{-1/4}\big)$. This makes ZSPO, to the authors' knowledge, the first RLHF algorithm with provable guarantees for general MDPs that does not require knowing the link function.

Load-bearing premise

The policy's expected return must be a smooth function of its network weights, so that small parameter nudges change the value almost linearly; without this, preference signs need not track the true gradient and the drift argument collapses.

Editorial extensions

If this is right

  • RLHF pipelines can drop the known-link assumption: logistic, probit, linear, and step-like link functions all fall under the same monotone anti-symmetric smooth class, so the algorithm avoids the preference-model mismatch that degrades DPO- and PPO-style baselines in the paper's experiments.
  • The unknown link adds only two controlled gaps to the rate: the term $\varsigma^{-1}(\sqrt{2/N})$ shrinks as the number of queried trajectory pairs grows, and $\varepsilon_D^\star$ shrinks roughly as $H/\sqrt D$ with the trajectory batch size.
  • The perturbation distance $\mu$ cannot be made arbitrarily small as in classic zeroth-order optimization; it must keep the perturbed policy distinguishable from the original, giving a concrete tuning rule $\mu^2 = \Theta(d^{-1}\max\{1/\sqrt N, H/\sqrt D\})$.
  • When the value function itself can be queried (no human feedback), the same sign-only update forms a new zeroth-order method matching the $\sqrt{d/T}$ rate of ZO-SGD in the limit $N,D\to\infty$.
  • Panelist quality enters the guarantee explicitly through $\sigma'(0)$: a sharper panelist (larger derivative at zero) directly improves the $N^{-1/4}$ coefficient.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sign-alignment mechanism is the true source of progress, then any feedback channel that reliably reports which policy is better — including noisy or ordinal crowd votes — should inherit the same convergence structure, so long as the reported sign is unbiased; this suggests treating sign accuracy as the central quantity when transferring ZSPO beyond the batched-preference model.
  • The distinguishability limit $\varepsilon_D^\star$ is environment-dependent (transition randomness and reward skewness), not just panelist quality; a practical consequence is that the minimal useful perturbation size should be calibrated per deployment, and algorithms that adapt $\mu$ against an online estimate of sign reliability would be a natural extension the paper does not explore.
  • One can test whether sign information is truly sufficient in harder function classes: for neural policies where $V$ is not L-smooth in $\theta$, the linearization in Eq. (3) fails exactly where the value gap is small, predicting a stall governed by $\varepsilon_D^\star/\mu$ — a pattern worth checking empirically against the smooth-tabular results reported here.
  • The single-bit nature of the transmitted sign suggests a federated or communication-limited variant in which workers send only preferred-or-not votes; such a variant would presumably keep the same rate up to the majority-vote term, which is an immediate but unproven extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper studies preference-based reinforcement learning (RLHF) with an unknown link function connecting trajectory returns to human preference probabilities. The authors propose Sign-SZPO (called ZSPO in the body), a zeroth-order policy optimization algorithm that, at each iteration, perturbs the policy parameter, samples trajectory batches under the current and perturbed policies, queries a panel for pairwise preferences, aggregates the one-bit feedback by majority vote to estimate the sign of the value function difference, and updates the parameter along a direction claimed to be positively correlated with the true policy gradient. The main theoretical result (Theorem 1) gives a polynomial convergence rate to a stationary policy of the true value function, with a rate involving the horizon H, dimension d, iterations T, number of comparison batches N, batch size D, and a new 'panelist distinguishability' constant ε*_D; Corollary 1 provides a concrete rate with explicit dependence on the link function's derivative at zero. The proof introduces a drift decomposition into D1 (sign alignment) and D2 (majority-vote estimation error) and avoids the standard smoothing-function framework. Experiments on a stochastic GridWorld compare ZSPO with RM+PPO, DPO, Online DPO, and ZPG under logistic and linear link functions.

Significance. If the main result holds, this is a meaningful step in RLHF theory: it provides, to the authors' claim, the first provable convergence guarantee for general MDPs without knowing the link function, interpolating between known-link and agnostic preference models. The algorithm is simple and the new proof framework (using the value function as Lyapunov function with event-based sign-alignment analysis) is a genuine technical novelty. The paper ships complete proofs of Proposition 1, Theorem 1, and Corollary 1 in the appendix, together with a detailed experimental setup; the proofs are internally consistent and the drift analysis is transparent. The distinguishability constant ε*_D is a useful conceptual tool for characterizing when batched preferences can reveal value function order. However, the scope of the central result is narrower than the abstract's 'mild conditions' suggests, because the sign-alignment mechanism rests on a global smoothness assumption on the policy parameterization. With the fixes proposed below (clarifying the scope of Assumption 3 and correcting the definition of ε*_D), the contribution would be solid.

major comments (2)
  1. [Section 5.1, Assumption 3; Appendix D.3 (Lemma 4) and D.4 (Lemma 5)] The sign-alignment mechanism is load-bearing: Eq. (3) and the event definitions E_{v,+}, E_{v,-}, E_{v,0} in Lemma 4 use the quadratic Taylor bound from Lemma 3 to conclude that sign[V(π_{θ+μv}) − V(π_θ)] = sign[⟨∇V(π_θ), v⟩] on the large-inner-product events. This requires V(π_θ) to be globally L-smooth in θ. For the motivating class of neural policy networks with ReLU activations, this smoothness can fail (∇_θ V may be discontinuous or lack a global Lipschitz constant), and then the negative drift in D1 is not guaranteed. The paper's only justification (Section 5.1) is that linearly realizable MDPs satisfy the assumption when the policy parameterization is smooth; Appendix B.3 discusses local-versus-global convergence but does not address this restriction. Please either (i) prove the sign-alignment under a weaker condition (e.g., using a smoothed value function), or (ii) explicitly delimit Theorem 1 to smoothly parameterized policies (e.g., tabular softmax, Gaussian policies) and add a limitation paragraph on non-smooth networks. Without one of these, the abstract's 'provably converges ... under mild conditions' overstates what is shown.
  2. [Definition 1, Section 5.1] The phrase 'maximum constant ε' is formally incorrect. The set of ε such that 'for all policies with V(π1)−V(π0) ≥ ε, E[ς] ≥ 1/2 ς((V(π1)−V(π0))/2)' is upward-closed, so its maximum is +∞; the proof of Proposition 1 in fact establishes ε*_D ≤ ε0, i.e., an upper bound on the infimum threshold. Please redefine ε*_D as the infimum (or 'critical gap') such that the property holds, and verify that Theorem 1's rate, which uses ε*_D, remains well-defined. This is a formal issue in the statement of the main theorem, although the intended meaning is clear from the proof.
minor comments (6)
  1. [Title and Abstract] The arXiv title is 'Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function' but the paper's header and abstract refer to 'ZSPO' and 'Provable Reinforcement Learning from Human Feedback with an Unknown Link Function'. Please unify the naming.
  2. [Algorithm 1, line 8] The majority vote sign is not defined when Σ_n(o_{t,n} − 1/2) = 0, which is possible when N is even. Define sign(0) explicitly (e.g., 0 or +1); with sign(0) = 0 the update is still well-defined and the Hoeffding bound in Lemma 5 covers ties. The experiments in Appendix A define per-panelist ties as o = 0, but the aggregate tie is not specified.
  3. [Appendix D.2] The text preceding Lemma 5 says 'the panelists' error term D1' but the term being bounded is D2; please correct this typo.
  4. [Corollary 1 and Appendix E] The proof in Appendix E uses a condition such as N ≥ 8L²/(σ'(0))² to control ς^{-1}(√(2/N)) via smoothness; this condition is not stated in Corollary 1. Please state the condition explicitly or use a bound that avoids an implicit restriction on N.
  5. [Definition 1] Consider renaming ε*_D to 'critical distinguishability gap' or similar, to avoid the maximum/minimum confusion noted in Major Comment 2.
  6. [References and Figure 1 caption] The related-work section contains the incomplete citation '(Greene, 2000,?)', and Figure 1's caption 'averaged over 103 repetitions' should be '10^3 repetitions' or '1000 repetitions'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence proof is self-contained from stated assumptions, and self-citations are contextual rather than load-bearing.

full rationale

The paper's derivation chain is self-contained. ZSPO's sign-based update (Algorithm 1, lines 2-9) is analyzed under Assumptions 1-3 and Definition 1. Lemma 4 proves a negative drift E[D1|F_t] >= sqrt(2/pi) ||grad V|| - mu L d using only the L-smoothness of the value function (Assumption 3) via the standard Taylor bound in Lemma 3, with the events E_{v,+}, E_{v,-}, E_{v,0} partitioning the Gaussian perturbation space. Lemma 5 bounds the majority-vote sign-error term D2 using Hoeffding's inequality and the definition of the panelist distinguishability epsilon*_D; it does not fit any parameter. Proposition 1 independently proves epsilon*_D = O~(H/sqrt(D)) from concentration of average trajectory rewards, so the distinguishability term in Theorem 1 is an analyzed quantity rather than an assumed or fitted input. The majority-vote term ς^{-1}(sqrt(2/N)) is converted in Corollary 1 to a parameter-free O(1/(σ'(0) sqrt(N))) bound using smoothness of the inverse deviation function near the origin, again by analysis. Hyperparameters mu and alpha are chosen according to explicit schedules (alpha_t = Theta(sqrt(H/(d t))); mu^2 = Theta(d^{-1} max{1/sqrt(N), H/sqrt(D)})) and are not fitted to the target gradient. The self-citations to ZPG (Zhang and Ying, 2025) are contextual: they motivate the zeroth-order approach, describe the known-link baseline, and appear as one of several citations for the standard smoothness assumption. The role of Assumption 3 is stated directly in the paper and is not imported from a self-citation chain. The legitimate robustness concern that Assumption 3 may fail for non-smooth policy parameterizations such as ReLU networks, noted also in Appendix B.3, is a correctness or modeling limitation, not a circularity: the theorem's statement explicitly conditions on Assumption 3 and derives the result from it rather than presupposing the conclusion. Accordingly, no step reduces, by construction or by self-citation, to the paper's own inputs, and the paper merits a circularity score of 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or algorithmic entities beyond the estimator; the key defined quantity ε*_D is a problem-dependent constant bounded in Proposition 1, not an invented entity.

free parameters (2)
  • perturbation distance μ_t = μ² = Θ(d^{-1} max{ς^{-1}(√(2/N)), ε*_D}) in Eq. (5); Corollary 1 uses μ² = Θ(d^{-1} max{N^{-1/2}, H D^{-1/2}})
    Algorithm hyperparameter; optimal choice in Eq. (5) depends on unknown ς and ε*_D, though Corollary 1 gives a parameter-free choice.
  • learning rate α_t = α_t = Θ(√(H/(dt)))
    Standard step-size schedule in zeroth-order analysis; not fitted to data but chosen for the proof.
assumptions (5)
  • domain assumption Preference feedback is generated by a fixed link function σ of the average reward difference between trajectory batches (Eq. 2), with σ strictly increasing, σ(0)=1/2, σ(-x)=1-σ(x) (Assumption 1).
    Defines the entire preference model; the paper relaxes knowledge of σ but still assumes its existence and monotonicity. If human preferences are not monotone in true reward, sign inference can fail.
  • domain assumption σ is L-smooth and σ'(0)>0 (Assumption 2).
    Used to bound ς^{-1}(√(2/N)) = O(1/(σ'(0)√N)) and to control distinguishability; inverse smoothness proof in Appendix E relies on it.
  • domain assumption Value function V(π_θ) is L-smooth as a function of the policy parameter θ (Assumption 3).
    Used in Lemma 3 for Taylor expansion and in the drift analysis. Not guaranteed for arbitrary neural policies; authors note linearly realizable MDPs satisfy it.
  • domain assumption Trajectory rewards are bounded in [0,H] and trajectory returns r(τ) define the value function.
    Needed for Hoeffding concentration in Proposition 1 and boundedness of V.
  • standard math Hoeffding's inequality, Khintchine's inequality, and standard smooth-function inequalities hold.
    Background results used in Lemmas 1-5 and Proposition 1; no issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function." pith.science (2026). https://pith.science/paper/THNIIJVD

@misc{pith2026250603066,
  author       = {Pith},
  title        = {Pith review of: Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/THNIIJVD}},
  note         = {Machine review of arXiv:2506.03066}
}
read the original abstract

The link function, which characterizes the relationship between the preference for two trajectories and their returns, is a crucial component in designing RL algorithms that learn from preference feedback. Most existing methods, both theoretical and empirical, assume that the link function is known (often a logistic function based on the Bradley-Terry model), which is arguably restrictive given the complex nature of preferences, especially those of humans. To avoid mis-specification, this paper studies preference-based RL with an unknown link function and proposes a novel zeroth-order policy optimization algorithm called Sign-SZPO. Unlike typical zeroth-order methods, which rely on the known link function to estimate the value function differences and form a gradient estimator, Sign-SZPO only estimates the sign of the value function difference. It then constructs a parameter update direction that is positively correlated with the true policy gradient, eliminating the need to know the link function exactly. Under mild conditions, Sign-SZPO provably converges to a stationary policy with a polynomial rate in the number of policy iterations and trajectories per iteration. Empirical evaluations further demonstrate the robustness of Sign-SZPO under link function mis-specifications.

Figures

Figures reproduced from arXiv: 2506.03066 by the authors.

Figure 1
Figure 1. GridWorld: (a) comparison of ZSPO and baselines without link function mismatch, and (b) comparison of ZSPO and baselines with link function mismatch. Results are averaged over 103 repetitions, and shaded areas are 95% confidence intervals. We then experimented with the setting with a link function mismatch, i.e., the true link function is linear while the baseline algorithms adopt a logistic link function, as shown … view at source ↗
Figure 2
Figure 2. a two-step MDP example: the reward depends on the state at the second planning step with two [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

87 extracted references · 50 canonical work pages

  1. [1]

    Reinforcement learning: An introduction, volume 1

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  2. [2]

    Controlled experiments on the web: survey and practical guide

    Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18: 0 140--181, 2009

    2009

  3. [3]

    Deep reinforcement learning from human preferences

    Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  4. [4]

    Al Sallab, Senthil Yogamani, and Patrick Pérez

    Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23 0 (6): 0 4909--4926, 2022. doi:10.1109/TITS.2021.3054625

    arXiv 2022

  5. [5]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedbac...

    2022

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  7. [7]

    Dynamic programming and stochastic control processes

    Richard Bellman. Dynamic programming and stochastic control processes. Information and Control, 1 0 (3): 0 228--239, 1958. ISSN 0019-9958. doi:https://doi.org/10.1016/S0019-9958(58)80003-0

    doi:10.1016/s0019-9958(58)80003-0 1958

  8. [8]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

Show all 87 references
  1. [9]

    Inverse reward design

    Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  2. [10]

    Reward design with language models

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023

    2023 arXiv

  3. [11]

    Defining and characterizing reward gaming

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, volume 35, pages 9460--9471. Curran Associates, Inc., 2022

    2022

  4. [12]

    Scaling laws for reward model overoptimization in direct alignment algorithms

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024 a

    2024 arXiv

  5. [13]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  6. [14]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728--53741. Curran As...

    2023

  7. [15]

    Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference

    Qining Zhang and Lei Ying. Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  8. [16]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  9. [17]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J \'e r \'e my Scheurer, Javier Rando Ramirez, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Trans....

    2023

  10. [18]

    Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Forty-fi...

    2024

  11. [19]

    From r to q^ * : Your language model is secretly a q-function

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q^ * : Your language model is secretly a q-function. In First Conference on Language Modeling, 2024 b

    2024

  12. [20]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. ISSN 00063444, 14643510

    1952

  13. [21]

    Random utility theory for social choice

    Hossein Azari, David Parks, and Lirong Xia. Random utility theory for social choice. Advances in Neural Information Processing Systems, 25, 2012

    2012

  14. [22]

    Modeling ordered choices: A primer, 2010

    WH Greene. Modeling ordered choices: A primer, 2010

    2010

  15. [23]

    Econometric analysis 4th edition

    William H Greene. Econometric analysis 4th edition. International edition, New Jersey: Prentice Hall, pages 201--215, 2000

    2000

  16. [24]

    Sensory evaluation of food: principles and practices

    Harry T Lawless and Hildegarde Heymann. Sensory evaluation of food: principles and practices. Springer Science & Business Media, 2010

    2010

  17. [25]

    Sensory evaluation techniques

    Morten C Meilgaard, B Thomas Carr, and Gail Vance Civille. Sensory evaluation techniques. CRC press, 1999

    1999

  18. [26]

    Nash learning from human feedback

    Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, C\^ o me Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J Mankowitz, Doina Precup,...

    2024

  19. [27]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intellige...

    2024

  20. [28]

    Models of human preference for learning reward functions

    Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro G Allievi. Models of human preference for learning reward functions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    2024

  21. [29]

    Reinforcement learning from human feedback without reward inference: Model-free algorithm and instance-dependent analysis

    Qining Zhang, Honghao Wei, and Lei Ying. Reinforcement learning from human feedback without reward inference: Model-free algorithm and instance-dependent analysis. Reinforcement Learning Journal, 2024 b

    2024

  22. [30]

    Preference-based online learning with dueling bandits: A survey

    Viktor Bengs, Robert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hullermeier. Preference-based online learning with dueling bandits: A survey. Journal of Machine Learning Research, 22 0 (7): 0 1--108, 2021

    2021

  23. [31]

    Is rlhf more difficult than standard rl? arXiv preprint arXiv:2306.14111, 2023

    Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rl? arXiv preprint arXiv:2306.14111, 2023

    2023 arXiv

  24. [32]

    A survey of reinforcement learning from human feedback

    Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke H \"u llermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 2023

    2023

  25. [33]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835--10866. PMLR, 23--29 Jul 2023

    2023

  26. [34]

    Model-free preference-based reinforcement learning

    Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-free preference-based reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30 0 (1), Mar. 2016. doi:10.1609/aaai.v30i1.10269

    2016 doi

  27. [35]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  28. [36]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    2024 arXiv

  29. [37]

    Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF . In Forty-first International Conference on Machine Learning, 2024

    2024

  30. [38]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  31. [39]

    Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL -constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL -constraint. In Forty-first International Conference on Machine Learning, 2024

    2024

  32. [40]

    Slic-hf: Sequence likelihood calibration with human feedback

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  33. [41]

    Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, and R. Srikant. Exploration-driven policy optimization in RLHF : Theoretical insights on efficient data utilization. In Forty-first International Conference on Machine Learning, 2024

    2024

  34. [42]

    Dueling rl: Reinforcement learning with trajectory preferences

    Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with trajectory preferences. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 6263--6...

    2023

  35. [43]

    Lee, and Wen Sun

    Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable offline preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024 a

    2024

  36. [44]

    Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D. Lee. Provable reward-agnostic preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  37. [45]

    Principled reinforcement learning with human feedback from pairwise or k-wise comparisons

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 4303...

    2023

  38. [46]

    Provably feedback-efficient reinforcement learning via active reward learning

    Dingwen Kong and Lin Yang. Provably feedback-efficient reinforcement learning via active reward learning. In Advances in Neural Information Processing Systems, volume 35, pages 11063--11078. Curran Associates, Inc., 2022

    2022

  39. [47]

    Making RL with preference-based feedback efficient via randomization

    Runzhe Wu and Wen Sun. Making RL with preference-based feedback efficient via randomization. In The Twelfth International Conference on Learning Representations, 2024

    2024

  40. [48]

    Reinforcement learning with human feedback: Learning dynamic choices via pessimism

    Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438, 2023

    2023 arXiv

  41. [49]

    PARL : A unified framework for policy alignment in reinforcement learning from human feedback

    Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. PARL : A unified framework for policy alignment in reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024

    2024

  42. [50]

    A framework for partially observed reward-states in rlhf

    Chinmaya Kausik, Mirco Mutti, Aldo Pacchiano, and Ambuj Tewari. A framework for partially observed reward-states in rlhf. arXiv preprint arXiv:2402.03282, 2024

    2024 arXiv

  43. [51]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046, 2024

    2024 arXiv

  44. [52]

    Preference-based reinforcement learning with finite-time guarantees

    Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. In Advances in Neural Information Processing Systems, volume 33, pages 18784--18794. Curran Associates, Inc., 2020

    2020

  45. [53]

    Zeroth-order optimization meets human feedback: Provable learning via ranking oracles

    Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. In The Twelfth International Conference on Learning Representations, 2024 a

    2024

  46. [54]

    Interactively optimizing information retrieval systems as a dueling bandits problem

    Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 1201–1208, New York, NY, USA, 2009. Association for Computing M...

    2009

  47. [55]

    Beat the mean bandit

    Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 241--248, 2011

    2011

  48. [56]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International ...

    2024

  49. [57]

    Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation

    Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceed...

    2022

  50. [58]

    A theoretical analysis of nash learning from human feedback under general kl-regularized preference

    Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv e-prints, pages arXiv--2402, 2024

    2024

  51. [59]

    Extragradient preference optimization (egpo): Beyond last-iterate convergence for nash learning from human feedback

    Runlong Zhou, Maryam Fazel, and Simon S Du. Extragradient preference optimization (egpo): Beyond last-iterate convergence for nash learning from human feedback. arXiv preprint arXiv:2503.08942, 2025

    2025 arXiv

  52. [60]

    Direct nash optimization: Teaching language models to self-improve with general preferences

    Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024

    2024 arXiv

  53. [61]

    Iterative nash policy optimization: Aligning llms with general preferences via no-regret learning

    Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning llms with general preferences via no-regret learning. arXiv preprint arXiv:2407.00617, 2024 c

    2024 arXiv

  54. [62]

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming

    Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23 0 (4): 0 2341--2368, 2013

    2013

  55. [63]

    Random gradient-free minimization of convex functions

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Found. Comput. Math., 17 0 (2): 0 527–566, apr 2017. ISSN 1615-3375. doi:10.1007/s10208-015-9296-2

    2017 doi

  56. [64]

    Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications

    Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Procee...

    2018

  57. [65]

    Zeroth-order stochastic variance reduction for nonconvex optimization

    Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 b

    2018

  58. [66]

    A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization

    Hanqin Cai, Yuchen Lou, Daniel Mckenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pag...

    2021

  59. [67]

    On the information-adaptive variants of the admm: an iteration complexity perspective

    Xiang Gao, Bo Jiang, and Shuzhong Zhang. On the information-adaptive variants of the admm: an iteration complexity perspective. Journal of Scientific Computing, 76: 0 327--363, 2018

    2018

  60. [68]

    Fine-tuning language models with just forward passes

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems, volume 36, pages 53038--53075. Curran Associates, Inc., 2023

    2023

  61. [69]

    Evolutionsstrategie

    Ingo Rechenberg. Evolutionsstrategie. Optimierung technischer Systeme nach Prinzipien derbiologischen Evolution, 1973

    1973

  62. [70]

    Evolution strategies as a scalable alternative to reinforcement learning

    Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017

    2017 arXiv

  63. [71]

    Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents

    Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. Advances in neural information processing systems, 31, 2018

    2018

  64. [72]

    o r \'e nyi, Paul Weng, Weiwei Cheng, and Eyke H \

    R \'o bert Busa-Fekete, Bal \'a zs Sz \"o r \'e nyi, Paul Weng, Weiwei Cheng, and Eyke H \"u llermeier. Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine learning, 97: 0 327--351, 2014

    2014

  65. [73]

    Preference-based policy learning

    Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases, pages 12--27, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg

    2011

  66. [74]

    sign SGD : Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. sign SGD : Compressed optimisation for non-convex problems. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages...

    2018

  67. [75]

    sign SGD via zeroth-order oracle

    Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. sign SGD via zeroth-order oracle. In International Conference on Learning Representations, 2019

    2019

  68. [76]

    Thurstone

    Louis L. Thurstone. A law of comparative judgment. Psychological Review, 34 0 (4): 0 273, 1927

    1927

  69. [77]

    Kenneth E. Train. Discrete choice methods with simulation. Cambridge university press, 2009

    2009

  70. [78]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018

    2018

  71. [79]

    Online rl in linearly q^ -realizable mdps is as easy as in linear mdps if you learn what to ignore

    Gellert Weisz, Andr\' a s Gy\" o rgy, and Csaba Szepesvari. Online rl in linearly q^ -realizable mdps is as easy as in linear mdps if you learn what to ignore. In Advances in Neural Information Processing Systems, volume 36, pages 59172--59205. Curran Associates, Inc., 2023

    2023

  72. [80]

    Sample-efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting

    Gen Li, Yuxin Chen, Yuejie Chi, Yuantao Gu, and Yuting Wei. Sample-efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting. In Advances in Neural Information Processing Systems, volume 34, pages 16671--16685. Curran Associates, Inc., 2021

    2021

  73. [81]

    Provably efficient reinforcement learning with linear function approximation

    Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137--2143. PMLR, 2020

    2020

  74. [82]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  75. [83]

    Bangrui Chen and Peter I. Frazier. Dueling bandits with weak regret. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 731--739. PMLR, 06--11 Aug 2017

    2017

  76. [84]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  77. [85]

    Convex Optimization

    Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge Univ. Press, New York, NY, 2004

    2004

  78. [86]

    Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition

    Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases, pages 795--811, Cham, 2016. Springer International Publishing. ISBN 978-...

    2016

  79. [87]

    On the global convergence rates of softmax policy gradient methods

    Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6820--6...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.