Pith. sign in

REVIEW 3 major objections 4 minor 34 references

A Descriptive and Normative Theory of Human Beliefs in RLHF

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Human labelers' beliefs about an agent's capabilities shift the preferences they give, and the paper bounds the resulting loss in the learned policy's return.

desk verdict The belief-based model and priming study are solid and novel, but Theorem 4.3's regret bound is unproven and false as stated, so the paper's headline normative claim does not hold. read the letter →

arxiv 2506.01692 v2 pith:4P5XPPGT submitted 2025-06-02 cs.AI cs.LG

Sylee Dandekar , Shripad Deshmukh , Frank Chiu , W. Bradley Knox , Scott Niekum This is my paper
classification cs.AIcs.LG
keywords reinforcementlearningfromhumanfeedbackbelief-basedpreferencemodelagent-labelerdisagreementregret-basedpreferencesbeliefsaboutagentcapabilitiesprimingpolicyperformanceboundnormativeidealbelief
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that RLHF preference models should include the labeler's belief about what the trained agent will actually be able to do, not just the reward or optimal value of the trajectories. It introduces a belief-based preference model in which preferences are generated from the advantage of an imagined post-training policy, defines agent-labeler disagreement as the difference between the labeler's belief and the normative ideal, and proves a bound on the expected return of the final policy in terms of that disagreement. A human study shows that priming people to think a self-driving car is safe or unsafe significantly changes the preferences they give. Synthetic gridworld experiments show that assuming the agent is optimal is often worse than assuming its true, limited capability.

What carries the argument

The load-bearing object is the belief-based preference model: preference probability is a softmax over discounted advantages $A^{\pi_{\text{belief}}}$ under the labeler's imagined post-training policy, replacing the optimal advantage $A^*$ used in regret-based preferences. The proof of the main bound uses the standard policy performance-difference identity, which expresses the return gap between two policies as a discounted occupancy-weighted difference in Q-values; the disagreement parameter $\delta$ is meant to control the Q-value difference at the single state where the policies differ. The synthetic experiments operationalize capability as epsilon-greedy noise, and the human study operationalizes belief change through safe/unsafe priming videos.

What would settle it

A two-state MDP in which the labeler's belief agrees with the ideal at the measured state-action pair (so $\delta=0$) but ranks a second action at that state as much better would put a deterministic RLHF step on the wrong action while the bound promises zero loss; such an MDP would disprove the bound as stated.

Watch

Extended reading notes

Core claim

The paper claims that preferences in RLHF are not determined by the trajectory alone: labelers extrapolate from what they see to what the agent will do afterward, and that extrapolation depends on what they believe about the agent's capabilities. It formalizes this with a belief-based preference model in which preference probability is a softmax over discounted advantages under an imagined post-training policy, and it defines the normative ideal as the belief that yields the highest-return post-RLHF policy among all possible preference assignments. Under noiseless single-transition preferences, a tabular policy, and a deterministic RLHF step, the paper proves that a disagreement of size $\delta$ at one state-action pair degrades expected return by no more than $\delta/(1-\gamma)$. Gridworld results with epsilon-greedy agents show that assuming optimality is suboptimal when the agent's true noise is large, and the human study shows that priming shifts preferences as the belief model predicts.

Load-bearing premise

The bound rests on the assumption that a labeler's wrong belief at a single decision point can only nudge the learned policy by that same small amount, and cannot make a different, much worse action look better.

Editorial extensions

If this is right

  • In regret-based RLHF with finite or noisy preference data, labelers should be told about the agent's real limitations instead of being asked to assume optimal behavior.
  • When the agent's policy class is restricted, preferences generated under an optimality assumption can yield lower post-training returns than preferences generated under the true capability model.
  • The penalty for belief mismatch scales as $1/(1-\gamma)$, so long-horizon tasks are more sensitive to what labelers believe about the agent.
  • Priming is a practical lever: the human study shows that simply showing safe or unsafe driving footage moves the preferences labelers give.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A conservative reading: the paper's practical guidance stands even if the main bound needs a stronger definition of disagreement, because the human study and gridworld results do not depend on the proof's specific inequality.
  • A testable refinement would be to define disagreement as the maximum error over all actions at the differing state, rather than at a single state-action pair, which would make the same proof strategy produce a valid bound.
  • The priming result suggests a calibration loop: elicit labelers' capability beliefs directly, feed that estimate back into preference instructions, and check whether preference distributions shift as predicted.
  • The $1/(1-\gamma)$ factor makes belief alignment a first-order concern for long-horizon and safety-critical tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a belief-based preference model for RLHF in which a human labeler's preferences depend on a Q-function Q^{π_belief} representing the labeler's belief about the agent's post-training capabilities. It defines an 'ideal' belief Q^{π*_belief} as any belief that would lead to the best post-RLHF policy, and states Theorem 4.3, which claims that a single state-action disagreement δ between the labeler's belief and the ideal belief degrades the expected return by at most δ/(1−γ). The paper also reports synthetic gridworld experiments with ε-greedy capability mismatch and a human study with safe/unsafe priming, finding that priming changes preferences. The conclusions recommend aligning labeler beliefs with agent capabilities.

Significance. If Theorem 4.3 were correct, the paper would provide a clean theoretical justification for a new and plausible descriptive/normative account of human preferences in RLHF, and the human study would be a useful first demonstration that such beliefs are manipulable. The empirical work is valuable and the research question is timely. However, the central theoretical result is false as stated: the proof uses an inequality that connects the disagreement δ to the true value function Q^{π*_post} with no supporting argument, and a simple tabular counterexample shows the claimed regret bound does not hold. Because the abstract and conclusions present the bound as the main normative contribution, the soundness of the paper's central claim is not established.

major comments (3)
  1. [Section 4.2, Theorem 4.3, Eqs. (5) and (11)] The step (11) is the load-bearing part of the proof, and it is false. The disagreement δ is defined by δ = |Q^{π*_belief}(s',a') − Q^{π_belief}(s',a')|, a distance between two belief Q-functions, but the inequality replaces Q^{π*_post}(s',πδ_post(s')) with Q^{π*_post}(s',π*_post(s')) − δ. Nothing in Definition 4.2 or the proof connects either belief Q-function to Q^{π*_post}, the true value of the post-RLHF policy under the ground-truth reward. A small δ at a switched action therefore imposes no bound on how much worse the switched action is under the true value function. Concretely, let γ = 0.9 and let actions G and B at s' lead deterministically to states y and x. Set rewards so that from y the ground-truth-optimal continuation yields 10 and from x the optimal continuation yields 0.02, with r(s',G,y)=0.01 and r(s',B,x)=0. Let the ideal belief's imagined policy be poor at y and x, giving Q^{π*_belief}(s',G)=0.01 and Q^{π*_belief}(s',B)=0, while the corresponding post-RLHF policy is optimal after s', so Q^{π*_post}(s',G)=9.01 and Q^{π*_post}(s',B)=0.018. Let the human belief agree with the ideal belief except at (s',B), where Q^{π_belief}(s',B)=0.018; then δ = 0.018. The human prefers B at s' because 0.018 > 0.01, so πδ_post(s') = B. With the start distribution concentrated at s', the true loss is 8.992, far exceeding δ/(1−γ) = 0.18. Thus Eq. (4) is contradicted by the construction, not merely left unproven.
  2. [Corollary 4.4 and Appendix A] The proof for multiple disagreements is not a proof: it invokes a worst case in which 'the agent will be stuck at state (s'',a'') from the beginning of the episode until the end of the horizon,' without deriving how multiple local disagreements combine. With disagreements at k different state-action pairs, the two policies can differ at k states, and the performance-difference formula would involve a sum of terms weighted by state-visitation frequencies; these terms can exceed max_i δ_i/(1−γ). The corollary therefore does not follow from Theorem 4.3 even if Theorem 4.3 were repaired.
  3. [Section 4, normative ideal definition] The definition of Q^{π*_belief} as any belief that 'results in the highest expected return of the post-RLHF policy' makes the ideal belief outcome-dependent. Combined with Definition 4.2, which selects the minimum disagreement when multiple ideal beliefs exist, the quantifier structure in Theorem 4.3 is unclear: one fixed Q^{π*_belief} must be chosen, but the bound is stated in terms of a δ that is defined relative to the set of all ideal beliefs. The paper acknowledges that the ideal may require oracle knowledge, but this is not a minor caveat; it means the theorem does not provide an actionable characterization of which beliefs are good, and the central normative claim currently rests on an undefined or arbitrarily selected oracle object.
minor comments (4)
  1. [Section 1 and proof of Theorem 4.3] There are several typographical slips, including 'requires requires' in the Figure 1 caption and 'Schulman et. at.' in the text; these should be corrected.
  2. [Section 6] The paper should clarify whether the 'extremely high confidence' filtering rule was fixed before data collection and how many responses were discarded at that step, since this affects the reported p-values.
  3. [Section 6] The statistical analysis averages each participant's responses over items, but the number of items per participant and the consistency across items are not reported; a short paragraph on this would improve reproducibility.
  4. [Section 5, Table 1] The table contains formatting irregularities (e.g., '1 .47') and the relationship between ε and ε' would be easier to read if the diagonal and off-diagonal entries were visually separated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is not self-referential, though Theorem 4.3 contains a serious proof gap that is a correctness issue rather than a circularity.

full rationale

The paper's normative ideal is definitional rather than derived: Q^{π*_belief} is defined as any belief that produces the highest post-RLHF expected return, so the statement that ideal beliefs maximize return is true by construction. However, this definition does not by itself make the paper circular, because Theorem 4.3 attempts a nontrivial bound on how much a small belief discrepancy δ degrades the expected return relative to that benchmark. The main theoretical result is not circular; it is invalid. Inequality (11) of the proof treats the belief-space discrepancy δ from Eq. (5), defined as |Q^{π*_belief}(s',a') − Q^{πbelief}(s',a')|, as if it lower-bounded the true value gap Q^{π*_post}(s',πδ_post(s')) − Q^{π*_post}(s',π*_post(s')). No assumption in Definition 4.2 or elsewhere connects either belief Q-function to Q^{π*_post}, so the bound is unsupported. That is a mathematical error, not a circular reduction. The synthetic experiments manipulate labeler belief (ϵ') and agent capability (ϵ) externally and train with Contrastive Preference Learning, so the observed diagonal performance pattern is not a fitted parameter renamed as a prediction. The human study uses a randomized priming intervention and reports a statistically significant difference, so it is also not circular. The only self-citation of note is Knox et al. (2024), which shares authors with this paper and supplies the regret-based preference model as background; the belief-based model, theory, gridworld experiments, and human study do not reduce to that citation. Consequently, no load-bearing circular step can be quoted from the paper, and the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 6 assumptions · 3 invented entities

The central theory rests on the belief-based model, a strong assumption about the RLHF learning rule, and the existence of an ideal belief defined by its own outcome. The synthetic experiments add hyperparameters α and γ; the human study adds selection rules. No free parameter is fitted to external benchmarks.

free parameters (2)
  • Inverse temperature α of the belief-based preference model = ∞ in Theorem 4.3; 10 in GridWorld experiments
    Controls the noise of preference sampling. The theorem assumes noiseless preferences (α→∞), while the experiments use α=10, so the empirical validation is in a different regime.
  • Discount factor γ = 0.7 in GridWorld experiments
    Selected by hand for the synthetic environment. The theorem's bound scales as 1/(1−γ), but no specific γ is claimed.
assumptions (6)
  • domain assumption Preferences are generated by a softmax (Bradley-Terry) over summed advantages or returns (Eqs. 1-3).
    Standard in the RLHF literature and used to define all preference models in the paper.
  • domain assumption The belief-based preference model (Eq. 3) accurately captures how humans generate preferences in practice.
    The paper provides indirect evidence via the priming study, but this is a modeling assumption that underlies the theory.
  • domain assumption In Theorem 4.3, RLHF produces a deterministic policy that respects all preferences and satisfies π_post(s) = argmax_a Q^{π_belief}(s,a).
    Strong assumption about the learning algorithm and data coverage; not guaranteed for finite data or function approximators.
  • ad hoc to paper The normative ideal belief Q^{π*_belief} exists and always leads to the best achievable post-RLHF policy π*_post.
    Introduced only for the theory; the existence and properties of Q^{π*_belief} are not independently established.
  • domain assumption Epsilon-greedy noise in the GridWorld is a faithful stand-in for real agent performance limitations.
    The paper uses this as a proxy for limited data, weak function approximation, or actuator noise.
  • domain assumption Priming changes participants' beliefs about agent capabilities rather than other psychological factors.
    The sentiment analysis is a manipulation check but does not isolate beliefs from mood, risk attitude, or demand effects.
invented entities (3)
  • Agent capability belief Q^{π_belief}
    purpose: Models how a human labeler evaluates actions under an imagined agent policy instead of the optimal policy.
    Introduced as a new construct; no external falsifiable handle outside this paper.
  • Normative ideal belief Q^{π*_belief}
    purpose: Defines the belief that would lead to the best post-RLHF policy over a fixed trajectory set.
    Defined by the outcome it produces, so it is not independently observable or measurable.
  • Best post-RLHF policy π*_post
    purpose: Reference point for the regret bound; the policy with maximum expected return under any preference assignment over the fixed trajectory set.
    Defined relative to the finite preference set and learning rule; not observable outside the paper's construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Descriptive and Normative Theory of Human Beliefs in RLHF." pith.science (2026). https://pith.science/paper/4P5XPPGT

@misc{pith2026250601692,
  author       = {Pith},
  title        = {Pith review of: A Descriptive and Normative Theory of Human Beliefs in RLHF},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4P5XPPGT}},
  note         = {Machine review of arXiv:2506.01692}
}
read the original abstract

Human preferences in RLHF are typically modeled as a function of the human's reward function or corresponding optimal state-action values. In this work, we propose that human beliefs about the capabilities of the agent being trained also play a key role in preference generation. We examine two questions related to this hypothesis, one descriptive and one normative, respectively: Do human labelers' beliefs about agent capabilities affect the preferences that they provide? And what is the ideal set of beliefs about an agent -- and resulting preferences -- for humans to have? We propose a new preference model that incorporates human beliefs and provide a normative theory that bounds the error on the final learned policy based on the mismatch between the human's beliefs and an idealized set of beliefs. We then confirm via a human study that beliefs about agent capabilities do, in fact, significantly affect preferences and can be influenced through simple interventions. Additionally, we empirically show through synthetic experiments that it is often suboptimal for human preference labelers to assume agent optimality. Collectively, these results theoretically and empirically demonstrate how reducing the mismatch between human beliefs and agent capabilities can lead to more performant RLHF and point toward new best practices for RLHF practitioners.

Figures

Figures reproduced from arXiv: 2506.01692 by the authors.

Figure 1
Figure 1. Illustration of scenario in which preference for a more optimal partial path can lead to a worse post-RLHF policy. (Path 1): The car drives along the edge of a cliff but straight to the destination. This path takes less time but requires requires greater capabilities (both during and after) to avoid catastrophic outcomes. (Path 2): The car takes a longer path far away from the cliff, reducing risk. The dotted lines … view at source ↗
Figure 2
Figure 2. An MDP in which a safe and a risky action are available. If A πbelief(s0, arisk) > Aπbelief(s0, asafe) when A π ∗ belief(s0, arisk) ≤ A π ∗ belief(s0, asafe), labelers will erroneously prefer the partial tra￾jectory (s0, arisk, srisk) over (s0, asafe, ssafe) under the belief-based preference model in Equation 3. Here we present an example to illustrate the effect of agent￾labeler disagreement. Consider the simple MD… view at source ↗
Figure 3
Figure 3. 7x7 GridWorld with start state (in yellow), two terminal cliff states (in red) and one terminal goal state (in green). In each cell, we mark the reward incurred for reaching the state. We first collect 100 random trajectories (enough to guaran￾tee a nearly optimal-within-class post-RLHF policy) of the agent traversing the gridworld environment that terminate either in the goal state or in the cliff state. We denote … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: shows that immediately after priming, participants primed with the unsafe priming had lower sentiment scores regarding the car’s capabilities than the participants in the safe priming group, suggesting that priming had an impact on their beliefs [PITH_FULL_IMAGE:figur…
Figure 5
Figure 5. Figure 5: shows the format of the survey that participants saw when giving their responses [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: An example pair of trajectories as shown to human labelers. Both trajectories show the car driving along the same path for the same amount of time. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Demographics of human labelers 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages

  1. [1]

    Concrete problems in ai safety, 2016

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in ai safety, 2016. URL https://arxiv.org/abs/1606.06565

    arXiv 2016

  2. [2]

    Atas, M., Felfernig, A., Polat-Erdeniz, S., Popescu, A., Tran, T. N. T., and Uta, M. Towards psychology-aware preference construction in recommender systems: Overview and research issues. J. Intell. Inf. Syst., 57 0 (3): 0 467–489, December 2021. ISSN 0925-9902. doi:10.1007/s10844-021-00674-5. URL https://doi.org/10.1007/s10844-021-00674-5

    work page doi:10.1007/s10844-021-00674-5 2021

  3. [3]

    S., Goo, W., Nagarajan, P., and Niekum, S

    Brown, D. S., Goo, W., Nagarajan, P., and Niekum, S. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, 2019. URL https://api.semanticscholar.org/CorpusID:119111734

    work page 2019

  4. [4]

    Human irrationality: both bad and good for reward inference, 2021

    Chan, L., Critch, A., and Dragan, A. Human irrationality: both bad and good for reward inference, 2021. URL https://arxiv.org/abs/2111.06956

    arXiv 2021

  5. [5]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  6. [6]

    Dominance statistics: Ordinal analyses to answer ordinal questions

    Cliff, N. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114: 0 494--509, 11 1993. doi:10.1037/0033-2909.114.3.494

    doi:10.1037/0033-2909.114.3.494 1993

  7. [7]

    Carla: An open urban driving simulator

    Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. Carla: An open urban driving simulator. In Conference on robot learning, pp.\ 1--16. PMLR, 2017

    work page 2017

  8. [8]

    Dunn, O. J. Multiple comparisons among means. Journal of the American statistical association, 56 0 (293): 0 52--64, 1961

    work page 1961

Show all 34 references
  1. [9]

    and van Huyssteen, G

    Eiselen, R. and van Huyssteen, G. B. A comparison of statistical tests for likert-type data: The case of swearwords. Journal of Open Humanities Data, Oct 2023. doi:10.5334/johd.132

    2023 doi

  2. [10]

    Gilovich T, Griffin D, K. D. Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press, 2002

    2002

  3. [11]

    and Zhang, Y

    Gong, Z. and Zhang, Y. What is it you really want of me? generalized reward learning with biased beliefs about domain dynamics. Proceedings of the AAAI Conference on Artificial Intelligence, 34: 0 2485--2492, 04 2020. doi:10.1609/aaai.v34i03.5630

    2020 doi

  4. [12]

    R., Millman, K

    Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del R \' i o, J. F., Wiebe, M., Peterson, P., G \' e rard...

    2020 doi

  5. [13]

    B., and Sadigh, D

    Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D. Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevie...

    2024

  6. [14]

    Ai alignment: A comprehensive survey

    Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023

    2023 arXiv

  7. [15]

    Maps of bounded rationality: Psychology for behavioral economics

    Kahneman, D. Maps of bounded rationality: Psychology for behavioral economics. The American Economic Review, 93 0 (5): 0 1449--1475, 2003. ISSN 00028282. URL http://www.jstor.org/stable/3132137

    2003

  8. [16]

    Kakade, S. M. and Langford, J. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002. URL https://api.semanticscholar.org/CorpusID:31442909

    2002

  9. [17]

    A Survey of Reinforcement Learning from Human Feedback , 2023

    Kaufmann, T., Weng, P., Bengs, V., and H \"u llermeier, E. A Survey of Reinforcement Learning from Human Feedback , 2023

    2023

  10. [18]

    B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A

    Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A. G. Models of human preference for learning reward functions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=hpKJkVoThY

    2024

  11. [19]

    and Wallis, W

    Kruskal, W. and Wallis, W. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, pp.\ 583--621, 1952

    1952

  12. [20]

    and Rabin, M

    Köszegi, B. and Rabin, M. Revealed Mistakes and Revealed Preferences, pp.\ 193--209. Oxford Academic, 04 2008. ISBN 9780195328318. doi:10.1093/acprof:oso/9780195328318.003.0008

    2008

  13. [21]

    and Slovic, P

    Lichtenstein, S. and Slovic, P. The Construction of Preference. Cambridge University Press, 2006

    2006

  14. [22]

    and Roy, B

    Marklund, H. and Roy, B. V. Choice between partial trajectories: Disentangling goals from beliefs, 2024. URL https://arxiv.org/abs/2410.22690

    2024 arXiv

  15. [23]

    D., Tang, Y., Geist, M., Mesnard, T., Fiegel, C., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D

    Munos, R., Valko, M., Calandriello, D., Gheshlaghi Azar, M., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Fiegel, C., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D. J., Precup, D., and Piot, B. N ash learning from human feedback. In Sala...

    2024

  16. [24]

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapi...

    2024 arXiv

  17. [25]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to f...

    2022

  18. [26]

    Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  19. [27]

    Python: Version 3.11, 2023

    Python Software Foundation . Python: Version 3.11, 2023. URL https://www.python.org/

    2023

  20. [28]

    D., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY,...

    2024

  21. [29]

    Where do you think you re going?: Inferring beliefs about dynamics from behavior

    Reddy, S., Dragan, A., and Levine, S. Where do you think you re going?: Inferring beliefs about dynamics from behavior. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume ...

    2018

  22. [30]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp.\ 1889--1897. PMLR, 2015

    2015

  23. [31]

    Sutton, R. S. and Barto, A. G. Reinforcement Learning: A n Introduction . MIT Press, Cambridge, MA, 1998

    1998

  24. [32]

    Tran, T. N. T., Felfernig, A., and Tintarev, N. Humanized recommender systems: State-of-the-art and research issues. ACM Transactions on Interactive Intelligent Systems, 11: 0 1--41, 07 2021. doi:10.1145/3446906

    2021 doi

  25. [33]

    E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt , S

    Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt , S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J...

    2020

  26. [34]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019. URL https://api.semanticscholar.org/CorpusID:202660943

    1909 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.