Pith. sign in

REVIEW 2 major objections 5 minor 39 references

Human pairwise preferences under limited attention cannot generally reveal true reward rankings, and learning is limited by attended information rather than label count.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 16:46 UTC pith:S4KIC3SA

load-bearing objection Clean theory of attention-limited pairwise labels: non-identification, ranking reversal under heterogeneous β, and β^{2} sample complexity, with solid proofs and two supportive public-data case studies. the 2 major comments →

arxiv 2607.04590 v1 pith:S4KIC3SA submitted 2026-07-06 cs.AI

Attention Limited Reward Learning

Wenqian Xing This is my paper
classification cs.AI
keywords reward learningBradley-Terryrational inattentionRLHFpairwise comparisonsattention-limited measurementcyclic preference datascalable oversight
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard reward learning from human comparisons treats each vote as a noisy readout of a true value difference. This paper models each vote instead as the output of a low-capacity attention channel: the observed log-odds equal a default tendency plus an attention multiplier times the deliberative reward gap. Under that channel, passive labels cannot separate reward from attention or defaults, and when attention varies across pairs the usual Bradley-Terry fit can reverse the true ranking even though every pairwise majority is correct. What determines sample complexity is not the number of labels but the total attended information they carry, which scales with the square of the attention multipliers. Case studies on Chatbot Arena votes and on a perceptual comparison task show the predicted cyclic signature that no scalar reward can represent, and that response times and gaze carry gap information that the binary labels themselves do not. The practical claim is that weak preference signals may mean evaluation difficulty rather than genuine indifference.

Core claim

Passive pairwise comparison data cannot generally distinguish reward, attention, and default tendencies; heterogeneous attention can make the population Bradley-Terry solution reverse the deliberative ranking even when every observed majority points the right way; and reward recovery is governed by total attended information, which scales as the square of the attention multipliers, not by the raw number of labels.

What carries the argument

The attention-scaled comparison channel: observed log-odds equal a default term plus an attention multiplier times the deliberative reward gap. This reduced form produces cycle obstructions to scalar representability, a weighted projection limit for Bradley-Terry fitting, rank-one local information matrices, and sample-complexity lower bounds controlled by attended information.

Load-bearing premise

The claim rests on the premise that real human comparison labels are well described by a reduced-form channel that multiplies the true reward gap by a query-specific attention weight and adds a default term.

What would settle it

Find a large human preference graph in which the cyclic energy of the log-odds field is statistically indistinguishable from sampling noise under a fitted Bradley-Terry model, or a controlled comparison task in which response times and gaze carry no additional information about gap magnitude once the binary labels are known.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that standard Bradley–Terry reward learning from pairwise human comparisons systematically misreads attention-limited labels. Motivated by Shannon rational inattention, it posits a reduced-form channel ℓ_z = η_z + β_z Δ*_z in which observed log odds mix deliberative reward gaps with query-specific attention and defaults. From this channel it derives: a cycle criterion for scalar representability (Prop. 1); a local projection characterization of the population BT fit that can reverse deliberative rankings even when every pairwise majority is correct (Prop. 2, Example 1); observational non-identification of reward, attention, and defaults from passive labels (Prop. 3); entropy–information separation and rank-one Fisher matrices (Props. 4–5); KL/Fano sample-complexity bounds governed by total attended information rather than label count (Prop. 6, Thm. 1); and a one-eighth-law KL price of cyclic energy discarded by any scalar reward (Prop. 7). Two case studies check signatures: excess cyclic energy in Chatbot Arena votes, and response-time/gaze information about gap magnitude absent from labels in a perceptual task.

Significance. If the reduced-form channel is a useful description of alignment-relevant feedback, the results are load-bearing for RLHF practice: they separate hard-because-close from hard-because-hidden comparisons, show that BT can recover the wrong ranking under heterogeneous attention, and reframe sample complexity as attended information rather than annotation volume. The theoretical core is cleanly executed: appendix proofs for Props. 1–7, Lemma 1, and Theorem 1 are elementary and complete (cycle telescoping, IFT for the local projection, outer-product Fisher matrices, Pinsker/Fano, Hodge decomposition). The one-eighth law is a parameter-free second-order prediction that matches Arena misfit to within about 10% even outside the small-ε regime. The paper is explicit about scope (Remark 1, Section 5 caveats) and states falsifiable geometric and process signatures rather than fitting the channel to produce the reversal. That combination of identification theory, information bounds, and priced cyclic loss is a genuine contribution to the foundations of preference-based alignment.

major comments (2)
  1. [Section 2.2, Definition 1, Remark 1] Definition 1 / Remark 1: The entire identification, projection, and information analysis is conditional on the reduced-form marginal channel ℓ_z = η_z + β_z Δ*_z. Lemma 1 only yields a conditional logit given the realized evidence state; Remark 1 correctly notes that the marginal label law equals this form only in a special environment. For alignment-relevant queries, deliberative Δ* typically requires integrating over unresolved uncertainty, so the gap between conditional and marginal can be material. The manuscript should either (i) state sufficient conditions under which the reduced form is a good approximation to the marginal choice probability for the intended RLHF setting, or (ii) more sharply bound which of Props. 1–7 survive under qualitatively different attention models (e.g., feature-based or threshold attention). Without that, the transfer claim in the abstract and conclusion
  2. [Section 5.1, Abstract] Section 5.1 / Abstract: The Arena analysis cleanly rejects every scalar score (bootstrap p = 0.008; LR p = 0.007) and prices the misfit with Prop. 7 to ratio 0.89. The paper itself notes that aggregation over prompts and annotators can generate cyclic energy without per-query attention heterogeneity. That alternative is not quantified (e.g., by prompt-stratified cycle energy or annotator-level Hodge residuals). Given that the abstract presents the result as exhibiting the predicted attention signature, the manuscript should either add a simple stratification check or rephrase the abstract/conclusion so that the finding is stated as rejection of scalar representability consistent with, but not diagnostic of, heterogeneous attention—matching the more careful language already in the last paragraph of §5.1.
minor comments (5)
  1. [Figure 1, Example 1] Figure 1 caption and Example 1: the figure uses R*_A(2), R*_B(1), R*_C(0) while the text of Example 1 uses the same values; the figure also shows β_BC = 5ε producing ℓ_BC = 5ε, which is consistent, but the cycle sum annotation 'ε + 5ε − ε ≠ 0' would be clearer if the oriented edges were labeled with signs matching the chosen orientation in the incidence matrix.
  2. [Proposition 2, Example 1] Proposition 2 is stated for ∥ℓ∥_∞ ≤ ε with remainder O(ε³). Example 1 notes that the exact FOCs preserve the reversal at ε = 0.4. A short remark on the range of ε where the ranking of r† remains reversed (or a one-line numerical check) would help readers judge relevance outside the asymptotic regime.
  3. [Section 5.2] Section 5.2: the mutual-information numbers (0.33 bits for sign, 0.0001 for magnitude from labels; 0.035 bits from RT) are useful; please state the estimator (e.g., histogram / KSG) and the permutation procedure more explicitly so the 0.035 figure is reproducible from the public aDDM data alone.
  4. [Related work] Related work: the connection to combinatorial Hodge ranking (Jiang et al.) is well credited; a brief pointer to recent preference-game / von Neumann-winner work (already cited as [11,21,32]) on how cyclic energy from attention differs operationally from cyclic energy from genuine intransitive preferences would help practitioners choose between scalar and non-scalar pipelines.
  5. [Equation (10)] Typos / notation: 'deliberatively' is used consistently; in Eq. (10) the objective L(r;ℓ) uses σ(ℓ_e) as the true probability, which is fine, but a one-word reminder that this is the population BT likelihood under the human channel would reduce confusion with the usual correctly-specified BT likelihood.

Circularity Check

0 steps flagged

No significant circularity: results are mathematical consequences of an explicitly stated reduced-form channel, not fits or self-citation chains renamed as predictions.

full rationale

The paper's load-bearing claims (cycle obstruction Prop. 1, local BT projection Prop. 2, ranking reversal Example 1, non-identification Prop. 3, Fisher/KL/Fano bounds Props. 4–6 and Thm. 1, cyclic KL price Prop. 7) are derived from the reduced-form channel ℓ_z = η_z + β_z Δ*_z (Definition 1), which is stated as a modeling choice motivated by—but not claimed to be uniquely forced by—the Shannon rational-inattention FOC (Lemma 1, Remark 1). Non-identification is a legitimate characterization of that model class, not a circular derivation: the paper does not claim to recover reward from passive labels and then succeed. Example 1 is a constructed counterexample under heterogeneous β, not a data fit. Empirical checks are non-circular: Arena cyclic energy is tested against a parametric bootstrap null under fitted BT (rejecting every scalar score), and the one-eighth law comparison (0.0034 vs 0.0038 bits) is a parameter-free second-order expansion applied to the same observed field as a numerical check of approximation quality outside the small-ε regime, not a fitted parameter rebranded as prediction. No self-citations by the author load-bear the argument; Hodge decomposition is attributed to Jiang et al. and rational inattention to the external literature. The channel-adequacy caveat is a scope/correctness issue, not circularity. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central claims rest on a reduced-form measurement channel motivated by Shannon rational inattention, standard graph and information-theoretic tools, and the interpretation that deliberative reward is the target of alignment. Free parameters are the per-query attention and default terms that the theory treats as unknown; invented entities are the channel itself and the deliberative-reward target. No numerical constants are fitted to force the theoretical conclusions; empirical thresholds are robustness checks.

free parameters (3)
  • per-query attention multipliers β_z
    Unknown nonnegative scalars that scale the deliberative gap in the reduced-form channel; not estimated from data in the theory, and the non-identification result treats them as free.
  • per-query default/salience terms η_z
    Additive log-odds shifts standing in for endogenous action tendency plus order/format effects; free in the general channel and absorbable into potential defaults under Corollary 1.
  • Arena edge vote threshold (primary 100 decisive votes)
    Analysis choice that selects which model pairs enter the comparison graph; robustness at 50 and 200 is reported but the primary graph depends on this cutoff.
axioms (5)
  • domain assumption Shannon rational-inattention first-order condition yields generalized logit with inverse information cost β and endogenous log prior odds α (Lemma 1).
    Standard result from Matějka–McKay / Sims; used only to motivate the reduced form, not as an exact identification of marginal label probabilities.
  • ad hoc to paper Observed labels follow the reduced-form attention-scaled channel q_z = σ(η_z + β_z Δ*_z) (Definition 1).
    The paper's core modeling postulate; holds exactly only in the conditional environment of Remark 1, otherwise taken as reduced form.
  • standard math A scalar Bradley–Terry reward represents edge log-odds iff all directed cycle sums vanish (Proposition 1 / incidence-matrix potential representation).
    Standard graph-potential fact; proved in appendix.
  • standard math Weighted combinatorial Hodge decomposition of edge fields into potential plus cyclic components (Jiang et al.).
    Used for the cyclic-energy pricing in Proposition 7; cited prior work.
  • domain assumption Deliberative reward R* is the evaluation the principal wants after full processing of reward-relevant information.
    Normative target of alignment; not observed, only inferred through the channel.
invented entities (2)
  • attention-scaled comparison channel (Definition 1) independent evidence
    purpose: Reduced-form generative model separating reward gap, attention multiplier, and default so that hard-because-close and hard-because-hidden can be distinguished in principle.
    Postulated measurement model for the paper; motivated by RI but not uniquely identified from first principles for alignment tasks. Partial external handle via psychometric slopes and response times in the perceptual case study.
  • deliberative reward R* no independent evidence
    purpose: The latent target quality the principal wants humans to apply after full information processing; distinct from fitted Bradley–Terry scores.
    Normative construct standard in alignment; not independently measured in the Arena study (only in the perceptual task where Δ is experimenter-set).

pith-pipeline@v1.1.0-grok45 · 24611 in / 3516 out tokens · 36682 ms · 2026-07-11T16:46:29.106854+00:00 · methodology

0 comments
read the original abstract

Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences. RLHF and related alignment pipelines typically model such comparisons with Bradley--Terry log-odds, where choice probabilities are governed by latent reward differences. This paper examines what this assumption misses through a reduced-form model motivated by rational inattention, in which each label is generated by a low-capacity evaluation channel. The model separates two forms of ambiguity that standard reward modeling tends to conflate: a comparison may be difficult because the two candidates are genuinely close in value, or because the relevant distinction is hard to detect under limited attention. We show that limited attention can fundamentally distort what pairwise comparisons reveal. In particular, passive comparison data cannot generally distinguish reward, attention, and default tendencies, and heterogeneous attention can make standard Bradley--Terry reward modeling recover misleading rankings. Our analysis shows that learning is governed not by the raw number of labels, but by the amount of attended information each label carries. A case study on human votes over language-model pairs from Chatbot Arena exhibits the predicted signature, a cyclic component of the comparison data that exceeds sampling noise and that no scalar reward can represent; a second case study on perceptual comparisons shows that response times and gaze carry gap information that the labels do not. This perspective suggests that human feedback should be treated not as direct revealed preference, but as an attention-limited measurement process: a weak preference signal may reflect hidden evaluation difficulty rather than genuine indifference.

Figures

Figures reproduced from arXiv: 2607.04590 by Wenqian Xing.

Figure 1
Figure 1. Figure 1: The projection reversal (Example 1). Left to right: deliberative reward R∗ , observed log odds ℓ produced by the attention channel, and the Bradley–Terry fit r † . Attention is heterogeneous across pairs, with βBC = 5ε but βAC = ε/2. Every observed majority favors the better item, yet the heterogeneity makes the cycle sum nonzero, so no scalar reward represents ℓ (Proposition 1); the fit is the projection … view at source ↗
Figure 2
Figure 2. Figure 2: Cyclic comparison information in Chatbot Arena votes ( [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Choices and response times in a perceptual comparison task [ [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 5 canonical work pages

  1. [1]

    Christiano, John Schulman, and Dan Mané

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety, 2016. URLhttps://arxiv.org/abs/1606.06565

    Pith/arXiv arXiv 2016

  2. [2]

    Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil˙ e Lukoši¯ ut˙ e, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jack Clark, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish,...

    Pith/arXiv arXiv 2022

  3. [3]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. doi: 10.1093/biomet/39. 3-4.324. 15

    work page doi:10.1093/biomet/39 1952

  4. [4]

    Revealed preference, rational inattention, and costly infor- mation acquisition.American Economic Review, 105(7):2183–2203, 2015

    Andrew Caplin and Mark Dean. Revealed preference, rational inattention, and costly infor- mation acquisition.American Economic Review, 105(7):2183–2203, 2015. doi: 10.1257/aer. 20140117

    doi:10.1257/aer 2015

  5. [5]

    Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob...

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.15217 2023

  6. [6]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An open platform for evaluating LLMs by human preference. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and F...

    2024

  7. [7]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems, volume 30, 2017. URLhttps://proceedings.neurips.cc/paper/2017/ hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html

    2017

  8. [8]

    Owain Evans, Andreas Stuhlmüller, and Noah D. Goodman. Learning the preferences of ignorant, inconsistent agents. InProceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. doi: 10.1609/aaai.v30i1.10010. URL https://ojs.aaai.org/index.php/ AAAI/article/view/10010

    work page doi:10.1609/aaai.v30i1.10010 2016

  9. [9]

    Discrete choice and rational inattention: A general equivalence result.International Economic Review, 61(4): 1569–1589, 2020

    Mogens Fosgerau, Emerson Melo, André de Palma, and Matthew Shum. Discrete choice and rational inattention: A general equivalence result.International Economic Review, 61(4): 1569–1589, 2020. doi: 10.1111/iere.12469

    work page doi:10.1111/iere.12469 2020

  10. [10]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 10835–10866. PMLR, 23...

    2023

  11. [11]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, vo...

  12. [12]

    URLhttps://proceedings.mlr.press/v238/gheshlaghi-azar24a.html

  13. [13]

    Cooper- ative inverse reinforcement learning

    Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooper- ative inverse reinforcement learning. InAdvances in Neural Information Processing 16 Systems, volume 29, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ c3395dd46c34fa7fd8d729d8cf88b7a8-Abstract.html

    2016

  14. [14]

    Christiano, and Dario Amodei

    Geoffrey Irving, Paul F. Christiano, and Dario Amodei. AI safety via debate, 2018. URL https://arxiv.org/abs/1805.00899

    Pith/arXiv arXiv 2018

  15. [15]

    Reward-rational (implicit) choice: A unifying formalism for reward learning

    Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. InAdvances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 2f10c1578a0706e06b6d7db6f0b4a6af-Abstract.html

    2020

  16. [16]

    Statistical ranking and combi- natorial Hodge theory.Mathematical Programming, 127(1):203–244, 2011

    Xiaoye Jiang, Lek-Heng Lim, Yuan Yao, and Yinyu Ye. Statistical ranking and combi- natorial Hodge theory.Mathematical Programming, 127(1):203–244, 2011. doi: 10.1007/ s10107-010-0419-x

    2011

  17. [17]

    Visual fixations and the computation and comparison of value in simple choice.Nature Neuroscience, 13(10):1292–1298, 2010

    Ian Krajbich, Carrie Armel, and Antonio Rangel. Visual fixations and the computation and comparison of value in simple choice.Nature Neuroscience, 13(10):1292–1298, 2010. doi: 10.1038/nn.2635

    work page doi:10.1038/nn.2635 2010

  18. [18]

    Scalable agent alignment via reward modeling: A research direction, 2018

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction, 2018. URLhttps://arxiv.org/ abs/1811.07871

    Pith/arXiv arXiv 2018

  19. [19]

    Duncan Luce.Individual Choice Behavior: A Theoretical Analysis

    R. Duncan Luce.Individual Choice Behavior: A Theoretical Analysis. Wiley, New York, 1959

    1959

  20. [20]

    Rational inattention to discrete choices: A new foundation for the multinomial logit model.American Economic Review, 105(1):272–298, 2015

    Filip Matějka and Alisdair McKay. Rational inattention to discrete choices: A new foundation for the multinomial logit model.American Economic Review, 105(1):272–298, 2015. doi: 10.1257/aer.20130047

    doi:10.1257/aer.20130047 2015

  21. [21]

    Conditional logit analysis of qualitative choice behavior

    Daniel McFadden. Conditional logit analysis of qualitative choice behavior. In Paul Zarembka, editor,Frontiers in Econometrics, pages 105–142. Academic Press, New York, 1974

    1974

  22. [22]

    Mankowitz, Doina Precup, and Bilal Piot

    Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. In Ruslan Salakhutdinov, Zico K...

    2024

  23. [23]

    Rank Centrality: Ranking from pairwise comparisons.Operations Research, 65(1):266–287, 2017

    Sahand Negahban, Sewoong Oh, and Devavrat Shah. Rank Centrality: Ranking from pairwise comparisons.Operations Research, 65(1):266–287, 2017. doi: 10.1287/opre.2016.1534

    doi:10.1287/opre.2016.1534 2017

  24. [24]

    Chris- tiano, Jan Leike, and Ryan Lowe

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with hu- man fe...

    2022

  25. [25]

    Manning, Stefano Er- mon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Er- mon, and Chelsea Finn. Direct preference optimization: Your language model is se- cretly a reward model. InAdvances in Neural Information Processing Systems, vol- ume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ a85b405ed65c6477a4fe8302b5e06ce7-...

    2023

  26. [26]

    Dragan, S

    Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. InRobotics: Science and Systems XIII, 2017. doi: 10.15607/RSS. 2017.XIII.053. URLhttps://www.roboticsproceedings.org/rss13/p53.html

    doi:10.15607/rss 2017

  27. [27]

    Shah and Martin J

    Nihar B. Shah and Martin J. Wainwright. Simple, robust and optimal ranking from pairwise comparisons.Journal of Machine Learning Research, 18(199):1–38, 2018. URLhttps://jmlr. org/papers/v18/16-206.html

    2018

  28. [28]

    On the feasibility of learning, rather than assuming, human biases for reward inference

    Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 5670–5679. PMLR, 09–15 Jun

  29. [29]

    URLhttps://proceedings.mlr.press/v97/shah19a.html

  30. [30]

    Christopher A. Sims. Implications of rational inattention.Journal of Monetary Economics, 50 (3):665–690, 2003. doi: 10.1016/S0304-3932(03)00029-1

    doi:10.1016/s0304-3932(03)00029-1 2003

  31. [31]

    A long way to go: Investigating length correlations in RLHF

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF. InProceedings of the First Conference on Language Modeling, 2024. doi: 10.48550/arXiv.2310.03716. URL https://openreview.net/forum?id=G8LaO1P0xv. Spotlight; arXiv:2310.03716

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.03716 2024

  32. [32]

    Defin- ing and characterizing reward gaming

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defin- ing and characterizing reward gaming. InAdvances in Neural Information Pro- cessing Systems, volume 35, pages 9460–9471, 2022. doi: 10.48550/arXiv. 2209.13085. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 3d719fee332caa23d5038b8a90e81796-Abstract-Conference.html

    Pith/arXiv arXiv doi:10.48550/arxiv 2022

  33. [33]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. InAdvances in Neural Information Processing Systems, vol- ume 33, pages 3008–3021, 2020. URLhttps://proceedings.neurips.cc/paper/2020/hash/ 1f89885d556929e98d3ef9b86448f951-Abst...

    2020

  34. [34]

    A minimaxi- malist approach to reinforcement learning from human feedback

    Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A minimaxi- malist approach to reinforcement learning from human feedback. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proceedings of the 41st International Conference on Machine Learning, volu...

    2024

  35. [35]

    The attentional drift diffusion model of simple perceptual decision-making.Frontiers in Neuroscience, 11:468, 2017

    Gabriela Tavares, Pietro Perona, and Antonio Rangel. The attentional drift diffusion model of simple perceptual decision-making.Frontiers in Neuroscience, 11:468, 2017. doi: 10.3389/ fnins.2017.00468. URL https://www.frontiersin.org/journals/neuroscience/articles/ 10.3389/fnins.2017.00468/full. 18

    work page doi:10.3389/fnins.2017.00468/full 2017

  36. [36]

    A survey of preference-based reinforcement learning methods.Journal of Machine Learning Research, 18 (136):1–46, 2017

    Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods.Journal of Machine Learning Research, 18 (136):1–46, 2017. URLhttps://jmlr.org/papers/v18/16-634.html

    2017

  37. [37]

    Principled reinforcement learning with human feedback from pairwise or K-wise comparisons

    Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or K-wise comparisons. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machin...

    2023

  38. [38]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences,

  39. [39]

    19 A Proofs A.1 Proof of Lemma 1 Proof

    URLhttps://arxiv.org/abs/1909.08593. 19 A Proofs A.1 Proof of Lemma 1 Proof. Fix z and suppress the query subscript. Writeu(a,ω) = uz(a,ω), µ= µz, κ= κz, and π=πz. Because the optimum is interior,π(a|ω)> 0and ¯π(a)> 0for both actions. The mutual information can be written as I(ω;a) = ∑ ω,a µ(ω)π(a|ω) logπ(a|ω)− ∑ a ¯π(a) log ¯π(a). For eachaandω, ∂I ∂π(a|...

    Pith/arXiv arXiv 1909