Pith. sign in

REVIEW 4 major objections 4 minor 62 references

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SARA claims that a single latent vector for the preferred set, used as a cosine-similarity anchor, is enough to compute RL rewards that stay reliable under high label noise.

desk verdict SARA's core idea is solid and the offline setup is careful, but the abstract claims statistics the paper never reports, so the headline robustness result is not yet backed. read the letter →

arxiv 2506.12529 v2 pith:O6PD75NU submitted 2025-06-14 cs.LG cs.AIstat.ML

Sara Rajaram , R. James Cotton , Fabian H. Sinz This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords preference-basedreinforcementlearningcontrastiverewardinferencecosinesimilaritylabelnoiserobustnessofflineshapingpreferencetransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a preference signal for reinforcement learning can be carried by a single learned vector: the latent representation of the whole preferred set. Instead of fitting a Bradley-Terry model to pairwise comparisons, SARA trains a contrastive transformer encoder to pull preferred trajectories together and push non-preferred ones apart, then defines the reward at every timestep as the cosine similarity between the encoded trajectory and that preferred-set latent. On continuous-control offline RL benchmarks with 20% of preference labels randomly flipped, the paper reports a 31% average improvement in policy evaluation rewards over three baselines, with lower variance across dataset variants. It also argues the same reward can filter low-quality trajectories, transfer preferences between agents with different morphologies, and shape rewards in online RL.

What carries the argument

The load-bearing object is the preferred-set latent $z^*_p$. A first transformer encodes each trajectory with positional encodings and average-pools over timesteps; the resulting per-trajectory encodings are randomly partitioned into $k$ subsets whose composition is reshuffled each epoch, and a second transformer encodes each subset into one latent. Training uses the SimCLR contrastive loss, which attracts latents of the same category and repels the two categories. At inference, the full preferred set is encoded in one forward pass to give $z^*_p$, and every timestep reward is the cosine similarity $\cos(z_t, z^*_p)$; the paper sets the optional negative-similarity term to zero in all experiments. This machinery does the work of absorbing label noise: because subset composition is shuffled, a mislabeled trajectory cannot dominate the contrastive objective, and because only set membership matters, pairwise ranking assumptions are dropped.

What would settle it

Train SARA on a synthetic preference dataset whose preferred set has two equally large, widely separated modes, such as clockwise and counterclockwise trajectories to the same target, then measure whether the cosine reward to a single $z^*_p$ still ranks both modes above non-preferred trajectories; if the average latent falls between the modes and both modes receive low similarity, the central claim fails. A quantitative version compares the correlation between SARA reward and true environment reward against baselines on this bimodal set, and checks whether the correlation drops below the single-mode baseline.

Watch

Extended reading notes

Core claim

On its own terms, SARA claims that a reward function adequate for RL can be read off directly from representation geometry: $r_t = \cos(z_t, z^*_p)$, where $z^*_p$ is the encoder's latent for the full set of preferred trajectories and $z_t$ is the latent of the trajectory up to time $t$. The encoder is trained with the SimCLR contrastive loss on randomly composed subsets of preferred and non-preferred trajectories, so the latent captures patterns that distinguish the categories rather than any particular pairwise ranking. The paper's central evidence is that with 20% label error, SARA rewards drive IQL policies that beat Preference Transformer, PT with adaptive denoising, and DPPO on five of six offline tasks, and that its reward correlates more strongly with the environment reward across noise rates. The same frozen latent supports trajectory filtering, preference transfer from a one-legged to a two-legged agent, and online reward shaping in a Reacher task.

Load-bearing premise

The whole reward signal rests on the assumption that one vector produced by encoding the entire preferred set is an adequate summary of the preference criterion, so that cosine similarity to that vector correctly ranks any new trajectory; if the preferred set is multimodal or dominated by mislabeled trajectories, that single anchor can be biased.

Editorial extensions

If this is right

  • With SARA's reward, any off-the-shelf offline RL algorithm can be trained from preferences, since a scalar reward is produced for every transition rather than only a policy objective.
  • At 20% label error, SARA's policy evaluation rewards average 31% higher than PT, PT+ADT, and DPPO, and its performance varies less across preference dataset variants (with or without neutral labels, human vs scripted, noisy vs clean).
  • The learned preferred latent is reusable: it ranks cold trajectories for filtering low-quality demonstrations and transfers hopper preferences to reward estimation for walker2d.
  • In online RL, adding the cosine-similarity reward to a known task reward changes the learned Reacher policy from the shortest clockwise path to the desired counterclockwise path even though the preference set only covered an easier target.
  • SARA accepts feedback that is not pairwise: a set of preferred trajectories alone is enough, so neutral or equal-preference labels do not need special handling beyond placing trajectories in both sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the single-latent summary holds up, preference-based RL shifts from ranking pairs to set-level prototype learning, which is a natural fit for LLM alignment data where many responses are preferred to one reference and labels are noisy.
  • A testable consequence the paper does not run: as the share of flipped labels in the preferred set approaches 50%, the contrastive objective has nothing consistent to align, so the reward should collapse; locating that breakpoint would calibrate the claimed robustness.
  • The cross-task transfer experiment suggests a more general recipe, namely mapping a simple agent's state-action dimensions into a larger agent by joint symmetry; a stronger test would use a learned correspondence instead of a crude doubling.
  • Because $r_t = \cos(z_t, z^*_p)$ is a dense, bounded reward, SARA doubles as a shaped exploration bonus; combining it with curiosity or entropy bonuses could be explored in tasks where preferences are sparse.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Similarity as Reward Alignment (SARA), a contrastive framework that learns a latent representation of preferred trajectories and assigns per-timestep rewards as the cosine similarity between the encoded trajectory and a single preferred-set latent z*_p. The method is evaluated on D4RL offline control tasks (Mujoco locomotion, Franka Kitchen, and Adroit pen), with IQL as the downstream RL algorithm, against PT, PT+ADT, and DPPO baselines across several preference-set variants: human vs. script labels, inclusion vs. exclusion of neutral preferences, and a 20% label-error condition. The paper also presents three additional applications: low-quality trajectory filtering for imitation learning, cross-task preference transfer from hopper to walker2d, and online reward shaping on a Deepmind Control Suite Reacher task. The central empirical claim is that with 20% label error SARA achieves a 31% mean relative gain in policy evaluation rewards over baselines, with a statistically significant improvement asserted in the abstract (Wilcoxon signed-rank, p < 0.01).

Significance. If the empirical claims are fully supported, SARA would be a simple and versatile preference-based RL method with a useful property: it produces an explicit reward signal without Bradley-Terry modeling, and it can be plugged into off-the-shelf offline and online RL algorithms. The paper has notable strengths: the code is publicly released; the offline evaluation uses 8 seeds with a carefully documented evaluation protocol (Appendix E.2); several dataset variants and labeler types are considered; and the additional applications, especially the morphologically different transfer experiment, are original. I also do not see a circularity defect in the reward construction: z*_p is learned from preference data and then evaluated on downstream policy returns, which is standard generalization, and alpha is not forced to a value that trivially produces the reported results. The main weakness is that the headline statistical claims are not backed by any reported test, the 31% aggregate is sensitive to the exact aggregation formula, and a promised correlation-to-environment-reward analysis is absent from the body.

major comments (4)
  1. [Abstract; Section 4 (Offline experiments summary)] The abstract's claim of 'statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01)' is nowhere reported in Section 4 or in the appendices. With the six primary tasks in Tables 1-4, the minimum achievable two-sided exact p-value for a paired Wilcoxon signed-rank test is 0.03125, so p < 0.01 cannot be obtained from that pairing. If the test instead pools the eight tasks including Adroit, or uses per-seed measurements, the paper must state the pairing, the sample size, the test statistic, and any multiple-comparison correction. As written, the headline significance claim is unsupported.
  2. [Section 4, 'Offline experiments summary' and Table 1] The '31% mean relative gain in policy evaluation rewards over baselines' is not reproducible as stated because no aggregation formula is given. Reconstructing from Table 1 as an unweighted average of relative gains over the 18 baseline entries gives approximately 30.9%, but this value is strongly influenced by the 194% relative gain on hopper-expert, where DPPO collapses to 28.92. Individual baseline relative gains range from about -1% to 194%, and against PT and PT+ADT alone the mean is about 15%. The paper should specify the aggregation formula, report per-task and per-baseline gains, and provide a robust summary such as the median gain or a confidence interval before claiming a 31% advantage.
  3. [Abstract; Section 4] The abstract promises that 'We also compute correlation to the environment rewards as a proxy for measuring alignment to the underlying preference criteria' and that 'SARA computed rewards display higher correlation across noise rates compared to baselines.' No such correlation analysis appears in Section 4 or in the appendices; Figure 3 is a t-SNE visualization, not a correlation measurement. This result must either be added with a concrete metric reported across noise rates, or the claim should be removed from the abstract.
  4. [Section 4, 'Robustness to dataset variants' and Figure 2] The robustness claim that 'SARA is quite consistent across dataset variants as well as performant' is supported only by a qualitative visual comparison in Figure 2. The paper does not quantify the stability of rewards or policy returns across dataset variants, nor does it test whether the differences in fluctuation between SARA and the baselines are statistically meaningful. Given that robustness to dataset variations is one of the paper's primary claims, please add a quantitative stability measure (e.g., variance or interquartile range across variant means, or a paired test across the variants) rather than relying on the vertical range in the figure.
minor comments (4)
  1. [Abstract] The abstract says 'on preference data with varying realistic noise rates,' but the offline experiments include only the original data and a single 20% label-flip condition. If additional noise rates were tested, they should be reported or cited; otherwise the wording should be adjusted.
  2. [Section 5, 'Online RL with reward shaping'] The online reward-shaping experiment is presented through a qualitative Figure 5 only. Please report numerical policy returns or success metrics for the shaped versus unshaped policy, along with the number of seeds, so that the reader can assess the strength of the effect.
  3. [Related work and Section 4] The paper describes OPPO, CPL, and IPL in the related work but does not compare against them empirically. Since the abstract claims strong performance over baselines, a sentence explaining why these methods are not included as baselines would help the reader calibrate the comparison.
  4. [Appendix A, Eq. (2)] The logistic form in Eq. (2) is described as motivation, but it is used to justify setting the reward to cos(z_t, z*_p) with alpha = 0. Please state explicitly that this is a heuristic reward and that no formal guarantee is claimed for the ordering induced by the cosine similarity, as the limitations paragraph already acknowledges.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SARA's reward is a design choice evaluated on external policy returns; the cited theory and hyperparameter choices are not self-defining.

full rationale

The paper's central derivation is self-contained and does not reduce to its inputs by construction. SARA defines a contrastive encoder trained with the SimCLR loss on preferred/non-preferred trajectory sets, then defines the reward as rt = cos(zt, z*p), where z*p is obtained by a forward pass over the full preferred set (Section 3, Steps 1-3). The downstream evaluation is the IQL policy's normalized environment returns (Section 4, Table 1), which is an external quantity not used as a training target for the encoder or for z*p. This is a standard design-to-generalization evaluation, not a fitted quantity renamed as a prediction. No load-bearing self-citation appears: the references to PT, DPPO, and Sun et al. are external prior work used for datasets, baselines, or theoretical grounding, and no uniqueness theorem from the present authors is invoked to force the choice of reward. Appendix A is admittedly heuristic and closes with alpha=0 being selected empirically ('Empirically, we found alpha = 0 to be optimal in all our experiments'), which is a hyperparameter-selection concern that could inflate reported performance but does not make the derivation circular, because the reward formula is not derived from the evaluation metric. The paper also explicitly flags the absence of a full theoretical treatment in its Limitations section. Finally, the abstract's Wilcoxon p<0.01 claim is not reproduced in the body and, with only six tasks, a paired two-sided Wilcoxon test cannot reach that threshold; this is an unsupported statistical claim and a correctness risk, but it is not an instance of circular reasoning. Overall, no circular step can be quoted or exhibited, so the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or causal entities are introduced; the method relies on a learned representation rather than an invented mechanism. The key free parameter is alpha, fixed at 0. The axioms are domain assumptions about the sufficiency of a single preferred latent, the noise model, the cosine metric, and the synthetic hopper-to-walker mapping used in one experiment.

free parameters (1)
  • alpha = 0
    Coefficient in the reward formula r_t = cos(z_t, z*_p) - alpha * cos(z_t, z*_n). The authors state alpha=0 was optimal in all experiments, making the reward purely cosine to the preferred latent.
assumptions (4)
  • domain assumption The full set of preferred trajectories can be summarized by a single latent z*_p, and cosine similarity to it is a suitable reward.
    SARA's reward is defined as this cosine similarity. The paper provides no theory that a single vector captures multimodal preferences; this is asserted in Section 3 Step 2.
  • domain assumption 20% label flipping is an adequate model of human labeling noise, and subset shuffling makes the encoder robust to it.
    Robustness is demonstrated by injecting random label flips. The paper does not study structured noise (e.g., systematic bias), so the claim is limited to this noise model.
  • domain assumption Cosine similarity preserves the relative order of preference over trajectories after the encoder is trained.
    The reward is used directly as RL reward. While the SimCLR training aligns same-category latents, there is no proof that the cosine metric ranks all trajectories consistently with the underlying preference.
  • ad hoc to paper Hopper state-action dimensions can be duplicated to map to walker due to joint symmetry.
    Used only for the cross-task transfer experiment. Appendix F states the mapping is not a physically realistic way to map the dimensions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning." pith.science (2026). https://pith.science/paper/O6PD75NU

@misc{pith2026250612529,
  author       = {Pith},
  title        = {Pith review of: Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O6PD75NU}},
  note         = {Machine review of arXiv:2506.12529}
}
read the original abstract

Preference-based Reinforcement Learning (PbRL) entails a variety of approaches for aligning models with human intent to alleviate the burden of reward engineering. However, most previous PbRL work has not investigated the robustness to labeler errors, inevitable with labelers who are non-experts or operate under time constraints. We introduce Similarity as Reward Alignment (SARA), a simple contrastive framework that is both resilient to noisy labels and adaptable to diverse feedback formats. SARA learns a latent representation of preferred samples and computes rewards as similarities to the learned latent. On preference data with varying realistic noise rates, we demonstrate competitive and more stable performance on continuous control offline RL benchmarks, with statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01). We also compute correlation to the environment rewards as a proxy for measuring alignment to the underlying preference criteria. We show that the SARA computed rewards display higher correlation across noise rates compared to baselines.

Figures

Figures reproduced from arXiv: 2506.12529 by the authors.

Figure 1
Figure 1. Step 1: preferred and non-preferred trajectories are extracted from labeled pairs. The first [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Average normalized policy evaluation rewards are shown for four preference set variants [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. T-SNE embedding of the latents for the hopper-medium-replay-v2 preference data, either [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Walker2d IQL eval rewards, running average and 8 seeds (see Section E.2). Shading [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Left: After training SARA encoder, we divide up the trajectories by their label (preferred vs [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 46 canonical work pages

  1. [1]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. In Rein- forcement Learning: An Introduction, chapter 17.6 Reinforcement Learning and the Future of Artificial Intelligence. MIT Press, Cambridge, MA, 2 edition, 2018. Chapter 17.6

    work page 2018

  2. [2]

    Reinforcement learning can be more efficient with multiple rewards

    Christoph Dann, Yishay Mansour, and Mehryar Mohri. Reinforcement learning can be more efficient with multiple rewards. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Resear...

    work page 2023

  3. [3]

    Learning Agile Robotic Locomotion Skills by Imitating Animals

    Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine. Learning Agile Robotic Locomotion Skills by Imitating Animals. In Robotics: Science and Systems XVI. Robotics: Science and Systems Foundation, July 2020. ISBN 978-0-9923747-6-1. doi: 10.15607/RSS.2020.XVI.064. URL http://www.roboticsproceedings.org/rss16/ p064.pdf

    doi:10.15607/rss.2020.xvi.064 2020

  4. [4]

    Deep object-centric represen- tations for generalizable robot learning

    Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric represen- tations for generalizable robot learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7111–7118, 2018. doi: 10.1109/ICRA.2018.8461196

    arXiv 2018

  5. [5]

    The ingredients of real world robotic reinforcement learning

    Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview. net/forum?id=rJe2syrtvS

    work page 2020

  6. [6]

    Preference Transformer: Modeling Human Preferences using Transformers for RL, March 2023

    Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference Transformer: Modeling Human Preferences using Transformers for RL, March 2023. URL http://arxiv.org/abs/2303.00957. arXiv:2303.00957 [cs]

    arXiv 2023

  7. [7]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, January 2020. URL http://arxiv.org/abs/1909.08593. arXiv:1909.08593 [cs]. 10

    arXiv 2020

  8. [8]

    GPT-4 Technical Report, March 2024

    OpenAI, Josh Achiam, Steven Adler, and et al. GPT-4 Technical Report, March 2024. URL http://arxiv.org/abs/2303.08774. arXiv:2303.08774 [cs]

    arXiv 2024

Show all 62 references
  1. [9]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, and et al. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Cu...

    2022

  2. [10]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, and et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025. URL http://arxiv.org/ abs/2501.12948. arXiv:2501.12948 [cs]

    2025 arXiv

  3. [11]

    Active Preference-Based Learning of Reward Functions

    Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active Preference-Based Learning of Reward Functions. In Robotics: Science and Systems XIII. Robotics: Science and Systems Foundation, July 2017. ISBN 978-0-9923747-3-0. doi: 10.15607/RSS.2017.XIII.053. URL http://w...

    2017 doi

  4. [12]

    Deep Reinforcement Learning from Human Preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information ...

    2017

  5. [13]

    A Survey of Preference-Based Reinforcement Learning Methods

    Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A Survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1–46, 2017. URL http://jmlr.org/papers/v18/16-634.html

    2017

  6. [14]

    Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 0006-3444. doi: 10.2307/2334029. URL https://www.jstor.org/stable/2334029. Publisher: [Oxford University Press, Biom...

    1952

  7. [15]

    Smith, and Pieter Abbeel

    Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: Feedback-Efficient Interactive Rein- forcement Learning via Relabeling Experience and Unsupervised Pre-training. In Proceedings of the 38th International Conference on Machine Learning , pages 6152–6163. PMLR, July

  8. [16]

    Few-shot preference learning for human-in-the-loop RL

    Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In 6th Annual Conference on Robot Learning, 2022. URL https://openreview.net/ forum?id=IKC5TfXLuW0

    2022

  9. [17]

    Bradley Knox, and Dorsa Sadigh

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Representations,

  10. [18]

    Inverse Preference Learning: Preference-based RL without a Reward Function

    Joey Hejna and Dorsa Sadigh. Inverse Preference Learning: Preference-based RL without a Reward Function. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 18806–18827. Curran Ass...

    2023

  11. [19]

    Direct Preference-based Policy Optimization without Reward Modeling

    Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct Preference-based Policy Optimization without Reward Modeling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing...

    2023

  12. [20]

    Beyond Reward: Offline Preference-guided Policy Optimization

    Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond Reward: Offline Preference-guided Policy Optimization. In Proceedings of the 40th International Conference on Machine Learning, pages 15753–15768. PMLR, July 2023. URL https://proceedings. mlr.press/v202/kang...

    2023

  13. [21]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in N...

    2023

  14. [22]

    Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning

    Sachit Kuhar, Shuo Cheng, Shivang Chopra, Matthew Bronars, and Danfei Xu. Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning. In Proceedings of The 7th Conference on Robot Learning, pages 1437–1449. PMLR, December 2023...

    2023

  15. [23]

    Rethinking reward modeling in preference-based large language model alignment

    Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference-based large language model alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=rfdblE10qm

    2025

  16. [24]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Ruslan Salakhutdinov, Zico...

    2024

  17. [25]

    Nash learning from human feedback

    Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J Mankowitz, Doina Precup, and...

    2024

  18. [26]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th I...

    2024

  19. [27]

    Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

    Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. Online Iterative Reinforcement Learning from Human Feedback with General Preference Model. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neur...

    2024

  20. [28]

    Intransitivity of preferences

    Amos Tversky. Intransitivity of preferences. Psychological Review, 76(1):31–48, January 1969. ISSN 1939-1471, 0033-295X. doi: 10.1037/h0026750. URL https://doi.apa.org/doi/ 10.1037/h0026750

    1969 doi

  21. [29]

    Kenneth O. May. Intransitivity, Utility, and the Aggregation of Preference Patterns. Econometrica, 22(1):1–13, 1954. URL https://onlinelibrary.wiley.com/doi/abs/ 0012-9682(195401)22:1&lt;1:IUATAO&gt;2.0.CO;2-P . 12

    1954

  22. [30]

    RIME: robust preference-based reinforcement learning with noisy preferences

    Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei-Yue Wang. RIME: robust preference-based reinforcement learning with noisy preferences. In Proceedings of the 41st International Conference on Machine Learning, volume 235 ofICML’24, pages 8229–8247, Vienna,...

    2024

  23. [31]

    AlpacaFarm: A Sim- ulation Framework for Methods that Learn from Human Feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. AlpacaFarm: A Sim- ulation Framework for Methods that Learn from Human Feedback. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. ...

    2023

  24. [32]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representa- tions, 2024. URL https://openreview.net/forum?id=dcjtMYkpXx

    2024

  25. [33]

    B-pref: Benchmarking preference- based reinforcement learning

    Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https://openreview.net/ forum?id=ps95-mkHF_

    2021

  26. [34]

    Rethinking decision transformer via hierarchical reinforcement learning

    Yi Ma, Jianye Hao, Hebin Liang, and Chenjun Xiao. Rethinking decision transformer via hierarchical reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML’24, pages 33730–33745, Vienna, Austria, July 2024. JMLR.org

    2024

  27. [35]

    WHEN SHOULD WE PREFER DECISION TRANSFORMERS FOR OFFLINE REINFORCE- MENT LEARNING? 2024

    Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. WHEN SHOULD WE PREFER DECISION TRANSFORMERS FOR OFFLINE REINFORCE- MENT LEARNING? 2024

    2024

  28. [36]

    OFFLINE REINFORCEMENT LEARNING WITH IMPLICIT Q-LEARNING

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. OFFLINE REINFORCEMENT LEARNING WITH IMPLICIT Q-LEARNING. 2022

    2022

  29. [37]

    Preference Alignment with Flow Matching

    Minu Kim, Yongsik Lee, Sehyeok Kang, Jihwan Oh, Song Chong, and Se-Young Yun. Preference Alignment with Flow Matching. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37...

    2024

  30. [38]

    Flow to better: Offline preference-based reinforcement learning via preferred trajectory generation

    Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better: Offline preference-based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openrevie...

    2024

  31. [39]

    Denoising Implicit Feedback for Recommendation

    Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. Denoising Implicit Feedback for Recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 373–381, Virtual Event Israel, March 2021. ACM. ISBN 978-1-4503-829...

    2021

  32. [40]

    mixup: BEYOND EMPIRICAL RISK MINIMIZATION

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: BEYOND EMPIRICAL RISK MINIMIZATION. 2018

    2018

  33. [41]

    Co-teaching: Robust training of deep neural networks with extremely noisy labels

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Ad...

    2018

  34. [42]

    Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML’20, pages 6448–6458

    Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML’20, pages 6448–6458. JMLR.org, July 2020

    2020

  35. [43]

    Learning from noisy labels with deep neural networks: A survey

    Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 34(11):8135–8153, 2023. doi: 10.1109/TNNLS.2022.3152527

    2023

  36. [44]

    Attention is All you Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neur...

    2017

  37. [45]

    A Simple Framework for Contrastive Learning of Visual Representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597–1607. PMLR, November 2020. URL https: //proceedings.mlr...

    2020

  38. [46]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning, February 2021

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, February 2021. URL http://arxiv.org/abs/ 2004.07219. arXiv:2004.07219 [cs]

    2021 arXiv

  39. [47]

    D4RL: Building Better Benchmarks for Offline Reinforcement Learning

    Ritwik Gupta. D4RL: Building Better Benchmarks for Offline Reinforcement Learning. URL http://bair.berkeley.edu/blog/2020/06/25/D4RL/

    2020

  40. [48]

    URL https://www.gymlibrary.dev/environments/ mujoco/index.html

    MuJoCo - Gym Documentation. URL https://www.gymlibrary.dev/environments/ mujoco/index.html

  41. [49]

    Offlinerl-kit: An elegant pytorch offline reinforcement learning library

    Yihao Sun. Offlinerl-kit: An elegant pytorch offline reinforcement learning library. https: //github.com/yihaosun1124/OfflineRL-Kit, 2023

    2023

  42. [50]

    Hierarchically decoupled imitation for morpho- logical transfer

    Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morpho- logical transfer. In Hal Daumé III and Aarti Singh, editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Researc...

    2020

  43. [51]

    PEARL: Zero-shot cross-task preference alignment and robust reward learning for robotic manipulation

    Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. PEARL: Zero-shot cross-task preference alignment and robust reward learning for robotic manipulation. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix B...

    2024

  44. [52]

    Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

    Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 25...

    2022

  45. [53]

    Learning robust perceptive locomotion for quadrupedal robots in the wild

    Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci- ence Robotics , 7(62):eabk2822, January 2022. doi: 10.1126/scirobotics.abk2822. URL https://www.science...

    2022 doi

  46. [54]

    dm_control: Software and tasks for continuous control

    Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https:// doi.org/...

    2020

  47. [55]

    URLB: Unsupervised reinforcement learning benchmark

    Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Rou...

    2021

  48. [56]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2016. URL https://arxiv.org/abs/ 1509.02971

    2016 arXiv

  49. [57]

    URL https://docs.pytorch.org/ docs/stable/generated/torch.nn.TransformerEncoder.html

    TransformerEncoder — PyTorch 2.7 documentation. URL https://docs.pytorch.org/ docs/stable/generated/torch.nn.TransformerEncoder.html

  50. [58]

    URL https://www.gymlibrary.dev/environments/ mujoco/hopper/

    Hopper - Gym Documentation. URL https://www.gymlibrary.dev/environments/ mujoco/hopper/

  51. [59]

    URL https://www.gymlibrary.dev/environments/ mujoco/walker2d/

    Walker2D - Gym Documentation. URL https://www.gymlibrary.dev/environments/ mujoco/walker2d/. A Motivation for SARA reward inference A full theoretical treatment is suggested for future works, but here we provide motivation for our proposed reward estimation detailed in Section...

  52. [62]

    For the Franka Kitchen tasks, we use the preference datasets from An et al

    from the PT human label repository. For the Franka Kitchen tasks, we use the preference datasets from An et al. [19] from the DPPO human label repository. Both repositories are MIT licensed. Kim et al. [6] and An et al. [19] created the preference datasets by sampling pairs of...

    2000

  53. [2021]

    ISSN: 2640-3498

    URL https://proceedings.mlr.press/v139/lee21i.html. ISSN: 2640-3498

  54. [2024]

    URL https://openreview.net/forum?id=iX1RjVQODj

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.