REVIEW 8 cited by
Preference Transformer: Modeling Human Preferences using Transformers for RL
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.
Forward citations
Cited by 8 Pith papers
-
SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
SP3O is a reward-model-free, critic-free, gradient-based RL algorithm that optimizes policies from segment-level preferences in stochastic MDPs via off-policy importance sampling and PPO-style clipping.
-
Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
SARA extracts rewards from cosine similarity to a contrastively learned latent of preferred trajectories and outperforms or matches baselines under label noise in continuous control benchmarks.
-
SimulPL: Aligning Human Preferences in Simultaneous Machine Translation
SimulPL adds latency-aware preference optimization to simultaneous machine translation and reports better human-aligned quality at low latency on three language pairs.
-
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
A variational Bayesian inverse RL objective, AVA, models token-level rewards and outperforms reward-difference baselines in LLM alignment on standard benchmarks.
-
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
LEMUR jointly learns a separate reward model for each teacher's preferences and uses them to train a population of multi-objective policies, beating baselines that merge feedback into one reward.
-
TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations
TREND combines cyclic small-loss selection among three reward models with few-shot demonstration pretraining and regularization, yielding robust policies on Meta-World tasks under up to 40 percent label noise.
-
Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
The paper derives convergence rates for sigmoid gating mixture-of-experts with quadratic scores and uses them to argue sigmoid self-attention is more sample-efficient than softmax, but the link to attention is an unpr...
-
Reward Models in Deep Reinforcement Learning: A Survey
A structured survey of reward modeling in deep RL, proposing a three-axis taxonomy and reviewing applications and evaluation methods.
Discussion (0). Continue with ORCID to comment.