RECORD
Query-Policy Misalignment in Preference-Based Reinforcement Learning
Machine Learning · Submitted 27 May 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via near on-policy query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. | background 1.00 | 0.04 | 0.03 | asserted 0.99 |
| s1 | To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. | result 0.53 | 0.86 | 0.24 | hedged possibility 0.78 |
| s2 | To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. | gap 0.56 | 0.83 | 0.16 | asserted 1.00 |
| s3 | We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. | result 0.50 | 0.87 | 0.39 | hedged possibility 0.60 |
| s4 | We show that this issue can be effectively addressed via near on-policy query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. | method 0.92 | 0.90 | 0.25 | asserted 0.99 |
| s5 | Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. | contribution 0.63 | 0.47 | 0.04 | asserted 1.00 |
| s6 | We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks. | result 0.98 | 0.96 | 0.61 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via near on-policy query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.
Human review
References
How the literature uses it
4 arXiv papers cite this in 8 sentences
Papers citing it · 4 sentences without a confident label
Uses its method or tool 1 paper
-
For all methods, we use unsupervised pre-training proposed by PEBBLE and the same hyper-parameters and network architectures as in original papers [ref] .
SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning · Implementation Details · 1.00
Cites it as background 2 papers
-
To address this problem, various query selection schemes have been proposed, such as entropy-based query selection [ref] , disagreement-based query selection [ref] , policy-aligned query selection [ref] , etc.
SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning · INTRODUCTION · 0.85
-
Prior work has studied improving query-efficiency by sampling the most informative behaviors to obtain preferences, such as maximizing behavior entropy [ref] , aligning the agent's actions with the queries [ref] , and allowing the agent to actively ask questions [ref] .
Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics · Reward Learning from Human Preferences · 0.86
What it cites · 5 sentences without a confident label
Cites as background 2 papers
-
PbRL provides a natural approach for humans (oracle overseer) to communicate desired behaviors with RL agents by making relative judgments between a pair of behaviors [ref] .
Recursively Summarizing Books with Human Feedback · Related Work · 0.98
-
Data augmentation has been widely used in many deep RL [ref] algorithms, and also has been seamlessly integrated into previous PbRL algorithms for consistency regularization [ref] .
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels · Data Augmentation for Reward Learning · 0.80
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.