REVIEW 7 cited by
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample- and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-policy learning. Specifically, we learn a reward model by actively querying a teacher's preferences between two clips of behavior and use it to train an agent. To enable off-policy learning, we relabel all the agent's past experience when its reward model changes. We additionally show that pre-training our agents with unsupervised exploration substantially increases the mileage of its queries. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods, including a variety of locomotion and robotic manipulation skills. We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.
Forward citations
Cited by 7 Pith papers
-
RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview
Rule-guided mismatch cues raise Human+AI rehab-assessment accuracy by 14% and cut harmful reliance; prospective embedding previews raise local model-edit gains from 11.5% to 36%, with global transfer often regressing.
-
HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation
HALO learns a vision-based navigation reward from human preference rankings on egocentric video, and an IQL policy using it beats several baselines in 10-trial real-world tests.
-
CueLearner: Bootstrapping and local policy adaptation from relative feedback
CueLearner learns a relative-feedback model from a small number of human directional corrections and uses it to guide off-policy RL exploration or refine a deployed policy.
-
CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
CLARIFY uses contrastive learning on preference data to embed trajectories, then rejection-samples queries that humans can distinguish clearly, improving offline preference-based RL.
-
Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning
RE-GoT combines graph-of-thoughts planning in LLMs with VLM feedback from rollout videos to automatically write and refine RL reward functions, beating prior LLM-based reward design on RoboGen and ManiSkill2.
-
Residual Reward Models for Preference-based Reinforcement Learning
Combining a hand-designed or learned prior reward with a preference-trained residual improves sample efficiency and final performance in preference-based reinforcement learning.
-
Active Query Selection for Crowd-Based Reinforcement Learning
Extending the Advise algorithm with variational crowd modelling and entropy-based query selection yields faster learning in small tabular RL tasks, especially highly constrained ones.
Discussion (0). Continue with ORCID to comment.