REVIEW 6 cited by
Reinforcement Learning from Human Feedback with Active Queries
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled preference data, which is expensive to collect. In this paper, inspired by the success of active learning, we address this problem by proposing query-efficient RLHF methods. We first formalize the alignment problem as a contextual dueling bandit problem and design an active-query-based proximal policy optimization (APPO) algorithm with an $\tilde{O}(d^2/\Delta)$ instance-dependent regret bound and an $\tilde{O}(d^2/\Delta^2)$ query complexity, where $d$ is the dimension of feature space and $\Delta$ is the sub-optimality gap over all the contexts. We then propose ADPO, a practical version of our algorithm based on direct preference optimization (DPO) and apply it to fine-tuning LLMs. Our experiments show that ADPO, while only making about half of queries for human preference, matches the performance of the state-of-the-art DPO method.
Forward citations
Cited by 6 Pith papers
-
When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design
Safety in surrogate-driven design requires oracle-only certification; rank preservation is the exact criterion for oracle use, and selection-aware auditing is the cheapest way to certify it.
-
Personalizing Large Language Model Agents with Small Policy Models
A factorized Bayesian Thompson-sampling layer outside a frozen agent learns per-user execution preferences from selected-action scalar feedback, with a Õ(d^{3/2}√n) regret bound against the best feasible action.
-
PILAF: Optimal Human Preference Sampling for Reward Modeling
A response-pair sampling scheme that interpolates current and reference model logits is proposed and claimed to align DPO gradients with the oracle reward gradient, with empirical gains in iterative and online DPO.
-
Federated Linear Dueling Bandits
A new federated linear dueling bandit algorithm with claimed sublinear regret, but the key proof step is invalid.
-
Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
A conceptual framework classifies human feedback to RL agents along nine dimensions and seven quality criteria, unifying human-centered, interface-centered, and model-centered design perspectives.
-
Active Human Feedback Collection via Neural Contextual Dueling Bandits
Neural-ADB claims an O~((d/T)^(1/2)) worst sub-optimality gap for active contextual dueling bandits with non-linear rewards, but the proof relies on a reversed matrix inequality.
Discussion (0). Continue with ORCID to comment.