REVIEW 8 cited by
Dataset Reset Policy Optimization for RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Reinforcement Learning (RL) from Human Preference-based feedback is a popular paradigm for fine-tuning generative models, which has produced impressive models such as GPT-4 and Claude3 Opus. This framework often consists of two steps: learning a reward model from an offline preference dataset followed by running online RL to optimize the learned reward model. In this work, leveraging the idea of reset, we propose a new RLHF algorithm with provable guarantees. Motivated by the fact that offline preference dataset provides informative states (i.e., data that is preferred by the labelers), our new algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing offline preference dataset into the online policy training procedure via dataset reset: it directly resets the policy optimizer to the states in the offline dataset, instead of always starting from the initial state distribution. In theory, we show that DR-PO learns to perform at least as good as any policy that is covered by the offline dataset under general function approximation with finite sample complexity. In experiments, we demonstrate that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH) dataset, the generation from DR-PO is better than that from Proximal Policy Optimization (PPO) and Direction Preference Optimization (DPO), under the metric of GPT4 win-rate. Code for this work can be found at https://github.com/Cornell-RL/drpo.
Forward citations
Cited by 8 Pith papers
-
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
CALM uses bilevel optimization to tune per-vocabulary temperature-like logit adjustments during LLM fine-tuning, and reports improved out-of-domain calibration for aligned language models.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Outcome-based Exploration for LLM Reasoning
Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.
-
Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
IRO trains a sequence of small value functions that reweight a frozen LLM's own samples, achieving test-time alignment gains that outperform Best-of-N search on AlpacaEval 2.0 and TL;DR.
-
Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment
SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.
-
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
A hybrid online-plus-offline preference optimization algorithm, HPO, provably needs fewer samples than pure online or offline RLHF in linear MDP settings.
-
GPO: Learning from Critical Steps to Improve LLM Reasoning
Guided Pivotal Optimization improves LLM reasoning post-training by resetting rollouts at an advantage-estimated critical step, beating each of five base optimizers on seven benchmarks in 34 of 35 comparisons.
-
Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
RLHF reward modeling satisfies pairwise majority and Condorcet consistency when each response pair is labeled once, because the maximum likelihood ranking then matches the Copeland rule.
Discussion (0). Continue with ORCID to comment.