Optimistic regression algorithms with Gibbs updates achieve high-probability KL-regret that degrades gracefully under pointwise KL misspecification for bandits and stagewise KL Bellman misspecification for episodic RL.
Value-incentivized preference optimization: A unified approach to online and offline rlhf.arXiv preprint arXiv:2405.19320
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
fields
cs.LG 2years
2026 2representative citing papers
PEPO is a single-step pessimistic ensemble algorithm for direct preference optimization that provably avoids over-optimization by depending only on single-policy concentrability without knowing the data distribution or learning an explicit reward model.
citing papers explorer
-
Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Optimistic regression algorithms with Gibbs updates achieve high-probability KL-regret that degrades gracefully under pointwise KL misspecification for bandits and stagewise KL Bellman misspecification for episodic RL.
-
Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution
PEPO is a single-step pessimistic ensemble algorithm for direct preference optimization that provably avoids over-optimization by depending only on single-policy concentrability without knowing the data distribution or learning an explicit reward model.