Pith. sign in

REVIEW 4 cited by

Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07314 v3 pith:T3CPF6FM submitted 2024-02-11 cs.LG stat.ML

Chenlu Ye , Wei Xiong , Yuheng Zhang , Hanze Dong , Nan Jiang , Tong Zhang This is my paper · ORCID
classification cs.LGstat.ML
keywords preferencelearningoraclegeneralfeedbackformulationframeworkhuman
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Optimistic regression algorithms with Gibbs updates achieve high-probability KL-regret that degrades gracefully under pointwise KL misspecification for bandits and stagewise KL Bellman misspecification for episodic RL.

  2. Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Outcome-based online RL is tractable under coverability with general function approximation, but there are MDPs where trajectory-level feedback costs exponentially more samples than per-step feedback.

  3. Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

    stat.ML 2025-07 conditional novelty 6.0 of 10

    Smoothed Best-of-N has finite-sample KL and regret bounds under imperfect reward models, and tuning its temperature can make its regret bound beat hard Best-of-N in the overoptimization regime.

  4. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

Pith tools