Pith. sign in

REVIEW 4 cited by

Policy Optimization in RLHF: The Impact of Out-of-preference Data

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.10584 v2 pith:OGZPKQGT submitted 2023-12-17 cs.LG

Ziniu Li , Tian Xu , Yang Yu This is my paper
classification cs.LG
keywords datarmb-pooptimizationpolicymodelout-of-preferencerewardgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+ is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally uses policy-generated data, and RMB-PO+ further leverages new, preference-free data. We examine the impact of such out-of-preference data. Our study, conducted through controlled and synthetic experiments, demonstrates that DPO performs poorly, whereas RMB-PO+ performs the best. In particular, even when providing the policy model with a good feature representation, we find that policy optimization with adequate out-of-preference data significantly improves performance by harnessing the reward model's generalization capabilities.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SGPO: Self-Generated Preference Optimization based on Self-Improver

    cs.CL 2025-07 conditional novelty 6.0 of 10

    SGPO uses one shared model to refine its own responses and then optimize with DPO on those self-generated preference pairs, outperforming DPO and SPIN on AlpacaEval 2.0 and Arena-Hard without external preference labels.

  2. Explicit Preference Optimization: No Need for an Implicit Reward Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.

  3. Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

    stat.ML 2025-06 conditional novelty 5.0 of 10

    RLHF reward modeling satisfies pairwise majority and Condorcet consistency when each response pair is labeled once, because the maximum likelihood ranking then matches the Copeland rule.

  4. AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

    q-bio.BM 2025-06 reject novelty 4.0 of 10

    DPO with contrastive sequence-annotation alignment improves GO term prediction by 2 to 4 percent relative F1-Max over supervised fine-tuning alone.

Pith tools