Pith. sign in

REVIEW 16 cited by

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.16436 v3 pith:LDGMNZ3E submitted 2024-05-26 cs.LG cs.AIstat.ML

Zhihan Liu , Miao Lu , Shenao Zhang , Boyi Liu , Hongyi Guo , Yingxiang Yang , Jose Blanchet , Zhaoran Wang This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords losspreferencealgorithmpolicyrewardoptimizationoveroptimizationaligning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Aligning generative models with human preference via RLHF typically suffers from overoptimization, where an imperfectly learned reward model can misguide the generative model to output undesired responses. We investigate this problem in a principled manner by identifying the source of the misalignment as a form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model; one that simultaneously minimizes the maximum likelihood estimation of the loss and a reward penalty term. Here, the reward penalty term is introduced to prevent the policy from choosing actions with spurious high proxy rewards, resulting in provable sample efficiency of the algorithm under a partial coverage style condition. Moving from theory to practice, the proposed algorithm further enjoys an equivalent but surprisingly easy-to-implement reformulation. Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines: (i) a preference optimization loss that directly aligns the policy with human preference, and (ii) a supervised learning loss that explicitly imitates the policy with a (suitable) baseline distribution. In the context of aligning large language models (LLM), this objective fuses the direct preference optimization (DPO) loss with the supervised fine-tuning (SFT) loss to help mitigate the overoptimization towards undesired responses, for which we name the algorithm Regularized Preference Optimization (RPO). Experiments of aligning LLMs demonstrate the improved performance of RPO compared with DPO baselines. Our work sheds light on the interplay between preference optimization and SFT in tuning LLMs with both theoretical guarantees and empirical evidence.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Improvement in Language Models: The Sharpening Mechanism

    cs.AI 2024-12 conditional novelty 7.0 of 10

    Self-improvement in language models can be understood as amortizing best-of-N inference-time selection, with minimax-optimal guarantees for SFT and provable coverage-free benefits for RL with exploration.

  2. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  3. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

  4. Policy-labeled Preference Learning: Is Preference Enough for RLHF?

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Policy-labeled Preference Learning models preferences with regret and behavior-policy labels, adds contrastive KL regularization, and reports improved RLHF performance on MetaWorld offline and online control tasks.

  5. Design Considerations in Offline Preference-based RL

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A unified theory of offline RLHF shows that loss curvature and data coverage control suboptimality, explaining why squared-loss IPO is more stable than logistic-loss DPO.

  6. Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

    cs.AI 2025-02 conditional novelty 6.0 of 10

    Sampling dispreferred completions proportionally to the current reward model, as in contrastive divergence, improves preference-optimization performance and is framed as NLL estimation.

  7. BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

    cs.LG 2025-01 reject novelty 6.0 of 10

    BRiTE is an EM-style algorithm that uses RL to sample high-likelihood reasoning chains and fine-tunes the LLM on them, with a theoretical 1/T convergence rate and mixed empirical gains.

  8. AlphaPO: Reward Shape Matters for LLM Alignment

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AlphaPO replaces SimPO's log reward with a length-normalized alpha-divergence reward; a slightly positive alpha improves AlpacaEval 2 length-controlled win rates by 7-10 percent on two instruct models.

  9. Teaching LLMs to Refine with Tools

    cs.CL 2024-12 conditional novelty 6.0 of 10

    CaP trains LLMs to fix chain-of-thought math solutions by producing program-of-thought code, and shows that DPO preference optimization is essential for the refinement to actually improve accuracy.

  10. Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A contrastive token-scoring method identifies 'critical tokens' in incorrect reasoning traces and penalizes them in DPO, yielding small but consistent accuracy gains on math benchmarks.

  11. Continual SFT Matches Multimodal RLHF with Negative Supervision

    cs.LG 2024-11 conditional novelty 6.0 of 10

    nSFT matches multimodal RLHF performance by converting rejected responses into corrective SFT data, without pairwise preference optimization.

  12. Rethinking DPO: The Role of Rejected Responses in Preference Misalignment

    cs.AI 2025-06 conditional novelty 5.0 of 10

    BDPO replaces the rejected response probability in the DPO loss denominator with a mixture of the learned and reference policies, yielding better chosen-response probability and better benchmark scores.

  13. DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs

    cs.SE 2024-11 conditional novelty 5.0 of 10

    DSTC builds preference pairs by selecting the hardest self-generated test that the best self-generated code passes, then fine-tunes code LMs with DPO or KTO to improve pass@1 accuracy.

  14. Robust Single-Stage Fully Sparse 3D Object Detection via Detachable Latent Diffusion

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    The submitted manuscript's abstract and full text are mismatched; the claimed 3D detection method is not present in the body.

  15. LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

    cs.CV 2025-06 conditional novelty 4.0 of 10

    LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.

  16. DPO-Shift: Shifting the Distribution of Direct Preference Optimization

    cs.CL 2025-02 conditional novelty 4.0 of 10

    DPO-Shift multiplies the rejected-response term in the DPO loss by a factor f(λ) below 1, shifting chosen-response probabilities upward at the expense of some reward margin.

Pith tools