Pith. sign in

REVIEW 8 cited by

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.00409 v2 pith:RCMQI7W7 submitted 2024-03-01 cs.LG cs.CL

Sayak Ray Chowdhury , Anush Kini , Nagarajan Natarajan This is my paper · ORCID
classification cs.LGcs.CL
keywords policypreferencemodelsnoisyrobustdatasethumanlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order $O(\frac{1}{1-2\epsilon}\sqrt{\frac{d}{n}})$, where $\epsilon < 1/2$ is flip rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Under linear-model assumptions, offline RLHF and DPO both reduce to logistic regression, and privatizing labels before corruption (LTC) carries an extra c(ε) factor in the error bounds compared to corrupting before pr...

  2. Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Under 20–40% random preference-label flips, PACMR-DPO—a VNet-reweighted DPO with a prompt-augmentation-consistency meta-objective and central-difference LoRA meta-gradients—outperforms cDPO, IPO, rDPO, and Dr.DPO in j...

  3. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

  4. MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.

  5. Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

    cs.LG 2025-10 conditional novelty 5.0 of 10

    MADPO replaces DPO's fixed temperature with an instance-level weight derived from a trained reward model, amplifying low-margin preference pairs and dampening high-margin pairs.

  6. On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

    cs.LG 2025-05 reject novelty 4.0 of 10

    Symmetric losses preserve action rankings under symmetric label noise, and the paper's claim that they also handle asymmetric noise is invalid.

  7. A Technical Survey of Reinforcement Learning Techniques for Large Language Models

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A survey of RL methods for LLMs that organizes the field by reward modeling, feedback source, and optimization strategy, with benchmark tables favoring a scalar-regression UNA variant over DPO and KTO in offline alignment.

  8. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools