Pith. sign in

REVIEW 15 cited by

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.00409 v2 pith:RCMQI7W7 submitted 2024-03-01 cs.LG cs.CL

Sayak Ray Chowdhury , Anush Kini , Nagarajan Natarajan This is my paper · ORCID
classification cs.LGcs.CL
keywords policypreferencemodelsnoisyrobustdatasethumanlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order $O(\frac{1}{1-2\epsilon}\sqrt{\frac{d}{n}})$, where $\epsilon < 1/2$ is flip rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Under linear-model assumptions, offline RLHF and DPO both reduce to logistic regression, and privatizing labels before corruption (LTC) carries an extra c(ε) factor in the error bounds compared to corrupting before pr...

  2. Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathways

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Safety refusals travel through shared cross-lingual 'pathways' inside LLMs, and fine-tuning only those pathway parameters (under 1%) sharply improves low-resource-language safety.

  3. Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    PU-DPO applies positive-unlabeled learning to preference optimization so that report generators learn to mention findings that are present but missing from noisy training reports.

  4. Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Under 20–40% random preference-label flips, PACMR-DPO—a VNet-reweighted DPO with a prompt-augmentation-consistency meta-objective and central-difference LoRA meta-gradients—outperforms cDPO, IPO, rDPO, and Dr.DPO in j...

  5. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

  6. MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.

  7. Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Filtering DPO training data by the absolute length-normalized reference model log-probability gap between preferred and rejected responses improves MT-Bench scores with 30-50% of the data.

  8. T-REG: Preference Optimization with Token-Level Reward Regularization

    cs.CL 2024-12 conditional novelty 6.0 of 10

    T-REG adds contrastive-prompt self-generated token-level rewards as a weighted regularizer to DPO and SimPO, improving Alpaca Eval 2 length-controlled win rate by up to 3.8% and Arena-Hard win rate by up to 4.4% over ...

  9. Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

    cs.LG 2025-10 conditional novelty 5.0 of 10

    MADPO replaces DPO's fixed temperature with an instance-level weight derived from a trained reward model, amplifying low-margin preference pairs and dampening high-margin pairs.

  10. On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

    cs.LG 2025-05 reject novelty 4.0 of 10

    Symmetric losses preserve action rankings under symmetric label noise, and the paper's claim that they also handle asymmetric noise is invalid.

  11. Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

    cs.AI 2025-05 reject novelty 4.0 of 10

    Training 2D-DPO with an expected loss over uniform segment-score perturbations yields higher win rates under score noise than a clean-trained 2D-DPO baseline, though the comparison is confounded.

  12. Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

    cs.LG 2024-12 reject novelty 4.0 of 10

    Cal-DPO modifies DPO by adding squared losses that anchor the policy's implicit rewards of chosen and rejected responses to plus or minus 1/(2*beta), reporting gains on reasoning, summarization, and dialogue benchmarks.

  13. A Technical Survey of Reinforcement Learning Techniques for Large Language Models

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A survey of RL methods for LLMs that organizes the field by reward modeling, feedback source, and optimization strategy, with benchmark tables favoring a scalar-regression UNA variant over DPO and KTO in offline alignment.

  14. An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems

    cs.CL 2024-12 unverdicted novelty 3.0 of 10

    A survey and position paper that reviews LLM prompting, RAG, and RL techniques and argues they could support open-ended implementation generation, without presenting new results.

  15. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools