Pith. sign in

REVIEW 11 cited by

Disentangling Length from Quality in Direct Preference Optimization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.19159 v2 pith:EMX4ERL7 submitted 2024-03-28 cs.CL cs.LG

Ryan Park , Rafael Rafailov , Stefano Ermon , Chelsea Finn This is my paper
classification cs.CLcs.LG
keywords lengthrlhfdirectverbosityapproachesbeenbiasesclassical
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation, while still maintaining improvements in model quality. We demonstrate these effects across datasets on summarization and dialogue, where we achieve up to 20\% improvement in win rates when controlling for length, despite the GPT4 judge's well-known verbosity bias.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  2. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  3. RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    RLearner-LLM's Hybrid-DPO fuses DeBERTa NLI and LLM verifier scores to deliver up to 6x higher NLI entailment than standard SFT while preserving answer coverage across academic domains.

  4. Bridging Offline and Online Reinforcement Learning for LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Semi-online DPO, which syncs the generation model every few update steps, performs nearly as well as fully online DPO and GRPO, while strongly beating offline DPO.

  5. ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Training a policy to prefer responses by optimizing only its own low-confidence (high-surprisal) tokens improves alignment over uniform token optimization in SimPO and DPO.

  6. Explicit Preference Optimization: No Need for an Implicit Reward Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.

  7. Unlocking Recursive Thinking of LLMs: Alignment via Refinement

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An offline alignment pipeline using reward-filtered self-refinement data and long chain-of-thought SFT raises an 8B model's AlpacaEval 2 win rate from 25.0% to 51.0% with roughly 14k training examples.

  8. Aligning Large Language Models with Implicit Preferences from User-Generated Content

    cs.CL 2025-06 conditional novelty 6.0 of 10

    PUGC turns unlabeled user-generated content into preference pairs by scoring policy responses against the source post as a reference, reaching 35.93% length-controlled win rate on AlpacaEval 2.0 with Mistral-7B-Instruct.

  9. MPO: Multilingual Safety Alignment via Reward Gap Optimization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MPO reduces jailbreak success in multilingual LLMs by regressing target-language reward gaps onto the English reward gap, outperforming DPO and related methods while preserving utility.

  10. Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A modified DPO loss with a hinge margin improves small LLM alignment on AlpacaEval by about 2 points over the APO-zero baseline.

  11. MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

    cs.CL 2025-06 conditional novelty 5.0 of 10

    An MoE alignment pipeline combining two SPE-DPO-trained experts and a learned routing network reports better safety and helpfulness scores than existing dual-preference alignment baselines.

Pith tools