Pith. sign in

REVIEW 5 cited by

Bootstrapping Language Models with DPO Implicit Rewards

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.09760 v2 pith:73Z72HUC submitted 2024-06-14 cs.CL cs.LG

Changyu Chen , Zichen Liu , Chao Du , Tianyu Pang , Qian Liu , Arunesh Sinha , Pradeep Varakantham , Min Lin This is my paper · ORCID
classification cs.CLcs.LG
keywords implicitpreferencerewardapproachdatasetmodelsrewardswork
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularized reward shaping to make the preference dataset length-unbiased; 2) experience replay to enhance the quality of the preference dataset. Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8$\\%$ in lengthcontrolled win rate on AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https://github.com/sail-sg/dice.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Explicit Preference Optimization: No Need for an Implicit Reward Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.

  2. Preference Optimization for Reasoning with Pseudo Feedback

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Generating pseudo test cases from frontier models or self-consistency and using them as DPO preference signals improves 7B LLMs on math and code benchmarks, sometimes beating 72B models.

  3. Coupled Variational Reinforcement Learning for Language Model General Reasoning

    cs.CL 2025-12 conditional novelty 5.0 of 10

    CoVRL trains an LLM on a mixture of question-only and answer-guided reasoning traces, using the model's own answer probability as reward, and reports consistent gains on math and general-reasoning benchmarks.

  4. Direct Advantage Regression: Aligning LLMs with Online AI Reward

    cs.AI 2025-04 conditional novelty 5.0 of 10

    DAR aligns LLMs through advantage-weighted supervised fine-tuning on online AI scalar rewards with dual KL regularization, and reports win-rate gains over online RLHF and online preference methods.

  5. Bag of Tricks for Inference-time Computation of LLM Reasoning

    cs.AI 2025-02 conditional novelty 4.0 of 10

    An empirical study showing that inference-time LLM reasoning methods are highly sensitive to sampling hyperparameters and reward-model choices, with recommended defaults of temperature 0.8 and top-p 0.9.

Pith tools