REVIEW 17 cited by
Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).
Forward citations
Cited by 17 Pith papers
-
What does AI consider praiseworthy?
LLM praise and critique of user-stated intentions are driven more by source trustworthiness than ideology, align broadly with human moral scores, and show no country-of-origin bias.
-
WorldPM: Scaling Human Preference Modeling
Preference modeling exhibits scaling laws for objective and adversarial tasks, with up to 5-14% gains when used as initialization for fine-tuning on smaller human-preference datasets.
-
Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
The authors show that DPO, IPO, DRO, PPO and SAC can be viewed through one loss-construction framework, but their claim that DPO's target distribution differs from the standard DPO target rests on an incomplete argument.
-
Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
Filtering DPO training data by the absolute length-normalized reference model log-probability gap between preferred and rejected responses improves MT-Bench scores with 30-50% of the data.
-
Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
The paper proposes confidence and critique metrics for LLM self-correction, finds a trade-off between them under prompting and in-context learning, and introduces a data-format transformation (CCT) that improves both ...
-
A Systematic Examination of Preference Learning through the Lens of Instruction-Following
Across 48K synthetic instruction prompts, DPO models train best on preference pairs with shared prefixes, high chosen-rejected contrast, and moderate prompt difficulty.
-
Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
RLHF training on LLMs shows diminishing returns from more response samples, larger reward models, and larger policy models, so it scales less efficiently than pretraining.
-
Self-Generated Critiques Boost Reward Modeling for Language Models
Self-generated, label-filtered critiques jointly trained with reward prediction improve preference accuracy and data efficiency without a stronger teacher model.
-
Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
BDPO replaces the rejected response probability in the DPO loss denominator with a mixture of the learned and reference policies, yielding better chosen-response probability and better benchmark scores.
-
AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs
AutoMixAlign adaptively reweights or resamples task data during DPO training to match specialist-model losses, improving average performance on helpfulness, coding, and safety benchmarks compared to standard DPO and m...
-
Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective
Fine-tuning LLMs on observational logs can amplify spurious patterns such as weekday effects, and a confounder-subtraction method, DeconfoundLM, restores causal signal in self-built simulations.
-
Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance
The paper proposes C3F, a two-axis framework rating GenAI compliance capability and standard criticality, and applies it to 15 models and 34 standards.
-
Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
RPO unifies DPO, IPO, SimPO, and RLOO as special cases of one reward-matching loss, and a synthetic-judge ablation finds online RPO-bwd outperforms RLOO.
-
Technical Report of TeleChat2, TeleChat2.5 and T1
The released T1-115B open-weight model outperforms OpenAI's o1-mini and GPT-4o on MATH500, AlignBench, and IFEval, despite using a standard dense transformer architecture.
-
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization
A DPO variant that kernelizes the preference loss and swaps KL for other divergences is claimed to improve alignment, but the math and evaluation do not support the state-of-the-art claim.
-
Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality
A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.
-
Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.
Discussion (0). Continue with ORCID to comment.