Pith. sign in

REVIEW 29 cited by

A General Theoretical Paradigm to Understand Learning from Human Preferences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.12036 v2 pith:RQZSILRI submitted 2023-10-18 cs.AI cs.LGstat.ML

Mohammad Gheshlaghi Azar , Mark Rowland , Bilal Piot , Daniel Guo , Daniele Calandriello , Michal Valko , Rémi Munos This is my paper
classification cs.AIcs.LGstat.ML
keywords preferenceslearningdatageneralhumanapproximationapproximationsassumes
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation. In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called $\Psi$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of $\Psi$PO) and to identify their potential pitfalls. We then consider another special case for $\Psi$PO by setting $\Psi$ simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.

Discussion (0). Sign in to comment.

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ORPO: Monolithic Preference Optimization without Reference Model

    cs.CL 2024-03 conditional novelty 8.0 of 10

    ORPO performs preference alignment during supervised fine-tuning via a monolithic odds ratio penalty, allowing 7B models to outperform larger state-of-the-art models on alignment benchmarks.

  2. CrossVLA: Cross-Paradigm Post-Training and Inference Optimization for Vision-Language-Action Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    CrossVLA develops a surrogate log-probability estimator for DPO on flow-matching VLAs, shows DoRA outperforming LoRA by +10.4 pp mean on LIBERO, and identifies inference bottlenecks with limited caching gains.

  3. CrossVLA: Cross-Paradigm Post-Training and Inference Optimization for Vision-Language-Action Models

    cs.CV 2026-05 conditional novelty 7.0 of 10

    CrossVLA introduces a surrogate log-probability estimator to enable DPO on flow-matching VLAs, reports DoRA yielding +10.4 pp mean gains over SFT on LIBERO with 600 trials, and shows inference caching limited to 21% s...

  4. TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Introduces TBPO, which derives a Bregman-divergence density-ratio matching objective for token-level preference optimization that generalizes DPO while preserving the induced optimal policy.

  5. TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    TBPO posits a token-level Bradley-Terry model and derives a Bregman-divergence density-ratio matching loss that generalizes DPO while preserving token-level optimality.

  6. Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Introduces Block-R1 benchmark, Block-R1-41K dataset, and a conflict score to handle domain-specific optimal block sizes in RL post-training of diffusion LLMs.

  7. Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Block-R1 formulates domain block size conflicts in multi-domain RL for dLLMs, releases a 41K-sample dataset with per-sample best block sizes and a conflict score, and provides a benchmark plus simple cross-domain trai...

  8. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  9. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  10. ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Mode-local surrogate entropy asymmetrically reweights token advantages in RLVR, improving LLM math and code reasoning over SAPO, DAPO, and GTPO.

  11. Towards Spec Learning: Inference-Time Alignment from Preference Pairs

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Proposes compiling preference pairs into readable natural-language specifications for inference-time LLM alignment, claiming outperformance over DPO on dense-preference domains.

  12. Weight-Space Geometry of Offline Reasoning Training

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Comparative weight-space analysis finds SFT/RFT/RIFT colinear with similar accuracy, DFT more divergent, GRPO partially orthogonal, and DPO near-orthogonal with highest GSM8K/AIME accuracy but using 10x smaller learning rate.

  13. FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    FlowPRO applies proximalized preference optimization to flow-matching VLAs with intervention-rollback data to reach higher success rates on long-horizon bimanual tasks without rewards or critics.

  14. AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    AdaDPO uses self-adaptive stop-gradient coefficients to balance preferred and dispreferred gradients in DPO, achieving higher AlpacaEval 2 win rates than standard DPO on Llama-3-8B-Instruct.

  15. TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TBPO derives a token-level preference optimization objective from sequence-level pairwise data via Bregman divergence ratio matching that generalizes DPO and improves alignment quality.

  16. Response Time Enhances Alignment with Heterogeneous Preferences

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.

  17. Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models

    cs.LG 2026-05 conditional novelty 6.0 of 10

    Gate-DPO attenuates gradients on low-probability rejected responses to reduce probability collapse and improve chosen-response likelihood during preference optimization.

  18. Adaptive Margin RLHF via Preference over Preferences

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.

  19. Process Reinforcement through Implicit Rewards

    cs.LG 2025-02 conditional novelty 6.0 of 10

    PRIME enables online process reward model updates in LLM RL using implicit rewards from rollouts and outcome labels, yielding 15.1% average gains on reasoning benchmarks and surpassing a stronger instruct model with 1...

  20. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 5.0 of 10

    By replaying teacher prefixes with a step-decay schedule, multi-turn on-policy distillation can run without live environment interaction, matching or slightly beating online OPD accuracy.

  21. Towards Spec Learning: Inference-Time Alignment from Preference Pairs

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Spec learning compiles brief instructions and preference pairs into readable natural-language specifications that condition LLMs at inference time and can outperform DPO on domains with dense preference signals.

  22. Constitutional On-Policy Safe Distillation

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    COPSD uses a Cross-SFT cold-start followed by constitution-conditioned distillation to achieve stronger safety-helpfulness balance and lower safety tax on reasoning than prior on-policy self-distillation methods.

  23. YFPO: A Preliminary Study of Yoked Feature Preference Optimization with Neuron-Guided Rewards for Mathematical Reasoning

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    YFPO augments standard preference optimization with neuron-level activation margins from math-related features to improve LLM reasoning on math tasks.

  24. Failure Modes of Maximum Entropy RLHF

    cs.LG 2025-09 unverdicted novelty 5.0 of 10

    Derives SimPO from MaxEnt RL and reports that MaxEnt RL in online RLHF exhibits frequent overoptimization and unstable KL dynamics across scales, unlike stable KL-constrained baselines.

  25. Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A modified DPO loss with a hinge margin improves small LLM alignment on AlpacaEval by about 2 points over the APO-zero baseline.

  26. Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Selecting preference pairs whose DPO implicit reward gap is small yields better LLM alignment than random or baseline selection while using only 10% of the data.

  27. (Towards) Scalable Reliable Automated Evaluation with Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.

  28. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A comprehensive reference book organizing existing techniques for agentic AI systems across LLM substrate, reasoning, agent design patterns, inter-agent coordination, and production deployment.

  29. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 1.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools