Pith. sign in

REVIEW 17 cited by

A Minimaximalist Approach to Reinforcement Learning from Human Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.04056 v2 pith:5VMG3SBS submitted 2024-01-08 cs.LG

Gokul Swamy , Christoph Dann , Rahul Kidambi , Zhiwei Steven Wu , Alekh Agarwal This is my paper · ORCID
classification cs.LG
keywords approachhumanlearningpreferencepreferenceswhileapproachesfeedback
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training and is therefore rather simple to implement. Our approach is maximalist in that it provably handles non-Markovian, intransitive, and stochastic preferences while being robust to the compounding errors that plague offline approaches to sequential prediction. To achieve the preceding qualities, we build upon the concept of a Minimax Winner (MW), a notion of preference aggregation from the social choice theory literature that frames learning from preferences as a zero-sum game between two policies. By leveraging the symmetry of this game, we prove that rather than using the traditional technique of dueling two policies to compute the MW, we can simply have a single agent play against itself while maintaining strong convergence guarantees. Practically, this corresponds to sampling multiple trajectories from a policy, asking a preference or teacher model to compare them, and then using the proportion of wins as the reward for a particular trajectory. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches while maintaining robustness to the intransitive and stochastic preferences that frequently occur in practice when aggregating human judgments.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

    cs.LG 2025-05 accept novelty 7.0 of 10

    NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.

  2. Visual Token Compression Enhances Robustness of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Pruning visual tokens farthest from the text feature space at selected 'robust' layers improves MLLM jailbreak defense (average +13.29% RAR) and slightly reduces hallucination.

  3. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  4. e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.

  5. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

  6. Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Value-incentivized exploration via best-response values gives near-optimal regret for NE/CCE in linear-model Markov games without explicit uncertainty bonuses.

  7. Design Considerations in Offline Preference-based RL

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A unified theory of offline RLHF shows that loss curvature and data coverage control suboptimality, explaining why squared-loss IPO is more stable than logistic-loss DPO.

  8. The Battling Influencers Game: Nash Equilibria Structure of a Potential Game and Implications to Value Alignment

    cs.GT 2025-02 conditional novelty 6.0 of 10

    A new potential game shows that when influencers compete to shape a receiver's aggregate opinion, any pure Nash equilibrium forces all but at most one influencer to the most extreme allowed action.

  9. Online Preference Alignment for Language Models via Count-based Exploration

    cs.LG 2025-01 conditional novelty 6.0 of 10

    COPO adds a pseudo-count exploration bonus to DPO-style preference optimization and reports improved online RLHF results on Zephyr and Llama-3.

  10. Clone-Robust AI Alignment

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A Voronoi-weighted maximum likelihood estimator for RLHF is robust to adding approximate clone responses, unlike the standard regularized MLE.

  11. Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A multimodal LLM can improve itself using only unlabeled images by self-generating questions, self-enhancing answers, and adding a description-alignment loss to DPO.

  12. Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective

    cs.LG 2026-01 conditional novelty 5.0 of 10

    The optimal reward for KL-regularized LLM alignment is a threshold function—reward B above a prompt-dependent cutoff, 0 below—which can be estimated from base-model samples and integrated into decoding-time alignment.

  13. Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learning

    cs.MA 2025-02 conditional novelty 5.0 of 10

    LLM-generated, agent-specific potential-based rewards accelerate sparse-reward MARL training in grid world and pistonball benchmarks.

  14. Jackpot! Alignment as a Maximal Lottery

    cs.AI 2025-01 conditional novelty 5.0 of 10

    Nash Learning from Human Feedback is shown to approximate the maximal lottery voting rule, which the authors argue better reflects majority preferences than Borda-like RLHF.

  15. Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

    cs.LG 2025-01 conditional novelty 5.0 of 10

    Applying a utility-inspired, threshold-based transformation to individual rewards before summing them improved the harmlessness of an RLHF-trained 2B language model without reducing helpfulness.

  16. Game Theory Meets LLM and Agentic AI: Reimagining Cybersecurity for the Age of Intelligent Threats

    cs.CR 2025-07 conditional novelty 4.0 of 10

    A conceptual chapter proposing LLM-Nash and LLM-Stackelberg games that shift equilibrium analysis from action spaces to prompt spaces for AI-driven cyber defense.

  17. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Pith tools