Pith. sign in

REVIEW 9 cited by

A Minimaximalist Approach to Reinforcement Learning from Human Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.04056 v2 pith:5VMG3SBS submitted 2024-01-08 cs.LG

Gokul Swamy , Christoph Dann , Rahul Kidambi , Zhiwei Steven Wu , Alekh Agarwal This is my paper · ORCID
classification cs.LG
keywords approachhumanlearningpreferencepreferenceswhileapproachesfeedback
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training and is therefore rather simple to implement. Our approach is maximalist in that it provably handles non-Markovian, intransitive, and stochastic preferences while being robust to the compounding errors that plague offline approaches to sequential prediction. To achieve the preceding qualities, we build upon the concept of a Minimax Winner (MW), a notion of preference aggregation from the social choice theory literature that frames learning from preferences as a zero-sum game between two policies. By leveraging the symmetry of this game, we prove that rather than using the traditional technique of dueling two policies to compute the MW, we can simply have a single agent play against itself while maintaining strong convergence guarantees. Practically, this corresponds to sampling multiple trajectories from a policy, asking a preference or teacher model to compare them, and then using the proportion of wins as the reward for a particular trajectory. We demonstrate that on a suite of continuous control tasks, we are able to learn significantly more efficiently than reward-model based approaches while maintaining robustness to the intransitive and stochastic preferences that frequently occur in practice when aggregating human judgments.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

    cs.LG 2025-05 accept novelty 7.0 of 10

    NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.

  2. Visual Token Compression Enhances Robustness of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Pruning visual tokens farthest from the text feature space at selected 'robust' layers improves MLLM jailbreak defense (average +13.29% RAR) and slightly reduces hallucination.

  3. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  4. e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.

  5. Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.

  6. Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Value-incentivized exploration via best-response values gives near-optimal regret for NE/CCE in linear-model Markov games without explicit uncertainty bonuses.

  7. Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective

    cs.LG 2026-01 conditional novelty 5.0 of 10

    The optimal reward for KL-regularized LLM alignment is a threshold function—reward B above a prompt-dependent cutoff, 0 below—which can be estimated from base-model samples and integrated into decoding-time alignment.

  8. Game Theory Meets LLM and Agentic AI: Reimagining Cybersecurity for the Age of Intelligent Threats

    cs.CR 2025-07 conditional novelty 4.0 of 10

    A conceptual chapter proposing LLM-Nash and LLM-Stackelberg games that shift equilibrium analysis from action spaces to prompt spaces for AI-driven cyber defense.

  9. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Pith tools