REVIEW 6 cited by
Jackpot! Alignment as a Maximal Lottery
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Jackpot! Alignment as a Maximal Lottery
read the original abstract
Reinforcement Learning from Human Feedback (RLHF), the standard for aligning Large Language Models (LLMs) with human values, is known to fail to satisfy properties that are intuitively desirable, such as respecting the preferences of the majority \cite{ge2024axioms}. To overcome these issues, we propose the use of a probabilistic Social Choice rule called \emph{maximal lotteries} as a replacement for RLHF. We show that a family of alignment techniques, namely Nash Learning from Human Feedback (NLHF) \cite{munos2023nash} and variants, approximate maximal lottery outcomes and thus inherit its beneficial properties. We confirm experimentally that our proposed methodology handles situations that arise when working with preferences more robustly than standard RLHF, including supporting the preferences of the majority, providing principled ways of handling non-transitivities in the preference data, and robustness to irrelevant alternatives. This results in systems that better incorporate human values and respect human intentions.
Forward citations
Cited by 6 Pith papers
-
Efficient Exploration for Iterative Nash Preference Optimization
An explicitly exploratory iterative NLHF method achieves O(sqrt(T)) regret for Nash equilibria under general preference models, removing the exponential KL dependence that plagues standard iterative approaches.
-
The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisions
The angular mean of voter scoring vectors satisfies long-run individual proportionality for sequential linear ranking decisions.
-
Nash without Numbers: A Social Choice Approach to Mixed Equilibria in Context-Ordinal Games
Context-ordinal Nash equilibria are defined via social choice aggregation of ordinal preferences, shown to exist under mild conditions, with regularization, approximation, regret notions, complexity results, and learn...
-
Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization
Sequential DPO-style training makes coalition policies reconstructable by arithmetic on singleton models, enabling linear-cost approximation of Shapley data values.
-
Fair Agents: Balancing Multistakeholder Alignment in Multi-Agent Personalization Systems
The authors propose a conceptual framework integrating stakeholder-LLM alignment methods, social choice-based aggregation for collective decisions, and stakeholder-centric evaluations to achieve fair multi-agent perso...
-
AI Alignment From Social Choice Perspectives
This survey examines applications of social choice theory to aggregating human feedback in AI alignment, identifying failure modes and expanding design options for disagreement.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.