Pith. sign in

REVIEW 6 cited by

Axioms for AI Alignment from Human Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.14758 v2 pith:SVCSQVGD submitted 2024-05-23 cs.GT cs.AIcs.LG

Luise Ge , Daniel Halpern , Evi Micha , Ariel D. Procaccia , Itai Shapira , Yevgeniy Vorobeychik , Junlin Wu This is my paper · ORCID
classification cs.GTcs.AIcs.LG
keywords axiomschoicelearningrewardsocialaggregationfailfeedback
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans. The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory. From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards. We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms. In response, we develop novel rules for learning reward functions with strong axiomatic guarantees. A key innovation from the standpoint of social choice is that our problem has a linear structure, which greatly restricts the space of feasible rules and leads to a new paradigm that we call linear social choice.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Internal Pluralism and the Limits of Pairwise Comparisons

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Under internal pluralism, forced local pairwise comparisons erase inseparable priorities and distort conflicted answers, while allowing indecision reports can sharply reduce queries needed to learn preference weights.

  2. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

    cs.LG 2025-05 accept novelty 7.0 of 10

    NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.

  3. Clone-Robust AI Alignment

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A Voronoi-weighted maximum likelihood estimator for RLHF is robust to adding approximate clone responses, unlike the standard regularized MLE.

  4. Jackpot! Alignment as a Maximal Lottery

    cs.AI 2025-01 conditional novelty 5.0 of 10

    Nash Learning from Human Feedback is shown to approximate the maximal lottery voting rule, which the authors argue better reflects majority preferences than Borda-like RLHF.

  5. Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

    cs.LG 2025-01 conditional novelty 5.0 of 10

    Applying a utility-inspired, threshold-based transformation to individual rewards before summing them improved the harmlessness of an RLHF-trained 2B language model without reducing helpfulness.

  6. Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning

    cs.LG 2025-08 reject novelty 3.0 of 10

    A proposed Counterfactual Trust Score aggregates drift, uncertainty, fairness violations, and counterfactual consistency into a single reward-model trust signal, evaluated only via a self-composed score on an unnamed ...

Pith tools