REVIEW 5 cited by
Direct Alignment with Heterogeneous Preferences
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous. We formalize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption. We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine improvements under different information settings, focusing on direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficient consistent direct loss exists in this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment.
Forward citations
Cited by 5 Pith papers
-
Internal Pluralism and the Limits of Pairwise Comparisons
Under internal pluralism, forced local pairwise comparisons erase inseparable priorities and distort conflicted answers, while allowing indecision reports can sharply reduce queries needed to learn preference weights.
-
Power and Limitations of Aggregation in Compound AI Systems
In a principal-agent model of compound AI, aggregation expands the set of outputs a designer can elicit exactly when one of three mechanisms — feasibility expansion, support expansion, or binding set contraction — hol...
-
Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.
-
Preference learning made easy: Everything should be understood through win rate
Under two axioms (preference-consistency and prevalence-consistency), the only distribution-grounded evaluation for preference learning is h-win rate, and most popular alignment methods can be classified by whether th...
-
Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers
A taxonomy-based survey of bidirectional game theory and LLM research, spanning evaluation, alignment, economic competition, and LLM-driven game solving.
Discussion (0). Continue with ORCID to comment.