Pith. sign in

REVIEW 10 cited by

Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.08358 v2 pith:X662LZJJ submitted 2023-12-13 cs.LG cs.AIstat.ML

Anand Siththaranjan , Cassidy Laidlaw , Dylan Hadfield-Menell This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords datahiddenlearningpreferencecontextrlhffeedbackhuman
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is not represented in the data used to train a preference model. This captures common issues of data collection, such as having human annotators with varied preferences, cognitive processes that result in seemingly irrational behavior, and combining data labeled according to different criteria. We prove that standard applications of preference learning, including reinforcement learning from human feedback (RLHF), implicitly aggregate over hidden contexts according to a well-known voting rule called Borda count. We show this can produce counter-intuitive results that are very different from other methods which implicitly aggregate via expected utility. Furthermore, our analysis formalizes the way that preference learning from users with diverse values tacitly implements a social choice function. A key implication of this result is that annotators have an incentive to misreport their preferences in order to influence the learned model, leading to vulnerabilities in the deployment of RLHF. As a step towards mitigating these problems, we introduce a class of methods called distributional preference learning (DPL). DPL methods estimate a distribution of possible score values for each alternative in order to better account for hidden context. Experimental results indicate that applying DPL to RLHF for LLM chatbots identifies hidden context in the data and significantly reduces subsequent jailbreak vulnerability. Our code and data are available at https://github.com/cassidylaidlaw/hidden-context

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

    cs.LG 2025-05 accept novelty 7.0 of 10

    NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.

  2. CTR-Driven Advertising Image Generation with Multimodal Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A CTR-driven advertising image generation pipeline that pre-trains an MLLM prompt model, trains an MLLM pairwise reward model on real click data, and fine-tunes with DPO plus a product-centric preference loss.

  3. Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes

    cs.CL 2025-01 conditional novelty 6.0 of 10

    LLMs systematically inflate Republican positions and deflate Democratic positions relative to human survey responses, consistent with representativeness heuristics.

  4. Clone-Robust AI Alignment

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A Voronoi-weighted maximum likelihood estimator for RLHF is robust to adding approximate clone responses, unlike the standard regularized MLE.

  5. Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A conceptual framework classifies human feedback to RL agents along nine dimensions and seven quality criteria, unifying human-centered, interface-centered, and model-centered design perspectives.

  6. Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

    stat.ML 2025-06 conditional novelty 5.0 of 10

    RLHF reward modeling satisfies pairwise majority and Condorcet consistency when each response pair is labeled once, because the maximum likelihood ranking then matches the Copeland rule.

  7. Jackpot! Alignment as a Maximal Lottery

    cs.AI 2025-01 conditional novelty 5.0 of 10

    Nash Learning from Human Feedback is shown to approximate the maximal lottery voting rule, which the authors argue better reflects majority preferences than Borda-like RLHF.

  8. Beyond the Binary: Capturing Diverse Preferences With Reward Regularization

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Reward models trained with a margin derived from synthetic LLM judgments better match aggregate human preferences than standard binary-trained reward models, mainly on subjective prompts.

  9. Active Query Selection for Crowd-Based Reinforcement Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    Extending the Advise algorithm with variational crowd modelling and entropy-based query selection yields faster learning in small tabular RL tasks, especially highly constrained ones.

  10. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools