Pith. sign in

REVIEW 8 cited by

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.00254 v2 pith:BXUE7VOQ submitted 2024-04-30 cs.AI cs.LG

Chanwoo Park , Mingyang Liu , Dingwen Kong , Kaiqing Zhang , Asuman Ozdaglar This is my paper · ORCID
classification cs.AIcs.LG
keywords humanfeedbackapproacheslearningpreferencepreferencesrewardrlhf
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

    cs.LG 2025-05 accept novelty 7.0 of 10

    NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.

  3. Collaborative and Efficient Fine-tuning: Leveraging Task Similarity

    cs.LG 2026-02 conditional novelty 6.0 of 10

    CoLoRA shares a low-rank adapter pair across users plus a small personal matrix, improving fine-tuning for similar tasks and providing a recovery guarantee.

  4. Uncertainty Quantification for Ranking with Heterogeneous Preferences

    stat.ME 2025-09 conditional novelty 6.0 of 10

    A debiased regularized estimator provides entrywise-accurate scores and uncertainty quantification for heterogeneous Bradley-Terry-Luce rankings.

  5. Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

    stat.ML 2025-06 conditional novelty 5.0 of 10

    RLHF reward modeling satisfies pairwise majority and Condorcet consistency when each response pair is labeled once, because the maximum likelihood ranking then matches the Copeland rule.

  6. MOSLIM:Align with diverse preferences in prompts through reward classification

    cs.CL 2025-05 reject novelty 5.0 of 10

    A prompt-controlled multi-objective alignment method using a multi-head classification reward model and a z-score reward mapping, claimed to work with off-the-shelf models.

  7. Context Engineering: A Practitioner Methodology for Structured Human-AI Collaboration

    cs.AI 2026-04 conditional novelty 4.0 of 10

    Structured five-role context packages and a four-phase pipeline were associated with cutting average AI task iterations from 3.8 to 2.0 and raising first-pass acceptance from 32% to 55% in an observational single-oper...

  8. T-POP: Test-Time Personalization with Online Preference Feedback

    cs.LG 2025-09 conditional novelty 4.0 of 10

    T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

Pith tools