Pith. sign in

REVIEW 15 cited by

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.00254 v2 pith:BXUE7VOQ submitted 2024-04-30 cs.AI cs.LG

Chanwoo Park , Mingyang Liu , Dingwen Kong , Kaiqing Zhang , Asuman Ozdaglar This is my paper · ORCID
classification cs.AIcs.LG
keywords humanfeedbackapproacheslearningpreferencepreferencesrewardrlhf
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Metric Distortion for Tournament Voting and Beyond

    cs.GT 2025-05 conditional novelty 8.0 of 10

    Deterministic tournament rules have metric distortion between 3.1128 and 3.9312, and deterministic k-tournament rules approach distortion 3 while randomized 3-tournament rules can beat 3.

  2. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  3. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

    cs.LG 2025-05 accept novelty 7.0 of 10

    NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.

  4. Collaborative and Efficient Fine-tuning: Leveraging Task Similarity

    cs.LG 2026-02 conditional novelty 6.0 of 10

    CoLoRA shares a low-rank adapter pair across users plus a small personal matrix, improving fine-tuning for similar tasks and providing a recovery guarantee.

  5. Uncertainty Quantification for Ranking with Heterogeneous Preferences

    stat.ME 2025-09 conditional novelty 6.0 of 10

    A debiased regularized estimator provides entrywise-accurate scores and uncertainty quantification for heterogeneous Bradley-Terry-Luce rankings.

  6. Pairwise Calibrated Rewards for Pluralistic Alignment

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A small ensemble of reward functions can be trained to match pairwise human preference frequencies, offering a practical route to pluralistic AI alignment.

  7. Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models

    cs.CL 2025-04 conditional novelty 6.0 of 10

    MPO uses a meta reward model to continuously rewrite the reward model's evaluation prompt during PPO training, and the resulting models beat static-prompt RLAIF baselines on four tasks.

  8. Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment

    cs.CL 2025-04 conditional novelty 6.0 of 10

    Persona-judge applies speculative decoding between two preference-prompted copies of the same LLM to achieve training-free personalized alignment.

  9. On the Way to LLM Personalization: Learning to Remember User Conversations

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Finetuning a LoRA adapter on self-generated question-answer pairs lets Llama 3 8B recall conversation topics with 81.5% accuracy, close to RAG at 83.5% but without retrieval.

  10. Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory

    stat.ML 2025-06 conditional novelty 5.0 of 10

    RLHF reward modeling satisfies pairwise majority and Condorcet consistency when each response pair is labeled once, because the maximum likelihood ranking then matches the Copeland rule.

  11. MOSLIM:Align with diverse preferences in prompts through reward classification

    cs.CL 2025-05 reject novelty 5.0 of 10

    A prompt-controlled multi-objective alignment method using a multi-head classification reward model and a z-score reward mapping, claimed to work with off-the-shelf models.

  12. Context Engineering: A Practitioner Methodology for Structured Human-AI Collaboration

    cs.AI 2026-04 conditional novelty 4.0 of 10

    Structured five-role context packages and a four-phase pipeline were associated with cutting average AI task iterations from 3.8 to 2.0 and raising first-pass acceptance from 32% to 55% in an observational single-oper...

  13. T-POP: Test-Time Personalization with Online Preference Feedback

    cs.LG 2025-09 conditional novelty 4.0 of 10

    T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

  14. Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

    cs.AI 2025-05 reject novelty 4.0 of 10

    Training 2D-DPO with an expected loss over uniform segment-score perturbations yields higher win rates under score noise than a clean-trained 2D-DPO baseline, though the comparison is confounded.

  15. FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF

    cs.LG 2024-12 reject novelty 4.0 of 10

    FedRLHF decentralizes RLHF across clients that locally shape rewards with private human feedback, with claimed convergence guarantees and a personalization-performance trade-off.

Pith tools