Pith. sign in

REVIEW 8 cited by

Personalized Language Modeling from Personalized Human Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05133 v3 pith:AVH2CKNV submitted 2024-02-06 cs.CL cs.AIcs.LG

Xinyu Li , Ruiyang Zhou , Zachary C. Lipton , Liu Leqi This is my paper
classification cs.CLcs.AIcs.LG
keywords personalizedpreferencesuserhumanllmsfeedbackrlhfcontent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs

    cs.CL 2025-08 conditional novelty 6.0 of 10

    PREF is a reference-free, two-stage LLM judge that personalizes a quality rubric with a user profile and scores candidates against it, beating reminder-only baselines on the PrefEval implicit preference subset.

  3. Learning the Value Systems of Societies from Preferences

    cs.AI 2025-07 conditional novelty 6.0 of 10

    The paper defines a society's value system as a shared grounding plus clustered group preferences, and introduces a deep-clustering method to learn it from pairwise choice data.

  4. Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions

    cs.HC 2025-07 conditional novelty 6.0 of 10

    Surfacing variations across multiple MLLM image descriptions increases blind and low vision users' detection of unreliable claims and reduces their over-trust in a single AI description.

  5. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

  6. Configurable Preference Tuning with Rubric-Guided Synthetic Data

    cs.CL 2025-06 conditional novelty 5.0 of 10

    CPT fine-tunes LLMs with DPO on rubric-guided synthetic preferences so that a system prompt can reconfigure output style at inference, with in-distribution accuracy gains over baselines.

  7. T-POP: Test-Time Personalization with Online Preference Feedback

    cs.LG 2025-09 conditional novelty 4.0 of 10

    T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

  8. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools