Pith. sign in

REVIEW 17 cited by

Personalized Language Modeling from Personalized Human Feedback

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05133 v3 pith:AVH2CKNV submitted 2024-02-06 cs.CL cs.AIcs.LG

Xinyu Li , Ruiyang Zhou , Zachary C. Lipton , Liu Leqi This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords personalizedpreferencesuserhumanllmsfeedbackrlhfcontent
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment

    cs.CL 2024-12 conditional novelty 7.0 of 10

    Latent DPO trains only a small preference encoder per user, reducing LLM personalization training time by 80-90% with comparable alignment quality.

  3. UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs

    cs.LG 2026-08 conditional novelty 6.0 of 10

    UserToolBench evaluates LLMs on personalized tool-use decisions with hidden user profiles and incomplete requests, and finds the best model reaches only 49.36% exact trajectory accuracy.

  4. PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs

    cs.CL 2025-08 conditional novelty 6.0 of 10

    PREF is a reference-free, two-stage LLM judge that personalizes a quality rubric with a user profile and scores candidates against it, beating reminder-only baselines on the PrefEval implicit preference subset.

  5. Learning the Value Systems of Societies from Preferences

    cs.AI 2025-07 conditional novelty 6.0 of 10

    The paper defines a society's value system as a shared grounding plus clustered group preferences, and introduces a deep-clustering method to learn it from pairwise choice data.

  6. Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions

    cs.HC 2025-07 conditional novelty 6.0 of 10

    Surfacing variations across multiple MLLM image descriptions increases blind and low vision users' detection of unreliable claims and reduces their over-trust in a single AI description.

  7. Can Large Language Models Understand Preferences in Personalized Recommendation?

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A new grouped-ranking benchmark finds current LLMs score near chance on personalized preference ranking once user rating bias and item quality are controlled.

  8. On the Way to LLM Personalization: Learning to Remember User Conversations

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Finetuning a LoRA adapter on self-generated question-answer pairs lets Llama 3 8B recall conversation topics with 81.5% accuracy, close to RAG at 83.5% but without retrieval.

  9. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

  10. Configurable Preference Tuning with Rubric-Guided Synthetic Data

    cs.CL 2025-06 conditional novelty 5.0 of 10

    CPT fine-tunes LLMs with DPO on rubric-guided synthetic preferences so that a system prompt can reconfigure output style at inference, with in-distribution accuracy gains over baselines.

  11. Personalized Preference Fine-tuning of Diffusion Models

    cs.LG 2025-01 conditional novelty 5.0 of 10

    PPD fine-tunes a single diffusion model to follow per-user preferences by conditioning on VLM-extracted embeddings, reporting 76-81% win rates over Stable Cascade with four examples per user.

  12. Beyond the Binary: Capturing Diverse Preferences With Reward Regularization

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Reward models trained with a margin derived from synthetic LLM judgments better match aggregate human preferences than standard binary-trained reward models, mainly on subjective prompts.

  13. T-POP: Test-Time Personalization with Online Preference Feedback

    cs.LG 2025-09 conditional novelty 4.0 of 10

    T-POP uses dueling-bandit token selection to learn a reward function online from pairwise user feedback, enabling test-time personalization of a frozen LLM without fine-tuning.

  14. LoRe: Personalizing LLMs via Low-Rank Reward Modeling

    cs.LG 2025-04 conditional novelty 4.0 of 10

    LoRe learns a shared low-rank reward basis plus per-user simplex weights and reports higher preference-prediction accuracy than personalized and monolithic baselines on three datasets.

  15. From General to Specific: Tailoring Large Language Models for Personalized Healthcare

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A method constructs patient-specific hard prompts, refined only by word deletion with a BERTScore reward, to make proprietary LLMs output more personalized medical responses.

  16. FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF

    cs.LG 2024-12 reject novelty 4.0 of 10

    FedRLHF decentralizes RLHF across clients that locally shape rewards with private human feedback, with claimed convergence guarantees and a personalization-performance trade-off.

  17. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools