REVIEW 13 cited by
Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have demonstrated exceptional capabilities in generalizing to new tasks in a zero-shot or few-shot manner. However, the extent to which LLMs can comprehend user preferences based on their previous behavior remains an emerging and still unclear research question. Traditionally, Collaborative Filtering (CF) has been the most effective method for these tasks, predominantly relying on the extensive volume of rating data. In contrast, LLMs typically demand considerably less data while maintaining an exhaustive world knowledge about each item, such as movies or products. In this paper, we conduct a thorough examination of both CF and LLMs within the classic task of user rating prediction, which involves predicting a user's rating for a candidate item based on their past ratings. We investigate various LLMs in different sizes, ranging from 250M to 540B parameters and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct comprehensive analysis to compare between LLMs and strong CF methods, and find that zero-shot LLMs lag behind traditional recommender models that have the access to user interaction data, indicating the importance of user interaction data. However, through fine-tuning, LLMs achieve comparable or even better performance with only a small fraction of the training data, demonstrating their potential through data efficiency.
Forward citations
Cited by 13 Pith papers
-
GrocLM: Grocery Category Recommendation in E-Commerce with Large Language Models
A LoRA-tuned LLM with trie-constrained decoding improves grocery category recommendation and yields a 7.5% cart-add lift in production.
-
VENOMREC: Cross-Modal Interactive Poisoning for Targeted Promotion in Multimodal LLM Recommender Systems
Synchronized text+image poisoning steers multimodal LLM recommenders to promote target items, reaching 0.73 mean exposure@20.
-
Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation
ContextSim uses life-simulation and explicit thought-training to make LLM user proxies more realistic, and claims that recommender systems tuned with these proxies improve real-world engagement.
-
KERAG_R: Knowledge-Enhanced Retrieval-Augmented Generation for Recommendation
KERAG_R improves LLM-based top-k recommendation by using a GAT to select relevant KG triples and incorporating them into instruction-tuned prompts, reporting gains over ten baselines on three datasets.
-
PageLLM: A Multi-Grained Reward Framework for Whole-Page Optimization with Large Language Models
A multi-grained reward framework fine-tunes an LLM with PPO to generate whole-page recommendations, showing that page-level and item-level reward heads are complementary.
-
GORACS: Group-level Optimal Transport-guided Coreset Selection for LLM-based Recommender Systems
GORACS selects small groups of fine-tuning examples via an optimal-transport and gradient-norm proxy objective, outperforming prior coreset methods for LLM-based recommendation.
-
Augment or Not? A Comparative Study of Pure and Augmented Large Language Model Recommenders
A survey and benchmark of LLM recommenders finds that augmenting LLMs with non-LLM techniques (semantic IDs, collaborative signals) generally improves sequential recommendation accuracy on Amazon'23.
-
Fair Document Valuation in LLM Summaries via Shapley Values
Cluster Shapley groups semantically similar documents via embeddings and computes cluster-level Shapley values, claiming better efficiency-accuracy trade-offs than Monte Carlo and Kernel SHAP on Amazon review summarization.
-
Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signals
A 7B model trained with synthetic reasoning demonstrations plus reinforcement learning infers explicit user preference descriptions from behavioral signals, improving personalized response judging and generation.
-
Personas within Parameters: Fine-Tuning Small Language Models with Low-Rank Adapters to Mimic User Behaviors
Persona-level LoRA fine-tuning lets a 3.8B small language model simulate MovieLens users about as accurately as a much larger frozen LLM, at lower cost.
-
CLAImate: AI-Enabled Climate Change Communication through Personalized and Localized Narrative Visualizations
A personalized, localized AI conversation system for climate communication shows modest factual accuracy and positive early feedback from 10 UK users.
-
Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing
LLM tutoring modestly accelerates RL convergence on average, with advice reuse saving wall-clock time but reducing stability.
-
Improving the Performance of Sequential Recommendation Systems with an Extended Large Language Model
Replacing Llama2 with Llama3.1 in the LlamaRec framework improved recommendation metrics by 8 to 39 percent on three public datasets.
Discussion (0). Sign in to comment.