REVIEW 5 cited by
Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for conversational recommendation, revealing the inadequacy of the existing evaluation protocol. It might over-emphasize the matching with the ground-truth items or utterances generated by human annotators, while neglecting the interactive nature of being a capable CRS. To overcome the limitation, we further propose an interactive Evaluation approach based on LLMs named iEvaLM that harnesses LLM-based user simulators. Our evaluation approach can simulate various interaction scenarios between users and systems. Through the experiments on two publicly available CRS datasets, we demonstrate notable improvements compared to the prevailing evaluation protocol. Furthermore, we emphasize the evaluation of explainability, and ChatGPT showcases persuasive explanation generation for its recommendations. Our study contributes to a deeper comprehension of the untapped potential of LLMs for CRSs and provides a more flexible and easy-to-use evaluation framework for future research endeavors. The codes and data are publicly available at https://github.com/RUCAIBox/iEvaLM-CRS.
Forward citations
Cited by 5 Pith papers
-
User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendation
SMTPO uses multi-task SFT to improve simulator feedback quality and RL with fine-grained rewards to optimize multi-turn preference reasoning in LLM-based conversational recommendation.
-
Evaluating Recabilities of Foundation Models: A Multi-Domain, Multi-Dataset Benchmark
A large-scale benchmark finds that in-domain fine-tuning works best for foundation model recommenders, while cross-dataset and multi-domain training help in new scenarios.
-
VRAgent-R1: Boosting Video Recommendation with MLLM-based Agents via Reinforcement Learning
VRAgent-R1 uses an MLLM agent to summarize videos and a reinforcement-learned agent to simulate user choices, improving video recommendation and user-decision simulation on MicroLens-100K.
-
Towards Fast Domain Adaptation and Fine-Grained User Simulation for Evaluating Conversational Recommender Systems
AdaptSim is an adaptive user simulator for CRS evaluation that combines automatic prompt generation, open actions, controlled text generation, and BFS-based pairwise comparison to produce realistic dialogues and asses...
-
A Language-Driven Framework for Improving Personalized Recommendations: Merging LLMs with Traditional Algorithms
An LLM-based re-ranking layer over SVD/SVD++ improves offline rating and ranking metrics on MovieLens-Latest-Small, but evaluation and baseline gaps limit the strength of the claim.
Discussion (0). Sign in to comment.