PersonaLens uses LLM-simulated users and an LLM judge to evaluate personalization in task-oriented conversational assistants across 111 tasks and 20 domains.
Personalization in Goal-Oriented Dialog
1 Pith paper cite this work. Polarity classification is still indexing.
abstract
The main goal of modeling human conversation is to create agents which can interact with people in both open-ended and goal-oriented scenarios. End-to-end trained neural dialog systems are an important line of research for such generalized dialog models as they do not resort to any situation-specific handcrafting of rules. However, incorporating personalization into such systems is a largely unexplored topic as there are no existing corpora to facilitate such work. In this paper, we present a new dataset of goal-oriented dialogs which are influenced by speaker profiles attached to them. We analyze the shortcomings of an existing end-to-end dialog system based on Memory Networks and propose modifications to the architecture which enable personalization. We also investigate personalization in dialog as a multi-task learning problem, and show that a single model which shares features among various profiles outperforms separate models for each profile.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
PersonaLens uses LLM-simulated users and an LLM judge to evaluate personalization in task-oriented conversational assistants across 111 tasks and 20 domains.