REVIEW 5 cited by
User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
One of the major impediments to the development of new task-oriented dialogue (TOD) systems is the need for human evaluation at multiple stages and iterations of the development process. In an effort to move toward automated evaluation of TOD, we propose a novel user simulator built using recently developed large pretrained language models (LLMs). In order to increase the linguistic diversity of our system relative to the related previous work, we do not fine-tune the LLMs used by our system on existing TOD datasets; rather we use in-context learning to prompt the LLMs to generate robust and linguistically diverse output with the goal of simulating the behavior of human interlocutors. Unlike previous work, which sought to maximize goal success rate (GSR) as the primary metric of simulator performance, our goal is a system which achieves a GSR similar to that observed in human interactions with TOD systems. Using this approach, our current simulator is effectively able to interact with several TOD systems, especially on single-intent conversational goals, while generating lexically and syntactically diverse output relative to previous simulators that rely upon fine-tuned models. Finally, we collect a Human2Bot dataset of humans interacting with the same TOD systems with which we experimented in order to better quantify these achievements.
Forward citations
Cited by 5 Pith papers
-
CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
SalesSim benchmarks MLLMs as retail user simulators, finds gaps in persona adherence and over-persuasion, and introduces UserGRPO RL to raise decision alignment by 13.8%.
-
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
On τ-bench, LLM user simulators are more cooperative, more verbose, and more lenient than real human users, so agent benchmarks that rely on them overstate real-world performance.
-
DiPS: Dialogue Policy Selection for High-Stakes Persuasion Agents
DiPS uses Implicit Q-Learning over dialogue history embeddings to select persuasion policies turn-by-turn, raising evacuation success above zero-shot LLM and RAG baselines in simulation and human studies.
-
Large-Scale ChatBot Validation Through Customer Digital Twin Simulations
Synthetic customer agents built from real bank data can mimic customer semantics and personality well enough to serve as scalable chatbot validation proxies.
-
ChatChecker: A Framework for Dialogue System Testing and Evaluation Through Non-cooperative User Simulation
A framework that combines LLM-based user personas, breakdown detection, and dialogue rating to test chatbots, with a non-cooperative simulator that triggers more breakdowns than cooperative baselines.
Discussion (0). Continue with ORCID to comment.