REVIEW 3 cited by
LLM Roleplay: Simulating Human-Chatbot Interaction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The development of chatbots requires collecting a large number of human-chatbot dialogues to reflect the breadth of users' sociodemographic backgrounds and conversational goals. However, the resource requirements to conduct the respective user studies can be prohibitively high and often only allow for a narrow analysis of specific dialogue goals and participant demographics. In this paper, we propose LLM Roleplay: a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. LLM Roleplay can be applied to generate dialogues with any type of chatbot and uses large language models (LLMs) to play the role of textually described personas. To validate our method, we collect natural human-chatbot dialogues from different sociodemographic groups and conduct a user study to compare these with our generated dialogues. We evaluate the capabilities of state-of-the-art LLMs in maintaining a conversation during their embodiment of a specific persona and find that our method can simulate human-chatbot dialogues with a high indistinguishability rate.
Forward citations
Cited by 3 Pith papers
-
Too Human to Model:The Uncanny Valley of LLMs in Social Simulation -- When Generative Language Agents Misalign with Modelling Principles
A position paper contends that LLM agents, despite their human-like talk, are often too rich in detail to serve as scientific models, and proposes conditions where they still excel.
-
ProfiLLM: An LLM-Based Framework for Implicit Profiling of Chatbot Users
ProfiLLM infers chatbot users' IT/cybersecurity proficiency from their prompts, achieving a rapid initial reduction in profiling error in synthetic and limited human evaluations.
-
DialogueForge: LLM Simulation of Human-Chatbot Dialogue
DialogueForge generates synthetic human-chatbot dialogues by pitting an inquirer LLM against a responder LLM, and finds that fine-tuned small models can approach GPT-4o-level realism on LLM-judged metrics.
Discussion (0). Sign in to comment.