LLM user simulators exhibit a disengagement deficit: they match real buyers but systematically overstate purchase intent among real non-buyers by reducing expressed resistance and increasing deliberation.
Mind the sim2real gap in user simulation for agentic tasks, 2026 b
12 Pith papers cite this work. Polarity classification is still indexing.
12
Pith papers citing it
citation-role summary
background 2
citation-polarity summary
years
2026 12roles
background 2polarities
background 2representative citing papers
User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.
BehaviorBench reconstructs 2,000 real wallets into 141k belief and 1.4M trade prediction tasks to test if personalization from history improves model performance over non-personalized baselines.
EVA-Bench supplies a simulation engine for bot-to-bot voice dialogues plus two composite metrics (EVA-A for accuracy, EVA-X for experience) evaluated on 213 enterprise scenarios, showing no tested system exceeds 0.5 on both pass@1 scores.
Real developer IDE traces differ substantially from LLM simulations in behavior and structure; current proactive assistants are unreliable on real traces, and simulated data cannot substitute for real data in training.
Using 85 controlled and 35 public LLMs, the authors show social-simulation accuracy generally improves with compute, but some behavioral and low-resource tasks do not scale.
Language models show superior memory to humans on psych experiments but can be adjusted via prompting and compaction to forget more human-like, yielding better user simulators.
DITTO uses RL with verbal feedback to train LLMs for human behavior simulation, reporting 36% average gains over base models and outperforming GPT-5.4 on 6 of 10 SOUL benchmark tasks.
Execution-grounded multi-turn OS trajectories from 4D intents and a role-locked simulator lift Qwen3-8B ClawEval pass@1 from 19.3 to 37.7, beating GPT-4o and Qwen3-32B zero-shot.
Grounding an LLM in verbalized transaction histories via Person–Environment prompting, continued pre-training, SFT, and GRPO yields stronger retail decision simulation than frontier models, with partial cross-domain transfer.
Fine-tuned simulators grounded in real human data produce LLM assistants that win more often against real users than those trained against role-playing simulators.
Evaluates multimodal foundation models as agents for power distribution defect detection across perception, reasoning, and tool usage using a custom benchmark.
citing papers explorer
-
Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes
LLM user simulators exhibit a disengagement deficit: they match real buyers but systematically overstate purchase intent among real non-buyers by reducing expressed resistance and increasing deliberation.
-
Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models
User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.
-
BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces
BehaviorBench reconstructs 2,000 real wallets into 141k belief and 1.4M trade prediction tasks to test if personalization from history improves model performance over non-personalized baselines.
-
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
EVA-Bench supplies a simulation engine for bot-to-bot voice dialogues plus two composite metrics (EVA-A for accuracy, EVA-X for experience) evaluated on 213 enterprise scenarios, showing no tested system exceeds 0.5 on both pass@1 scores.
-
An Empirical Study of Proactive Coding Assistants in Real-World Software Development
Real developer IDE traces differ substantially from LLM simulations in behavior and structure; current proactive assistants are unreliable on real traces, and simulated data cannot substitute for real data in training.
-
Will Scaling Improve Social Simulation with LLMs?
Using 85 controlled and 35 public LLMs, the authors show social-simulation accuracy generally improves with compute, but some behavioral and low-resource tasks do not scale.
-
Simulating Human Memory with Language Models
Language models show superior memory to humans on psych experiments but can be adjusted via prompting and compaction to forget more human-like, yielding better user simulators.
-
Reinforcing Human Behavior Simulation via Verbal Feedback
DITTO uses RL with verbal feedback to train LLMs for human behavior simulation, reporting 36% average gains over base models and outperforming GPT-5.4 on 6 of 10 SOUL benchmark tasks.
-
ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories
Execution-grounded multi-turn OS trajectories from 4D intents and a role-locked simulator lift Qwen3-8B ClawEval pass@1 from 19.3 to 37.7, beating GPT-4o and Qwen3-32B zero-shot.
-
Large Behavior Model: A Promptable Digital Twin of the Retail Customer
Grounding an LLM in verbalized transaction histories via Person–Environment prompting, continued pre-training, SFT, and GRPO yields stronger retail decision simulation than frontier models, with partial cross-domain transfer.
-
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
Fine-tuned simulators grounded in real human data produce LLM assistants that win more often against real users than those trained against role-playing simulators.
-
Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models
Evaluates multimodal foundation models as agents for power distribution defect detection across perception, reasoning, and tool usage using a custom benchmark.