pith. sign in

arxiv: 2605.20204 · v1 · pith:SVJR56AFnew · submitted 2026-04-07 · 💻 cs.HC · cs.AI

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

Ming Zhu , Juntao Tan , Rithesh Murthy , Jielin Qiu , Liangwei Yang , Wenting Zhao , Silvio Savarese , Shelby Heinecke
show 1 more author
Huan Wang
This is my paper

Pith reviewed 2026-05-21 10:17 UTC · model grok-4.3

classification 💻 cs.HC cs.AI
keywords user simulationagent benchmarkingbehavioral profilessimulation fidelityreality gapLLM evaluationconversation data
0
0 comments X

The pith

Grounded profiles from real conversations raise user simulator fidelity from 24% to 45% and expose hidden agent failures.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current LLM user simulators struggle with either overly formal outputs or exaggerated behaviors triggered by instructions, making them weak stand-ins for actual people when testing agents. The work extracts thousands of executable profiles directly from large sets of authentic human conversations and uses those to condition the simulators instead. This produces substantially closer matches to observed human behavior across multiple dimensions in a controlled benchmark spanning dozens of domains. The same grounded approach then functions as a stricter test that uncovers specific ways agents break down, patterns that remain hidden when simulators cooperate too readily.

Core claim

Extracting 7,275 executable behavioral profiles from more than 14,000 real human-LLM conversations supplies concrete grounding for LLM simulators; when these profiles replace unconstrained generation or hand-crafted directives, behavioral match rates on a 600-conversation fidelity benchmark across 71 domains rise from 24.2 percent to 45.3 percent, and the resulting simulations reveal three distinct agent failure mechanisms that produce consistent 3.2 to 3.5 percent drops in task success on TauBench evaluations.

What carries the argument

Executable behavioral profiles extracted from the WildChat dataset that replace unconstrained defaults or hand-crafted directives when conditioning LLM user simulators.

If this is right

  • Grounded simulation supplies a more valid stress test that lowers measured agent performance relative to cooperative simulators.
  • Existing agent benchmarks that rely on directive-driven simulators may overestimate real-world effectiveness.
  • Three concrete failure modes in agent behavior become detectable only when user simulation matches observed human patterns.
  • Replacing hand-crafted directives with data-derived profiles reduces unwanted variation across different base simulator models.
  • Higher-fidelity user simulation supports more reliable end-to-end evaluation of agent systems before deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The profile extraction method could be reused to create reusable test suites for agent categories beyond the current benchmark.
  • If the profiles prove representative, the same grounding technique might improve simulation in other human-AI interaction settings.
  • Periodic refresh of the profile set with newer conversation data would keep the simulators aligned with evolving user habits.
  • Evaluation protocols may need to shift emphasis from cooperative test cases toward realistic stress conditions to maintain validity.

Load-bearing premise

The 7,275 behavioral profiles pulled from the WildChat dataset give a representative and unbiased picture of how real users actually behave across many different tasks and domains.

What would settle it

Run the identical TauBench tasks with live human users instead of the simulators and check whether the measured task-success degradation and the three specific failure mechanisms appear at the same rates.

Figures

Figures reproduced from arXiv: 2605.20204 by Huan Wang, Jielin Qiu, Juntao Tan, Liangwei Yang, Ming Zhu, Rithesh Murthy, Shelby Heinecke, Silvio Savarese, Wenting Zhao.

Figure 1
Figure 1. Figure 1: Linguistic trait clusters across 7,273 users. (a) t-SNE projection of 48-dimensional [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Two key effects on agent task success rate across 6 simulator models. (a) Persona [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Demographic distributions across 7,273 WildChat users. The population skews [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Profile statistics across 7,273 users. Left: most profiles contain 8–10 commands [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Conversation domain distribution (top 25 of 1,012 domains). Entertainment & [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Persona sampling variance across three independent random runs (v7, v8, v9) with [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Demographic-based persona sampling. High-Edu and Oldest personas generally [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Perfect User vs. other conditions. The Perfect User consistently outperforms [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
read the original abstract

LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users), while hand-crafted behavioral directives trigger Directive Amplification, where models hyper-interpret instructions into unnatural behavioral extremes that vary dramatically across simulator models. We present RealUserSim, the first user simulation framework grounded in real behavioral data. From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles and use them to ground LLM simulators. A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3% across five behavioral dimensions. Agent evaluation on TauBench with 6 simulator models and extensive analysis shows that grounded simulation acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation), while Directive Amplification in existing benchmarks produces unrealistic behavior that compromises the validity of agent evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces RealUserSim, a user simulation framework that extracts 7,275 executable behavioral profiles from the WildChat dataset of 14,000+ human-LLM conversations to ground LLM simulators. It proposes a fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls, reporting an increase in behavioral match rate from 24.2% to 45.3% across five dimensions, and evaluates the approach on TauBench with six simulator models, claiming it surfaces three failure mechanisms (with mean task-success degradation of -3.2% to -3.5%) that cooperative simulators miss while avoiding issues like Formalism Ceiling and Directive Amplification.

Significance. If the central claims hold, this work offers a concrete path toward more valid end-to-end agent benchmarking by replacing unconstrained or hand-crafted simulators with data-grounded ones. The use of external datasets (WildChat, TauBench) without self-referential fitting, combined with quantitative fidelity metrics and degradation analysis, strengthens the empirical case; the anti-leakage controls in PT3 are a positive design choice that could raise standards for simulation realism if the profile extraction proves representative.

major comments (2)
  1. [§3 and §4.1] §3 (Behavioral Profile Extraction) and §4.1 (PT3 Benchmark): The headline fidelity improvement (24.2% → 45.3%) and the TauBench stress-test results both depend on the claim that the 7,275 profiles extracted from WildChat faithfully represent task-oriented, multi-turn, goal-directed user behaviors across the 71+ domains. The manuscript provides no validation (e.g., comparison of interaction-style distributions or domain coverage statistics) showing that open-ended WildChat chats preserve the relevant behavioral statistics for the agents under test; if the extraction procedure systematically under-samples certain interaction patterns, both the match-rate gain and the reported failure mechanisms could be artifacts of the source distribution rather than genuine realism.
  2. [TauBench Evaluation] TauBench Evaluation section: The mean task-success degradation of -3.2% to -3.5% is presented as evidence that grounded simulation acts as a realistic stress test revealing three invisible failure mechanisms. However, the section does not report per-model breakdowns, confidence intervals, or statistical tests for the degradation, nor does it demonstrate that the three mechanisms are absent under cooperative simulators after controlling for prompt length or instruction strength; without these, it is difficult to rule out that the degradation simply reflects higher variance or different prompt sensitivity rather than improved ecological validity.
minor comments (2)
  1. [Abstract] The abstract states that PT3 uses 'anti-leakage controls' but does not specify their exact implementation (e.g., train/test split on profiles or domain hold-out); a one-sentence clarification would improve reproducibility.
  2. [PT3 Benchmark] Table or figure presenting the five behavioral dimensions used for match-rate calculation should include inter-annotator agreement or definition of each dimension to allow readers to assess the 45.3% figure.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We are grateful to the referee for the constructive feedback on our manuscript. We have carefully considered the major comments and provide point-by-point responses below. Where the comments identify areas for improvement, we have revised the manuscript accordingly to enhance clarity and rigor.

read point-by-point responses

  1. Referee: [§3 and §4.1] §3 (Behavioral Profile Extraction) and §4.1 (PT3 Benchmark): The headline fidelity improvement (24.2% → 45.3%) and the TauBench stress-test results both depend on the claim that the 7,275 profiles extracted from WildChat faithfully represent task-oriented, multi-turn, goal-directed user behaviors across the 71+ domains. The manuscript provides no validation (e.g., comparison of interaction-style distributions or domain coverage statistics) showing that open-ended WildChat chats preserve the relevant behavioral statistics for the agents under test; if the extraction procedure systematically under-samples certain interaction patterns, both the match-rate gain and the reported failure mechanisms could be artifacts of the source distribution rather than genuine realism.

    Authors: We agree that explicit validation of the source distribution is important to support the generalizability of our findings. In the revised manuscript, we have added a new analysis in Section 3 that compares key interaction-style metrics (such as average turns per conversation, goal specificity, and multi-turn persistence) between the WildChat-derived profiles and a reference set of task-oriented dialogues. We also include domain coverage statistics demonstrating that the profiles span the 71+ domains in PT3 with high overlap. These additions address the concern and reduce the likelihood that the results are artifacts of the source data. revision: yes

  2. Referee: [TauBench Evaluation] TauBench Evaluation section: The mean task-success degradation of -3.2% to -3.5% is presented as evidence that grounded simulation acts as a realistic stress test revealing three invisible failure mechanisms. However, the section does not report per-model breakdowns, confidence intervals, or statistical tests for the degradation, nor does it demonstrate that the three mechanisms are absent under cooperative simulators after controlling for prompt length or instruction strength; without these, it is difficult to rule out that the degradation simply reflects higher variance or different prompt sensitivity rather than improved ecological validity.

    Authors: We acknowledge the need for more detailed statistical reporting. The revised version now includes per-model task success rates with standard deviations and 95% confidence intervals in an updated table. We have also performed and reported paired statistical tests confirming the significance of the degradation. Furthermore, we added a controlled experiment matching prompt lengths and instruction strengths across simulator types, showing that the three failure mechanisms remain absent in cooperative simulators under these controls. This supports our interpretation of improved ecological validity. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical results rest on external datasets

full rationale

The paper extracts 7,275 behavioral profiles from the external WildChat dataset of 14,000+ human-LLM conversations and evaluates fidelity via the PT3 benchmark on 600 conversations plus agent performance on TauBench. Reported gains (24.2% to 45.3% match rate) and task-success degradations (-3.2% to -3.5%) are direct empirical comparisons against these independent sources. No equations, fitted parameters renamed as predictions, self-definitional steps, or load-bearing self-citations appear in the abstract or described chain. The central claims remain data-driven and externally benchmarked rather than reducing to internal definitions or prior author results by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the representativeness of WildChat conversations as authentic behavioral data and the validity of PT3 as a fidelity measure; no explicit free parameters, new axioms beyond standard assumptions, or invented entities are described in the abstract.

axioms (1)
  • domain assumption WildChat contains authentic human-LLM conversations that can be processed into representative executable behavioral profiles.
    Invoked when extracting the 7,275 profiles to ground simulators.

pith-pipeline@v0.9.0 · 5772 in / 1372 out tokens · 50216 ms · 2026-05-21T10:17:06.231654+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages · 2 internal anchors

  1. [1]

    Attention is All you Need , volume =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser,. Attention is All you Need , volume =. Advances in Neural Information Processing Systems , pages =

    work page

  2. [2]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    International Conference on Learning Representations , year=

    WildChat: 1M ChatGPT Interaction Logs in the Wild , author=. International Conference on Learning Representations , year=

    work page

  4. [4]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , pages=

    Personalizing Dialogue Agents: I have a dog, do you have pets too? , author=. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , pages=

    work page

  5. [5]

    The Knowledge Engineering Review , volume=

    A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , author=. The Knowledge Engineering Review , volume=

    work page

  6. [6]

    Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue , pages=

    Neural User Simulation for Corpus-based Policy Optimisation of Spoken Dialogue Systems , author=. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue , pages=

    work page

  7. [7]

    Nature , volume=

    Role-Play with Large Language Models , author=. Nature , volume=

    work page

  8. [8]

    Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , year=

    Generative Agents: Interactive Simulacra of Human Behavior , author=. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , year=

    work page

  9. [9]

    8 Jason Chuang, Margaret E

    From Persona to Personalization: A Survey on Role-Playing Language Agents , author=. arXiv preprint arXiv:2404.18231 , year=

    work page arXiv

  10. [10]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    work page 2024

  11. [11]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Character-LLM: A Trainable Agent for Role-Playing , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2023

  12. [12]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  13. [13]

    Du, Chengyu and Wang, Xintao and Chen, Aili and Li, Weiyuan and Xu, Rui and Liu, Junteng and Huang, Zishan and Tian, Rong and Sun, Zijun and others , journal=

    work page

  14. [14]

    Advances in Neural Information Processing Systems , year=

    Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning , author=. Advances in Neural Information Processing Systems , year=

    work page

  15. [15]

    arXiv preprint arXiv:2511.07338 , year=

    DeepPersona: A Generative Engine for Scaling Deep Synthetic Personas , author=. arXiv preprint arXiv:2511.07338 , year=

    work page arXiv

  16. [16]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Implicit Behavioral Alignment of Language Agents in High-Stakes Crowd Simulations , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2025

  17. [17]

    PersonaGym: Evaluating Persona Agents and

    Samuel, Vinay and Zou, Henry Peng and Zhou, Yue and Chaudhari, Shreyas and Kalyan, Ashwin and Rajpurohit, Tanmay and Deshpande, Ameet and Narasimhan, Karthik R and Murahari, Vishvak , booktitle=. PersonaGym: Evaluating Persona Agents and

    work page

  18. [18]

    Shea, Ryan and Lu, Yunan and Qiu, Liang and Yu, Zhou , booktitle=

    work page

  19. [19]

    Ren, Ruiyang and Qiu, Peng and Qu, Yingqi and Liu, Jing and Zhao, Wayne Xin and Wu, Hua and Wen, Ji-Rong and Wang, Haifeng , booktitle=

    work page

  20. [20]

    Bougie, Nicolas and Watanabe, Narimasa , booktitle=

    work page

  21. [21]

    Kim, Minseo and Im, Sujeong and Choi, Junseong and Lee, Junhee and Shim, Chaeeun and Choi, Edward , journal=

    work page

  22. [22]

    $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment

    ^2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment , author=. arXiv preprint arXiv:2506.07982 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2025

  24. [24]

    Zhou, Yifei and Jiang, Song and Tian, Yuandong and Weston, Jason and Levine, Sergey and Sukhbaatar, Sainbayar and Li, Xian , journal=

    work page

  25. [25]

    Bian, Haonan and Yao, Zhiyuan and Hu, Sen and Xu, Zishan and Zhang, Shaolei and Guo, Yifu and Yang, Ziliang and Han, Xueran and Wang, Huacan and Chen, Ronghao , journal=

    work page

  26. [26]

    Jiayang, Cheng and Ru, Dongyu and Qiu, Lin and Li, Yiyang and Cao, Xuezhi and Song, Yangqiu and Cai, Xunliang , booktitle=

    work page

  27. [27]

    , journal=

    Li, Hanyu and Liu, Haoyu and Zhu, Tingyu and Guo, Tianyu and Zheng, Zeyu and Deng, Xiaotie and Jordan, Michael I. , journal=

    work page

  28. [28]

    Deng, Mingyi and Huang, Lijun and Fan, Yani and Zhang, Jiayi and Ren, Fashen and Bai, Jinyi and Yang, Fuzhen and others , journal=

    work page

  29. [29]

    Jiang, Tanqiu and Wang, Yuhui and Liang, Jiacheng and Wang, Ting , journal=

    work page

  30. [30]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Effective Red-Teaming of Policy-Adherent Agents , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2025

  31. [31]

    Shang, Yu and Liu, Peijie and Yan, Yuwei and Wu, Zijing and Sheng, Leheng and Yu, Yuanqing and Jiang, Chumeng and Zhang, An and Xu, Fengli and Wang, Yu and Zhang, Min and Li, Yong , journal=

    work page

  32. [32]

    Ye, Jingheng and Jiang, Yong and Wang, Xiaobin and Li, Yinghui and Li, Yangning and Zheng, Hai-Tao and Xie, Pengjun and Huang, Fei , journal=

    work page

  33. [33]

    Lost in Simulation:

    Seshadri, Preethi and Cahyawijaya, Samuel and Odumakinde, Ayomide and Singh, Sameer and Goldfarb-Tarrant, Seraphina , journal=. Lost in Simulation:

    work page

  34. [34]

    Mind the

    Zhou, Xuhui and Sun, Weiwei and Ma, Qianou and Xie, Yiqing and Liu, Jiarui and Du, Weihua and Welleck, Sean and Yang, Yiming and Neubig, Graham and Wu, Sherry Tongshuang and Sap, Maarten , journal=. Mind the

    work page