Pith. sign in

REVIEW 4 major objections 7 minor 17 cited by

This paper establishes that LLM-based user simulators create an 'easy mode' for agents, inflating measured success rates above human baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:47 UTC pith:TFZXCHBF

load-bearing objection Careful human study with a convincing central result—simulated users are too easy on agents and LLM judges too lenient—though the LLM-filtered human baseline deserves explicit robustness analysis. the 4 major comments →

arxiv 2603.11245 v2 pith:TFZXCHBF submitted 2026-03-11 cs.AI

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou , Weiwei Sun , Qianou Ma , Yiqing Xie , Jiarui Liu , Weihua Du , Sean Welleck , Yiming Yang
show 3 more authors
Graham Neubig Sherry Tongshuang Wu Maarten Sap
This is my paper
classification cs.AI
keywords Sim2Real gapuser simulationLLM-as-judgeagent evaluationUser-Sim Indextau-benchinteractive benchmarkshuman validation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the large language models widely used to simulate users in interactive agent benchmarks do not behave like real people, and that this gap systematically flatters the agents being tested. Running the full customer-service protocol of the τ-bench benchmark with 451 human participants and 31 LLM simulators, the authors find that most general-purpose simulators yield higher agent success rates than the 63.6% human baseline, with the best simulator scoring only 76.0 on their new User-Sim Index versus 92.9 for humans. The paper also finds that LLM judges inflate interaction quality ratings while rule-based rewards are nearly orthogonal to what humans actually perceive as success or quality. If right, benchmarks built on unvalidated simulators overestimate agent competence and miss the ambiguity, impatience, and frustration that make real interactions difficult.

Core claim

The central discovery is that LLM-based user simulators are not faithful proxies for human users: they are more cooperative, more verbose, stylistically uniform, and they front-load complete information; they rarely express genuine uncertainty or push back when the agent errs. Concretely, humans succeed against the evaluated agent 63.6% of the time, while the majority of general-purpose simulators produce higher success rates, some reaching 77.8%, and the best simulator's composite alignment score is 76.0 against a human self-agreement of 92.9. The paper formalizes this as a 'Sim2Real gap' and introduces a taxonomy plus the User-Sim Index (USI) to quantify behavioral and evaluative fidelity.

What carries the argument

The User-Sim Index (USI), a composite 0–100 score, aggregates four behavioral dimensions (communication style, information pattern, clarification behavior, error reaction) using Sørensen–Dice coefficients between simulator and human feature distributions, an outcome-calibration term based on Expected Calibration Error, and an evaluative-alignment term based on mean absolute error between simulated and human post-interaction survey ratings. The key experimental move is role-swap: humans and each LLM simulator interact with the same fixed agent on the same tasks, making the user simulator the only variable.

Load-bearing premise

The human baseline is treated as ground truth, but it comes from crowd workers role-playing with the task instructions visible, and their traces were filtered by an LLM judge that rejected some valid traces; if that filtered role-play is not representative of real users, the entire gap measurement is against the wrong reference.

What would settle it

Collect a human baseline in a setting where users do not see the task instruction, or draw from real customer-service logs, then recompute USI and the success-rate inflation. If the gap between LLM simulators and that baseline shrinks to near zero, the 'easy mode' claim fails; if it grows, the claim is strengthened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarks that rely on unvalidated LLM simulators likely overestimate agent competence; agent success rates should be read against human baselines.
  • LLM-as-judge feedback in interactive settings is asymmetrically lenient on experience dimensions, so it should not be used as the sole quality signal.
  • Rule-based rewards that check exact database states miss human-perceived success and failure; they need human-calibrated alternatives.
  • General capability does not reliably translate to simulation fidelity; the paper finds no consistent monotonic relationship between model quality and USI except within one model family.
  • The USI metric offers a quantitative validation step for future user simulators before they are deployed in agent benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same taxonomy could be applied to other interactive benchmarks beyond customer service; if it were, similar 'easy mode' inflation might appear in coding, medical, or social-interaction agent evaluations.
  • The human baseline itself was gathered through role-playing with task instructions visible; a hidden-behavior baseline or field deployment could plausibly show an even larger gap.
  • Simulator behavior likely affects not just evaluation but agent training: agents optimized against overly cooperative simulated users may become brittle when deployed with real people.
  • A direct test of the practical cost would be to train an agent on simulated users and then measure its success with real users; the drop would quantify the Sim2Real gap in deployment terms.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper formalizes the notion of a Sim2Real gap for LLM-based user simulators in interactive agent evaluation, decomposes it into four behavioral dimensions (D1–D4) and two evaluative dimensions (outcome calibration and evaluative alignment), and introduces the User-Sim Index (USI) as a composite 0–100 score. The authors instantiate this framework on τ-bench, collecting 451 human participants across 165 tasks in three independent annotation batches and benchmarking 31 LLM simulators (proprietary, open-source, and specialized). Their headline findings are that LLM simulators behave differently from humans along all four behavioral dimensions; that LLM evaluators are lenient on interaction quality but conservative on task success; that τ-bench's rule-based reward is largely orthogonal to human-perceived quality; and that most general-purpose simulators inflate agent success rates above a 63.6% human baseline, creating an 'easy mode' for evaluated agents. The paper also reports that the best simulator reaches USI 76.0 versus 92.9 for humans, and that higher general model capability does not consistently translate into more faithful user simulation.

Significance. If the results hold, this is a substantial and timely contribution. It is, to my knowledge, the first study to run the full τ-bench protocol with real humans at this scale, measuring both behavioral fidelity and evaluative reliability against a common reference. The strengths are notable: the human study is large and carefully designed with three independent annotation batches; the taxonomy is grounded in established pragmatics and communication theory; the comparison spans 31 simulators across three model families; concrete interaction examples make the behavioral gaps vivid; and the appendix provides a robustness check across an alternative agent. The central warning—that unvalidated LLM simulators can create artificially easy evaluation environments and that LLM judges and rule-based rewards misrepresent human experience—is practically important for the growing number of agent benchmarks that rely on simulated users. The paper is a useful corrective to the default assumption that LLM simulators are faithful human proxies, and its main qualitative claims are likely to influence evaluation practice.

major comments (4)
  1. [Appendix A.1 / Table 2] The human reference baseline is load-bearing and is filtered by an LLM judge. The paper states 'we only keep the pass traces' after calibrating GPT-5 on N=51 author-labeled interactions, with Cohen's κ=0.6 and 6/40 human-valid traces rejected (FN rate 15%). All human-side behavioral metrics, the 63.6% success baseline in Figure 8, ECE, and USI in Table 1 are computed on this filtered set. If the rejected valid traces were disproportionately terse, frustrated, ambiguous, or task-failing—precisely the behaviors that distinguish D4 and that motivate the 'easy mode' claim—the reference distribution shifts and the magnitude, and possibly the direction, of the simulator-over-human gap changes. Please report the raw number of interactions collected per batch, the rejection rate, and the task-success rates and D1–D4 metric values for rejected versus kept traces. At minimum, provide a sensitivity
  2. [§3.2, Eq. (2)] The expected calibration error formula is central to USI, but the 'difficulty binning' is not specified: the number of bins, the binning procedure (equal frequency, equal width, or task-specific difficulty estimates), and whether bins are constructed from human success rates, simulator success rates, or a fixed task property are all absent. This makes ECE and the resulting USI values non-reproducible as reported. Please define the binning scheme precisely and report sensitivity of Table 1 and Figure 8 to reasonable choices (e.g., 3–10 bins, equal-frequency vs. equal-width).
  3. [§3.2, Eq. (3) / Table 1] USI uses unweighted equal weights for D1–D4, ECE, and Eval, but no justification or sensitivity analysis is given. The headline claim that the best simulator reaches USI 76.0 versus 92.9 for humans, and the model rankings in Table 1, depend on this particular aggregation. Because the six dimensions are highly heterogeneous in scale and variance, equal weighting is an arbitrary modeling choice. Please provide a sensitivity analysis (e.g., perturb weights, drop one dimension, or report component-wise dominance) to establish that the USI rankings and the overall gap magnitude are not artifacts of the chosen weights. This is especially important because the paper introduces USI as a general-purpose index.
  4. [Appendix A.1 / §5] The human participants are Prolific workers role-playing with the full task instruction and role-playing guidelines visible in a side panel. The paper calls this a 'real human' baseline, but it is a specific kind of role-play behavior, not necessarily how an actual customer with private, partial information would behave. LLM simulators also receive the task instruction, so the comparison is internally consistent, but the external-validity claim implicit in 'real user behavior' is stronger than the protocol supports. Please temper the wording (e.g., 'human role-play baseline' rather than 'real users') and, if feasible, add a smaller naturalistic condition where users receive only the information a real customer would have. At minimum, discuss this limitation explicitly in the main text.
minor comments (7)
  1. [§2.2] The phrase 'AA VE' in the related-work paragraph appears to be a typo for 'AAVE'. Please fix.
  2. [Figure 2] The figure caption states 'Human-Human agreement: 95.6%; Human-LLM agreement: 29.7% ~ 81.1%' but neither the agreement measure nor the calculation is defined in the text. Please add a pointer to the relevant section or appendix.
  3. [Figure 4] The caption says 'n=165×3 batches'; it would be clearer to write n=495, matching the text in Figure 5.
  4. [Table 1] The 'Human (inter-ann.)' row should state explicitly how the inter-annotator scores were computed (e.g., each batch vs. the aggregate of the other two) and how this relates to the human baseline in Figure 8. Additionally, the text says some models are marked '†' for missing survey data, but no such markers appear in Table 1; either add them or remove the sentence.
  5. [Appendix A.1] The annotation interface exposes intermediate agent reasoning and tool traces to the human participant. This could make the human's behavior different from a real customer who only sees the agent's final messages. The paper mentions this in passing; please discuss it as a potential limitation and, ideally, analyze whether it affects the D1–D4 results.
  6. [Figure 3] The horizontal axis labels are visually confusing (e.g., stray '1.0' and overlapping '8.5 10.0'). Please clean the axis labels and ensure all metric values are legible.
  7. [§5, footnote 2] The claim that specialized models fall below the human baseline 'due to their limited instruction-following capability' is an interpretation; the presented USI scores are correlational. Please soften the causal wording or provide evidence (e.g., instruction-following benchmarks for these models).

Circularity Check

0 steps flagged

No circular derivation: the central Sim2Real gap claim is measured against independently collected human interaction data, not derived from the authors' own prior results or fitted to the conclusion.

full rationale

The paper's derivation chain is self-contained against its own human data. The central quantitative claims—that LLM simulators create an 'easy mode' and inflate agent success rates—are supported by direct comparisons between simulator outputs and the human baseline: the human success rate (63.6%) is independently collected from 451 Prolific participants, and Figure 8 reports raw agent success rates per simulator against that baseline. The USI is an author-defined composite (USI = (D1+D2+D3+D4+(1-ECE)*100+Eval)/6), but it is not fitted to or optimized for the paper's conclusion; it simply aggregates per-dimension Dice coefficients, ECE, and evaluative MAE, all computed against the human traces. No parameter is fitted to the data and then renamed as a prediction. The self-citations (e.g., Zhou et al. 2024; Sun et al. 2025) appear as contextual motivation for LLM-based user simulation, not as load-bearing evidence for the gap measurements; there is no uniqueness theorem or ansatz imported from the authors' prior work. Appendix A.1 does flag a limitation: the human baseline is filtered by an LLM judge with a conservative bias (6/40 valid traces rejected) and only 'pass' traces are kept. This is a potential empirical validity threat to the reference distribution, but it is not circularity: the paper does not use the LLM judge to construct the very outcome it then claims to predict, and the bias is openly disclosed rather than hidden. If the filtered baseline is unrepresentative, the quantitative magnitude of the gap could shift, but that is a correctness concern, not a reduction-by-construction. The behavioral and evaluative comparisons are measured, not derived from definitions; therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claim rests on two families of assumptions: (1) the human sample is a valid ground truth, and (2) the author-defined metrics and aggregation choices faithfully measure fidelity. The human-data assumption is load-bearing and complicated by LLM-based filtering; the metric choices are arbitrary but not fitted to the conclusion.

free parameters (4)
  • USI equal weights = 1/6 each for D1, D2, D3, D4, (1-ECE)*100, and Eval
    The User-Sim Index aggregates six dimensions with equal weight; no sensitivity analysis or data-driven weighting is provided, so the composite score depends on an arbitrary choice.
  • Behavioral metric thresholds = e.g., short turn <=3 words; repeated trigram >5 occurrences; formal if em-dash; LIWC/NRC lexicon membership
    Hand-set thresholds and lexicon assignments define the D1–D4 metrics; different thresholds could change the reported gaps.
  • ECE difficulty binning = unspecified
    Expected Calibration Error groups tasks into bins by difficulty, but the number of bins and the difficulty definition are not specified, which affects ECE values and thus USI.
  • LLM judge pass threshold = 80/100
    Human traces were filtered by a GPT-5 quality judge with a score threshold of 80; this is a hand-set threshold that shapes the human baseline.
axioms (4)
  • domain assumption Prolific crowd workers role-playing with the full task instruction visible are a faithful proxy for real users of agentic customer-service systems.
    All human-vs-simulator comparisons use this role-played human sample as ground truth; if it is not representative, the central gap measurements are biased.
  • domain assumption LIWC2015 and NRC lexicons plus regex rules validly measure the intended behavioral constructs (politeness, uncertainty, frustration, etc.).
    The D1–D4 metrics rely on these tools; lexical proxies may miss or misclassify pragmatic behavior.
  • domain assumption The fixed agent (GPT-5.2) and the τ-bench task set are representative interaction partners for measuring simulator fidelity.
    Generalizability is tested with only one alternative agent and five simulators (Table 3), so the main rankings are conditional on the choice of agent.
  • ad hoc to paper USI's equal weighting and ECE binning are valid aggregations of the six dimensions.
    No theoretical or empirical justification is given for equal weights; the composite index is an author-defined summary rather than an externally established measure.
invented entities (1)
  • User-Sim Index (USI) no independent evidence
    purpose: Composite 0–100 metric aggregating behavioral, outcome-calibration, and evaluative dimensions to quantify simulator–human alignment.
    USI is introduced in this paper and its validity is demonstrated only on this study's data; it has no external benchmark or pre-registered validation.

pith-pipeline@v1.3.0-alltime-deepseek · 20752 in / 11429 out tokens · 116661 ms · 2026-08-04T05:47:37.985110+00:00 · methodology

0 comments
read the original abstract

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full $\tau$-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Figures

Figures reproduced from arXiv: 2603.11245 by Graham Neubig, Jiarui Liu, Maarten Sap, Qianou Ma, Sean Welleck, Sherry Tongshuang Wu, Weihua Du, Weiwei Sun, Xuhui Zhou, Yiming Yang, Yiqing Xie.

Figure 1
Figure 1. Figure 1: User-Sim Index (USI) vs. Chatbot Arena Elo Score for LLM simulators. Solid lines and shaded regions [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of Sim2Real gaps in user simulation. We highlight the dimensions where the gaps between [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-metric behavioral comparison for selected models (GPT-4o, Qwen3-235B, CoSER, UserLM-8b) and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Score distributions for human annotators and GPT-5.1 across quality dimensions ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-dimension human quality ratings by reward group ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Annotation interface used in the human study. The right panel displays the task instructions and role [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Human task success judgment (Yes / No / Policy-constrained) vs. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Agent success rates (%) on τ -bench when paired with each user simulator. Models are grouped by category (proprietary, open-source, specialized). The dashed line indicates the human baseline. Most general-purpose LLM simulators inflate agent success rates above the human baseline, while specialized user-simulation models fall below it. Customer: Um, hi. I actually need to return my luggage set, but I want … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes

    cs.AI 2026-06 accept novelty 8.0

    LLM user simulators exhibit a disengagement deficit: they match real buyers but systematically overstate purchase intent among real non-buyers by reducing expressed resistance and increasing deliberation.

  2. EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

    cs.SD 2026-05 accept novelty 8.0

    EVA-Bench introduces a simulation-plus-scoring framework for voice agents that reveals no tested system exceeds 0.5 on both accuracy and experience metrics at pass@1.

  3. Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models

    cs.AI 2026-04 unverdicted novelty 8.0

    User-turn generation reveals that LLMs' interaction awareness is largely decoupled from task accuracy, remaining near zero in deterministic settings even as accuracy scales to 96.8% on GSM8K.

  4. Will Scaling Improve Social Simulation with LLMs?

    cs.CL 2026-07 conditional novelty 7.0

    Scaling improves LLM social simulation fidelity in most opinion and behavior tasks but not for human cognitive bias calibration or low-resource domains.

  5. ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

    cs.CL 2026-06 conditional novelty 7.0

    ISE creates 23,132 execution-grounded multi-turn OS agent trajectories via intent simulation and live execution, improving agent performance on ClawEval from 19.3 to 37.7 pass@1 with Qwen3-8B.

  6. BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces

    cs.AI 2026-06 unverdicted novelty 7.0

    BehaviorBench reconstructs 2,000 real wallets into 141k belief and 1.4M trade prediction tasks to test if personalization from history improves model performance over non-personalized baselines.

  7. EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

    cs.SD 2026-05 unverdicted novelty 7.0

    EVA-Bench supplies a simulation engine for bot-to-bot voice dialogues plus two composite metrics (EVA-A for accuracy, EVA-X for experience) evaluated on 213 enterprise scenarios, showing no tested system exceeds 0.5 o...

  8. An Empirical Study of Proactive Coding Assistants in Real-World Software Development

    cs.SE 2026-05 unverdicted novelty 7.0

    Real developer IDE traces differ substantially from LLM simulations in behavior and structure; current proactive assistants are unreliable on real traces, and simulated data cannot substitute for real data in training.

  9. Will Scaling Improve Social Simulation with LLMs?

    cs.CL 2026-07 conditional novelty 6.0

    Using 85 controlled and 35 public LLMs, the authors show social-simulation accuracy generally improves with compute, but some behavioral and low-resource tasks do not scale.

  10. ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

    cs.CL 2026-06 conditional novelty 6.0

    Execution-grounded multi-turn OS trajectories from 4D intents and a role-locked simulator raise Qwen3-8B ClawEval pass@1 from 19.3 to 37.7, beating GPT-4o and Qwen3-32B.

  11. Simulating Human Memory with Language Models

    cs.CL 2026-05 unverdicted novelty 6.0

    Language models show superior memory to humans on psych experiments but can be adjusted via prompting and compaction to forget more human-like, yielding better user simulators.

  12. Reinforcing Human Behavior Simulation via Verbal Feedback

    cs.LG 2026-05 unverdicted novelty 6.0

    DITTO uses RL with verbal feedback to train LLMs for human behavior simulation, reporting 36% average gains over base models and outperforming GPT-5.4 on 6 of 10 SOUL benchmark tasks.

  13. ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

    cs.CL 2026-06 conditional novelty 5.5

    Execution-grounded multi-turn OS trajectories from 4D intents and a role-locked simulator lift Qwen3-8B ClawEval pass@1 from 19.3 to 37.7, beating GPT-4o and Qwen3-32B zero-shot.

  14. Large Behavior Model: A Promptable Digital Twin of the Retail Customer

    cs.AI 2026-07 conditional novelty 5.0

    Grounding an LLM in verbalized transaction histories via Person–Environment prompting, continued pre-training, SFT, and GRPO yields stronger retail decision simulation than frontier models, with partial cross-domain transfer.

  15. Large Behavior Model: A Promptable Digital Twin of the Retail Customer

    cs.AI 2026-07 conditional novelty 5.0

    A language model trained on verbalized retail transactions with continued pre-training, SFT, and GRPO outperforms frontier LLMs on customer behavior prediction tasks by grounding decisions in explicit behavioral evide...

  16. Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants

    cs.CL 2026-05 unverdicted novelty 5.0

    Fine-tuned simulators grounded in real human data produce LLM assistants that win more often against real users than those trained against role-playing simulators.

  17. Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models

    cs.AI 2026-06 unverdicted novelty 3.0

    Evaluates multimodal foundation models as agents for power distribution defect detection across perception, reasoning, and tool usage using a custom benchmark.

Reference graph

Works this paper leans on

50 extracted references · 21 linked inside Pith · cited by 12 Pith papers

  1. [1]

    Simuser: Simulating user behavior with large language models for recommender system evaluation

    Nicolas Bougie and Narimasa Watanabe. Simuser: Simulating user behavior with large language models for recommender system evaluation. In Annual Meeting of the Association for Computational Linguistics, 2025. URL https://aclanthology.org/2025.acl-industry.5/

    2025

  2. [2]

    Levinson

    Penelope Brown and Stephen C. Levinson. Politeness: Some Universals in Language Usage. Cambridge University Press, 1987

    1987

  3. [3]

    Code with me or for me? how increasing ai automation transforms developer workflows, 2025

    Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig. Code with me or for me? how increasing ai automation transforms developer workflows, 2025. URL https://arxiv.org/abs/2507.08149

    Pith/arXiv arXiv 2025

  4. [4]

    Clark and Susan E

    Herbert H. Clark and Susan E. Brennan. Grounding in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley (eds.), Perspectives on Socially Shared Cognition, pp.\ 127--149. American Psychological Association, 1991

    1991

  5. [5]

    User simulation with large language models for evaluating task-oriented dialogue

    Sam Davidson, Salvatore Romeo, Raphael Shu, James Gung, Arshit Gupta, Saab Mansour, and Yi Zhang. User simulation with large language models for evaluating task-oriented dialogue. arXiv preprint arXiv:2309.13233, 2023. URL https://arxiv.org/abs/2309.13233

    Pith/arXiv arXiv 2023

  6. [6]

    Simulatorarena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Conference on Empirical Methods in Natural Language Processing, 2025

    Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. Simulatorarena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Conference on Empirical Methods in Natural Language Processing, 2025. URL https://arxiv.org/abs/2510.05444

    arXiv 2025

  7. [7]

    Evaluating large language models as generative user simulators for conversational recommendation

    Se eun Yoon, Zhankui He, Jessica Maria Echterhoff, and Julian McAuley. Evaluating large language models as generative user simulators for conversational recommendation. In North American Chapter of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2403.09738

    Pith/arXiv arXiv 2024

  8. [8]

    Agent-based modelling meets generative ai in social network simulations

    Antonino Ferraro, Antonio Galli, Valerio La Gatta, Marco Postiglione, Gian Marco Orlando, Diego Russo, Giuseppe Riccio, Antonio Romano, and Vincenzo Moscato. Agent-based modelling meets generative ai in social network simulations. In International Conference on Social Networks Analysis and Mining, 2024. URL https://arxiv.org/abs/2411.16031

    Pith/arXiv arXiv 2024

  9. [9]

    Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor

    Kate Forbes-Riley and Diane Litman. Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Communication, 53 0 (9--10): 0 1115--1136, 2011

    2011

  10. [10]

    Communication accommodation theory

    Howard Giles and Tania Ogay. Communication accommodation theory. In Bryan B. Whaley and Wendy Samter (eds.), Explaining Communication: Contemporary Theories and Exemplars, pp.\ 293--310. Lawrence Erlbaum, 2007

    2007

  11. [11]

    Paul Grice

    H. Paul Grice. Logic and conversation. In Peter Cole and Jerry L. Morgan (eds.), Syntax and Semantics, Vol. 3: Speech Acts, pp.\ 41--58. Academic Press, 1975

    1975

  12. [12]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1321--1330. PMLR, 06--11 Aug 2017. URL https://proceedings.mlr.press/v70/guo17a.html

    2017

  13. [13]

    Real or robotic? assessing whether llms accurately simulate qualities of human responses in dialogue

    Jonathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramaniam, Junghwan Kim, Bowen Yi, Dustin Wright, Abraham Israeli, Anders Giovanni M ller, Lechen Zhang, and David Jurgens. Real or robotic? assessing whether llms accurately simulate qualities of human responses in dialogue. ArXiv, abs/2409.083...

    Pith/arXiv arXiv 2024

  14. [14]

    Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2024

  15. [15]

    Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning

    Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.00922

    Pith/arXiv arXiv 2024

  16. [16]

    Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen

    Xiujun Li, Zachary C. Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688, 2016. URL https://arxiv.org/abs/1612.05688

    Pith/arXiv arXiv 2016

  17. [17]

    Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL, 2025. URL https://arxiv.org/ab...

    Pith/arXiv arXiv 2025

  18. [18]

    Mohammad and Peter D

    Saif M. Mohammad and Peter D. Turney. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29 0 (3): 0 436--465, 2013. URL https://doi.org/10.1111/j.1467-8640.2012.00460.x

    arXiv 2013

  19. [19]

    Flipping the dialogue: Training and evaluating user language models

    Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. Flipping the dialogue: Training and evaluating user language models. arXiv preprint arXiv:2510.06552, 2025. URL https://arxiv.org/abs/2510.06552

    arXiv 2025

  20. [20]

    Pennebaker, Ryan L

    James W. Pennebaker, Ryan L. Boyd, Kayla Jordan, and Kate Blackburn. The development and psychometric properties of LIWC2015 . Technical report, University of Texas at Austin, 2015

    2015

  21. [21]

    Userbench: An interactive gym environment for user-centric agents

    Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. Userbench: An interactive gym environment for user-centric agents. ArXiv, abs/2507.22034, 2025. URL https://arxiv.org/abs/2507.22034

    Pith/arXiv arXiv 2025

  22. [22]

    Bases: Large-scale web search user simulation with large language model based agents

    Ruiyang Ren, Peng Qiu, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Bases: Large-scale web search user simulation with large language model based agents. In Findings of the Association for Computational Linguistics: EMNLP, 2024. URL https://arxiv.org/abs/2402.17505

    Pith/arXiv arXiv 2024

  23. [23]

    Personagym: Evaluating persona agents and llms

    Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari. Personagym: Evaluating persona agents and llms. In Findings of the Association for Computational Linguistics: EMNLP, 2025. URL https://arxiv.org/abs/2407.18416

    Pith/arXiv arXiv 2025

  24. [24]

    A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies

    Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review, 21 0 (2): 0 97--126, 2006. URL https://doi.org/10.1017/S0269888906000944

    doi:10.1017/s0269888906000944 2006

  25. [25]

    Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments, 2024

    Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments, 2024. URL https://arxiv.org/abs/2405.07960

    Pith/arXiv arXiv 2024

  26. [26]

    Reliable llm-based user simulator for task-oriented dialogue systems

    Ivan Sekulic, Silvia Terragni, Victor Guimaraes, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. Reliable llm-based user simulator for task-oriented dialogue systems. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT), 2024. URL https://arxiv.org/abs/2402.13374

    Pith/arXiv arXiv 2024

  27. [27]

    Lost in simulation: Llm-simulated users are unreliable proxies for human users in agentic evaluations

    Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb-Tarrant. Lost in simulation: Llm-simulated users are unreliable proxies for human users in agentic evaluations. arXiv preprint arXiv:2601.17087, 2026. URL https://arxiv.org/abs/2601.17087

    arXiv 2026

  28. [28]

    Completion collaboration: Scaling collaborative effort with agents, 2025

    Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu, and David Sontag. Completion collaboration: Scaling collaborative effort with agents, 2025. URL https://arxiv.org/abs/2510.25744

    arXiv 2025

  29. [29]

    Why johnny can't use agents: Industry aspirations vs

    Pradyumna Shome, Sashreek Krishnan, and Sauvik Das. Why johnny can't use agents: Industry aspirations vs. user realities with ai agent software. ArXiv, abs/2509.14528, 2025. URL https://arxiv.org/abs/2509.14528

    Pith/arXiv arXiv 2025

  30. [30]

    Exploring human error recovery strategies: Implications for spoken dialogue systems

    Gabriel Skantze. Exploring human error recovery strategies: Implications for spoken dialogue systems. Speech Communication, 45 0 (3): 0 325--341, 2005

    2005

  31. [31]

    Simulating user satisfaction for the evaluation of task-oriented dialogue systems

    Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021. URL https://dl.acm.org/doi/10.1145/3404835.3463241

    arXiv 2021

  32. [32]

    Training proactive and personalized llm agents, 2025

    Weiwei Sun, Xuhui Zhou, Weihua Du, Xingyao Wang, Sean Welleck, Graham Neubig, Maarten Sap, and Yiming Yang. Training proactive and personalized llm agents, 2025. URL https://arxiv.org/abs/2511.02208

    arXiv 2025

  33. [33]

    Do llms exhibit human-like response biases? a case study in survey design

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. Do llms exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12: 0 1011--1026, 2024

    2024

  34. [34]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017. URL https://arxiv.org/abs/1703.06907

    Pith/arXiv arXiv 2017

  35. [35]

    Interactive agents to overcome underspecificity in software engineering

    Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. Interactive agents to overcome underspecificity in software engineering. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=X2yzXtH4wp

    2026

  36. [36]

    User behavior simulation with large language model-based agents

    Lei Wang, Jingsen Zhang, Hao Yang, Zhi-Yuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji rong Wen. User behavior simulation with large language model-based agents. ACM Transactions on Information Systems, 2025 a . URL https://dl.acm.org/doi/10.1145/3708985

    doi:10.1145/3708985 2025

  37. [37]

    Coser: A comprehensive literary dataset and framework for training and evaluating llm role-playing and persona simulation, 2026

    Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, and Yanghua Xiao. Coser: A comprehensive literary dataset and framework for training and evaluating llm role-playing and persona simulation, 2026. URL https://arxiv.org/abs/2502.09082

    arXiv 2026

  38. [38]

    Human vs

    Zhefan Wang, Ning Geng, Zhiqiang Guo, Weizhi Ma, and Min Zhang. Human vs. agent in task-oriented conversations. Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, 2025 b . URL https://arxiv.org/abs/2509.17619

    arXiv 2025

  39. [39]

    Humanlm: Simulating users with state alignment beats response imitation, 2026

    Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. Humanlm: Simulating users with state alignment beats response imitation, 2026. URL https://arxiv.org/abs/2603.03303

    arXiv 2026

  40. [40]

    -bench: A benchmark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. URL https://arxiv.org/abs/2406.12045

    Pith/arXiv arXiv 2024

  41. [41]

    Usimagent: Large language models for simulating search users

    Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, and Jiaxin Mao. Usimagent: Large language models for simulating search users. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024. URL https://arxiv.org/abs/2403.09142

    Pith/arXiv arXiv 2024

  42. [42]

    Exploring human-like thinking in search simulations with large language models

    Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Zixuan Yang, and Jiaxin Mao. Exploring human-like thinking in search simulations with large language models. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025. URL https://arxiv.org/abs/2504.07570

    Pith/arXiv arXiv 2025

  43. [43]

    Sim-to-real transfer in deep reinforcement learning for robotics: A survey

    Wenshuai Zhao, Jorge Pe \ n a Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In IEEE Symposium Series on Computational Intelligence, 2020. URL https://arxiv.org/abs/2009.13303

    Pith/arXiv arXiv 2020

  44. [44]

    SOTOPIA : Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA : Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mM7VurbA4r

    2024

  45. [45]

    Tom-swe: User mental modeling for software engineering agents

    Xuhui Zhou, Valerie Chen, Zora Zhiruo Wang, Graham Neubig, Maarten Sap, and Xingyao Wang. Tom-swe: User mental modeling for software engineering agents. ArXiv, abs/2510.21903, 2025 a . URL https://arxiv.org/abs/2510.21903

    arXiv 2025

  46. [46]

    Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, Ronan Le Bras, and Maarten Sap

    Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank F. Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, Ronan Le Bras, and Maarten Sap. HAICOSYSTEM : An ecosystem for sandboxing safety risks in human-ai interactions. In Second Conference on Language Modeling, 2025 b . URL https://arxiv.org/abs/2409.16427

    Pith/arXiv arXiv 2025

  47. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  48. [48]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  49. [49]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  50. [50]

    score": <0-100>, // 80 is the passing score

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2020