Pith. sign in

REVIEW 2 major objections 5 minor 53 references

Tool-use agents are being tested on the wrong tasks: real requests are vague, infeasible, or risky, and this paper shows that the missing skill — asking the user — is the main bottleneck, with the best agent at 48.6% and oracle knowledge at

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:29 UTC pith:OSJFKTE6

load-bearing objection Solid, well-engineered benchmark that extends AppWorld into user-in-the-loop territory; the interaction-difficulty claim holds, but the I-TGC metric's dependence on an unvalidated LLM user mapping is a genuine soft spot that should be fixed before the numbers become canonical. the 2 major comments →

arxiv 2607.20536 v1 pith:OSJFKTE6 submitted 2026-07-10 cs.AI cs.CLcs.LGcs.MA

AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

Junzhi Chen , Harsh Trivedi , Jane Pan , Michael JQ Zhang , Tejas Srinivasan , Niranjan Balasubramanian , Ashish Sabharwal This is my paper
classification cs.AI cs.CLcs.LGcs.MA
keywords user-in-the-loop agentstool-use benchmarkperturbation-based transformationsimulated userinteraction evaluationclarification and confirmationAppWorld-UL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that standard tool-use benchmarks understate the real difficulty of deploying agents because they hand agents complete instructions, while real users give vague, infeasible, or high-stakes requests that require dialogue. It proposes AppWorld-UL, a benchmark of 516 tasks in a stateful nine-app environment, built by a perturbation method: take a well-defined task, remove or alter a specific piece of information, and record the missing information as a user knowledge set K that the agent must elicit. A simulated user, driven by an LLM but constrained to answer only from K, supplies the missing knowledge; a new metric, I-TGC, checks both task completion and whether the agent asked for all of K. The empirical claim is that interaction is the main difficulty: the best system reaches 48.6% I-TGC overall, oracle knowledge raises that to 78.1%, and a user who withholds information drops it to 12.8%.

Core claim

The paper's central claim is that the difficulty of user-in-the-loop tool use is measurable and largely attributable to interaction: a state-of-the-art code agent reaches only 48.6% I-TGC overall and 35.7% on compositional tasks, while the same agent given all user knowledge in advance reaches 78.1%, and a user who refuses information drops it to 12.8%. The benchmark formalizes each interactive task as a tuple (I, S0, E, K), where K is the user-knowledge set of question-answer pairs that the agent must elicit through dialogue. The simulated user is constrained to answer only questions that map to K and to deflect others, and the evaluation adds an interaction unit test to the usual goal-comp

What carries the argument

The central object is the User Knowledge Set K — a set of question-answer pairs containing precisely the information removed from the original autonomous task by perturbation. K does three jobs: it defines the simulated user's knowledge boundaries, it lets the benchmark know exactly what the agent does not know and therefore should ask, and it grounds a programmatic interaction metric by recording which questions the agent actually asked. The perturbation method is the other half: by introducing known gaps into instructions, initial states, and evaluation criteria, the authors create underspecified, infeasible, and confirmation-requiring tasks whose missing information is exactly K.

Load-bearing premise

The evaluation's interaction metric depends on the simulated user correctly recognizing when an agent's question maps to one of the hand-authored knowledge-set entries; the paper manually checked 312 responses for naturalness but did not independently validate the correctness of this mapping, so a mis-mapping would unfairly penalize agents that asked the right thing in different words.

What would settle it

Have human annotators independently re-score a sample of logged agent–user dialogues, marking for each agent question whether it maps to a K entry and whether the simulated user's answer supplies exactly that entry; recompute I-TGC from these judgments. If agreement with the simulator's mapping is low, or the human-scored I-TGC diverges by more than a few points, the benchmark's interaction metric is not measuring what it claims.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Non-interactive success rates will overstate deployed performance: if the agent cannot ask for missing knowledge, it fails even when it can navigate all the APIs.
  • Interaction skill is separable from tool-use skill: the oracle ablation bounds how much of the gap is attributable to the need to ask, providing a target for future work.
  • Any stateful benchmark with programmatic evaluation can be turned into a user-in-the-loop benchmark via the same perturbation recipe, so the methodology has leverage beyond AppWorld-UL.
  • Asking the right questions is strongly diagnostic: on successful tasks agents recall all expected questions, while on failed tasks recall drops to 37–40%, so interaction quality predicts task success.
  • Compositional tasks — combining clarification, infeasibility, and approval — are disproportionately harder than single-type tasks, so scaling to realistic mixed interactions is the next challenge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The recall-only interaction metric deliberately ignores extra questions; an agent that asks many irrelevant questions can still score well if it also asks the required ones. A cost-weighted variant suggested by the paper's own efficiency analysis would reward asking fewer, better questions.
  • Because K explicitly enumerates what the agent should ask, AppWorld-UL is a natural source of synthetic supervision: successful dialogues show exactly which questions should have been asked, which could be used to train or fine-tune interactive agents.
  • The paper does not test multi-party knowledge — a user who knows some facts and a colleague who knows others. Since K is a flat set, extending it to per-person knowledge would test whether agents track who knows what.
  • The observed 'hallucinated self-resolution' failure suggests a sharper evaluation dimension: penalize agents for inventing missing facts rather than asking. A future version could add decoy items that tempt guessing, making the interaction metric more sensitive.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper introduces AppWorld-UL, a benchmark of 516 user-in-the-loop tasks built by perturbing 34 AppWorld scenarios to require one or more of three interaction types: clarification for underspecified goals, infeasibility communication, and confirmation-seeking. Each task includes a hand-authored user knowledge set K; a constrained LLM-based simulated user answers only questions mapped to K. The paper proposes I-TGC/I-SGC, which add a recall-of-K unit test to AppWorld's programmatic evaluation. In experiments across six LLMs and two scaffolds, the best system (Code Agent with Claude Opus 4.7) reaches 48.6% I-TGC overall and 35.7% on the compositional subset. Ablations show that withholding K drops GPT-5.5 Code TGC to 12.8%, while providing oracle K raises it to 78.1%, supporting the claim that interaction is a major source of difficulty.

Significance. AppWorld-UL is a timely and useful contribution. It moves beyond simple clarification benchmarks by adding infeasibility and confirmation interactions, and it inherits AppWorld's 475-API stateful environment and programmatic evaluation. The perturbation methodology is principled: because K is known by construction, the authors can check solvability with stubbed solutions, simulate a constrained user, and score interaction quality programmatically. The code/data release and the strong oracle/hidden ablation are particular strengths. However, the validity of I-TGC as a measure of interaction quality is not yet established, because it depends on an LLM's mapping from agent questions to K that is not independently validated; this is the main technical risk in accepting the headline numbers.

major comments (2)
  1. [§4.2, §6.6; Table 3] Both the simulated user's behavior and the I-TGC unit test rely on the same LLM mapping from agent questions to K. The manual evaluation in §6.6 judges naturalness, not mapping correctness; Table 3 shows LLM-user consistency, not accuracy against ground truth. A systematic over-strict mapping would both deflect legitimate agent questions and mark them as unasked, falsely inflating the apparent interaction difficulty. I recommend a human annotation study of mapping correctness on a stratified sample of agent turns, reporting accuracy per interaction type, and a sensitivity analysis in which I-TGC is recomputed under alternative mappings.
  2. [§5.1 (Validating Solvability); §4.1] The solvability check verifies that the stubbed solution using all K pairs passes E', but it does not verify that each K pair is necessary—i.e., that no TGC-passing trajectory avoids asking it. Because I-TGC requires all K questions to have been asked, an agent that finds a legitimate alternative solution (AppWorld explicitly allows multiple valid paths) would be penalized even though it completed the task. Please add a per-K-pair necessity check or a meta-evaluation showing that removing any K pair causes the reference solution to fail, and report how many tasks have alternative TGC-passing trajectories without all K questions.
minor comments (5)
  1. [Appendix A] Typo: 'takeawy' should be 'takeaway'.
  2. [Figure 5 caption] The phrase 'interaction requirements account for most of the difficulty' is stronger than the evidence, since the Oracle condition changes both the presence of K and its presentation format (auxiliary QA context vs. instruction text). Please qualify the caption.
  3. [Table 2] The missing GLM 5.1 FC entry is footnoted in the text but should be marked within the table itself to avoid ambiguity.
  4. [§4.1] 'Non-conflicting perturbations' is defined only by example; please state the formal condition (e.g., disjoint modifications to S0, I, E, K) used in the automatic check.
  5. [Figures 3 and 4] The y-axis labels 'avg recall' and 'avg precision' are unclear about the aggregation unit (tasks vs. questions); please define in the caption.

Circularity Check

1 steps flagged

The interaction-importance conclusion is partly definitional (I-TGC counts missing K questions as failure), while the headline difficulty numbers remain independent measurements.

specific steps
  1. self definitional [§5.1 Evaluation Metrics; §6.4 Ablation Study]
    "We introduce the corresponding “interactive” variants I-TGC and I-SGC, which effectively add a unit test that checks that all the required questions in K were asked. ... The Hidden Knowledge setting yields much lower performance (TGC: 12.8%, SGC: 2.3%), confirming that K is necessary for task completion—tasks cannot be solved through environmental reasoning alone. Conversely, Oracle Knowledge improves performance (TGC: 78.1%, SGC: 58.3%), showing that removing interaction requirements leads to far more successful task completion, indicating that user-in-the-loop dynamics significantly contribu"

    The ablation's central conclusion is entailed by construction: perturbations are defined by moving the information needed for success into K (“The removed specification ... becomes part of K. Now the agent must query the user ... to pass the original evaluation tests”), and solvability validation requires “all knowledge in K is actually used for task completion.” I-TGC then adds a unit test that failing to ask the K questions is a failure. Hence an agent that does not elicit K necessarily fails both TGC and I-TGC; “correct user-interaction is crucial” is thus a restatement of the task-definition rather than an independent empirical discovery. The direction of the effect is guaranteed, though the reported magnitudes (78.1 vs. 12.8) still contain empirical content about how much agents can c

full rationale

This is a benchmark paper, not a quantitative derivation, so most circularity patterns do not apply. There are no fitted parameters, no prediction that reduces to a fitted input, and no uniqueness theorem imported from the authors. The reliance on AppWorld is a legitimate reuse of a released, code-backed environment with programmatic tests; self-citation there is not load-bearing. The one mild definitional loop concerns the ablation narrative: because tasks are constructed so that K is necessary and I-TGC explicitly penalizes missing K questions, the claim that interaction is essential is partly true by design, not solely by experiment. However, the headline results (best system 48.6% I-TGC, 35.7% on compositional tasks) come from actual agent runs against programmatic state-based tests, and the numerical gap between Oracle and Hidden conditions is not fully determined a priori. The skeptic's concern about the LLM mapping from agent questions to K is a validity/reliability risk rather than circularity: consistent simulator LLMs and a small naturalness review do not validate mapping correctness, but this does not make the benchmark's measured difficulty equivalent to its inputs. Overall score 3: one partial, definitional reduction in an auxiliary finding, while the central benchmark contribution retains independent empirical content.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The benchmark's central claims rest on the simulated user being a faithful proxy for a real user with exactly the knowledge K, and on the programmatic interaction metric correctly detecting whether the agent asked for that knowledge. The paper provides some evidence (solution validation, user-LLM robustness, manual review of naturalness) but independent verification of the question-mapping step is absent. No numeric free parameters were fitted; the hand-designed knowledge sets and QA pairs are dataset content, not fitted model parameters.

axioms (4)
  • domain assumption The simulated user's knowledge boundaries and responses accurately represent a real user who knows exactly the information in K and nothing else.
    §4.2 describes the constrained LLM user; if real users would provide more/less info, measured difficulty shifts.
  • domain assumption AppWorld's simulated apps and APIs faithfully model realistic tool-use environment.
    The paper inherits AppWorld (Trivedi et al., 2024) without re-validation (§5).
  • ad hoc to paper The I-TGC unit test that all K questions were asked is a valid measure of interaction quality.
    This metric is newly defined in §5.1; a different metric could change agent rankings.
  • ad hoc to paper The perturbation and manual task construction do not introduce hidden ambiguities beyond the intended knowledge gaps.
    Authors manually verify solvability (§5.1), but no independent verification.

pith-pipeline@v1.3.0-alltime-deepseek · 37197 in / 12918 out tokens · 140380 ms · 2026-08-02T07:29:22.938254+00:00 · methodology

0 comments
read the original abstract

Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.g., to ask clarification questions, prompt for confirmation, and inform the user when the instruction is infeasible. However, current benchmarks for evaluating agent-user interactions do not capture the diversity of such interactions. Further, they operate in small environments with few, often non-state-changing, APIs. To address this gap, we introduce AppWorld-UL, a ``user-in-the-loop'' benchmark of 516 challenging tasks requiring diverse agent-user interactions. Building upon the AppWorld framework with 9 popular simulated apps like Amazon and Spotify, we systematically modify original tasks to introduce ambiguities and constraints that necessitate various types of agent-user interaction. User behavior is simulated by an LLM prompted to respond with carefully designed knowledge boundaries, offering more reliable simulation than the unconstrained or overly rigid alternatives used in prior work. Our evaluation reveals that a state-of-the-art LLM, Claude Opus 4.7, achieves only 48.6% success on AppWorld-UL, and only 35.7% on the harder, compositional subset. On the stricter, scenario-level metric, compositional task performance drops to only 21.3%. Our analysis reveals that correct user-interaction is crucial for success. This demonstrates the benchmark's difficulty and its potential to advance research on user-in-the-loop tool-use agents.

Figures

Figures reproduced from arXiv: 2607.20536 by Ashish Sabharwal, Harsh Trivedi, Jane Pan, Junzhi Chen, Michael JQ Zhang, Niranjan Balasubramanian, Tejas Srinivasan.

Figure 1
Figure 1. Figure 1: Three phenomena that necessitate agent-user interaction in AppWorld-UL. Underspecification: clarifying which path to take when multiple paths are available. Infeasibility: commu￾nicating infeasibility when it arises. Need for Approval: Seek￾ing explicit confirmation before executing actions with high cost. Agent-Environment interactions are hidden. 1. Introduction LLM-based agents have demonstrated strong … view at source ↗
Figure 2
Figure 2. Figure 2: Left and middle: Perturbation process for converting a well-defined autonomous agent task T in a stateful environment into user-in-the-loop tasks (T1, T2, T3) that induce three real-world agent–user interaction types. These perturbations are also composed together to induce multiple types of agent–user interactions in the same task. Right: Example agent trajectory illustrating the resulting agent–user inte… view at source ↗
Figure 4
Figure 4. Figure 4: Interaction Deviation: difference between the number of questions asked and the number of expected questions (optimal = 0). The code agent asks efficiently on successful tasks but under￾or over-asks on failed tasks. action. Hidden Knowledge: The simulated user refuses to provide any information, forcing it to proceed without K. In both Oracle and Hidden Knowledge conditions, we evaluate only goal completio… view at source ↗
Figure 5
Figure 5. Figure 5: The large performance drop from Default to Hidden confirms user knowledge (K) is necessary, while the large per￾formance improvement from Default to Oracle shows interaction requirements account for most of the difficulty. The Hidden Knowledge setting yields much lower perfor￾mance (TGC: 12.8%, SGC: 2.3%), confirming that K is nec￾essary for task completion—tasks cannot be solved through environmental reas… view at source ↗
Figure 6
Figure 6. Figure 6: Performance–Efficiency Trade-off. I-TGC versus num￾ber of user interactions and total cost per task. CO and FC refer to the Code and Function Calling agents, respectively. Increased interaction or cost does not guarantee higher performance. Interestingly, more user interactions do not necessarily im￾prove performance. Some agents achieve lower scores de￾spite asking more questions, suggesting they ask irre… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 7 linked inside Pith

  1. [1]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  2. [2]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  3. [3]

    M. J. Kearns , title =

  4. [4]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  5. [5]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  6. [6]

    Suppressed for Anonymity , author=

  7. [7]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  8. [8]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  9. [9]

    2024 , url =

    Subramonyam, Hari and Pea, Roy and Pondoc, Christopher and Agrawala, Maneesh and Seifert, Colleen , title =. 2024 , url =

    2024

  10. [10]

    2024 , isbn =

    Guo, Jiajing and Mohanty, Vikram and Piazentin Ono, Jorge H and Hao, Hongtao and Gou, Liang and Ren, Liu , title =. 2024 , isbn =

    2024

  11. [11]

    2025 , isbn =

    He, Gaole and Demartini, Gianluca and Gadiraju, Ujwal , title =. 2025 , isbn =

    2025

  12. [13]

    2025 , url =

    Epperson, Will and Bansal, Gagan and Dibia, Victor C and Fourney, Adam and Gerrits, Jack and Zhu, Erkang (Eric) and Amershi, Saleema , title =. 2025 , url =

    2025

  13. [14]

    2024 , isbn =

    Mozannar, Hussein and Bansal, Gagan and Fourney, Adam and Horvitz, Eric , title =. 2024 , isbn =

    2024

  14. [15]

    2024 , isbn =

    Kim, Taewan and Shin, Donghoon and Kim, Young-Ho and Hong, Hwajung , title =. 2024 , isbn =

    2024

  15. [16]

    2022 , url =

    Chung, John Joon Young and Kim, Wooseok and Yoo, Kang Min and Lee, Hwaran and Adar, Eytan and Chang, Minsuk , title =. 2022 , url =

    2022

  16. [17]

    2024 , eprint=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. 2024 , eprint=

    2024

  17. [18]

    2025 , eprint=

    ^2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment , author=. 2025 , eprint=

    2025

  18. [22]

    Gonzalez and Shishir G

    Huanzhi Mao and Fanjia Yan and Charlie Cheng-Jie Ji and Jason Huang and Vishnu Suresh and Yixin Huang and Xiaowen Yu and Joseph E. Gonzalez and Shishir G. Patil , url =

  19. [25]

    Harsh Trivedi and Tushar Khot and Mareike Hartmann and Ruskin Manku and Vinty Dong and Edward Li and Shashank Gupta and Ashish Sabharwal and Niranjan Balasubramanian , booktitle=

  20. [26]

    Merrill, Mike A and Shaw, Alexander G and Carlini, Nicholas and Li, Boxuan and Raj, Harsh and Bercovich, Ivan and Shi, Lin and Shin, Jeong Yeon and Walshe, Thomas and Buchanan, E Kelly and others , journal=

  21. [27]

    Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig , booktitle=

    Shuyan Zhou and Frank F. Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig , booktitle=. 2024 , url=

    2024

  22. [28]

    2024 , url=

    Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan , booktitle=. 2024 , url=

    2024

  23. [29]

    Introducing

    Google , url =. Introducing

  24. [30]

    Introducing

    Kimi , url =. Introducing

  25. [31]

    Introducing

    Anthropic , url =. Introducing

  26. [32]

    Introducing

    OpenAI , url =. Introducing

  27. [33]

    2023 , url=

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik R Narasimhan and Yuan Cao , booktitle=. 2023 , url=

    2023

  28. [34]

    Introducing C laude S onnet 4.5, 2025

    Anthropic. Introducing C laude S onnet 4.5, 2025. URL https://www.anthropic.com/news/claude-sonnet-4-5

    2025

  29. [35]

    ^2 -bench: Evaluating conversational agents in a dual-control environment, 2025

    Barres, V., Dong, H., Ray, S., Si, X., and Narasimhan, K. ^2 -bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982

    Pith/arXiv arXiv 2025

  30. [36]

    Chung, J. J. Y., Kim, W., Yoo, K. M., Lee, H., Adar, E., and Chang, M. TaleBrush : Sketching stories with generative pretrained language models. In CHI, 2022. URL https://doi.org/10.1145/3491102.3501819

    arXiv 2022

  31. [37]

    C., Fourney, A., Gerrits, J., Zhu, E

    Epperson, W., Bansal, G., Dibia, V. C., Fourney, A., Gerrits, J., Zhu, E. E., and Amershi, S. Interactive debugging and steering of multi-agent AI systems. In CHI, 2025. URL https://doi.org/10.1145/3706598.3713581

    arXiv 2025

  32. [38]

    Introducing G emini 3, 2025

    Google. Introducing G emini 3, 2025. URL https://blog.google/products-and-platforms/products/gemini/gemini-3-collection/

    2025

  33. [39]

    H., Hao, H., Gou, L., and Ren, L

    Guo, J., Mohanty, V., Piazentin Ono, J. H., Hao, H., Gou, L., and Ren, L. Investigating interaction modes and user agency in human- LLM collaboration for domain-specific data analysis. In CHI, 2024. ISBN 9798400703317. URL https://doi.org/10.1145/3613905.3651042

    arXiv 2024

  34. [40]

    Plan-then-execute: An empirical study of user trust and team performance when using LLM agents as a daily assistant

    He, G., Demartini, G., and Gadiraju, U. Plan-then-execute: An empirical study of user trust and team performance when using LLM agents as a daily assistant. In CHI, 2025. ISBN 9798400713941. URL https://doi.org/10.1145/3706598.3713218

    arXiv 2025

  35. [41]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE -bench: Can language models resolve real-world github issues? In ICLR, 2024. URL https://openreview.net/forum?id=VTF8yNQM66

    2024

  36. [42]

    DiaryMate : Understanding user perceptions and experience in human- AI collaboration for personal journaling

    Kim, T., Shin, D., Kim, Y.-H., and Hong, H. DiaryMate : Understanding user perceptions and experience in human- AI collaboration for personal journaling. In CHI, 2024. ISBN 9798400703300. URL https://doi.org/10.1145/3613904.3642693

    arXiv 2024

  37. [43]

    Introducing Kimi K2 Thinking , 2025

    Kimi. Introducing Kimi K2 Thinking , 2025. URL https://moonshotai.github.io/Kimi-K2/thinking.html

    2025

  38. [44]

    Sketch2Code : Evaluating vision-language models for interactive web design prototyping, 2024

    Li, R., Zhang, Y., and Yang, D. Sketch2Code : Evaluating vision-language models for interactive web design prototyping, 2024. URL https://arxiv.org/abs/2410.16232

    Pith/arXiv arXiv 2024

  39. [45]

    ToolSandbox : A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities, 2025

    Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., Wang, Z., and Pang, R. ToolSandbox : A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities, 2025. URL https://arxiv.org/abs/2408.04682

    Pith/arXiv arXiv 2025

  40. [46]

    User interaction patterns and breakdowns in conversing with LLM -powered voice assistants

    Mahmood, A., Wang, J., Yao, B., Wang, D., and Huang, C.-M. User interaction patterns and breakdowns in conversing with LLM -powered voice assistants. International Journal of Human-Computer Studies, 195: 0 103406, 2025. ISSN 1071-5819. doi:https://doi.org/10.1016/j.ijhcs.2024.103406. URL https://www.sciencedirect.com/science/article/pii/S1071581924001897

    arXiv 2025

  41. [47]

    C.-J., Huang, J., Suresh, V., Huang, Y., Yu, X., Gonzalez, J

    Mao, H., Yan, F., Ji, C. C.-J., Huang, J., Suresh, V., Huang, Y., Yu, X., Gonzalez, J. E., and Patil, S. G. BFCL V3 : Multi-turn & multi-step function calling evaluation, 2024. URL https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html

    2024

  42. [48]

    A., Shaw, A

    Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., et al. Terminal-Bench : Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  43. [49]

    Reading between the lines: Modeling user behavior and costs in AI -assisted programming

    Mozannar, H., Bansal, G., Fourney, A., and Horvitz, E. Reading between the lines: Modeling user behavior and costs in AI -assisted programming. In CHI, 2024. ISBN 9798400703300. URL https://doi.org/10.1145/3613904.3641936

    arXiv 2024

  44. [50]

    Introducing GPT-5 , 2025

    OpenAI. Introducing GPT-5 , 2025. URL https://openai.com/index/introducing-gpt-5/

    2025

  45. [51]

    UserBench : An interactive gym environment for user-centric agents, 2025

    Qian, C., Liu, Z., Prabhakar, A., Liu, Z., Zhang, J., Chen, H., Ji, H., Yao, W., Heinecke, S., Savarese, S., Xiong, C., and Wang, H. UserBench : An interactive gym environment for user-centric agents, 2025. URL https://arxiv.org/abs/2507.22034

    Pith/arXiv arXiv 2025

  46. [52]

    Qwen3 : Think deeper, act faster, 2025

    Qwen. Qwen3 : Think deeper, act faster, 2025. URL https://qwen.ai/blog?id=qwen3

    2025

  47. [53]

    Lost in simulation: LLM -simulated users are unreliable proxies for human users in agentic evaluations, 2026

    Seshadri, P., Cahyawijaya, S., Odumakinde, A., Singh, S., and Goldfarb-Tarrant, S. Lost in simulation: LLM -simulated users are unreliable proxies for human users in agentic evaluations, 2026. URL https://arxiv.org/abs/2601.17087

    arXiv 2026

  48. [54]

    Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with LLM s

    Subramonyam, H., Pea, R., Pondoc, C., Agrawala, M., and Seifert, C. Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with LLM s. In CHI, 2024. URL https://doi.org/10.1145/3613904.3642754

    arXiv 2024

  49. [55]

    A pp W orld: A controllable world of apps and people for benchmarking interactive coding agents

    Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., and Balasubramanian, N. A pp W orld: A controllable world of apps and people for benchmarking interactive coding agents. In ACL, 2024

    2024

  50. [56]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. ReAct : Synergizing reasoning and acting in language models. In ICLR, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X

    2023

  51. [57]

    -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024

    Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/2406.12045

    Pith/arXiv arXiv 2024

  52. [58]

    F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G

    Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. WebArena : A realistic web environment for building autonomous agents. In ICLR, 2024. URL https://openreview.net/forum?id=oKn9c6ytLx

    2024

  53. [59]

    SWEET-RL : Training multi-turn LLM agents on collaborative reasoning tasks, 2025

    Zhou, Y., Jiang, S., Tian, Y., Weston, J., Levine, S., Sukhbaatar, S., and Li, X. SWEET-RL : Training multi-turn LLM agents on collaborative reasoning tasks, 2025. URL https://arxiv.org/abs/2503.15478

    Pith/arXiv arXiv 2025