Pith. sign in

REVIEW 4 major objections 5 minor 16 cited by

UserBench: An Interactive Gym Environment for User-Centric Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UserBench, an interactive benchmark with simulated users who reveal preferences implicitly and incrementally, finds that leading LLM agents fully satisfy all user intents only about 20% of the time and actively elicit under 30% of…

desk verdict UserBench is a useful new benchmark for user-centric agents, but the headline '20% full alignment' claim is not supported by the reported metrics and needs a fix before this is citable. read the letter →

arxiv 2507.22034 v1 pith:G6USPTWD submitted 2025-07-29 cs.AI cs.CLcs.LG

Cheng Qian , Zuxin Liu , Akshara Prabhakar , Zhiwei Liu , Jianguo Zhang , Haolin Chen , Heng Ji , Weiran Yao
show 4 more authors
Shelby Heinecke Silvio Savarese Caiming Xiong Huan Wang
This is my paper
classification cs.AIcs.CLcs.LG
keywords user-centricagentspreferenceelicitationinteractivebenchmarksimulatedusertravelplanningLLMalignmentmulti-turninteractiontooluse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that current LLM agents can plan and use tools but are poor at interactive user understanding. It introduces UserBench, a Gymnasium-style environment where simulated users start with underspecified travel goals, reveal preferences only gradually and often indirectly, and reward agents that both uncover these preferences and pick cost-effective options. Across leading open- and closed-source models, the authors find a large gap between task execution and user alignment: fully aligned answers occur about 20% of the time on average, and even the best model actively uncovers under 30% of preferences. The claim matters because it isolates a missing capability—proactive, preference-driven dialogue—and offers a standard environment to measure and train it.

What carries the argument

The load-bearing machinery is UserBench's simulated-user protocol. A GPT-4o-driven oracle user holds a private list of preferences across up to five travel aspects (flight, hotel, apartment, car rental, restaurant), each expressed through a bank of curated indirect statements. Agents act through three tool calls: search, which queries the option database; action, which talks to the user and whose utterance is classified into four intent types, with only concrete preference-targeted questions drawing out a preference; and answer, which submits option IDs. The option database mixes best, correct, wrong, and noise choices, and the reward scheme pays 1.0 for the cheapest preference-satisfying best option, 0.8 for another correct option, and partial credit for valid searches and preference-eliciting actions. This machinery turns 'did the agent finish the task' into 'did the agent find out what the user wanted and act on it'.

What would settle it

Run the same 471 test scenarios with human users playing the client and compare preference-elicitation rates and final alignment scores; if the human numbers diverge materially from the GPT-4o-simulated ones, or if outside raters frequently disagree with the environment's four-way intent labeling of agent utterances, UserBench's conclusions about user alignment would not transfer to real interaction.

Watch

Extended reading notes

Core claim

The paper claims that when an agent must discover what a user actually wants through conversation, today's leading LLMs are substantially weaker than their tool-use scores suggest. On UserBench's travel-planning tasks, where the simulated user starts with an underspecified goal and discloses preferences implicitly and incrementally, no tested model produces answers aligned with all user intents more than about a quarter of the time; across all models the average full-alignment rate is 20%. Even the best model actively elicits fewer than 30% of the user's preferences, and valid clarifying questions are much rarer than valid database searches. The paper also reports that one-answer-per-aspect decisions drop scores by roughly 40% on average relative to multi-choice answering, and that allowing more turns or more sampling does not consistently improve alignment. These observations support the paper's central claim: current agents are reliable tool executors but brittle at the collaborative, proactive part of user-centered reasoning.

Load-bearing premise

The benchmark's validity hinges on the GPT-4o user simulator revealing and withholding preferences the way real people do in underspecified conversations.

Editorial extensions

If this is right

  • If UserBench captures user-centric ability, then standard tool-use benchmarks overstate agent readiness for real, under-specified requests.
  • The low active-elicitation rates imply that instruction-tuned models need explicit training or reward pressure to ask targeted clarifying questions rather than guess.
  • Since extra turns and extra sampling do not raise alignment, further gains must come from better dialogue planning or goal-tracking, not from longer rollouts.
  • The environment's 2,651 held-out training scenarios plus customizable rewards make it a plausible training ground for reinforcement-learning methods that reward each elicited preference and penalize premature answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would validate UserBench against real human users in the same travel scenarios; the paper does not present such a human study, so whether GPT-4o's preference-reveal timing matches human behavior remains an open empirical question.
  • The same scaffold could be ported to other under-specified domains such as code generation or document drafting, where the binding constraint may likewise be eliciting requirements rather than executing them.
  • The paper's active versus passive elicitation split suggests a sharper, currently unreported metric: preferences elicited per valid clarifying question, which would separate persistent question-askers from one-shot lucky askers.
  • One testable prediction is that agents trained in this environment with per-preference partial credit and a late-answer penalty will exceed the reported 20% full-alignment rate, since the paper's own ablations indicate that guessing, not comprehension, drives many correct answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces UserBench, a Gymnasium-based benchmark for evaluating LLM agents in multi-turn, preference-driven travel-planning interactions. Simulated users begin with underspecified goals and reveal implicit preferences incrementally, either in response to targeted agent questions or proactively after off-topic stretches. The authors contribute a scenario-generation pipeline (4K+ claimed scenarios), a customizable tool-use environment with correctness/noise/wrong options, and an evaluation of 11 open- and closed-source models under single-choice and multi-choice answer settings. The central empirical claim is that current agents achieve full alignment with all user intents only about 20% of the time and elicit fewer than 30% of user preferences through active interaction, alongside a reported 40% score drop when only one option per aspect is allowed. The paper also presents analyses of difficulty tiers, preference concentration, turn budgets, sampling, and distractor counts.

Significance. If the central claims were properly supported, UserBench would address a real gap: most agent benchmarks measure task completion rather than proactive user-intent understanding, and the environment's modular Gym interface, public code/data release, and configurable reward and search-noise parameters are genuine strengths. The difficulty-tier analyses, timing-weighted evaluation, and the observation that tool-use validity rates exceed preference-elicitation rates are useful empirical contributions that could guide future user-centric agent training. However, the headline quantitative claims are currently not verifiable from the reported metrics, and the evaluation's dependence on GPT-4o for scenario generation, user simulation, and intent classification raises construct-validity concerns that need explicit handling.

major comments (4)
  1. [Abstract; §4.1; Table 3]
  2. [§3.5; Table 2]
  3. [Abstract; §4.1; Table 3; Table 4; Appendix C]
  4. [§3.3; §4.1; Appendix B; Figures 8–10]
minor comments (5)
  1. [§3.3]
  2. [Figure 1; §3.5; §6]
  3. [Table 1]
  4. [§5; Figure 5; Figure 6]
  5. [Appendix B]

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: UserBench reports empirical measurements, and no reported metric reduces to its inputs by construction.

full rationale

The paper is an empirical benchmark study, not a derivation. Every reported quantity (score, Best Exist Rate, Correct Exist Rate, Valid Search/Action Attempt, Preference Elicited) is computed from recorded agent interactions against fixed ground-truth preference lists, rule-based option comparisons, and GPT-4o utterance classifications. None of these metrics is defined in terms of the quantity it is claimed to measure, and no fitted parameter is renamed as a prediction. The use of GPT-4o both as user simulator and as one of the tested models is a simulator-validity and self-consistency limitation, not a logical circularity, because the simulator's responses are driven by prompts and by the ground-truth preference inventory independent of the tested model's outputs. Self-citations (e.g., Qian et al. 2024b) appear only as related-work motivation and are not load-bearing for any measured result. The abstract's 20% 'fully align with all user intents' claim is not traceable to a defined per-scenario metric, and Section 3.5 says 417 test points while Table 2 reports 471; these are correctness and verifiability risks, not circularity. No equation-level circular step can be quoted from the paper.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The environment's simulated user is a procedural construction, but not a claimed entity. The benchmark rests on hand-set design choices (simulator prompts, elicitation intervals, option counts, reward weights) rather than fitted parameters.

free parameters (6)
  • Elicitation Interval = 3 turns
    Hand-set hyperparameter in Appendix C: the simulated user proactively reveals a preference after 3 consecutive off-topic turns. It directly controls how many preferences are passively disclosed and thus inflates the Preference Elicited metric.
  • Max Steps = 20
    Maximum number of interaction turns; caps how many preferences can be elicited and therefore affects all scores.
  • Search Failure Interval = 5
    Every 5th search returns a system error (Appendix C); this hand-set noise level changes effective search difficulty.
  • Wrong Choice Number = 10
    Number of distractor options per search result; a design choice that influences selection difficulty.
  • Noise Choice Number = 5
    Number of unrealistic or irrelevant options per search result; affects distractor load.
  • Reward weights (best/correct) = 1.0/0.8
    Scoring weights chosen by authors; different weights change score magnitudes and the perceived size of the alignment gap.
assumptions (3)
  • domain assumption The GPT-4o user simulator behaves like a real human user when revealing and withholding preferences.
    All preference elicitation and alignment scores are measured against this simulator; no human validation or comparison is reported. See Section 3.3 and Figures 8 to 10.
  • domain assumption The GPT-4o intent classifier reliably maps agent utterances to the four types used for action validity.
    The 'Valid Action Attempt' and 'Active Preference Elicited' metrics depend on this classifier's type 1 judgments. Figure 8.
  • domain assumption Curated implicit expressions correspond one-to-one with preferences and with option properties.
    The data pipeline claims validation of implicitness and clarity (Section 3.1), but no human agreement numbers are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UserBench: An Interactive Gym Environment for User-Centric Agents." pith.science (2026). https://pith.science/paper/G6USPTWD

@misc{pith2026250722034,
  author       = {Pith},
  title        = {Pith review of: UserBench: An Interactive Gym Environment for User-Centric Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6USPTWD}},
  note         = {Machine review of arXiv:2507.22034}
}
read the original abstract

Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, enabling them to solve complex tasks. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored. To address this gap, we introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions. UserBench features simulated users who start with underspecified goals and reveal preferences incrementally, requiring agents to proactively clarify intent and make grounded decisions with tools. Our evaluation of leading open- and closed-source LLMs reveals a significant disconnect between task completion and user alignment. For instance, models provide answers that fully align with all user intents only 20% of the time on average, and even the most advanced models uncover fewer than 30% of all user preferences through active interaction. These results highlight the challenges of building agents that are not just capable task executors, but true collaborative partners. UserBench offers an interactive environment to measure and advance this critical capability.

Figures

Figures reproduced from arXiv: 2507.22034 by the authors.

Figure 1
Figure 1. The pipeline of UserBench Construction, including data gathering, preference curation, tool augmentation, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The score distribution of tested models across [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. When aspect number is settled, more user [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: When total user preferences are settled, less [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Increasing the number of turns allowed in in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Increasing the number of sampling time raises the maximum score but average score shows little change or even slightly drops. heuristics or early guesses rather than deeply engag￾ing with user intent. While this may occasionally yield fast correct answers, it more ofte…
Figure 7
Figure 7. Figure 7: System and user prompt for evaluating whether the agent’s search request aligns with the ground truth [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: System and user prompt for evaluating the agent’s utterance type and identifying preference-related [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: System and user prompt for responding to agent requests by implicitly revealing the user’s preference. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: System and user prompt for proactively and implicitly introducing a user preference into the conversation. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: System and user prompt for generating neutral and natural user responses when no preferences are [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: System prompt used to instruct the UserBench agent (single-choice setting). [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: System prompt used to instruct the agent (multi-choice setting). [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Initial user message provided in the UserBench simulation. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Good example showing successful hotel planning with clear preference elicitation, relevant search, and [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Good example showing Claude-4-Sonnet’s ability to elicit implicit preferences, adapt follow-up questions, [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Negative example showing Deepseek-V3’s failure to elicit meaningful preferences or recover from [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good h...

  2. SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    SENTINEL generates targeted tasks from model failures in a Controller-Proposer-Solver loop, raising Pass^1 from 66.4 to 74.9 on Tau2-Bench Retail and outperforming standard RL.

  3. CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    CRAB-Bench and RUSE create a new evaluation framework for LLM agents on constraint-graph tasks with realistic human-like user behaviors, reporting 61% pass@1 for the best model and up to 57% further drops under RUSE.

  4. ProactBench: Beyond What The User Asked For

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    ProactBench measures LLM conversational proactivity in three phases using 198 multi-agent dialogues and finds recovery behavior hard to predict from existing benchmarks.

  5. Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation

    cs.CL 2026-04 unverdicted novelty 7.0 of 10

    Multi-turn simulations of grounded support narratives show LLMs reduce teaching-oriented responses as estimated user distress rises, with community context also shaping strategy choice.

  6. HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    HiL-Bench shows frontier AI agents fail to ask for help on incomplete tasks, recovering only a fraction of full-information performance, but RL training on Ask-F1 reward improves judgment and transfers across domains.

  7. PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

    cs.AI 2026-03 unverdicted novelty 7.0 of 10

    PERMA is a new benchmark using temporally ordered events, text variability, and linguistic alignment to evaluate LLM memory agents on persona consistency beyond simple retrieval.

  8. Mind the Sim2Real Gap in User Simulation for Agentic Tasks

    cs.AI 2026-03 conditional novelty 7.0 of 10

    On τ-bench, LLM user simulators are more cooperative, more verbose, and more lenient than real human users, so agent benchmarks that rely on them overstate real-world performance.

  9. Long-Horizon Embodied Decision-Making via Multimodal Memory Compression

    cs.CV 2026-08 conditional novelty 6.0 of 10

    DunphyBench tests long-horizon, preference-driven house selection in virtual homes; MeMento, a preference-conditioned memory compressor, raises VLM agent accuracy by 7.18% and cuts memory by 85.38%.

  10. Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A clarification-first 3D agent, trained by simulated multi-turn dialogue, reaches 60.4% and 43.3% success on single- and multi-step 3D tool tasks, more than doubling prior baselines.

  11. AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

    cs.AI 2026-07 conditional novelty 6.0 of 10

    AppWorld-UL perturbs AppWorld's autonomous tasks into 516 user-interaction-requiring tasks; SOTA agents succeed on only 48.6% of them.

  12. WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agents

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    WRIT is a synthesis pipeline that generates write-read intensive trajectories along axes of write-decision count and per-decision evidence burden, enabling a 4B model to outperform GPT-5.1 on τ²-bench with reduced inf...

  13. ComPASS: Towards Personalized Agentic Social Support via Tool-Augmented Companionship

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    ComPASS creates tool-augmented LLM agents for substantive social support, releases the first personalized benchmark ComPASS-Bench, and fine-tunes ComPASS-Qwen to outperform its base model while matching larger LLMs.

  14. Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    This survey categorizes agentic environments for LLMs by eight attributes and domains, introduces symbolic and neural synthesis paradigms with evaluation, and outlines four agent evolution pathways plus three environm...

  15. UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mind

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    UserHarness reframes ToM as explicit user-mind reconstruction and reports up to 95.94% macro accuracy on five benchmarks with over 15% relative gains.

  16. RecoWorld: Building Simulated Environments for Agentic Recommender Systems

    cs.IR 2025-09 conditional novelty 5.0 of 10

    A design proposal, not a tested system: a dual-view simulation loop in which an LLM-simulated user issues reflective instructions when about to disengage, and an instruction-following recommender adapts to maximize si...

Reference graph

Works this paper leans on

44 extracted references · 14 canonical work pages · cited by 16 Pith papers

  1. [1]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. tau2-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982

    arXiv 2025

  2. [2]

    Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. https://openreview.net/forum?id=qV83K9d5WB Large language models as tool makers . In The Twelfth International Conference on Learning Representations

    work page 2024

  3. [3]

    Maximillian Chen, Ruoxi Sun, Tomas Pfister, and Sercan O Arik. 2025. Learning to clarify: Multi-turn conversations with action-based contrastive self-training. In Proceedings of ICLR

    work page 2025

  4. [4]

    H.H. Clark. 1996. Using Language. ACLS Humanities E-Book. Cambridge University Press

    work page 1996

  5. [5]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    2024

  6. [6]

    Nicholas Farn and Richard Shin. 2023. Tooltalk: Evaluating tool-usage in a conversational setting. arXiv preprint arXiv:2311.10775

    arXiv 2023

  7. [7]

    Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. 2024. Aligning llm agents by learning latent preference from user edits. In Advances in Neural Information Processing Systems (NeurIPS)

    work page 2024

  8. [8]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

Show all 44 references
  1. [9]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  2. [10]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128

    2023 arXiv

  3. [11]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  4. [12]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  5. [13]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    2025 arXiv

  6. [14]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102--3116

    2023

  7. [15]

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383

    2025 arXiv

  8. [16]

    Bill Yuchen Lin, Yuntian Deng, Khyathi R Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770

    2024 arXiv

  9. [17]

    Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2023. We're afraid language models aren't modeling ambiguity. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  10. [18]

    Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. 2024. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:...

    2024

  11. [19]

    Jiarui Lu and et al. 2024. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682

    2024 arXiv

  12. [20]

    Yaxi Lu, Shenzhi Yang, Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, et al. 2025. Proactive agent: Shifting llm agents from reactive responses to active assistance. In The Thirteenth International Conference on Learning Representations

    2025

  13. [21]

    Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. 2025. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03601

    2025 arXiv

  14. [22]

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. 2025 a . Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958

    2025 arXiv

  15. [23]

    Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. 2025 b . Smart: Self-aware agent for tool overuse mitigation. arXiv preprint arXiv:2502.11435

    2025 arXiv

  16. [24]

    Cheng Qian, Hongyi Du, Hongru Wang, Xiusi Chen, Yuji Zhang, Avirup Sil, Chengxiang Zhai, Kathleen McKeown, and Heng Ji. 2025 c . Modelingagent: Bridging llms and mathematical modeling for real-world challenges. arXiv preprint arXiv:2505.15068

    2025 arXiv

  17. [25]

    Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6922--6939

    2023

  18. [26]

    Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, et al. 2024 a . Escapebench: Pushing language models to think outside the box. arXiv preprint arXiv:2412.13549

    2024 arXiv

  19. [27]

    Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, et al. 2024 b . Tell me more! towards implicit user intention understanding of language model driven agents. arXiv preprint arXiv:2402.09205

    2024 arXiv

  20. [28]

    Cheng Qian, Shihao Liang, Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, and Maosong Sun. 2024 c . Investigate-consolidate-exploit: A general strategy for inter-task agent self-evolution. arXiv preprint arXiv:2401.13996

    2024 arXiv

  21. [29]

    Cheng Qian, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2024 d . Toolink: Linking toolkit creation and using through chain-of-solving on open-source model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2024

  22. [30]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  23. [31]

    Harmanpreet Singh, Nikhil Verma, Yixiao Wang, Manasa Bharadwaj, Homa Fashandi, Kevin Ferreira, and Chul Lee. 2024. Personal large language model agents: A case study on tailored travel planning. In Proceedings of EMNLP (Industry Track)

    2024

  24. [32]

    Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. 2025. Acting less is reasoning more! teaching model to act efficiently. arXiv preprint arXiv:2504.14870

    2025 arXiv

  25. [33]

    Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024 a . A user-centric multi-intent benchmark for evaluating large language models. In Proceedings of EMNLP

    2024

  26. [34]

    Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024 b . Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In Proceedings of ICLR

    2024

  27. [35]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122

    2024 arXiv

  28. [36]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045

    2024 arXiv

  29. [37]

    Fung, Hao Peng, and Heng Ji

    Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. 2024. Craft: Customizing llms by creating and retrieving from specialized toolsets. In Proc. The Twelfth International Conference on Learning Representations (ICLR2024)

    2024

  30. [38]

    Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. 2024 a . xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215

    2024 arXiv

  31. [39]

    Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, et al. 2024 b . Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint arXiv:2408.07060

    2024 arXiv

  32. [40]

    Michael JQ Zhang, WB Knox, and Eunsol Choi. 2024 c . Modeling future conversation turns to teach llms to ask clarifying questions. arXiv preprint arXiv:2410.13788

    2024 arXiv

  33. [41]

    Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. 2025. Do llms recognize your preferences? evaluating personalized preference following in llms. In Proceedings of ICLR

    2025

  34. [42]

    Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. 2025. Multiagentbench: Evaluating the collaboration and competition of llm agents. arXiv preprint arXiv:2503.01935

    2025 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.