REVIEW 20 cited by
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox
Forward citations
Cited by 20 Pith papers
-
Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools
Canary tools turn a binary wrong-tool error into a typed diagnosis of the reasoning weakness, and a six-type taxonomy separates models by capability.
-
Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills
Graph of Skills retrieves dependency-aware skill bundles from large libraries via offline graph construction and hybrid inference-time search, improving agent rewards by 43.6% while cutting tokens by 37.8% on benchmarks.
-
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
On τ-bench, LLM user simulators are more cooperative, more verbose, and more lenient than real human users, so agent benchmarks that rely on them overstate real-world performance.
-
AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use
AppWorld-UL perturbs AppWorld's autonomous tasks into 516 user-interaction-requiring tasks; SOTA agents succeed on only 48.6% of them.
-
The A-R Behavioral Space: Execution-Level Profiling of Tool-Using Language Model Agents in Organizational Deployment
Execution and refusal in tool-using LLMs are separable dimensions whose joint distribution shifts systematically across risk regimes and autonomy scaffolds, made visible in A-R space.
-
Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls
A simulated tool environment with argument validation, response synthesis, and task-state feedback improves LLM tool-call accuracy at test time.
-
One Tool Is Enough: Reinforcement Learning for Repository-Level LLM Agents
Repository-level issue localization can be done by a single jump-to-definition tool trained with reinforcement learning, achieving strong results on SWE-bench despite using only open-weights models.
-
Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic-Procedural Memory
An LLM agent that builds a tool-transition graph with state summaries from past experience improves tool selection and RL exploration by large margins on multi-turn benchmarks.
-
UserBench: An Interactive Gym Environment for User-Centric Agents
A new multi-turn agent benchmark shows that current LLMs elicit fewer than 30% of user preferences and reach full intent alignment only about 20% of the time.
-
ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution
ASPERA generates a benchmark of 250 executable assistant tasks and finds that LLMs, even with full API documentation, solve only 10 to 80 percent of them.
-
Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems
LLM tool agents fail at parameter filling in five recurring ways; perturbing tool documents and user queries drives most failures, and invented parameter names are tied to the model rather than the input.
-
Quo Vadis, World Modeling?
An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.
-
How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $\tau$-bench
IRMA reformulates tool-agent inputs with memory, domain constraints, and tool suggestions, and reports improved tau-bench pass^5 reliability over ReAct, function calling, and self-reflection.
-
Agent Identity Evals: Measuring Agentic Identity
Introduces Agent Identity Evals (AIE), five similarity-based metrics for LMA identity stability, with pilot experiments showing identifiability always at zero and no statistical support.
-
Apple Intelligence Foundation Language Models: Tech Report 2025
Apple's 3B on-device and larger server language models match or beat several similarly sized open models on MMLU, MMMLU, and MGSM, using new efficiency techniques like KV-cache sharing and 2-bit quantization.
-
Teaching a Language Model to Speak the Language of Tools
LoRA fine-tuning of BgGPT models on a bilingual Bulgarian function-calling dataset yields large gains on a self-built 120-case benchmark while keeping knowledge benchmarks stable.
-
Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities
A survey that groups graph-empowered AI agent research into planning, execution, memory, and multi-agent coordination, plus agents-for-graphs and applications.
-
CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions
A new 109-conversation, 86-API benchmark for LLM function-calling in multi-turn dialogue shows top models at about 40% accuracy and near-zero performance on chains of 4+ calls.
-
PyTOD: Programmable Task-Oriented Dialogue with Execution Feedback
A solvable one-state model of a dynamic molecular switch is claimed to combine synapse-like switching with proven convergence and fading memory for stable neuromorphic computation.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
Discussion (0). Sign in to comment.