REVIEW 6 cited by
On the Tool Manipulation Capability of Open-source Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent studies on software tool manipulation with large language models (LLMs) mostly rely on closed model APIs. The industrial adoption of these models is substantially constrained due to the security and robustness risks in exposing information to closed LLM API services. In this paper, we ask can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation, with practical amount of human supervision. By analyzing common tool manipulation failures, we first demonstrate that open-source LLMs may require training with usage examples, in-context demonstration and generation style regulation to resolve failures. These insights motivate us to revisit classical methods in LLM literature, and demonstrate that we can adapt them as model alignment with programmatic data generation, system prompts and in-context demonstration retrievers to enhance open-source LLMs for tool manipulation. To evaluate these techniques, we create the ToolBench, a tool manipulation benchmark consisting of diverse software tools for real-world tasks. We demonstrate that our techniques can boost leading open-source LLMs by up to 90% success rate, showing capabilities competitive to OpenAI GPT-4 in 4 out of 8 ToolBench tasks. We show that such enhancement typically requires about one developer day to curate data for each tool, rendering a recipe with practical amount of human supervision.
Forward citations
Cited by 6 Pith papers
-
DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.
-
SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
SciAgent-8B, fine-tuned on trajectories synthesized from a tool dependency graph, outperforms Qwen3-VL-235B-Instruct on SciAgentBench, a new 259-task benchmark for multi-step scientific tool-use.
-
MassTool: A Multi-Task Search-Based Tool Retrieval Framework for Large Language Models
A multi-task retriever that combines tool-usage detection with query-centered graph and search-based modules improves tool retrieval accuracy over prior baselines.
-
A Formal Hierarchical Architecture for Agentic Orchestration with Stack-Based Execution and Lazy Discovery
A rooted skill tree with LIFO stack execution and lazy child-manifest discovery bounds per-step schema tokens independent of global tool count N while supporting nested resumption and branch isolation.
-
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
CRITICTOOL, a benchmark of 2,740 tool-calling error scenarios built from BFCL and T-Eval with GPT-4o-based error injection, finds most LLMs rarely recover from tool-use errors, with GPT-4o best at 69.01 overall and to...
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
Discussion (0). Sign in to comment.