REVIEW 15 cited by
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.
Forward citations
Cited by 15 Pith papers
-
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories
Offline DPO on expert states with one-step student negatives and schema-preserving augmentation improves LLM agents beyond SFT and can match online GRPO without environment interaction.
-
Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Citation-faithfulness metrics for AI science agents are verifier-dependent (3–18% on identical outputs), and a split-conformal guard provides a finite-sample catch-rate guarantee anchored on human gold.
-
FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
FinToolBench couples 760 executable financial APIs with 295 tool-required queries and scores agents on execution success plus timeliness, intent, and domain compliance, with a finance-aware retrieval baseline (FATR).
-
Efficient Multi-round LLM Inference over Disaggregated Serving
AMPD adaptively routes incremental prefill tasks between prefill and decode workers and reorders queued prefill jobs to improve SLO attainment for multi-round LLM inference under prefill-decode disaggregation.
-
MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
MUA-RL adds an LLM-simulated user to the RL rollout loop for multi-turn tool use, improving small Qwen3 models on TAU2, BFCL-V3 Multi Turn, and ACEBench Agent.
-
Self-Challenging Language Model Agents
A language model agent can generate its own verifiable training tasks and improve its tool-use success rate by about 2x without human-annotated data.
-
Enhancing Tool Learning in Large Language Models with Hierarchical Error Checklists
HiTEC improves LLM tool calling by embedding hierarchical error checklists in prompts or using them to generate negative examples for KTO fine-tuning.
-
Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
Fine-tuned small language models can reach high accuracy on function calling, but zero-shot and few-shot performance is poor, and the study's few-shot results are compromised by using test-set examples in the prompt.
-
When2Call: When (not) to Call Tools
When2Call measures when language models should call tools versus ask questions or refuse, and shows that RPO training substantially improves this decision-making.
-
CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
CallNavi is a new benchmark for LLM function calling with unfiltered, nested, multi-step API tasks; GPT-4o leads the leaderboard and a two-step routing pipeline improves fine-tuned models.
-
Reducing Tool Hallucination via Reliability Alignment
A reliability alignment framework, Relign, that expands the LLM tool-use action space with indecisive actions reduces tool hallucination rates and improves task success on the new RelyToolBench benchmark.
-
Execution-First Synthetic Tool-Use Trace Generation for LLM Agents
An execution-first data generator that validates tool traces before synthesizing user tasks improves Qwen agents' tool use and answer quality, with masked reasoning supervision beating full-trace supervision.
-
CoDec: Prefix-Shared Decoding Kernel for LLMs
CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.
-
Structured Agent Distillation for Large Language Model
Structured Agent Distillation splits teacher trajectories into reasoning and action spans for span-specific distillation, yet the stated loss reduces to masked token-level KL and the reported gains are confounded by b...
-
Toward Efficient Agents: Memory, Tool learning, and Planning
A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.
Discussion (0). Continue with ORCID to comment.