Pith. sign in

REVIEW 5 cited by

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.14033 v3 pith:CH7J2YGB submitted 2023-12-21 cs.CL

Zehui Chen , Weihua Du , Wenwei Zhang , Kuikun Liu , Jiangning Liu , Miao Zheng , Jingming Zhuo , Songyang Zhang
show 3 more authors
Dahua Lin Kai Chen Feng Zhao
This is my paper
classification cs.CL
keywords t-evalllmssteptoolutilizationcapabilityevaluateevaluation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLM) have achieved remarkable performance on various NLP tasks and are augmented by tools for broader applications. Yet, how to evaluate and analyze the tool-utilization capability of LLMs is still under-explored. In contrast to previous works that evaluate models holistically, we comprehensively decompose the tool utilization into multiple sub-processes, including instruction following, planning, reasoning, retrieval, understanding, and review. Based on that, we further introduce T-Eval to evaluate the tool utilization capability step by step. T-Eval disentangles the tool utilization evaluation into several sub-domains along model capabilities, facilitating the inner understanding of both holistic and isolated competency of LLMs. We conduct extensive experiments on T-Eval and in-depth analysis of various LLMs. T-Eval not only exhibits consistency with the outcome-oriented evaluation but also provides a more fine-grained analysis of the capabilities of LLMs, providing a new perspective in LLM evaluation on tool-utilization ability. The benchmark will be available at https://github.com/open-compass/T-Eval.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.

  2. WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

    cs.CL 2026-07 conditional novelty 6.0 of 10

    WorkSurface-Bench measures surface routing separately from answer correctness and finds near-perfect routing still leaves 25–44% answer errors across four LLM backbones.

  3. Qwen-Audio-VAE Technical Report

    eess.AS 2026-07 conditional novelty 5.0 of 10

    A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.

  4. Teaching a Language Model to Speak the Language of Tools

    cs.IR 2025-06 conditional novelty 5.0 of 10

    LoRA fine-tuning of BgGPT models on a bilingual Bulgarian function-calling dataset yields large gains on a self-built 120-case benchmark while keeping knowledge benchmarks stable.

  5. Evaluation and Benchmarking of LLM Agents: A Survey

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A review that proposes a two-dimensional taxonomy for evaluating LLM agents and highlights enterprise-specific evaluation gaps.

Pith tools