Pith. sign in

REVIEW 5 major objections 6 minor 81 references

Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tool-MVR: verified data plus reflection learning lets a 7B model outperform GPT-4 on tool-use benchmarks.

desk verdict A useful data-quality and reflection pipeline for tool learning, but the headline numbers depend on an unstated train/test split that must be confirmed before the results can be trusted. read the letter →

arxiv 2506.04625 v1 pith:3UL255G6 submitted 2025-06-05 cs.CL

Zhiyuan Ma , Jiayu Liu , Xianzhen Luo , Zhenya Huang , Qingfu Zhu , Wanxiang Che This is my paper
classification cs.CL
keywords toollearninglargelanguagemodelsinstructiondataverificationmeta-verificationreflectionAPIcallingsystem2reasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a 7B open-source language model can be made to out-plan, out-invoke, and out-reflect GPT-4 in tool-use tasks by fixing the training corpus and teaching error correction, rather than by scaling up. The authors identify two root causes of weak tool agents: instruction datasets full of unsolvable queries and hallucinated API calls, and static imitation learning that never trains on mistakes. They address the first with Multi-Agent Meta-Verification (MAMV), a GPT-4-driven pipeline that validates APIs, queries, and API-call trajectories and turns ToolBench into the higher-quality ToolBench-V. They address the second with Exploration-based Reflection Learning (EXPLORE), which generates ToolBench-R from failed steps and teaches an $\text{Error}\rightarrow\text{Reflection}\rightarrow\text{Correction}$ loop. On StableToolBench the resulting Tool-MVR reaches an 83.8% pass rate, 15.3 points above GPT-4, and on the new RefineToolBench it corrects 58.9% of errors versus 9.1% for ToolLLM.

What carries the argument

The load-bearing machinery is the two-stage training pipeline: MAMV (Multi-Agent Meta-Verification) is a GPT-4-powered pipeline with three specialized agents: APIOptAgent for API validation, simulation, and documentation refinement, QueryVerifyAgent for feasibility and quality scoring, and APICallAgent for trajectory construction with Format, Semantic, and Execution checkers. EXPLORE (Exploration-based Reflection Learning) is the second mechanism: it samples step-level wrong actions from verified trajectories, feeds the tool's error observation to GPT-4 to generate a structured reflection, and stores the tuple (wrong action, feedback, reflection, corrected action) as a training instance. The training objective is the sum of an action-prediction loss $\mathcal{L}_V$ over ToolBench-V and a reflection-plus-correction loss $\mathcal{L}_R$ over ToolBench-R, combined as $\mathcal{L} = \mathcal{L}_V + \lambda \mathcal{L}_R$ with $\lambda=1$ and a 10:1 data ratio, so the model learns deliberate planning, precise invocation, and feedback-driven correction in one supervised pass.

What would settle it

Download the released ToolBench-V and ToolBench-R data, match their query and trajectory pairs against the 765 StableToolBench tasks and the 178 I3 cases of RefineToolBench, then retrain with any overlapping items removed; if the 83.8% pass rate and 58.9% correction rate collapse toward the roughly 47-60% range of the no-verification baselines, the reported gains are a test-set artifact rather than a learned skill.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the quality bottleneck in tool-augmented LLMs is not model scale but the verification of the three meta-components of tool use: the API pool, the user query, and the execution trajectory. MAMV upgrades ToolBench by testing 11,302 APIs against real calls, simulating the invalid ones, filtering 62,147 generated queries down to 23,903 solvable ones, and re-annotating trajectories with step-level format, semantic, and execution checks, raising query validity from 52.7% to 98.8% and trajectory accuracy from 25.6% to 81.3%. EXPLORE then samples wrong actions from the verified trajectories, lets GPT-4 turn tool feedback into structured reflections, and yields 3,625 instances where the model learns to recognize and correct its own mistakes. Trained jointly on the 11,765 verified instances and the reflection set, Tool-MVR (Qwen-2.5-7B) attains an 83.8% average pass rate on StableToolBench, 15.3 points above GPT-4, while using 31.4% fewer API calls, and a 58.9% error-correction rate on RefineToolBench against GPT-4's 51.1% and ToolLLM's 9.1%.

Load-bearing premise

The headline results assume ToolBench-V and ToolBench-R were built only from the ToolBench training split, so no StableToolBench test query or trajectory, and no RefineToolBench I3 case derived from one, leaked into the fine-tuning set; the paper never states this split.

Editorial extensions

If this is right

  • A 7B or 8B open-weight model can exceed a frontier closed model on tool-use benchmarks if its training data is verified and its reflection ability is trained directly, which weakens the assumption that tool mastery requires proprietary scale.
  • Instruction quality dominates quantity: Tool-MVR uses 15,390 examples versus ToolLLM's 73,423, so dataset cleaning is a cheaper path to improvement than collecting more data.
  • Teaching error correction from execution feedback generalizes to unseen tools and categories, as shown by the highest pass rate (86.7%) on the G1-Tool scenario.
  • Fewer API calls per task (3.48-3.99 versus 5.19-5.92 for GPT-4 in the reported runs) means lower latency and cost for deployed tool agents.
  • A dedicated reflection benchmark can separate the ability to recognize an error from the ability to fix it, and the reported gap (58.9% versus 9.1% correction) suggests reflection is a trainable skill rather than an emergent one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the train/test split is clean, the method's logic transfers beyond tool use: any domain where instruction corpora are synthetically generated, such as code repair, retrieval-augmented generation, or planning, could be upgraded by the same verify-then-reflect recipe with domain-specific checkers.
  • The paper leaves implicit that its evaluation judge is itself a GPT-4 model, the same family that generated the verified trajectories; a fair reader should test whether the reported wins survive a judge-agnostic or open-weight evaluator.
  • A natural extension is to mine errors at inference time rather than only at data-construction time, turning EXPLORE into an online loop that keeps expanding ToolBench-R as the deployed model encounters new API failures.
  • The I3 portion of RefineToolBench is built from StableToolBench trajectories, so a stricter test would construct reflection cases from tools and queries never seen in training; the I1/I2 single-tool numbers (53-54% error-correction rate) may be the more honest estimate of reflection in novel settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Tool-MVR, a tool-augmented LLM built by fine-tuning open-source backbones (Qwen-2.5-7B and LLaMA-3.1-8B) on two newly constructed datasets. The first, ToolBench-V, is produced by a Multi-Agent Meta-Verification (MAMV) pipeline that validates APIs, filters user queries, and verifies API-call trajectories. The second, ToolBench-R, is produced by an Exploration-based Reflection Learning (EXPLORE) algorithm that samples error steps, generates reflections, and constructs an "error -> reflection -> correction" training signal. The authors report state-of-the-art results on StableToolBench (83.8% average Pass Rate for the Qwen-based model, exceeding GPT-4 by 15.3%) and introduce a new benchmark, RefineToolBench, on which their model achieves a 58.9% average Error Correction Rate. They also report a 31.4% reduction in average API calls per task compared to GPT-4.

Significance. The work addresses an important problem: instruction data quality for tool-augmented LLMs and the largely missing capability of error recovery through reflection. The proposed pipeline, combining meta-verification with exploration-based reflection learning, is a plausible and potentially valuable design, and the reported results, if they hold under clean data hygiene and controlled evaluation, would be notable: a 7B open-source model outperforming GPT-4 on a tool-use benchmark while using fewer API calls. The release of code and data is a concrete strength that makes the claims checkable. The proposed RefineToolBench could also be a useful resource for the community. However, the significance is conditional on resolving the data-provenance and evaluation-bias concerns raised below, because the headline gains depend crucially on a clean train/test split and on evaluation procedures that do not systematically favor the proposed model.

major comments (5)
  1. [Section 3.2.2 and Section 4.1.1] The manuscript never states whether the 62,147 initial queries from which ToolBench-V was filtered originate exclusively from ToolBench's training split. Because StableToolBench (Section 4.1.1) is a curated subset of ToolBench, any overlap between the queries used for ToolBench-V/ToolBench-R and the 765 StableToolBench evaluation tasks would make the 'unseen' results in Tables 2 and 3, including the headline 83.8% Pass Rate, an in-distribution measurement. Please state the precise data split used, and provide an overlap analysis (by query identifier or exact text) between ToolBench-V, ToolBench-R, and the StableToolBench test set using the released data.
  2. [Section 4.1.2 and Section 3.3.2] RefineToolBench I3 cases are built by running the Stage 1 multi-agent system on StableToolBench tasks and injecting errors, while ToolBench-R is constructed through EXPLORE using ToolBench-V trajectories as references. The paper does not establish that the I3 cases are disjoint from ToolBench-R or ToolBench-V; indeed, Section 4.6.4 states that I3 cases 'contain familiar multi-tool tasks,' which suggests distributional overlap. If any I3 cases or their underlying trajectories appear in the training data, the 79.8% I3 Error Correction Rate is partially an in-distribution number. The authors should clarify the disjointness and, ideally, evaluate on error types not represented in the training distribution.
  3. [Section 4.4 and Table 1] The quality scores in Table 1 are obtained with GPT-4 after the same GPT-4-based MAMV pipeline has filtered and refined the data. This makes the reported improvements in query validity (52.7% to 98.8%) and trajectory accuracy (25.6% to 81.3%) partly self-confirming, because the evaluator and the filtering/refinement mechanism share the same model family and evaluation criteria. Please provide human-annotated quality judgments on a random sample, or use an independent verifier, to confirm that the quality gains are not an artifact of evaluator-model bias.
  4. [Section 4.2, Section 4.5 and Section 4.7] The paper states that all baselines are combined with DFSDT for inference, but it does not state whether Tool-MVR also uses DFSDT. If Tool-MVR is evaluated with greedy decoding while the baselines use a search procedure, the Pass Rate comparisons in Tables 2 and 3 and the API-call efficiency comparison in Figure 4 are not controlled for inference strategy, and the 'reducing API calls by 31.4%' claim would be expected from the absence of search rather than from the learned model alone. Please specify the exact inference protocol for Tool-MVR and, if it differs from the baselines, include a controlled comparison using the same search strategy for all models.
  5. [Section 4.1 and Appendix A] All reported evaluation metrics (Pass Rate, Win Rate, ERR, ECR) are computed by gpt-4-turbo with custom prompts, and the same model family was used to generate the training trajectories and reflections in ToolBench-V and ToolBench-R. Although the same evaluator is applied to every model, this setup can still favor Tool-MVR if the evaluator's format preferences align with the training data distribution. Please report agreement with human evaluation on a subset of these judgments to establish that the metric is not systematically biased toward the proposed model.
minor comments (6)
  1. [Table 2] Several rows in Table 2 are typeset without separators (e.g., '79.179.279.086.9'), making the values difficult to parse; please reformat the table.
  2. [Section 4.5.3] The ablation discussion says 'the substantially lower performance of w/o Stage 1&2 (L) (47.3%)', but Table 2 shows 47.3% is the average for w/o Stage 1&2 (Q), while the L row averages 60.1%; the model label appears to be incorrect.
  3. [Section 3.2.1] The API pool statistics (11,302 original APIs, 6,770 validated, 6,334 enhanced, and 4,500+ simulated) need a clarifying set-relationship statement; as written, these totals do not form a partition and the overlap is ambiguous.
  4. [Section 3.2.2] The text mentions 'approximately 80,000 initial queries' and then an 'initial pool of 62,147' without explaining the relationship; please clarify whether the latter is after an initial API/format filter.
  5. [Section 4.3] Eq. (8) contains a single loss weight λ, but the text says the authors 'set λ=1 and maintain a 10:1 ratio between ToolBench-V and ToolBench-R'; please clarify whether the ratio is implemented via data sampling or via separate weighting, since it is not visible in Eq. (8).
  6. [References] Reference [8] is a survey on spectral clustering and appears unrelated to tool-augmented LLMs; please check the citation mapping.

Circularity Check

2 steps flagged · score 5.0 of 10

Data-quality and reflection metrics are partly self-confirming, but the headline StableToolBench result rests on an independent external benchmark.

  1. fitted input called prediction [Section 3.2.2 (QueryVerifyAgent) and Section 4.4 / Table 1]
    "The Quality Assessor then evaluates remaining queries across three dimensions: semantic clarity, information completeness, and reasoning complexity, using a 1-10 scoring scale. We retain only high-quality queries (scores 8-10) to ensure query solvability, complexity, and completeness. ... for user queries, MAMV increases the overall quality score from 52.7% to 98.8%"

    The 98.8% overall quality is not an independent measurement: the same GPT-4-based Quality Assessor that filtered ToolBench-V by retaining only queries it scored 8-10 is then used, under the Quality Matters protocol, to score the filtered dataset. The filter and the evaluator share the same model family and overlapping criteria (solvability, completeness, semantic clarity), so the reported gain from 52.7% to 98.8% is largely the filter's own label distribution rather than an externally demonstrated improvement. The same mechanism applies to the API-call trajectory score: APICallAgent's GPT-4 Semantic and Execution Checkers pre-filter the trajectories that Table 1 then certifies as 81.3% accurate.

  2. fitted input called prediction [Section 4.1.2 (RefineToolBench construction) vs. Sections 3.3.1-3.3.2 (EXPLORE and ToolBench-R)]
    "For multi-tool scenarios (I3), we first obtain complete reasoning trajectories from StableToolBench using our Stage 1 multi-agent system, then use GPT-4 to introduce errors in tool selection and parameter content at randomly selected steps. ... This pattern aligns with the training data distribution, as I3 cases from StableToolBench contain familiar multi-tool tasks"

    The I3 subset of the authors' own RefineToolBench is produced by the same Stage 1 APICallAgent that generated the verified trajectories used as references for EXPLORE and ToolBench-R, with GPT-4 injecting errors in both the training reflections and the I3 test cases. The paper explicitly concedes that I3 contains familiar multi-tool tasks, so the headline 79.8% I3 error-correction rate is an in-distribution measurement of the training pipeline rather than an independent assessment of reflection generalization. Because the reported 58.9% average error-correction rate includes this self-constructed I3 subset, the aggregate reflection claim is partly self-confirming; I1 and I2 provide stronger independent signal but are a minority of cases.

full rationale

The paper does not rely on a load-bearing self-citation chain: citations to the authors' own prior work are contextual, and no uniqueness theorem is imported from their own papers. The strongest external anchor is StableToolBench, an independently constructed benchmark from prior work, which keeps the central claim that Tool-MVR outperforms GPT-4 from being circular. However, two claimed results are partly self-confirming. First, the Table 1 data-quality gains are measured with the same GPT-4 model family that filtered and verified ToolBench-V, so the jump from 52.7% to 98.8% query quality and from 25.6% to 81.3% trajectory accuracy is substantially a property of the filter rather than an external validation. Second, RefineToolBench's I3 subset is built from the authors' own Stage 1 system with GPT-4 error injection, and the paper itself states that these tasks are familiar from training, making the 79.8% I3 error-correction rate an in-distribution consistency check. The additional risk that StableToolBench test queries leaked into ToolBench-V because the paper never states whether the 62,147-query pool was restricted to ToolBench's training split is a data-hygiene and correctness issue, not a circularity of the derivation; as written, it cannot be adjudicated here. Overall, the circularity is partial and confined to the quality and reflection sub-claims, while the principal comparison against StableToolBench retains independent empirical content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim's empirical support rests on GPT-4 as data generator, filter, and judge, and on a train/test split that is never explicitly defined. The hand-chosen hyperparameters are peripheral, while the reliance on GPT-4-based verification and evaluation is the main unstated dependency.

free parameters (3)
  • lambda (loss weight) = 1
    Hand-chosen in Eq. 8 to balance ToolBench-V and ToolBench-R losses; no validation sweep is shown.
  • ToolBench-V to ToolBench-R data ratio = 10:1
    Implementation choice in Section 4.3; no ablation is presented for this ratio.
  • query quality retention threshold = 8-10
    Only queries with GPT-4-assigned quality scores between 8 and 10 are kept (Appendix B.2); the threshold is chosen by hand.
assumptions (4)
  • domain assumption GPT-4-based validators, query assessors, and trajectory checkers correctly identify solvable queries and accurate API calls.
    The entire MAMV pipeline in Section 3.2 relies on GPT-4 judgments without human verification or an independent validation set.
  • domain assumption Custom gpt-4-turbo prompt templates (Tables 5-7) yield a valid measure of pass rate, win rate, and error recovery.
    Section 4.1 and Appendix A introduce new evaluation prompts instead of using official StableToolBench scripts, so metric validity is assumed.
  • domain assumption Simulated API responses from the Simulator are semantically faithful replacements for real API outputs.
    Section 3.2.1 uses a GPT-4-based simulator for 4,500+ unavailable APIs; both training and evaluation depend on these synthetic responses.
  • domain assumption RefineToolBench error cases are solvable and representative of real tool-use failures.
    Section 4.1.2 constructs the benchmark using GPT-4 to insert errors; there is no human verification or external benchmark comparison.
invented entities (1)
  • RefineToolBench
    purpose: Benchmark for evaluating tool reflection, measuring error recognition and correction rates.
    Introduced by the authors and evaluated with GPT-4 prompts; no external validation or comparison to existing reflection benchmarks is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning." pith.science (2026). https://pith.science/paper/3UL255G6

@misc{pith2026250604625,
  author       = {Pith},
  title        = {Pith review of: Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UL255G6}},
  note         = {Machine review of arXiv:2506.04625}
}
read the original abstract

Empowering large language models (LLMs) with effective tool utilization capabilities is crucial for enabling AI agents to solve complex problems. However, current models face two major limitations: (1) unreliable tool planning and invocation due to low-quality instruction datasets (e.g., widespread hallucinated API calls), and (2) weak tool reflection abilities (over 90% of errors cannot be corrected) resulting from static imitation learning. To address these critical limitations, we propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations. Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories to construct ToolBench-V, a new high-quality instruction dataset that addresses the limitation of unreliable tool planning and invocation. Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback through a dynamic "Error -> Reflection -> Correction" learning paradigm, resulting in our reflection dataset ToolBench-R and addressing the critical weakness in tool reflection. Finally, we obtain Tool-MVR by finetuning open-source LLMs (e.g., Qwen-7B) on both ToolBench-V and ToolBench-R. Our experiments demonstrate that Tool-MVR achieves state-of-the-art performance on StableToolBench, surpassing both ToolLLM (by 23.9%) and GPT-4 (by 15.3%) while reducing API calls by 31.4%, with strong generalization capabilities across unseen tools and scenarios. Additionally, on our proposed RefineToolBench, the first benchmark specifically designed to evaluate tool reflection capabilities, Tool-MVR achieves a 58.9% error correction rate, significantly outperforming ToolLLM's 9.1%.

Figures

Figures reproduced from arXiv: 2506.04625 by the authors.

Figure 1
Figure 1. Tool learning examples: given user queries and APIs, models generate API calls for answers. The green trajectory shows [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of Multi-Agent Meta-Verification (MAMV) framework. MAMV creates a new high-quality tool instruction [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Exploration-based Reflection Learning (EXPLORE): [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average number of API calls required by different [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: This filtering process results in 11,776 queries from G1 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 5
Figure 5. Figure 5: Distribution of query quality scores across different groups (G1, G2, G3) in ToolBench. The yellow bars indicate [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 46 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Erkan Bas,ar, Xin Sun, Iris Hendrickx, Jan de Wit, Tibor Bosse, Gert-Jan De Bruijn, Jos A Bosch, and Emiel Krahmer. 2025. How Well Can Large Language Models KDD ’25, August 3–7, 2025, Toronto, ON, Canada. Ma et al. Reflect? A Human Evaluation of LLM-generated Reflections for Motivational Interviewing Dialogues. InProceedings of the 31st International Conf...

    2025

  3. [3]

    Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. 2024. Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview. net/forum?id=ZIpdu0cHYu

    work page 2024

  4. [4]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Pro- gram of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.Transactions on Machine Learning Research(2023)

    work page 2023

  5. [5]

    Zhi-Yuan Chen, Shiqi Shen, Guangyao Shen, Gong Zhi, Xu Chen, and Yankai Lin

    work page

  6. [6]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InInternational Conference on Learning Representations (ICLR)

    2024

  7. [7]

    Hao Ding, Ziwei Fan, Ingo Guehring, Gaurav Gupta, Wooseok Ha, Jun Huan, Linbo Liu, Behrooz Omidvar-Tehrani, Shiqi Wang, and Hao Zhou. 2024. Reasoning and planning with large language models in code development. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6480–6490

    work page 2024

  8. [8]

    Ling Ding, Chao Li, Di Jin, and Shifei Ding. 2024. Survey of spectral clustering based on graph theory.Pattern Recognition151 (2024), 110366. doi:10.1016/j. patcog.2024.110366

    arXiv 2024

Show all 81 references
  1. [9]

    Yu Du, Fangyun Wei, and Hongyang Zhang. [n. d.]. AnyTool: Self-Reflective, Hi- erarchical Agents for Large-Scale API Calls. InForty-first International Conference on Machine Learning

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    2024 arXiv

  3. [11]

    Jonathan St BT Evans. 2003. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences7, 10 (2003), 454–459

    2003

  4. [12]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ...

    2024

  5. [13]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning. PMLR, 10764–10799

    2023

  6. [14]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al . [n. d.]. CRITIC: Large Language Models Can Self-Correct with Tool- Interactive Critiquing. InThe Twelfth International Conference on Learning Repre- sentations

  7. [15]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2024. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. InThe Twelfth International Conference on Learning Representations

    2024

  8. [16]

    Zhicheng Guo, Sijie Cheng, Hao Wang, and et al. 2024. StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 11143–11...

    2024 doi

  9. [17]

    Ruixin Hong, Xinyu Pang, and Changshui Zhang. 2024. Advances in reasoning by prompting large language models: A survey.Cybernetics and Intelligence(2024)

    2024

  10. [18]

    Shadi Iskander, Sofia Tolmach, Ori Shapira, Nachshon Cohen, and Zohar Karnin

  11. [19]

    Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Jia Xu, Linjian Mo, and Min Zhang. 2025. Test-time Computing: from System-1 Thinking to System-2 Thinking.arXiv preprint arXiv:2501.02497(2025)

    2025 arXiv

  12. [20]

    InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs. InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing. 4958–4976

    2024

  13. [21]

    Nan Jiang, Xiaopeng Li, Shiqi Wang, Qiang Zhou, Soneya Binta Hossain, Baishakhi Ray, Varun Kumar, Xiaofei Ma, and Anoop Deoras. 2024. LeDex: Training LLMs to Better Self-Debug and Explain Code. InThe Thirty-eighth An- nual Conference on Neural Information Processing Systems. h...

    2024

  14. [22]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung

  15. [23]

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024. From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning. In Proceedings of the 2024 Conference of the North...

    2024

  16. [24]

    Jiayu Liu, Zhenya Huang, Qi Liu, Zhiyuan Ma, Chengxiang Zhai, and Enhong Chen. 2025. Knowledge-Centered Dual-Process Reasoning for Math Word Prob- lems With Large Language Models.IEEE Transactions on Knowledge and Data Engineering37, 6 (2025), 3457–3471. doi:10.1109/TKDE.2025.3556367

    2025

  17. [25]

    2011.Thinking, fast and slow

    Daniel Kahneman. 2011.Thinking, fast and slow. Macmillan

    2011

  18. [26]

    Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. 2024. SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models. InAdvances in Neural Information Processing Systems, Vol. 37

    2024

  19. [27]

    Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, et al

  20. [28]

    Jiayu Liu, Zhenya Huang, Zhiyuan Ma, Qi Liu, Enhong Chen, Tianhuang Su, and Haifeng Liu. 2023. Guiding Mathematical Reasoning via Mastering Commonsense Formula Knowledge. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1477–1488

    2023

  21. [29]

    Zuxin Liu, Thai Quoc Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R N, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024. APIGen: Automated PIpeline for ...

    2024

  22. [30]

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2024. Chameleon: Plug-and-play composi- tional reasoning with large language models.Advances in Neural Information Processing Systems36 (2024)

    2024

  23. [31]

    Zhiyuan Ma, Zhenya Huang, Jiayu Liu, Minmao Wang, Hongke Zhao, and Xin Li. 2025. Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning.Proceedings of the AAAI Conference on Artificial Intelligence39, 23 (Apr. 2025), 24821–24830. doi:10.1609/aaai.v39i23.34664

    2025 doi

  24. [32]

    Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Min Ma, et al. 2024. Coachlm: Automatic instruction revisions improve the data quality in llm instruction tuning. In2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 5184– 5197

    2024

  25. [33]

    OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https: //openai.com/blog/chatgpt. Accessed: 2024-01-30

    2022

  26. [34]

    Hongbin Pei, Yuheng Xiong, Pinghui Wang, Jing Tao, Jialun Liu, Huiqi Deng, Jie Ma, and Xiaohong Guan. 2024. Memory disagreement: A pseudo-labeling measure from training dynamics for semi-supervised graph learning. InProceedings of the ACM Web Conference 2024. 434–445

    2024

  27. [35]

    Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. Reasoning with large language models, a survey.arXiv preprint arXiv:2407.11511(2024)

    2024

  28. [36]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  29. [37]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems36 (2024)

    2024

  30. [38]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. Tool Learning with Large Language Models: A Survey.arXiv preprint arXiv:2405.17935(2024)

    2024 arXiv

  31. [39]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. InThirty-seventh Conference on Neural Infor- mation Processing Systems. https://openrevi...

    2023

  32. [40]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems36 (2024)

    2024

  33. [41]

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, et al. 2024. Tool Learning with Foundation Models.ACM Comput. Surv.57, 4, Article 101 (Dec. 2024), 40 pages. doi:10.1145/3704435

    2024 doi

  34. [42]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al . [n. d.]. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. InThe Twelfth International Conference on Learning Representations

  35. [43]

    Xubin Ren, Jiabin Tang, Dawei Yin, Nitesh Chawla, and Chao Huang. 2024. A survey of large language models for graphs. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6616–6626

    2024

  36. [44]

    Matthew Renze and Erhan Guven. 2024. Self-Reflection in LLM Agents: Effects on Problem-Solving Performance.arXiv preprint arXiv:2405.06682(2024). Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning KDD ’25, August 3–7, 2025, Toronto, ON, Canada

    2024 arXiv

  37. [45]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2023), 68539–68551

    2023

  38. [46]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. InSC20: Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

    2020

  39. [47]

    RapidAPI. 2024. RapidAPI Hub - The World’s Largest API Hub. https://rapidapi. com/hub. Accessed: 2024-01-30

    2024

  40. [48]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems36 (2024)

    2024

  41. [49]

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei...

    2024 doi

  42. [50]

    Keith E Stanovich and Richard F West. 2000. Individual differences in reasoning: Implications for the rationality debate?Behavioral and brain sciences23, 5 (2000), 645–665

    2000

  43. [51]

    Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, Lingyong Yan, Haibo Shi, Dawei Yin, Pengjie Ren, Suzan Verberne, and Zhaochun Ren. 2024. Learning to Use Tools via Cooperative and Interactive Agents. InFindings of the Association for Computational Linguistics: EMNLP 2024. 10642–10657

    2024

  44. [52]

    Zhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng, Xiuyi Chen, Zhumin Chen, Dawei Yin, Suzan Verberne, and Zhaochun Ren. 2025. Tool learning in the wild: Empowering language models as automatic tool agents. InProceedings of the ACM on Web Conference 2025. 2222–2237

    2025

  45. [53]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Jiale Li, et al. [n. d.]. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. InICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  46. [54]

    Qinzhuo Wu, Wei Liu, Jian Luan, and Bin Wang. 2024. ToolPlanner: A Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedback. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 18315–18339

    2024

  47. [55]

    Tong Xiao, Jiayu Liu, Zhenya Huang, Jinze Wu, Jing Sha, Shijin Wang, and Enhong Chen. 2024. Learning to Solve Geometry Problems via Simulating Human Dual-Reasoning Process. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, K...

    2024

  48. [56]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. [n. d.]. Executable Code Actions Elicit Better LLM Agents. InForty-first International Conference on Machine Learning

  49. [57]

    Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. 2024. What are tools anyway? a survey from the language model perspective.arXiv preprint arXiv:2403.15452(2024)

    2024 arXiv

  50. [58]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, et al. 2024. Qwen2. 5 Technical Report.arXiv preprint arXiv:2412.15115(2024)

    2024 arXiv

  51. [59]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. [n. d.]. ReAct: Synergizing Reasoning and Acting in Language Models. InThe Eleventh International Conference on Learning Representations

  52. [60]

    Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, and Enhong Chen. 2024. Dataset Regeneration for Sequential Recommendation. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Barcelona, Spain)(KDD ’24). Associa...

    2024

  53. [61]

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Proces...

    2024 doi

  54. [62]

    Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. 2024. Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle. InAdvances in Neural Information Processing Systems, Vol. 37

    2024

  55. [63]

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning. InThe Twelfth International Conference on Learning Representations

    2024

  56. [64]

    Chan, Raymond Chi-Wing Wong, and Haiqin Yang

    Weixu Zhang, Yifei Wang, Yuanfeng Song, Victor Junqiu Wei, Yuxing Tian, Yiyan Qi, Jonathan H. Chan, Raymond Chi-Wing Wong, and Haiqin Yang. 2024. Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey.IEEE Trans. on Knowl. and Data Eng.36, 11 (Nov. 2...

    2024

  57. [65]

    Hongke Zhao, Likang Wu, Yuqing Shan, Zonghan Jin, Yuanpei Sui, Zipeng Liu, Nan Feng, Minqiang Li, and Wei Zhang. 2024. A comprehensive survey of large language models in management: Applications, challenges, and opportunities. Challenges, and Opportunities (August 14, 2024)(2024)

    2024

  58. [66]

    Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. 2024. StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs.arXiv preprint arXiv:2410.07745(2024)

    2024 arXiv

  59. [67]

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Kan Ren, Dong- sheng Li, and Deqing Yang. [n. d.]. EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction. InICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  60. [68]

    Yuze Zhao, Tianyun Ji, Wenjun Feng, Zhenya Huang, Qi Liu, Zhiding Liu, Yixiao Ma, Kai Zhang, and Enhong Chen. 2025. Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment. InThe Thirteenth Interna- tional Conference on Learning Representations. ht...

    2025

  61. [69]

    Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Mei, et al. 2024. Recommender Systems in the Era of Large Language Models (LLMs).IEEE Transactions on Knowledge and Data Engineering36, 11 (2024), 6889–6907. doi:10.1109/TKDE. 2024.3392335

    2024

  62. [70]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo

  63. [71]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models.arXiv preprint arXiv:2303.18223(2023)

    2023 arXiv

  64. [72]

    Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. 2024. RePair: Automated Program Repair with Process-based Feedback. InFindings of the Association for Computational Linguistics ACL 2024. 16415–16429

    2024

  65. [76]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Yixin Cao, Yang Feng, and Deyi Xiong (Eds.)

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Yixin Cao, Yang Feng, and Deyi Xiong (Eds.). Association for Computational Linguist...

    doi

  66. [77]

    Input param- eters missing

    Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. [n. d.]. ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search. InThe Twelfth International Conference on Learning Representations. ...

    2025

  67. [78]

    Answer Quality (60 points):Completeness (20pts): Contains all required information, fully addresses query requirements; Accuracy (20pts): Information correctness, alignment with API responses; Clarity (10pts): Well-structured presentation, easy to understand; Error Handling (1...

  68. [79]

    content":

    Execution Efficiency (40 points):Tool Usage (15pts): Failed API calls (-2pts each), redundant calls (-1pt each), appropriate tool selection; Execution Path (15pts): Logical progression, minimal steps, efficient goal achievement; Strategy (10pts): Tool selection planning, respo...

    2000

  69. [80]

    Explanation of decision

  70. [81]

    Can you fetch the flight data for the company AZU on June 15th, 2022?

    JSON response with decision and quality score Example: Query: “Can you fetch the flight data for the company AZU on June 15th, 2022?” Tools: Flight data API with company and date parameters Output: {“decision”: “Solvable”, “quality_score”: 7} Input Required:Query text, Availab...

    2022

  71. [2023]

    InFindings of the Association for Computational Linguistics: EMNLP 2023

    Towards mitigating LLM hallucination via self reflection. InFindings of the Association for Computational Linguistics: EMNLP 2023. 1827–1843

    2023

  72. [2024]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Towards Tool Use Alignment of Large Language Models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 1382–1400

    2024

  73. [2025]

    InThe Thirteenth International Conference on Learning Representations

    ToolACE: Winning the Points of LLM Function Calling. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/ forum?id=8EB8k6DdCU

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.