Pith. sign in

REVIEW 3 major objections 6 minor 50 references

Community-maintained tools with higher intrinsic quality raise tool-using agent scores by 6–22% relative to a smaller curated toolbox.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 19:58 UTC pith:OLTRMJXM

load-bearing objection Solid open infrastructure for tool reliability and community maintenance; the headline 6–22% gains are real under their protocol but not cleanly isolated as “intrinsic accuracy” versus coverage and subsetting. the 3 major comments →

arxiv 2604.00137 v2 pith:OLTRMJXM submitted 2026-03-31 cs.AI cs.SE

Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents

Hy Dang , Quang Dao , Meng Jiang This is my paper
classification cs.AI cs.SE
keywords tool-using AI agentsintrinsic tool accuracycommunity-driven toolboxLLM agentstool reliabilityagent evaluationopen-source toolstool standardization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Tool-using language models fail for two distinct reasons: the agent may choose or call tools badly, and the tools themselves may be wrong, unstable, or unsafe. Most prior work treats tools as fixed and reliable and concentrates on tool-use skill. This paper argues that intrinsic tool accuracy—correctness, stability, and safety—must be measured and maintained on its own, and it introduces an open, community-driven toolbox that standardizes interfaces, runs living test suites, and refreshes reliability reports as tools change. In head-to-head agent experiments with the base model held fixed, swapping in the community toolbox with more task-specific tools yields relative gains of about 6% to 22% over an existing curated toolbox across several agent styles. A public demo and contribution path are meant to keep tests and tools evolving rather than freezing them at release.

Core claim

End-to-end reliability of tool-integrated language models depends as much on the tools’ own correctness, stability, and safety as on how well agents select and invoke them. A community-driven toolbox that standardizes schemas, continuously evaluates tools with contributed tests, and exposes those tools through interchangeable agent workflows produces consistent downstream gains—relative improvements of 6% to 22% over a prior curated toolbox—across prompting, reason-and-act, planner–executor, and multi-agent setups, with the largest lifts on tasks that require external actions.

What carries the argument

OpenTools: a dual-loop design that separates a tool accuracy/maintenance loop (unified JSON schemas, living test suites, community contributions, regression and availability reports) from an agentic workflow that plugs the same toolbox into interchangeable agent policies and emits structured traces so failures can be attributed to tool-use errors versus tool-side faults.

Load-bearing premise

The reported end-to-end gains are mainly due to better tool correctness and reliability, not simply to offering many more tools and different task-specific subsets than the baseline toolbox.

What would settle it

Hold tool count and task-specific tool subsets fixed between the two toolboxes and re-run the same agent policies; if the 6–22% relative gains largely vanish, coverage—not intrinsic accuracy—was driving the result.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On hard agent tasks that require external actions, better tool breadth and reliability can matter more than a stronger base model alone.
  • Tools can be improved and re-tested without rewriting agent policies, and agents can be swapped without rewriting tools.
  • Living community test suites make API drift, nondeterminism, and silent regressions visible over time rather than only at release.
  • Prior reliability scores can guide which tools an agent is allowed to use or prefer before a run.
  • Public run-and-contribute interfaces lower the barrier for turning observed failures into shared tests.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If intrinsic tool accuracy is a real bottleneck, closed fixed tool catalogs may lag open continuously tested ones even when the agent policy is identical.
  • A clean ablation that matches tool count and forced subsets would separate quality from mere coverage—an isolation the main results leave partly open.
  • Routine agent debugging should split failures into policy mistakes versus tool regressions using structured call traces of the kind described here.
  • Non-executing risk checks plus optional review may become a standard gate for shared agent tool registries as contribution volume grows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces OpenTools, an open-source, community-driven toolbox and dual-workflow framework for tool-integrated LLM agents. It distinguishes tool-use accuracy from intrinsic tool accuracy (correctness, stability, robustness) and operationalizes the latter via standardized JSON-schema interfaces, community-contributed tests, continuous evaluation/maintenance, and a public web demo. Empirically, agents using OpenTools-T are compared to the same policies with the OctoTools toolbox on 15 benchmarks across five task groups and multiple agent architectures (Prompting, ReAct, OctoTools, MultiAgent), reporting 6%–22% relative average gains and larger gains on the Agent (GAIA-Text) task. The manuscript also releases code, a contribution protocol, and demo infrastructure under Apache 2.0.

Significance. If the systems contribution holds, OpenTools is a useful infrastructure paper: it makes intrinsic tool reliability a first-class, community-maintainable object rather than an implicit assumption of tool-learning work. Strengths include an open repository, standardized wrappers, a public interactive demo for tests/feedback, multi-framework agent evaluation with three-trial means±std (Appendix Table 2), and explicit separation of tool maintenance from agent policy. The reported end-to-end gains under fixed base models and policies are practically relevant for agent builders. The work is significant primarily as reproducible systems/infrastructure and as an empirical demonstration that toolbox choice matters; its stronger mechanistic claim about intrinsic accuracy is more provisional and needs tighter support.

major comments (3)
  1. Abstract and §4 attribute the 6%–22% relative gains primarily to ‘intrinsic tool accuracy’ / ‘higher-quality’ tools. Table 4 shows OpenTools-T has 42 tools vs 13 in OctoTools-T (16 vs 1 program-based; 19 vs 8 API-based). §3.1.2 further uses greedy task-specific subsetting for OpenTools-T versus paper-reported best sets for OctoTools-T. No experiment holds tool count/coverage fixed, swaps only tools with higher measured R(τ) on the same suite, or correlates per-tool reliability with end-to-end deltas. The Agent-task jumps (e.g., MultiAgent 53.92→66.14) are exactly where specialized coverage (maze solvers, etc.) would help regardless of reliability signals. Please either (i) add coverage-matched or reliability-only ablations, or (ii) reframe the central claim as gains from a larger, community-maintained, task-specific toolbox—of which intrinsic accuracy is one unisolated component.
  2. Workflow 1 defines reliability profiles R(τ) and continuous regression testing (§2.1), and tool cards can expose ‘current intrinsic accuracy,’ yet Tables 1–2 and the analysis never report suite-level accuracy, availability, or regression metrics for the tools actually used, nor show that agents preferred higher-R(τ) tools. Without those measurements, the dual-workflow architecture’s evaluation loop is described but not shown to drive the agentic results. At minimum, report per-tool or per-category reliability scores for OpenTools-T vs OctoTools-T tools used on the benchmarks, and state whether reliability signals were enabled during agent runs.
  3. The Abstract and introduction list safety and non-executing risk inspection as part of intrinsic accuracy, but §3–4 contain no safety, sandboxing, or adversarial-tool evaluation. Either provide a minimal safety/risk evaluation consistent with the abstract’s claims, or narrow the abstract/intro language so experimental claims match what is measured (correctness/coverage/end-to-end accuracy).
minor comments (6)
  1. Table 1 formatting is hard to parse (slash-separated model scores packed into cells; MultiAgent only reports gpt-5-mini). Consider separate columns or panels for each base model.
  2. §3.1.2 says zero-shot outperforms CoT in preliminary experiments and relegates the comparison to Appendix Table 3; a one-sentence note in the main text on why CoT hurts the stronger model would help readers.
  3. Figure 1 is dense; the two workflows are clear in text but the figure labels (1)/(2)/(3) repeat across panels and can confuse first-time readers.
  4. Minor inconsistencies: Abstract mentions MCP and ‘non-executing risk inspection / advisory LLM review,’ while the body emphasizes schemas, tests, and the web demo more than MCP; align terminology across abstract and §2.
  5. Typos/style: ‘OPENTOOLS’ capitalization varies; ‘Feedbacks’ in Figure 1; occasional missing spaces after periods in the arXiv text. Standard copy-edit pass recommended.
  6. Related Work could more explicitly position against StableToolBench, ToolFuzz, and other tool-stability/testing efforts already cited, clarifying what is new in the community maintenance loop versus prior tool libraries.

Circularity Check

0 steps flagged

No circularity: empirical toolbox comparison with measured end-to-end gains, not a derivation that reduces predictions to inputs.

full rationale

OpenTools is a systems/empirical paper. Its load-bearing claim is that equipping agents with OpenTools-T yields 6%–22% relative gains over OctoTools-T across frameworks (Abstract; Table 1; §4). That claim is supported by held-out benchmark measurements under fixed base LLMs and agent policies, not by a mathematical derivation, fitted parameter renamed as prediction, or uniqueness theorem. Reliability profiles R(τ) are defined as estimates from external test suites T (§2.1.1), not as quantities that force the Table 1 scores by construction. Self-citations (e.g., Dang et al. 2025 in Related Work) are ordinary framing and do not underwrite the gain numbers. Confounding between tool coverage (42 vs 13 tools) and intrinsic accuracy is a causal-attribution concern, not circularity under the stated patterns. No step reduces Eq./claim X to its own inputs by definition.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 3 invented entities

As a systems/empirical paper the load-bearing content is mostly engineering design choices and experimental protocol assumptions rather than free physical constants. The central claim rests on the separability of tool-use vs intrinsic accuracy, the fairness of the OctoTools-T comparison under unequal tool counts, and the premise that community-maintained tests will keep reliability signals current.

free parameters (2)
  • Task-specific tool subset size / greedy selection policy for OpenTools-T
    Authors perform greedy tool selection for OpenTools-T and use paper-reported best tool sets for OctoTools-T; the exact subset cardinalities and selection thresholds are design choices that affect measured gains.
  • Number of independent trials (n=3) and base-model choice
    Reported means±std rest on three trials with fixed proprietary models; trial count and model versions are experimenter choices that influence variance estimates and absolute scores.
axioms (4)
  • domain assumption Tool-integrated LLM reliability decomposes into tool-use accuracy and intrinsic tool accuracy (correctness, stability, safety), and the latter is currently underexamined.
    Stated in the abstract and Introduction as the motivating distinction that justifies the framework.
  • domain assumption A unified natural-language description + typed JSON argument schema + structured output contract is sufficient for plug-and-play use across heterogeneous agent frameworks.
    Section 2.1.1 Tool Standardization; contrasted with OctoTools metadata.
  • domain assumption Community-contributed test cases, once accepted by maintainers, produce living reliability signals that track non-stationary tool behavior (API drift, nondeterminism, regressions).
    Section 2.1.2 Community-Driven Maintenance; core premise of the maintenance loop.
  • ad hoc to paper Comparing agents under identical policies while swapping only the toolbox isolates the effect of tool quality/coverage on end-to-end task accuracy.
    Experimental design in Section 3; the unequal tool counts make this isolation incomplete.
invented entities (3)
  • OpenTools (toolbox + maintenance loop + contribution protocol + web demo) no independent evidence
    purpose: Provide standardized, continuously evaluated open tools for LLM agents and a community process for tests and new tools.
    The primary artifact introduced by the paper; independent evidence is the public repo/demo and reported experiments, not an external physical prediction.
  • Reliability profile R(τ) no independent evidence
    purpose: Summarize a tool’s correctness and robustness under a suite of tests for use in monitoring and optional routing.
    Defined in Section 2.1 Problem Definition as the object estimated by the evaluation loop.
  • Tool Accuracy / Maintenance Loop vs Agentic Workflow dual architecture no independent evidence
    purpose: Decouple continuous tool validation from agent orchestration so tools and policies can evolve independently.
    Figure 1 and Section 2 design goals; structural invention of the system.

pith-pipeline@v1.1.0-grok45 · 18832 in / 3049 out tokens · 36804 ms · 2026-07-14T19:58:37.479470+00:00 · methodology

0 comments
read the original abstract

Tool-integrated LLMs retrieve information, perform computations, and take real-world actions, but their reliability depends on both tool-use accuracy and intrinsic tool accuracy, including tool correctness, stability, and safety. While prior work primarily emphasizes tool use, intrinsic tool accuracy remains underexamined. We introduce OpenTools, a community-driven and maintainable toolbox for discovering, using, evaluating, and contributing open-source tools. OpenTools standardizes tool interfaces, converts documented Python functions into reviewable bundles, supports maintainer-triggered evaluation, and combines non-executing risk inspection with optional advisory LLM review. A public web demo allows users to run tools and agents, inspect evidence, contribute tests, and submit tools for maintainer review, while MCP enables controlled access from external applications. Experiments show that community-contributed, task-specific tools yield relative gains of 6% to 22% over an existing toolbox across multiple agent architectures, highlighting the importance of intrinsic tool accuracy.

Figures

Figures reproduced from arXiv: 2604.00137 by Hy Dang, Meng Jiang, Quang Dao.

Figure 1
Figure 1. Figure 1: OPENTOOLS supports two complementary workflows. Top (maintenance): OpenTools stores Tools and Tests; community modules collect new tools/tests/feedback, which are reviewed by verifier(s) (tool cre￾ators/maintainers) to accept, curate, and update the canonical toolboxes and evaluation suites; tool-evaluation modules then re-run standardized checks to refresh intrinsic reliability signals (accuracy, availabi… view at source ↗
Figure 4
Figure 4. Figure 4: Agent Playground UI: select an agent and base [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 2
Figure 2. Figure 2: Test-case contribution UI: submission instruc [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Test-case authoring/execution UI: run tools [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 16 linked inside Pith

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.219 Can knowledge graphs reduce hallucinations in LLM s? : A survey . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p...

    doi:10.18653/v1/2024.naacl-long.219 2024

  4. [4]

    S \"o ren Arlt, Xuemei Gu, and Mario Krenn. 2025. Towards autonomous quantum physics research using llm agents with access to intelligent tools. arXiv preprint arXiv:2511.11752

    arXiv 2025

  5. [5]

    Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. 2024. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16259--16273

    2024

  6. [6]

    Hy Dang, Tianyi Liu, Zhuofeng Wu, Jingfeng Yang, Haoming Jiang, Tao Yang, Pei Chen, Zhengyang Wang, Helen Wang, Huasheng Li, Bing Yin, and Meng Jiang. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.1242 Improving large language models function calling and interpretability via guided-structured templates . In Proceedings of the 2025 Conference on Empiri...

    doi:10.18653/v1/2025.emnlp-main.1242 2025

  7. [7]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, and 1 others. 2024. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985

    Pith/arXiv arXiv 2024

  8. [8]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2

    Pith/arXiv arXiv 2023

  9. [9]

    Deepanway Ghosal, Vernon Toh, Yew Ken Chia, and Soujanya Poria. 2025. Algopuzzlevqa: Diagnosing multimodal reasoning challenges of language models with algorithmic multimodal puzzles. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa...

    2025

  10. [10]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  11. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  12. [12]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  13. [13]

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.664 S table T ool B ench: Towards stable large-scale benchmarking on tool learning of large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 11143-...

    doi:10.18653/v1/2024.findings-acl.664 2024

  14. [14]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286

    Pith/arXiv arXiv 2020

  15. [15]

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. 2024. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. Advances in Neural Information Processing Systems, 37:139348--139379

    2024

  16. [16]

    Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, and 1 others. 2024. A survey of safety and trustworthiness of large language models through the lens of verification and validation. Artificial Intelligence Review, 57(7):175

    2024

  17. [17]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2023. Metatool benchmark: Deciding whether to use tools and which to use. arXiv preprint arXiv: 2310.03128

    Pith/arXiv arXiv 2023

  18. [18]

    Dongsuk Jang, Ziyao Shangguan, Kyle Tegtmeyer, Anurag Gupta, Jan T Czerminski, Sophie Chheang, and Arman Cohan. 2025. Medtutor: A retrieval-augmented llm system for case-based medical education. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 319--353

    2025

  19. [19]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827--1843

    2023

  20. [20]

    Haipeng Jiang, Kailong Ren, Zimo Yin, Zhetao Sun, Xin Gan, Guangyi Lv, Ming He, Peng Wang, Congli Yin, Hong Pan, and 1 others. 2026. Lemon agent technical report. arXiv preprint arXiv:2602.07092

    arXiv 2026

  21. [21]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    Pith/arXiv arXiv 2025

  22. [22]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  23. [23]

    Toolscan: A benchmark for characterizing errors in tool-use llms

    Shirley Kokane, Ming Zhu, Tulika Manoj Awalgaonkar, Jianguo Zhang, Akshara Prabhakar, Thai Quoc Hoang, Zuxin Liu, Rithesh RN, Liangwei Yang, Weiran Yao, and 1 others. Toolscan: A benchmark for characterizing errors in tool-use llms. In ICLR 2025 Workshop on Building Trust in Language Models and Applications

    2025

  24. [24]

    Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and 1 others. 2024 a . Mmedagent: Learning to use medical tools with multi-modal agent. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8745--8760

    2024

  25. [25]

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. 2024 b . Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875, 1

    Pith/arXiv arXiv 2024

  26. [26]

    Nathan Lile. 2024. Math twenty four (24s game) dataset. https://huggingface.co/datasets/nlile/24-game. Hugging Face Datasets

    2024

  27. [27]

    Adam Dahlgren Lindstr \"o m and Savitha Sam Abraham. 2022. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358

    Pith/arXiv arXiv 2022

  28. [28]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650--1654. IEEE

    2021

  29. [29]

    Qiang Liu, Xinlong Chen, Yue Ding, Bowen Song, Weiqiang Wang, Shu Wu, and Liang Wang. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.1063 Attention-guided self-reflection for zero-shot hallucination detection in large language models . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21016--21032, Suzhou,...

    doi:10.18653/v1/2025.emnlp-main.1063 2025

  30. [30]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255

    Pith/arXiv arXiv 2023

  31. [31]

    Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. 2025. Octotools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502.11271

    Pith/arXiv arXiv 2025

  32. [32]

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations

    2023

  33. [33]

    Ivan Milev, Mislav Balunovi \'c , Maximilian Baader, and Martin Vechev. 2025. Toolfuzz--automated agent tool testing. arXiv preprint arXiv:2503.04479

    Pith/arXiv arXiv 2025

  34. [34]

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014

    Pith/arXiv arXiv 2023

  35. [35]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2024. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544--126565

    2024

  36. [36]

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, and 1 others. 2024. Tool learning with foundation models. ACM Computing Surveys, 57(4):1--40

    2024

  37. [37]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://doi.org/10.48550/ARXIV.2307.16789 Toolllm: Facilitating large language models to master 16000+ real-world apis . CoRR, abs/2307.16789

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.16789 2023

  38. [38]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  39. [39]

    Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. 2024. Scifibench: Benchmarking large multimodal models for scientific figure interpretation. Advances in Neural Information Processing Systems, 37:18695--18728

    2024

  40. [40]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539--68551

    2023

  41. [41]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024 a . Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning

    2024

  42. [42]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024 b . Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266--95290

    2024

  43. [43]

    Cailin Winston and Ren \'e Just. 2025. A taxonomy of failures in tool-augmented llms. In 2025 IEEE/ACM International Conference on Automation of Software Test (AST), pages 125--135. IEEE

    2025

  44. [44]

    Mengsong Wu, YaFei Wang, Yidong Ming, Yuqi An, Yuwei Wan, Wenliang Chen, Binbin Lin, Yuqiang Li, Tong Xie, and Dongzhan Zhou. 2025. Chemagent: Enhancing llms for chemistry and materials science through tree-search based tool learning. arXiv preprint arXiv:2506.07551

    Pith/arXiv arXiv 2025

  45. [45]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023 a . Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

    2023

  46. [46]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023 b . React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  47. [47]

    Chengyue Yu, Siyuan Lu, Chenyi Zhuang, Dong Wang, Qintong Wu, Zongyue Li, Runsheng Gan, Chunfeng Wang, Siqi Hou, Gaochi Huang, and 1 others. 2025. Aworld: Orchestrating the training recipe for agentic ai. arXiv preprint arXiv:2508.20404

    Pith/arXiv arXiv 2025

  48. [48]

    Xiang Zhang and Dujian Ding. 2024. Supervised chain of thought. arXiv preprint arXiv:2410.14198

    Pith/arXiv arXiv 2024

  49. [49]

    Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024. Chain of preference optimization: Improving chain-of-thought reasoning in llms. Advances in Neural Information Processing Systems, 37:333--356

    2024

  50. [50]

    Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. 2025. https://aclanthology.org/2025.findings-naacl.205/ K now A gent: Knowledge-augmented planning for LLM -based agents . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3709--3732, Albuquerque, New Mex...

    2025