Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Integrating External Tools with Large Language Models to Improve Accuracy

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Athena, a framework that lets an LLM call external APIs, reaches 83% accuracy on math and 88% on science questions from the MMLU collection, beating all tested standalone models.

desk verdict Standard tool-calling agent with plausible but unverified headline numbers; the causal claim about tool integration is untested because there's no no-tool control. read the letter →

arxiv 2507.08034 v1 pith:VLU7AMCB submitted 2025-07-09 cs.CL cs.AIcs.LG

Nripesh Niketan , Hadj Batatia This is my paper
classification cs.CLcs.AIcs.LG
keywords toolintegrationlargelanguagemodelsexternalAPIsmathematicalreasoningscientificMMLUbenchmarkeducationalAIAthenaframework
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that giving a large language model access to external tools, such as calculators, web search, weather data, calendars, and scholarly-article lookup, makes it markedly more accurate on reasoning questions than any standalone model of the same generation. Its Athena framework scores 83% on 100 mathematical-reasoning questions and 88% on 100 scientific-reasoning questions drawn from the MMLU collection, while the best tested baseline model scores 67% and 79%. The authors read this as evidence that tool integration supplies capabilities that model scaling alone does not. A sympathetic reader would care because this points to a low-cost route to accuracy: augmenting an existing model with APIs instead of waiting for a larger base model.

What carries the argument

The load-bearing mechanism is a schema registry plus an iterative tool-use loop. Each tool is described by a structured schema, naming its purpose, accepted arguments, and output type, so the LLM knows what each tool can do. When a query arrives, the system decides whether the query matches a tool; if so, it extracts the required parameters, calls the external API, converts the raw result back into conversational text, and resubmits the updated query to the LLM. The loop continues until the model is satisfied that the question is answered.

What would settle it

Record every tool invocation during the MMLU runs and re-run the same questions through Athena with the tool responses suppressed; if accuracy stays near 83% and 88% rather than falling toward the 67% and 79% baselines, the tools are not responsible for the gain.

Watch

Extended reading notes

Core claim

The central claim is that a modest LLM, wired to external APIs through a tool-registration loop, can outperform substantially larger standalone models on educational reasoning benchmarks. On 100 math questions and 100 science questions from the MMLU collection, Athena reaches 83% and 88% accuracy respectively; the best baseline, LLaMA-Large, reaches 67% and 79%. The paper attributes the gains mainly to computational tools such as calculators for multistep arithmetic and to search APIs for factual lookups beyond the model's training data. Its closing conclusion is that tool integration provides capabilities that cannot be achieved through model scaling alone.

Load-bearing premise

The result rests on the assumption that the accuracy gap comes from the external tools themselves and not from the different underlying model, prompting, or scaffolding used in Athena; the paper does not run an otherwise identical version with the tools removed.

Editorial extensions

If this is right

  • Educators could obtain high accuracy on math and science multiple-choice questions without the largest commercial models, by pairing a mid-size model with a calculator and search tool.
  • The smaller gain in science (9 points over the best baseline) than in math (16 points) suggests modern LLMs have largely caught up on factual scientific recall but still depend on external computation for calculation-heavy problems.
  • Because tools are added through schemas rather than retraining, the same scaffold can be extended to new APIs, so accuracy gains should generalize to any domain with a reliable external service.
  • The reported pattern implies that tool integration remains valuable even as base models improve, because each new model generation still lacks dedicated computational or up-to-date information access.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: compare Athena against the same prompt and scaffold with all tool calls disabled; that ablation is what would isolate the tool contribution from the contribution of the middleware and the model selection.
  • If the tools, not the scaffold, drive the gains, then accuracy per dollar should be obtainable by adding APIs to a small model rather than paying for a larger one; the paper's numbers make that hypothesis testable but do not test it.
  • With only 100 questions per domain, the headline gaps may carry wide uncertainty; running the same comparison on the full MMLU test split would tell whether the margin is stable or driven by sample selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes Athena, a LangChain-based framework that integrates external APIs (Wolfram Alpha, Google SERPer, ArXiv, OpenWeatherMap, Google Calendar) with LLMs hosted on the Unify platform. It evaluates Athena on hand-selected MMLU math (33/34 questions per level) and science (16 or 17 per subject) subsets, reporting 83% math and 88% science accuracy versus bare LLM baselines, with LLaMA-Large best at 67% and 79%. The paper concludes that tool integration provides capabilities beyond model scaling.

Significance. If the reported results were causally attributable to tool access, the paper would provide a useful data point for tool-augmented LLM agents in educational question answering. However, the current design cannot support the central causal claim: the Athena system and the baselines differ in model identity, scaffold, prompt formatting, and tool access simultaneously. The paper provides no controlled ablation, no error bars or significance tests, and contains a concrete arithmetic inconsistency in the science question count. These issues substantially limit its contribution.

major comments (4)
  1. [§3.3–3.4] The evaluation does not include a no-tool control for Athena. Section 2.1 states that deciding which LLM to deploy and how to integrate external tool responses is managed internally by LangChain on the Unify platform, so Athena differs from each baseline in base model choice, agent scaffold, and prompt structure in addition to tool access. The reported accuracy gap cannot therefore be attributed to tool integration; an ablation that toggles tools while holding model, scaffold, and prompt fixed is necessary to support the paper's central claim.
  2. [§3.2] The science dataset count is internally inconsistent: 16 questions each for three high-school subjects (48) plus 17 each for three college subjects (51) totals 99, not the stated 100. This affects the reported 88% accuracy (88/100 vs 88/99 or 87/99) and the statement that Athena answered only 12 science questions incorrectly; the correct count and corresponding accuracy need to be reconciled.
  3. [§3.4] The claim that tool integration 'provides capabilities that cannot be achieved through model scaling alone' is unsupported by the experimental design, which compares Athena against a heterogeneous collection of baseline models of different sizes and families rather than a scaling series of the same model family with and without tool access. Supporting this claim would require a controlled scaling comparison or at least a matched-capacity tool/no-tool comparison.
  4. [§3.3] The evaluation reports point estimates on small, hand-picked subsets (33 or 34 math questions per level; at most 100 science questions) with no confidence intervals, repeated runs, or statistical tests. The differences between Athena and baselines are therefore not established as statistically reliable, and the sample selection criteria are not described.
minor comments (6)
  1. [§2.1, Listing 2] In the example JSON prompt, the key is misspelled as '"answe r"' instead of '"answer"', and the surrounding quotation marks and line breaks are malformed.
  2. [§2.1] The identity and version of the underlying LLM(s) used inside Athena through Unify/LangChain are never reported, making the results impossible to reproduce or interpret.
  3. [§3.3] Baseline model designations such as 'GPT-4o', 'LLaMA-Large', 'Mistral-Large', and 'Phi-Large' lack version numbers, access dates, and provider details, preventing exact replication.
  4. [§3.1] The manuscript does not report which of the five integrated tools were actually invoked for the math and science questions, nor how often; a tool-invocation log would help assess the framework's behavior.
  5. [abstract/§3.4] The abstract's claim that the approach 'significantly improves performance' is not supported by any statistical significance testing; the wording should be tempered unless such tests are added.
  6. [§3.2] The procedure for selecting exactly 33 and 34 mathematics questions from the Elementary, High School, and College datasets is not described; a random sampling seed or inclusion criterion should be specified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper makes an empirical comparison that is confounded by scaffold and model-choice differences, but no claim reduces by construction to its own inputs or to a self-citation chain.

full rationale

The paper does not contain a mathematical derivation, fitted parameter, or imported uniqueness theorem, so the classic circularity failure modes do not arise. The central claim, that tool integration provides capabilities beyond model scaling alone, rests on comparing Athena (a LangChain agent over Unify) with bare baseline LLMs, with no ablation that disables tools while holding the scaffold, model, and prompt fixed; this is a causal-inference and confound problem, not a circularity. The paper defines no quantity in terms of another quantity it later claims to predict, and it does not cite its own prior results as load-bearing evidence. The arithmetic inconsistency in the stated science question count (16+16+16+17+17+17=99, not 100) is a correctness concern, not circularity. Accordingly, an honest non-finding is appropriate: score 0, with the caveat that the empirical attribution is unverified but not circular.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the fairness and representativeness of a small, hand-picked evaluation and on the correct operation of an unspecified tool-selection pipeline. No fitted parameters or new entities are introduced.

assumptions (3)
  • domain assumption The selected MMLU subsets are representative of general mathematical and scientific reasoning ability.
    The paper hand-selects 67 math and 99-100 science questions without randomization or justification of representativeness (Section 3.2).
  • domain assumption The baseline models were evaluated under identical conditions to Athena except for tool access.
    The paper asserts this (Section 3.3) but provides no logs or configurations to verify it.
  • domain assumption The tool-selection mechanism in RunMonitoring correctly identifies when a tool is needed and extracts correct parameters.
    The description of keyword matching and intent recognition (Section 2.1) is high-level and not validated separately.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Integrating External Tools with Large Language Models to Improve Accuracy." pith.science (2026). https://pith.science/paper/VLU7AMCB

@misc{pith2026250708034,
  author       = {Pith},
  title        = {Pith review of: Integrating External Tools with Large Language Models to Improve Accuracy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLU7AMCB}},
  note         = {Machine review of arXiv:2507.08034}
}
read the original abstract

This paper deals with improving querying large language models (LLMs). It is well-known that without relevant contextual information, LLMs can provide poor quality responses or tend to hallucinate. Several initiatives have proposed integrating LLMs with external tools to provide them with up-to-date data to improve accuracy. In this paper, we propose a framework to integrate external tools to enhance the capabilities of LLMs in answering queries in educational settings. Precisely, we develop a framework that allows accessing external APIs to request additional relevant information. Integrated tools can also provide computational capabilities such as calculators or calendars. The proposed framework has been evaluated using datasets from the Multi-Modal Language Understanding (MMLU) collection. The data consists of questions on mathematical and scientific reasoning. Results compared to state-of-the-art language models show that the proposed approach significantly improves performance. Our Athena framework achieves 83% accuracy in mathematical reasoning and 88% in scientific reasoning, substantially outperforming all tested models including GPT-4o, LLaMA-Large, Mistral-Large, Phi-Large, and GPT-3.5, with the best baseline model (LLaMA-Large) achieving only 67% and 79% respectively. These promising results open the way to creating complex computing ecosystems around LLMs to make their use more natural to support various tasks and activities.

Figures

Figures reproduced from arXiv: 2507.08034 by the authors.

Figure 1
Figure 1. Architecture of the proposed Athena framework showing the integration of external tools with LLMs through [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Mathematical reasoning accuracy comparison between Athena framework and state-of-the-art language [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Scientific reasoning accuracy comparison between Athena framework and state-of-the-art language models [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 4 canonical work pages

  1. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  2. [2]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Luyu Gao, Zhuyun Dai, and Jamie Callan. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 36, 2023

    work page 2023

  4. [4]

    Retrieval-augmented generation for knowledge- intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in neural information processing systems , 33:9459–9474, 2020

    2020

  5. [5]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  6. [6]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [7]

    Naturalproofs: Mathematical theorem proving in natural language

    Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021

    arXiv 2021

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 21 references
  1. [9]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  2. [10]

    Internet-augmented dialogue generation

    Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. International Conference on Machine Learning, pages 8460–8478, 2021

    2021

  3. [11]

    Lamda: Language models for dialog applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022

    2022 arXiv

  4. [12]

    Check your facts and try again: Improving large language models with external knowledge and automated feedback

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813, 2023....

    2023 arXiv

  5. [13]

    Unifying large language models and knowledge graphs: A roadmap

    Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering , 2023

    2023

  6. [14]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  7. [15]

    Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295, 2023

    2023 arXiv

  8. [16]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023

    2023 arXiv

  9. [17]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023

    2023 arXiv

  10. [18]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022

    2022 arXiv

  11. [19]

    Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings

    Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554, 2023

    2023 arXiv

  12. [20]

    Taskmatrix.ai: Completing tasks by connecting foundation models with millions of apis

    Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix.ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434, 2023

    2023 arXiv

  13. [21]

    Unify: The complete llm platform, 2024

    Unify. Unify: The complete llm platform, 2024. Accessed: 2024-01-15. 9

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.