Pith. sign in

REVIEW 2 major objections 4 minor 55 references

Among 21 open-source LLMs, none solves more than 40% of web API invocation tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 15:12 UTC pith:QSPPGZ32

load-bearing objection A genuinely useful execution-based benchmark for LLM web API integration code, with a real capability gap finding that is probably robust in direction but fragile at the exact 40% threshold; worth engaging seriously. the 2 major comments →

arxiv 2509.20172 v7 pith:QSPPGZ32 submitted 2025-09-24 cs.SE cs.LG

Benchmarking Web API Integration Code Generation

Daniel Maninger , Leon Chemnitz , Amir Molzam Sharifloo , Jannis Brugger , Mira Mezini This is my paper
classification cs.SE cs.LG
keywords large language modelscode generationweb API integrationbenchmarkOpenAPIhallucinationfunctional correctnessAxios
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

WAPIIBench asks large language models to complete a short JavaScript program that must invoke a web API endpoint described by a natural-language comment, then executes the generated code in a sandbox and compares the captured HTTP request—method, URL, headers, query, and body—against a hand-curated ground truth. The central empirical finding is that this task is far harder for current models than local code completion: the best open-source model produces a fully correct request 30% of the time when it must choose the endpoint, and 40% when the endpoint is supplied, while the strongest commercial model reaches 60% and 77%. Models frequently hallucinate URLs and parameter names, with 14–39% of generated URLs being illegal and argument precision and recall lagging well behind what one would expect from a capable code model. The paper argues that web API integration is a distinct benchmark category because correctness depends on coordinating multiple argument locations against an externally documented specification, not on matching a single function signature.

Core claim

The authors claim that current open-source LLMs, when prompted with only the API name and a task comment, do not have reliable memorized knowledge of how to call web APIs. Across 395 tasks drawn from four real-world OpenAPI-specified services, the best open-source model (Code Llama 70B) produces an HTTP configuration that exactly matches the ground truth on 30% of full-completion tasks and 40% of argument-completion tasks; the strongest commercial model, GPT-4o, scores 60% and 77%. The error analysis shows the failures are not random syntax errors but systematic missteps: models pick URLs that are not defined in the specification (up to 39% of executable outputs), pass arguments the endpoint

What carries the argument

The central object is the request configuration c: a JSON structure modeling the HTTP method, full URL, headers, query parameters, and request body that a given task requires. The pipeline's novel step is its mock adapter, which executes the generated JavaScript in a sandbox, intercepts the outgoing Axios request, and serializes it into a captured configuration c′. Correctness is then computed by comparing each element of c′ with the ground-truth c (exact per-field match) and by validating c′ against the OpenAPI specification to separate 'illegal but plausible' from 'legal but wrong' outputs. This element-wise comparison yields the fine-grained metrics—correct URL, illegal URL, method correc

Load-bearing premise

The entire comparison rests on the premise that each task has exactly one correct HTTP request configuration and that exact match against that configuration is both necessary and sufficient for functional correctness; constraints that live only in prose, or tasks with multiple valid requests, would break the scoring.

What would settle it

Give each open-source model the 395 WAPIIBench tasks with the relevant OpenAPI specification excerpt appended to the prompt (without the ground truth). If the correct-implementation rate stays at or below 40%, the paper's ceiling reflects a task-level difficulty independent of closed-book recall; if it rises well above 40%, the ceiling is specifically a memorization limitation, and the benchmark's headline result would be expected to dissolve once context is supplied.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If a model cannot pick a correct endpoint from memory, API-integration assistants that rely on closed-book generation will keep producing failed requests; injecting the relevant OpenAPI spec via retrieval or explicit context becomes the obvious remediation, and the benchmark provides a way to measure its effect.
  • The two experimental setups give a clean decomposition: endpoint selection versus argument assembly. Because argument completion is substantially easier for all models, future progress can be attributed to one of these skills rather than confounded overall scores.
  • The fine-grained error metrics (illegal URLs, illegal arguments, missing arguments) define a concrete checklist for constrained decoding or static verification: a guard that rejects endpoints not present in the specification, or arguments not in the endpoint's schema, would immediately eliminate a large share of the observed failures.
  • Because the dataset contains one sample per endpoint and the evaluation is fully automated, the benchmark can be extended to new APIs, languages, or HTTP libraries without re-designing the correctness analysis; the configuration abstraction carries over.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The reported percentages measure only the outgoing request; the paper explicitly does not evaluate response handling. An end-to-end task that also required parsing the response would likely lower the success rates, so the 30–40% figures should be read as an upper bound on full integration correctness.
  • Editorial extension: Because the ground truth is a single exact configuration, two models that both solve a task but choose different valid parameter placements or equivalent header syntax would be scored as wrong. A human-equivalence rejudging of a sample of mismatched outputs would quantify how many 'incorrect' answers are actually acceptable requests.
  • Editorial extension: The benchmark only uses base (non-instruction-tuned) models and no in-context specification. Re-running the same pipeline with instruction-tuned models or with the OpenAPI spec in context would separate memorization failure from instruction-following failure; if the latter is the bottleneck, the 40% ceiling may not transfer to modern chat-based assistants.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces WAPIIBench, a benchmark with 395 samples covering four real-world OpenAPI-based APIs (Asana, Google Calendar, Google Sheets, Slack). Each sample pairs a natural-language task t with a hand-curated ground-truth request configuration c. The authors evaluate 21 open-source LLMs and two commercial models on JavaScript/Axios code completion in two setups (full completion and argument completion). Generated code is executed in a mock environment and the captured request configuration c' is compared to c via exact and element-wise metrics (Table II). The main empirical claim is that no evaluated open-source model solves more than 40% of tasks: Code Llama 70B reaches 30% (full completion) and 40% (argument completion), while GPT-4o reaches 60% and 77%, respectively. The paper also analyzes error types such as hallucinated endpoints, illegal arguments, and missing arguments, and releases the dataset and evaluation pipeline as open-source artifacts.

Significance. If the central claim is accepted, WAPIIBench is a valuable, first-of-its-kind benchmark for an underexplored task. The experimental design is generally careful: code is executed safely in a controlled environment, metrics distinguish total vs. executable samples, the dataset was manually curated (9 automated fixes, 58 manual fixes), and the dataset-generating model (Gemini 1.5 Pro) is excluded from the reported results. The public artifact and detailed per-API tables further support reproducibility. The central weakness is the reliance on exact configuration match as a proxy for functional correctness, which the authors acknowledge but do not validate against human judgment; this matters because the headline claim is stated in terms of solving tasks.

major comments (2)
  1. [§II-D, Table II; §II-A] The 'Correct implementations' metric is an exact match between the captured configuration c' and the hand-curated ground-truth c (Table II). The manual curation in §II-A establishes that the curated c is one correct configuration for each t, but it does not establish that it is the unique one. Since the headline claim (no open-source model >40%) is a binary statement over this metric, the lack of a human-validated re-judgment of non-matching outputs makes the reported rates lower bounds rather than demonstrated upper bounds. For Code Llama 70B in argument completion, correct implementations (t)=0.40 (≈158/395); re-judging the 237 non-matching outputs by human raters (with t and the OpenAPI spec) would determine whether any are functionally correct but configuration-divergent. Without this validation, the claim 'only 40% completely right' is not fully supported. A human re-judgment is a c
  2. [§II-C and §II-D; §III-C.2] The paper describes the evaluation as measuring 'functional correctness' (§II-C), but the automated analysis only captures the outgoing request configuration, not response handling, and it cannot enforce usage constraints that appear only in free-form textual descriptions in the API specification (§III-C.2). This gap is acknowledged in the limitations, but the abstract and conclusion use stronger language ('solve', 'completely right'). The metric is more precisely 'configuration-exact match' than 'functional correctness'. Either the terminology should be softened, or the human re-judgment proposed above should be used to show that exact match is necessary and sufficient for the task as stated. This is not just a wording issue: it affects how readers interpret the central quantitative claims.
minor comments (4)
  1. [Abstract] The sentence 'None of the evaluated open-source models was able to solve more than 40% of the tasks' should be qualified with 'under exact configuration matching' to avoid overstating the result.
  2. [Appendix F, Tables VI/VII] Some cells in the provided full result tables appear as concatenated values (e.g., '0.001.00' and '0.001' in the rendering). Ensure the camera-ready tables are properly typeset and each numeric entry is clearly separated.
  3. [§III-C.3] The dataset's synthetic nature—optional parameters used sparingly, placeholder values—limits transferability to real-world API integration, as the authors note. Consider stating more explicitly in the conclusion that the headline percentages apply to this synthetic task distribution, not to web API integration in general.
  4. [§IV] The related work claim 'We are not aware of an evaluation method able to provide results as fine-grained as ours' is strong. It is acceptable as a claim, but consider softening to 'to our knowledge' to match the surrounding style.

Circularity Check

0 steps flagged

No significant circularity: WAPIIBench evaluates independent model outputs against externally sourced, manually curated ground-truth configurations.

full rationale

The paper's central claim—that no evaluated open-source LLM solves more than 40% of WAPIIBench tasks—is an empirical measurement, not a derivation from the benchmark's own inputs. Ground-truth configurations c are synthesized from external OpenAPI specifications and then manually checked for consistency with the natural-language tasks t (Section II-A). The evaluated models are independent of the dataset generation process: Gemini 1.5 Pro, the model used to create the dataset, is explicitly excluded from the reported comparisons (Appendix F). Correctness is operationally defined as element-wise match of the captured request configuration c' to the hand-curated c (Section II-D). This is a standard benchmark evaluation procedure rather than a circular argument: the ground truth is not derived from the models under test, and no parameter is fitted to the model outputs. The acknowledged limitations—e.g., that free-form textual constraints in API specifications are not captured by the automated analysis and that tasks may in principle admit more than one valid configuration (Section III-C)—are threats to construct validity and external validity, not evidence of circularity. Self-citations in the related-work discussion (e.g., reference [48]) are not load-bearing for the empirical claims. Therefore, no circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claim involves no fitted parameters or hand-tuned constants; all reported numbers are measured. The listed axioms are the domain assumptions needed for the benchmark's validity, primarily around the definition of ground truth, the faithfulness of the mock execution, and the representativeness of the dataset. None of these introduces an entity or force a result.

axioms (6)
  • domain assumption The OpenAPI specifications used (Asana, Google Calendar, Google Sheets, Slack) are accurate enough, after manual fixes, to define a valid ground truth.
    Section II-A: the authors encountered and manually fixed errors in the real-world specifications, and correctness analysis validates configurations against the same specifications.
  • domain assumption Each natural language task t has exactly one correct request configuration c.
    Section II-A: manual checks were performed 'to ensure that there is one unique c for each t'; this uniqueness is required for exact-match correctness scoring.
  • domain assumption Functional correctness of API invocation code is fully captured by the outgoing HTTP request configuration; response handling is out of scope.
    Section III-C.1 explicitly states 'we evaluate only the correctness of the outgoing request, not how the incoming response is handled.'
  • domain assumption The Mock interception environment faithfully serializes the request that generated code would send, and the excess-code truncation heuristic does not distort results.
    Section II-C describes the Mock and the heuristic truncation; no validation against a real HTTP server is reported.
  • domain assumption Evaluating base (non-instruction-tuned) models with a single zero-shot prompt is a fair way to measure API-invocation ability.
    Appendix D justifies excluding instruction-tuned models because they produce unnatural completions; Section II-B chooses a single prompt for all models.
  • domain assumption The four selected APIs and 395 synthetic tasks are representative enough to draw general conclusions about LLM performance on web API invocation code.
    Section III-C.3 acknowledges the limited number of APIs and the synthetic nature of tasks, noting transferability may be limited.

pith-pipeline@v1.3.0-alltime-deepseek · 26409 in / 10589 out tokens · 76757 ms · 2026-08-04T15:12:42.299124+00:00 · methodology

0 comments
read the original abstract

API integration is a cornerstone of our digital infrastructure, enabling software systems to connect and interact. However, as shown by many studies, writing or generating correct code to invoke APIs, particularly web APIs, is challenging. Although large language models (LLMs) have become popular in software development, their effectiveness in automating the generation of web API integration code remains unexplored. In order to address this, we present WAPIIBench, a dataset and evaluation pipeline designed to assess the ability of LLMs to generate web API invocation code. Our experiments with several open-source LLMs reveal that generating API invocations poses a significant challenge, resulting in hallucinated endpoints, incorrect argument usage, and other errors. None of the evaluated open-source models was able to solve more than 40% of the tasks.

Figures

Figures reproduced from arXiv: 2509.20172 by Amir Molzam Sharifloo, Daniel Maninger, Jannis Brugger, Leon Chemnitz, Mira Mezini.

Figure 1
Figure 1. Figure 1: Benchmark design for evaluating the capabilities of LLMs in generating web API invocation code. 1) Based on an [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance comparison between selected models for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of a web API and its usage. Top: Excerpt from the Google Calendar OpenAPI specification. Left: JavaScript code to send a request to this API using the Axios library. Right: Configuration object that describes the request sent. For our evaluation, we pair the task description (comment in the JavaScript code) with the configuration to create an input–output sample. API required additional instruction… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 14 linked inside Pith

  1. [1]

    API management market by platform, service – global forecast to 2029,

    MarketsandMarkets, “API management market by platform, service – global forecast to 2029,” MarketsandMarkets Research Private Ltd.,

    2029

  2. [2]

    2024 state of the API report,

    Postman, “2024 state of the API report,” Postman, Inc., 2024. [Online]. Available: https://www.postman.com/state-of-api/2024/

    2024

  3. [3]

    What Makes APIs Hard to Learn? Answers from Developers,

    M. P. Robillard, “What Makes APIs Hard to Learn? Answers from Developers,”IEEE Softw., vol. 26, no. 6, pp. 27–34, 2009

    2009

  4. [4]

    A systematic mapping study of API usability evaluation methods,

    I. Rauf, E. Troubitsyna, and I. Porres, “A systematic mapping study of API usability evaluation methods,”Comput. Sci. Rev., vol. 33, pp. 49–68,

  5. [5]

    The effects of generative ai on high skilled work: Evidence from three field experiments with software developers,

    Z. K. Cui, M. Demirer, S. Jaffe, L. Musolff, S. Penget al., “The effects of generative ai on high skilled work: Evidence from three field experiments with software developers,”Available at SSRN 4945566, 2024

    2024

  6. [6]

    What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study,

    S. Dou, H. Jia, S. Wu, H. Zheng, W. Zhouet al., “What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study,”CoRR, vol. abs/2407.06153, 2024, arXiv: 2407.06153. [Online]. Available: https://doi.org/10.48550/arXiv.2407.06153

    doi:10.48550/arxiv.2407.06153 2024

  7. [7]

    Bugs in large language models generated code: an empirical study,

    F. Tambon, A. M. Dakhel, A. Nikanjam, F. Khomh, and o. Desmarais, “Bugs in large language models generated code: an empirical study,” Empir. Softw. Eng., vol. 30, no. 3, p. 65, 2025. [Online]. Available: https://doi.org/10.1007/s10664-025-10614-4

    doi:10.1007/s10664-025-10614-4 2025

  8. [8]

    Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions,” in43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022. IEEE, 2022, pp. 754–768

    2022

  9. [9]

    Do Users Write More Insecure Code with AI Assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do Users Write More Insecure Code with AI Assistants?” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023, W. Meng, C. D. Jensen, C. Cremers, and E. Kirda, Eds. ACM, 2023, pp. 2785–2799

    2023

  10. [10]

    AI copilot code quality: Evaluating 2024’s increased defect rate with data,

    GitClear, “AI copilot code quality: Evaluating 2024’s increased defect rate with data,” GitClear, 2025. [Online]. Available: https: //www.gitclear.com/ai assistant code quality 2025 research

    2024

  11. [11]

    Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?

    T. Y . Zhuo, X. Du, Z. Xing, J. Sun, H. Quanet al., “Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?”CoRR, vol. abs/2309.07804, 2023, arXiv: 2309.07804. [Online]. Available: https://doi.org/10.48550/arXiv.2309.07804

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.07804 2023

  12. [12]

    StarCoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkovet al., “StarCoder: may the source be with you!”Trans. Mach. Learn. Res., vol. 2023, 2023. [Online]. Available: https://openreview.net/forum?id=KoFOg41haE

    2023

  13. [13]

    StarCoder 2 and The Stack v2: The Next Generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirieret al., “StarCoder 2 and The Stack v2: The Next Generation,”CoRR, vol. abs/2402.19173, 2024, arXiv: 2402.19173. [Online]. Available: https://doi.org/10.48550/arXiv.2402.19173

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.19173 2024

  14. [14]

    DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Donget al., “DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence,”CoRR, vol. abs/2401.14196, 2024, arXiv: 2401.14196. [Online]. Available: https://doi.org/10.48550/arXiv.2401.14196

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.14196 2024

  15. [15]

    DeepSeek- Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence,

    DeepSeek-AI, Q. Zhu, D. Guo, Z. Shao, D. Yanget al., “DeepSeek- Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence,”CoRR, vol. abs/2406.11931, 2024, arXiv: 2406.11931. [Online]. Available: https://doi.org/10.48550/arXiv.2406.11931

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.11931 2024

  16. [16]

    Qwen2.5-Coder Technical Report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liuet al., “Qwen2.5-Coder Technical Report,”CoRR, vol. abs/2409.12186, 2024, arXiv: 2409.12186. [Online]. Available: https://doi.org/10.48550/arXiv.2409.12186

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.12186 2024

  17. [17]

    Code Llama: Open Foundation Models for Code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gatet al., “Code Llama: Open Foundation Models for Code,”CoRR, vol. abs/2308.12950, 2023, arXiv: 2308.12950. [Online]. Available: https: //doi.org/10.48550/arXiv.2308.12950

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.12950 2023

  18. [18]

    The Llama 3 Herd of Models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahleet al., “The Llama 3 Herd of Models,”CoRR, vol. abs/2407.21783, 2024, arXiv: 2407.21783. [Online]. Available: https://doi.org/10.48550/ arXiv.2407.21783

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  19. [19]

    The state of API 2020 report,

    SmartBear, “The state of API 2020 report,” SmartBear Software, 2020. [Online]. Available: https://smartbear .com/resources/ebooks/the-state-of- api-2020-report/

    2020

  20. [20]

    Most used programming languages among developers worldwide as of 2024,

    Statista, “Most used programming languages among developers worldwide as of 2024,” Statista, Inc., 2024. [Online]. Avail- able: https://www .statista.com/statistics/793628/worldwide-developer- survey-most-used-languages/

    2024

  21. [21]

    State of JavaScript 2022,

    S. Greif and E. Burel, “State of JavaScript 2022,” Devographics, 2022. [Online]. Available: https://2022.stateofjs.com/en-US/

    2022

  22. [22]

    Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm,

    L. Reynolds and K. McDonell, “Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm,” inCHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts, Y . Kitamura, A. Quigley, K. Isbister, and T. Igarashi, Eds. ACM, 2021, pp. 314:1–314:7

    2021

  23. [23]

    Out of the BLEU: How should we assess quality of the Code Generation models?

    M. Evtikhiev, E. Bogomolov, Y . Sokolov, and T. Bryksin, “Out of the BLEU: How should we assess quality of the Code Generation models?”J. Syst. Softw., vol. 203, p. 111741, 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111741

    arXiv 2023

  24. [24]

    Evaluating Large Language Models Trained on Code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pintoet al., “Evaluating Large Language Models Trained on Code,”CoRR, vol. abs/2107.03374, 2021, arXiv: 2107.03374. [Online]. Available: https://arxiv.org/abs/2107.03374

    Pith/arXiv arXiv 2021

  25. [25]

    When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Prevention,

    L. Guo, Y . Wang, E. Shi, W. Zhong, H. Zhanget al., “When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Prevention,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, M. Christakis and M. Pradel, Eds. ACM, 2024, pp. 1073–1085

    2024

  26. [26]

    CodeT5+: Open Code Large Language Models for Code Understanding and Generation,

    Y . Wang, H. Le, A. Gotmare, N. D. Q. Bui, J. Liet al., “CodeT5+: Open Code Large Language Models for Code Understanding and Generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Association for Computational Linguistics, 202...

    doi:10.18653/v1/2023.emnlp-main.68 2023

  27. [27]

    Building Natural Language Interfaces to Web APIs,

    Y . Su, A. H. Awadallah, M. Khabsa, P. Pantel, M. Gamonet al., “Building Natural Language Interfaces to Web APIs,” inProceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 - 10, 2017, E.-P. Lim, M. Winslett, M. Sanderson, A. W.-C. Fu, J. Sun, J. S. Culpepper, E. Lo, J. C. Ho, D. Donato, R. Agra...

    2017

  28. [28]

    Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs,

    Y . Su, A. H. Awadallah, M. Wang, and R. W. White, “Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs,” inThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, K. Collins-Thompson, Q. Mei, B. D. Davison, Y . Liu, and E. Yilmaz...

    2018

  29. [29]

    Compositional Generalization for Natural Language Interfaces to Web APIs,

    S. Hosseini, A. H. Awadallah, and Y . Su, “Compositional Generalization for Natural Language Interfaces to Web APIs,”CoRR, vol. abs/2112.05209, 2021, arXiv: 2112.05209. [Online]. Available: https: //arxiv.org/abs/2112.05209

    Pith/arXiv arXiv 2021

  30. [30]

    Dialog2API: Task-Oriented Dialogue with API Description and Example Programs,

    R. Shu, E. Mansimov, T. Alkhouli, N. Pappas, S. Romeoet al., “Dialog2API: Task-Oriented Dialogue with API Description and Example Programs,”CoRR, vol. abs/2212.09946, 2022, arXiv: 2212.09946. [Online]. Available: https://doi.org/10.48550/arXiv.2212.09946

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.09946 2022

  31. [31]

    On Mitigating Code LLM Hallucinations with API Documentation,

    N. Jain, R. Kwiatkowski, B. Ray, M. K. Ramanathan, and V . Kumar, “On Mitigating Code LLM Hallucinations with API Documentation,” in47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025. IEEE, 2025, pp. 237–248

    2025

  32. [32]

    Private- library-oriented code generation with large language models,

    D. Zan, B. Chen, Y . Gong, J. Cao, F. Zhanget al., “Private- library-oriented code generation with large language models,”Knowl. Based Syst., vol. 326, p. 113934, 2025. [Online]. Available: https://doi.org/10.1016/j.knosys.2025.113934

    arXiv 2025

  33. [33]

    De-Hallucinator: Iterative Grounding for LLM-Based Code Completion,

    A. Eghbali and M. Pradel, “De-Hallucinator: Iterative Grounding for LLM-Based Code Completion,”CoRR, vol. abs/2401.01701, 2024, arXiv: 2401.01701. [Online]. Available: https://doi.org/10.48550/ arXiv.2401.01701

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.01701 2024

  34. [34]

    Applying RLAIF for Code Generation with API-usage in Lightweight LLMs,

    S. Dutta, S. Mahinder, R. Anantha, and B. Bandyopadhyay, “Applying RLAIF for Code Generation with API-usage in Lightweight LLMs,” CoRR, vol. abs/2406.20060, 2024, arXiv: 2406.20060. [Online]. Available: https://doi.org/10.48550/arXiv.2406.20060

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.20060 2024

  35. [35]

    Survey on Evaluation of LLM-based Agents,

    A. Yehudai, L. Eden, A. Li, G. Uziel, Y . Zhaoet al., “Survey on Evaluation of LLM-based Agents,”CoRR, vol. abs/2503.16416, 2025, arXiv: 2503.16416. [Online]. Available: https://doi.org/10.48550/ arXiv.2503.16416

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.16416 2025

  36. [36]

    Toolformer: Language Models Can Teach Themselves to Use Tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeliet al., “Toolformer: Language Models Can Teach Themselves to Use Tools,” inAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann, A. Globerson, K. Saenko,...

    2023

  37. [37]

    RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs,

    Y . Song, W. Xiong, D. Zhu, C. Li, K. Wanget al., “RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs,”CoRR, vol. abs/2306.06624, 2023, arXiv: 2306.06624. [Online]. Available: https://doi.org/10.48550/arXiv.2306.06624

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.06624 2023

  38. [38]

    API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs,

    M. Li, Y . Zhao, B. Yu, F. Song, H. Liet al., “API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Association for Computational Linguistics, 2023, pp. 3102–3116. [Online]. ...

    doi:10.18653/v1/2023.emnlp-main.187 2023

  39. [39]

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yanet al., “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=dHng2O0Jjr

    2024

  40. [40]

    Gorilla: Large Language Model Connected with Massive APIs,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large Language Model Connected with Massive APIs,” inAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M...

    2024

  41. [41]

    Berkeley function calling leaderboard,

    F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil et al., “Berkeley function calling leaderboard,” Blog, 2024. [Online]. Available: https://gorilla .cs.berkeley.edu/blogs/8 berkeley function calling leaderboard.html

    2024

  42. [42]

    ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Models,

    H. Ding, S. Tao, L. Pang, Z. Wei, J. Gaoet al., “ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Models,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehva...

    2025

  43. [43]

    Unveiling Memorization in Code Models,

    Z. Yang, Z. Zhao, C. Wang, J. Shiet al., “Unveiling Memorization in Code Models,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering. Lisbon Portugal: ACM, Apr. 2024, pp. 1–13. [Online]. Available: https://dl.acm.org/doi/10.1145/3597503.3639074

    arXiv 2024

  44. [44]

    Exploring and Evaluating Hallucinations in LLM-Powered Code Generation,

    F. Liu, Y . Liu, L. Shi, H. Huang, R. Wanget al., “Exploring and Evaluating Hallucinations in LLM-Powered Code Generation,”CoRR, vol. abs/2404.00971, 2024, arXiv: 2404.00971. [Online]. Available: https://doi.org/10.48550/arXiv.2404.00971

    doi:10.48550/arxiv.2404.00971 2024

  45. [45]

    SAFEWAPI: web API misuse detector for web applications,

    S. Bae, H. Cho, I. Lim, and S. Ryu, “SAFEWAPI: web API misuse detector for web applications,” inProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE- 22), Hong Kong, China, November 16 - 22, 2014, S.-C. Cheung, A. Orso, and M.-A. D. Storey, Eds. ACM, 2014, pp. 507–517

    2014

  46. [46]

    A Large-scale Study on API Misuses in the Wild,

    X. Li, J. Jiang, S. Benton, Y . Xiong, and L. Zhang, “A Large-scale Study on API Misuses in the Wild,” in14th IEEE Conference on Software Testing, Verification and Validation, ICST 2021, Porto de Galinhas, Brazil, April 12-16, 2021. IEEE, 2021, pp. 241–252

    2021

  47. [47]

    APICAD: Augmenting API Misuse Detection through Specifications from Code and Documents,

    X. Wang and L. Zhao, “APICAD: Augmenting API Misuse Detection through Specifications from Code and Documents,” in45th IEEE/ACM In- ternational Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 245–256

    2023

  48. [48]

    jGuard: Program- ming Misuse-Resilient APIs,

    S. Binder, K. Narasimhan, S. Kernig, and M. Mezini, “jGuard: Program- ming Misuse-Resilient APIs,” inProceedings of the 15th ACM SIGPLAN International Conference on Software Language Engineering, SLE 2022, Auckland, New Zealand, December 6-7, 2022, B. Fischer, L. Burgue ˜no, and W. Cazzola, Eds. ACM, 2022, pp. 161–174

    2022

  49. [49]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,

    P. S. H. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” inAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, ...

    2020

  50. [50]

    Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search,

    C. Hokamp and Q. Liu, “Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, R. Barzilay and M.-Y . Kan, Eds. Association for Computational Linguistics, 2017, pp. 1535–1546

    2017

  51. [51]

    A General-Purpose Algorithm for Constrained Sequential Inference,

    D. Deutsch, S. Upadhyay, and D. Roth, “A General-Purpose Algorithm for Constrained Sequential Inference,” inProceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, M. Bansal and A. Villavicencio, Eds. Association for Computational Linguistics, 2019, pp. 482–492. [Online]. Available: ...

    doi:10.18653/v1/k19-1045 2019

  52. [2019]

    Available: https://doi.org/10.1016/j.cosrev.2019.05.001

    [Online]. Available: https://doi.org/10.1016/j.cosrev.2019.05.001

    doi:10.1016/j.cosrev.2019.05.001 2019

  53. [2020]

    Available: https://proceedings .neurips.cc/paper/2020/ hash/6b493230205f780e1bc26945df7481e5-Abstract.html

    [Online]. Available: https://proceedings .neurips.cc/paper/2020/ hash/6b493230205f780e1bc26945df7481e5-Abstract.html

    2020

  54. [2023]

    Available: http://papers .nips.cc/paper files/paper/2023/ hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html

    [Online]. Available: http://papers .nips.cc/paper files/paper/2023/ hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html

    2023

  55. [2024]

    Available: https://www .marketsandmarkets.com/Market- Reports/api-management-market-178266736.html

    [Online]. Available: https://www .marketsandmarkets.com/Market- Reports/api-management-market-178266736.html