Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned small language models turn natural-language requests into correct JSON function calls 85-87 percent of the time; zero-shot and few-shot prompting mostly fail at format adherence.

desk verdict Useful empirical measurements with a believable core, undermined by test-set leakage in the few-shot arm and a prompt-injection condition that is really just input noise. read the letter →

arxiv 2504.19277 v1 pith:XDTKCH2S submitted 2025-04-27 cs.AI cs.SE

Ishan Kavathekar , Raghav Donakanti , Ponnurangam Kumaraguru , Karthik Vaidhyanathan This is my paper · ORCID
classification cs.AIcs.SE
keywords smalllanguagemodelsfunctioncallingfine-tuningLoRApromptinjectionJSONformatadherenceedgedeploymentlatency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Function calling asks a model to convert a natural-language request plus a list of tool descriptions into a structured JSON call that a system can execute. This paper tests whether small language models can do that reliably, comparing zero-shot prompting, three-example few-shot prompting, and fine-tuning on five small models, and then measuring resistance to prompt injection and performance on an edge device. The central result is that fine-tuning makes two of the five models strong function-call generators—DeepSeek-Coder-1.3B reaches 99.44 percent JSON parsability and 85.43 percent task accuracy, Phi-3-mini reaches 99.62 percent and 87.27 percent—while the other three models fail even after fine-tuning because their outputs never parse as JSON. The stake is practical: if format adherence can be secured, small models could run function calling locally on phones and embedded devices instead of relying on large cloud models.

What carries the argument

The central mechanism is the structured JSON output contract: every prompt instructs the model to answer with a JSON function call, and the evaluation separates syntactic compliance from semantic correctness using metrics for JSON parsability, function selection, argument completeness, and argument value correctness. LoRA (low-rank adaptation) fine-tuning on 55,000 function-calling samples is the intervention that internalizes the output contract for DeepSeek-Coder and Phi-3-mini. The edge experiments use GGUF, a quantized weight format, at 4-bit precision, which is what makes deployment on the edge device memory-feasible.

What would settle it

Check the fine-tuning and test splits for near-identical or paraphrastic queries; if overlap exists, rerun the evaluation on a freshly collected disjoint test set and see whether task accuracy stays near 85-87 percent. Separately, rerun the few-shot condition with examples taken from the training split rather than the test split and compare against the reported 55.65 percent task accuracy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that format adherence, not task understanding, is the main obstacle for small models. The two successful models already know enough to pick the right function and arguments after fine-tuning; what fine-tuning adds is the ability to emit well-formed JSON, lifting parsability from near zero to above 99 percent and task accuracy to 85-87 percent. Prompt injection with a random character string costs them only about 1-2 percentage points, whereas the few-shot setting loses 13-16 percent on the same perturbation. On an edge device, 4-bit quantization cuts memory by roughly a factor of five but latency grows by an order of magnitude, so the models remain usable but not yet real-time.

Load-bearing premise

The reported numbers assume the evaluation queries were unseen during pretraining and that the three few-shot examples drawn from the test set do not leak answers to the queries being scored.

Editorial extensions

If this is right

  • Two small models can act as local function-calling engines, reaching about 85-87 percent task accuracy with JSON parsability above 99 percent.
  • Zero-shot and few-shot prompting are not a reliable path for small models on this task, because most outputs fail the JSON format before semantics are even scored.
  • Fine-tuning does not guarantee format adherence: three of the five models remain at zero on JSON parsability after the same recipe.
  • Prompt-injection resistance improves with fine-tuning, with only 1-2 percent drops compared with 13-16 percent in the few-shot setting.
  • Edge deployment is memory-feasible but latency-bound: 4-bit quantization cuts memory about fivefold, while inference time rises roughly tenfold over server hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the format-adherence failure suggests a cheap, model-agnostic fix—constrained decoding or grammar-guided generation could push the three failing models above zero without fine-tuning.
  • Inference: because the three few-shot examples were selected from the test set, the reported few-shot gains should be read as upper bounds until the condition is rerun with training-set examples.
  • Inference: the 1-2 percent prompt-injection drop covers only a single simple attack; multi-turn or indirect injections could degrade fine-tuned models much more.
  • Inference: the results support a system design in which specialized fine-tuned small models handle function calls locally and only ambiguous or out-of-distribution queries are routed to larger models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This exploratory empirical study evaluates five small language models (1.3B-3.8B parameters) on function-call generation using zero-shot prompting, few-shot prompting, and LoRA fine-tuning, with and without a condition the authors call 'prompt injection'. The authors also deploy GGUF-quantized versions of selected models on a Qualcomm edge device and report latency and memory usage. The headline results are that zero-shot performance is near zero for all models, few-shot substantially improves DeepSeek-Coder, and fine-tuning yields high JSON parsability and task accuracy for DeepSeek-Coder-1.3B (99.44% parsability, 85.43% task accuracy) and Phi-3-mini (99.62% parsability, 87.27% task accuracy). The paper additionally reports that fine-tuned models degrade only 1-2% under the 'prompt injection' condition, and it releases the fine-tuned models and a replication package.

Significance. If the reported results hold, the paper provides a useful empirical data point for a genuinely important question: whether small, locally deployable models can be made usable for structured function-call generation. The authors deserve credit for releasing fine-tuned models and a replication package, and for measuring latency and memory on real edge hardware, which is uncommon in this literature. However, two of the study's headline claims are currently threatened by methodological issues: the few-shot examples are drawn from the test set (Section 4.4), and the so-called prompt-injection condition is actually input noise rather than an injection attack. These issues do not by themselves invalidate the fine-tuned accuracy numbers, but they do invalidate the few-shot comparison and the prompt-injection robustness conclusion as currently stated. With those experiments corrected or relabeled, the study could be a useful contribution to the empirical SE/NLP literature on small-model function calling.

major comments (3)
  1. [Section 4.4, Table 2] The few-shot condition uses three examples that, by the authors' own description, were 'carefully selected to cover representative scenarios within the test set' and held constant across all datapoints. Because the same test set is used for evaluation, these few-shot prompts expose the model to ground-truth task/answer patterns from the evaluation distribution; the reported few-shot gains (e.g., DeepSeek-Coder task accuracy rising from 1.11% to 55.65% and JSON parsability from 7.34% to 89.38%) are therefore inflated and cannot be interpreted as genuine in-context learning. RQ2 and the zero-shot-to-few-shot comparison in Section 6 rest on this measurement. The authors should either draw the examples from a held-out split that is disjoint from the test set or re-report the few-shot results without test-set examples.
  2. [Section 4.4, Table 2, RQ3] The 'prompt injection' condition consists of appending a string of non-sensical random characters (alphanumeric, special, and unicode) to the user query. No adversarial instruction is added, so the condition tests robustness to input noise, not prompt injection as defined in Section 2.4 (where injection involves 'instructions or adversarial text'). Consequently, the 1-2% drops under this condition do not support the claims in RQ3, the abstract, and Section 6 that fine-tuned models are robust to prompt injection attacks. The authors should either conduct genuine injection attacks (e.g., explicit override instructions) or relabel the experiment as noise/perturbation robustness and temper the corresponding claims.
  3. [Section 5 (RQ4), Table 4] The text states that DeepSeek-Coder requires 5385.89MB on the server, but Table 4 lists 2,570.24MB for DeepSeek-Coder on the server, and 5385.89MB does not match any row in the table. This is an internal inconsistency in the memory comparison that supports the '5x less memory on edge' claim. Please correct the figure and ensure all memory numbers in the text agree with the table.
minor comments (6)
  1. [Table 2, Correct Ratio row] For DeepSeek-Coder zero-shot, Correct Ratio (0.0470) exceeds Task Accuracy (0.0111); since Correct Ratio is the proportion of datapoints with Task Accuracy exactly 1, this is arithmetically impossible if both are computed on the same denominator. Please clarify the denominators or fix the numbers.
  2. [Section 4.4] The statement that few-shot and fine-tuning experiments used a temperature 'identical to that of zero-shot' is vague; state the value (presumably 0) explicitly for all settings.
  3. [Tables 2 and 3] Labeling and formatting are inconsistent across tables (e.g., 'Parsibility' vs 'Parsibility', 'Finetuned' vs 'Fine-tuned', '0.468' vs '0.4680', '0.001' vs '0.0010'); standardize these for readability.
  4. [Section 4.5] The AVC metric applies exact equality to argument values, but values in the dataset may be lists or nested objects; please state how equality is computed for such cases.
  5. [Figure 3] Figure 3 is referenced in Section 6 after the failure patterns are already described in Section 5; consider introducing it where the patterns are first mentioned, and ensure the cropped model outputs are legible in the camera-ready version.
  6. [Section 7] The external-validity discussion acknowledges possible test-data exposure for pretrained models, but it does not discuss the test-set-sourced few-shot examples; that threat should be acknowledged and mitigated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the study reports direct empirical measurements on a fixed test set, and no predicted quantity reduces by construction to a fitted input or to a self-citation.

full rationale

This paper is an empirical benchmark study, not a derivation from a model or theory. The central quantities (JSON Parsability, Task Accuracy, Correct Ratio, FSP, ACS, AVC; Eqs. 1-3) are defined as counting/F1 aggregates over model outputs on a held-out 5,000-sample test set, and the fine-tuned models are trained on a separate 55,000-sample subset before being evaluated on that test set. There is no equation or fitting step in which a reported 'prediction' is algebraically identical to an input, nor any parameter that is fitted to the target metric and then re-reported as a result. The few-shot condition does present a real methodological threat: Section 4.4 states that the three examples 'were carefully selected to cover representative scenarios within the test set,' so the few-shot prompt contains test-set content and RQ2's few-shot numbers may be inflated by leakage. This is a benchmark-validity and data-leakage problem, not circular reasoning: the measured outputs are still genuine model outputs, and the leakage does not make the metric equal to the input by construction. Likewise, the 'prompt injection' condition appends non-sensical random characters to the user prompt; that is a weak operationalization of prompt injection and a construct-validity concern, but it is not circular. The citations to the Salesforce XLAM dataset and EvalPlus leaderboard are external sources of data and model selection, and no load-bearing claim is justified solely by a self-citation. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard empirical evaluation assumptions: the dataset's ground truth is correct, the curated subset is representative, and the metrics are meaningful. No free parameters or invented entities are introduced; the LoRA hyperparameters are standard choices, not fitted to the data.

assumptions (4)
  • domain assumption The Salesforce XLAM dataset provides correct ground truth function calls.
    The study relies on this dataset's answers as ground truth for all metrics; if the answers are wrong, all accuracy numbers are unreliable.
  • domain assumption The curated subset of 5,000 test samples is representative of the full dataset's diversity.
    The paper filters samples to fit context lengths and does not report how many were removed; if the filtering introduces bias, results may not generalize.
  • domain assumption The evaluation metrics capture function-call correctness.
    Task Accuracy is an F1 over predicted function calls, computed only on JSON-parseable outputs; it is a reasonable but imperfect proxy, and ignoring unparsable outputs could skew the results.
  • domain assumption EvalPlus leaderboard ranking is a valid proxy for function-calling capability.
    The paper selects models based on coding benchmarks; if coding skill does not transfer to function calling, the model selection is biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling." pith.science (2026). https://pith.science/paper/XDTKCH2S

@misc{pith2026250419277,
  author       = {Pith},
  title        = {Pith review of: Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDTKCH2S}},
  note         = {Machine review of arXiv:2504.19277}
}
read the original abstract

Function calling is a complex task with widespread applications in domains such as information retrieval, software engineering and automation. For example, a query to book the shortest flight from New York to London on January 15 requires identifying the correct parameters to generate accurate function calls. Large Language Models (LLMs) can automate this process but are computationally expensive and impractical in resource-constrained settings. In contrast, Small Language Models (SLMs) can operate efficiently, offering faster response times, and lower computational demands, making them potential candidates for function calling on edge devices. In this exploratory empirical study, we evaluate the efficacy of SLMs in generating function calls across diverse domains using zero-shot, few-shot, and fine-tuning approaches, both with and without prompt injection, while also providing the finetuned models to facilitate future applications. Furthermore, we analyze the model responses across a range of metrics, capturing various aspects of function call generation. Additionally, we perform experiments on an edge device to evaluate their performance in terms of latency and memory usage, providing useful insights into their practical applicability. Our findings show that while SLMs improve from zero-shot to few-shot and perform best with fine-tuning, they struggle significantly with adhering to the given output format. Prompt injection experiments further indicate that the models are generally robust and exhibit only a slight decline in performance. While SLMs demonstrate potential for the function call generation task, our results also highlight areas that need further refinement for real-time functioning.

Figures

Figures reproduced from arXiv: 2504.19277 by the authors.

Figure 1
Figure 1. Overview of the study design illustrating different components of the research methodology. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Sample datapoint from the Salesforce-XLAM [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Responses of Deepseek-Coder and Phi-3-mini across settings highlighting incorrect ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of latency (in sec) and Memory usage [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWEnergy: An Empirical Study on Energy Efficiency in Agentic Issue Resolution Frameworks with SLMs

    cs.SE 2025-12 conditional novelty 6.0 of 10

    Small-language-model agents on existing issue-resolution frameworks solve almost no SWE-bench tasks and spend most of their energy in failed, repetitive loops; framework design, not model size, dominates the energy bill.

Reference graph

Works this paper leans on

69 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gu- nasekara, et al. 2024. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00121 (2024)

    arXiv 2024

  2. [2]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ah- mad Awan, Nguyen Bach, Amit Bahree, and Arash Bakhtiari et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL] https://arxiv.org/abs/2404.14219

    arXiv 2024

  3. [3]

    Divyansh Agarwal, Alexander Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, and Chien-Sheng Wu. 2024. Prompt Leakage effect and mitigation strategies for multi- turn LLM Applications. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track , Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimori...

    doi:10.18653/v1/2024.emnlp-industry 2024

  4. [4]

    Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang

  5. [5]

    Muhammad Azeem Akbar, Arif Ali Khan, Najmul Islam, and Sajjad Mahmood

    work page

  6. [6]

    Satanjeev Banerjee and Alon Lavie. 2004. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. Proceedings of ACL-WMT (2004), 65–72

    work page 2004

  7. [7]

    Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Ka- panipathi, and Luis A Lastras. 2024. API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs. arXiv preprint arXiv:2402.15491 (2024)

    arXiv 2024

  8. [8]

    Hezekiah J Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi

Show all 69 references
  1. [9]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, ...

    2020

  2. [10]

    Saheed A Busari and Emmanuel Letier. 2017. Radar: A lightweight tool for require- ments and architecture decision analysis. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) . IEEE, 552–562

    2017

  3. [11]

    Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering (1994), 528–532

    1994

  4. [12]

    Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu

  5. [13]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205.14135

    2022 arXiv

  6. [14]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318–30332

    2022

  7. [15]

    Georgi Gerganov. [n. d.]. GGML. https://github.com/ggerganov/ggml

  8. [16]

    Georgi Gerganov. 2023. GGUF. https://github.com/ggerganov/ggml/blob/master/ docs/gguf.md

    2023

  9. [17]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Int...

    2023

  10. [18]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543 (2023)

    2023 arXiv

  11. [19]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wen- feng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196 ...

    2024 arXiv

  12. [20]

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. StableToolBench: Towards Stable Large- Scale Benchmarking on Tool Learning of Large Language Models. arXiv preprint arXiv:2403.07714 (2024)

    2024 arXiv

  13. [21]

    Zixu Hao, Huiqiang Jiang, Shiqi Jiang, Ju Ren, and Ting Cao. 2024. Hybrid SLM and LLM for Edge-Cloud Collaborative Inference. Proceedings of the Workshop on Edge and Mobile Foundation Models (2024). https://api.semanticscholar.org/ CorpusID:270405086

    2024

  14. [22]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  15. [23]

    Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme

    J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019. Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Associa...

    2019

  16. [24]

    Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al . 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog 1, 3 (2023), 3. Sm...

    2023

  17. [25]

    Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 1161–1173

    2021

  18. [26]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  19. [27]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih ...

    2021

  20. [28]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: ...

    2021 doi

  21. [29]

    Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. Cctest: Testing and repairing code completion systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1238–1250

    2023

  22. [30]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  23. [31]

    Yishuai Lin, Philippe Descamps, Nicolas Gaud, Vincent Hilaire, and Abderrafiaa Koukam. 2015. Multi-agent system for intelligent scrum project management. Integrated Computer-Aided Engineering 22, 3 (2015), 281–296

    2015

  24. [32]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.n...

    2023

  25. [33]

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24) . 1831–1847

    2024

  26. [34]

    Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024. APIGen: Automated Pipeline for Ge...

    2024 arXiv

  27. [35]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, and et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv:2402.19173 [cs.SE] https://arxiv.org/abs/2402.19173

    2024 arXiv

  28. [36]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al . 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)

    2021 arXiv

  29. [37]

    Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. 2023. Developer- intent driven code comment generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 768–780

    2023

  30. [38]

    Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul Kim, Hanieh Deil- amsalehy, Namyong Park, Mike Rimer, Zhehao Zhang, Huanrui Yang, Ryan A

    Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zheng- mian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, Junda Wu, Ashish Singh, Yu Wang, Jiuxiang Gu, Franck Dernoncourt, Nesreen K. Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul...

    2024 arXiv

  31. [39]

    NousResearch. 2023. Nous-Hermes-13b. https://huggingface.co/NousResearch/ Nous-Hermes-13b

    2023

  32. [40]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  33. [41]

    Eugenio Parra, Jose Luis de la Vara, and Luis Alonso. 2018. Analysis of require- ments quality evolution. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings . 199–200

    2018

  34. [42]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Go- rilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334 (2023)

    2023 arXiv

  35. [43]

    Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, and Nathan Cooper. [n. d.]. Stable Code 3B. [https://huggingface.co/ stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b)

  36. [44]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023)

    2023 arXiv

  37. [45]

    Murray Shanahan. 2024. Talking about large language models. Commun. ACM 67, 2 (2024), 68–79

    2024

  38. [46]

    Yisheng Song, Ting Wang, Puyu Cai, Subrota K Mondal, and Jyoti Prakash Sahoo

  39. [47]

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301 (2023)

    2023 arXiv

  40. [48]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et al. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621 (2024)

    2024 arXiv

  41. [49]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024. A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhanceme...

    2024 arXiv

  42. [50]

    Yiwei Wang, Muhao Chen, Nanyun Peng, and Kai-Wei Chang. 2024. DeepEdit: Knowledge Editing as Decoding with Constraints. arXiv:2401.10471 [cs.CL]

    2024 arXiv

  43. [51]

    Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) 53, 3 (2020), 1–34

    2020

  44. [52]

    Simon Willison. 2023. Delimiters won’t save you from prompt injection. https: //simonwillison.net/2023/May/11/delimiters-wont-save-you/

    2023

  45. [53]

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders Wesslén, et al. 2012. Experimentation in software engineering . Vol. 236. Springer

    2012

  46. [54]

    Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao

  47. [55]

    Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. 2024. On protecting the data privacy of large language models (llms): A survey. arXiv preprint arXiv:2403.05156 (2024)

    2024 arXiv

  48. [56]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley Function Calling Leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_ leaderboard.html

    2024

  49. [57]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    2022 arXiv

  50. [58]

    Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023. Tool- coder: Teach code generation models to use api search tools. arXiv preprint arXiv:2305.04032 (2023)

    2023 arXiv

  51. [59]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  52. [60]

    Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. 2024. A Review on Edge Large Language Models: Design, Execution, and Applications. arXiv:2410.11845 [cs.DC] https://arxiv.org/abs/2410.11845

    2024 arXiv

  53. [61]

    arXiv:2402.18649 [cs.CR] https://arxiv.org/abs/2402.18649

    A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems. arXiv:2402.18649 [cs.CR] https://arxiv.org/abs/2402.18649

    arXiv

  54. [62]

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics 12 (2024), 1556–1577

    2024

  55. [68]

    Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. 2025. ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario. arXiv preprint arXiv:2501.10132 (2025)

    2025 arXiv

  56. [2021]

    arXiv preprint arXiv:2103.06333 (2021)

    Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333 (2021)

    2021 arXiv

  57. [2022]

    arXiv preprint arXiv:2209.02128 (2022)

    Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv preprint arXiv:2209.02128 (2022)

    2022 arXiv

  58. [2023]

    A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. Comput. Surveys 55, 13s (2023), 1–40

    2023

  59. [2024]

    Software: Practice and Experience 54, 2 (2024), 257–280

    DevOps project management success factors: A decision-making frame- work. Software: Practice and Experience 54, 2 (2024), 257–280

    2024

  60. [2025]

    arXiv:2408.05002 [cs.SE] https://arxiv.org/abs/2408.05002

    An Empirical Study on Challenges for LLM Application Developers. arXiv:2408.05002 [cs.SE] https://arxiv.org/abs/2408.05002

    arXiv

  61. [3059]

    doi:10.18653/v1/2021.emnlp-main.243

    2021 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.