Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Stepwise validated generation of synthetic tool-use data lifts small models to match or beat zero-shot 32B models on function calling.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:45 UTC pith:LXZXW7IK

load-bearing objection Useful open framework for function-calling data generation; the role-wise ablation is the strong result, but the BFCL headline leans on in-distribution data. the 3 major comments →

arxiv 2607.29250 v1 pith:LXZXW7IK submitted 2026-07-31 cs.CL

Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation

Goutham Ramakrishnan , Megha Sharma This is my paper
classification cs.CL
keywords function callingsynthetic data generationsmall language modelsdata qualitytool usemulti-turn agentschain-of-thoughtdomain adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the quality of synthetic training data, not model size, is the binding constraint on small language models' ability to call functions and act as agents. It answers yes, and introduces Data Turnstile, an open framework that generates tool-use training data by decomposing each interaction into a directed graph of typed roles—user query, thinking trace, API call, execution output, assistant response—each generated and validated separately with error-feedback retries. The thesis is that this stepwise, validate-before-generate discipline produces data so clean that a 0.6B-parameter model fine-tuned on it approaches or beats zero-shot models many times larger. Concretely, a Turnstile-trained Qwen3-0.6B scores 75.9% on the BFCL single-turn benchmark (versus 67.4% for its base model with thinking), and a 1.7B model trained on Turnstile data about the Telecom domain surpasses a zero-shot 32B model on the τ2-bench agentic benchmark (31.1% vs 27.4%). The paper also finds that chain-of-thought reasoning helps multi-turn diagnosis but hurts single-turn calls, a nuance with practical implications for when to enable thinking.

Core claim

The central claim is that a method for generating synthetic function-calling data can substitute for model capacity: small models fine-tuned on Turnstile-generated data perform on par with or better than zero-shot models up to 53 times larger on tool-use benchmarks. The mechanism is the interaction template—a directed acyclic graph of roles with typed outputs—that forces every API call to be schema-valid, every observation to be well-formed, and every reasoning trace to be checked by a validate-before-generate step, with retries on failure and early aborts for irrecoverable problems. The authors show the methodology effect directly: on the same API pool, Turnstile-generated data improves a 0

What carries the argument

The central object is the interaction template: a DAG (V,E,Θ) where each node is a role (USER, API CALL, API OBS, ASSISTANT, THINKING), edges encode dependencies, and Θ supplies generation context such as API definitions and personas. The framework generates roles sequentially, validating each before generating the next, retrying with error feedback on failures, and optionally aborting early. It also defines 'Issues' and 'Scenarios' to compose policy-adherent multi-turn workflows, and supports dynamic perturbations (simulated failures, clarification requests). The validated stepwise loop is what turns a fallible teacher model into a reliable data producer, achieving ~84% generation success w

Load-bearing premise

The whole pipeline rests on the teacher model (here Qwen2.5-32B-Instruct) generating semantically correct role content and on the LLM-as-judge validation catching hallucinations; if the teacher or judge systematically produces or passes flawed API observations and reasoning traces, downstream SFT gains disappear even though the structural decomposition is sound.

What would settle it

Generate a Turnstile dataset with a teacher model known to hallucinate API parameters (or with the judge disabled), fine-tune the same SLM, and check whether BFCL/τ2-bench scores fall back to base levels; if a structurally valid but semantically corrupted dataset still yields large gains, the stepwise validation is not the active ingredient. Conversely, hand-verify a random sample of Turnstile API observations for semantic consistency with the interaction; if false success declarations or oracle leakage appear at rates near the single-shot baseline, the validate-before-generate claims are over

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Small models trained on Turnstile data can close the gap to large models on function calling, enabling low-latency, on-device agentic deployments without proprietary APIs.
  • Turnstile's execution-free validation means teams can generate training data for private or custom APIs without a working backend, unlike execution-verified pipelines.
  • The framework provides fine-grained control over complexity, API diversity, and refusal behavior (irrelevance injection), allowing targeted fixes for known failure modes such as hallucinated parameters and over-triggering.
  • The task-dependent role of chain-of-thought suggests a practical rule: disable thinking for single-turn calls, enable it for multi-step diagnosis.
  • Since the framework is open-source and runs with open-weight teachers, synthetic data generation can be done on-premise and iterated rapidly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the result generalizes across model families beyond the Qwen3 series (which the authors note as future work), data-quality engineering would become a first-class alternative to scaling for agentic tasks.
  • The validate-before-generate principle may transfer to other sparse-supervision domains, such as code generation or structured output constraints, where a decomposition into typed roles with per-step checks could reduce hallucination.
  • Because the teacher and judge are themselves language models, the framework's ceiling is set by their competence; a direct test would be to generate data with a deliberately weak teacher and measure how quickly downstream gains erode.
  • The finding that CoT hurts single-turn function calling suggests that reasoning traces should be curriculum- or task-gated, not uniformly included, and that 'thinking mode' is not a free lunch.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes Data Turnstile, an open-source framework for generating synthetic function-calling training data. The central idea is to decompose a multi-turn tool-use interaction into a DAG of typed generation roles (user, reasoning, API call, API observation, assistant), generate each role independently with constraints and per-step validation, and retry with error feedback on failure. The authors argue this improves data quality and diversity over single-shot generation and execution-verified pipelines, while using only open-weight teacher models. They evaluate by fine-tuning Qwen3-0.6B/1.7B/4B on Turnstile-generated data and testing on BFCL v3 single-turn and tau2-bench Telecom multi-turn. Headline results are that Qwen3-0.6B fine-tuned on Turnstile OOD+ID data reaches 75.9% on BFCL (base think: 67.4%; Qwen3-4B think: 79.9%), and Turnstile-trained SLMs beat zero-shot Qwen2.5-32B on the tau2-bench Telecom domain. The framework and a ~100K-interaction dataset are released.

Significance. If the claims hold, the contribution is practically significant: an open-source, policy-aware data synthesis pipeline that works with non-frontier teachers, with explicit structure and diversity controls, is a useful addition to the tool-use data-generation toolkit. The paper has real strengths: the controlled single-shot ablation in Appendix C isolates the role-wise decomposition; the tau2-bench results include confidence intervals; the diversity metrics (Table 3) provide a quantitative comparison against open-source data; and the released code/data make the approach reproducible. The Limitations section appropriately acknowledges that generated-data quality is bounded by the teacher LLM. However, the central 'data quality compensates for model capacity' claim is currently presented with an in-distribution BFCL result as its headline, and the key BFCL comparison lacks uncertainty quantification.

major comments (3)
  1. [Abstract; §3.2, Table 2; Conclusion] The headline BFCL number 75.9% is the Turnstile-OOD+ID condition, where ID is Turnstile data generated from BFCL's own API definitions. The OOD-only condition is 72.9%, which is still a real gain over the base think-mode 0.6B (67.4%) but leaves a 7.0pp gap to Qwen3-4B (79.9%), not the 4.0pp gap implied by the abstract. The +3pp from ID data is plausibly schema/format memorization rather than general data quality. The paper is transparent in Table 2, but the abstract and conclusion do not carry the OOD-only caveat. Please report the OOD-only result as the headline or state the ID/OOD decomposition explicitly in the abstract and conclusion.
  2. [§3.3, Table 2, Table 1] The claim that the Raw-OS vs. Turnstile-OS comparison is the 'most controlled' and that the +15.3pp gain is 'attributable to generation methodology alone' is not supported by the listed experimental details. Raw-OS lacks CoT traces, while Turnstile-OS contains them; Table 1 shows much larger volume for Turnstile data (xLAM: 230K vs 59K; Glaive: 199K vs 78K); and §3.1 adds distractor APIs during Turnstile training. Any of these factors could contribute to the gain. Please run a volume/CoT-matched ablation or explicitly acknowledge these confounds and restate the conclusion accordingly.
  3. [Table 2] The BFCL results are reported without confidence intervals or significance tests, unlike the tau2-bench results in Table 4. The BFCL per-category test sizes are small (visible in the single-digit percentages), so key deltas such as OOD 72.9% vs OOD+ID 75.9% (+3.0pp) and OS 70.4% vs OOD 72.9% (+2.5pp) may be within sampling noise. The absence of CIs is particularly consequential because the abstract's 'approaching Qwen3-4B' claim rests on these small differences. Please add bootstrap confidence intervals or per-item significance tests, or soften the claims accordingly.
minor comments (4)
  1. [Abstract] The phrase 'fine-tuned on Turnstile data' for the 75.9% result should specify 'Turnstile OOD+ID data' to match Table 2.
  2. [§3.4] The human assessment reports scores of 4.4/3.8/4.5 on 100 interactions, but does not state how the interactions were sampled, whether annotations were independent, or inter-annotator agreement. Please add these details.
  3. [§4.3] The tool-call weighted SFT gains of +1.6 to +3.9pp are all within the 95% CIs. The text says the positive impact 'suggests this direction is worth exploring,' which is appropriately cautious; consider avoiding the phrase 'consistent gains' earlier in the same paragraph, since the effect is not statistically significant.
  4. [Appendix C] The single-shot baseline is run without retries, whereas the role-wise comparison includes retries (96.4% vs 89.4% pass rates). The quality differences may partly reflect the retry mechanism rather than role decomposition per se. Please state this explicitly or add a no-retry role-wise baseline.

Circularity Check

0 steps flagged

No circularity: benchmark results are empirical and externally evaluated; BFCL in-domain caveat is disclosed and does not reduce to a fitted input.

full rationale

The paper's claims are empirical SFT results on external benchmarks, not derivations from fitted constants or from an ansatz imported via self-citation. The headline BFCL number (75.9%) uses Turnstile data generated from BFCL's own API schemas, which is disclosed in §3.2 as Experiment 3 ('in-distribution (ID) Turnstile data, generated from BFCL's APIs') and reported in Table 2 as 'Turnstile-OOD+ID'. This is in-domain training, not circularity: the synthetic data is generated from API definitions, not from the benchmark's test items, and the paper separately reports an OOD-only result (72.9%) and a controlled methodology comparison (Turnstile-OS 70.4% vs Raw-OS 55.1%, same APIs, different generation method). The τ2-bench Telecom experiments likewise use data synthesized from the policy document and 14 APIs, which is the intended domain-adaptation setting, not an input-output tautology. No load-bearing self-citation chain, uniqueness theorem, or ansatz smuggled via citation is present. The acknowledged limitation that generated data quality depends on the teacher LLM is a correctness/falsifiability caveat, not a circular step. The BFCL in-domain configuration weakens the abstract's generalization framing somewhat, but it is transparently disclosed and does not make the reported prediction equivalent to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new theoretical entities are postulated. The framework's components (roles, templates, Issues/Scenarios) are software abstractions validated empirically through benchmark results and ablations, not unfalsifiable constructs.

free parameters (3)
  • Template distribution weights = 50% single-API single-turn; 25% multi-API single-turn; 25% multi-turn (example in §2.3)
    Chosen by hand to control data mix; this distribution affects diversity and downstream benchmark performance, but is not fitted to the benchmark.
  • Tool-call SFT loss multiplier = 5
    Applied 5x loss on API call tokens; 'multiplier selected via validation' (§4.2). This hyperparameter is fit to the τ2-bench validation set and affects reported gains.
  • Retry budget
    Predefined but unspecified retry limit per role; affects success rate and data quality (§2.2, §3.5). Chosen by hand.
axioms (4)
  • domain assumption The teacher model (Qwen2.5-32B-Instruct) can generate semantically correct role outputs for arbitrary API specs and policies.
    Paper's Limitations state 'quality of generated data depends on teacher LLM capability' (§6). If the teacher produces flawed observations or reasoning, data quality collapses regardless of structural validation.
  • domain assumption LLM-as-judge validation reliably detects hallucinated parameters and implausible observations.
    Validate-before-generate (§2.2) relies on the LLM to catch issues beyond structural checks; §3.5 reports only 13% of errors were caught by LLM-as-judge, so this mechanism's reliability is load-bearing for hallucination control.
  • domain assumption The teacher model has not memorized the evaluation benchmarks, so generated data does not leak test answers.
    No contamination analysis is provided. If Qwen2.5-32B-Instruct has seen BFCL/τ2-bench tasks in pretraining, benchmark gains could be inflated.
  • domain assumption BFCL and τ2-bench are valid proxies for real-world function-calling performance.
    The paper's practical conclusions rely on these benchmarks measuring skills that matter for deployment: schema compliance, policy adherence, multi-turn diagnosis.

pith-pipeline@v1.3.0-daily-deepseek · 16995 in / 18630 out tokens · 165200 ms · 2026-08-03T10:45:36.288510+00:00 · methodology

0 comments
read the original abstract

Small language models (SLMs) are attractive for agentic deployment due to low latency, reduced cost, and on-device privacy, yet they struggle with tool-use tasks where training data is scarce and noisy. Unlike larger models, SLMs cannot compensate for low-quality supervision through sheer capacity, making data quality the critical bottleneck. We present Data Turnstile, an open-source framework that takes user-defined API specifications and generates high-quality synthetic training data for function calling. Turnstile decomposes multi-turn tool-use interactions into constrained, stepwise generation with validation and error-feedback loops, providing fine-grained control over API diversity, conversation complexity, and output correctness. We demonstrate effectiveness of domain adaptation with Turnstile data on two challenging function calling benchmarks. On the BFCL single-turn benchmark, a Qwen3-0.6B fine-tuned on Turnstile data without chain-of-thought achieves 75.9% overall accuracy (versus 67.4% for the base model with thinking enabled), closing the gap with thinking-enabled Qwen3-1.7B (78.4%) and Qwen3-4B (79.9%) despite being 3$\times$ and 7$\times$ smaller respectively. On $\tau^2$-bench, a multi-turn agentic benchmark, Turnstile-trained Qwen3-1.7B achieves 31.1% pass^1 on the Telecom domain, improving 4.7$\times$ over its 6.6% base and surpassing Qwen2.5-32B-Instruct (27.4%), a model 19$\times$ larger. Turnstile-trained Qwen3-0.6B achieves 24.6%, improving 7$\times$ over its 3.5% base and approaching the 32B model (53$\times$ larger). We release Data Turnstile along with a dataset spanning 1,000+ APIs and 100K+ multi-turn interactions.

Figures

Figures reproduced from arXiv: 2607.29250 by Goutham Ramakrishnan, Megha Sharma.

Figure 1
Figure 1. Figure 1: τ 2 -bench multi-turn results: Domain adaptation with Turnstile data helps SLMs beat zero-shot 32B models. Gaps. Existing approaches for generating synthetic tool-use data face several limitations. End-to-end generation methods such as ToolBench [11] typically prompt an LLM to produce entire multi-turn conversations in a ‘single￾shot’, i.e. through a single call to the LLM. This leads to quality degradatio… view at source ↗
Figure 2
Figure 2. Figure 2: Data Turnstile Overview: role-wise generation with per-step validation and error-feedback based retry. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multi-turn data generation for τ 2 -bench Telecom: reusable Issues compose into Scenarios with dynamic perturbations. API CALL, API OBS, ASST), capturing variety in interaction patterns; (d) Arg Richness: argument-value variety per tool. It is the fraction of unique argument-value combinations when sampling n calls per tool (repeated 50 times), averaged across all qualifying tools. We choose n=10 to retain… view at source ↗
Figure 4
Figure 4. Figure 4: τ 2 -bench Telecom: Error Analysis across different dimensions for Turnstile SFT models (0.6B, 1.7B, 4B). capabilities critical for multi-turn diagnosis: (1) hypothesis formation (reasoning about possible causes before act￾ing), (2) evidence-based progression (interpreting API results to confirm or reject hypotheses), and (3) multi-issue discovery (after resolving one problem, reasoning that additional iss… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 3 linked inside Pith

  1. [1]

    Function calling in large language models: Industrial practices, challenges, and future directions.ACM Comput

    Maolin Wang, Yingyi Zhang, Bowen Yu, Bingguang Hao, Cunyin Peng, Yicheng Chen, Wei Zhou, Jinjie Gu, Chenyi Zhuang, Ruocheng Guo, Wanyu Wang, and Xiangyu Zhao. Function calling in large language models: Industrial practices, challenges, and future directions.ACM Comput. Surv., 58(9), February 2026

    2026

  2. [2]

    Lane, and Mengwei Xu

    Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Demystifying small language models for edge deployment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistic...

    2025

  3. [3]

    Small language models are the future of agentic ai.ArXiv, abs/2506.02153, 2025

    Peter Belc ´ak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Lin, and Pavlo Molchanov. Small language models are the future of agentic ai.ArXiv, abs/2506.02153, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  5. [5]

    Gonzalez

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 11 Data Turnstile: A Scalable Open Framework for Function-Calling Data GenerationA PREPRINT

    2024

  6. [6]

    Hammer: Robust function-calling for on-device lan- guage models via function masking, 2024

    Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, Jun Wang, and Weinan Zhang. Hammer: Robust function-calling for on-device lan- guage models via function masking, 2024

    2024

  7. [7]

    xLAM: A family of large action models to empower AI agent systems

    Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Zhiwei Liu, Yihao Feng, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xLAM: A family of large action models to empower AI agent systems. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Confere...

    2025

  8. [8]

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning.ArXiv, abs/2206.14486, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Shah, Harkirat Singh Behl, Xin Wang, S´ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuan-Fang Li

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C’esar Teodoro Mendes, Allison Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, S. Shah, Harkirat Singh Behl, Xin Wang, S´ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuan-Fang Li. Textbooks are all you need.ArXiv, abs/2306.11644, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    APIGen: Automated PIpeline for generating verifiable and diverse function- calling datasets

    Zuxin Liu, Thai Quoc Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R N, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. APIGen: Automated PIpeline for generating verifiable and diverse function- calling datasets. InThe Thirty-eight Confe...

    2024

  11. [11]

    On the tool manipulation capability of open-source large language models, 2023

    Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models, 2023

    2023

  12. [12]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gon- zalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InAdvances in Neural Information Processing Systems, 2024

    2024

  13. [13]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ 2-bench: Evaluating conversa- tional agents in a dual-control environment, 2025

    2025

  14. [14]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA,

  15. [15]

    Denny Zhou, Nathanael Sch ¨arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  16. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  17. [17]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Association

    2022

  18. [18]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Arney Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Efficient llm inference via chunked prefills.SIGOPS Oper. Syst. Rev., 59(1):9–16, August 2025

    2025

  19. [19]

    Glaive function calling v2 dataset.https://huggingface.co/datasets/glaiveai/ glaive-function-calling-v2, 2023

    Glaive AI. Glaive function calling v2 dataset.https://huggingface.co/datasets/glaiveai/ glaive-function-calling-v2, 2023

    2023

  20. [20]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    2025

  21. [21]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY , USA, 2020. Association for Computing Machinery

    2020

  22. [22]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022

    2022

  23. [23]

    APIGen-MT: Agentic pipeline for multi-turn data generation via simulated agent- human interplay

    Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Manoj Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Quoc Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong. APIGen-MT: Agentic pipeline for multi-turn data generation via simulated agent- human interplay. InThe Thirty-ninth Annual Co...

    2026

  24. [24]

    Magnet: Multi-turn tool-use data synthesis and distillation via graph translation

    Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, and Tomas Pfister. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meetin...

    2025

  25. [25]

    ToolFlow: Boosting LLM tool-calling through natural and coherent dialogue synthesis

    Zezhong Wang, Xingshan Zeng, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. ToolFlow: Boosting LLM tool-calling through natural and coherent dialogue synthesis. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com...

    2025

  26. [26]

    Funreason-mt technical report: Advanced data synthesis solution for real-world multi-turn tool-use, 2025

    Zengzhuang Xu, Bingguang Hao, Zechuan Wang, Yuntao Wen, Xinyi Xu, Yang Liu, Long Chen, Dong Wang, Maolin Wang, Tong Zhao, Yicheng Chen, Cunyin Peng, Jinjie Gu, Leilei Gan, Xiangyu Zhao, Chenyi Zhuang, and Shi Gu. Funreason-mt technical report: Advanced data synthesis solution for real-world multi-turn tool-use, 2025

    2025

  27. [27]

    Toolweave: Structured synthesis of complex multi-turn tool-calling dialogues, 2026

    Dinesh Khandelwal, Gnana Prakash Punnavajhala, GPS Bhargav, Gaurav Pandey, Sachin Joshi, Hima Karanam, and Dinesh Raghu. Toolweave: Structured synthesis of complex multi-turn tool-calling dialogues, 2026

    2026

  28. [28]

    Dis- tilabel: An ai feedback (aif) framework for building datasets with and for llms.https://github.com/ argilla-io/distilabel, 2024

    ´Alvaro Bartolom´e Del Canto, Gabriel Mart ´ın Bl´azquez, Agust´ın Piqueres Lajar´ın, and Daniel Vila Suero. Dis- tilabel: An ai feedback (aif) framework for building datasets with and for llms.https://github.com/ argilla-io/distilabel, 2024

    2024

  29. [29]

    Data synthesizer for function calling.https://github.com/Azure-Samples/ function-calling-data-synthesizer, 2025

    Azure-Samples. Data synthesizer for function calling.https://github.com/Azure-Samples/ function-calling-data-synthesizer, 2025. Accessed: 2026-07-22

    2025

  30. [30]

    Camel: Communicative agents for ”mind” exploration of large language model society

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for ”mind” exploration of large language model society. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  31. [31]

    Lastras, and Pavan Kapanipathi

    Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, and Pavan Kapanipathi. NEST- FUL: A benchmark for evaluating LLMs on nested sequences of API calls. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet P...

    2025

  32. [32]

    T-eval: Evaluating the tool utilization capability of large language models step by step

    Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. T-eval: Evaluating the tool utilization capability of large language models step by step. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Comp...

    2024

  33. [33]

    Can you order me a large pepperoni pizza from McDonald’s?

    Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: a benchmark for general tool agents.Advances in Neural Information Processing Systems, pages 75749–75790, 2024. 13 Data Turnstile: A Scalable Open Framework for Function-Calling Data GenerationA PREPRINT A Think vs No-Think: Qualitative Analysis We present qualitati...

    2024

  34. [2022]

    Curran Associates Inc