Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Advancing SLM Tool-Use Capability using Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that GRPO reinforcement learning, with a hand-designed reward that requires exact JSON, function names, and arguments, lifts tool-use accuracy of small language models by 6x to 21x on the xLAM evaluation set.

desk verdict Plausible idea, but the evaluation metric appears to be built from the same components as the training reward, so the reported 6x-21x gains don't yet establish real tool-use improvement. read the letter →

arxiv 2509.04518 v2 pith:PKL6HWM3 submitted 2025-09-03 cs.CL

Dhruvi Paprunia , Vansh Kharidia , Pankti Doshi This is my paper
classification cs.CL
keywords smalllanguagemodelsGRPOtoolusefunctioncallingreinforcementlearningrewardmodelingJSONoutputcapability-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that small language models (SLMs) can become reliable tool users without resorting to large-model-scale compute. It fine-tunes Qwen 2.5 1.5B, Qwen 2.5 3B, and Llama 3.2 3B with Group Relative Policy Optimization (GRPO) on 4,000 samples from the xLAM dataset, using a reward function that pays only for valid JSON, correct function names, and correct arguments, and returns zero for any extraneous text. The reported result is a jump in overall tool-use accuracy from 3.4% to 26.6% for the 1.5B model, 6.1% to 71.1% for the 3B Qwen, and 0.98% to 22.0% for Llama 3.2 3B, with JSON validity reaching 100%. If these numbers hold, GRPO is a compute-light route to equipping resource-constrained models with tool-use capability. The authors present the strict zero-reward penalty, not a complex shaped reward, as the decisive design choice.

What carries the argument

The load-bearing object is the hand-crafted reward function, formalized as R_final = R_fn + R_args + R_json, with R_final set to zero if any extraneous text exists. R_fn rewards each correct function name at 0.375 divided by the expected number of calls, R_args rewards correct argument ratios at 0.5 divided by the expected number of calls, and R_json adds 0.125 for valid JSON; extra predicted calls scale both main rewards down. This strict, capability-aware reward is optimized with GRPO, which replaces PPO's critic network with group-based advantage estimation, reducing memory cost during fine-tuning. The mechanism works by making partial successes worthless unless the full structured call i

What would settle it

Re-run the evaluation on a fresh, untouched sample from the same xLAM distribution, or on queries with tool lists shuffled, and check whether the 6x-21x gain persists; if accuracy drops to base-model levels, the original split or the exact-match reward was the source of the gain.

Watch

Extended reading notes

Core claim

The central claim is that a simple, capability-aware reward optimized by GRPO can teach small language models to emit tool calls that exactly match expected JSON: correct tool name, correct arguments, and no extra text. The reward starts by establishing valid JSON, then concentrates weight on argument correctness (0.5) over function-name correctness (0.375), with a strict zero for extraneous text and a scaling penalty for over-generating tool calls. On the xLAM evaluation split, the authors report overall accuracy improvements of roughly 6.28x, 10.65x, and 21.65x for Qwen 2.5 1.5B, Qwen 2.5 3B, and Llama 3.2 3B respectively, with JSON validity at 100% after training. The intended moral is th

Load-bearing premise

The reported gains depend on the 4,000/1,000 train/test split of the xLAM dataset being a faithful random split with no leakage, and on exact function-name and argument matching being a trustworthy measure of tool-use skill.

Editorial extensions

If this is right

  • If the reported gains are real, small models on edge hardware can handle function calling for real APIs after GRPO training on a few thousand examples.
  • The zero-reward extraneous-text penalty appears to eliminate a common tool-call failure mode: models that produce correct JSON but wrap it in explanation.
  • Because GRPO drops the critic network, the same recipe scales to larger models and longer tool-use sequences with less memory than PPO.
  • Near-perfect JSON validity across all three models suggests the reward correctly shapes output structure before task accuracy improves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report the random seed or split procedure; a natural extension is to test the same reward on multiple random splits of the xLAM data to see whether the 4,000/1,000 split is reproducing the gain.
  • If the exact-match metric is the driver, the recipe likely transfers to other tool-call benchmarks, but it may be gameable: a model could learn template-matching quirks of the dataset rather than general tool selection.
  • The strict zero-reward policy might be brittle on noisier real-world queries, where a graded partial reward could be needed for paraphrased but correct outputs; the paper argues the opposite is true.
  • Because only three small models are tested, the natural next experiment is application to 7B-13B models and to live APIs with multi-turn tool selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using Group Relative Policy Optimization (GRPO) with a hand-designed reward function to fine-tune small language models (Qwen 2.5 1.5B, Qwen 2.5 3B, Llama 3.2 3B) for tool-use. The reward decomposes accuracy into JSON validity, function-name correctness, and argument correctness, with a strict zero penalty for extraneous text. The authors report large improvements in 'Overall Accuracy' on a subset of SalesforceXLAM (6.28x, 10.65x, and 21.65x across the three models, Tables II–IV) and near-perfect JSON validity after GRPO training. They describe the reward as 'capability-aware' because its weights were adjusted based on observed training behavior. The central claim is that GRPO is a compute-efficient way to improve SLM tool-use capability.

Significance. If the reported gains could be taken at face value, the paper would make a useful practical contribution: a low-resource RL method for improving structured function-calling behavior in SLMs. The reward decomposition is clearly described and the strict zero-penalty mechanism is a sensible response to a common failure mode. However, in its current form the paper does not establish the stated contribution. The evaluation metric is built from the same components as the training reward, no independent or execution-based validation is provided, no supervised fine-tuning baseline is reported, and there are no confidence intervals, multiple seeds, or reproducibility artifacts. The empirical claims are therefore not supported as evidence of genuine tool-use improvement.

major comments (4)
  1. [Section VI.B vs Section IV.B, Eqs. (1)-(9)] The evaluation metric 'Overall Accuracy' is defined only as correctness of function name, arguments, and JSON structure. These are exactly the terms that define the training reward (Rjson, Rfn, Rargs, plus the zero penalty for extraneous text). No separate scoring procedure, external benchmark, or execution-based validation is described. Consequently Tables II–IV report the model's performance on the same objective it was trained to optimize; the 6.28x, 10.65x, and 21.65x gains do not by themselves establish improved tool-use capability. An independent evaluation—for example actual API/tool execution, schema validation, or a standard function-calling benchmark such as BFCL—is needed to break this circularity.
  2. [Section IV.A and V.A-B] The reward weights (0.125, 0.375, 0.5) and the zero-penalty policy were adjusted after observing the model's learning behavior on the training data. Since the same hand-tuned objective is then used as the evaluation metric, the comparison is not neutral: the metric has effectively been fitted to the training set. The paper should report multiple training seeds, a held-out reward-design procedure, or ablations over reward weights to show that the gains are robust and not an artifact of reward overfitting.
  3. [Section II.F and VI.A] The paper contrasts the proposed RL approach with 'conventional fine-tuning approaches' and claims that GRPO is preferable, but no supervised fine-tuning (SFT) baseline is reported. Without an SFT baseline trained on the same 4,000 samples and evaluated on the same 1,000 test samples, the paper cannot substantiate the claimed advantage of GRPO over standard fine-tuning. This is a load-bearing omission because the contribution is framed as an alternative to supervised fine-tuning.
  4. [Section III.B and Tables II-IV] No confidence intervals, number of runs, or random-seed/split details are reported. The test set has 1,000 samples, and single-run accuracies such as 26.6%, 71.1%, and 22.0% may be subject to non-negligible variance. The paper also does not release code, data, or checkpoints. These omissions make the quantitative claims unverifiable. Provide multi-seed statistics, a reproducible data split, and ideally release code/data or at least the exact split procedure.
minor comments (5)
  1. [Abstract/Introduction] 'LLMs are usually at tool use' appears to be a typo for 'LLMs are usually good at tool use'.
  2. [Tables II-IV] The text refers to 'Table 2', 'Table 3', and 'Table 4', but the tables are numbered II, III, and IV. Use consistent numbering.
  3. [Section VI.A.1] The model names are inconsistent: the text mentions 'Qwen 2.3 3B' and 'Qwen 2 1.5B Instruct', while the rest of the paper and tables use 'Qwen 2.5 1.5B' and 'Qwen 2.5 3B'. Please correct the model names.
  4. [References] Reference [10] contains a placeholder DOI ('10.1109/TPAMI.2024.1234567'); this should be updated to the actual DOI or removed.
  5. [Figure 2] Figure 2 is labeled 'Completion Length' but its role in the training narrative is not explained in the text. Please clarify what the plot shows and how it supports the 'Aha' moment claim.

Circularity Check

1 steps flagged · score 6.0 of 10

Evaluation metric is the training reward, so the reported accuracy gains are not independent evidence of tool-use improvement.

  1. self definitional [Section IV.B (reward definition) and Section VI.B (evaluation metrics)]
    "The formulas below were created to measure how accurately the model generates tool calls. [Section IV.B] ... Overall Accuracy: Measures the correctness of the predicted function name, arguments, and JSON structure. [Section VI.B]"

    Overall Accuracy is described using exactly the three components the GRPO reward is built from: valid JSON structure, correct function names (Eq. 3), and correct arguments (Eqs. 4-5), with Section IV.B explicitly stating that the reward formulas were created to measure tool-call accuracy. The reward weights were hand-tuned on the training data based on observed model behavior (Section IV.A and V.B), so the reported held-out accuracy is a measurement of the same objective the policy was trained to maximize. The 6.28x/10.65x/21.65x gains therefore do not establish improved tool-use capability independently of the reward; they show that GRPO increased the value of the hand-designed objective that also serves as the evaluation metric.

full rationale

The paper's central quantitative claim is an empirical RL result: GRPO fine-tuning improves a held-out accuracy metric. However, that metric is not independent. The reward function of Eqs. 1-9 and the 'Overall Accuracy' of Section VI.B share the same ingredients (JSON validity, function-name matching, argument matching), and Section IV.B says the reward formulas were created to measure tool-call accuracy. Since the reward weights were iteratively tuned to the models' training behavior (Section IV.A), the reported accuracy gains are largely a consequence of optimizing the metric being reported. This is a self-definitional evaluation rather than a derivation from first principles. No self-citation or imported uniqueness theorem is involved; the circularity is confined to the reward-as-metric design. A score of 6 reflects partial circularity: the training/evaluation loop is closed by construction, though the result still contains an empirical held-out component and would be strengthened by an external tool-execution benchmark or an independent, pre-registered accuracy metric.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on hand-chosen reward weights (0.125, 0.375, 0.5) that are tailored to the training data, and on the assumptions that GRPO is a valid optimizer and that the xLAM benchmark plus the reward-like accuracy metric measure tool-use faithfully. No new entities are introduced.

free parameters (3)
  • Rjson (JSON validity reward weight) = 0.125
    Reduced from 0.5 after observing that JSON formatting was learned quickly (Section IV.A); set by hand.
  • Rfn (function name reward weight) = 0.375
    Allocated based on observed difficulty; hand-chosen.
  • Rargs (argument correctness reward weight) = 0.5
    Assigned higher weight because argument names were found harder than function names (Section IV.A).
assumptions (3)
  • domain assumption GRPO is a stable and effective RL optimizer for language models
    Paper relies on Shao et al. (2024) and DeepSeek-R1 (2025); no comparison to other RL algorithms in this work.
  • domain assumption The Salesforce xLAM dataset and the defined accuracy metric correctly measure tool-use capability
    The evaluation uses a subset of xLAM and a metric composed of the same elements as the reward; no independent benchmark is used.
  • ad hoc to paper The hand-tuned reward weights and strict zero penalty do not cause reward hacking and generalize to the test split
    No analysis of reward hacking or robustness to reward weight changes is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing SLM Tool-Use Capability using Reinforcement Learning." pith.science (2026). https://pith.science/paper/PKL6HWM3

@misc{pith2026250904518,
  author       = {Pith},
  title        = {Pith review of: Advancing SLM Tool-Use Capability using Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKL6HWM3}},
  note         = {Machine review of arXiv:2509.04518}
}
read the original abstract

In an era where tool-augmented AI agents are becoming increasingly vital, our findings highlight the ability of Group Relative Policy Optimization (GRPO) to empower SLMs, which are traditionally constrained in tool use. The ability to use tools effectively has become a defining feature of Large Language Models (LLMs), allowing them to access external data and internal resources. As AI agents grow more sophisticated, tool-use capabilities have become indispensable. While LLMs have made significant progress in this area, Small Language Models (SLMs) still face challenges in accurately integrating tool use, especially in resource-constrained settings. This study investigates how Reinforcement Learning, specifically Group Relative Policy Optimization (GRPO), can enhance the tool-use accuracy of SLMs. By designing a well-defined reward system that reinforces structured JSON output, correct tool selection, and precise parameter usage, we demonstrate that GRPO enables SLMs to achieve significant improvements in tool-use capabilities (function calling/JSON output). Our approach provides a computationally efficient training method that enhances SLMs practical deployment in real-world AI applications.

Figures

Figures reproduced from arXiv: 2509.04518 by the authors.

Figure 1
Figure 1. Dataset Snapshot (GRPO), for fine-tuning Small Language Models. This ap￾proach trains SLMs to master structured tool use—ensuring valid JSON output, precise tool selection, and accurate param￾eter specification. It also penalizes extra text with a strict zero￾reward mechanism. This efficient, adaptable solution signifi￾cantly elevates SLM tool-use accuracy, making them reliable and scalable for real-world applicatio… view at source ↗
Figure 3
Figure 3. Qwen 2.5 1.5B Reward [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Comparison of Models Before and After GRPO training [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    FM-Agent is the first framework to automate compositional Hoare reasoning for large systems by having LLMs derive natural-language function specs from caller intent and then generate tests that found 522 new bugs in s...

  2. FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    FM-Agent automates top-down natural-language Hoare reasoning with LLMs and reports 522 new bugs in large, previously tested systems up to 143k LoC.

  3. UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    UniToolCall unifies tool-use data and evaluation for LLM agents, enabling fine-tuned models to reach 93% single-turn precision on a challenging benchmark with distractors.

  4. UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

    cs.AI 2026-04 conditional novelty 5.0 of 10

    A unified tool-learning pipeline (22k tools, 390k hybrid trajectories, QAOA eval) lets fine-tuned Qwen3-8B reach 93% single-turn Strict Precision under Hybrid-20, beating GPT/Gemini/Claude.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    xLAM: A Family of Large Action Models to Empower AI Agent Systems

    J. Zhanget al., "xLAM: A Family of Large Action Models to Empower AI Agent Systems." Available: https://github.com/SalesforceAIResearch/ xLAM

    work page

  2. [2]

    Small Language Models: Survey, Measurements, and Insights,

    Z. Luet al., "Small Language Models: Survey, Measurements, and Insights,"ArXiv, vol. abs/2409.15790, 2024. Available: https://api. semanticscholar.org/CorpusID:272832447

    arXiv 2024

  3. [3]

    A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness,

    F. Wanget al., "A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness,"ArXiv, vol. abs/2411.03350, 2024. Available: https://api.semanticscholar.org/ CorpusID:273850126

    arXiv 2024

  4. [4]

    A Survey of Small Language Models,

    C. V . Nguyenet al., "A Survey of Small Language Models,"ArXiv, vol. abs/2410.20011, 2024. Available: https://arxiv.org/abs/2410.20011

    arXiv 2024

  5. [5]

    A Survey on Large Language Model Based Autonomous Agents,

    L. Wang, C. Ma, X. Fenget al., "A Survey on Large Language Model Based Autonomous Agents,"Front. Comput. Sci., vol. 18, 2024. Available: https://doi.org/10.1007/s11704-024-40231-1

    doi:10.1007/s11704-024-40231-1 2024

  6. [6]

    Report on a General Problem-Solving Program,

    A. Newell, J. C. Shaw, and H. A. Simon, "Report on a General Problem-Solving Program," inIFIP Congress, 1959. Available: https: //api.semanticscholar.org/CorpusID:35199622

    work page 1959

  7. [7]

    Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,

    J. McCarthy, "Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,"Commun. ACM, vol. 3, no. 4, pp. 184–195, Apr. 1960. Available: https://doi.org/10.1145/367177. 367199pe

    work page doi:10.1145/367177 1960

  8. [8]

    Language Models are Unsupervised Multitask Learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language Models are Unsupervised Multitask Learners," 2019. Avail- able: https://api.semanticscholar.org/CorpusID:160025533

    work page 2019

Show all 24 references
  1. [9]

    ’Alexa, Do You Know Anything?’ The Impact of an Intelligent Assistant on Team Interactions and Creative Performance Under Time Scarcity,

    S. Shaikh and I. Cruz, "’Alexa, Do You Know Anything?’ The Impact of an Intelligent Assistant on Team Interactions and Creative Performance Under Time Scarcity," Dec. 2019. Available: https://doi.org/10.48550/ arXiv.1912.12914

    2019 doi arXiv

  2. [10]

    End-to- End Autonomous Driving: Challenges and Frontiers,

    L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li, "End-to- End Autonomous Driving: Challenges and Frontiers,"IEEE Transac- tions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10164–10185, Dec. 2024. Available: https://doi.org/10.1109/TPAMI. 2024.1234567

    2024

  3. [11]

    AI Agents That Matter,

    S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan, "AI Agents That Matter," arXiv preprint, 2024. Available: https://arxiv.org/ abs/2407.01502

    2024 arXiv

  4. [12]

    Toolformer: Language Models Can Teach Themselves to Use Tools,

    T. Schick, J. Dwivedi-Yu, R. Dessí, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, "Toolformer: Language Models Can Teach Themselves to Use Tools," inProc. 37th Int. Conf. Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, 2...

    2023

  5. [13]

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, "ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs," inProc. 12th Int. Conf. Le...

    2024

  6. [14]

    Training Language Models to Follow Instructions with Human Feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, "Training Language Models to Follow Instructions...

    2022

  7. [15]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo, "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models," arXiv preprint,

  8. [16]

    DeepSeek-R1: Incen- tivizing Reasoning Capability in LLMs via Reinforcement Learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, et al., "DeepSeek-R1: Incen- tivizing Reasoning Capability in LLMs via Reinforcement Learning," a...

    2025 arXiv

  9. [17]

    Proximal Policy Optimization Algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal Policy Optimization Algorithms,"ArXiv, vol. abs/1707.06347,

    arXiv

  10. [18]

    TinyAgent: Function Calling at the Edge,

    L. E. Erdoganet al., "TinyAgent: Function Calling at the Edge," inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Miami, Florida, USA, Nov. 2024, pp. 80–88. Available: https://aclanthology.org/2024. emnlp-demo.9/

    2024

  11. [19]

    Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks,

    G. Manduzio, F. Galatolo, M. G. C. A. Cimino, E. Scilingo, and L. Cominelli, "Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks,"ArXiv, Oct. 2024. Available: https://doi. org/10.48550/arXiv.2410.18890

    2024 doi arXiv

  12. [20]

    Qwen2.5 Technical Report,

    A. Yanget al., "Qwen2.5 Technical Report,"CoRR, vol. abs/2412.15115,

    arXiv

  13. [21]

    LLaMA: Open and Efficient Foundation Lan- guage Models,

    H. Touvronet al., "LLaMA: Open and Efficient Foundation Lan- guage Models," Feb. 2023. Available: https://doi.org/10.48550/arXiv. 2302.13971

    2023 doi arXiv

  14. [23]

    Available: https://doi.org/10.48550/arXiv.2412.15115

    doi arXiv

  15. [2017]

    Available: https://api.semanticscholar.org/CorpusID:28695052

  16. [2024]

    Available: https://arxiv.org/abs/2402.03300

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.