{"id":"99d341ac-57c9-4fd4-a6c5-e4d1c7909705","arxiv_id":"2505.00024","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Rule-based RL with a binary format-and-tool-match reward produces tool-calling LLMs that outperform GPT-4o on BFCL, API-Bank, and ACEBench, and can rival or beat SFT-then-RL under equal data budgets.","lead":"Researchers at NVIDIA show that rule-based reinforcement learning, with a simple binary reward for correct tool calls, can train language models to use external tools without human-written reasoning examples. The resulting 7B and 14B models score higher than GPT-4o on several function-calling benchmarks, though by small margins.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No error bars across any experiment; 'pure RL > SFT-then-RL' and 'clearly outperform GPT-4o' rest on 0.07–0.85-point single-run gaps that may be training noise.","rationale":"Good-faith reading: The paper proposes a sensible application of rule-based RL to tool calling, and the benchmark results are plausibly positive. The reward-proxy concern raised by the reader is real but secondary: even if the binary reward is biased, the empirical comparison is what it is, and the model's performance on benchmarks could still be valid. The more immediate threat to the central claim is the absence of any uncertainty quantification. All headline conclusions are based on single runs, and the margins are small. This is not unfalsifiable—it can be checked by multi-seed training—and it is the first thing that would settle whether the paradigm claim is real. The reader's conditions (release code, error bars, soften phrasing) already cover this; our read does not change the verdict, hence UNCHANGED. We set agreement_with_reader to partial because the reader's weakest_assumption was the reward proxy, whereas we locate the weakest point in the statistical support for the empirical claims.","tokens_in":17269,"tokens_out":9942,"duration_ms":99038,"concrete_test":"Retrain the four recipes in Table 3 with at least three independent seeds each (varying only the random seed; identical hyperparameters) and evaluate on BFCL Non-Live and Live. Report mean and standard deviation per recipe, and run a paired bootstrap or signed test between RL and Reason-SFT+RL. Separately, retrain Tool-N1-7B with two additional seeds and report the spread of overall BFCL accuracy; if the 0.85% margin over GPT-4o falls within that spread, the 'clearly outperform' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims rest on single-run point estimates with no uncertainty quantification. In Table 3, the key comparison between Reason-SFT+RL (83.17) and pure RL (83.24) is a 0.07-point difference; the largest gap in that table is 0.53 points. The main BFCL overall gap between Tool-N1-7B and GPT-4o is 0.85 points (84.82 vs 83.97). The paper reports no standard deviations, confidence intervals, or multiple training seeds. Since evaluation uses fixed test sets and presumably greedy decoding, the dominant variance is training stochasticity. Published BFCL evaluations show that such margins are commonly within ±1–2 points across runs. Thus the abstract's 'clearly outperform' and Finding 2's 'pure RL demonstrates stronger performance' are not supported by the evidence as presented. This is load-bearing because the paper's contribution is precisely the claim that rule-based RL with a binary reward is a superior training paradigm; if these gaps are noise, that conclusion collapses.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Tool-N1, a series of tool-calling LLMs trained with rule-based reinforcement learning (GRPO) using a binary reward that checks output format and exact tool-call matching, without requiring distilled reasoning traces. The authors evaluate on BFCL, API-Bank, and ACEBench, and compare SFT, RL, and SFT-then-RL training recipes on a curated 5,518-instance subset of ToolACE. They claim that Tool-N1-7B/14B clearly outperform GPT-4o and that pure RL matches or exceeds the SFT-then-RL paradigm for tool calling.","tokens_in":17385,"tokens_out":6133,"duration_ms":57990,"significance":"If the empirical claims hold, this is a useful result for the tool-learning community: it suggests that a simple verification reward can substitute for expensive SFT on distilled reasoning trajectories, and it provides a controlled comparison of training recipes. The paper uses external benchmarks with official scripts, evaluates across model scales and backbones, and includes ablations on reward granularity and data composition. The main limitation is that the headline comparisons rest on single-run point estimates with gaps as small as 0.07 points, so the significance of the central findings depends on the stability of those numbers.","major_comments":[{"comment":"No uncertainty quantification is provided for any experiment, and the central comparisons rest on single-run point estimates. The key recipe comparison in Table 3 shows pure RL (83.24) ahead of Reason-SFT+RL (83.17) by 0.07 points, and the main BFCL gap between Tool-N1-7B and GPT-4o is 0.85 points (Table 2). These margins are within typical run-to-run variability for such benchmarks, so the abstract's 'clearly outperform' and Finding 2's 'stronger performance' are not supported by the evidence as presented. Please report mean±std over at least three independent training seeds, or otherwise demonstrate that the gaps are stable.","section":"§5.1, Tables 2–3"},{"comment":"The phrasing 'does not necessarily outperform' is a tautology, while the stronger claim that pure RL 'demonstrates stronger performance than any other combinations' is contradicted by the 0.07-point gap in Table 3. Moreover, the recipe study uses a single 5,518-instance dataset and a single evaluation benchmark, so the general conclusion about the 'widely adopted SFT-then-RL paradigm' is over-generalized. Either provide additional evidence across datasets and benchmarks or temper the conclusion to 'pure RL is competitive with SFT-then-RL under equal data budgets.'","section":"§5.3.2, Finding 2"},{"comment":"The reward is described as assessing 'functional correctness' of tool invocations, but it actually grants reward 1 only on an exact match of the tool name and all argument key-value pairs against a single ground-truth call. If the annotation is noisy or more than one tool call would satisfy the query, this binary reward is a biased proxy for functional correctness. This matters because the claimed advantage over SFT is attributed to the reward's flexibility. Please either rename the criterion as 'exact match against the reference tool call' or add a robustness analysis (e.g., a multi-annotation study or evaluation of alternative correct calls).","section":"§4.3, Eq. (5)"}],"minor_comments":[{"comment":"The template example shows `[\"name\": <function-name>, ...]` which is not valid JSON; use the array-of-objects syntax `[{\"name\": ..., \"arguments\": ...}]` as in Appendix D.","section":"§4.2, Appendix E"},{"comment":"The sentence 'We utilize a subsets of ToolACE' is ungrammatical, and Table 3 does not state which benchmark the 'Avg' column refers to.","section":"§5.1"},{"comment":"The notation `𝒵 = {𝑧𝑖}𝐼 𝑖=𝑖` contains a typo; it should be `𝒵 = {𝑧𝑖}_{𝑖=1}^{𝐼}`.","section":"Eq. (1)"},{"comment":"The phrase 'an vertain response length' should be 'a certain response length', and Section 1 contains 'simple-tructured' which should be 'simple-structured'.","section":"§5.4"},{"comment":"Figures 5 and 6 would benefit from error bars or an explicit statement that each curve is a single run; without this, the reader cannot assess the stability of the scaling and backbone comparisons.","section":"Figures 5 and 6"},{"comment":"The header layout of Table 4 is difficult to parse; please clearly separate the fine-grained and binary reward columns with subheadings.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to interest the tool-learning community, and the controlled recipe comparison is a useful contribution. The main risk is that the headline claims are built on single-run point estimates; if the authors can provide multi-seed results or otherwise demonstrate stability, the contribution would be solid. I would also encourage the authors to strengthen Finding 2, as currently the 'does not necessarily outperform' wording is too weak to be a finding, while the stronger wording is not supported by the 0.07-point gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a genuinely useful empirical study, and the stress-test note is right that the headline claims are under-supported. Rule-based RL with a binary reward is applied to general multi-tool calling, not just a single search or code tool, and the paper includes a controlled comparison of No-Reason SFT, Reason-SFT, pure RL, and SFT-then-RL on a shared 5,518-instance dataset. That controlled comparison is the real contribution. It is also evaluated on external benchmarks with official scripts, which earns credit. The soft spots are real but not fatal. There are no error bars, confidence intervals, or multiple seeds anywhere. The central recipe comparison in Table 3 is a 0.07-point gap between pure RL (83.24) and Reason-SFT+RL (83.17), and the headline BFCL gap over GPT-4o is 0.85 points. Published BFCL evaluations show that margins like this are commonly within run-to-run noise. So the abstract's 'clearly outperform GPT-4o' and Finding 2's 'demonstrates stronger performance' are too strong for the evidence as presented. The abstract's 'does not necessarily outperform' is fine; the Finding is not. The promised code and exact data subsets are also unreleased, which is a concrete condition for reproducibility. One assumption worth naming: the binary reward gives full credit only on exact match to a single ground-truth tool call. If several tool calls are functionally valid, which is common, the reward is a biased proxy. That is a standard training objective and not circular, but it does mean the claimed RL generalization advantage depends on the quality of the xLAM/ToolACE ground truths. Overall the paper deserves a serious referee. The core finding, that rule-based RL with lightweight verification rewards is a viable tool-calling training paradigm, is plausible and well-motivated, and the controlled comparison is worth publishing even if the precise ordering of recipes shifts with uncertainty. I would send it to review, but I would require the authors to report training-seed variance or confidence intervals, release code and exact data subsets, and soften the comparative claims to match the actual evidence. I'd bring it to a reading group: it is timely, and the recipe comparison is a useful reference point.","headline":"A useful empirical study of rule-based RL for tool calling, but the headline claims rest on single-run gaps with no error bars, so treat them as provisional.","tokens_in":641,"tokens_out":666,"would_cite":true,"duration_ms":21720,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a language model to call tools with a binary reward that checks only format and exact match produces 7B and 14B models that outperform GPT-4o on function-calling benchmarks.","keywords":["tool calling","function calling","rule-based reinforcement learning","GRPO","binary reward","LLM reasoning","SFT vs RL","tool-use generalization"],"falsifier":"Construct a test set from queries that admit two or more equally valid tool calls for the same user intent; if a model trained with Eq. (5) fails on the alternative valid calls while a model trained with an execution-based verifier succeeds, the exact-match uniqueness assumption is the bottleneck.","tokens_in":16979,"feed_emoji":"🛠️","tokens_out":14535,"duration_ms":116174,"temperature":0.7,"pith_summary":"This paper asks whether a language model can learn to call external tools from a reward that only checks that its output has the right format and that the tool call exactly matches a single annotated answer, with no supervision on intermediate reasoning. The authors report that it can: models trained this way, Tool-N1-7B and Tool-N1-14B, outperform GPT-4o on the BFCL, API-Bank, and ACEBench benchmarks. They also compare supervised fine-tuning (SFT), reinforcement learning (RL), and their combination on 5,518 distilled reasoning trajectories, and find that pure RL matches or outperforms the widely used SFT-then-RL pipeline. If these results hold, lightweight verifiable rewards could replace expensive distilled reasoning annotations when training tool-using language models.","feed_headline":"Binary-reward RL lets 7B LLMs beat GPT-4o at tool calls","feed_subtitle":"No reasoning annotations needed; pure RL matches or beats the standard SFT-then-RL recipe.","key_machinery":"The load-bearing mechanism is the binary rule-based reward of Eq. (5) combined with the GRPO policy update. The reward is computed after parsing the model's output into two tagged components: a <think> block and a <tool_call> block. Format correctness requires both tags to be present, and tool-call correctness requires the parsed call to match the ground-truth dictionary in tool name and in every argument key-value pair, with argument order free to vary. This single sparse signal, which gives no reward to the reasoning text itself, is what pushes the policy to discover its own reasoning strategies, and the paper's ablation shows that a fine-grained reward giving partial credit for format or function-name matches hurts performance, which the authors attribute to reward hacking.","core_discovery":"This paper establishes that rule-based reinforcement learning with a binary reward is a strong training signal for tool-calling language models. The reward, defined in Eq. (5), returns 1 only if the model output wraps its reasoning in <think> tags, wraps the tool call in <tool_call> tags, and the parsed tool call exactly matches the ground-truth tool name and all argument key-value pairs; argument order is free to vary because matching is done on parsed dictionaries. The reasoning text itself receives no reward. Trained with the GRPO algorithm on data from xLAM and a ToolACE subset, the resulting models reach average accuracies of 84.82 (7B) and 85.97 (14B) on the Berkeley Function Calling Leaderboard (BFCL), against 83.97 for GPT-4o, and score 82.19 on API-Bank and 87.00 on ACEBench, again above GPT-4o. The paper further reports that in a controlled comparison on 5,518 distilled reasoning trajectories, pure RL (83.24 average BFCL) slightly edges out Reason-SFT+RL (83.17) and clearly beats both SFT variants, indicating that the commonly adopted SFT-then-RL recipe is not necessary for tool-calling performance.","pith_inferences":["A natural extension the paper does not test is applying the same binary exact-match reward to other agentic actions with a unique verifiable target, such as SQL generation or GUI automation.","Because the reward depends on clean, unique ground-truth annotations, deliberately corrupting a fraction of the training tool calls would likely erode the RL advantage; the paper's ablations do not vary annotation quality.","The SFT-vs-RL comparison uses only 5,518 trajectories, so the 'pure RL is enough' conclusion is conditional on data budget; with substantially more SFT data the ordering could reverse.","The flat response-length curve across training suggests tool calling rewards short, focused reasoning rather than long chains, which would make the optimal rollout budget for tool agents different from math-reasoning RL."],"forward_implications":["Tool-calling capability can be trained with a sparse binary reward, so curated reasoning traces are not a prerequisite for strong function-calling performance.","The widely used SFT-then-RL recipe is not automatically superior to pure RL for tool calling; practitioners can skip the SFT warm-up without losing accuracy.","Open 7B and 14B models trained with this recipe outperform a closed frontier model (GPT-4o) on standard function-calling benchmarks.","The method scales with model size and transfers across backbones, with Qwen-based models benefiting more than LLaMA-based ones at the same scale.","Longer reasoning traces are not necessary for tool-use quality; response length stayed flat during training, contrasting with math-reasoning RL runs."],"supporting_citations":[{"why":"Supplies the R1-style rule-based reinforcement learning paradigm and the GRPO objective that Tool-N1 trains with.","marker":"[7]"},{"why":"Provides the GRPO algorithm and rule-based reward design for mathematical reasoning that the paper adapts to tool calling.","marker":"[31]"},{"why":"One of the two training data sources (ToolACE subset) used for the RL training.","marker":"[17]"},{"why":"The other training data source (xLAM single-turn dataset) used for the RL training.","marker":"[49]"},{"why":"The main evaluation benchmark (BFCL) whose official scoring script the paper uses for its headline comparisons.","marker":"[41]"},{"why":"API-Bank benchmark used to evaluate the trained models in the Call mode.","marker":"[13]"},{"why":"ACEBench benchmark used for additional evaluation, and prior work on tool learning.","marker":"[2]"},{"why":"Prior investigation showing SFT may hinder R1-style training; the paper's SFT-vs-RL study directly builds on and contrasts with this finding.","marker":"[3]"}],"fun_headline_variants":["Binary-reward RL: 7B model tops GPT-4o on tool calls","Pure RL outperforms SFT+RL for tool calling","No annotations needed: RL wins tool calling","Tool-N1: Rule-based RL beats GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that every training query has exactly one correct tool call and that the dataset's ground-truth annotations are clean and complete, because the reward gives full credit only to an exact match with that single annotation.","fun_headline_variants_meta":{"raw":{"variants":["Binary-reward RL: 7B model tops GPT-4o on tool calls","Pure RL outperforms SFT+RL for tool calling","No annotations needed: RL wins tool calling","Tool-N1: Rule-based RL beats GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2904,"prompt_tokens":1027,"completion_tokens":1877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1806}},"tokens_in":643,"tokens_out":1877,"duration_ms":16342,"temperature":1.0,"reasoning_tokens":1806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:25:48.244463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set from queries that admit two or more equally valid tool calls for the same user intent; if a model trained with Eq. (5) fails on the alternative valid calls while a model trained with an execution-based verifier succeeds, the exact-match uniqueness assumption is the bottleneck.","supporting_citations":[{"cited_title":"Toolace: Winning the points of llm function calling.International Conference on Learning Representations, 2024","cited_arxiv_id":null,"evidence_quote":"One of the two training data sources (ToolACE subset) used for the RL training."},{"cited_title":"xlam: A family of large action models to empower ai agent systems","cited_arxiv_id":null,"evidence_quote":"The other training data source (xLAM single-turn dataset) used for the RL training."},{"cited_title":"Patil, Ion Stoica, and Joseph E","cited_arxiv_id":null,"evidence_quote":"The main evaluation benchmark (BFCL) whose official scoring script the paper uses for its headline comparisons."}],"review_version":1}