{"id":"b75f5248-a483-468e-aa5b-a65e14f8de58","arxiv_id":"2504.14870","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A tool-use efficiency reward for RL-trained LLM agents reduces tool calls by up to 68% and improves tool productivity by up to 215% without significant accuracy loss.","lead":"This paper proposes a reinforcement learning reward that penalizes unnecessary tool calls while still rewarding correct answers, and applies it to language model agents that use search and code tools. The method cuts tool use by up to 68 percent and raises a new efficiency metric, tool productivity, by up to 215 percent, while keeping accuracy roughly unchanged.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline search results are confounded by a prompt change: OTC-PO's 40–68% tool-call reductions on NQ/HotpotQA run with a modified system prompt that explicitly instructs fewer calls, while Search-R1 baselines use the original prompt; no ablation separates reward from instruction.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing confound, and the manuscript's own Appendix C confirms the prompt wording. I therefore agree with the reader. The concern is not an internal mathematical inconsistency or an out-of-consensus position; it is an experimental control problem: the independent variable (reward) is changed together with a highly relevant instruction. Because the prompt explicitly tells the model to make every search call count and to answer directly when no further knowledge is needed, it plausibly produces the observed 40–68% TC reduction even without OTC-PO's reward. The code experiments, where the template is unchanged, are a genuine mitigating data point and should be credited; they prevent the method from being entirely unsupported. However, the paper's headline efficiency figures and most of the search analysis rely on the confounded comparison, so the central empirical claim that the reward drives the tool-call reduction is not established. The theoretical appendix D.2 also leans on an external paper [34] and does not prove that the product reward preserves accuracy under finite-sample optimization, but the prompt confound alone is decisive. The appropriate verdict remains the reader's REJECT; the required fix is a concrete prompt×reward ablation, after which a CONDITIONAL or ACCEPT could be reconsidered.","tokens_in":18553,"tokens_out":6361,"duration_ms":57315,"concrete_test":"Run a 2×2 ablation on Qwen2.5-7B-Base for NQ and HotpotQA, with three seeds each: (A) Search-R1 correctness-only reward with the original Search-R1 template; (B) Search-R1 reward with the Appendix C template; (C) OTC-PO reward with the original template; and (D) OTC-PO reward with the Appendix C template (the paper's configuration). If condition (B) alone yields TC near 1.0 and TP near 0.43 on NQ, the prompt explains the headline reduction and the reward adds little; if (B) remains near Search-R1's TC≈3.28 while (D) drops to ≈1.04, the confound is resolved and the reward is causally responsible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline claim attributes OTC-PO's up to 68.3% tool-call reduction and up to 215.4% tool-productivity gain to the proposed reward (Eq. 7), but §4.1 and Appendix C reveal an unablated intervention: the OTC search runs use a modified Search-R1 template that includes \"You need to make every search call count and gain helpful results\" and \"If you find no further external knowledge needed, you can directly provide the answer\". The Search-R1 baselines use their original template. These instructions are a direct behavioral prior to minimize tool calls, so the observed TC/TP differences are compatible with prompt conditioning alone. No experiment holds the prompt fixed and varies only the reward, nor varies the reward under the original prompt; Tables 5 and 6 manipulate model type and α but not this variable. The code-as-tool experiments (Table 2) use an unchanged ToRL template and are therefore less confounded and provide partial evidence for the reward's effect, but the largest advertised numbers come from the confounded search setting. Consequently, the central claim that the OTC-PO reward drives the efficiency gains is not supported as submitted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Optimal Tool Call-controlled Policy Optimization (OTC-PO), an RL framework that augments correctness rewards with a tool-use reward so that models learn to answer correctly with fewer external tool calls. Two instantiations are presented: OTC-PPO and OTC-GRPO. Experiments on NQ and HotpotQA (search as tool) and AIME/MATH (code as tool) report large reductions in average tool calls (up to 68.3%) and large increases in the newly proposed tool productivity metric (up to 215.4%) while roughly preserving answer accuracy. The paper also introduces the notion of cognitive offloading and argues that fewer tool calls can strengthen internal reasoning.","tokens_in":18912,"tokens_out":4524,"duration_ms":40927,"significance":"If the reported effects are real and attributable to the proposed reward, the paper addresses a practically important problem: tool-integrated RL agents can be made substantially cheaper and faster without sacrificing accuracy. The paper is one of the first to explicitly optimize tool-use efficiency in RL, and the code-as-tool experiments (Table 2) use an unchanged ToRL template, providing cleaner evidence than the search experiments. The out-of-domain evaluation (Table 4) and the behavioral case studies are useful additions. However, the significance of the headline claims depends on isolating the reward's effect from a concurrent change to the system prompt in the search experiments, which the current manuscript does not do.","major_comments":[{"comment":"The search experiments are confounded because the OTC runs use a modified Search-R1 template that explicitly instructs the model to minimize tool calls (\"You need to make every search call count and gain helpful results\" and \"If you find no further external knowledge needed, you can directly provide the answer\"), while the Search-R1 baselines use the original template. Section 4.1 describes this as a \"slight change\" and provides no ablation that holds the template fixed and varies only the reward. As a result, the up to 68.3% TC reduction and up to 215.4% TP gain in Table 1 cannot be attributed to the OTC-PO reward rather than to the added instruction. A prompt-only control, or experiments running OTC-PO with the original template and Search-R1 with the modified template, is needed to support the central claim.","section":"§4.1, Appendix C, Table 1"},{"comment":"The theoretical justification defines θ*_eff as the maximizer of E[α·r_tool] alone, omitting the correctness factor r_phi from Eq. (7). The subsequent claim that Acc(p*_θeff) = 1 is therefore unsupported: maximizing r_tool alone can be achieved by a response with zero tool calls and no answer. The appeal to Theorem 4.2 of Arora and Zanette [34] requires a careful check of its assumptions for the actual multiplicative reward r_tool^phi = α·r_tool·r_phi, and the statement that the only difference is the response definition y = (τ, a) is not sufficient to establish that the theorem applies.","section":"Appendix D.2, Eq. (10)"},{"comment":"In OTC-GRPO, the optimal number n is approximated by the minimum number of tool calls among correct trajectories sampled by the current policy, making the target a moving quantity that can drift toward ever-fewer calls during training without independent grounding. Combined with the monotonically decreasing r_tool in OTC-PPO (Eq. 5), this may overly penalize useful tool use. The paper does not analyze how n evolves over training or provide evidence that the moving target does not cause accuracy to degrade in settings beyond the reported benchmarks.","section":"§3.3, Eq. (6)"},{"comment":"The tool productivity metric TP = (number of correct answers) / (total tool calls) is essentially the quantity optimized by Eq. (7), since the reward is proportional to r_tool for correct answers and zero for incorrect ones. Reporting TP gains as the headline result is therefore partly circular. The informative evidence is the joint EM/TC pairs in Tables 1, 2, and 4, and the paper should frame its contribution through those paired numbers and accuracy-at-fixed-cost comparisons rather than through TP alone.","section":"§4.1, tool productivity metric"}],"minor_comments":[{"comment":"There is a typographical error: \"it only need to call tolls when necessary\" should read \"tools\" instead of \"tolls\".","section":"§4.1 Implementation Details"},{"comment":"The definitions of ME, LE, MA, LA, and AE are used without a clear caption or in-text explanation; the sentence \"OTC-GRPO and Search-R1-PPO achieves same results with same tool calls in 42.54% cases\" is particularly hard to interpret and should be clarified.","section":"Table 3"},{"comment":"The panels would benefit from labeled axes and units, especially the running-time panel, to make the reported training-time comparisons interpretable.","section":"Figure 3"},{"comment":"The phrase \"as r_phi is mainly the indicator function about correctness and format\" after Eq. (10) does not fix the omission of r_phi from the optimization objective; the derivation should be rewritten so that the maximizer is defined with respect to Eq. (7) as written.","section":"Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The prompt-change confound in the search experiments is the main obstacle to accepting the paper's headline claims. The ToRL-based results (Table 2) are cleaner and suggest the method has promise, but the abstract and Section 4.2 currently state the confounded numbers as the primary evidence. I would ask the authors to add the missing ablations and to revise the theoretical justification before the paper appears in a final form. If the ablations show that the prompt alone accounts for most of the effect, the paper should be substantially reframed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the headline search results are not a clean test of the reward. Section 4.1 admits the OTC search runs use a modified Search-R1 template that explicitly says to make every search count and to answer directly when possible, while the Search-R1 baselines use their original template. The stress-test note is right. Without an ablation holding the prompt fixed, the 40–68% tool-call reductions and the 215% TP gain cannot be attributed to OTC-PO's reward. That is a load-bearing flaw for the paper's central claim.\n\nNow the credit. The reward design itself is sensible: a multiplicative tool-use bonus that only activates on correct answers, borrowed from Arora and Zanette's efficiency RL but applied to tool-call count instead of reasoning length. Tool productivity is a reasonable summary metric for cost-adjusted accuracy. And the code-as-tool experiments (Table 2, Tables 7–8) use the unchanged ToRL template, so they are far less confounded. There OTC-GRPO cuts tool calls by roughly half to two-thirds while keeping EM on AIME and MATH essentially flat. That is real evidence the reward can change behavior. The out-of-domain generalization study is also broader than most work in this area.\n\nThe soft spots, in proportion. The prompt confound is the big one; it is mechanical to fix with a single ablation and the authors should have run it. The theoretical justification in Appendix D.2 is sloppy: they define the optimal policy as \\(\\arg\\max E[\\alpha \\cdot r_{\\text{tool}}]\\), dropping the correctness term, and then assert accuracy is preserved. As written it is not a proof, just a reference to [34] that does not quite carry over. The \"optimal\" tool-call count \\(n\\) is estimated from the policy's own correct trajectories, so it is a moving target; not fatal, but it means the reward is partly self-referential. Also, there are no seeds reported anywhere, so the 0.5–1 point EM differences are within noise.\n\nWho this is for: people working on RL for tool-augmented LLMs, especially those interested in efficiency rewards. It deserves a serious referee because the code experiments give a plausible demonstration and the prompt confound is fixable. My recommendation: send it out for review, but require a prompt-controlled ablation, multi-seed runs, and a corrected derivation before acceptance. As submitted, the strongest advertised numbers are not supported, but there is a usable idea underneath.","headline":"The search-table numbers are confounded by a prompt change that itself tells the model to make fewer calls; the code experiments give the reward a cleaner test, and the core idea is worth taking seriously despite the messy write-up.","tokens_in":19378,"tokens_out":1659,"would_cite":false,"duration_ms":17610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewarding minimal tool calls cuts search and code use by up to 68.3% while keeping answer accuracy.","keywords":["tool-integrated reasoning","reinforcement learning","tool calls","tool productivity","cognitive offloading","reward shaping","PPO","GRPO"],"falsifier":"Take the OTC search training and rerun it with the original baseline system prompt, removing the instruction to make every search call count; if average tool calls and tool productivity move back toward the baseline, the reward is not the causal driver.","tokens_in":18361,"feed_emoji":"🔧","tokens_out":8796,"duration_ms":77664,"temperature":0.7,"pith_summary":"This paper argues that tool-using language models should be trained to get the answer right with as few external tool calls as possible, not merely to get the answer right. It proposes OTC-PO, a reinforcement-learning reward that multiplies the ordinary correctness reward by a tool-efficiency factor, so a correct answer reached in fewer steps is rewarded more. Across search and code-interpreter benchmarks on base models of 1.5B to 7B parameters, the paper reports tool-call reductions up to 68.3% and gains in tool productivity (correct answers per tool call) up to 215.4%, with accuracy roughly unchanged. The payoff is practical: external tool calls dominate latency and cost, so the same accuracy at a fraction of the calls would make tool-integrated reasoning substantially cheaper to train and serve.","feed_headline":"Tool use drops up to 68% when RL rewards minimal tool calls","feed_subtitle":"A tool-cost-aware reward keeps answer accuracy while cutting search and code calls by up to two thirds.","key_machinery":"The central object is the tool-integrated reward $r^{tool}_\\phi=\\alpha\\,r_{tool}\\,r_\\phi$, where $r_\\phi$ is the usual correctness (plus optional format) reward and $r_{tool}\\in[0,1]$ measures how the trajectory's tool-call count $m$ compares with an estimated optimal count $n$. The multiplicative form is the load-bearing design: when the answer is wrong, $r_\\phi=0$ and the tool term is disabled, which prevents the model from being rewarded simply for calling no tools; when the answer is right, fewer calls yield a higher reward. In the group variant, $n$ is approximated online as the minimum number of tool calls among correct sampled trajectories; in the single-trajectory variant, the reward decays as $\\cos(m\\pi/(2m+c))$ as $m$ grows. This reward is what carries the claimed shift from optimizing correctness alone to optimizing tool productivity.","core_discovery":"The central claim is that there is, for each question and model, a minimal number of tool calls needed to reach a correct answer, and that a policy trained with a reward reflecting that minimum will keep accuracy while using fewer tools. OTC-PO instantiates this by assigning higher reward to correct trajectories with fewer tool calls and zero tool-efficiency reward to incorrect answers, so the model cannot game the metric by avoiding tools. In the group-based variant, the estimated optimal call count is the minimum observed among correct trajectories for the same question, updated across training epochs; in the single-trajectory variant, reward decays with the number of calls. The paper reports exact-match accuracy that is comparable to, and sometimes better than, the accuracy-only baselines on NQ, HotpotQA, and two AIME math sets, while average tool calls drop by 23% to 68%, and it reads the pattern as evidence that penalizing offloading develops internal reasoning rather than simply making the model terser.","pith_inferences":["The reported search results come from runs with a modified system prompt that explicitly tells the model to make every search call count and to answer directly when no external knowledge is needed; the baselines keep their original prompt, and no ablation holds the prompt fixed. Until that prompt change is controlled, part of the measured reduction may be instruction-following rather than reward l","If the reward is the true cause, an obvious failure mode to watch for is under-searching: on hard or out-of-domain questions the model may answer from parametric memory to protect its tool-productivity score, and the current exact-match numbers do not fully rule this out.","Tool productivity as defined (correct answers divided by total tool calls) rewards a model that calls a tool once on easy questions and zero times on difficult ones it gets wrong; a difficulty-stratified version of the metric would be a sharper test of whether efficiency reflects skill.","The same multiplicative reward should transfer to other costly agent actions, such as long code executions, API calls, or environment resets, whenever a correct outcome with fewer actions can be observed; testing that would require no change to the reward formula, only to what counts as a tool call."],"forward_implications":["If the central claim is right, tool-integrated agents can be trained and served at substantially lower cost, because tool execution, not token generation, is the dominant per-call expense.","The reported pattern that larger models cut tool calls more suggests over-reliance on tools grows with model capacity and can be corrected by the reward rather than by prompt engineering alone.","Accuracy and efficiency need not be traded off: on most benchmarks the method matches the accuracy-only baseline while improving tool productivity several-fold.","Out-of-domain question sets still show the efficiency gains, implying the learned policy transfers beyond the training distribution without extra annotation.","Because the reward plugs into both PPO and GRPO and works for search and code interpreters, the same recipe should attach to any RL pipeline with a countable costly action."],"supporting_citations":[{"why":"Supplies the search-tool RL training setup, the Search-R1 baseline whose tool-call counts and prompts OTC-PO is compared against, and the NQ/HotpotQA benchmark choice.","marker":"[14]"},{"why":"Supplies the code-as-tool RL setup, the ToRL baseline, and the AIME benchmark configurations used for the code experiments.","marker":"[15]"},{"why":"Provides the theoretical justification, adapted by the paper, that scaling a correctness reward by an efficiency coefficient preserves accuracy.","marker":"[34]"},{"why":"Defines the Qwen base models on which all search and code experiments are run.","marker":"[18]"}],"fun_headline_variants":["RL reward cuts tool calls by 68% at same accuracy","Minimal tool calls, maximal reasoning: OTC-PO","Tool productivity up 215% with RL efficiency reward","Less tool use, same answers: RL does it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline comparison changes both the reward and the search system prompt, and it never ablated the prompt change, so the causal role of the reward itself is not established.","fun_headline_variants_meta":{"raw":{"variants":["RL reward cuts tool calls by 68% at same accuracy","Minimal tool calls, maximal reasoning: OTC-PO","Tool productivity up 215% with RL efficiency reward","Less tool use, same answers: RL does it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1673,"prompt_tokens":1041,"completion_tokens":632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":657,"tokens_out":632,"duration_ms":5158,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:38:21.262198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the OTC search training and rerun it with the original baseline system prompt, removing the instruction to make every search call count; if average tool calls and tool productivity move back toward the baseline, the reward is not the causal driver.","supporting_citations":[],"review_version":1}