Pith. sign in

REVIEW 4 major objections 4 minor 20 cited by

Acting Less is Reasoning More! Teaching Model to Act Efficiently

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rewarding minimal tool calls cuts search and code use by up to 68.3% while keeping answer accuracy.

desk verdict The search-table numbers are confounded by a prompt change that itself tells the model to make fewer calls; the code experiments give the reward a cleaner test, and the core idea is worth taking seriously despite the messy write-up. read the letter →

arxiv 2504.14870 v2 pith:C6AL6HGQ submitted 2025-04-21 cs.AI cs.CL

Hongru Wang , Cheng Qian , Wanjun Zhong , Xiusi Chen , Jiahao Qiu , Shijue Huang , Bowen Jin , Mengdi Wang
show 2 more authors
Kam-Fai Wong Heng Ji
This is my paper · ORCID
classification cs.AIcs.CL
keywords tool-integratedreasoningreinforcementlearningtoolcallsproductivitycognitiveoffloadingrewardshapingPPOGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that tool-using language models should be trained to get the answer right with as few external tool calls as possible, not merely to get the answer right. It proposes OTC-PO, a reinforcement-learning reward that multiplies the ordinary correctness reward by a tool-efficiency factor, so a correct answer reached in fewer steps is rewarded more. Across search and code-interpreter benchmarks on base models of 1.5B to 7B parameters, the paper reports tool-call reductions up to 68.3% and gains in tool productivity (correct answers per tool call) up to 215.4%, with accuracy roughly unchanged. The payoff is practical: external tool calls dominate latency and cost, so the same accuracy at a fraction of the calls would make tool-integrated reasoning substantially cheaper to train and serve.

What carries the argument

The central object is the tool-integrated reward $r^{tool}_\phi=\alpha\,r_{tool}\,r_\phi$, where $r_\phi$ is the usual correctness (plus optional format) reward and $r_{tool}\in[0,1]$ measures how the trajectory's tool-call count $m$ compares with an estimated optimal count $n$. The multiplicative form is the load-bearing design: when the answer is wrong, $r_\phi=0$ and the tool term is disabled, which prevents the model from being rewarded simply for calling no tools; when the answer is right, fewer calls yield a higher reward. In the group variant, $n$ is approximated online as the minimum number of tool calls among correct sampled trajectories; in the single-trajectory variant, the reward decays as $\cos(m\pi/(2m+c))$ as $m$ grows. This reward is what carries the claimed shift from optimizing correctness alone to optimizing tool productivity.

What would settle it

Take the OTC search training and rerun it with the original baseline system prompt, removing the instruction to make every search call count; if average tool calls and tool productivity move back toward the baseline, the reward is not the causal driver.

Watch

Extended reading notes

Core claim

The central claim is that there is, for each question and model, a minimal number of tool calls needed to reach a correct answer, and that a policy trained with a reward reflecting that minimum will keep accuracy while using fewer tools. OTC-PO instantiates this by assigning higher reward to correct trajectories with fewer tool calls and zero tool-efficiency reward to incorrect answers, so the model cannot game the metric by avoiding tools. In the group-based variant, the estimated optimal call count is the minimum observed among correct trajectories for the same question, updated across training epochs; in the single-trajectory variant, reward decays with the number of calls. The paper reports exact-match accuracy that is comparable to, and sometimes better than, the accuracy-only baselines on NQ, HotpotQA, and two AIME math sets, while average tool calls drop by 23% to 68%, and it reads the pattern as evidence that penalizing offloading develops internal reasoning rather than simply making the model terser.

Load-bearing premise

The paper's headline comparison changes both the reward and the search system prompt, and it never ablated the prompt change, so the causal role of the reward itself is not established.

Editorial extensions

If this is right

  • If the central claim is right, tool-integrated agents can be trained and served at substantially lower cost, because tool execution, not token generation, is the dominant per-call expense.
  • The reported pattern that larger models cut tool calls more suggests over-reliance on tools grows with model capacity and can be corrected by the reward rather than by prompt engineering alone.
  • Accuracy and efficiency need not be traded off: on most benchmarks the method matches the accuracy-only baseline while improving tool productivity several-fold.
  • Out-of-domain question sets still show the efficiency gains, implying the learned policy transfers beyond the training distribution without extra annotation.
  • Because the reward plugs into both PPO and GRPO and works for search and code interpreters, the same recipe should attach to any RL pipeline with a countable costly action.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported search results come from runs with a modified system prompt that explicitly tells the model to make every search call count and to answer directly when no external knowledge is needed; the baselines keep their original prompt, and no ablation holds the prompt fixed. Until that prompt change is controlled, part of the measured reduction may be instruction-following rather than reward l
  • If the reward is the true cause, an obvious failure mode to watch for is under-searching: on hard or out-of-domain questions the model may answer from parametric memory to protect its tool-productivity score, and the current exact-match numbers do not fully rule this out.
  • Tool productivity as defined (correct answers divided by total tool calls) rewards a model that calls a tool once on easy questions and zero times on difficult ones it gets wrong; a difficulty-stratified version of the metric would be a sharper test of whether efficiency reflects skill.
  • The same multiplicative reward should transfer to other costly agent actions, such as long code executions, API calls, or environment resets, whenever a correct outcome with fewer actions can be observed; testing that would require no change to the reward formula, only to what counts as a tool call.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Optimal Tool Call-controlled Policy Optimization (OTC-PO), an RL framework that augments correctness rewards with a tool-use reward so that models learn to answer correctly with fewer external tool calls. Two instantiations are presented: OTC-PPO and OTC-GRPO. Experiments on NQ and HotpotQA (search as tool) and AIME/MATH (code as tool) report large reductions in average tool calls (up to 68.3%) and large increases in the newly proposed tool productivity metric (up to 215.4%) while roughly preserving answer accuracy. The paper also introduces the notion of cognitive offloading and argues that fewer tool calls can strengthen internal reasoning.

Significance. If the reported effects are real and attributable to the proposed reward, the paper addresses a practically important problem: tool-integrated RL agents can be made substantially cheaper and faster without sacrificing accuracy. The paper is one of the first to explicitly optimize tool-use efficiency in RL, and the code-as-tool experiments (Table 2) use an unchanged ToRL template, providing cleaner evidence than the search experiments. The out-of-domain evaluation (Table 4) and the behavioral case studies are useful additions. However, the significance of the headline claims depends on isolating the reward's effect from a concurrent change to the system prompt in the search experiments, which the current manuscript does not do.

major comments (4)
  1. [§4.1, Appendix C, Table 1] The search experiments are confounded because the OTC runs use a modified Search-R1 template that explicitly instructs the model to minimize tool calls ("You need to make every search call count and gain helpful results" and "If you find no further external knowledge needed, you can directly provide the answer"), while the Search-R1 baselines use the original template. Section 4.1 describes this as a "slight change" and provides no ablation that holds the template fixed and varies only the reward. As a result, the up to 68.3% TC reduction and up to 215.4% TP gain in Table 1 cannot be attributed to the OTC-PO reward rather than to the added instruction. A prompt-only control, or experiments running OTC-PO with the original template and Search-R1 with the modified template, is needed to support the central claim.
  2. [Appendix D.2, Eq. (10)] The theoretical justification defines θ*_eff as the maximizer of E[α·r_tool] alone, omitting the correctness factor r_phi from Eq. (7). The subsequent claim that Acc(p*_θeff) = 1 is therefore unsupported: maximizing r_tool alone can be achieved by a response with zero tool calls and no answer. The appeal to Theorem 4.2 of Arora and Zanette [34] requires a careful check of its assumptions for the actual multiplicative reward r_tool^phi = α·r_tool·r_phi, and the statement that the only difference is the response definition y = (τ, a) is not sufficient to establish that the theorem applies.
  3. [§3.3, Eq. (6)] In OTC-GRPO, the optimal number n is approximated by the minimum number of tool calls among correct trajectories sampled by the current policy, making the target a moving quantity that can drift toward ever-fewer calls during training without independent grounding. Combined with the monotonically decreasing r_tool in OTC-PPO (Eq. 5), this may overly penalize useful tool use. The paper does not analyze how n evolves over training or provide evidence that the moving target does not cause accuracy to degrade in settings beyond the reported benchmarks.
  4. [§4.1, tool productivity metric] The tool productivity metric TP = (number of correct answers) / (total tool calls) is essentially the quantity optimized by Eq. (7), since the reward is proportional to r_tool for correct answers and zero for incorrect ones. Reporting TP gains as the headline result is therefore partly circular. The informative evidence is the joint EM/TC pairs in Tables 1, 2, and 4, and the paper should frame its contribution through those paired numbers and accuracy-at-fixed-cost comparisons rather than through TP alone.
minor comments (4)
  1. [§4.1 Implementation Details] There is a typographical error: "it only need to call tolls when necessary" should read "tools" instead of "tolls".
  2. [Table 3] The definitions of ME, LE, MA, LA, and AE are used without a clear caption or in-text explanation; the sentence "OTC-GRPO and Search-R1-PPO achieves same results with same tool calls in 42.54% cases" is particularly hard to interpret and should be clarified.
  3. [Figure 3] The panels would benefit from labeled axes and units, especially the running-time panel, to make the reported training-time comparisons interpretable.
  4. [Appendix D.2] The phrase "as r_phi is mainly the indicator function about correctness and format" after Eq. (10) does not fix the omission of r_phi from the optimization objective; the derivation should be rewritten so that the maximizer is defined with respect to Eq. (7) as written.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reward and TP metric are aligned by design, but the central claims do not reduce to their inputs; the main caveats are a search-prompt confound and a weak theoretical appendix, not circular reductions.

full rationale

The paper's derivation chain is not circular in the sense defined here. The tool-productivity metric (TP = ΣI{correct}/Σtc) is the quantity the reward is designed to improve (Eq. 7: r_tool^φ = α·r_tool·r_φ), but that is an objective/evaluation alignment, not a reduction by construction: the training could fail, and the reported EM numbers are independent of the efficiency objective. The per-question optimal count n is estimated online as min(C) over correct rollouts (Sec. 3.3) and used as a moving reward target; this is a self-referential RL target, but it is not a fitted parameter that is later renamed as a test prediction. The search experiments do contain a real confound: Appendix C adds 'You need to make every search call count... If you find no further external knowledge needed, you can directly provide the answer' to the OTC search template while Search-R1 baselines use their original template, so the large TC/TP reductions on NQ/HotpotQA cannot be cleanly attributed to the reward. However, the code-as-tool experiments (Table 2) keep the ToRL template unchanged and still show 41–67% TC reductions, and the accuracy-preservation claim is separately measured, so the central result does not collapse to a single input. Appendix D.2's 'theoretical justification' is formally invalid (Eq. 10 drops r_φ and then concludes accuracy is preserved), but that is an unsupported proof step, not a circular definition. Minor self-citations to SMART and ToolRL in related work are not load-bearing. Overall: no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on hand-set hyperparameters c, alpha, and C, plus a per-question estimate n of optimal tool calls that is derived from the model's own sampled trajectories rather than from an external ground truth. No new physical or formal entities are introduced. The theoretical accuracy guarantee is imported from prior work without verifying its assumptions.

free parameters (4)
  • c (smooth constant in OTC-PPO tool reward) = set to maximum tool limit (4 for search, 3 for code)
    Controls decay rate of r_tool in Eq. 5; chosen by hand, no tuning procedure reported.
  • alpha (reward scale) = 1
    Scales r_tool in Eq. 7; Table 6 shows alpha=2 and 3 degrade both EM and TP, so alpha is a sensitive hand-set hyperparameter.
  • C (max tool calls) = 4 for search, 3 for code
    Limits trajectory length; paper shows ToRL-GRPO's tool calls grow with C while OTC-GRPO stays stable, so C is a choice that interacts with the method.
  • n (estimated optimal tool calls per question) = running minimum of correct group trajectories during training
    The 'optimal' number is approximated from the policy's own sampled outputs in OTC-GRPO; this data-derived estimate defines the reward peak and is updated across epochs. It is fitted to model behavior rather than specified a priori.
assumptions (4)
  • domain assumption For each question and model there exists an optimal (minimal) number of tool calls needed to reach a correct answer.
    Stated in Section 3.3 and used as the basis for the reward design.
  • ad hoc to paper The minimum number of tool calls observed among correct trajectories in a GRPO group approximates this true optimum.
    No proof is given; the reward peak is placed at this observed minimum, which can shift with policy behavior (Section 3.3, Eq. 6).
  • ad hoc to paper The multiplicative reward r_tool*r_phi preserves accuracy, following Theorem 4.2 of Arora and Zanette [34], and all assumptions of that theorem hold when the response includes tool trajectories.
    Appendix D.2 adopts the prior theorem without verifying its assumptions; the written derivation maximises alpha*r_tool alone, omitting r_phi.
  • ad hoc to paper The modified Search-R1 template that instructs the model to make every search call count has no significant effect on tool-call behavior relative to the original template.
    This is implicit in the experimental comparison (Section 4.1); if false, the reported reductions are explained by the prompt rather than the reward.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Acting Less is Reasoning More! Teaching Model to Act Efficiently." pith.science (2026). https://pith.science/paper/C6AL6HGQ

@misc{pith2026250414870,
  author       = {Pith},
  title        = {Pith review of: Acting Less is Reasoning More! Teaching Model to Act Efficiently},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6AL6HGQ}},
  note         = {Machine review of arXiv:2504.14870}
}
read the original abstract

Tool-integrated reasoning (TIR) augments large language models (LLMs) with the ability to invoke external tools during long-form reasoning, such as search engines and code interpreters, to solve tasks beyond the capabilities of internal reasoning. While reinforcement learning (RL) has shown promise in training such agents, most of existing approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use. This often leads to excessive tool calling, incurring high computational costs and hindering the development of internal reasoning capabilities - a phenomenon known as \textit{cognitive offloading}. To this end, we propose Optimal Tool Call-controlled Policy Optimization (OTC-PO), a simple yet effective RL-based framework that encourages models to produce accurate answers with minimal tool calls. Our method introduces a tool-integrated reward that jointly considers answer correctness and corresponding tool use behavior of model to reach that answer. To validate the effectiveness, we introduce the metric of \textit{tool productivity}, defined as the ratio between the number of correct answers and the total number of tool calls across all test cases. This metric reflects how efficiently tool usage contributes to successful task completion, with higher values indicating smarter and more autonomous reasoning. We instantiate this framework within both Proximal Policy Optimization (PPO) and Group Relative Preference Optimization (GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and Qwen-Math across multiple QA benchmarks show that our approach reduces tool calls by up to 68.3\% and improves tool productivity by up to 215.4\%, while maintaining comparable answer accuracy.

Figures

Figures reproduced from arXiv: 2504.14870 by the authors.

Figure 1
Figure 1. A case study using search as a tool on Qwen-7B-Base model, where all four methods arrive at the correct answer but exhibit distinct behaviors in internal reasoning and external tool usage. external tools like search engines, code interpreters and even physical robots. By bridging language understanding with external capabilities, TIR significantly expands the problem-solving scope of LLMs beyond internal knowledge-o… view at source ↗
Figure 2
Figure 2. An overview of OTC-GRPO Algorithm. rϕ(q, y) = rcorrect = 1 if a = ˆa else 0, (4) where a is the extracted predicted answer from the response y and aˆ is the ground truth answer. Some of them further consider the format reward by assigning additional reward besides the correctness such as rformat if the model generate the response in the required format. We discuss this in Appendix. 3.3 OTC-PO: Optimal Tool Call-cont… view at source ↗
Figure 3
Figure 3. Left: Changes of number of search calls during the training; Middle: Running time analysis; and Right: Changes of response length during the training [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Left: the illustration of tool-use reward functions in OTC-PPO; and Right: the illustration of tool-use reward functions in OTC-GRPO. D.2 Theoretical Justification We mainly follow the justification (Section 4.2) in the paper [34] to showcase our proposed reward design…
Figure 5
Figure 5. Figure 5: The Out-of-domain performance of OTC-PO and Search-R1 in TP. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Left: Training reward changes of ToRL-GRPO; and Right: Training reward changes of OTC-GRPO [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: An example that OTC-GRPO provide correct answer solely rely on its own internal [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: An example that OTC-GRPO uses less tool calls but with more accurate search to arrive a [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: An example of OTC-GRPO use less tool calls to get the correct answer. Question: Find all values of $x$ that satisfy the equation $x = \\!\\sqrt{11-2x} + 4$. Answer: 5 ToRL-GRPO: To solve the equation \\(x = \\sqrt{11 - 2x} + 4\\), we can follow these steps ...... ```py…
Figure 10
Figure 10. Figure 10: An example of OTC-GRPO does not need any external tool calls to get the correct answer. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BAP-SQL: Budget-Aware Observation Planning for Agentic Text-to-SQL

    cs.AI 2026-08 conditional novelty 7.0 of 10

    BAP-SQL improves tight-budget text-to-SQL accuracy by estimating query cost, rewriting expensive queries, and enforcing hard limits with a runtime shield.

  2. The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    The Piggyback Hypothesis attributes emergent misalignment to chat-template tokens piggybacking finetuned behavior; Token-Regularized Finetuning (TReFT) mitigates it by regularizing prefix token representations.

  3. Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    For six deep search agents on BrowseComp-Plus, answer accuracy tracks cumulative retrieval recall, not search effort, and failures split into missing-evidence and evidence-misuse gaps.

  4. ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Reinforcement-learned, type-aware visual tool calls improve multimodal scientific claim verification on SCIVER and MuSciClaims.

  5. Tool-Adaptive LLM Reranker

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Pointwise LLM reranking cast as an agentic MDP with language-preserving warm-up and asymmetric cost-aware RL yields adaptive tool use, SOTA NDCG, and high throughput.

  6. TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework

    cs.IR 2025-11 conditional novelty 6.0 of 10

    TeaRAG shows that hybrid chunk+triplet retrieval with Personalized PageRank and an iterative process-aware DPO reward keeps QA accuracy while cutting reasoning tokens by roughly 60%.

  7. ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    ParallelSearch applies reinforcement learning with parallel-decomposition rewards to teach LLM search agents to issue independent sub-queries concurrently, lifting QA accuracy by 2.9% on average and 12.7% on paralleli...

  8. MetaAgent: Toward Self-Evolving Agent via Tool Meta-Learning

    cs.AI 2025-08 reject novelty 6.0 of 10

    The paper advertises a self-evolving tool-using agent that beats baselines, but the body contains a different math paper, leaving the performance claims unsupported.

  9. UserBench: An Interactive Gym Environment for User-Centric Agents

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A new multi-turn agent benchmark shows that current LLMs elicit fewer than 30% of user preferences and reach full intent alignment only about 20% of the time.

  10. Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A 3B agent trained with supervised tool-use traces and reinforcement learning answers week-long egocentric video questions by dynamically selecting hierarchical retrieval, video-LLM, and VLM tools.

  11. Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Tool-Star combines cold-start supervised fine-tuning with a multi-tool self-critic reinforcement learning algorithm and hierarchical rewards to improve LLM tool-use reasoning.

  12. Time-R1: Towards Comprehensive Temporal Reasoning in LLMs

    cs.CL 2025-05 reject novelty 6.0 of 10

    A 3B model trained by staged reinforcement learning with rule-based rewards claims to outperform 671B models on temporal prediction and generation, though test-set checkpoint selection and synthetic training data weak...

  13. Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    CRPO turns on-policy self-distillation into group-wise contrastive learning gated by student–teacher entropy gaps, improving multi-turn agentic LLM post-training over GRPO, ARPO, and OPSD.

  14. Leanabell-Prover-V2: Verifier-integrated Reasoning for Formal Theorem Proving via Reinforcement Learning

    cs.AI 2025-07 reject novelty 5.0 of 10

    Verifier-integrated reinforcement learning with multi-turn reflection improves 7B-scale Lean 4 theorem proving by 2 to 3 points on MiniF2F at pass@128.

  15. R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning

    cs.CL 2025-05 conditional novelty 5.0 of 10

    R1-Searcher++ uses SFT cold-start plus reinforcement learning with group and memorization rewards to teach Qwen-2.5-7B to balance internal knowledge and external retrieval, improving accuracy and reducing retrieval calls.

  16. CoRT: Code-integrated Reasoning within Thinking

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Inserting targeted hints into a few training examples teaches reasoning models to compute with Python instead of text, improving accuracy and cutting token use by 30 to 50 percent.

  17. SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

    cs.CL 2025-05 conditional novelty 4.0 of 10

    SPA-RL attributes a single final task reward to individual agent steps and uses the attributions plus an executability signal as dense rewards for PPO, yielding modest success-rate gains on WebShop, ALFWorld, and VirtualHome.

  18. Toward Efficient Agents: Memory, Tool learning, and Planning

    cs.AI 2026-01 conditional novelty 3.0 of 10

    A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.

  19. Large Language Models for Planning: A Comprehensive and Systematic Survey

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.

  20. Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey

    cs.LG 2025-05 unverdicted novelty 2.0 of 10

    A survey that organizes LLM-based complex problem solving into three components, multi-step reasoning, domain knowledge, and result verification, and reviews methods and challenges across four application domains.

Reference graph

Works this paper leans on

57 extracted references · 34 canonical work pages · cited by 20 Pith papers

  1. [34]

    Training language models to reason efficiently, 2025

    Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025

    2025

  2. [1]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022

    2022

  3. [2]

    Openai o1 system card, 2024

    OpenAI Team. Openai o1 system card, 2024

    2024

  4. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI Team. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  5. [4]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  6. [5]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  7. [6]

    Pan, Zeming Liu, and Kam-Fai Wong

    Hongru W ANG, Deng Cai, Wanjun Zhong, Shijue Huang, Jeff Z. Pan, Zeming Liu, and Kam-Fai Wong. Self-reasoning language models: Unfold hidden reasoning chains with few reasoning catalyst. In Workshop on Reasoning and Planning for Large Language Models, 2025

    work page 2025

  8. [7]

    O1 replication journey: A strategic progress report – part 1, 2024

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. O1 replication journey: A strategic progress report – part 1, 2024

    work page 2024

Show all 57 references
  1. [8]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    2025

  2. [9]

    ToRA: A tool-integrated reasoning agent for mathematical problem solving

    Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2024

    2024

  3. [10]

    Start: Self-taught reasoner with tools, 2025

    Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools, 2025

    2025

  4. [11]

    Large language models as source planner for personalized knowledge-grounded dialogues

    Hongru Wang, Minda Hu, Yang Deng, Rui Wang, Fei Mi, Weichao Wang, Yasheng Wang, Wai-Chung Kwan, Irwin King, and Kam-Fai Wong. Large language models as source planner for personalized knowledge-grounded dialogues. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings ...

    2023

  5. [12]

    Smart: Self-aware agent for tool overuse mitigation

    Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation. arXiv preprint arXiv:2502.11435, 2025

    2025 arXiv

  6. [13]

    Pan, and Kam-Fai Wong

    Hongru Wang, Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. Empowering large lan- guage models: Tool learning for real-world interaction. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page ...

    2024

  7. [14]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    2025

  8. [15]

    Torl: Scaling tool-integrated rl, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl, 2025

    2025

  9. [16]

    Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for llms via reinforcement learning, 2025

    2025

  10. [17]

    Cognitive offloading

    Evan F Risko and Sam J Gilbert. Cognitive offloading. Trends in cognitive sciences, 20(9):676– 688, 2016. 10

    2016

  11. [18]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025

  12. [19]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  13. [20]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  14. [21]

    CREATOR: Tool creation for disentangling abstract and concrete reasoning of large language models

    Cheng Qian, Chi Han, Yi Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. CREATOR: Tool creation for disentangling abstract and concrete reasoning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistic...

    2023

  15. [22]

    Pan, and Kam-Fai Wong

    Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Z. Pan, and Kam-Fai Wong. AppBench: Planning of multiple APIs from various APPs for complex user instruction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conf...

    2024

  16. [23]

    Investigate-consolidate-exploit: A general strategy for inter-task agent self-evolution

    Cheng Qian, Shihao Liang, Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, and Maosong Sun. Investigate-consolidate-exploit: A general strategy for inter-task agent self-evolution. arXiv preprint arXiv:2401.13996, 2024

    2024 arXiv

  17. [24]

    Fung, Hao Peng, and Heng Ji

    Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. Craft: Cus- tomizing llms by creating and retrieving from specialized toolsets, 2024

    2024

  18. [25]

    Escapebench: Pushing language models to think outside the box

    Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, et al. Escapebench: Pushing language models to think outside the box. arXiv preprint arXiv:2412.13549, 2024

    2024 arXiv

  19. [26]

    Self-dc: When to reason and when to act? self divide-and- conquer for compositional unknown questions, 2025

    Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guanhua Chen, and Kam fai Wong. Self-dc: When to reason and when to act? self divide-and- conquer for compositional unknown questions, 2025

    2025

  20. [27]

    Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to m...

    2023

  21. [28]

    Toolink: Linking toolkit creation and using through chain-of-solving on open-source model

    Cheng Qian, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. Toolink: Linking toolkit creation and using through chain-of-solving on open-source model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang...

    2024

  22. [29]

    Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2025

    Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2025

    2025

  23. [30]

    Retool: Reinforcement learning for strategic tool use in llms, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025

    2025

  24. [31]

    Toolrl: Reward is all tool learning needs

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint, 2025. 11

    2025

  25. [32]

    SMARTCAL: An approach to self-aware tool-use evaluation and calibration

    Yuanhao Shen, Xiaodan Zhu, and Lei Chen. SMARTCAL: An approach to self-aware tool-use evaluation and calibration. In Franck Dernoncourt, Daniel Preo¸ tiuc-Pietro, and Anastasia Shimorina, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proc...

    2024

  26. [33]

    Adaptive tool use in large language models with meta-cognition trigger, 2025

    Wenjun Li, Dexun Li, Kuicai Dong, Cong Zhang, Hao Zhang, Weiwen Liu, Yasheng Wang, Ruiming Tang, and Yong Liu. Adaptive tool use in large language models with meta-cognition trigger, 2025

    2025

  27. [35]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in ...

    2025

  28. [36]

    Theory and application of reward shaping in reinforcement learning

    Adam Daniel Laud. Theory and application of reward shaping in reinforcement learning . University of Illinois at Urbana-Champaign, 2004

    2004

  29. [37]

    Integrating pretrained language model for dialogue policy evaluation

    Hongru Wang, Huimin Wang, Zezhong Wang, and Kam-Fai Wong. Integrating pretrained language model for dialogue policy evaluation. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6692–6696, 2022

    2022

  30. [38]

    L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    2025

  31. [39]

    Harnessing the reasoning economy: A survey of efficient reasoning for large language models, 2025

    Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, and Kam-Fai Wong. Harnessing the reasoning economy: A survey of efficient reasoning for large language models, 2025

    2025

  32. [40]

    Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning, 2025

    Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning, 2025

    2025

  33. [41]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  34. [42]

    Left: as the number of tool calls increases, thertool decreases accordingly. Thus when multiple trajectories leads to correct answer, the one that use less tools will get higher reward and the one that do not use tool will get the highest reward; 2) Right: It is obvious that t...

  35. [43]

    We need to find the US city where the Bellagio resort, owned by MGM Resorts Interna- tional, is located

  36. [44]

    The Bellagio resort is a well-known hotel and casino located in Las Vegas, Nevada

  37. [45]

    Animorphs

    Therefore, the US city where the Bellagio resort is located is Las Vegas. <answer> Las Vegas </answer>(Tool Calls = 0) Figure 7: An example that OTC-GRPO provide correct answer solely rely on its own internal reasoning without any external tool calls while Search-R1 extensivel...

  38. [46]

    We need to find a science fantasy young adult series

  39. [47]

    The series is told in first person

  40. [48]

    Animorphs

    It has a set of companion books narrating the stories of enslaved worlds and alien species. Let’s search for this information: <search> What science fantasy young adult series, told in first person, has a set of companion books narrating the stories of enslaved worlds and alie...

  41. [49]

    Isolate the square root term

  42. [50]

    Square both sides to eliminate the square root

  43. [51]

    Solve the resulting equation

  44. [52]

    Let's go through these steps systematically

    Verify the solutions by substituting back into the original equation to ensure they satisfy it. Let's go through these steps systematically

  45. [53]

    Isolate the square root term:\n\\[ x - 4 = \\sqrt{11 - 2x} \\]

  46. [54]

    Square both sides:\n\\[ (x - 4)^2 = 11 - 2x \\]

  47. [55]

    Expand and simplify the equation:\n\\[ x^2 - 8x + 16 = 11 - 2x \\] [ x^2 - 6x + 5 = 0 \\]

  48. [56]

    Solve the quadratic equation:\n\\[ (x - 5)(x - 1) = 0 \\] So, the solutions are \\(x = 5\\) and \\(x = 1\\)

  49. [57]

    Verify the solutions by substituting back into the original equation. For \\(x = 5\\):\n\\[ 5 = \\sqrt{11 - 2 \\cdot 5} + 4 \\]\n\\[ 5 = \\sqrt{1} + 4 \\]\n\\[ 5 = 1 + 4 \\]\n\\[ 5 = 5 \\]\nThis is true, so \\(x = 5\\) is a valid solution.\n\nFor \\(x = 1\\):\n\\[ 1 = \\sqrt{1...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.