Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that separating a reasoning planner from a tool-calling agent improves multi-hop question answering, achieving state-of-the-art exact match on Bamboogle after reinforcement fine-tuning on only 180 samples.

desk verdict A sensible but confounded study: the pre/post RL gains are real, but the Bamboogle SOTA claim needs an ablation to hold. read the letter →

arxiv 2507.01489 v1 pith:HOELECPD submitted 2025-07-02 cs.AI cs.MA

Yanfei Zhang This is my paper
classification cs.AIcs.MA
keywords multi-hopquestionansweringreinforcementlearninghierarchicalagenttoolcallingGRPOlargelanguagemodelswebsearchsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-hop question answering generally requires an agent to interleave thinking with web searches, but earlier RL-trained agents do both in one loop and therefore reason over raw, noisy search output. This paper proposes Agent-as-tool, which splits the job: a Planner does the natural-language reasoning and decides what to look up, while a separate Toolcaller agent carries out web searches and returns short structured summaries. The Planner is fine-tuned with GRPO on just 180 samples from HotpotQA and 2WikiMultiHopQA, and the resulting model achieves 63.2% exact match and 75.2% cover exact match on Bamboogle, exceeding the Search-R1 baseline by 4.8 and 3.2 points. The paper argues that this hierarchical decoupling makes reasoning cleaner and reinforcement learning far more sample-efficient.

What carries the argument

The load-bearing mechanism is the two-tier Planner/Toolcaller hierarchy plus observation masking. The Planner is the policy being trained: it reasons in <think>...</think> tags and issues sub-queries inside <tool calling>...</tool calling> tags. The Toolcaller is a separate, fixed chat agent that executes the search and returns processed observations in <obs>...</obs> blocks. In RL, those observation blocks are replaced by a zero-embedding <fim pad> token during reward modeling, so credit flows only to the Planner's reasoning and query-construction decisions. Training uses GRPO with 12 rollouts per prompt, a batch of 3, and 60 steps (180 samples total), with a reward of +F1 for correctly formatted answers and -2 for malformed ones.

What would settle it

Run the comparison with the tool-output variable held constant: give Search-R1 the same GPT-4o-mini-generated structured summaries that Agent-as-tool's Toolcaller produces, or replace the Toolcaller with Qwen-2.5-7B-Instruct. If Bamboogle exact match for Search-R1 rises from 58.4% toward Agent-as-tool's 63.2%, or if Agent-as-tool's score falls when its Toolcaller is weakened, the confound is confirmed and the hierarchical claim is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that tool execution and verbal reasoning should be assigned to different agents in an RL-trained system. The Planner emits <think> reasoning and <tool calling> natural-language sub-queries; the Toolcaller, a CAMEL-style chat agent powered by GPT-4o-mini, retrieves top-k search results and returns them as structured <obs> observations. During GRPO fine-tuning the observation blocks are masked with a special token so the reward reaches only the Planner's outputs, preserving credit assignment. With 60 training steps on 180 samples (12 rollouts each), the fine-tuned Planner improves over its untuned base on all four benchmarks by an average of 2.5% in exact match and, on Bamboogle, beats the strongest compared baseline Search-R1.

Load-bearing premise

The load-bearing premise is that the measured gains come from the hierarchical split itself, yet Agent-as-tool and Search-R1 differ in a second respect: Agent-as-tool's planner receives pre-digested summaries from a stronger, fixed model (GPT-4o-mini), and that variable is never isolated.

Editorial extensions

If this is right

  • If the hierarchical design is responsible for the gains, RL-based agents for search-heavy tasks should separate reasoning from tool execution instead of using a single coupled ReAct-style loop.
  • Reinforcement fine-tuning on as few as 180 samples yields a consistent improvement across all four benchmarks, suggesting multi-hop search agents can be adapted cheaply.
  • Reasoning over structured, summarized observations rather than raw search output produces clearer, more decomposable reasoning traces and avoids the model being misled by irrelevant symbols.
  • The architecture is extensible: the Toolcaller can be swapped for calculators, code interpreters, or MCP-based tool servers, and the Planner can evolve into a tool orchestrator.
  • Observation masking preserves credit assignment in the hierarchical setup and could support other multi-agent RL training schemes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison with Search-R1 changes two variables at once: the hierarchical structure and the presence of a strong summarizer (GPT-4o-mini) that writes structured observations, so the isolated effect of the hierarchy is not identified by the reported numbers.
  • A direct test would feed Search-R1 the same structured summaries or give Agent-as-tool raw search results, which would separate the benefit of structured observations from the benefit of hierarchical RL training.
  • The design also implies a cheaper deployment pattern, where a fixed or small model handles tool-interface noise while the trained policy concentrates on reasoning.
  • Observation masking may transfer to other hierarchical RL settings where a subordinate agent's outputs should not leak reward into the policy gradient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes 'Agent-as-tool', a hierarchical framework that separates a reasoning Planner from a tool-calling Toolcaller. The Toolcaller, implemented as a CAMEL-style agent powered by GPT-4o-mini, summarizes web-search results into structured observations. The Planner is fine-tuned with GRPO on 180 samples from HotpotQA and 2WikiMultiHopQA, with observation masking to preserve credit assignment. The authors report state-of-the-art results on Bamboogle (63.2% EM, 75.2% CEM) over Search-R1, while acknowledging lower exact-match results on HotpotQA, 2WikiMultiHopQA, and MuSiQue.

Significance. The conceptual split of planning and tool execution is a sensible architectural proposal, and the claim that only 180 samples and 2160 rollouts are needed for a stable gain is interesting. The paper is transparent about its training protocol and includes qualitative examples. However, the headline empirical claim is not yet established: the head-to-head comparison with Search-R1 is confounded by the Toolcaller's use of GPT-4o-mini, and the reported numbers lack error bars. The framework and the pre/post-RL comparison are worth seeing with proper ablations, but in its current form the evidence is insufficient to support 'state-of-the-art'.

major comments (4)
  1. [§3.1.2, §4.1.4, Table 1] The headline claim of state-of-the-art Bamboogle performance rests on a comparison that changes two variables at once. §3.1.2 specifies that the Toolcaller is a CAMEL-style GPT-4o-mini agent returning structured summaries, while §4.1.4 states that for Search-R1, search results are 'directly returned as the answer' without such a Toolcaller. Table 1 shows that Agent-as-tool-Base (no GRPO) already reaches 60.0 EM on Bamboogle, above Search-R1's 58.4 EM. The reported 4.8-point gain of Agent-as-tool-Instruct over Search-R1 is therefore consistent with the GPT-4o-mini summarizer accounting for most or all of the improvement, rather than the hierarchical design or the RL fine-tuning. Please add an ablation that fixes the observation model—e.g., a Toolcaller implemented with Qwen-2.5-7B-Instruct, or Search-R1 fed GPT-4o-mini-generated summaries—and report the pre-RL version of that ablated setting.
  2. [§4.1.5, Table 1, Table 2] All quantitative results are single numbers with no error bars, no test-set sizes, and no significance tests. The differences that support the paper's conclusions (e.g., +3.2 EM on Bamboogle from RL in Table 2; +2.2 EM on HotpotQA; +4.8 EM over Search-R1 in Table 1) are a few percentage points on datasets that are known to be small; without variance estimates they cannot be distinguished from run-to-run noise. Please provide means and standard deviations over at least three independent RL runs/seeds, or bootstrap confidence intervals, and report the number of evaluation examples per dataset.
  3. [§3.2.2–§3.2.3, Eq. (2)] The causal story that 'hierarchical decoupling' produces the gains is underdetermined because the reward function (Eq. 2) combines F1 scoring with a −2 formatting penalty and the training procedure includes observation masking (§3.2.2). A non-hierarchical GRPO baseline trained with the same reward, prompt format, and 180-sample budget is needed to show that the architecture, rather than the reward shaping or the extra training signal, is responsible for the improvement. Without such a baseline, the paper's Section 4.2 claim that 'the Agent-as-tool ... is a more effective framework' is not supported.
  4. [§4.1.2, Figure 2] The report that training is 60 steps on 180 samples with 12 rollouts per prompt is transparent, but the loss curve in Figure 2 is unstable for the first 30 steps, and no validation accuracy is shown during training. It is therefore unclear whether the final checkpoint is representative, and whether the 180-sample selection (described only as 'same ratio as R1-searcher') matters. Please report the sample-selection procedure and show results for at least two different selections or seeds.
minor comments (5)
  1. [Abstract] The abstract contains grammatical errors ('had achieved exceptionally well performance', 'the chain of reasoning was solely relied on') that should be corrected.
  2. [Eq. (2)] The F1 score used for the reward is never defined for free-form answers (e.g., token-level, character-level, or based on a normalization step); please specify the exact computation.
  3. [§4.1.5] The CEM ('cover exact match') metric is not defined; please provide a definition or a citation to the original metric.
  4. [§4.1.4] There is a typo 'information retrival' (should be 'information retrieval'), and the text contains inconsistent spacing in 'T oolcaller', 'W eb Search', and 'Agent-as-tool-Base + W eb Search'.
  5. [§4.2] Cross-reference errors: 'The qualitative results are shown in 1' should read 'Table 1', and 'shown in 2' should read 'Table 2'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL derivation and evaluation are self-contained; the SOTA comparison is confounded but not circular.

full rationale

The paper's central derivation is not circular. The GRPO objective (Eq. 1), the reward function (Eq. 2), and the observation-masking scheme are fixed independently of the evaluation outcome. Training data are drawn from HotpotQA and 2WikiMultiHopQA, while Bamboogle is used only for evaluation, so no fitted parameter is later renamed as a prediction. The framework's components (Planner and Toolcaller) are defined behaviorally, not in terms of the benchmark results, and the claimed efficiency (180 training samples) is an empirical outcome rather than a construction. The main weakness is a confound: the Toolcaller uses GPT-4o-mini to summarize search results while the Search-R1 baseline receives raw search outputs, so the Bamboogle gain over Search-R1 may be attributable to the summarizer rather than to hierarchical RL. A confound, however, is not circularity: it does not make the stated result equivalent to its inputs by definition. There are also no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The paper's claims are underdetermined by the experiments, not entailed by them, so the correct circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on several unverified assumptions: the GPT-4o-mini Toolcaller's summaries are accurate and sufficient, the reward shape (F1 with a -2 formatting penalty) drives reasoning rather than formatting improvements, and the small 180-sample training set transfers to unseen benchmarks. No code, data, or prompt templates are provided, so these assumptions cannot be checked independently. There are no newly postulated physical or conceptual entities beyond the architectural roles of Planner and Toolcaller.

free parameters (4)
  • Reward penalty for malformed output = -2
    Equation (2); hand-chosen constant that shapes the RL objective and may affect formatting-related metric gains.
  • Number of GRPO training steps = 60
    Section 4.1.2; chosen by hand, loss stabilized after ~30 steps, but no sensitivity analysis is provided.
  • Rollouts per prompt = 12
    Section 4.1.1; fixed for all prompts, no tuning or variance reporting.
  • Max tool-calling rounds per rollout = 10
    Section 4.1.1; an arbitrary stop condition that may cap the number of reasoning hops.
assumptions (4)
  • domain assumption The Toolcaller (GPT-4o-mini) returns accurate and sufficient summaries of web search results.
    Section 3.1.2 states the Toolcaller 'returns structured summaries to the Planner'; the whole framework relies on these summaries being faithful and complete.
  • ad hoc to paper Rewarding only the final answer with F1 and a formatting penalty is sufficient to improve multi-hop reasoning.
    Section 3.2.3 defines the reward function; there is no validation that this reward shape, rather than formatting effects, drives the reported gains.
  • domain assumption The evaluation metrics EM and CEM capture the intended reasoning ability.
    Section 4.1.5 uses these metrics but does not define CEM or discuss its reliability.
  • domain assumption The 180-sample training set, drawn from HotpotQA and 2WikiMultiHopQA, is representative enough to generalize to MuSiQue and Bamboogle.
    Section 4.1.2 describes the small training set; no analysis of domain shift is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning." pith.science (2026). https://pith.science/paper/HOELECPD

@misc{pith2026250701489,
  author       = {Pith},
  title        = {Pith review of: Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOELECPD}},
  note         = {Machine review of arXiv:2507.01489}
}
read the original abstract

Large Language Models (LLMs) have emerged as one of the most significant technological advancements in artificial intelligence in recent years. Their ability to understand, generate, and reason with natural language has transformed how we interact with AI systems. With the development of LLM-based agents and reinforcement-learning-based reasoning models, the study of applying reinforcement learning in agent frameworks has become a new research focus. However, all previous studies face the challenge of deciding the tool calling process and the reasoning process simultaneously, and the chain of reasoning was solely relied on the unprocessed raw result with redundant information and symbols unrelated to the task from the tool, which impose a heavy burden on the model's capability to reason. Therefore, in our research, we proposed a hierarchical framework Agent-as-tool that detach the tool calling process and the reasoning process, which enables the model to focus on the verbally reasoning process while the tool calling process is handled by another agent. Our work had achieved comparable results with only a slight reinforcement fine-tuning on 180 samples, and had achieved exceptionally well performance in Bamboogle with 63.2% of exact match and 75.2% in cover exact match, exceeding Search-R1 by 4.8% in exact match and 3.2% in cover exact match.

Figures

Figures reproduced from arXiv: 2507.01489 by the authors.

Figure 1
Figure 1. The trajectory of a single sample from a batch of questions processed in different [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training progress of the Agent-as-tool model showing loss convergence over training [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The Agent-as-tool-Instruct could reason with less fuzzy and more structured observations, comparing with the Search-R1 + Web Search which would need to reason with the unprocessed web search results with other unrelated details. Search-R1 was misled by the unprocessed web search results to reason with the wrong answer for the second hop (Princess Feodora of Leiningen), while the Agent-as-tool-Instruct has applied th… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The Agent-as-tool-Instruct + Web Search could correctly decompose the question to identify the first hop and the second hop, comparing with the Agent-as-tool￾Base + Web Search which barely decompose the question and try to ask about the whole question in another manner…

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    CARL trains a critic for segment-level credit assignment from binary outcomes in LLM tool-use trajectories, yielding 6.7-9.7 point accuracy gains and 53% fewer calls on solvable questions across five benchmarks.

  2. IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents

    cs.AI 2026-05 conditional novelty 7.0 of 10

    IdleSpec improves LLM agent accuracy by generating and aggregating speculative plans during idle time between tool calls and observations using complementary drafting strategies.

  3. GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    GEAR reshapes GRPO trajectory advantages using divergence signals from a ground-truth-conditioned teacher to create adaptive token- and segment-level credit regions.

  4. Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory

    cs.CL 2025-11 unverdicted novelty 7.0 of 10

    Evo-Memory is a new benchmark for self-evolving memory in LLM agents across task streams, with baseline ExpRAG and proposed ReMem method that integrates reasoning, actions, and memory updates for continual improvement.

  5. GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    GEAR adaptively reweights GRPO advantages in LLM RL by using divergence spikes from self-distillation to define semantic segments and modulate local credit.

  6. Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory

    cs.CL 2025-11 unverdicted novelty 6.0 of 10

    Evo-Memory is a new streaming benchmark and evaluation framework for self-evolving memory in LLM agents, unifying over ten memory modules and introducing the ReMem pipeline for continual improvement on multi-turn and ...

  7. OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systems

    cs.AI 2026-04 reject novelty 3.0 of 10

    A review-plus-demo claiming agentic capabilities emerge from system integration, backed by a 15-task benchmark whose C1→C3 performance gap is largely built into the test design.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [1]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  2. [3]

    Ho, A.-K

    X. Ho, A.-K. D. Nguyen, S. Sugawara, and A. Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060,

    arXiv 2011

  3. [6]

    URL https://arxiv.org/abs/2503.09516. G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for” mind” exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008, 2023a. M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li. Api-bank: A comprehensive be...

    arXiv

  4. [7]

    URL https://doi.org/10.5281/zenodo.15186407. B. Liu, X. Li, J. Zhang, J. Wang, T. He, S. Hong, H. Liu, S. Zhang, K. Song, K. Zhu, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.01990,

    arXiv

  5. [8]

    URL https://arxiv.org/abs/2210.03350. C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-T¨ ur, G. Tur, and H. Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958,

    arXiv

  6. [9]

    URL https://arxiv.org/abs/2412.15115. 9 Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  7. [12]

    Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600,

    arXiv

  8. [13]

    URL https://arxiv.org/abs/2210.03629. J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762,

    arXiv

Show all 15 references
  1. [14]

    Zheng, S

    Y. Zheng, S. Sun, L. Qiu, D. Ru, C. Jiayang, X. Li, J. Lin, B. Wang, Y. Luo, R. Pan, et al. Openresearcher: Unleashing ai for accelerated scientific research. arXiv preprint arXiv:2408.06941,

    arXiv

  2. [15]

    Zheng, D

    Y. Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160,

    arXiv

  3. [2020]

    S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6,

    arXiv

  4. [2022]

    URL https://arxiv.org/abs/2108.00573. L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallou´ edec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl,

    arXiv

  5. [2023]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Weli- hinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

    arXiv

  6. [2024]

    H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J.-R. Wen. R1- searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592,

    arXiv

  7. [2025]

    URL https://arxiv.org/abs/2504.04736. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.