{"id":"43158190-d8cd-4fbd-bc43-34bf431a315e","arxiv_id":"2607.15660","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ToolVerse trains LLM agents via RL on 422 executable MCP environments and graph-based long-horizon tasks, with a turn-aware advantage estimator that improves multi-turn tool-use benchmarks over GRPO baselines.","lead":"ToolVerse is a framework that scales agentic reinforcement learning to hundreds of real-world tool environments (422 MCPs, ~4,400 tools), generates long-horizon task chains from tool dependency graphs, and introduces a turn-level credit assignment method (TARA). The authors report consistent gains over base LLMs on function-calling benchmarks, though the TARA-specific advantage is not uniform across all settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shared user simulator between training and tau2-Bench/ACEBench evaluation may inflate reported gains; re-evaluate with a held-out user model.","rationale":"The central claim is empirical: ToolVerse consistently improves long-horizon tool-use performance, and TARA beats GRPO. The most load-bearing condition is that the evaluation measures real generalization rather than overfitting to the training setup. The strongest threat is that the same DeepSeek-V3.2 model acts as the user in both RL training and two of the three benchmark evaluations. This creates a direct distribution overlap: the policy can memorize the simulator's dialogue patterns, and the evaluation then rewards that memorization. The reader's weakest_assumption focused on mock-environment fidelity, which is a valid external-validity concern, but the internal validity of the reported benchmark results is more immediately threatened by the user-simulator leakage. The reader did note this leakage in the rationale, so my agreement is partial rather than full. The recommended verdict remains CONDITIONAL because the concern can be addressed by a concrete re-evaluation experiment; it does not require rejecting the framework outright. The paper's engineering contributions, including automated environment construction, the DUS task generator, and turn-level advantage, are credible and reproducible in principle, so the appropriate response is to condition acceptance on the additional held-out-user evaluation and on reporting variance across the four runs.","tokens_in":838,"tokens_out":988,"duration_ms":49553,"concrete_test":"Fix a trained policy (e.g., Qwen3-8B + TARA from Table 2). Re-run tau2-Bench and ACEBench-Agent evaluations with a user simulator not used in training: either the benchmark's original/reference user model (from the official repositories) or a different LLM such as GPT-4.1 or Qwen3-32B, keeping all other settings identical and running the same number of seeds. Compare the TARA, GRPO, and base-model scores against Table 2. If the ToolVerse gain over the base model or the TARA-over-GRPO gap shrinks materially or becomes statistically indistinguishable, the claimed universal gains are inflated by user-simulator overfitting.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim that ToolVerse yields universal gains and TARA improves over GRPO depends on the fairness of the evaluation. Section 5.1 states that DeepSeek-V3.2 is deployed as the user simulator during RL training and then the same model is used as the user for tau2-Bench and ACEBench-Agent evaluations. A policy trained against a fixed user model can exploit its idiosyncratic clarification behavior, phrasing, and tolerance for partial information; evaluating with that same model measures fit to the training user, not general tool-use capability. No control is reported in which the user is a held-out model or the benchmark's canonical simulator. This concern is reinforced by Table 2: Qwen3-4B BFCL Overall shows GRPO at 28.50 vs TARA at 28.25, a counterexample to the blanket 'TARA outperforms GRPO' claim, suggesting measured differences are small and sensitive to evaluation distribution. The paper reports four runs but no error bars, so it is impossible to separate genuine improvement from simulator-specific noise. If gains shrink or vanish under a different user model, the central claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ToolVerse, a framework for scaling executable agentic reinforcement-learning environments. It automatically converts raw MCP tool schemas into mock dictionary-backed Python/MCP environments (422 toolsets, 4,438 tools), constructs a tool dependency graph, uses Dynamic Unlocking Sampling to synthesize long-horizon multi-turn tasks (the GUST dataset), and proposes Turn-Aware Relative Advantage (TARA), a turn-level advantage estimator that combines local correctness and gated future returns. Experiments with Qwen3-4B, Qwen3-8B, and Qwen2.5-14B are evaluated on BFCL-v3 Multi-Turn, τ2-Bench, and ACEBench-Agent, with ablations and comparisons to public baselines. The central claim is that ToolVerse produces consistent, universal performance gains and that TARA substantially improves over naive GRPO.","tokens_in":17582,"tokens_out":4583,"duration_ms":45956,"significance":"If the claims hold, the paper would make a useful contribution: an automated pipeline from raw tool schemas to executable environments, a dependency-aware task-generation method, and a relatively simple dense-credit RL variant. Strengths include the scale of the environment/task construction, the executable verification pipeline, teacher Pass@8 filtering, and evaluation on multiple external benchmarks. However, the current evidence does not support the strong 'universal gains' and 'TARA is consistently superior' claims. Two benchmarks use the same user simulator for training and evaluation, no error bars or significance tests are reported despite four runs, one row in Table 2 contradicts the TARA > GRPO statement, and the theoretical appendix overclaims. These concerns are substantial but addressable with additional experiments and a revised exposition.","major_comments":[{"comment":"The same DeepSeek-V3.2 user simulator used for RL training is also deployed as the user for τ2-Bench and ACEBench-Agent. This confounds the measured gains: policies trained against that simulator can exploit its specific clarification behavior, phrasing, and tolerance, so the improvements may reflect fit to the training user rather than general tool-use capability. No control with a held-out user model or with the benchmark's canonical simulator is reported. Because the central claim of universal gains rests substantially on these two benchmarks, a user-model sensitivity analysis is needed.","section":"Sec. 5.1 (evaluation protocol); Table 2"},{"comment":"The claim that TARA 'consistently' outperforms GRPO is not supported by the reported numbers. For Qwen3-4B on BFCL Overall, GRPO gives 28.50 and TARA gives 28.25. More generally, four runs are claimed but no variance, confidence intervals, or significance tests are provided; many deltas are a few points (e.g., Qwen3-8B τ2-Bench: 30.10 vs 32.37; BFCL: 35.25 vs 37.50). Without measures of uncertainty, the 'substantial gains' assertion is unverifiable.","section":"Table 2; Table 4"},{"comment":"The theoretical analysis overstates what is proved. Theorem A.4 requires the group-mean local reward μ_local>0, i.e., at least one correct rollout at each turn. When all rollouts fail, A_local=0 and A_future can be 0, so the total advantage is zero, not negative; the proof's conclusion of strict negativity does not follow. In addition, Theorem A.2 compares variances of raw return-to-go estimates, but GRPO and TARA use group-normalized advantages; variance reduction of the unnormalized value does not imply variance reduction of the advantage estimator used in the policy update. These claims should be restated or removed.","section":"Appendix A.2–A.4, Eq. (2)–(5)"},{"comment":"Transferability of the trained policies depends on the fidelity of the mock environments. The dictionary databases contain only 3–5 records per scenario, and roughly 20% of toolsets are discarded because they are 'difficult to faithfully mock.' This is acknowledged as a limitation, but its impact on the benchmark gains is not quantified. A diagnostic comparing behavior on more stateful environments, or on a held-out set of real API traces, would materially strengthen the generalization claim. As written, the possibility remains that reported gains partly reflect adaptation to simplified mocks.","section":"Sec. 3.1, Sec. 4.2, Appendix B.1"}],"minor_comments":[{"comment":"The abstract says 'nearly 400 real-world MCPs' while the introduction and Table 6 report 422 environments and 4,438 tools. Please harmonize the numbers.","section":"Abstract vs. Section 1/Table 6"},{"comment":"There are typos and formatting inconsistencies, e.g., 'assigment' in the abstract, 'severa' in the abstract, 'Defination' in Appendix F, and inconsistent capitalization of 'ToolVerse.'","section":"Throughout"},{"comment":"Equation (5) defines λ∈[0,1), but Appendix A.1 states λ∈[0.5,1.0], while Figure 5 varies λ down to 0.1. Please clarify the allowed range.","section":"Eq. (5) vs. Appendix A.1"},{"comment":"The x-axis labels for λ and γ appear to be missing in the figure; the caption also says 'Varying with = 0.5' without naming the fixed hyperparameter.","section":"Figure 5"},{"comment":"The comparison with public baselines uses Qwen2.5-7B-Instruct-TARA, while the main experiments use Qwen2.5-14B-Instruct and Qwen3 models. Please clarify the training recipe and report variance for these runs.","section":"Table 3"},{"comment":"KL divergence loss is disabled (kl_loss=False). This is an unusual choice for GRPO-style training and should be justified, especially since many GRPO variants rely on the KL penalty for stability.","section":"Appendix B.5"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the shared user simulator between training and evaluation; this should be fixed before acceptance, e.g., by adding a held-out user model or the benchmarks' original simulators. The authors should also provide confidence intervals or significance tests for all claimed improvements. If these can be addressed, the paper's scale and tooling contributions would likely be a good fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ToolVerse is a serious engineering contribution. The authors turned ~400 real MCP tool definitions into executable mock environments, built a graph-based task sampler (DUS) that respects tool dependencies, and added a turn-level advantage estimator (TARA) with a consistency gate. That combination is new, and the GUST dataset (422 environments, ~4.4k tools, ~3k tasks) is a useful resource if released. The empirical gains over base models are real-looking: Qwen3-8B goes from 28.88 to 37.50 on BFCL-v3 and from 27.87 to 32.37 on τ²-Bench. The ablations support the claim that future-credit gating matters more than local-only credit. This is not a reshuffling of known parts.\n\nThe soft spots are significant but not fatal. First, the same DeepSeek-V3.2 model is used as the user simulator during RL training and as the user simulator for τ²-Bench and ACEBench-Agent evaluation. That is a direct distribution match. A policy trained against a fixed user model can exploit its quirks—clarification behavior, tolerance for ambiguity—so those two benchmarks measure fit to the training user as much as general tool-use capability. BFCL doesn't use the simulator, so it's the cleanest evidence, and the gains there are meaningful. Second, no error bars, despite claiming four independent runs. The differences between GRPO and TARA are often under a point (28.50 vs 28.25 on BFCL for Qwen3-4B, one case where GRPO wins). Without variance, you can't tell signal from noise. Third, the theory in Appendix A is overstated. Theorem A.2 is a trivial variance-scaling consequence of multiplying future return by λ<1; it says nothing about bias or policy quality. Theorem A.4's distractor suppression assumes μ_local>0, i.e., at least one correct rollout in the group. If all rollouts fail, the local advantage is set to zero and the gate gives zero too, so the 'guaranteed non-positive' claim breaks. The empirical ablation is more convincing than the theorem; I'd cut or rewrite the theory.\n\nThe mock fidelity concern—only 3–5 records per DB, ~20% of toolsets discarded—is real but acknowledged. The external BFCL results suggest transfer, so I wouldn't call it fatal, just a boundary on the claim.\n\nWho is this for? People building agentic RL environments or credit-assignment methods. It deserves a serious referee. Send it to review, but the revision needs to use a held-out user model for evaluation, report variance, and fix the theory. Engage with it.","headline":"Good engineering, useful data, but the shared user simulator and missing error bars make the headline claims weaker than they look.","tokens_in":18148,"tokens_out":2533,"would_cite":true,"duration_ms":25919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning on 4,438 automatically synthesized mock API tools improves long-horizon tool use, and a turn-level gated advantage estimator beats trajectory-level credit assignment.","keywords":["agentic reinforcement learning","tool-integrated reasoning","long-horizon tasks","tool dependency graph","Dynamic Unlocking Sampling","turn-level credit assignment","Model Context Protocol (MCP) environments","GUST dataset"],"falsifier":"Run the same task templates and dialogue histories on live, stateful API backends instead of dictionary mocks, training identical models with TARA and with trajectory-level GRPO; if the TARA-vs-GRPO gap disappears or reverses on the live backend, the mocked-environment transfer premise is falsified.","tokens_in":17161,"feed_emoji":"🛠️","tokens_out":8510,"duration_ms":80411,"temperature":0.7,"pith_summary":"The paper tries to establish that agentic reinforcement learning can be scaled to long-horizon tool use by training on automatically synthesized, executable mock environments rather than hand-built single-tool simulators. The authors build ToolVerse, which converts roughly 422 real-world Model Context Protocol (MCP) toolsets — about 4,438 tools — into dictionary-backed Python environments, generates multi-turn tasks by sampling paths through a tool dependency graph, and trains LLM agents with a new turn-level advantage estimator called TARA instead of a single trajectory-level reward. They report consistent gains over baseline RL on BFCL-v3 Multi-Turn, τ2-Bench, and ACEBench-Agent across three model scales, and they show TARA outperforms naive group-relative optimization, especially on multi-turn and multi-step subtasks. If correct, the work offers a recipe for scaling agentic RL environments and a credit-assignment method that could generalize beyond tool use.","feed_headline":"Training on 4,438 mocked API tools lifts long-horizon agent scores","feed_subtitle":"Graph-built long-horizon tasks plus turn-level credit assignment beat trajectory-level RL on three agentic benchmarks.","key_machinery":"Two mechanisms carry the argument. First, a tool dependency graph (TDG) built by an LLM from tool schemas, combined with the Dynamic Unlocking Sampling (DUS) algorithm, which maintains a ready queue of zero in-degree tools and samples trajectory stages in topological order; this 'unlocks' each tool only after its prerequisites execute, generating coherent golden traces that are then replayed and filtered. Second, Turn-Aware Relative Advantage (TARA), which splits the advantage at each turn into a local term (binary turn reward, group-normalized) and a future term (discounted future rewards multiplied by a consistency gate equal to the current turn reward), then fuses them with weight lambda.","core_discovery":"The paper's central discovery, on its own terms, is that three components together — automated conversion of static tool schemas into executable dictionary-backed MCP environments, graph-guided task synthesis with Dynamic Unlocking Sampling, and the Turn-Aware Relative Advantage (TARA) estimator with a consistency gate — produce what the authors call 'universal performance gains' across diverse benchmarks and model architectures. TARA computes a binary reward per turn against a golden trace, normalizes it within the group at that turn, adds a gated discounted future term, and fuses the two. The ablation shows full TARA beats both the trajectory-level GRPO baseline and the local-only variant,","pith_inferences":["The local-plus-gated-future advantage decomposition is not specific to tool calls; any trajectory with a known reference plan (web navigation, robotics, dialogue) could use the same scheme, though the paper only tests API tool use.","The paper's environment-scaling result implies that adding near-duplicate toolsets would give diminishing returns; a direct test is to measure benchmark gains while increasing only the number of redundant environments.","Because the mock databases are deterministic dictionaries with 3–5 records, trained policies may be brittle to noisy, non-deterministic, or out-of-distribution real APIs; injecting stochastic or missing responses into the mocks during training is a natural extension.","The paper's own limitation note says predefined dependencies bound emergent behavior, so a worthwhile extension is to let the TDG grow with agent-discovered tool combinations and check whether long-horizon capability improves further."],"forward_implications":["Agentic RL environments can be scaled to thousands of executable tools with an automated pipeline, replacing hand-built single-tool simulators for many domains.","Turn-level relative advantages with a consistency gate give denser, more stable training signal than a single trajectory-level reward for long-horizon tool use.","Models trained on the GUST dataset transfer to three external multi-turn agentic benchmarks, with the largest gains on ACEBench-Agent's multi-turn subtasks.","Increasing training environment diversity from 100 to 422 toolsets improves benchmark scores, suggesting diversity rather than extra rollouts drives generalization.","The GUST dataset itself is a reusable resource of 3–7 task data items per environment, verified by replay and teacher-agent filtering."],"fun_headline_variants":["4,500-tool MCP universe trains agents for long-horizon tasks","Agentic RL scales to 4,500 tools with graph-sampled tasks","ToolVerse: 400 MCPs, 4,500 tools, longer agent training","Turn-aware credit assignment boosts long-horizon tool use","Graph-built tasks and TARA lift agent scores across benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper assumes that small dictionary-based mock databases (3–5 records per scenario) and LLM-inferred tool dependency graphs capture enough real API behavior that policies trained on them transfer to real-world tool use; the paper itself notes about 20% of toolsets cannot be faithfully mocked this way.","fun_headline_variants_meta":{"raw":{"variants":["4,500-tool MCP universe trains agents for long-horizon tasks","Agentic RL scales to 4,500 tools with graph-sampled tasks","ToolVerse: 400 MCPs, 4,500 tools, longer agent training","Turn-aware credit assignment boosts long-horizon tool use","Graph-built tasks and TARA lift agent scores across benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":3960,"prompt_tokens":744,"completion_tokens":3216,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":3118}},"tokens_in":488,"tokens_out":3216,"duration_ms":20161,"temperature":1.0,"reasoning_tokens":3118,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:38:56.133234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same task templates and dialogue histories on live, stateful API backends instead of dictionary mocks, training identical models with TARA and with trajectory-level GRPO; if the TARA-vs-GRPO gap disappears or reverses on the live backend, the mocked-environment transfer premise is falsified.","supporting_citations":[],"review_version":1}