{"id":"a230e93a-3126-4a3b-b079-628ef65fd064","arxiv_id":"2506.04625","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 7B open-source model fine-tuned on GPT-4-verified tool-use trajectories plus reflection data reports state-of-the-art pass rates on StableToolBench and high error-correction rates on a new benchmark.","lead":"This paper introduces Tool-MVR, a tool-augmented language model trained on a meta-verified instruction dataset and a reflection-learning dataset. On the StableToolBench benchmark it reports higher pass rates than GPT-4 with fewer API calls, and it introduces a new benchmark for tool-use reflection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states that ToolBench-V/RefineToolBench are built only from ToolBench's training split; if StableToolBench test queries entered SFT data, the headline gains are memorization.","rationale":"The reader's verdict already identifies the decisive issue: the training/evaluation split is never documented. I agree. Section 3.2.4 reports 11,765 ToolBench-V instances and Section 3.3.2 reports 3,625 ToolBench-R instances, but neither section says where in ToolBench the source queries came from. StableToolBench (Section 4.1.1) is explicitly derived from ToolBench, and its G1-Tool/G1-Cat/G3-Ins scenarios are only meaningful if the model has not seen those queries during SFT. If the MAMV filtering in Section 3.2.2 ran on the full ToolBench rather than the train split, the 'unseen' generalization numbers (e.g., 86.7% on G1-Tool) could be memorization rather than generalization. The construction of RefineToolBench I3 in Section 4.1.2 makes the risk concrete: the paper uses 'our Stage 1 multi-agent system' (the same APICallAgent/MAMV machinery that generates ToolBench-V) to obtain complete trajectories from StableToolBench, then injects errors. Any overlap between those test-domain trajectories and the training data would invalidate the 79.8% I3 ECR. This is not an accusation; the released repository may show a clean train-only filter. But the manuscript, as written, leaves the central claim unprotected at exactly its load-bearing point. I would not change the reader's CONDITIONAL verdict; the required clarification is a one-line split statement plus a released overlap check.","tokens_in":30339,"tokens_out":6241,"duration_ms":68028,"concrete_test":"Download the released ToolBench-V and ToolBench-R JSON files from the paper's GitHub and the official StableToolBench test set. Normalize query text (strip whitespace, lowercase) and compute exact-match and embedding-similarity overlap between (a) ToolBench-V queries and the 765 StableToolBench queries, and (b) ToolBench-R <wa,wo,ra> triples and the I3 trajectories generated from StableToolBench. Also verify from the data-building scripts that the source ToolBench queries were taken from the train split only. If any overlap > 0, or the scripts do not enforce the split, the headline claims are contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—a 7B open-source model beating GPT-4 on StableToolBench and achieving 58.9% ECR on RefineToolBench—requires a clean separation between the data used to build ToolBench-V/ToolBench-R and the StableToolBench test set. The paper never states this split. Section 3.2.2 describes filtering 62,147 ToolBench queries to 23,903 and then 11,765 instances, but does not say whether the 62,147 came from the training split only. StableToolBench (Section 4.1.1) is a 765-query subset of ToolBench; if the MAMV pipeline ran on the full ToolBench (train+test), then the same queries evaluated in StableToolBench are in the SFT training data, making the G1-Tool/G1-Cat/G3-Ins 'unseen' claims false and inflating Pass Rate. The same risk applies to RefineToolBench: Section 4.1.2 says I3 cases are built by running 'our Stage 1 multi-agent system' on StableToolBench and injecting errors. If those StableToolBench queries or the resulting trajectories also appear in ToolBench-R, the 79.8% I3 ECR becomes an in-distribution measurement. The release links code and data, so the split can be checked, but as written the evidence is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Tool-MVR, a tool-augmented LLM built by fine-tuning open-source backbones (Qwen-2.5-7B and LLaMA-3.1-8B) on two newly constructed datasets. The first, ToolBench-V, is produced by a Multi-Agent Meta-Verification (MAMV) pipeline that validates APIs, filters user queries, and verifies API-call trajectories. The second, ToolBench-R, is produced by an Exploration-based Reflection Learning (EXPLORE) algorithm that samples error steps, generates reflections, and constructs an \"error -> reflection -> correction\" training signal. The authors report state-of-the-art results on StableToolBench (83.8% average Pass Rate for the Qwen-based model, exceeding GPT-4 by 15.3%) and introduce a new benchmark, RefineToolBench, on which their model achieves a 58.9% average Error Correction Rate. They also report a 31.4% reduction in average API calls per task compared to GPT-4.","tokens_in":30627,"tokens_out":7926,"duration_ms":80833,"significance":"The work addresses an important problem: instruction data quality for tool-augmented LLMs and the largely missing capability of error recovery through reflection. The proposed pipeline, combining meta-verification with exploration-based reflection learning, is a plausible and potentially valuable design, and the reported results, if they hold under clean data hygiene and controlled evaluation, would be notable: a 7B open-source model outperforming GPT-4 on a tool-use benchmark while using fewer API calls. The release of code and data is a concrete strength that makes the claims checkable. The proposed RefineToolBench could also be a useful resource for the community. However, the significance is conditional on resolving the data-provenance and evaluation-bias concerns raised below, because the headline gains depend crucially on a clean train/test split and on evaluation procedures that do not systematically favor the proposed model.","major_comments":[{"comment":"The manuscript never states whether the 62,147 initial queries from which ToolBench-V was filtered originate exclusively from ToolBench's training split. Because StableToolBench (Section 4.1.1) is a curated subset of ToolBench, any overlap between the queries used for ToolBench-V/ToolBench-R and the 765 StableToolBench evaluation tasks would make the 'unseen' results in Tables 2 and 3, including the headline 83.8% Pass Rate, an in-distribution measurement. Please state the precise data split used, and provide an overlap analysis (by query identifier or exact text) between ToolBench-V, ToolBench-R, and the StableToolBench test set using the released data.","section":"Section 3.2.2 and Section 4.1.1"},{"comment":"RefineToolBench I3 cases are built by running the Stage 1 multi-agent system on StableToolBench tasks and injecting errors, while ToolBench-R is constructed through EXPLORE using ToolBench-V trajectories as references. The paper does not establish that the I3 cases are disjoint from ToolBench-R or ToolBench-V; indeed, Section 4.6.4 states that I3 cases 'contain familiar multi-tool tasks,' which suggests distributional overlap. If any I3 cases or their underlying trajectories appear in the training data, the 79.8% I3 Error Correction Rate is partially an in-distribution number. The authors should clarify the disjointness and, ideally, evaluate on error types not represented in the training distribution.","section":"Section 4.1.2 and Section 3.3.2"},{"comment":"The quality scores in Table 1 are obtained with GPT-4 after the same GPT-4-based MAMV pipeline has filtered and refined the data. This makes the reported improvements in query validity (52.7% to 98.8%) and trajectory accuracy (25.6% to 81.3%) partly self-confirming, because the evaluator and the filtering/refinement mechanism share the same model family and evaluation criteria. Please provide human-annotated quality judgments on a random sample, or use an independent verifier, to confirm that the quality gains are not an artifact of evaluator-model bias.","section":"Section 4.4 and Table 1"},{"comment":"The paper states that all baselines are combined with DFSDT for inference, but it does not state whether Tool-MVR also uses DFSDT. If Tool-MVR is evaluated with greedy decoding while the baselines use a search procedure, the Pass Rate comparisons in Tables 2 and 3 and the API-call efficiency comparison in Figure 4 are not controlled for inference strategy, and the 'reducing API calls by 31.4%' claim would be expected from the absence of search rather than from the learned model alone. Please specify the exact inference protocol for Tool-MVR and, if it differs from the baselines, include a controlled comparison using the same search strategy for all models.","section":"Section 4.2, Section 4.5 and Section 4.7"},{"comment":"All reported evaluation metrics (Pass Rate, Win Rate, ERR, ECR) are computed by gpt-4-turbo with custom prompts, and the same model family was used to generate the training trajectories and reflections in ToolBench-V and ToolBench-R. Although the same evaluator is applied to every model, this setup can still favor Tool-MVR if the evaluator's format preferences align with the training data distribution. Please report agreement with human evaluation on a subset of these judgments to establish that the metric is not systematically biased toward the proposed model.","section":"Section 4.1 and Appendix A"}],"minor_comments":[{"comment":"Several rows in Table 2 are typeset without separators (e.g., '79.179.279.086.9'), making the values difficult to parse; please reformat the table.","section":"Table 2"},{"comment":"The ablation discussion says 'the substantially lower performance of w/o Stage 1&2 (L) (47.3%)', but Table 2 shows 47.3% is the average for w/o Stage 1&2 (Q), while the L row averages 60.1%; the model label appears to be incorrect.","section":"Section 4.5.3"},{"comment":"The API pool statistics (11,302 original APIs, 6,770 validated, 6,334 enhanced, and 4,500+ simulated) need a clarifying set-relationship statement; as written, these totals do not form a partition and the overlap is ambiguous.","section":"Section 3.2.1"},{"comment":"The text mentions 'approximately 80,000 initial queries' and then an 'initial pool of 62,147' without explaining the relationship; please clarify whether the latter is after an initial API/format filter.","section":"Section 3.2.2"},{"comment":"Eq. (8) contains a single loss weight λ, but the text says the authors 'set λ=1 and maintain a 10:1 ratio between ToolBench-V and ToolBench-R'; please clarify whether the ratio is implemented via data sampling or via separate weighting, since it is not visible in Eq. (8).","section":"Section 4.3"},{"comment":"Reference [8] is a survey on spectral clustering and appears unrelated to tool-augmented LLMs; please check the citation mapping.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central concern is data provenance: the paper does not state whether ToolBench-V and ToolBench-R are built exclusively from ToolBench training data, and the new RefineToolBench I3 is constructed from StableToolBench itself. Since the authors release code and data, these issues are verifiable, but as written the evidence for the headline claims is conditional. I would also ask the editor to verify reference [8], which appears to be an unrelated spectral clustering survey, and to check whether the claim that RefineToolBench is the 'first' reflection benchmark is adequately supported by prior literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this paper if you care about tool-augmented LLMs. It does two things that are genuinely useful: a multi-agent verification pipeline to clean up ToolBench, and an exploration-based reflection learning method that trains models to correct API errors. The reported numbers are strong—a 7B Qwen model beating GPT-4 on StableToolBench by 15.3% and cutting API calls by 31.4%—but the biggest number depends on something the paper never says.\n\nThe MAMV pipeline is a solid practical contribution. The authors correctly identify that ToolBench has serious quality issues (57% unsolvable queries, 74% hallucinated trajectories, citing Quality Matters) and they build a sensible multi-agent verification framework to address them. Their quality scores jump from 52.7% to 98.8% (query) and 25.6% to 81.3% (trajectory), though those scores are measured by the same GPT-4 model family that did the filtering—so treat them as upper bounds. The EXPLORE method for reflection learning is a good idea: sampling error steps, generating reflections, and training on 'error→reflection→correction' triples. The new benchmark RefineToolBench, while synthetic, fills a gap.\n\nThe load-bearing problem is the split. The paper never states that ToolBench-V is built only from ToolBench's training split. It starts with 62,147 queries from ToolBench, filters to 23,903, then to 11,765 instances, but there is no sentence saying 'we excluded the StableToolBench test queries.' Worse, the I3 cases of RefineToolBench are explicitly built by running the Stage 1 system on StableToolBench, and the model is trained on ToolBench-R. If any of those test queries or their trajectories are in the training data, the G1-Tool/G1-Cat 'unseen' claims are false, and the I3 ECR of 79.8% is in-distribution. The authors may have done the right thing—they release code and data—but the paper as written is ambiguous, and the ambiguity is exactly where the headline numbers could break.\n\nSecondary issues: evaluation uses custom gpt-4-turbo prompts rather than the official StableToolBench evaluator, so the comparison to published baselines may not be like-for-like; and the quality assessment is self-confirming to a degree.\n\nWho is it for? People building tool-use agents or instruction data pipelines. The ideas are worth knowing even if the exact numbers need verification.\n\nRecommendation: send it to peer review, but require the authors to state the split explicitly, use the official evaluator, and report variance. If the split is clean, this is a meaningful result; if not, it's an artifact.","headline":"A useful data-quality and reflection pipeline for tool learning, but the headline numbers depend on an unstated train/test split that must be confirmed before the results can be trusted.","tokens_in":31160,"tokens_out":4275,"would_cite":true,"duration_ms":47109,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tool-MVR: verified data plus reflection learning lets a 7B model outperform GPT-4 on tool-use benchmarks.","keywords":["tool learning","large language models","instruction data verification","meta-verification","reflection learning","tool reflection","API calling","system 2 reasoning"],"falsifier":"Download the released ToolBench-V and ToolBench-R data, match their query and trajectory pairs against the 765 StableToolBench tasks and the 178 I3 cases of RefineToolBench, then retrain with any overlapping items removed; if the 83.8% pass rate and 58.9% correction rate collapse toward the roughly 47-60% range of the no-verification baselines, the reported gains are a test-set artifact rather than a learned skill.","tokens_in":30155,"feed_emoji":"🛠️","tokens_out":7726,"duration_ms":68289,"temperature":0.7,"pith_summary":"This paper claims that a 7B open-source language model can be made to out-plan, out-invoke, and out-reflect GPT-4 in tool-use tasks by fixing the training corpus and teaching error correction, rather than by scaling up. The authors identify two root causes of weak tool agents: instruction datasets full of unsolvable queries and hallucinated API calls, and static imitation learning that never trains on mistakes. They address the first with Multi-Agent Meta-Verification (MAMV), a GPT-4-driven pipeline that validates APIs, queries, and API-call trajectories and turns ToolBench into the higher-quality ToolBench-V. They address the second with Exploration-based Reflection Learning (EXPLORE), which generates ToolBench-R from failed steps and teaches an $\\text{Error}\\rightarrow\\text{Reflection}\\rightarrow\\text{Correction}$ loop. On StableToolBench the resulting Tool-MVR reaches an 83.8% pass rate, 15.3 points above GPT-4, and on the new RefineToolBench it corrects 58.9% of errors versus 9.1% for ToolLLM.","feed_headline":"7B model beats GPT-4 at tool use after data clean-up","feed_subtitle":"Verified instructions plus reflection training lift StableToolBench pass rate to 83.8% and cut API calls by 31.4%.","key_machinery":"The load-bearing machinery is the two-stage training pipeline: MAMV (Multi-Agent Meta-Verification) is a GPT-4-powered pipeline with three specialized agents: APIOptAgent for API validation, simulation, and documentation refinement, QueryVerifyAgent for feasibility and quality scoring, and APICallAgent for trajectory construction with Format, Semantic, and Execution checkers. EXPLORE (Exploration-based Reflection Learning) is the second mechanism: it samples step-level wrong actions from verified trajectories, feeds the tool's error observation to GPT-4 to generate a structured reflection, and stores the tuple (wrong action, feedback, reflection, corrected action) as a training instance. The training objective is the sum of an action-prediction loss $\\mathcal{L}_V$ over ToolBench-V and a reflection-plus-correction loss $\\mathcal{L}_R$ over ToolBench-R, combined as $\\mathcal{L} = \\mathcal{L}_V + \\lambda \\mathcal{L}_R$ with $\\lambda=1$ and a 10:1 data ratio, so the model learns deliberate planning, precise invocation, and feedback-driven correction in one supervised pass.","core_discovery":"On its own terms, the paper's central discovery is that the quality bottleneck in tool-augmented LLMs is not model scale but the verification of the three meta-components of tool use: the API pool, the user query, and the execution trajectory. MAMV upgrades ToolBench by testing 11,302 APIs against real calls, simulating the invalid ones, filtering 62,147 generated queries down to 23,903 solvable ones, and re-annotating trajectories with step-level format, semantic, and execution checks, raising query validity from 52.7% to 98.8% and trajectory accuracy from 25.6% to 81.3%. EXPLORE then samples wrong actions from the verified trajectories, lets GPT-4 turn tool feedback into structured reflections, and yields 3,625 instances where the model learns to recognize and correct its own mistakes. Trained jointly on the 11,765 verified instances and the reflection set, Tool-MVR (Qwen-2.5-7B) attains an 83.8% average pass rate on StableToolBench, 15.3 points above GPT-4, while using 31.4% fewer API calls, and a 58.9% error-correction rate on RefineToolBench against GPT-4's 51.1% and ToolLLM's 9.1%.","pith_inferences":["If the train/test split is clean, the method's logic transfers beyond tool use: any domain where instruction corpora are synthetically generated, such as code repair, retrieval-augmented generation, or planning, could be upgraded by the same verify-then-reflect recipe with domain-specific checkers.","The paper leaves implicit that its evaluation judge is itself a GPT-4 model, the same family that generated the verified trajectories; a fair reader should test whether the reported wins survive a judge-agnostic or open-weight evaluator.","A natural extension is to mine errors at inference time rather than only at data-construction time, turning EXPLORE into an online loop that keeps expanding ToolBench-R as the deployed model encounters new API failures.","The I3 portion of RefineToolBench is built from StableToolBench trajectories, so a stricter test would construct reflection cases from tools and queries never seen in training; the I1/I2 single-tool numbers (53-54% error-correction rate) may be the more honest estimate of reflection in novel settings."],"forward_implications":["A 7B or 8B open-weight model can exceed a frontier closed model on tool-use benchmarks if its training data is verified and its reflection ability is trained directly, which weakens the assumption that tool mastery requires proprietary scale.","Instruction quality dominates quantity: Tool-MVR uses 15,390 examples versus ToolLLM's 73,423, so dataset cleaning is a cheaper path to improvement than collecting more data.","Teaching error correction from execution feedback generalizes to unseen tools and categories, as shown by the highest pass rate (86.7%) on the G1-Tool scenario.","Fewer API calls per task (3.48-3.99 versus 5.19-5.92 for GPT-4 in the reported runs) means lower latency and cost for deployed tool agents.","A dedicated reflection benchmark can separate the ability to recognize an error from the ability to fix it, and the reported gap (58.9% versus 9.1% correction) suggests reflection is a trainable skill rather than an emergent one."],"supporting_citations":[{"why":"Supplies the ToolBench dataset and the ToolLLM baselines that MAMV verifies and that Tool-MVR is compared against.","marker":"[37]"},{"why":"Documents the query and trajectory quality problems in ToolBench and supplies the evaluation protocol used to score ToolBench-V.","marker":"[18]"},{"why":"Defines StableToolBench, the main benchmark whose pass-rate and win-rate numbers anchor the headline comparison.","marker":"[16]"},{"why":"Provides the DPO-based TP-LLaMA baseline that also learns from inference-tree errors, against which reflection learning is compared.","marker":"[3]"},{"why":"Provides the PPO-based StepTool baseline used in the main comparison.","marker":"[61]"},{"why":"Motivates APICallAgent's use of Python function calls instead of JSON for more accurate trajectory construction.","marker":"[51]"},{"why":"Establishes the thought-action-observation interaction format that APICallAgent and the reflection loop build on.","marker":"[59]"},{"why":"Identifies the Qwen-2.5-7B backbone used for the best-performing Tool-MVR model.","marker":"[58]"}],"fun_headline_variants":["7B tool model tops GPT-4 via verified data and self-reflection","Tool-MVR: 7B LLM beats GPT-4 via meta-verification and reflection","Data verification and error-correction: 7B tool model surpasses GPT-4","Verified APIs and self-correction: 7B model beats GPT-4 on tool tasks","Quality data + reflection lifts 7B model past GPT-4 in tool use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results assume ToolBench-V and ToolBench-R were built only from the ToolBench training split, so no StableToolBench test query or trajectory, and no RefineToolBench I3 case derived from one, leaked into the fine-tuning set; the paper never states this split.","fun_headline_variants_meta":{"raw":{"variants":["7B tool model tops GPT-4 via verified data and self-reflection","Tool-MVR: 7B LLM beats GPT-4 via meta-verification and reflection","Data verification and error-correction: 7B tool model surpasses GPT-4","Verified APIs and self-correction: 7B model beats GPT-4 on tool tasks","Quality data + reflection lifts 7B model past GPT-4 in tool use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3986,"prompt_tokens":1162,"completion_tokens":2824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":778,"completion_tokens_details":{"reasoning_tokens":2713}},"tokens_in":778,"tokens_out":2824,"duration_ms":17404,"temperature":1.0,"reasoning_tokens":2713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:37:30.051284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Download the released ToolBench-V and ToolBench-R data, match their query and trajectory pairs against the 765 StableToolBench tasks and the 178 I3 cases of RefineToolBench, then retrain with any overlapping items removed; if the 83.8% pass rate and 58.9% correction rate collapse toward the roughly 47-60% range of the no-verification baselines, the reported gains are a test-set artifact rather than a learned skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the query and trajectory quality problems in ToolBench and supplies the evaluation protocol used to score ToolBench-V."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DPO-based TP-LLaMA baseline that also learns from inference-tree errors, against which reflection learning is compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates APICallAgent's use of Python function calls instead of JSON for more accurate trajectory construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the thought-action-observation interaction format that APICallAgent and the reflection loop build on."}],"review_version":1}