{"id":"6644e352-9012-4089-b6b4-e6bef860f881","arxiv_id":"2505.07512","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ToolACE-DEV decomposes tool learning into documentation adaption, tool generation, and invocation, then lets the model self-train on its own majority-voted invocations, improving function-calling accuracy across three model families.","lead":"This paper presents a training recipe that lets small language models improve their own tool-calling skills by generating practice problems and fine-tuning on their own best answers. The authors report that a 7-billion-parameter open model reaches near the level of a 22-billion-parameter specialist and of GPT-4 turbo on a widely used function-calling benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 5's majority-voted self-labels are never checked for semantic correctness, so the claim that self-evolution matches GPT-4-based data synthesis rests on an unvalidated ground truth.","rationale":"The paper's ablations and cross-model experiments are genuine empirical work, and the consistent upward BFCL curves in Figure 2 are suggestive. However, the entire novelty is self-evolution, and the self-evolution loop is only as good as its pseudo-labels. The majority-vote heuristic from self-consistency has not been validated for open-ended tool calls, where correct invocations can be expressed in multiple valid JSON forms and a systematic error can win the vote. The rule checker handles syntax, not semantics. The absence of an oracle check means the central claim is conditional: if an audit later shows high voted-label accuracy, the claim stands; if not, the paper's headline comparison to advanced-model distillation is unsupported. I therefore keep the reader's conditional verdict rather than moving it.","tokens_in":15912,"tokens_out":10212,"duration_ms":97877,"concrete_test":"Oracle-audit the self-generated labels: for each evolution round, take a random sample of 150 (query, candidate tools, voted invocation) triplets; have an independent judge (human or a stronger model with executable API checks) label each invocation as semantically correct or not, with the main failure modes being wrong tool, wrong/missing arguments, and missing parallel calls. If round-1 to round-3 voted-label accuracy is below ~80% or does not rise with rounds, the self-evolution training signal in Eq. 6 is unreliable and the Figure 2 gains cannot be attributed to genuine tool-calling improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the self-evolution loop (Eq. 6) improves tool invocation as effectively as synthesizing data with advanced models. The loop's only quality controls are majority vote over five self-generated invocations (Eq. 5) and a rule checker that removes format/type errors. Neither verifies semantic correctness: a majority sample can call a plausible but wrong tool, fill a required argument with a plausible but wrong value, or omit necessary parallel calls, and the checker cannot detect this because it has no external grounding. The paper never reports the oracle accuracy of the resulting (query, candidate tools, voted invocation) triplets. If the model has a systematic bias, majority voting amplifies rather than corrects it, and iterated fine-tuning on these pseudo-labels can improve fluency on the model's own output distribution while leaving true tool-calling ability unchanged or worse. Figure 2's consistent BFCL gains are therefore not sufficient evidence for the 'self-evolution as effective as advanced-model data synthesis' claim until the labels themselves are shown to be correct. The unspecified source of the 10,000 queries per round (Sec. 4.1) compounds this: if those queries resemble the BFCL test distribution, even distribution-fitting could inflate the benchmark numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ToolACE-DEV, a three-stage framework for tool learning in lightweight LLMs: (i) tool documentation adaption, (ii) query-aware tool generation plus invocation, and (iii) a self-evolution loop in which the model generates candidate tools, samples multiple invocations, and fine-tunes on majority-voted pseudo-labels. Experiments on Llama-3.1-8B, Qwen2.5 variants (1.5B, 3B, 7B), and Mistral-7B report consistent improvements on BFCL, API-Bank, and T-Eval, with the 8B model reaching an Overall score of 82.44 on BFCL. The authors claim that self-evolution is as effective as data synthesis with advanced models, and that task decomposition is key to the gains.","tokens_in":16141,"tokens_out":3196,"duration_ms":28732,"significance":"If the central claim is validated, the framework offers a practical route to reducing dependence on costly advanced-model data synthesis for tool calling, and the decomposition into documentation adaption, tool generation, and invocation is a plausible and potentially transferable design. The paper deserves credit for the breadth of the empirical study: ablations across model families and scales, three benchmarks, and clear per-stage analysis. The self-generated training signal, however, is the load-bearing component, and the manuscript does not yet establish that this signal is semantically correct or that the benchmark comparisons are apples-to-apples. These gaps are fixable in principle, making the work a strong candidate for major revision rather than rejection.","major_comments":[{"comment":"The self-evolution loop selects the training target for each query by majority voting over five invocations sampled from the model itself, with only a rule checker for format/type errors. The paper never reports the oracle accuracy of these voted labels, nor does it compare them against any external ground truth. If the model has a systematic bias for a class of queries, majority voting can amplify rather than correct that bias, and iterated fine-tuning on such pseudo-labels can improve fluency on the model's own output distribution without improving genuine tool-calling ability. Since the central claim is that self-evolution matches advanced-model data synthesis, the authors should validate the pseudo-labels on a held-out set (e.g., by measuring agreement with human or LLM-as-judge correctness, or by executing calls in a sandbox) and report the accuracy of the voted invocations per evolution round.","section":"Section 3.4, Eq. (5)"},{"comment":"The source of the 10,000 incoming queries per self-evolution round is not specified. If these queries overlap with the BFCL test distribution, or even resemble it in domain and style, the observed BFCL gains could partly reflect distribution fitting rather than improved tool-calling ability. The paper should state where the queries come from, how they were sampled, and what decontamination checks were performed against BFCL, API-Bank, and T-Eval. At minimum, the authors should demonstrate that queries from different rounds do not duplicate the benchmark test cases.","section":"Section 4.1"},{"comment":"The comparison with the BFCL leaderboard is problematic. The paper states that evaluation focused exclusively on single-turn AST data, yet the baseline scores in Table 1 are taken from the official leaderboard, whose Overall score aggregates additional categories (e.g., multi-turn and executable-style cases). A score computed on a subset with different composition is not directly comparable to the published Overall score of other models. The authors must either recompute all baseline scores on the same single-turn AST subset used for ToolACE-DEV, or evaluate ToolACE-DEV on the full leaderboard protocol. In addition, the sentence 'the best results (may not be at the third round) are adopted in Table 1' implies test-set-based model selection across evolution rounds; if the round was chosen by looking at BFCL scores, the reported numbers are optimistically biased and should be replaced by a prespecified selection rule (e.g., a validation split) or by reporting all rounds.","section":"Table 1 and Section 4.1"},{"comment":"The claim that 'the self-evolution process is as effective as data synthesis with advanced models' is not directly supported by the presented experiments. The comparison is between ToolACE-DEV (which includes the two supervised stages plus self-evolution) and ToolACE-8B, which was trained on a different dataset size and composition. The ablation in Table 3 shows that Invo.+Gen. w. Adaption already reaches 81.67 Overall, while the full ToolACE-DEV reaches 82.44, an increment of 0.77 points. To substantiate the equivalence claim, the authors need a controlled comparison: for example, replacing self-evolution with an equal number of additional GPT-4-synthesized samples, or training a variant that uses only GPT-4-synthesized data at the same total scale as the self-evolved data. Without such a comparison, the headline claim overreaches.","section":"Section 4.2 and Table 3"}],"minor_comments":[{"comment":"The phrase 'first self-evolutionary framework designed to enhance LLMs' tool-invocation capabilities' is an overclaim given prior self-training work for tool use (e.g., Toolformer); consider softening to 'a self-evolutionary framework' and discussing the distinction more carefully.","section":"Abstract and Section 1"},{"comment":"The text refers to a '1.8B model,' but the corresponding figure and experiments use Qwen2.5-1.5B. Please correct the model size or clarify the configuration.","section":"Section 4.5 and Figure 3(b)"},{"comment":"The phrase 'leading to a hypothesize' should be 'leading to a hypothesis.'","section":"Section 4.3"},{"comment":"There are minor typos in the references and appendices, such as 'V olume' in the T-Eval citation and duplicated system prompts in Figures 6 and 7; a careful proofread is recommended.","section":"References and prompts"},{"comment":"The term 'adaption' is used throughout; 'adaptation' is the more standard spelling in English. Consistency would improve readability.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The BFCL comparison issue is the most serious: comparing a subset-based score to the official leaderboard Overall is a methodology problem that, if not fixed, would undermine the headline result. The self-label validation and query-source disclosure are also essential. I believe these can be addressed within the scope of a revision, so I do not recommend rejection, but the revision must include concrete evidence, not just discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a serious empirical paper and deserves a real referee, but the central claim—that self-evolution matches GPT-4-synthesized data for tool calling—is not yet proven. The design is genuinely new: decomposing tool learning into documentation adaption, query-aware tool generation, and invocation, then letting the model iterate on its own data. The ablations are clean and the BFCL gains are consistent across LLaMA, Qwen, and Mistral backbones. That consistency is real evidence, not an artifact. The LoRA setup is modest but fine.\n\nWhere it gets soft: Eq. 5 selects training labels by majority vote over five self-generated invocations. The rule checker catches format errors and some hallucinated tool names or wrong argument types, but it does not verify semantic correctness of arguments or whether the chosen call actually answers the query. So the loop can fit the model's own output distribution while true tool-calling ability plateaus. The paper never reports oracle accuracy on the self-generated triplets, and the source of the 10,000 queries per round is never stated. If those queries resemble the BFCL test distribution, part of the gain could be distribution-fitting. The BFCL comparison also mixes a single-turn AST subset with leaderboard overall scores, and the authors pick the best of three evolution rounds. Each of these is addressable; together they make the headline conditional.\n\nNone of this sinks the paper. The two supervised stages contribute clear gains, and self-evolution adds a consistent few points on top. The decomposition is worth copying. But the phrase 'self-evolution process is as effective as data synthesis with advanced models' is stronger than the evidence.\n\nWho should read it: people building tool-augmented assistants on small open models, and anyone working on self-training loops generally. I'd cite it with a caveat.\n\nRecommendation: send it to peer review. Ask for (1) an oracle or human check on a sample of self-generated labels, (2) the query source and a contamination check, (3) a consistent BFCL evaluation protocol, and (4) confidence intervals. With those, the claim would be credible.","headline":"A solid, well-ablated empirical framework whose headline claim about self-evolution needs an external check on the self-generated labels before I'd take it at face value.","tokens_in":16722,"tokens_out":2702,"would_cite":true,"duration_ms":24723,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-evolution lets an 8B model beat larger rivals at tool calling.","keywords":["tool learning","self-evolution","tool invocation","function calling","data synthesis","lightweight language models","majority voting","tool generation"],"falsifier":"On a held-out set of several hundred queries with hand-verified correct invocations, run one evolution round and measure the accuracy of the majority-voted calls that become training data; if that accuracy is low or no better than the model's raw accuracy, the loop is training on its own noise. A sharper test compares the same base model fine-tuned on self-generated triplets versus an equal number of verified human or advanced-model triplets: the paper's claim predicts the self-generated set is not noticeably worse.","tokens_in":15684,"feed_emoji":"🛠️","tokens_out":7577,"duration_ms":66334,"temperature":0.7,"pith_summary":"ToolACE-DEV asks whether a lightweight language model can improve its own tool-calling ability without a stronger teacher. The paper's answer is yes, provided the task is first decomposed: the model learns to reconstruct tool documentation, to generate query-relevant candidate tools, and to invoke them. After that preparation, the model writes its own new training examples, generating tools for fresh queries, sampling five invocations, and keeping the majority vote, and is fine-tuned on them. On the BFCL function-calling leaderboard, an 8B model trained this way reaches 82.44 overall, above several larger and closed-source models and on par with a much larger mixture-of-experts model. The central claim is that this self-evolution loop contributes real gains on top of the supervised stages, matching the effect of synthesizing data with an advanced model.","feed_headline":"Self-evolution lets an 8B model beat larger rivals at tool calling","feed_subtitle":"ToolACE-DEV trains on its own generated queries and tools, hitting 82.44 overall on the BFCL leaderboard.","key_machinery":"The load-bearing mechanism is the decomposition of tool learning into two complementary training objectives plus a preparation task, followed by a self-consistency-based evolution loop. The preparation task, tool documentation adaption, trains the model to reconstruct a full tool definition from a one-line description, teaching the syntax of tool schemas. The two decomposed objectives are query-aware tool generation, turning a query into a set of candidate tools, and tool invocation, picking tools and filling arguments. In each evolution round the model generates candidate tools with sampling, draws five invocations, applies majority voting to choose the pseudo-ground-truth call, rejects only format errors with a rule checker, and updates itself with both objectives on the new triplets. The loop is what allows the model to expand its own tool set instead of relying on a fixed pool.","core_discovery":"The paper claims that tool learning for lightweight LLMs does not need to be a one-way distillation from advanced models. It decomposes the usual single objective into three teachable pieces: adapting to tool documentation, generating candidate tools from a query, and invoking those tools. Once an 8B model has these pieces, it can run a self-evolution loop: for each new query it invents its own tools, samples five possible invocations, takes the majority vote as ground truth, filters only format-level errors with a rule checker, and fine-tunes on the resulting triplets. The reported result is an 82.44 overall score on BFCL, exceeding Llama-3-70B-Instruct and several larger or closed models and matching xLAM-8x22B-r, with gains that persist across Llama, Qwen, and Mistral backbones. The paper states its conclusion as: self-evolution is as effective as data synthesis with advanced models.","pith_inferences":["The paper never checks whether majority-voted invocations are semantically correct; verifying them against a small set of human-annotated queries would separate genuine self-evolution from self-distillation.","The 10,000 queries per evolution round are not described as coming from a distribution different from BFCL, so a clean test would use held-out-domain queries to see whether the gains transfer rather than reflect benchmark-like fitting.","The success of majority voting suggests the model's errors are mostly unsystematic; where errors are systematic, such as a recurring argument-filling mistake, the loop would amplify them, and adding a verifier for a subset of generated calls could make the loop safer.","If the loop holds beyond function calling, it provides a recipe for self-improvement in any structured output task with a checkable schema, such as SQL generation or API orchestration."],"forward_implications":["A 7-8B model can improve its function-calling accuracy without access to costly advanced models, cutting expense and sidestepping privacy restrictions that prohibit sending user queries to an external service.","Because the model invents candidate tools for fresh queries, it is not limited to a fixed tool pool, which targets zero-shot generalization to unseen scenarios.","Gains are larger on the harder, user-contributed live subset of BFCL than on the synthetic non-live subset, suggesting the self-generated training data is not trivially easy.","Self-evolution gains diminish with each round, and smaller models such as 1.5B and 3B stop improving or start to fluctuate, indicating a scale threshold for the benefit.","The same decomposition improves different model families, Llama, Qwen, and Mistral, so the mechanism is not tied to one base model."],"supporting_citations":[{"why":"Supplies the ToolACE dataset of 26,522 tool documents used for the documentation-adaption stage and the ToolACE-8B model that serves as a key comparison baseline.","marker":"Liu et al. 2024a"},{"why":"Provides the BFCL benchmark and leaderboard scores that measure the tool-invocation accuracy central to the paper's comparisons.","marker":"Yan et al. 2024"},{"why":"Introduces self-consistency decoding, the top-k sampling plus majority-vote procedure used to produce pseudo-ground-truth invocations in each evolution round.","marker":"Wang et al. 2023"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method used to train all models in the experiments.","marker":"Hu et al. 2022"},{"why":"Provides the prior observation that self-play gains diminish with iterations, which the paper uses to explain the flattening of its evolution curves.","marker":"Chen et al. 2024d"}],"fun_headline_variants":["Decompose, self-evolve: 8B model hits 82.4 on BFCL","Tool learning without distillation: 8B self-evolves","Self-improving tool learning: 8B outdoes 70B models","Self-evolving loop turns 8B into a tool-calling rival"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop trusts the model's own majority-voted invocations as correct training labels, and the rule checker only removes format errors, so if the majority is often semantically wrong for some class of queries, fine-tuning on those triplets amplifies the error.","fun_headline_variants_meta":{"raw":{"variants":["Decompose, self-evolve: 8B model hits 82.4 on BFCL","Tool learning without distillation: 8B self-evolves","Self-improving tool learning: 8B outdoes 70B models","Self-evolving loop turns 8B into a tool-calling rival"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000349,"raw_usage":{"total_tokens":1877,"prompt_tokens":886,"completion_tokens":991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":907}},"tokens_in":502,"tokens_out":991,"duration_ms":8464,"temperature":1.0,"reasoning_tokens":907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:14:58.912294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of several hundred queries with hand-verified correct invocations, run one evolution round and measure the accuracy of the majority-voted calls that become training data; if that accuracy is low or no better than the model's raw accuracy, the loop is training on its own noise. A sharper test compares the same base model fine-tuned on self-generated triplets versus an equal number of verified human or advanced-model triplets: the paper's claim predicts the self-generated set is not noticeably worse.","supporting_citations":[],"review_version":1}