{"id":"5adbf25c-df4f-4efc-a720-b2a64cd3b843","arxiv_id":"2412.01130","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning experiments show instruction-following data improves LLM function-calling accuracy and relevance detection, a Decision Token plus synthetic negative examples helps non-relevant cases, and a tailored translation pipeline improves Traditional Chinese function calling.","lead":"This paper tests practical recipes for making large language models call functions correctly: how to format prompts, whether to mix in instruction-following data, a special 'decision token' to flag when to use a tool, and a translation pipeline for Traditional Chinese. The takeaway for builders is that adding instruction-following data improves function-call accuracy and relevance detection, and a small amount of translated data helps multilingual tool calling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main claim is confounded: Table 1 compares FC-110k alone with FC-110k+IF-110k, so the AST/Relevance gains may come from doubling dataset size, not from instruction-following content; no quantity-matched or content-matched control is reported.","rationale":"The reader's weakest assumption concerns the correctness of synthetic non-function-call labels and the self-built ZHTW benchmark. Those are valid secondary concerns about the Decision Token and multilingual results. However, the most load-bearing weakness is in the paper's primary claim: Table 1's comparison does not isolate 'instruction-following data' as a variable because the FC-only rows use half as much total training data. This confound is prior to the NF-1k label question, since it undermines the headline conclusion regardless of how the synthetic negatives are constructed. A quantity-matched or content-matched control would settle the attribution. The reader's verdict of CONDITIONAL is appropriate and not changed by this analysis; the conditionality should be understood to include the need for such controls before the instruction-following claim can be accepted.","tokens_in":13527,"tokens_out":5723,"duration_ms":51701,"concrete_test":"Retrain the Table 1 row (d) configuration (FC-110k only) twice: (i) with 220k FC examples (e.g., add another 110k from the same APIGen/glaive sources) and (ii) with FC-110k plus 110k generic chat/instruction data that contains no function-call syntax. Compare AST Summary and Relevance Detection on BFCL against rows (b)/(c). If either control matches the ~85/50 values, the claimed effect is not specific to instruction-following content; if neither does, the conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline finding is that instruction-following data significantly improves function-calling accuracy and relevance detection (Abstract, Section 4.2, Section 5). The evidence is Table 1(b) vs (d) and (c) vs (e): training with IF-110k+FC-110k yields AST Summary 85.25/85.94 and Relevance Detection 49.58/39.58, versus 74.62/74.50 and 38.33/27.08 with FC-110k only. In both comparisons, the treatment adds 110k instruction-following instances, but it also doubles the training corpus and adds substantial data diversity. A model trained on only 110k function-calling examples may overfit to tool-call syntax and lose the ability to decline irrelevant functions; any additional diverse text—another 110k FC examples, generic chat, or domain text—could plausibly produce the same regularization/generalization gains. The paper reports no control that fixes dataset size or data domain, so the specific contribution of 'instruction-following' content is underdetermined. The claim in Section 4.2 that IF data helps the model 'understand the semantic structure of the prompts' is an untested hypothesis. A related but secondary issue is that Table 2 shows adding the Decision Token alone decreases Relevance Detection (49.58 to 37.50); the positive result appears only after adding NF-1k, and no NF-only ablation isolates the token. The Table 1 confound is the most load-bearing because it concerns the paper's primary claimed contribution.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how to improve the function-calling abilities of an LLM by varying prompt formats, mixing instruction-following data with function-calling data, introducing a pair of special Decision Tokens, adding synthetic non-function-call data, incorporating chain-of-thought reasoning, and using a translation pipeline for Traditional Chinese. The authors fine-tune Breeze-7B with LoRA on datasets IF-110k and FC-110k, evaluate on BFCL, MT-Bench, and a Traditional Chinese function-calling benchmark, and report three main claims: instruction-following data improves function-calling accuracy and relevance detection; the Decision Token combined with synthetic non-function-call data improves relevance detection; and the translation pipeline improves Traditional Chinese function calling. The experimental results are given in Tables 1-3.","tokens_in":13875,"tokens_out":2808,"duration_ms":27013,"significance":"If the claims were fully established, the paper would provide useful engineering guidance for practitioners building function-calling models, particularly regarding data mixing and relevance detection. However, the current evidence is weakened by confounded comparisons, missing ablations, lack of statistical rigor, and the use of a benchmark built in part by the authors. The contribution is therefore more of a promising empirical direction than a definitive conclusion.","major_comments":[{"comment":"The central claim that instruction-following data improves function-calling accuracy and relevance detection is not supported by the reported design. In Table 1, rows (b) versus (d) and (c) versus (e) compare FC-110k alone with FC-110k plus IF-110k, so the treatment manifoldly changes both the data content and the total training corpus size (110k vs. 220k examples). Without a quantity-matched or content-matched control, such as FC-220k without IF data or an additional 110k generic chat examples, the observed gains in AST Summary and Relevance Detection cannot be attributed specifically to instruction-following content; they could result from simple data-volume increases, regularization, or greater prompt diversity. Please add a control that isolates the effect of the IF data and, preferably, report multiple seeds.","section":"4.2, Table 1"},{"comment":"The Decision Token claim is underdetermined because no ablation isolates the token from the synthetic non-function-call data. In Table 2, adding the Decision Token alone actually lowers Relevance Detection in the dedicated-role condition (49.58 to 37.50) and raises it in the system-role condition (39.58 to 47.50), while the large increases appear only after NF-1k is added (57.50 and 65.42). Since no NF-only condition is reported, it is possible that the synthetic non-function-call data, not the Decision Token, drives the benefit. Please run an NF-only ablation and report all intermediate configurations.","section":"4.3, Table 2"},{"comment":"The synthetic non-function-call data generation relies on the assumption that after removing the original helpful function func_A, the remaining functions func_B and func_C are irrelevant to the user query. This assumption is not validated in the paper. If the remaining functions are semantically overlapping or generic enough to answer the query, the generated 'answer directly' labels are incorrect, and the measured Relevance Detection gains from NF-1k could reflect label noise rather than improved relevance detection. Please provide a validation procedure, such as human or model-based checks of the NF-1k labels, and report the estimated label accuracy.","section":"3.2"},{"comment":"All conclusions are based on point estimates from a single base model (Breeze-7B) without confidence intervals, significance tests, or multiple random seeds, yet the prose uses language such as 'significantly enhances' and 'significant improvements.' This is especially concerning for the translation claim in Table 3, where the ZHTW benchmark is constructed by Lee et al. (2024), whose author list includes two authors of this paper. Please report variance information (e.g., seeds or bootstrap CIs), temper the significance language accordingly, and disclose or mitigate the potential benchmark-author overlap by using an independent evaluation where possible.","section":"4.1 and 4.5, Tables 1-3"}],"minor_comments":[{"comment":"There is a typo in 'trained onIF-110k + FC-110k-Reason'; a space is missing after 'on'.","section":"4.4"},{"comment":"The phrase 'an non-function-call case generation pipeline' contains a grammatical error; it should be 'a non-function-call case generation pipeline.'","section":"4.5"},{"comment":"The 'Use of Data?' columns use check and cross symbols, but the symbols are not defined in the caption or the text. Please define them explicitly.","section":"Table 1"},{"comment":"The MT-Bench scores in Table 1 are reported as point estimates from a GPT-4o judge, but no information is given about judge stability or variance. A brief note or additional runs would improve confidence in the instruction-following comparison.","section":"4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable empirical study, but its headline claims currently outrun the experimental design. The most important fix is a quantity-matched control for Table 1; without it, the primary contribution is not established. I would also urge the editor to require the authors to address the benchmark-author overlap for the ZHTW evaluation and to either provide confidence intervals or soften the significance language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this. The headline claim—instruction-following data improves function-calling accuracy and relevance detection—is plausible but underdetermined, because the evidence in Table 1 compares FC-110k alone against FC-110k plus IF-110k, so the treatment adds data volume and diversity as well as IF content. There is no quantity- or domain-matched control. Second, the Decision Token is the least convincing piece of the recipe: in Table 2, adding the token alone actually lowers relevance detection in the dedicated-role prompt (49.58 to 37.50) and only rises after the synthetic non-function-call data (NF-1k) is added, with no NF-only ablation to isolate the token's contribution.\n\nWhat the paper does well: it is a clear recipe-level study of function-calling fine-tuning, comparing two prompt formats, testing CoT reasoning (and honestly reporting a null result), and proposing a sensible translation pipeline with explicit rules not to translate function names. The gains on Traditional Chinese are large, and the paper positions itself against ToolACE and related work instead of ignoring them.\n\nThe soft spots beyond the confound: the synthetic NF-1k labels assume that after deleting the helpful function from a multi-function prompt, the remaining functions are irrelevant to the query; that is plausible but unvalidated. There are no confidence intervals or significance tests, and the base model is the authors' own Breeze-7B. No code or data is released, and the Traditional Chinese benchmark (Lee et al., 2024) was built by two of this paper's authors, so that result should be treated as preliminary until independently evaluated. None of these are fatal; they are addressable.\n\nWho this is for: practitioners assembling function-calling fine-tuning data will find the prompt-format comparison and the NF-1k generation trick useful. It deserves a serious referee. I would send it to review with a request for a quantity-matched control (extra FC or generic chat data) and an NF-only ablation, and ideally an independent evaluation of the ZHTW benchmark.","headline":"Useful recipe paper on function-calling fine-tuning, but its headline claim about instruction-following data is confounded with dataset size, and the Decision Token's benefit actually comes from synthetic negative data.","tokens_in":14411,"tokens_out":3348,"would_cite":false,"duration_ms":27124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that instruction-following data, when mixed into function-calling fine-tuning, substantially improves both the structural accuracy of generated tool calls and the model's ability to detect when no tool is relevant.","keywords":["function calling","decision token","instruction-following data","relevance detection","multilingual translation","chain-of-thought reasoning","fine-tuning","synthetic data"],"falsifier":"Take a set of function-call samples, delete the called function, but leave in place one remaining function that is a near-duplicate (same purpose, different name). Fine-tune with the Decision Token + NF-1k recipe and test on relevance detection; if the model does not beat the baseline on this test, the synthetic-label assumption fails.","tokens_in":13307,"feed_emoji":"🤖","tokens_out":4487,"duration_ms":32964,"temperature":0.7,"pith_summary":"This paper tries to show that the way you format function descriptions and, more importantly, the data you mix into fine-tuning determine how well a large language model can call tools. Its central claim is that instruction-following data is not a distraction: adding 110k instruction-following examples to 110k function-calling examples raises function-call structural accuracy from 74.62 to 85.25 on the Berkeley Function Calling Leaderboard and raises relevance detection from 38.33 to 49.58. It also proposes a Decision Token, a pair of special output tokens the model emits before answering or calling a tool, which makes it easy to create synthetic 'no tool needed' training data and improves the model's ability to refrain from calling irrelevant tools. A dedicated translation pipeline for converting English function-calling data into Traditional Chinese is shown to lift a Traditional Chinese benchmark substantially. If true, these results say that reliable tool use is more about training-data composition and a clean decision point than about prompt format alone.","feed_headline":"Adding instruction data lifts tool-call accuracy by 10 points","feed_subtitle":"New fine-tuning recipe shows instruction data improves both tool-use accuracy and knowing when to answer directly.","key_machinery":"The central mechanism is the Decision Token: a pair of special tokens, <|answer|> and <|use_tool|>, that the model is trained to output first before generating a direct response or a tool call. Because next-token prediction is a classification, the Decision Token turns the 'should I call a tool?' judgement into an explicit binary classification. It also enables a cheap data-generation trick: take a sample where function A was called, delete A and keep functions B and C, assume B and C are irrelevant, and label the sample as an answer-directly case; training on such synthetic non-function-call data teaches the model to refrain from hallucinated calls.","core_discovery":"On the paper's own terms, the core discovery is that mixing instruction-following data into function-calling fine-tuning significantly improves both function-calling accuracy and relevance detection, and that a learned decision token combined with synthetic non-function-call data further improves the model's ability to know when not to call any tool. The evidence is a set of fine-tuning experiments on Breeze-7B evaluated on the Berkeley Function Calling Leaderboard and a Traditional Chinese translation of it.","pith_inferences":["An implication the authors leave implicit is that the same synthetic-negative trick generalizes to any slot that can be framed as a binary decision (e.g., whether to retrieve, whether to escalate), not just tool calling.","The instruction-following benefit may come from the model learning prompt-semantic structure rather than from the content of the instructions; a testable extension would be to mix in instruction data with tool-related vocabulary removed and see if the gain persists.","The Decision Token could serve as a cheap confidence signal: the probability assigned to <|use_tool|> versus <|answer|> might be used at inference time to decide when to defer to a fallback, a use the paper does not explore.","The paper's Traditional Chinese gain suggests the biggest multilingual bottleneck is the format of the data rather than the language itself, which would predict that the same pipeline works for other low-resource languages with minimal adaptation."],"forward_implications":["Instruction-following data should be considered an essential ingredient of function-calling fine-tuning, not a separate capability to be trained in a separate stage.","Putting function descriptions in a dedicated role, rather than mixing them into the system prompt, makes relevance detection easier to learn (49.58 vs 39.58 on the paper's baseline).","The Decision Token recipe can be used to generate non-function-call data from any existing function-call dataset at scale, without needing hand-written negative responses.","A translation pipeline that preserves function names and only translates arguments where sensible can extend function-calling models to a new language with a relatively small amount (19k examples) of translated data.","The translation pipeline also improves Traditional Chinese relevance detection, not just structural calling accuracy."],"supporting_citations":[{"why":"Supplies the BFCL benchmark and the AST Summary and Relevance Detection metrics used for all English function-calling evaluation.","marker":"Yan et al., 2024"},{"why":"Source of the Open ORCA instruction-following data (IF-110k) whose addition is shown to improve function-calling accuracy and relevance detection.","marker":"Longpre et al., 2023"},{"why":"Source of the APIGen function-calling data (part of FC-110k) used in fine-tuning.","marker":"Liu et al., 2024b"},{"why":"Breeze-7B, the base model fine-tuned in all experiments; its Traditional Chinese strengths make the ZHTW evaluation possible.","marker":"Hsu et al., 2024"},{"why":"Provides the Traditional Chinese translation of BFCL (ZHTW leaderboard) used to measure multilingual function-calling performance.","marker":"Lee et al., 2024"}],"fun_headline_variants":["Instruction data boosts LLM tool-call accuracy","Decision token teaches LLMs when to skip tool calls","New recipe: instruction data plus decision token","Multilingual function calling via translation pipeline","Mix instruction data to lift function-calling and relevance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic non-function-call data labels assume that after deleting the helpful function, the remaining functions are truly irrelevant to the user query, so the 'answer directly' label is correct.","fun_headline_variants_meta":{"raw":{"variants":["Instruction data boosts LLM tool-call accuracy","Decision token teaches LLMs when to skip tool calls","New recipe: instruction data plus decision token","Multilingual function calling via translation pipeline","Mix instruction data to lift function-calling and relevance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1208,"prompt_tokens":791,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":407,"tokens_out":417,"duration_ms":4109,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:39:52.953206+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of function-call samples, delete the called function, but leave in place one remaining function that is a near-duplicate (same purpose, different name). Fine-tune with the Decision Token + NF-1k recipe and test on relevance detection; if the model does not beat the baseline on this test, the synthetic-label assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Traditional Chinese translation of BFCL (ZHTW leaderboard) used to measure multilingual function-calling performance."}],"review_version":1}