{"id":"05e12a21-e8ee-4675-93bf-570ba4c5c520","arxiv_id":"2505.20016","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TTPA teaches LLMs correct tool calls by sampling near-tie token choices, scoring them with an error taxonomy, and training with DPO.","lead":"Large language models often fail to call external tools because of tiny errors: a missing brace, a wrong field name, a bad date type. This paper presents a training method that generates fine-grained preference data around such errors and uses it to align a model's token-level choices, reporting gains on tool-use benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No SFT-only baseline isolates the proposed preference alignment from the effect of the reversed-construction dataset, so the ToolBench gains could be driven entirely by data quality rather than by the claimed token-level alignment.","rationale":"The reader's weakest_assumption focused on the error taxonomy and weights being a valid, complete quality measure. That is an important issue because the same F is used for training and for the in-house test set, creating a circular evaluation. However, the single most load-bearing gap is the absence of any SFT-only or component ablation that would attribute the measured improvements to the proposed token-level preference alignment rather than to the reversed-constructed data itself. The paper's Table 5 only contrasts the base model and the full TTPA, so the 36-point jump on ToolBench could be explained by supervised learning on clean, carefully generated instruction data. This confound affects the interpretation of the external benchmark results (ToolBench and BFCL), not just the custom metric, and it directly targets the novelty claim. The reader did mention 'no component ablation' in the rationale, so there is partial agreement, but the reader did not identify it as the weakest assumption. The proposed test is concrete, feasible with existing assets, and would decisively show whether TPS/ESM/DPO add value beyond the reversed dataset. Because this is an addressable missing experiment rather than a demonstration that the method fails, the conditional acceptance verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":17103,"tokens_out":8939,"duration_ms":99663,"concrete_test":"Train Qwen2.5-7B-Instruct with supervised fine-tuning only on the same 3,895 reversed-construction instruction instances, using the identical hyperparameters from Table 6 (learning rate 1e-4, 5 epochs, LoRA rank 16), without TPS, ESM, or DPO. Evaluate this SFT-only model on ToolBench (I1-instruction and I1-tool, vanilla subsets), BFCL, and the custom test set. If the SFT-only model's ToolBench vanilla scores are within 2 points of TTPA's 86.0/85.0, the preference-alignment components are not responsible for the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TTPA significantly improves tool-use performance is not supportable without an SFT-only control on the same generated instruction data. The framework bundles three components: reversed dataset construction (3,895 instruction instances), token-level preference sampling (TPS), and error-oriented scoring (ESM) with DPO (8,550 preference pairs). The only ablation, Table 5, compares the base Qwen2.5-7B-Instruct against the full TTPA model; it never isolates the contribution of the preference-alignment components. Since the reversed construction intentionally produces clean, answerable, leakage-free queries, even plain SFT on those 3,895 instances could plausibly explain the large jump from roughly 50% to 86% on ToolBench I1-instruction: the model simply learns the tool-call format and the mapping from query phrases to arguments. If SFT-only matched TTPA, then the specific novelties (TPS, ESM, DPO) contribute nothing measurable, and the paper's central claim collapses to a data construction recipe. The reader's concern about the error taxonomy is real but downstream: a biased F would corrupt the custom test set, whereas the missing SFT-only baseline undermines attribution on the external benchmarks too. This omission is directly addressable and is the most load-bearing gap in the argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TTPA, a training framework for tool-use LLMs that combines three components: reversed dataset construction, token-level preference sampling (TPS), and an error-oriented scoring mechanism (ESM). The reversed construction generates queries from scenario-and-answer traces; TPS samples candidate tool calls from near-tie token distributions; ESM scores tool calls by a hand-set taxonomy of six error types. The authors fine-tune Qwen2.5-7B-Instruct with DPO on the resulting preference data and evaluate on ToolBench, BFCL, and a custom test set, reporting large gains over baselines—most strikingly a 57.8% tool-name accuracy on the custom test set versus 43.0% for GPT-4o-mini.","tokens_in":17416,"tokens_out":6059,"duration_ms":59858,"significance":"If the central claim holds, TTPA would provide a practical recipe for enabling 7B-scale models to reach frontier-level tool-use performance on several benchmarks, which is of clear practical value. The reversed data construction idea and the effort to model token-level uncertainty in structured outputs are interesting and could influence future tool-learning work. The manuscript is transparent about its hyperparameters, provides full prompt templates in the appendix, includes a case study of BFCL failures, and states a limitations section honestly. However, the current evidence is not yet sufficient: the only ablation (Table 5) omits an SFT-only control on the generated instruction data, and the custom test set is scored by the same reward function used for training, producing a circular in-house evaluation. These gaps must be closed before the contribution is established.","major_comments":[{"comment":"The ablation study compares only the base Qwen2.5-7B-Instruct with the full TTPA model. Because TTPA bundles reversed dataset construction (3,895 instruction instances) with token-level preference sampling and DPO (8,550 preference pairs), the reported gains—e.g., 46.3% to 86.0% on ToolBench I1-instruction average—cannot be attributed to the preference-alignment components without an SFT-only control trained on the same 3,895 instances. Such a control is essential to rule out the possibility that the improvement comes entirely from the higher quality of the reversed-construction instruction data, which would leave the central claim unsupported. Please add this baseline.","section":"§5.3, Table 5"},{"comment":"The custom test set is scored with the same error-oriented function F used as the DPO training signal. Consequently, Table 4's headline numbers (e.g., 57.8% tool-name accuracy for TTPA vs. 43.0% for GPT-4o-mini) partly measure how well the model optimized its own reward, not general tool-use ability. To support the claim of improved tool use, the custom test set should be evaluated with an independent metric—such as exact match against golden calls or human judgment—or the paper should explicitly label this test set as an in-distribution diagnostic and base its generalization claims on ToolBench and BFCL only.","section":"§4.3, Table 4 and §3.4, Eq. (6)"},{"comment":"The token-level preference sampling procedure is underspecified. Eq. (1) writes C_K_sam ~ P_pred I(Dist < epsilon), which is not a well-defined sampling distribution, and the text says K is 'dynamically determined based on the specific probability' but never defines this relationship. Since TPS is one of the two main novel components, a precise algorithm (with the threshold epsilon, the number of perturbed tokens per step, and the rule for computing K) is required for reproducibility.","section":"§3.3, Eqs. (1)-(2)"},{"comment":"The error taxonomy and the weights omega_i in Eq. (6) are introduced without validation. The weights are described as 'empirically set based on preliminary observations' (A.3) and no sensitivity analysis is provided. Because F is used both to construct preference pairs and to score the custom test set, a mis-specified F will bias the in-house results. Please add a sensitivity analysis over the weights and, if possible, a human agreement study on a sample of tool-call ratings.","section":"§3.4 and A.3, Table 7"}],"minor_comments":[{"comment":"The symbol delta_ei is defined as a 0/1 indicator in Eq. (3) and then as a weighted average over parameters in Eq. (4); please use distinct notation or clarify that Eq. (4) applies only to parameter-type and parameter-value errors.","section":"§3.4, Eqs. (3)-(4)"},{"comment":"The table lists five weights, but Figure 2 defines six error types; please clarify the mapping from error types to weights.","section":"Table 7"},{"comment":"The radar chart lacks numeric labels; reporting the underlying numbers would make the claim of maintained general performance verifiable.","section":"Figure 3"},{"comment":"The sampling ratio formula 'Ratio = Pt i=0 x / t' is ambiguous; please define x, t, and the summation index precisely.","section":"A.6"},{"comment":"The text contains typos such as 'preferrence' (Introduction) and 'Malage' (Figure 5) instead of 'Malaga'; please proofread.","section":"§1 and Figure 5"},{"comment":"The paper mentions an anonymous GitHub link but no URL is given; please include the repository link or state that code will be released upon publication.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The missing SFT-only baseline is the key technical issue: without it, the paper cannot attribute the reported gains to the proposed preference-alignment components rather than to the reversed-construction data. The circularity of the custom test-set evaluation is a second concern that undermines the headline numbers. Both issues are addressable within the manuscript's scope, so I recommend major revision rather than rejection. The paper would be suitable if the authors add the control, re-frame or re-evaluate the custom test set, and clarify the sampling algorithm."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe one thing to know: TTPA bundles a genuinely interesting data-generation idea (reverse the flow: scenario → tool calls → answer → query) with token-level DPO preference pairs scored by a hand-built error taxonomy. The central claim—that token-level preference alignment improves tool use—is plausible, but the paper does not actually isolate it. The only ablation is base Qwen vs full TTPA, so the big ToolBench jumps (50%→86%) could be caused by the cleaner reversed dataset alone, with the preference alignment contributing nothing measurable. The stress-test note is right, and this is the load-bearing gap.\n\nWhat is new and good: the reversed construction is a simple fix for leakage and unanswerable queries in synthetic tool-use data, and worth borrowing even if the rest of the framework failed. Near-tie token sampling is a sensible way to create fine-grained preference pairs for structured outputs, and the error taxonomy is a reasonable, explicit reward signal. The paper also reports on external benchmarks (ToolBench, BFCL) and is honest about the mixed BFCL results, including a plausible case study of evaluation-system false negatives.\n\nWhere it is soft:\n- No SFT-only control trained on the same 3,895 reversed instances. This is the most important missing experiment.\n- No trajectory-level DPO baseline (the paper cites CHEN et al. 2024 but never compares against it).\n- The custom test set uses the same weighted error score F that defines the DPO pairs, so the in-house gains are partly a measure of fitting the training objective. On the external benchmarks the gains are real but moderate: five of six ToolBench subsets, two of five BFCL subsets.\n- The error weights (3,3,1,2,2) are hand-set with no sensitivity analysis; the paper admits this in Appendix A.3.\n- No code or data link appears in the text, despite the anonymous-GitHub sentence.\n\nNone of these sink the idea, and the limitations section is honest about compute and static-tool coverage. But as written, the attribution claim is not supported. The fix is a small set of experiments: add SFT-only, add one trajectory-level DPO control, and report sensitivity to the error weights. If those hold, this is a solid contribution to tool learning and to structured-output alignment more generally.\n\nI would send it to peer review with those requirements. It deserves referee time; it just needs to earn the central claim.\n\nBest.","headline":"Plausible recipe—reversed data construction plus token-level DPO—but without an SFT-only baseline the paper cannot attribute its ToolBench gains to the preference alignment itself.","tokens_in":17916,"tokens_out":2564,"would_cite":true,"duration_ms":29738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token-level preference alignment for tool calls lets a 7B model match or beat frontier API models on tool-use tests.","keywords":["token-level preference alignment","tool learning","function calling","direct preference optimization","synthetic data construction","error taxonomy","LLM fine-tuning"],"falsifier":"A reader could settle the claim by taking the model trained under TTPA and re-evaluating it on held-out real user queries using a scoring function that does not share TTPA's error taxonomy, such as exact-match parse success or human judgment of whether the returned call would execute correctly; if the large gap over baselines disappears under that independent metric, the reported gains are an artifact of training and evaluating with the same hand-weighted score.","tokens_in":16929,"feed_emoji":"🔧","tokens_out":6090,"duration_ms":55780,"temperature":0.7,"pith_summary":"The paper is trying to establish that fine-grained, token-level preference data—not whole-trajectory preferences—is the right training signal for LLM tool use. It proposes a fully automated pipeline: reverse-generate tool-use scenarios, sample near-tie token alternatives during decoding, grade each sampled call with an error taxonomy, and fine-tune with direct preference optimization. If correct, this would let a 7B open model reach or exceed the tool-calling accuracy of much larger proprietary models after moderate fine-tuning, while using data that is cheap to produce and contains no leaked tool names. The framework's value is that it turns tool-calling quality into a structured, interpretable score that can drive both training and evaluation.","feed_headline":"Token-level training lifts 7B tool-calling past GPT-4o-mini","feed_subtitle":"Fine-grained preference pairs and an error taxonomy lift tool-name accuracy from 43.0% to 57.8%.","key_machinery":"The central mechanism is the error-oriented scoring function $F(t_{\\mathrm{call}})=\\sum_{i}\\omega_i\\delta_{e_i}(t_{\\mathrm{call}})$, a weighted sum over six tool-call error types: format errors, wrong tool name, missing required parameters, wrong parameter names, wrong parameter types, and wrong parameter values, with weights $3,3,1,2,2$. This score is used twice: to build preferred/dispreferred pairs during token-level preference sampling, and to grade the authors' custom test set. The other two components are reversed dataset construction, which generates scenario, tool calls, answer, and query in that order to avoid leakage and unsolvable queries, and token-level preference sampling, which perturbs a small number of near-tie top tokens during decoding to create candidate tool calls that the scoring function then ranks.","core_discovery":"The paper's central discovery, on its own terms, is that aligning a tool-calling LLM with token-level preferences—rather than whole-trajectory preferences—substantially reduces tool-call errors, and that the alignment signal can be produced automatically without human annotation. Training the 7B base model with TTPA yields 57.8% tool-name accuracy, 81.3% parameter-selection accuracy, and 74.2% parameter-content accuracy on the authors' held-out test set, versus 43.0% for the strongest baseline on tool-name accuracy. The same model matches or beats strong baselines on public benchmarks while keeping general reasoning scores stable, and the paper attributes the gains to the combination of reversed dataset construction, token-level preference sampling, and the error-oriented scoring mechanism.","pith_inferences":["The paper leaves open whether the error weights are optimal; a testable extension is to calibrate the six weights against human preference judgments, or to learn them from a held-out reward model.","Because the reversed data generator is itself a strong LLM, the pipeline's ceiling may be inherited from that generator; probing data diversity and generator artifacts would clarify how much the framework adds beyond distillation.","The same error-oriented scoring could be applied to other strictly structured outputs, such as SQL, schemas, and config files, where single-token errors are also catastrophic and trajectory-level rewards are too coarse.","The token-level near-tie sampling only explores tokens that are already close in probability, so it may miss rare but catastrophic errors; combining it with adversarial perturbation is a natural next test."],"forward_implications":["A 7B-parameter open model can match or beat frontier closed API models on tool selection, parameter filling, and result parsing after fine-tuning on 8,550 automatically built preference pairs.","Synthetic tool-use data can be generated without expensive filtering: reversing query and answer construction guarantees answerable queries and avoids leakage.","Token-level preference signals transfer across datasets and base models, so the pipeline could be reused for new tool sets without human preference annotations.","The six-type error taxonomy provides an interpretable evaluation metric for structured outputs beyond tool calls, such as JSON generation or API orchestration.","Gains on robustness subsets, where the model's pass rate fluctuates less than 2% compared with over 5% for baselines, suggest improved stability under query shortening, query lengthening, and tool-order shuffling."],"supporting_citations":[{"why":"Supplies the DPO objective that TTPA applies to token-level preference pairs.","marker":"Rafailov et al., 2023"},{"why":"Provides the ToolBench benchmark and the forward data-generation approach that reversed dataset construction inverts.","marker":"Qin et al., 2023b"},{"why":"ToolACE serves as a strong tool-learning baseline and a dialogue-based forward-generation dataset.","marker":"Liu et al., 2024a"},{"why":"APIGen/xLAM provides an automated pipeline baseline and a model compared in the experiments.","marker":"Liu et al., 2024b"},{"why":"Supplies the BFCL benchmark with independent AST-based evaluation for generalization tests.","marker":"Patil et al., 2024"},{"why":"Hammer2.0 acts as a state-of-the-art tool-learning baseline.","marker":"Lin et al., 2024"},{"why":"Represents the trajectory-level preference sampling method that token-level sampling argues against.","marker":"CHEN et al., 2024"},{"why":"Provides the GPT-4o family used both as the data generator and as the strongest closed API baseline.","marker":"OpenAI, 2023"}],"fun_headline_variants":["Token-level tool training lifts 7B accuracy by 15 points","Automatic error scoring aligns tool calls, beats GPT-4o-mini","7B model beats GPT-4o-mini with token-level preferences","Token-level alignment trims tool errors, outperforms GPT-4o-mini","No human labels needed: Token-level tool training boosts 7B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-built error categories and their fixed weights (3, 3, 1, 2, 2) capture what makes a tool call good or bad; if that scoring is incomplete or miscalibrated, both the DPO training signal and the in-house test-set metric are biased, so the reported gains would not transfer to real tool use.","fun_headline_variants_meta":{"raw":{"variants":["Token-level tool training lifts 7B accuracy by 15 points","Automatic error scoring aligns tool calls, beats GPT-4o-mini","7B model beats GPT-4o-mini with token-level preferences","Token-level alignment trims tool errors, outperforms GPT-4o-mini","No human labels needed: Token-level tool training boosts 7B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001375,"raw_usage":{"total_tokens":5538,"prompt_tokens":876,"completion_tokens":4662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":4565}},"tokens_in":492,"tokens_out":4662,"duration_ms":37086,"temperature":1.0,"reasoning_tokens":4565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:01:37.141362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by taking the model trained under TTPA and re-evaluating it on held-out real user queries using a scoring function that does not share TTPA's error taxonomy, such as exact-match parse success or human judgment of whether the returned call would execute correctly; if the large gap over baselines disappears under that independent metric, the reported gains are an artifact of training and evaluating with the same hand-weighted score.","supporting_citations":[],"review_version":1}