{"id":"49eee0fe-27e7-444e-af21-2bff8115dc4d","arxiv_id":"2505.04072","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning Qwen2.5-7B on PTool-synthesized data raises its PTBench overall score from 0.074 to 0.268, including on users not seen in training.","lead":"Personalized tool invocation asks a model to choose between functionally similar tools based on a user's profile, and to fill in missing arguments from that profile. The authors build the PTool data-synthesis pipeline and the PTBench benchmark, and then show that fine-tuning an open model on PTool data sharply improves such invocation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PTBench's split holds out users but not tools: train and test share the same 15 platforms and 360 APIs, so untrained-user 'generalization' (0.2660) and the win over GPT-4-turbo (0.2678 vs 0.1847) are only shown inside the closed PTool tool universe; unseen-tool generalization is never tested.","rationale":"I read the central claim as having two parts: (1) SFT on PTool data improves personalized tool invocation relative to the base model and to zero-shot API baselines on PTBench; and (2) this improvement generalizes beyond training users without harming general abilities. Part (1) is well supported within the benchmark: the effect is large (0.0738 to 0.2678 for overall accuracy), the test set is manually verified, and the trained-versus-untrained gap is small (0.2701 versus 0.2660), which argues against query-level leakage. Part (2) is where the design is weakest: the untrained split is out-of-distribution only along the user dimension, because the same 15 platforms and 360 APIs appear in both training and test. A model can memorize platform-level preference mappings from 7,096 training queries and apply them to novel profiles; the paper never tests generalization to unseen platforms or scenarios, and the Limitations section omits this point. Since tool descriptions are in-context at inference, the model could in principle learn a generalizable rule, but the design as reported does not distinguish that from tool-set memorization. I considered other candidates: missing error bars (secondary, given the large effect sizes), the small number of untrained users (relevant but not deciding), and the zero-shot nature of API baselines (a real fairness limitation, but one that only matters if the tuned model's advantage is expected to transfer beyond its fine-tuning distribution, which is exactly what the tool-holdout test probes). I therefore keep the reader's CONDITIONAL verdict rather than moving it, and make the condition concrete: either run the tool-holdout experiment or explicitly scope the generalization and cross-model claims to the closed synthetic universe. Credit where due: the manual verification of 1,083 test samples, the released benchmark, the user-level held-out split, the ablations on profile components, and the general-capability checks are all genuine evidence, and I am not questioning the within-benchmark numbers.","tokens_in":16055,"tokens_out":17117,"duration_ms":159819,"concrete_test":"Retrain the LoRA model with the released PTool pipeline, excluding one full scenario (or one platform per scenario) from the training set, and evaluate on the PTBench samples whose ground-truth platform is the held-out tool. Report platform accuracy and overall accuracy on that held-out subset for the retrained model, for raw Qwen2.5-7B-Instruct, and for zero-shot GPT-4-turbo. If platform accuracy on never-seen platforms collapses toward the raw or GPT-4-turbo level (0.3795 and 0.5484 in the full set, respectively), the fine-tuned advantage is tied to the memorized tool universe and the paper's claims must be scoped to the closed synthetic distribution; if the advantage persists, the learned profile-to-tool-attribute mapping generalizes along the tool dimension.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is not merely that PTBench is synthetic; it is that the fine-tuned model's advantage is measured inside a tool set the training already exposed. Table 1 shows the train and test splits share the same 5 scenarios, 15 platforms, and 360 APIs. The 'untrained users' split (6 users, 609 queries) holds out user profiles only; those users act on the exact platforms and APIs seen during SFT. In the Tool Preference task, the model can shortcut by memorizing name-level associations (for example, which platform is the budget one, which is the fast-delivery one) from 7,096 training queries and apply them to any new profile. The reported 0.7374 platform accuracy, 0.7341 profile-value accuracy, and 0.2660 untrained overall accuracy are all consistent with profile-to-known-platform memorization, rather than with a generalizable profile-to-tool-attribute rule. The cross-model comparison compounds this: GPT-4-turbo and the other API baselines are zero-shot and never saw this tool distribution, so 'a 7B model surpasses GPT-4-turbo' compares an in-distribution tuned model with out-of-distribution zero-shot models. The manuscript's Limitations section lists only limited scenario coverage and task coverage; it does not flag the shared tool universe between train and test, nor the absence of any tool/scenario hold-out or real-user validation. This is the sharpest testable form of the reader's benchmark-validity concern, and it makes the paper's unqualified generalization claim narrower than the evidence supports. If the trained model has instead learned a generalizable rule, the tool-holdout test below would show it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of Personalized Tool Invocation, decomposing it into Tool Preference (selecting among functionally similar tools based on user preferences) and Profile-dependent Query (inferring missing tool arguments from a user profile). The authors propose PTool, a three-stage LLM-based synthesis framework that generates tools via a hierarchical API tree, constructs user profiles through bottom-up feature clustering and top-down assignment, and produces query-solution pairs via a two-agent role-playing pipeline with rule-based and model-based verification. They then build PTBench, a 1,083-sample, manually verified benchmark drawn from the same PTool distribution, and fine-tune Qwen2.5-7B with LoRA on 7,096 PTool queries. The fine-tuned model is reported to reach 0.2678 overall accuracy on PTBench, outperforming GPT-4-turbo (0.1847) and other API/open-source baselines, including 0.2660 on held-out untrained users, while also improving BFCL non-live scores without degrading MMLU, HumanEval, GSM8K, or CSQA.","tokens_in":16435,"tokens_out":6179,"duration_ms":59738,"significance":"If the results hold, this is a useful first benchmark and data-synthesis framework for personalized tool invocation, a genuinely underexplored aspect of tool use that combines user profiling with function calling. The public release of the benchmark, the manual verification of the test set, and the external BFCL gain are concrete strengths that support the practical value of the contribution. However, the main quantitative claims are measured inside a closed synthetic distribution that the training data and test data share, so the headline numbers should be interpreted as in-distribution improvements rather than demonstrated real-world generalization. The benchmark itself is likely to be a useful resource for future work, and the paper's task formulation is a reasonable starting point, but the evaluation design currently overstates the strength of the generalization evidence.","major_comments":[{"comment":"The train/test split holds out users but not tools or scenarios: both the training set and PTBench share the same 5 scenarios, 15 platforms, and 360 APIs, and the 'untrained users' split (6 users, 609 queries) only means those user profiles were absent from SFT. The platforms and APIs in those queries are exactly the ones seen during training. The reported platform accuracy of 0.7374 and untrained-user overall accuracy of 0.2660 are therefore consistent with the model memorizing name-level platform/attribute associations from 7,096 training queries and applying them to new profiles, rather than with a generalizable profile-to-tool-attribute rule. This makes the cross-user generalization claim in Finding 3 and the headline win over GPT-4-turbo (0.2678 vs 0.1847) an in-tool-universe result. The authors should add a held-out platform or scenario split, or an API-level hold-out, to support the generalization claim, or explicitly restrict the conclusions to the closed tool universe. The Limitations section does not currently mention this shared-universe issue.","section":"§5.1, Table 1; Findings 3 and 4"},{"comment":"Equations (2)-(11) define accuracy rates but leave the notion of a 'correct sample' underspecified. It is not stated whether parameter names and values are compared by exact string matching, by semantic equivalence, or by partial credit; how the multiple parameters of a single invocation are aggregated into a sample-level correctness score; or how platform accuracy is treated when the correct platform is chosen together with the wrong API. In addition, all results are from single runs with no error bars, so differences of a few points in Tables 2 and 3 cannot be distinguished from noise. An operational definition of the evaluation script is needed for the benchmark numbers to be reproducible and meaningful.","section":"Appendix A.1; Tables 2-5"},{"comment":"The headline comparison against API models is asymmetric. The fine-tuned Qwen2.5-7B model has been trained on PTool queries and on the exact platforms and APIs used in PTBench, while GPT-4-turbo, GPT-4o, DeepSeek, Claude, and the open-source baselines are evaluated zero-shot on a synthetic tool distribution they have never seen. The statement in Finding 2 that most models 'fail to select suitable tools' and the overall accuracy comparison in Table 2 therefore compare an in-distribution tuned model with out-of-distribution zero-shot models. To make the cross-model comparison informative, the authors should add a few-shot or in-context baseline that provides API models with a small set of PTool examples, or explicitly caveat that the advantage reflects in-distribution training rather than general superiority in personalized tool invocation.","section":"§5.2, Finding 2; Table 2"},{"comment":"The limitations paragraph mentions only limited scenario coverage and limited task coverage. It does not acknowledge that the evaluation is carried out on a test set synthesized by the same pipeline that produced the training data, that no real user data or behavioral logs are used, and that no tool/scenario generalization test exists. Since the abstract claims that training on PTool improves 'personalized tool invocation capabilities' in general, the manuscript should either add an external validation signal, such as a semi-realistic or real-user evaluation, or explicitly state that the demonstrated gains are confined to the PTool distribution. The BFCL improvement in §5.5 is a useful external signal but does not measure personalization, so it cannot by itself establish real-world validity for the personalization claim.","section":"Limitations"}],"minor_comments":[{"comment":"The 'All w/o History' bullet says 'The behavioral history is given,' but the variant name indicates that behavioral history is omitted; the text should be corrected to say 'omitted.'","section":"§5.3"},{"comment":"The definition of untrained overall accuracy says it 'indicate the personalized tool selection ability on trained users'; 'trained' should be replaced with 'untrained.'","section":"Appendix A.1, Eq. (10)"},{"comment":"'CommonSenceQA' is a typo and should be 'CommonsenseQA.'","section":"§5.5"},{"comment":"The 'Trained' and 'Untrained' overall columns are not defined in the captions; a note should indicate that these refer to the user split described in §5.1.","section":"Tables 2, 4, 5"},{"comment":"The vertical axis of the bar charts is not labeled, so the reader cannot determine the scale or the metric being reported.","section":"Figure 3"},{"comment":"The user agent is described as using basic features and implicit preferences, but the evaluation prompt in Figure 7 exposes only basic_features and user_history; please clarify whether implicit preferences are visible to the solver and whether this matches the actual evaluation setup.","section":"§4.3 and Figure 7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper defines personalized tool invocation as a task, builds a synthesis pipeline (PTool), and releases a manually verified benchmark (PTBench). The task definition is genuinely new in the tool-invocation literature, and the benchmark fills a real gap. The authors also do useful extras: ablations separating basic features from behavioral history, a scaling study, and a check that fine-tuning doesn't degrade general abilities. That is more than most benchmark papers do, and the public release is a plus.\n\nThe soft spot is the evaluation design. PTool generates both the training data and the test benchmark, and your stress-test note is right that the train and test sets share the same 5 scenarios, 15 platforms, and 360 APIs (Table 1). So the held-out-user split tests generalization to new profiles only; it does not test whether the learned platform/preference mapping transfers to unseen tools. The big win over GPT-4-turbo (0.2678 vs 0.1847) compares a fine-tuned in-distribution model against zero-shot API models that never saw this synthetic tool distribution. That is a real confound, and the paper's Limitations section does not flag it. The absence of error bars or repeated runs is a smaller concern, but the margins are large enough that the main effect is probably not noise.\n\nNone of this kills the paper. The central result—that SFT on PTool data improves PTBench performance without hurting MMLU/GSM8K/BFCL—is credible as far as it goes. The problem is the claim is stated too broadly. If the authors reframe PTBench as a synthetic sandbox for method development, add a holdout split that excludes some tools or scenarios from training, and report variance, the contribution would be solid. As is, it's a promising but self-referential benchmark.\n\nI'd send it to peer review; a good referee can push for exactly these fixes. I would cite it for the task formulation, and it's worth a reading-group discussion on benchmark validity.","headline":"Useful new task formulation and a real benchmark, but the train/test circularity makes the headline generalization claim narrower than stated; still deserves peer review.","tokens_in":17005,"tokens_out":4080,"would_cite":true,"duration_ms":42016,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a 7B open model on synthesized personalized tool-invocation data makes it surpass GPT-4-turbo on the new PTBench benchmark without hurting general abilities.","keywords":["personalized tool invocation","tool preference","profile-dependent query","PTool","PTBench","data synthesis","supervised fine-tuning","tool calling"],"falsifier":"Run the fine-tuned model on real logged tool-invocation requests with ground-truth platform preferences and profile-sourced parameter values; if its accuracy is no higher than a generic instruction-tuned baseline, the claimed transfer to real users fails.","tokens_in":15852,"feed_emoji":"🧰","tokens_out":9598,"duration_ms":81709,"temperature":0.7,"pith_summary":"This paper introduces the concept of personalized tool invocation, which requires an LLM to choose among functionally similar tools according to a user's preferences and to fill query-omitted tool parameters from the user's profile. To study it, the authors build PTool, a three-stage synthetic-data pipeline, and PTBench, a manually verified benchmark of 1,083 samples across 80 users and five everyday scenarios. The central claim is that supervised fine-tuning a 7B open-source model on PTool data raises PTBench overall accuracy from 0.0738 to 0.2678, above GPT-4-turbo's 0.1847, and that the gain carries to users not seen in training. The authors also report no degradation on standard general and function-calling benchmarks. The reason to care is that, if true, personalized tool use could be trained from synthetic profiles rather than collected from real users.","feed_headline":"7B model beats GPT-4-turbo on personalized tool calls","feed_subtitle":"Profile-aware tool selection and parameter filling improve after fine-tuning, with no general-capability loss.","key_machinery":"PTool, the carried mechanism, is a three-stage data synthesis framework: tool generation uses depth-first expansion over an API tree, with several platforms per scenario to create functionally interchangeable tools; user profile construction clusters platform parameters into a feature tree and assigns values top-down, producing basic features, implicit preferences, and simulated behavioral history; query and solution generation uses a user agent and an assistant agent, followed by rule-based and LLM-based verification, to create profile-grounded queries and tool calls. PTBench is its manually checked test subset, with separate trained and untrained user splits. This machinery lets the authors compare fine-tuned and API models on the same personalized distribution and attribute the gains to the synthetic data.","core_discovery":"The paper's central discovery is that a small open-source model can be taught personalized tool invocation by synthetic data alone. After parameter-efficient fine-tuning on PTool-generated data, Qwen2.5-7B-Instruct reaches 0.2678 overall accuracy on PTBench, compared with 0.1847 for GPT-4-turbo; on platform selection, the fine-tuned model scores 0.7374 versus 0.5484, and on profile-sourced parameter values 0.7341 versus 0.6832. On six held-out users the overall score is 0.2660, nearly matching the trained-user score of 0.2701, which the authors take as evidence of generalization rather than memorization. They further find that removing behavioral history from the user profile degrades tool-preference accuracy more than removing basic features degrades tool-invocation accuracy, and that the trained model's scores on MMLU, HumanEval, GSM8K, CommonsenseQA, and BFCL non-live do not fall relative to the base model.","pith_inferences":["If PTBench's distribution transfers, the same recipe could be used in deployed assistants: a 7B-class model fine-tuned on synthetic profiles can deliver personalized tool selection and parameter filling without collecting real user data.","The strongest untested risk is distribution shift: the benchmark fixes platform catalogues and profile schemas, so a direct follow-up is to add new platforms or new profile fields after training and measure whether the gains survive.","The ablation pattern suggests a testable extension: training on profiles with carefully curated behavioral histories may matter more than adding more basic features, since removing history hurts platform accuracy most."],"forward_implications":["A 7B open-source model fine-tuned with PTool data reaches 0.2678 overall PTBench accuracy, above GPT-4-turbo's 0.1847, with platform accuracy 0.7374 versus 0.5484.","The gains generalize to six unseen users: 0.2660 overall accuracy on untrained users, close to the trained-user performance of 0.2701.","Training does not degrade general abilities: MMLU, HumanEval, GSM8K, CommonsenseQA, and BFCL non-live scores are maintained or improved relative to the base Qwen2.5-7B-Instruct model.","Profile-dependent parameter filling is harder than query-dependent filling for all models, but the PTool-trained model improves profile-value accuracy to 0.7341, above GPT-4-turbo's 0.6832.","Removing behavioral history from the user profile disproportionately hurts platform accuracy, while removing basic features hurts tool-invocation accuracy."],"supporting_citations":[{"why":"Supplies the API-tree synthesis approach PTool extends and is used as the ToolACE baseline.","marker":"Liu et al., 2025"},{"why":"Defines ToolLLM, a reference tool-invocation method and one of the comparison baselines.","marker":"Qin et al., 2024"},{"why":"Introduces Hammer2.1-7b, a function-calling baseline whose results PTBench reports.","marker":"Lin et al., 2024"},{"why":"Introduces the xLAM action-model family used as a fine-tuned baseline.","marker":"Zhang et al., 2024b"},{"why":"Supplies Qwen2.5, the base model for fine-tuning and the scaling study.","marker":"Team, 2024a,b"},{"why":"Defines BFCL non-live, used to show basic function calling is not harmed.","marker":"Yan et al., 2024"},{"why":"Defines MMLU, used to show general knowledge is preserved.","marker":"Hendrycks et al., 2021b"},{"why":"Defines GSM8K, used to show math reasoning is preserved.","marker":"Cobbe et al., 2021"},{"why":"Defines HumanEval, used to show coding ability is preserved.","marker":"Chen et al., 2021"},{"why":"Defines CommonsenseQA, used to show reasoning ability is preserved.","marker":"Talmor et al., 2019"}],"fun_headline_variants":["Fine-tuned 7B model beats GPT-4-turbo on personalized tool calls","Synthetic data teaches 7B model personalized tool use, beats GPT-4","Profile-aware 7B model outperforms GPT-4-turbo on tools","Tiny model beats GPT-4, generalizes to unseen users for tools","Synthetic data alone yields 7B model that beats GPT-4-turbo on tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's synthetic user profiles, platform sets, and missing-parameter patterns are assumed to stand in for real-world personalized tool use.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned 7B model beats GPT-4-turbo on personalized tool calls","Synthetic data teaches 7B model personalized tool use, beats GPT-4","Profile-aware 7B model outperforms GPT-4-turbo on tools","Tiny model beats GPT-4, generalizes to unseen users for tools","Synthetic data alone yields 7B model that beats GPT-4-turbo on tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00089,"raw_usage":{"total_tokens":3838,"prompt_tokens":944,"completion_tokens":2894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2794}},"tokens_in":560,"tokens_out":2894,"duration_ms":20818,"temperature":1.0,"reasoning_tokens":2794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:38:17.687951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fine-tuned model on real logged tool-invocation requests with ground-truth platform preferences and profile-sourced parameter values; if its accuracy is no higher than a generic instruction-tuned baseline, the claimed transfer to real users fails.","supporting_citations":[],"review_version":1}