{"id":"e6f135ba-1512-433b-942d-69e6e1c6b60a","arxiv_id":"2505.13176","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new evaluation benchmark shows that leading LLMs perform poorly at tool selection when they must jointly reason about user profiles and environmental factors.","lead":"A new benchmark, ToolSpectrum, tests whether large language models can choose the right app and settings based on a user's profile and current environment, rather than just following the instruction literally. It finds that even top models score low on this task, especially when they must weigh user details and environmental context together.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Joint-condition F1 gap may be an artifact of GPT-4o-generated gold labels; the benchmark's oracle and evaluator are the same model.","rationale":"The paper introduces a novel and practically motivated benchmark, evaluates 18 models, and includes a human-validated preference analysis (Kappa 0.85); these are genuine strengths. However, the load-bearing assumption for the headline claim about joint reasoning is that the gold tool calls in ToolSpectrum are correct, unbiased, and uniquely determined. The reader's verdict already flags this, and I agree. The risk is not merely academic: because GPT-4o both creates the gold labels and is the best-performing model, the benchmark cannot independently establish that tested models 'prioritize one dimension at the expense of the other.' GPT-4o's own errors in the joint condition (31% policy-compliance failures in the sampled error analysis, Section 5.3) suggest that the reference standard may itself embody a one-at-a-time reasoning bias. A human-annotation study with inter-annotator agreement metrics would settle this by measuring whether the gold labels in the Both subset are actually less consensual than those in the single-factor subsets, and whether the gold outputs systematically favor one factor. If the proposed check shows high agreement and a balanced factor distribution, the concern would be resolved and the paper's claims would stand; if not, the paper would need human-annotated gold labels for the joint subset or a matched human-preference evaluation before the central claim can be accepted. Since the reader already made the verdict conditional on exactly this assumption, the verdict should remain UNCHANGED; the concrete test would determine whether it should later be upgraded to ACCEPT or downgraded to REJECT.","tokens_in":20264,"tokens_out":6513,"duration_ms":59933,"concrete_test":"Draw a random sample of 100 items from Profile & Environment and 50 from each of Profile-only and Environment-only. For each item, two expert annotators (blinded to provenance) rank the GPT-4o gold output against 2-3 plausible alternatives (e.g., an output using only profile cues, one using only environment cues, and one random tool call) and state whether a single best answer exists. Compute Fleiss' kappa and the percentage of items where the human majority chooses the gold output. If kappa on the Both subset is substantially lower than on the single-factor subsets, or if the human majority rejects the gold output on more than 20% of Both items, the F1 gap is confounded by label ambiguity. As a secondary check, count how often the gold output's parameter selection is determined by profile vs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ToolSpectrum's central empirical claim—that LLMs cannot jointly reason about profiles and environments—is measured as F1 against gold tool calls that GPT-4o generated (Section 3.2.4) and then filtered by GPT-4o quality scores (Section 3.2.5). The same model family is therefore both the oracle that defines 'correct' personalized tool calls and the strongest performer in the benchmark. This is particularly concerning for the Profile & Environment subset: GPT-4o's own error analysis (Section 5.3) shows that in 31% of sampled errors it fails to interpret domain policies in precisely this joint condition. If GPT-4o's joint reasoning is imperfect, its gold labels in the 'Both' subset are likely noisy or systematically biased toward whichever factor GPT-4o happened to attend to. Consequently, the low OP F1 (~0.45-0.50) for the best models in the Both condition may reflect label ambiguity or a reference standard that favors one dimension, rather than a genuine deficiency of the tested models. The paper does not report inter-annotator agreement on the gold labels themselves, and the manual quality check in Section 3.2.5 scores plausibility on a 1-10 scale, not whether the gold output is uniquely reasonable or unbiased across profile/environment dimensions. Without such validation, the headline conclusion is conditional on GPT-4o being a reliable oracle for joint reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ToolSpectrum, a benchmark for evaluating LLMs on personalized tool utilization, defined as selecting apps, APIs, required parameters, and optional parameters while taking into account user profiles, environmental factors, and their combination. The benchmark is constructed via a GPT-4o-driven pipeline: GPT-4o generates user instructions, tool-call results (under Profile, Environment, and Profile & Environment conditions), and then filters the data using its own quality scores with a threshold of 8, removing 21.8% of samples. The authors evaluate 16 open-source and API-based models, reporting F1 scores at four granularity levels (APP, API, RP, OP). They find that models perform relatively well on coarse-grained tool selection but poorly on optional parameters, and that the combined Profile & Environment condition is the most difficult, with the best models scoring only about 0.45-0.50 on OP. A secondary win-rate analysis (with GPT-4o as judge) is used to argue that personalization improves tool-utilization effectiveness. The paper also includes an error analysis and prompt-engineering experiments.","tokens_in":20569,"tokens_out":8445,"duration_ms":76776,"significance":"ToolSpectrum addresses a genuine gap in the tool-learning literature: existing benchmarks mostly ignore overlapping toolsets and the influence of user profiles and environmental context on tool choice. The decomposition of personalization into profile, environment, and their joint effect, together with the four-level F1 hierarchy, is sensible and potentially useful for the community. The paper provides broad model coverage, and the qualitative pattern that joint profile/environment reasoning is harder than either dimension alone is consistent with intuition and with the reported tables. If the gold-label validity is established, this could become a useful resource for studying personalized tool agents. However, the current construction relies heavily on GPT-4o both as the oracle that defines correct tool calls and as the judge for the win-rate analysis, which undermines the absolute scores and, to some extent, the cross-condition comparisons. The release of data and code is a clear strength that aids further validation.","major_comments":[{"comment":"Gold tool-call results are generated by GPT-4o using the prompt template in Figure 13 and then filtered by GPT-4o's own 1-10 quality scores with a threshold of 8, removing 21.8% of samples. GPT-4o is also the strongest evaluated model in Table 3. Because the oracle that defines 'correct' personalized calls is the same model family being measured, the reported F1 scores, including the low OP scores (~0.45-0.50) in the Both condition, reflect consistency with GPT-4o's generation policy rather than an independent standard. The manual quality assessment at the end of §3.2.5 scores plausibility on a 1-10 scale, and the paper reports no inter-annotator agreement on whether the gold output is uniquely reasonable or unbiased across the profile/environment dimensions. This matters concretely: §5.3 shows GPT-4o itself misinterprets domain policies in 31% of its sampled errors in the combined condition, so the gold labels in that subset are likely noisy exactly where the headline conclusion is drawn. I ask the authors to (i) report inter-annotator agreement on a sample of gold labels with instructions to judge not merely plausibility but also whether an alternative tool call would be equally valid; (ii) analyze dimension bias in the gold labels (e.g., how often the gold call follows profile vs environment when the two conflict); and (iii) re-evaluate at least the Both subset with gold labels generated by a different model or with human-authored labels to show the main pattern is not an artifact of GPT-4o self-consistency.","section":"§3.2.4, §3.2.5, Table 3"},{"comment":"The average number of optional parameters in the gold outputs differs greatly across conditions: 1.31 in Profile, 6.79 in Environment, and 6.67 in Both. Since OP F1 is computed over per-sample gold optional-parameter dictionaries, a model that perfectly predicts the single optional parameter in a Profile sample receives F1=1.0 for that sample, whereas in an Environment sample it must correctly predict about 6.8 parameters to achieve the same per-sample F1. The paper's cross-condition ranking (Profile ≈ Environment > Both) is therefore confounded by the number of gold OPs per condition; the Both condition not only combines two personalized factors but also has many more OPs to predict than Profile. Please report OP F1 normalized by the number of gold OPs per sample (or per-parameter precision/recall) and decompose the OP gap into a label-length effect and a joint-reasoning effect. Without this, the central claim that joint reasoning is the bottleneck for OP scores is not fully supported.","section":"Table 2 and §4.2"},{"comment":"The win-rate analysis in Figure 3 uses GPT-4o as the judge (prompt in Figure 15), with human-GPT-4o agreement measured on 100 samples (κ=0.85). A single aggregate κ of 0.85 on 100 samples is not sufficient to rule out systematic judge bias in favor of the personalized outputs, particularly because those outputs follow the same generation template used to create the gold labels. The conclusion that 'personalization improves effectiveness' would be strengthened by reporting win rates with an alternate judge (e.g., Claude or a different strong model) and by showing per-condition human agreement rather than one pooled number. This is a load-bearing check on the RQ1 claim, since the same model that served as oracle and primary evaluated model also serves as the sole automated judge.","section":"§5.1, Figure 15"}],"minor_comments":[{"comment":"The column header 'PP' under the Profile condition is a typo for 'OP'; the text and all other columns use OP.","section":"Table 3"},{"comment":"The text states that ToolSpectrum features '23 APPs, 42 APIs,' while Table 2 reports #APPs = 22/22/20 and #APIs = 39/40/33 across the three subsets; please reconcile these numbers.","section":"§3.3"},{"comment":"The sentence 'In this section (5.5), we explore...' references a nonexistent Section 5.5; it should refer to Section B.3.","section":"§B.3"},{"comment":"Requirement 4 says 'if you feel that the user's instruction violates the user's profile, set the APP, API, and params to null,' but the task definition in §3.1 states that None should be returned when the instruction violates the application's policy, not the user's profile; please correct the wording.","section":"Figure 13"},{"comment":"The caption of Figure 4 does not specify which models are averaged to obtain the per-domain performance; please state the model set or provide per-model results to make the domain analysis reproducible.","section":"Figure 4"},{"comment":"The rows 'Avg. Params' and 'Avg. OPs' are not formally defined; please clarify whether these are averages over samples and over which fields, so that readers can interpret the large OP-count differences.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile gap and the resource could become useful, but the GPT-4o-as-oracle-and-judge issue is serious enough that the claims cannot be accepted as stated. I would not reject: the authors can likely address the concern by adding human validation of gold labels, cross-model generation checks, and a confound analysis for the OP metric. The benchmark's scale is modest but adequate for a new task; the main risk is that the headline conclusion about joint reasoning is currently conditional on GPT-4o's own biases. I recommend the editor ask for the additional validation experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper gives the tool-augmented LLM community a genuinely new benchmark, and its main empirical pattern is probably real. It is the first to combine user profiles and environmental factors in a tool-selection task, and the formalization into APP/API/RP/OP is a clean way to separate where models fail. The finding that all models -- open and closed -- collapse on optional parameters when both factors are present, dropping to around 0.45-0.50 OP F1, is consistent across a wide model sweep and is the kind of result that should make people in the subfield pay attention.\n\nWhat it does well: the data construction pipeline is careful in the right places. The toolset covers 9 domains with overlapping apps and APIs, seed data is used to keep generation controllable, and there is manual review plus a quality filter. The error analysis is also a plus; it breaks down where GPT-4o itself fails, including the 31% category where it cannot interpret domain policies under joint conditions. The prompting experiments, especially hierarchical prompting, are a sensible start on mitigation.\n\nThe soft spots are real but not disqualifying. The biggest one is exactly what the stress-test note flags: the gold tool calls are generated by GPT-4o (Section 3.2.4), filtered by GPT-4o quality scores (Section 3.2.5), and then GPT-4o is evaluated against those same labels. For the Profile & Environment subset, where GPT-4o's own error rate on policy interpretation is high, the labels in the 'Both' condition may be noisy or biased toward whichever dimension GPT-4o happened to attend to. The manual check is 50 samples per domain scored for plausibility, not for whether the gold answer is uniquely reasonable. That is a genuine gap. The Kappa 0.85 on the win-rate comparison is decent, but that is still GPT-4o judging its own outputs against non-personalized ones, so it does not fully address the circularity. Also, the dataset is small -- 450, 220, and 330 samples -- and the repo URL differs between the abstract and the full text, which is annoying for reproducibility.\n\nI want to be fair: the central empirical pattern holds up better than the stress-test note suggests. The fact that non-GPT models show the same qualitative drop in the joint condition means it is not purely an artifact of GPT-4o rewarding itself. But because the benchmark's validity depends on the gold labels being genuinely correct personalized choices, the circularity has to be addressed with human-validated labels or alternative-reference analysis before the headline numbers can be taken at face value.\n\nFor whom: researchers working on tool-augmented LLMs or LLM personalization will want to read this. It deserves a serious referee -- the task definition and the failure pattern are worth engaging with. I would send it to peer review with a request for label validation and a fix to the reproducibility details, but I would not desk-reject it.","headline":"The first benchmark for personalized tool selection under joint profile-plus-environment constraints, with a real empirical finding -- but the gold labels and the judge are the same model family being tested, so the headline numbers are conditional on GPT-4o's reliability as an oracle.","tokens_in":21102,"tokens_out":1912,"would_cite":true,"duration_ms":22014,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that personalized tool utilization—choosing among functionally similar tools using user profile and environment—is a distinct capability current LLMs lack, and introduces ToolSpectrum to measure it.","keywords":["personalized tool utilization","tool learning","LLM benchmark","user profile","environmental context","domain policy","tool selection","context-aware personalization"],"falsifier":"Take a random sample of ToolSpectrum items and have independent human experts choose the tool call they would make for the stated profile and environment; if expert choices systematically disagree with the GPT-4o-generated ground-truth labels (for instance, Cohen's kappa well below 0.8), the benchmark measures agreement with GPT-4o rather than correctness of personalization. The complementary check is a live user study in which people in the profiled situations choose between the ground-truth label and a model's output, testing whether the labels actually correspond to higher satisfaction.","tokens_in":20083,"feed_emoji":"🧰","tokens_out":9040,"duration_ms":82336,"temperature":0.7,"pith_summary":"The paper claims that tool-augmented large language models need a capability existing benchmarks do not test: choosing among functionally similar tools based on who the user is and what situation they are in. It introduces ToolSpectrum, a 1,000-sample benchmark spanning nine domains, with user profiles, environmental contexts, and overlapping apps and APIs, and formalizes personalized tool utilization as a structured tool-call output. Experiments show that adding personalization improves tool-use effectiveness over non-personalized baselines, but that even strong models like GPT-4o and DeepSeek-R1 drop sharply when user profile and environment must be weighed together, often prioritizing one dimension at the expense of the other. If the paper is right, it identifies a concrete bottleneck for deploying LLM agents in user-facing services.","feed_headline":"Tool choice with personal context stumps even top LLMs","feed_subtitle":"New benchmark ToolSpectrum shows models score near 0.30 on app choice when profile and environment must be weighed together.","key_machinery":"The central object is the ToolSpectrum benchmark itself. It operationalizes personalization by splitting it into user profile (demographics, personality, preference) and environment (natural environment, digital environment, and app domain policy), then builds three data splits—Profile-only, Environment-only, and Profile & Environment—so the individual and joint contributions of the two factors can be measured separately. The measuring instrument is an F1 score computed at four hierarchical levels: APP and API capture whether the right app and service interface were chosen, RP (required parameters) captures instruction following, and OP (optional personalized parameters) captures whether personalization actually changed the call. This decomposition is what lets the paper attribute performance drops to the joint-reasoning problem rather than to tool-call syntax or format errors.","core_discovery":"ToolSpectrum defines personalized tool utilization as a task: given a user instruction $I$, a user profile $P$ (demographics, personality, preferences), an environment $E$ (weather, time, network, and app domain policies), and a toolset $T$ of functionally overlapping apps and APIs, the model must return a dictionary $\\{\\mathrm{APP}, \\mathrm{API}, \\mathrm{RP}, \\mathrm{OP}\\}$, or the value $\\mathrm{None}$ when the combination violates a domain policy. The paper argues that existing benchmarks only test whether a model can pick a tool that executes an instruction, not whether it picks the right tool for this user in this situation. On 1,000 samples across nine domains, GPT-4o reaches 0.80–0.81 F1 on app selection when only one personalization factor is present, but falls to 0.32 when profile and environment must be considered together, with optional personalized parameters dropping to 0.45; the paper reads this as evidence that current LLMs cannot jointly reason about both dimensions and often prioritize one at the expense of the other. It also reports that adding personalized information improves tool-use effectiveness over non-personalized baselines, with the gain larger for GPT-4o than for GPT-3.5.","pith_inferences":["The measured joint-reasoning deficit may partly reflect GPT-4o's own label-generation biases; a benchmark generated by a different model or by human experts could produce different rankings and should be tested.","A natural extension is multi-turn personalization, where the profile and environment update after each tool call (for example, a canceled flight changes the travel context); ToolSpectrum currently evaluates one-shot decisions.","The profile/environment decomposition could be used to diagnose model-specific biases, since some models may systematically favor profile over environment or vice versa, which is testable with the existing data splits.","Because the authors note that excessive context length is a major limitation of the benchmark, retrieval or hierarchical toolset pruning may matter more than raw reasoning ability for this task; this is testable with the released code."],"forward_implications":["Tool-selection benchmarks should include overlapping toolsets with contextual factors, because functional correctness alone overstates real-world agent competence.","Simply scaling model size will not close the personalization gap: Qwen2.5-72B shows little improvement over Qwen2.5-32B on ToolSpectrum.","Hierarchical prompting—first predicting the domain, then giving only the relevant toolset—improves F1 by reducing context noise, giving a cheap baseline for future work.","Domain-policy compliance is a major failure mode in combined profile-and-environment scenarios (31% of sampled errors), so personalized tool use and constraint handling are linked.","The effectiveness gain from personalization grows with model strength, so stronger base models will likely amplify the benefit of personalized tool-use training."],"supporting_citations":[{"why":"Supplies the AppBench multi-API planning framework and the hierarchical F1 evaluation that ToolSpectrum adapts to measure personalized tool calls.","marker":"Wang et al., 2024b"},{"why":"Provides τ-bench's domain-policy and user-environment interaction design, which ToolSpectrum extends with richer profiles and overlapping toolsets.","marker":"Yao et al., 2024"},{"why":"LaMP is the personalization benchmark ToolSpectrum contrasts with: it covers user profiles but no tools or environment.","marker":"Salemi et al., 2024"},{"why":"PersonaChat supplies the user-profile representation that ToolSpectrum's profile definition builds on.","marker":"Zhang et al., 2018"},{"why":"ToolBench represents the existing tool-call benchmarks that select tools by instruction alone, the gap ToolSpectrum targets.","marker":"Xu et al., 2023"},{"why":"The tool-learning survey frames the broader research context and motivates the benchmark's focus on tool-augmented LLMs.","marker":"Qin et al., 2024"}],"fun_headline_variants":["Personalized tool pick: LLMs fail when user and context collide","When profile meets environment, LLMs pick wrong app","ToolSpectrum: personalizing tools needs joint reasoning, LLMs fail","GPT-4o drops to 0.32 when both user and context matter","New benchmark shows LLMs can't weigh user profile and environment jointly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-generated tool-call labels, after GPT-4o quality filtering and a small manual check, are genuinely correct personalized choices; if GPT-4o's judgments are biased or misaligned with real user satisfaction, the reported F1 scores and win rates do not measure actual personalized tool utilization.","fun_headline_variants_meta":{"raw":{"variants":["Personalized tool pick: LLMs fail when user and context collide","When profile meets environment, LLMs pick wrong app","ToolSpectrum: personalizing tools needs joint reasoning, LLMs fail","GPT-4o drops to 0.32 when both user and context matter","New benchmark shows LLMs can't weigh user profile and environment jointly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000434,"raw_usage":{"total_tokens":2243,"prompt_tokens":1009,"completion_tokens":1234,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":1142}},"tokens_in":625,"tokens_out":1234,"duration_ms":8584,"temperature":1.0,"reasoning_tokens":1142,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:18:02.265035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ToolSpectrum items and have independent human experts choose the tool call they would make for the stated profile and environment; if expert choices systematically disagree with the GPT-4o-generated ground-truth labels (for instance, Cohen's kappa well below 0.8), the benchmark measures agreement with GPT-4o rather than correctness of personalization. The complementary check is a live user study in which people in the profiled situations choose between the ground-truth label and a model's output, testing whether the labels actually correspond to higher satisfaction.","supporting_citations":[],"review_version":1}