ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models

HaiFeng Wang; Hongru Wang; Yuanfang Guo; Yuhang Guo; Yunhong Wang; Zeming Liu; Zihao Cheng

REVIEW 3 major objections 6 minor 1 cited by

ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that personalized tool utilization—choosing among functionally similar tools using user profile and environment—is a distinct capability current LLMs lack, and introduces ToolSpectrum to measure it.

desk verdict The first benchmark for personalized tool selection under joint profile-plus-environment constraints, with a real empirical finding -- but the gold labels and the judge are the same model family being tested, so the headline numbers are conditional on GPT-4o's reliability as an oracle. read the letter →

arxiv 2505.13176 v2 pith:4HKL7LFE submitted 2025-05-19 cs.CL cs.AI

Zihao Cheng , Hongru Wang , Zeming Liu , Yuhang Guo , Yuanfang Guo , Yunhong Wang , Haifeng Wang This is my paper · ORCID

classification cs.CLcs.AI

keywords personalizedtoolutilizationlearningLLMbenchmarkuserprofileenvironmentalcontextdomainpolicyselectioncontext-awarepersonalization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that tool-augmented large language models need a capability existing benchmarks do not test: choosing among functionally similar tools based on who the user is and what situation they are in. It introduces ToolSpectrum, a 1,000-sample benchmark spanning nine domains, with user profiles, environmental contexts, and overlapping apps and APIs, and formalizes personalized tool utilization as a structured tool-call output. Experiments show that adding personalization improves tool-use effectiveness over non-personalized baselines, but that even strong models like GPT-4o and DeepSeek-R1 drop sharply when user profile and environment must be weighed together, often prioritizing one dimension at the expense of the other. If the paper is right, it identifies a concrete bottleneck for deploying LLM agents in user-facing services.

What carries the argument

The central object is the ToolSpectrum benchmark itself. It operationalizes personalization by splitting it into user profile (demographics, personality, preference) and environment (natural environment, digital environment, and app domain policy), then builds three data splits—Profile-only, Environment-only, and Profile & Environment—so the individual and joint contributions of the two factors can be measured separately. The measuring instrument is an F1 score computed at four hierarchical levels: APP and API capture whether the right app and service interface were chosen, RP (required parameters) captures instruction following, and OP (optional personalized parameters) captures whether personalization actually changed the call. This decomposition is what lets the paper attribute performance drops to the joint-reasoning problem rather than to tool-call syntax or format errors.

What would settle it

Take a random sample of ToolSpectrum items and have independent human experts choose the tool call they would make for the stated profile and environment; if expert choices systematically disagree with the GPT-4o-generated ground-truth labels (for instance, Cohen's kappa well below 0.8), the benchmark measures agreement with GPT-4o rather than correctness of personalization. The complementary check is a live user study in which people in the profiled situations choose between the ground-truth label and a model's output, testing whether the labels actually correspond to higher satisfaction.

Watch

Extended reading notes

Core claim

ToolSpectrum defines personalized tool utilization as a task: given a user instruction $I$, a user profile $P$ (demographics, personality, preferences), an environment $E$ (weather, time, network, and app domain policies), and a toolset $T$ of functionally overlapping apps and APIs, the model must return a dictionary $\{\mathrm{APP}, \mathrm{API}, \mathrm{RP}, \mathrm{OP}\}$, or the value $\mathrm{None}$ when the combination violates a domain policy. The paper argues that existing benchmarks only test whether a model can pick a tool that executes an instruction, not whether it picks the right tool for this user in this situation. On 1,000 samples across nine domains, GPT-4o reaches 0.80–0.81 F1 on app selection when only one personalization factor is present, but falls to 0.32 when profile and environment must be considered together, with optional personalized parameters dropping to 0.45; the paper reads this as evidence that current LLMs cannot jointly reason about both dimensions and often prioritize one at the expense of the other. It also reports that adding personalized information improves tool-use effectiveness over non-personalized baselines, with the gain larger for GPT-4o than for GPT-3.5.

Load-bearing premise

The load-bearing premise is that the GPT-4o-generated tool-call labels, after GPT-4o quality filtering and a small manual check, are genuinely correct personalized choices; if GPT-4o's judgments are biased or misaligned with real user satisfaction, the reported F1 scores and win rates do not measure actual personalized tool utilization.

Editorial extensions

If this is right

Tool-selection benchmarks should include overlapping toolsets with contextual factors, because functional correctness alone overstates real-world agent competence.
Simply scaling model size will not close the personalization gap: Qwen2.5-72B shows little improvement over Qwen2.5-32B on ToolSpectrum.
Hierarchical prompting—first predicting the domain, then giving only the relevant toolset—improves F1 by reducing context noise, giving a cheap baseline for future work.
Domain-policy compliance is a major failure mode in combined profile-and-environment scenarios (31% of sampled errors), so personalized tool use and constraint handling are linked.
The effectiveness gain from personalization grows with model strength, so stronger base models will likely amplify the benefit of personalized tool-use training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The measured joint-reasoning deficit may partly reflect GPT-4o's own label-generation biases; a benchmark generated by a different model or by human experts could produce different rankings and should be tested.
A natural extension is multi-turn personalization, where the profile and environment update after each tool call (for example, a canceled flight changes the travel context); ToolSpectrum currently evaluates one-shot decisions.
The profile/environment decomposition could be used to diagnose model-specific biases, since some models may systematically favor profile over environment or vice versa, which is testable with the existing data splits.
Because the authors note that excessive context length is a major limitation of the benchmark, retrieval or hierarchical toolset pruning may matter more than raw reasoning ability for this task; this is testable with the released code.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper gives the tool-augmented LLM community a genuinely new benchmark, and its main empirical pattern is probably real. It is the first to combine user profiles and environmental factors in a tool-selection task, and the formalization into APP/API/RP/OP is a clean way to separate where models fail. The finding that all models -- open and closed -- collapse on optional parameters when both factors are present, dropping to around 0.45-0.50 OP F1, is consistent across a wide model sweep and is the kind of result that should make people in the subfield pay attention.

What it does well: the data construction pipeline is careful in the right places. The toolset covers 9 domains with overlapping apps and APIs, seed data is used to keep generation controllable, and there is manual review plus a quality filter. The error analysis is also a plus; it breaks down where GPT-4o itself fails, including the 31% category where it cannot interpret domain policies under joint conditions. The prompting experiments, especially hierarchical prompting, are a sensible start on mitigation.

The soft spots are real but not disqualifying. The biggest one is exactly what the stress-test note flags: the gold tool calls are generated by GPT-4o (Section 3.2.4), filtered by GPT-4o quality scores (Section 3.2.5), and then GPT-4o is evaluated against those same labels. For the Profile & Environment subset, where GPT-4o's own error rate on policy interpretation is high, the labels in the 'Both' condition may be noisy or biased toward whichever dimension GPT-4o happened to attend to. The manual check is 50 samples per domain scored for plausibility, not for whether the gold answer is uniquely reasonable. That is a genuine gap. The Kappa 0.85 on the win-rate comparison is decent, but that is still GPT-4o judging its own outputs against non-personalized ones, so it does not fully address the circularity. Also, the dataset is small -- 450, 220, and 330 samples -- and the repo URL differs between the abstract and the full text, which is annoying for reproducibility.

I want to be fair: the central empirical pattern holds up better than the stress-test note suggests. The fact that non-GPT models show the same qualitative drop in the joint condition means it is not purely an artifact of GPT-4o rewarding itself. But because the benchmark's validity depends on the gold labels being genuinely correct personalized choices, the circularity has to be addressed with human-validated labels or alternative-reference analysis before the headline numbers can be taken at face value.

For whom: researchers working on tool-augmented LLMs or LLM personalization will want to read this. It deserves a serious referee -- the task definition and the failure pattern are worth engaging with. I would send it to peer review with a request for label validation and a fix to the reproducibility details, but I would not desk-reject it.

Referee Report

3 major / 6 minor

Summary. The paper introduces ToolSpectrum, a benchmark for evaluating LLMs on personalized tool utilization, defined as selecting apps, APIs, required parameters, and optional parameters while taking into account user profiles, environmental factors, and their combination. The benchmark is constructed via a GPT-4o-driven pipeline: GPT-4o generates user instructions, tool-call results (under Profile, Environment, and Profile & Environment conditions), and then filters the data using its own quality scores with a threshold of 8, removing 21.8% of samples. The authors evaluate 16 open-source and API-based models, reporting F1 scores at four granularity levels (APP, API, RP, OP). They find that models perform relatively well on coarse-grained tool selection but poorly on optional parameters, and that the combined Profile & Environment condition is the most difficult, with the best models scoring only about 0.45-0.50 on OP. A secondary win-rate analysis (with GPT-4o as judge) is used to argue that personalization improves tool-utilization effectiveness. The paper also includes an error analysis and prompt-engineering experiments.

Significance. ToolSpectrum addresses a genuine gap in the tool-learning literature: existing benchmarks mostly ignore overlapping toolsets and the influence of user profiles and environmental context on tool choice. The decomposition of personalization into profile, environment, and their joint effect, together with the four-level F1 hierarchy, is sensible and potentially useful for the community. The paper provides broad model coverage, and the qualitative pattern that joint profile/environment reasoning is harder than either dimension alone is consistent with intuition and with the reported tables. If the gold-label validity is established, this could become a useful resource for studying personalized tool agents. However, the current construction relies heavily on GPT-4o both as the oracle that defines correct tool calls and as the judge for the win-rate analysis, which undermines the absolute scores and, to some extent, the cross-condition comparisons. The release of data and code is a clear strength that aids further validation.

major comments (3)

[§3.2.4, §3.2.5, Table 3] Gold tool-call results are generated by GPT-4o using the prompt template in Figure 13 and then filtered by GPT-4o's own 1-10 quality scores with a threshold of 8, removing 21.8% of samples. GPT-4o is also the strongest evaluated model in Table 3. Because the oracle that defines 'correct' personalized calls is the same model family being measured, the reported F1 scores, including the low OP scores (~0.45-0.50) in the Both condition, reflect consistency with GPT-4o's generation policy rather than an independent standard. The manual quality assessment at the end of §3.2.5 scores plausibility on a 1-10 scale, and the paper reports no inter-annotator agreement on whether the gold output is uniquely reasonable or unbiased across the profile/environment dimensions. This matters concretely: §5.3 shows GPT-4o itself misinterprets domain policies in 31% of its sampled errors in the combined condition, so the gold labels in that subset are likely noisy exactly where the headline conclusion is drawn. I ask the authors to (i) report inter-annotator agreement on a sample of gold labels with instructions to judge not merely plausibility but also whether an alternative tool call would be equally valid; (ii) analyze dimension bias in the gold labels (e.g., how often the gold call follows profile vs environment when the two conflict); and (iii) re-evaluate at least the Both subset with gold labels generated by a different model or with human-authored labels to show the main pattern is not an artifact of GPT-4o self-consistency.
[Table 2 and §4.2] The average number of optional parameters in the gold outputs differs greatly across conditions: 1.31 in Profile, 6.79 in Environment, and 6.67 in Both. Since OP F1 is computed over per-sample gold optional-parameter dictionaries, a model that perfectly predicts the single optional parameter in a Profile sample receives F1=1.0 for that sample, whereas in an Environment sample it must correctly predict about 6.8 parameters to achieve the same per-sample F1. The paper's cross-condition ranking (Profile ≈ Environment > Both) is therefore confounded by the number of gold OPs per condition; the Both condition not only combines two personalized factors but also has many more OPs to predict than Profile. Please report OP F1 normalized by the number of gold OPs per sample (or per-parameter precision/recall) and decompose the OP gap into a label-length effect and a joint-reasoning effect. Without this, the central claim that joint reasoning is the bottleneck for OP scores is not fully supported.
[§5.1, Figure 15] The win-rate analysis in Figure 3 uses GPT-4o as the judge (prompt in Figure 15), with human-GPT-4o agreement measured on 100 samples (κ=0.85). A single aggregate κ of 0.85 on 100 samples is not sufficient to rule out systematic judge bias in favor of the personalized outputs, particularly because those outputs follow the same generation template used to create the gold labels. The conclusion that 'personalization improves effectiveness' would be strengthened by reporting win rates with an alternate judge (e.g., Claude or a different strong model) and by showing per-condition human agreement rather than one pooled number. This is a load-bearing check on the RQ1 claim, since the same model that served as oracle and primary evaluated model also serves as the sole automated judge.

minor comments (6)

[Table 3] The column header 'PP' under the Profile condition is a typo for 'OP'; the text and all other columns use OP.
[§3.3] The text states that ToolSpectrum features '23 APPs, 42 APIs,' while Table 2 reports #APPs = 22/22/20 and #APIs = 39/40/33 across the three subsets; please reconcile these numbers.
[§B.3] The sentence 'In this section (5.5), we explore...' references a nonexistent Section 5.5; it should refer to Section B.3.
[Figure 13] Requirement 4 says 'if you feel that the user's instruction violates the user's profile, set the APP, API, and params to null,' but the task definition in §3.1 states that None should be returned when the instruction violates the application's policy, not the user's profile; please correct the wording.
[Figure 4] The caption of Figure 4 does not specify which models are averaged to obtain the per-domain performance; please state the model set or provide per-model results to make the domain analysis reproducible.
[Table 2] The rows 'Avg. Params' and 'Avg. OPs' are not formally defined; please clarify whether these are averages over samples and over which fields, so that readers can interpret the large OP-count differences.

Circularity Check

1 steps flagged · score 5.0 of 10

ToolSpectrum's gold labels are generated and filtered by GPT-4o, and GPT-4o is also the top evaluated model; the central joint-reasoning gap is therefore partly a self-consistency artifact.

other [Section 3.2.4 (Tool Result Collection), Section 3.2.5 (Quality Assessment), Section 4.1 (Models), Section 4.2 (Main Results)]
"Specifically, we input the user instruction I∈I, predefined toolsets T, and either a user profile p∈P, an environment e∈E, or both into GPT-4o to obtain the corresponding tool call results. This process generates three distinct datasets: Profile, Environment, and Profile & Environment. ... we first use GPT-4o to score each data sample across three dimensions. ... We then exclude data points with an average score below 8, removing 21.8% of the data. ... the API-Based models include OpenAI GPT API (gpt-3.5-turbo-16k-0613, gpt-4o-20241120)."

The reference 'correct' tool call is defined by GPT-4o's own output and then filtered by GPT-4o's own quality score, while GPT-4o is one of the evaluated models. Its F1 scores, and the aggregate conclusion that LLMs cannot jointly reason about profiles and environments, are therefore measures of agreement with a reference distribution drawn from GPT-4o itself, not with an independent ground truth. This is most problematic in the Profile & Environment subset, where Section 5.3 reports that 31% of GPT-4o's sampled errors involve failing to interpret domain policy; those same failures can be embedded in the gold labels. Consequently, the low OP F1 scores in the 'Both' condition may reflect label bias or noise from the generator rather than a genuine limitation of current models.

full rationale

ToolSpectrum is a benchmark paper, not a formal derivation, so the classical circularity patterns (self-citation chains, imported uniqueness theorems, ansatz smuggling) are absent. The one substantive circularity is the oracle/evaluator overlap: GPT-4o generates the gold tool calls (Section 3.2.4), GPT-4o scores and filters them (Section 3.2.5), GPT-4o is evaluated against them (Section 4.1), and GPT-4o is used as the judge in the win-rate analysis (Section 5.1). This makes the central claim about poor joint reasoning partly a self-consistency artifact. Some independent content remains: even when the reference is its own generation distribution, GPT-4o's OP F1 in the joint condition is only about 0.45, and the human agreement on the pairwise preference judgments is high. However, the gold labels themselves are never validated against an external standard or human annotation of uniqueness or bias across profile and environment dimensions, so the headline limitation is not robustly established. Score 5 reflects partial circularity in the benchmark's construction rather than a full collapse of the derivation chain.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The benchmark's validity depends on the decomposition of personalization into profile and environment, on GPT-4o as an oracle for ground truth, and on the F1 metric. These are design choices rather than empirically established facts, and they collectively determine what ToolSpectrum measures. No fitted parameters in the physics sense are present, but arbitrary thresholds and sampling choices shape the dataset.

free parameters (2)

Quality filtering score threshold = 8 (out of 10)
Data with average GPT-4o score below 8 are removed (Section 3.2.5), affecting the composition of the benchmark. The threshold is chosen by hand and is not empirically justified.
Demographic sampling distribution parameters = not specified
Age, height, and weight are sampled from normal distributions (Section 3.2.3) but means and standard deviations are not given, so the user profile diversity is uncontrolled and unreproducible.

assumptions (4)

domain assumption User profile and environment are the two key dimensions of personalization for tool utilization.
The task definition in Section 3.1 and data collection in Section 3.2.2 rest on this decomposition; the benchmark's scope depends on these dimensions being the right ones.
domain assumption GPT-4o can generate valid and consistent ground-truth personalized tool calls.
Ground truth is produced by GPT-4o with prompt templates in Section 3.2.4; if these labels are arbitrary or biased, model performance scores are not meaningful.
domain assumption F1 scores on exact hierarchical fields (APP, API, RP, OP) are an appropriate measure of personalized tool utilization.
Section 4.1 uses F1 without specifying the matching procedure; the validity of all model comparisons rests on this metric.
domain assumption Domain policies are objective rules that can be unambiguously applied.
Many samples require applying policies like age restrictions (Section 3.2.2); the benchmark assumes these policies are clear and universally interpretable.

invented entities (1)

Synthetic app and API catalog (e.g., Ctrip, Temu, Freshippo)
purpose: Provide a controlled toolset with overlapping functionality to measure personalized selection.
These apps/APIs are generated and curated by the authors; they are not real-world services, so there is no external evidence for their properties. They serve as test fixtures, not as claims about reality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models." pith.science (2026). https://pith.science/paper/4HKL7LFE

@misc{pith2026250513176,
  author       = {Pith},
  title        = {Pith review of: ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HKL7LFE}},
  note         = {Machine review of arXiv:2505.13176}
}

read the original abstract

While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions, overlooking the context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs' capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool utilization significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code are available at https://github.com/Chengziha0/ToolSpectrum.

Figures

Figures reproduced from arXiv: 2505.13176 by the authors.

**Figure 1.** An example from our proposed ToolSpectrum, illustrating the effects of user profile and environment on personalized tool utilization. This illustrates three distinct scenarios, considering profile-only , environment-only , and combined profile and environment factors . et al., 2025). Recent researches demonstrate the effectiveness of this integration in diverse domains, including travel planning (Xie et al., 2024),… view at source ↗

**Figure 2.** The overall construction process of ToolSpectrum, including (a) Toolset Collection, (b) Profile and Environment Collection, (c) Tool-call Result Collection, and (d) Quality Assessment. 3 . These domains include shopping, entertainment, travel, delivery, grocery, knowledge, news, health and finance. In each domain, we leverage GPT-4o to generate initial designs for Apps and APIs with similar functions. We then manual… view at source ↗

**Figure 3.** Win rates for personalized vs. nonpersonalized settings in GPT-3.5-turbo and GPT-4o. 5.1 The Importance of Personalization in Tool Utilization This section investigates how integrating personalized factors enhances the effectiveness of tool utilization. Specifically, we compare the results of tool utilization with and without personalized information. We then use GPT-4o to evaluate which output from the two appro… view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Performance comparison of different models across various domains for three distinct data types. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: The performance gap between hierarchical [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

**Figure 6.** Figure 6: The relationship between the model parame [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: Error examples and specific errors are highlighted in [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: Prompt Template for Generating Consumption Preference. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: Prompt Template for Generating Use Habit. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: Prompt Template for Generating Content Preference. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Prompt Template for Scoring Profile [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Prompt Template for Generating Instruction. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: Prompt Template for Generating Tool-call Results. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

**Figure 14.** Figure 14: Prompt Template for Scoring Tool-call Result. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

**Figure 15.** Figure 15: Prompt Template for Selecting the Superior Results. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
cs.CL 2025-07 conditional novelty 5.0 of 10

MemTool is a short-term memory framework with three modes (autonomous, workflow, hybrid) that lets LLM agents add and remove tools across multi-turn conversations, evaluated over 100 turns on 13+ models.

Reference graph

Works this paper leans on

27 extracted references · 24 canonical work pages · cited by 1 Pith paper

[1]

The profile must include the following attributes: gender, height, age, weight, occupation, in- come level, and preferences (including consumption preferences, content consumption preferences, and app usage history preferences)

work page
[2]

International Jour- nal of Advanced Computer Science and Applications, 15(3)

Research on personalized recommendation algorithms based on user profile. International Jour- nal of Advanced Computer Science and Applications, 15(3). Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lava...

arXiv 2023
[3]

In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , pages 796– 806

Doing personal laps: Llm-augmented dialogue construction for personalized multi-session conver- sational search. In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , pages 796– 806. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage m...

arXiv 2022
[4]

In Proceedings of the ACM on Web Conference 2024, pages 1464–1473

Cognitive personalized search integrating large language models with an efficient memory mecha- nism. In Proceedings of the ACM on Web Conference 2024, pages 1464–1473. Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma, Jinyang Liu, Guohao Cai, Xi Xiao, and Rui Zhang

work page 2024
[5]

Do not include keys in the keyword dictionary, as the generated instructions can only contain values

work page
[6]

### Input: ##API Description {api description} ## Keywords: {keywords} ### Output: Figure 12: Prompt Template for Generating Instruction

Do not explicitly express the content of the API Description. ### Input: ##API Description {api description} ## Keywords: {keywords} ### Output: Figure 12: Prompt Template for Generating Instruction. Prompt Template for Generating Tool-call Results You are a personalized tool assistant. I will provide you with user instructions, a user profile, an externa...

work page
[7]

This means: - Demographic Coherence: The combination of age, gender, height, weight, occupation, and income should form a believable profile

The score reflects the [plausibility] and [internal consistency] of the profile. This means: - Demographic Coherence: The combination of age, gender, height, weight, occupation, and income should form a believable profile. For example, a 20-year-old high school dropout is unlikely to have a high income. Similarly, a very short, elderly person is unlikely ...

work page
[8]

The scoring range is integer values from 1 to 10, where 1 is highly implausible and 10 is perfectly plausible

work page

Show all 27 references

[9]

Input: {profile} Output: Figure 11: Prompt Template for Scoring Profile

You must return only an integer value (1-10) and no other output. Input: {profile} Output: Figure 11: Prompt Template for Scoring Profile. Prompt Template for Generating Instruction I am creating a dataset for tool calls, and I will input some keywords for you. Please help me ...
[10]

The input includes an API description and keywords. The API description is an introduction to the current application scenario; Keywords are a dictionary containing some keywords, and you need to concatenate them into instructions that fit the user’s usage habits
[11]

The generated instruction is in the first person
[12]

The generated instructions must include these keywords and cannot be modified in any way
[13]

Do not output any irrelevant content, instructions should not contain any irrelevant symbols, and only natural language is allowed
[16]

The input consists of user instructions, user profile, and the external environment. The user instruction is a string that describes the user’s needs; the user profile is a dictionary that includes the user’s gender, height, age, weight, occupation, income level, and preferenc...
[17]

You should comprehensively consider the user instruction, profile, and environment to give the most appropriate tool invocation result
[18]

APP": "APP

The output format should be a dictionary containing the app name, API name, corresponding parameters, and the explanation, as shown in the following format: { "APP": "APP", "API": "API", "params": { "key1": "value1", "key2": "value2", ... } "explanation": "EXPLANATION" }
[19]

There will be a policy field in the environment, and if you feel that the user’s instruction violates the user’s profile, set the APP, API, and params to null
[20]

Only output one dictionary, do not output any other content. ### Input ## User Instruction {instruction} ## User profile {profile} ## Environment {environment} ## Toolset {toolset} ### Output Figure 13: Prompt Template for Generating Tool-call Results. Prompt Template for Scor...
[21]

Please judge whether the tool calling results meet expectations based on user instructions, user profile, external environment, and tool calling results
[22]

The output format is separated by commas

Please rate the following three aspects separately, ranging from 1 to 10 points. The output format is separated by commas
[23]

You should evaluate whether the tool calling results meet expectations from three aspects: - Whether the tool calling results can solve the user’s needs; - Whether the tool calling results match the user’s profile; - Whether the tool calling results align with the external environment
[24]

Do not output any irrelevant content, only output the answer. Below are the user instructions, user profile, external environment, and tool calling results I’m in- putting: ### Input ## User Instruction {instruction} ## User profile {profile} ## Environment {environment} ## To...
[25]

The input includes a description of the toolset, user instructions, and results A and B
[26]

You need to determine which result better meets the user’s needs
[27]

You can only output A or B, do not output any other irrelevant content. ### Input ## Toolset {toolset} ## User Profile {profile} ## Response A {response_A} ## Response B {response_B} ### Output Figure 15: Prompt Template for Selecting the Superior Results
[2022]

In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR ’22, page 2912–2923, New York, NY , USA

Bars: Towards open benchmarking for recom- mender systems. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR ’22, page 2912–2923, New York, NY , USA. Association for Computing Machinery. A ToolSpetrum D...

2023
[2024]

In European Conference on Information Retrieval, pages 364–381

Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pages 364–381. Springer. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, ...

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

The profile must include the following attributes: gender, height, age, weight, occupation, in- come level, and preferences (including consumption preferences, content consumption preferences, and app usage history preferences)

work page

[2] [2]

International Jour- nal of Advanced Computer Science and Applications, 15(3)

Research on personalized recommendation algorithms based on user profile. International Jour- nal of Advanced Computer Science and Applications, 15(3). Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lava...

arXiv 2023

[3] [3]

In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , pages 796– 806

Doing personal laps: Llm-augmented dialogue construction for personalized multi-session conver- sational search. In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , pages 796– 806. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage m...

arXiv 2022

[4] [4]

In Proceedings of the ACM on Web Conference 2024, pages 1464–1473

Cognitive personalized search integrating large language models with an efficient memory mecha- nism. In Proceedings of the ACM on Web Conference 2024, pages 1464–1473. Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma, Jinyang Liu, Guohao Cai, Xi Xiao, and Rui Zhang

work page 2024

[5] [5]

Do not include keys in the keyword dictionary, as the generated instructions can only contain values

work page

[6] [6]

### Input: ##API Description {api description} ## Keywords: {keywords} ### Output: Figure 12: Prompt Template for Generating Instruction

Do not explicitly express the content of the API Description. ### Input: ##API Description {api description} ## Keywords: {keywords} ### Output: Figure 12: Prompt Template for Generating Instruction. Prompt Template for Generating Tool-call Results You are a personalized tool assistant. I will provide you with user instructions, a user profile, an externa...

work page

[7] [7]

This means: - Demographic Coherence: The combination of age, gender, height, weight, occupation, and income should form a believable profile

The score reflects the [plausibility] and [internal consistency] of the profile. This means: - Demographic Coherence: The combination of age, gender, height, weight, occupation, and income should form a believable profile. For example, a 20-year-old high school dropout is unlikely to have a high income. Similarly, a very short, elderly person is unlikely ...

work page

[8] [8]

The scoring range is integer values from 1 to 10, where 1 is highly implausible and 10 is perfectly plausible

work page