Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis

Ik-hwan Kim; Jisoo Mok; Sangkwon Park; Sungroh Yoon

REVIEW 3 major objections 5 minor 17 references

Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HiCUPID benchmark tests whether LLMs can truly personalize responses to a user.

desk verdict A solid new personalization benchmark with a persuasive evaluation story that currently overclaims human alignment on a 100-sample anchor and an unspecified proxy-training split. read the letter →

arxiv 2506.01262 v1 pith:V4VMTLN6 submitted 2025-06-02 cs.CL

Jisoo Mok , Ik-hwan Kim , Sangkwon Park , Sungroh Yoon This is my paper

classification cs.CL

keywords LLMpersonalizationbenchmarkconversationalAIpersonalizedassistantLLM-as-a-judgeretrieval-augmentedgenerationdirectpreferenceoptimizationlongcontext

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HiCUPID, a synthetic conversational benchmark built to test whether large language models can act as personalized assistants. The authors argue that existing personalization datasets only cover classification or shallow persona tasks, whereas a real assistant must infer user information from dialogue, combine multiple pieces of information, handle long histories, and respond proactively. HiCUPID supplies per-user personas, profiles, and schedules embedded in dialogues, plus single- and multi-info question-answer pairs. The paper further argues that a small Llama-3.2-3B evaluator trained on GPT-4o judgments scores responses in a way that closely tracks human preferences, unlike BLEU and ROUGE-L. If these claims hold, HiCUPID gives the field a standardized way to measure and improve LLM personalization.

What carries the argument

The load-bearing object is the HiCUPID synthetic user model, in which each user has structured metadata (25 persona dimensions, five profile fields, ten dated schedules) and a dialogue history of roughly 15,000 to 17,000 tokens that reveals that metadata only implicitly. On top of this, the evaluation machinery is a judge pipeline: GPT-4o compares a model response against a ground-truth personalized answer with criteria of personalization and logical validity, and the resulting preferences fine-tune a Llama-3.2-3B proxy evaluator. The score S is the model win rate over ground truth plus half the tie rate, and the paper uses this score to compare zero/few-shot inference, retrieval (BM25, Contriever), SFT, DPO, and SFT+DPO.

What would settle it

Collect a larger, pre-registered human-preference study on several hundred responses across multiple models and QA types, then compute agreement between human choices and the Llama-3.2 proxy; if agreement drops well below the reported levels or humans systematically prefer responses the proxy ranks lower, the central claim that the proxy mirrors human preferences fails. A second check: if training the same proxy architecture on judgments from a different LLM judge yields substantially different scores on the same responses, the evaluation is judge-dependent rather than human-aligned.

Watch

Extended reading notes

Core claim

The central discovery is a benchmark, HiCUPID, whose design operationalizes five desiderata of a personalized AI assistant: adherence to user information, understanding of implicit information, reasoning from multiple information sources, long-context modeling, and proactiveness. Each synthetic user is defined by 25 personas, a five-field profile, and ten schedules; dialogues reveal this information only implicitly, and QA pairs test whether a model can retrieve and combine it. The paper shows that a GPT-4o-based judge agrees with human preference judgments on 100 zero-shot responses, and that distilling that judge into Llama-3.2-3B preserves the agreement. Under this evaluation, off-the-shelf LLMs score well below the ground-truth personalized answers, retrieval-based methods help, supervised fine-tuning helps, and DPO applied alone collapses under most settings, while SFT followed by DPO gives the largest gains.

Load-bearing premise

The benchmark's validity rests on the assumption that GPT-4o's judgments of personalization and logical validity reliably stand in for human preferences, since GPT-4o also generated the ground-truth answers and the proxy evaluator is trained on GPT-4o labels.

Editorial extensions

If this is right

HiCUPID can serve as the first open benchmark for training and evaluating personalized assistants across the five desiderata at once.
The distilled Llama-3.2 evaluator makes personalization scoring inexpensive enough to run at scale, supporting reward modeling or model selection during development.
The strong long-context penalty shows that models need better memory or retrieval as conversations grow beyond a few thousand tokens.
SFT+DPO's consistent gains over SFT suggest that preference tuning after grounded fine-tuning is a useful recipe for personalization.
BLEU and ROUGE-L rankings mislead when judging personalization, so human-aligned preference scores should replace or supplement them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit is that because HiCUPID ships a generic non-personalized answer for every QA pair, the dataset can be reused directly as preference data for reward-model training and RLHF-style alignment, not only for evaluation.
The synthetic users are generated from sampled profiles; if the method scales, fresh users can be generated on demand, which could keep the benchmark from becoming stale as models improve.
The human-anchor study covers only 100 zero-shot persona QA pairs, so a larger human study across more models and question types is needed to confirm that the Llama-3.2 proxy generalizes beyond the reported setting.
A testable extension: because the proxy is trained on GPT-4o judgments, swapping the judge for another LLM and comparing the resulting proxies would reveal how much of the measured personalization is judge-dependent.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the benchmark is real and useful; the evaluator-alignment claim is not yet supported. I checked the appendix after reading the stress-test note, and the concern holds: A6 says the 400k proxy-training samples come 'from the inference results' in the listed settings, but never says those responses are drawn from the train split. The 100 human-evaluated persona QA pairs are from Test Set 1. If the zero-shot and few-shot responses used for proxy training include Test Set 1 responses, Figure 2 is not an independent test of human alignment. Even if the authors intended a train-only split, they need to state it and check for leakage.

What the paper does well: HiCUPID fills a real gap. The combination of persona, profile, and schedule information, implicit revelation through dialogue, multi-hop persona+profile questions, and a 17k-token context is genuinely more demanding than LaMP or PChatbot. The construction pipeline is detailed, the prompt templates are all in the appendix, the ablation study is broad, and the limitations section is honest about negation issues, DPO failures, and privacy risks. The dataset and code are released. That is a substantial resource.

The soft spots are in the evaluation story. First is the split ambiguity above, which is load-bearing because the central claim is that the Llama-3.2 proxy 'closely mirrors human preferences.' Second, the human anchor is thin: 10 volunteers per model, 100 persona QA pairs, no inter-annotator agreement, no confidence intervals, no correlation coefficient. The bar chart in Figure 2 looks similar across human and model judges, but that is not a measured alignment. Third, the main results in Tables 3 and 4 have no error bars; single runs make the rankings less certain.

None of this sinks the benchmark. The dataset and the experimental findings--for example, that SFT works far better than DPO for this task, and that retrieval helps but not enough--are worth having. The citation pattern is appropriate; the authors engage with the relevant personalization literature. The paper is clearly written and the thinking is coherent.

Who is this for? Researchers working on personalized assistants and LLM evaluation. It deserves a serious referee. I would send it to review, but with the explicit request that the authors clarify the proxy training split, check for overlap with the human-evaluated responses, and either enlarge the human validation or report agreement metrics and confidence intervals. With those changes, the evaluation claims become credible.

Referee Report

3 major / 5 minor

Summary. This manuscript introduces HiCUPID, a synthetic, GPT-4o-generated conversational benchmark for training and evaluating LLMs as personalized assistants. Each of 1,500 synthetic users is defined by 25 personas, five profile attributes, and ten schedules; the user's information is revealed naturally through dialogue histories of roughly 17k tokens on average, and per-user QA pairs probe single-info (persona, schedule) and multi-info (persona+profile) reasoning. The design is organized around five desiderata (AUI, UII, MI, LC, PR), and the paper provides two automated evaluation protocols (GPT-4o as judge, and a distilled Llama-3.2-3B proxy) anchored by a 100-item human evaluation. Extensive experiments with GPT-4o-mini, Llama-3.1-8B, Mistral-7B, and Qwen-2.5-7B under zero/few-shot prompting, BM25 and Contriever retrieval, SFT, DPO, and SFT+DPO show that SFT-based methods improve personalization, DPO alone collapses on most models, retrieval helps single-info tasks, and long context remains a bottleneck; BLEU and ROUGE scores are shown to contradict human preferences on this benchmark.

Significance. The paper's strengths are real and should be acknowledged explicitly: the dataset construction pipeline is detailed and reproducible (all prompt templates are included), the scale is substantial (1,500 users, roughly 60k dialogues, 40 QA pairs per user, two held-out settings), the empirical study is broad with unusually transparent hyperparameter reporting, and the authors release the dataset, code, and proxy evaluator. The central quantitative findings, such as SFT+DPO being the most reliable recipe, DPO alone being unstable, and BLEU/ROUGE ranking models differently from preference-based evaluation, are interesting and clearly presented. The load-bearing claim, however, is that the Llama-3.2 proxy 'closely mirrors human preferences' (Abstract and Section 4.2) and that GPT-4o evaluation 'closely follows' human preference (Section 4.1); as detailed in the major comments, the supporting evidence is currently too thin and ambiguous, and the validation protocol contains a potential training/validation overlap, to carry that claim as it stands.

major comments (3)

[Section 4.2, Appendix A6, Figure 2] Appendix A6 states that the 400k training samples for the Llama-3.2 proxy evaluator are taken 'from the inference results in the following experimental settings,' a list that explicitly includes the zero-shot and few-shot inference settings used to generate the model responses evaluated in Section 4.1, and Section 4.1's human study is conducted on 100 zero-shot persona QA pairs from Test Set 1. The manuscript never states whether these 100 responses (or the rest of the Test Set 1 and Test Set 2 evaluations) were excluded from the proxy's training set. If they were included, then Figure 2 compares the proxy against human judgments on inputs whose GPT-4o labels were seen during training, so it measures train-set agreement rather than human alignment, and the abstract's 'closely mirrors human preferences' claim is unsupported. This missing split information is load-bearing because the entire human-alignment claim rests on Figure 2 being an independent test; please specify the exact provenance of the 400k samples, confirm disjointness from the human-evaluated items, and re-run the Figure 2 validation on a strictly held-out set, retraining the proxy if necessary.
[Section 4.1, Figure 2] The human anchor consists of 100 persona QA pairs rated by 10 volunteer annotators per model, and the paper reports no inter-annotator agreement (e.g., no Cohen's kappa among annotators, no majority-vote consistency), no confidence intervals around the reported S scores, and no per-item agreement statistic between humans and either automated judge. The four S values in Figure 2 differ from the corresponding GPT-4o values by up to 5.3 points (GPT-4o-mini: human 25.6 versus S_GPT 29.4), while adjacent model comparisons under human evaluation (Mistral 22.8 versus Qwen 25.3) span a similar range, so without error bars the claim that GPT-4o and the proxy 'closely follow' human preferences cannot be assessed quantitatively. Moreover, only persona QA pairs are human-validated, although the proxy is claimed to be human-aligned for schedule and multi-info QA pairs as well; the paper should either extend the human validation to those QA types or restrict the alignment claim to persona QA pairs. Given that GPT-4o generated the ground-truth answers and the proxy is distilled from GPT-4o's judgments, the 100-item human comparison is the only independent anchor in the evaluation chain, so the missing agreement statistics and confidence intervals are load-bearing rather than cosmetic.
[Section 3, Appendix A3(e), Figure A14] The five desiderata are presented as a complete characterization of a personalized assistant without justification (Section 3), and the proactiveness desideratum (PR) is not directly operationalized: the evaluation prompts in Figure A14 score only 'Personalization' and 'Logical Validity,' and Appendix A3(e) argues that PR is satisfied because the QA pairs were 'designed' such that the LLM must provide proactive answers. This makes the PR checkmark in Table 1 an assertion about the dataset construction rather than a measured property of model responses, and it implies that the same evaluation score would be obtained even if the proactivity requirement were absent from the prompts. Please either add a direct scoring criterion or rubric excerpt for proactivity, or explicitly argue with concrete examples how the personalization criterion entails proactivity for the QA pairs in HiCUPID; otherwise the claim that HiCUPID 'properly reflects the challenges' of a personalized assistant is overstated relative to the evaluation protocol.

minor comments (5)

[Section 5.1] The sentence explaining the total score ('the schedule score is halved to match its range with the scores of the two remaining QA pair types') is confusing as written, since halving maps the schedule score to [0,50] while persona and multi-info scores remain in [0,100]; the intended effect is to bring the ground-truth reference score to 50 for all three QA types, and the wording should say so directly.
[Table 3] In the Qwen-2.5-7B 0-shot row, the multi-info S_GPT value appears as '17' without a decimal, unlike the surrounding entries in the same table; use a consistent number format.
[Section 2] The paragraph stating that a detailed discussion of personalization approaches 'has been moved to Appendix' does not give the appendix section number; add a pointer to Appendix A1.
[References] Several bibliography entries lack publication years (for example, the LoRA reference by Hu et al. and the 'Editing personality for large language models' entry by Mao et al.); complete these entries.
[Conclusion] The Conclusion calls HiCUPID 'the first open-source benchmark' for personalized assistants, which overstates the novelty given that LaMP and PChatbot are open resources with partially overlapping goals; consider qualifying the claim as the first long-context conversational benchmark covering all five desiderata.

Circularity Check

2 steps flagged · score 5.0 of 10

The proxy evaluator's human-alignment claim is not independently established: its 400k training samples are GPT-4o labels from the same four-model zero-shot settings used in Figure 2, with no stated disjointness from the 100 human-rated Test Set 1 pairs, and the gold answers and the judge are both GPT-4o.

fitted input called prediction [Section 4.2, Appendix A6, Figure 2]
"A total of 400k GPT-4o evaluation samples are used to train the Llama-3.2-3B-based proxy evaluation model. The samples are from the inference results in the following experimental settings: ... GPT-4o-mini: Zero- and few-shot inference with prompt in the user role. ... Llama, Mistral, Qwen: Zero- and few-shot inference with prompt in the user role; BM25- and Contriever-based utterance-level retrieval with k = 5; SFT (LR=1e-4) and DPO (LR=1e-5) with LoRAr = 256. ... The results in Figure 2 again show that our proxy evaluator estimates human preference as closely as its teacher model."

The proxy is fitted to 400k GPT-4o evaluation labels collected from the zero-/few-shot inference results of the same four models that Figure 2 uses for the human-preference comparison (100 zero-shot persona QA pairs from Test Set 1). The paper never states that these 400k training samples exclude Test Set 1 or the 100 human-rated pairs, and the listed training settings (zero-shot, user-role prompt, same models) match the validation setting. If any Test Set 1 zero-shot responses are in the training sample, Figure 2 is not an independent alignment test: the proxy has already seen those exact responses and their GPT-4o labels, so its 'human alignment' reduces to reproducing training labels.
self definitional [Introduction, Section 3.3, Section 4.1]
"we introduce “HiCUPID,” a new synthetic, GPT-4o-generated dataset ... All QA pairs include a personalized and a general answer ... we replace human evaluators with GPT-4o, whose preference is known to be aligned with that of a human"

The ground-truth personalized answers in every QA pair are written by GPT-4o, and the evaluation protocol then asks GPT-4o to choose between a model response and that GPT-4o-written gold answer. Thus S_GPT is, by construction, a measure of GPT-4o's preference between a model's output and GPT-4o's own output, not an independent measure of human preference. The only human anchor is 100 zero-shot persona QA pairs from Test Set 1; for all other QA-pair scores in Tables 3 and 4, no human judgment is involved. The benchmark's 'personalization quality' is therefore circularly defined: the model that generates the gold standard is also the model that judges whether a response is personalized.

full rationale

The dataset construction itself is not circular: HiCUPID is a synthetic dataset generated from explicit user metadata, and the five desiderata are stated design goals rather than derived results. The circularity lies in the evaluation chain. First, the Llama-3.2 proxy evaluator is trained on 400k GPT-4o evaluation samples drawn from the inference results of the exact experimental settings (zero-/few-shot, BM25/Contriever, SFT/DPO) used to produce the model responses that the proxy is later claimed to evaluate in a human-aligned way. Since the paper does not specify that these training samples are disjoint from Test Set 1 — the source of the 100 human-evaluated pairs in Figure 2 — the reported human alignment of the proxy may be measuring in-distribution memorization of GPT-4o labels rather than an independent agreement with human preferences. Second, GPT-4o is both the generator of the dataset's gold personalized answers and the judge that compares model responses against those gold answers; the metric therefore encodes GPT-4o's own stylistic preferences, with only a 100-sample human check as external grounding. These two issues do not make the dataset contribution circular, and there is a genuine human anchor, but they render the central claim that the proxy 'closely mirrors human preferences' only partially supported. Scoring is 5 rather than higher because the evaluation is not a pure identity: the 100 human-evaluated responses provide some external signal, and the proxy's agreement with GPT-4o is measured on real held-out model outputs; the circularity is partial and stems from missing disjointness guarantees and same-model gold/judge construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The benchmark's validity rests on the assumptions that GPT-4o's preferences approximate human preferences, that synthetic dialogues are realistic, and that the five desiderata are the right decomposition. The evaluation score weighting and S metric tie-weight are hand-defined. Several training hyperparameters were selected on the validation performance of one model, making the reported best results partly optimized rather than pre-registered.

free parameters (3)

S metric tie weighting = 0.5
Section 4.1 defines S as model win rate plus 0.5 times tie rate; the 0.5 weight is a modeling choice made by the authors.
Total score weighting = persona 25/40, schedule 10/40, multi-info 5/40 with schedule halved
Section 5.1 defines the weighted average; weights follow QA pair counts, not user importance.
Training hyperparameters for best results = SFT LR=1e-4 r=256; DPO LR=1e-6 r=128; SFT+DPO LR=3e-6 r=256
Section A7 reports the best settings found by search on the Llama model; the 'best' results are therefore partly optimized on development data.

assumptions (4)

domain assumption GPT-4o preferences approximate human preferences for assistant responses.
Section 4.1: 'their close alignment with human evaluation makes GPT-4o's evaluation a well-accepted alternative to human evaluation'; validated only on 100 samples in this paper.
domain assumption Synthetic GPT-4o-generated dialogues and QA pairs are realistic enough to reveal personalization ability.
Section 3.2: prompts are heavily engineered; the paper provides prompt templates but no independent realism validation.
ad hoc to paper The five desiderata (AUI, UII, MI, LC, PR) form a complete characterization of a personalized assistant.
Section 3 defines them as the organizing framework; no external validation of completeness.
domain assumption The proxy evaluator can be trained on GPT-4o's judgments without losing human alignment.
Section 4.2 distills GPT-4o preferences into Llama-3.2-3B; this assumes distillation preserves the human-aligned signal.

invented entities (1)

Five desiderata (AUI, UII, MI, LC, PR)
purpose: Define the benchmark's test coverage and the criteria for a personalized assistant.
New taxonomy introduced in Section 3; no independent evidence that these categories are the right or complete set.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis." pith.science (2026). https://pith.science/paper/V4VMTLN6

@misc{pith2026250601262,
  author       = {Pith},
  title        = {Pith review of: Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4VMTLN6}},
  note         = {Machine review of arXiv:2506.01262}
}

read the original abstract

Personalized AI assistants, a hallmark of the human-like capabilities of Large Language Models (LLMs), are a challenging application that intertwines multiple problems in LLM research. Despite the growing interest in the development of personalized assistants, the lack of an open-source conversational dataset tailored for personalization remains a significant obstacle for researchers in the field. To address this research gap, we introduce HiCUPID, a new benchmark to probe and unleash the potential of LLMs to deliver personalized responses. Alongside a conversational dataset, HiCUPID provides a Llama-3.2-based automated evaluation model whose assessment closely mirrors human preferences. We release our dataset, evaluation model, and code at https://github.com/12kimih/HiCUPID.

Figures

Figures reproduced from arXiv: 2506.01262 by the authors.

**Figure 1.** Configuration of HiCUPID. (1) A user u is characterized by a set of metadata or personal information (P u ∪ Qu ∪ Su ), which is only used for evaluation. (2) For each personal information, we create a dialogue that implicitly hints at it. All dialogues are aggregated into Du . (3) To probe whether the LLM picked up on one piece of personal information, we create a single-info QA pair for each persona or schedule. To… view at source ↗

**Figure 2.** Evaluation of 100 zero-shot model-generated responses with human evaluators [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Visualization of responses from the Llama model after SFT and DPO training. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 6.** Figure 6: Influence of the retrieval setting on the per [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 5.** Figure 5: How changing the number of few-shot demon [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

[1]

First considers the user's profile and explicitly references it to deliver a tailored answer

work page
[2]

role": "user

Then considers the user's characteristics and explicitly references them to further customize the response. - General Answer: The assistant provides a response that does not take the user's profile or characteristics into account and remains general. - Question is related to their given profile and characteristics. - Personalized answer must consider both...

work page 2025
[4]

arXiv preprint arXiv:2310.03304

Learning personalized story evaluation. arXiv preprint arXiv:2310.03304. T Wolf. 2019. Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771. Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020. Mind: A large- scale dataset...

arXiv 2019
[6]

Li et al

first fine-tunes the LLM with the LoRA adapter on user-specific information and then ap- plies RAG. Li et al. (Li et al., 2023b) and Tang et al. (Tang et al., 2023) utilize data-driven methods to extract persona information more compactly to reduce noisy and fine-grained learning signals. A2 Persona Dimensions of HiCUPID Following is the list of persona d...

work page 2023
[10]

Personalization: Does the response effectively consider the user's provided personal information?

work page
[11]

- Ensure the comparisons are concise and directly address the criteria

Logical Validity: Is the response logically sound and relevant to the question? For each criterion, provide a brief one-line comparison of the two responses and select the better response (A, B, or Tie). - Ensure the comparisons are concise and directly address the criteria. - If both answers are equally strong or weak in a category, mark it as a Tie. - D...

work page
[12]

Personalization: [Brief comparison of A and B]

work page
[13]

The goal of this survey is to evaluate which sample response addresses the question better

Logical Validity: [Brief comparison of A and B] Better Response: [A/B/Tie] Input: - User's Personal Information: {metadata} - Question: {question} - Answer (A): {answer_a} - Answer (B): {answer_b} Prompt for GPT-4o A/B Evaluation on Persona and Multi-Info QA Pairs You will be provided a user's question to an AI assistant, a user's profile, and two sample ...

work page

Show all 17 references

[14]

Do not use bold font

Personalization: Does the answer acknowledge and appropriately consider the user's schedule? Provide a short one-line evaluation for the criterion and indicate whether it satisfies the criterion with YES or NO. Do not use bold font. Output Format:
[15]

Personalization: [Short one-line evaluation] Satisfaction: [YES/NO] Example 1: - User's Schedule: - Datetime: 11 AM on March 13, 2025 - Event: Attend a networking brunch with fellow sports journalists and industry experts. - Question: Can you add a meeting with the editorial t...

2025
[16]

Satisfaction: YES Example 2: - User's Schedule: - Datetime: 11 AM on March 13, 2025 - Event: Attend a networking brunch with fellow sports journalists and industry experts

Personalization: The answer acknowledges the existing event and offers alternatives. Satisfaction: YES Example 2: - User's Schedule: - Datetime: 11 AM on March 13, 2025 - Event: Attend a networking brunch with fellow sports journalists and industry experts. - Question: Can you...

2025
[17]

Satisfaction: NO Input: - User's Schedule: - Datetime: 9 AM on January 13, 2025 - Event: Conduct an in-depth analysis session for an upcoming sports documentary

Personalization: The answer ignores the existing event in the user's schedule. Satisfaction: NO Input: - User's Schedule: - Datetime: 9 AM on January 13, 2025 - Event: Conduct an in-depth analysis session for an upcoming sports documentary. - Question: Can you add a meeting wi...

2025
[25]

Personalization

Finance A3 How HiCUPID Tests 5 Desiderata of Personalized Assistant Below, we detail how our dataset configuration and evaluation criteria in the evaluation prompt (Fig- ure A14) together probe the five desiderata of a personalized assistant, outlined in Section 3. (a) Adheren...

1998
[2020]

arXiv preprint arXiv:2009.03300

Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learnin...

2009 arXiv
[2021]

Improving dialog systems for negotiation with personality modeling. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 681–693....

2019 arXiv
[2023]

is this document relevant?. . . probably

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6. Konstantina Christakopoulou, Alberto Lalama, Cj Adams, Iris Qu, Yifat Amir, Samer Chucri, Pierce V ollucci, Fabio Soldo, Dina Bseiso, Sar...

2023 arXiv
[2024]

Advances in Neu- ral Information Processing Systems, 36

Direct preference optimization: Your language model is secretly a reward model. Advances in Neu- ral Information Processing Systems, 36. Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. 2023. Integrati...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

First considers the user's profile and explicitly references it to deliver a tailored answer

work page

[2] [2]

role": "user

Then considers the user's characteristics and explicitly references them to further customize the response. - General Answer: The assistant provides a response that does not take the user's profile or characteristics into account and remains general. - Question is related to their given profile and characteristics. - Personalized answer must consider both...

work page 2025

[3] [4]

arXiv preprint arXiv:2310.03304

Learning personalized story evaluation. arXiv preprint arXiv:2310.03304. T Wolf. 2019. Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771. Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020. Mind: A large- scale dataset...

arXiv 2019

[4] [6]

Li et al

first fine-tunes the LLM with the LoRA adapter on user-specific information and then ap- plies RAG. Li et al. (Li et al., 2023b) and Tang et al. (Tang et al., 2023) utilize data-driven methods to extract persona information more compactly to reduce noisy and fine-grained learning signals. A2 Persona Dimensions of HiCUPID Following is the list of persona d...

work page 2023

[5] [10]

Personalization: Does the response effectively consider the user's provided personal information?

work page

[6] [11]

- Ensure the comparisons are concise and directly address the criteria

Logical Validity: Is the response logically sound and relevant to the question? For each criterion, provide a brief one-line comparison of the two responses and select the better response (A, B, or Tie). - Ensure the comparisons are concise and directly address the criteria. - If both answers are equally strong or weak in a category, mark it as a Tie. - D...

work page

[7] [12]

Personalization: [Brief comparison of A and B]

work page

[8] [13]

The goal of this survey is to evaluate which sample response addresses the question better

Logical Validity: [Brief comparison of A and B] Better Response: [A/B/Tie] Input: - User's Personal Information: {metadata} - Question: {question} - Answer (A): {answer_a} - Answer (B): {answer_b} Prompt for GPT-4o A/B Evaluation on Persona and Multi-Info QA Pairs You will be provided a user's question to an AI assistant, a user's profile, and two sample ...

work page