{"id":"eded3360-485b-446b-878f-0c1473c5facd","arxiv_id":"2506.01262","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HiCUPID is a synthetic benchmark for personalized LLM assistants, with multi-hop questions and a distilled Llama-3.2 evaluator that closely matches human preferences on a small validation set.","lead":"This paper introduces HiCUPID, a new synthetic benchmark of chat histories and questions that tests whether AI assistants can remember a user's personality, profile, and schedule when answering. It also provides a small automated judge that rates assistant responses, and shows today's assistants rarely deliver truly personalized answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proxy-evaluator validation may be circular: the 400k training samples for the Llama-3.2 evaluator are not stated to be disjoint from the 100 human-evaluated Test Set 1 responses, so Figure 2 may not be an independent alignment test.","rationale":"The reader identified the self-referential GPT-4o loop as the weakest assumption, and I agree that this is the central risk. However, the more concrete and checkable version of that concern is a potential train/evaluation split overlap: the proxy evaluator is trained on GPT-4o evaluations of model responses, and the only human validation uses 100 responses from Test Set 1. If the proxy training data include Test Set 1 (or the exact 100 responses), Figure 2 is not an independent validation and the 'well-aligned with human preferences' claim collapses. The paper does not disclose the split used to create the 400k training samples, making this a missing-support issue rather than a settled defect. The reader's CONDITIONAL verdict already requires addressing such weaknesses before taking the evaluation claims at face value, so my read does not change the verdict. I would, however, add a specific condition: the authors must confirm or rule out overlap between proxy training data and the human-evaluated set, and ideally expand the human validation beyond 100 zero-shot persona QA pairs to include other QA types and training methods.","tokens_in":34125,"tokens_out":7175,"duration_ms":79828,"concrete_test":"Inspect the released code and data at https://github.com/12kimih/HiCUPID to determine which QA split (train, Test Set 1, or Test Set 2) was used to construct the 400k proxy-training samples. Specifically, check whether any of the 100 zero-shot persona responses used in the Figure 2 human evaluation appear in the proxy training file. If any do, recompute the proxy-vs-human agreement on a newly collected human evaluation of responses excluded from proxy training. If the splits are disjoint, report the inter-annotator agreement and per-QA-type (persona, schedule, multi-info) agreement to strengthen the alignment claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the Llama-3.2 proxy 'closely mirrors human preferences' (Abstract, Section 4.2) is validated in Figure 2 using 100 persona QA pairs from Test Set 1. Appendix A6 says the proxy is trained on 400k GPT-4o evaluation samples 'from the inference results in the following experimental settings,' which explicitly include zero-shot inference from GPT-4o-mini, Llama, Mistral, and Qwen. If those zero-shot responses were generated on Test Set 1 (the same split from which the 100 human-evaluated pairs are drawn), then the proxy has already seen the exact model responses and their GPT-4o labels before being compared to human judgments. The paper never specifies whether the proxy training split is disjoint from Test Set 1 or Test Set 2. Even if the proxy was trained only on the train split, the S_Llama scores in Tables 3 and 4 evaluate methods that closely resemble the training distribution, and the 100-sample human anchor with no per-model agreement metrics or confidence intervals cannot rule out systematic bias. This missing split information is load-bearing because the entire 'human-aligned' claim rests on Figure 2 being an independent test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces HiCUPID, a synthetic, GPT-4o-generated conversational benchmark for training and evaluating LLMs as personalized assistants. Each of 1,500 synthetic users is defined by 25 personas, five profile attributes, and ten schedules; the user's information is revealed naturally through dialogue histories of roughly 17k tokens on average, and per-user QA pairs probe single-info (persona, schedule) and multi-info (persona+profile) reasoning. The design is organized around five desiderata (AUI, UII, MI, LC, PR), and the paper provides two automated evaluation protocols (GPT-4o as judge, and a distilled Llama-3.2-3B proxy) anchored by a 100-item human evaluation. Extensive experiments with GPT-4o-mini, Llama-3.1-8B, Mistral-7B, and Qwen-2.5-7B under zero/few-shot prompting, BM25 and Contriever retrieval, SFT, DPO, and SFT+DPO show that SFT-based methods improve personalization, DPO alone collapses on most models, retrieval helps single-info tasks, and long context remains a bottleneck; BLEU and ROUGE scores are shown to contradict human preferences on this benchmark.","tokens_in":34438,"tokens_out":12858,"duration_ms":126812,"significance":"The paper's strengths are real and should be acknowledged explicitly: the dataset construction pipeline is detailed and reproducible (all prompt templates are included), the scale is substantial (1,500 users, roughly 60k dialogues, 40 QA pairs per user, two held-out settings), the empirical study is broad with unusually transparent hyperparameter reporting, and the authors release the dataset, code, and proxy evaluator. The central quantitative findings, such as SFT+DPO being the most reliable recipe, DPO alone being unstable, and BLEU/ROUGE ranking models differently from preference-based evaluation, are interesting and clearly presented. The load-bearing claim, however, is that the Llama-3.2 proxy 'closely mirrors human preferences' (Abstract and Section 4.2) and that GPT-4o evaluation 'closely follows' human preference (Section 4.1); as detailed in the major comments, the supporting evidence is currently too thin and ambiguous, and the validation protocol contains a potential training/validation overlap, to carry that claim as it stands.","major_comments":[{"comment":"Appendix A6 states that the 400k training samples for the Llama-3.2 proxy evaluator are taken 'from the inference results in the following experimental settings,' a list that explicitly includes the zero-shot and few-shot inference settings used to generate the model responses evaluated in Section 4.1, and Section 4.1's human study is conducted on 100 zero-shot persona QA pairs from Test Set 1. The manuscript never states whether these 100 responses (or the rest of the Test Set 1 and Test Set 2 evaluations) were excluded from the proxy's training set. If they were included, then Figure 2 compares the proxy against human judgments on inputs whose GPT-4o labels were seen during training, so it measures train-set agreement rather than human alignment, and the abstract's 'closely mirrors human preferences' claim is unsupported. This missing split information is load-bearing because the entire human-alignment claim rests on Figure 2 being an independent test; please specify the exact provenance of the 400k samples, confirm disjointness from the human-evaluated items, and re-run the Figure 2 validation on a strictly held-out set, retraining the proxy if necessary.","section":"Section 4.2, Appendix A6, Figure 2"},{"comment":"The human anchor consists of 100 persona QA pairs rated by 10 volunteer annotators per model, and the paper reports no inter-annotator agreement (e.g., no Cohen's kappa among annotators, no majority-vote consistency), no confidence intervals around the reported S scores, and no per-item agreement statistic between humans and either automated judge. The four S values in Figure 2 differ from the corresponding GPT-4o values by up to 5.3 points (GPT-4o-mini: human 25.6 versus S_GPT 29.4), while adjacent model comparisons under human evaluation (Mistral 22.8 versus Qwen 25.3) span a similar range, so without error bars the claim that GPT-4o and the proxy 'closely follow' human preferences cannot be assessed quantitatively. Moreover, only persona QA pairs are human-validated, although the proxy is claimed to be human-aligned for schedule and multi-info QA pairs as well; the paper should either extend the human validation to those QA types or restrict the alignment claim to persona QA pairs. Given that GPT-4o generated the ground-truth answers and the proxy is distilled from GPT-4o's judgments, the 100-item human comparison is the only independent anchor in the evaluation chain, so the missing agreement statistics and confidence intervals are load-bearing rather than cosmetic.","section":"Section 4.1, Figure 2"},{"comment":"The five desiderata are presented as a complete characterization of a personalized assistant without justification (Section 3), and the proactiveness desideratum (PR) is not directly operationalized: the evaluation prompts in Figure A14 score only 'Personalization' and 'Logical Validity,' and Appendix A3(e) argues that PR is satisfied because the QA pairs were 'designed' such that the LLM must provide proactive answers. This makes the PR checkmark in Table 1 an assertion about the dataset construction rather than a measured property of model responses, and it implies that the same evaluation score would be obtained even if the proactivity requirement were absent from the prompts. Please either add a direct scoring criterion or rubric excerpt for proactivity, or explicitly argue with concrete examples how the personalization criterion entails proactivity for the QA pairs in HiCUPID; otherwise the claim that HiCUPID 'properly reflects the challenges' of a personalized assistant is overstated relative to the evaluation protocol.","section":"Section 3, Appendix A3(e), Figure A14"}],"minor_comments":[{"comment":"The sentence explaining the total score ('the schedule score is halved to match its range with the scores of the two remaining QA pair types') is confusing as written, since halving maps the schedule score to [0,50] while persona and multi-info scores remain in [0,100]; the intended effect is to bring the ground-truth reference score to 50 for all three QA types, and the wording should say so directly.","section":"Section 5.1"},{"comment":"In the Qwen-2.5-7B 0-shot row, the multi-info S_GPT value appears as '17' without a decimal, unlike the surrounding entries in the same table; use a consistent number format.","section":"Table 3"},{"comment":"The paragraph stating that a detailed discussion of personalization approaches 'has been moved to Appendix' does not give the appendix section number; add a pointer to Appendix A1.","section":"Section 2"},{"comment":"Several bibliography entries lack publication years (for example, the LoRA reference by Hu et al. and the 'Editing personality for large language models' entry by Mao et al.); complete these entries.","section":"References"},{"comment":"The Conclusion calls HiCUPID 'the first open-source benchmark' for personalized assistants, which overstates the novelty given that LaMP and PChatbot are open resources with partially overlapping goals; consider qualifying the claim as the first long-context conversational benchmark covering all five desiderata.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the missing split information in Appendix A6: if the proxy was trained on Test Set 1 zero-shot responses, the Figure 2 validation is invalid, and that question cannot be resolved from the paper text alone. I would ask the authors to verify the split against their released code and data before the revision is considered. The dataset itself is a useful resource and the empirical findings are interesting; if the split is confirmed as disjoint and the human-alignment statistics (per-item agreement, confidence intervals, and ideally validation on additional QA types) are added, I would be comfortable moving toward acceptance. As it stands, the abstract's strongest claims outrun the evidence presented in the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the benchmark is real and useful; the evaluator-alignment claim is not yet supported. I checked the appendix after reading the stress-test note, and the concern holds: A6 says the 400k proxy-training samples come 'from the inference results' in the listed settings, but never says those responses are drawn from the train split. The 100 human-evaluated persona QA pairs are from Test Set 1. If the zero-shot and few-shot responses used for proxy training include Test Set 1 responses, Figure 2 is not an independent test of human alignment. Even if the authors intended a train-only split, they need to state it and check for leakage.\n\nWhat the paper does well: HiCUPID fills a real gap. The combination of persona, profile, and schedule information, implicit revelation through dialogue, multi-hop persona+profile questions, and a 17k-token context is genuinely more demanding than LaMP or PChatbot. The construction pipeline is detailed, the prompt templates are all in the appendix, the ablation study is broad, and the limitations section is honest about negation issues, DPO failures, and privacy risks. The dataset and code are released. That is a substantial resource.\n\nThe soft spots are in the evaluation story. First is the split ambiguity above, which is load-bearing because the central claim is that the Llama-3.2 proxy 'closely mirrors human preferences.' Second, the human anchor is thin: 10 volunteers per model, 100 persona QA pairs, no inter-annotator agreement, no confidence intervals, no correlation coefficient. The bar chart in Figure 2 looks similar across human and model judges, but that is not a measured alignment. Third, the main results in Tables 3 and 4 have no error bars; single runs make the rankings less certain.\n\nNone of this sinks the benchmark. The dataset and the experimental findings--for example, that SFT works far better than DPO for this task, and that retrieval helps but not enough--are worth having. The citation pattern is appropriate; the authors engage with the relevant personalization literature. The paper is clearly written and the thinking is coherent.\n\nWho is this for? Researchers working on personalized assistants and LLM evaluation. It deserves a serious referee. I would send it to review, but with the explicit request that the authors clarify the proxy training split, check for overlap with the human-evaluated responses, and either enlarge the human validation or report agreement metrics and confidence intervals. With those changes, the evaluation claims become credible.","headline":"A solid new personalization benchmark with a persuasive evaluation story that currently overclaims human alignment on a 100-sample anchor and an unspecified proxy-training split.","tokens_in":34933,"tokens_out":2446,"would_cite":true,"duration_ms":29421,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HiCUPID benchmark tests whether LLMs can truly personalize responses to a user.","keywords":["LLM personalization","benchmark","conversational AI","personalized assistant","LLM-as-a-judge","retrieval-augmented generation","direct preference optimization","long context"],"falsifier":"Collect a larger, pre-registered human-preference study on several hundred responses across multiple models and QA types, then compute agreement between human choices and the Llama-3.2 proxy; if agreement drops well below the reported levels or humans systematically prefer responses the proxy ranks lower, the central claim that the proxy mirrors human preferences fails. A second check: if training the same proxy architecture on judgments from a different LLM judge yields substantially different scores on the same responses, the evaluation is judge-dependent rather than human-aligned.","tokens_in":33945,"feed_emoji":"🤖","tokens_out":4000,"duration_ms":39343,"temperature":0.7,"pith_summary":"The paper introduces HiCUPID, a synthetic conversational benchmark built to test whether large language models can act as personalized assistants. The authors argue that existing personalization datasets only cover classification or shallow persona tasks, whereas a real assistant must infer user information from dialogue, combine multiple pieces of information, handle long histories, and respond proactively. HiCUPID supplies per-user personas, profiles, and schedules embedded in dialogues, plus single- and multi-info question-answer pairs. The paper further argues that a small Llama-3.2-3B evaluator trained on GPT-4o judgments scores responses in a way that closely tracks human preferences, unlike BLEU and ROUGE-L. If these claims hold, HiCUPID gives the field a standardized way to measure and improve LLM personalization.","feed_headline":"New benchmark scores how well LLMs personalize answers","feed_subtitle":"HiCUPID pairs a 1,500-user conversational dataset with a cheap Llama-based judge that tracks human preferences.","key_machinery":"The load-bearing object is the HiCUPID synthetic user model, in which each user has structured metadata (25 persona dimensions, five profile fields, ten dated schedules) and a dialogue history of roughly 15,000 to 17,000 tokens that reveals that metadata only implicitly. On top of this, the evaluation machinery is a judge pipeline: GPT-4o compares a model response against a ground-truth personalized answer with criteria of personalization and logical validity, and the resulting preferences fine-tune a Llama-3.2-3B proxy evaluator. The score S is the model win rate over ground truth plus half the tie rate, and the paper uses this score to compare zero/few-shot inference, retrieval (BM25, Contriever), SFT, DPO, and SFT+DPO.","core_discovery":"The central discovery is a benchmark, HiCUPID, whose design operationalizes five desiderata of a personalized AI assistant: adherence to user information, understanding of implicit information, reasoning from multiple information sources, long-context modeling, and proactiveness. Each synthetic user is defined by 25 personas, a five-field profile, and ten schedules; dialogues reveal this information only implicitly, and QA pairs test whether a model can retrieve and combine it. The paper shows that a GPT-4o-based judge agrees with human preference judgments on 100 zero-shot responses, and that distilling that judge into Llama-3.2-3B preserves the agreement. Under this evaluation, off-the-shelf LLMs score well below the ground-truth personalized answers, retrieval-based methods help, supervised fine-tuning helps, and DPO applied alone collapses under most settings, while SFT followed by DPO gives the largest gains.","pith_inferences":["An implication the authors leave implicit is that because HiCUPID ships a generic non-personalized answer for every QA pair, the dataset can be reused directly as preference data for reward-model training and RLHF-style alignment, not only for evaluation.","The synthetic users are generated from sampled profiles; if the method scales, fresh users can be generated on demand, which could keep the benchmark from becoming stale as models improve.","The human-anchor study covers only 100 zero-shot persona QA pairs, so a larger human study across more models and question types is needed to confirm that the Llama-3.2 proxy generalizes beyond the reported setting.","A testable extension: because the proxy is trained on GPT-4o judgments, swapping the judge for another LLM and comparing the resulting proxies would reveal how much of the measured personalization is judge-dependent."],"forward_implications":["HiCUPID can serve as the first open benchmark for training and evaluating personalized assistants across the five desiderata at once.","The distilled Llama-3.2 evaluator makes personalization scoring inexpensive enough to run at scale, supporting reward modeling or model selection during development.","The strong long-context penalty shows that models need better memory or retrieval as conversations grow beyond a few thousand tokens.","SFT+DPO's consistent gains over SFT suggest that preference tuning after grounded fine-tuning is a useful recipe for personalization.","BLEU and ROUGE-L rankings mislead when judging personalization, so human-aligned preference scores should replace or supplement them."],"supporting_citations":[{"why":"Supplies the GPT-4o model used to generate the synthetic dialogues and QA pairs and to provide human-aligned preference judgments.","marker":"Achiam et al., 2023"},{"why":"Supplies the Llama-3.1-8B and Llama-3.2-3B models used as base generation models and as the distilled proxy evaluator.","marker":"Dubey et al., 2024"},{"why":"Provides PersonaHub, the collection of 1,500 sampled individuals from which HiCUPID's user profiles are extrapolated.","marker":"Ge et al., 2024"},{"why":"LaMP is the main existing personalized generation benchmark that HiCUPID contrasts with, highlighting its lack of conversational multi-info and long-context demands.","marker":"Salemi et al., 2024b"},{"why":"DPO is the preference-optimization method evaluated in the experiments, shown to fail alone but to improve results after SFT.","marker":"Rafailov et al., 2024"},{"why":"BLEU is used as a traditional n-gram metric whose rankings contradict human preference in the paper's evaluation.","marker":"Papineni et al., 2002"},{"why":"ROUGE-L is used as the second traditional metric whose scores mislead when judging personalized responses.","marker":"Lin, 2004"}],"fun_headline_variants":["HiCUPID benchmark: LLMs lag on personalization","Personalization benchmark ranks LLMs below ground truth","New open benchmark and judge probe LLM personalization","LLMs fall short on personalized reasoning in HiCUPID","Open-source benchmark exposes LLM personalization limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that GPT-4o's judgments of personalization and logical validity reliably stand in for human preferences, since GPT-4o also generated the ground-truth answers and the proxy evaluator is trained on GPT-4o labels.","fun_headline_variants_meta":{"raw":{"variants":["HiCUPID benchmark: LLMs lag on personalization","Personalization benchmark ranks LLMs below ground truth","New open benchmark and judge probe LLM personalization","LLMs fall short on personalized reasoning in HiCUPID","Open-source benchmark exposes LLM personalization limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1599,"prompt_tokens":851,"completion_tokens":748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":670}},"tokens_in":467,"tokens_out":748,"duration_ms":8168,"temperature":1.0,"reasoning_tokens":670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:46:34.462575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a larger, pre-registered human-preference study on several hundred responses across multiple models and QA types, then compute agreement between human choices and the Llama-3.2 proxy; if agreement drops well below the reported levels or humans systematically prefer responses the proxy ranks lower, the central claim that the proxy mirrors human preferences fails. A second check: if training the same proxy architecture on judgments from a different LLM judge yields substantially different scores on the same responses, the evaluation is judge-dependent rather than human-aligned.","supporting_citations":[],"review_version":1}