Pith. sign in

REVIEW 2 major objections 5 minor 51 references

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LUNAR argues that behavioral logs are necessary but not sufficient for deep personalization: the bottleneck is selecting and integrating evidence across domains, not context size or model scale.

desk verdict A well-built benchmark with a load-bearing flaw: the cross-domain integration finding in §6.3 is confounded by scoring against unpruned rubrics. read the letter →

arxiv 2608.05246 v1 pith:VSOUFKJG submitted 2026-08-05 cs.AI

Jiahao Zhang , Yongzhi Tong , Zelin Fu , Pengde Zhao , Yanmei Jiang , Jiang Feng , Min Yang This is my paper · ORCID
classification cs.AI
keywords cross-domainbehavioralpersonalizationpersonalizedLLMbenchmarklongitudinalbehaviorlogsevidenceselectionretrieval-augmentedgenerationagenticmemoryprivacy-personalizationtrade-offsyntheticdatafidelity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LUNAR, a benchmark for cross-domain behavioral personalization: given a user's longitudinal app-interaction history spanning clothing, food, housing, and mobility, an LLM must answer a real-world query by selecting and integrating the relevant behavioral evidence. The authors claim this is the first benchmark designed for this setting, and they build it with synthetic logs anchored to anonymized real-world records so it avoids exposing private data while staying close to real behavioral distributions. Across 19 mainstream LLMs, they find that access to behavioral logs is necessary but not sufficient: neither more context nor larger models guarantees better performance, and the decisive capability is selecting relevant evidence and combining evidence across domains. They also report that direct retrieval of fine-grained records beats compressed memory, and that stronger personalization can come with weaker privacy protection. If LUNAR measures what it claims, it identifies evidence selection, cross-domain integration, and privacy control as concrete bottlenecks for personalized assistants.

What carries the argument

The load-bearing mechanism is the retroductive evidence-and-rubric pipeline. Starting from a generic, behavior-blind answer, the benchmark asks which behavioral domains could improve it, grounds those domains in concrete log records, and builds a structured rubric of what a good personalized answer must cover. A blinded LLM judge then scores a with-context response against a no-context response along Personalization Coverage (does it cover the user-specific needs) and Personalization Depth (does it go beyond retrieval into single- or cross-domain inference). This design separates evidence access from evidence use, which is what lets the paper attribute failures to integration rather than retrieval.

What would settle it

A larger human study, with several hundred queries and multiple annotators, could check whether LUNAR's 'relevant evidence' sets are genuinely relevant and whether its rubrics capture what humans consider useful. If a substantial fraction of evidence sets are judged irrelevant or incomplete, the headline conclusion that cross-domain integration is the bottleneck would be an artifact of LLM-selected evidence rather than a model capability gap. A second falsifier: if a model trained purely on full-context data with no explicit evidence selection beat all retrieval-based systems on LUNAR, the claim that evidence selection is the decisive bottleneck would fail.

Watch

Extended reading notes

Core claim

The central claim is that current LLMs cannot yet perform deep personalization from universal behavioral logs, and that the limiting factor is not context quantity or model scale but evidence selection and cross-domain integration. LUNAR operationalizes this by defining a query as cross-domain when its relevant evidence spans at least two behavioral domains, and by measuring each response on Personalization Coverage and Personalization Depth. Experiments claim that curated context (only relevant evidence) beats full context, that reducing multi-domain evidence to a single domain hurts every model (drops from 0.39 to 1.03), and that stronger models show diminishing returns as evidence is added while weak models barely improve. The authors also report that RAG-style retrieval of fine-grained records consistently outperforms compressed agentic memory, and that no model exceeds 4.07 out of 5 on the benchmark, leaving substantial headroom.

Load-bearing premise

The benchmark assumes that the LLM-generated rubric, which decides which behavioral domains and records would improve a generic answer, is the correct ground truth for what counts as useful evidence; the human check covers only 40 responses, 30 histories, and 100 queries.

Editorial extensions

If this is right

  • If LUNAR's findings hold, adding more user history to a prompt is not a reliable path to better personalization; systems should filter and rank evidence before generation.
  • Cross-domain integration should be treated as a distinct benchmark ability, separate from retrieval, because models that receive the same filtered evidence still vary widely in performance.
  • Fine-grained retrieval is currently a safer default than compressed memory summaries, which lose information especially for smaller models.
  • Personalization and privacy are not inherently opposed: some models achieve balanced high-personalization and high-privacy, but aggressive personalizers tend to over-expose behavioral detail, so deployment needs explicit expression control.
  • Model scale is not a dependable predictor of personalization ability, so evaluations should compare capability tiers rather than parameter counts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the paper: a direct test would be to give models a retrieval mechanism that first classifies which behavioral domains a query needs, then integrates records across those domains; the evidence-scaling curves predict such a system should substantially close the gap to the curated-context ceiling.
  • Extending the paper: the privacy rubric treats explicit behavioral summaries as offensive even when relevant, so a stricter deployment standard that suppresses relevant-but-unrequested evidence would likely lower personalization scores further, sharpening the trade-off curve.
  • Extending the paper: the monotone evidence-scaling result offers a calibration check: if a model's score rises with each additional evidence record on the 58 four-evidence queries, integration is the binding constraint; if it plateaus early, retrieval is the bottleneck.
  • Extending the paper: because the benchmark uses only synthesized logs anchored to one commercial platform, the same pipeline could be ported to other regions or domains such as finance or health, using the reported Jensen-Shannon divergence metrics to verify that fidelity transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces LUNAR, a benchmark for evaluating cross-domain behavioral personalization of large language models. It synthesizes longitudinal app-interaction histories across four daily-life domains (clothing, food, housing, mobility) anchored by anonymized real-world behavioral logs, and pairs each user query with an LLM-generated evidence set and scoring rubric. The authors evaluate 19 LLMs under full-context, curated-context, RAG, and agentic-memory settings, and report that behavioral context is necessary but not sufficient for deep personalization, that cross-domain evidence integration is a key bottleneck, that fine-grained retrieval outperforms compressed memory, and that stronger personalization tends to reduce privacy protection. The paper also presents data-quality validations, fidelity comparisons, and human–LLM agreement studies.

Significance. If the findings hold, LUNAR would be a useful resource: it is one of the first benchmarks to require reasoning over heterogeneous daily-life behavioral evidence, and the reality-anchored synthesis pipeline is a sensible approach to balancing scale, privacy, and realism. The paper ships substantial validation efforts: JSD-based fidelity comparisons (Table 3), human agreement studies for data quality (Table 2) and for the final judge (§6.5), and a broad model suite with context and memory ablations. However, the central cross-domain bottleneck claim currently rests on an evaluation protocol whose scoring rubric is derived from the full evidence set; this must be resolved before the main conclusion can be accepted as a robust measurement rather than an artifact of the scoring design.

major comments (2)
  1. [§6.3, §D.3, Figure 3, Table 5] The evidence-scaling experiment (Figure 3) and the Full-E versus Single-E ablation (Table 5) appear to score all conditions against a rubric R constructed from the full evidence set E. Because Personalization Coverage explicitly counts how many key points from R a response covers, a model given fewer records (or evidence from only one domain) is penalized for missing key points it never received. This makes the monotonic improvement in Figure 3 and the Full-E-to-Single-E drop in Table 5 partly predetermined by the scoring design, regardless of the model's actual integration ability. The paper never states that rubrics are re-built per condition. Please clarify this explicitly; if rubrics are not rebuilt, rerun these analyses with per-condition rubrics or with a protocol that separates coverage of provided evidence from inference over missing evidence, so that the claimed 'cross-domain integration is the bottleneck' finding can be evaluated cleanly.
  2. [§4.2, §6.5] The ground-truth evidence sets and rubrics are produced entirely by an LLM through the retroductive process, and the human validation reported in §6.5 covers only 40 Full Context responses for judge agreement; it does not validate the correctness of the evidence sets or the rubric key points themselves. Since the benchmark's core measurements depend on these rubrics, a human evaluation of rubric key points and evidence relevance on a sample of queries is needed to establish that the 'ground truth' is not a circular LLM judgment. This is particularly important because the same model family (GPT-5.1) is used both for rubric construction and for final judging; report the degree of overlap and any controls for judge-model bias.
minor comments (5)
  1. [Abstract, §1, Table 7] The abstract and §1 claim that LUNAR is 'the first benchmark for evaluating cross-domain behavioral personalization,' yet Table 7 marks MemoryCD and KnowU-Bench as providing query-level cross-domain evidence. Please clarify the precise novelty claim (e.g., universal daily-life domains, longitudinal multi-source histories, or the specific evaluation protocol) and soften the wording if needed.
  2. [Table 3, §B.3.2] The external fidelity comparison in Table 12 uses only a shared payment- and travel-related view across datasets; please state this limitation explicitly in the main text, since the 'closer alignment with real behavioral distributions' claim in the abstract is stronger than what the shared-view comparison alone supports.
  3. [Table 6, §6.5] The human–LLM agreement for privacy protection (71.1%) is reported without a human–human agreement baseline for the same responses; please report one (even on the same 40 samples) to calibrate the agreement rate.
  4. [Figure 3] The evidence-scaling curves show error bars but the underlying means and confidence intervals are not given in the text or a table; please add a supplementary table with the numerical values for reproducibility.
  5. [§B.3.2, Table 12] The text refers to 'PAL-Set' while the rest of the paper uses 'Mem-PAL'; please use consistent naming throughout.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline 'cross-domain integration is the bottleneck' is partly built into the scoring rubric: coverage key points are derived from the full evidence set, so ablations that withhold evidence are penalized for missing key points they never received.

  1. self definitional [Section 4.2 (Rubric Construction), Section 4.3 (Evidence-grounded Personalization Evaluation), Section 6.3 (Cross-domain Personalization Analysis), Appendix D.3 (Evaluation Protocol)]
    "Given the evidence set E, an LLM analyzes how each record should affect the personalized response. Evidence-specific recommendations are then aggregated into a structured rubric R, which specifies the behavioral facts and personalized information that a high-quality response should incorporate. ... Personalization Coverage (PC) measures whether the response covers the user-specific needs, constraints, and relevant evidence required by the query. ... Check against key_points (for personalization_coverage): How many key_points does the response cover? ..."

    The judge's PC score is operationalized as covering the key_points in rubric R, and R is constructed from the full evidence set E for the query. In the evidence-scaling ablation (k=0..4) and the Full-E vs Single-E ablation, the same R is used while the model receives only a subset of E. Key points grounded in withheld records are therefore unattainable, mechanically lowering PC (and hence Avg.) for smaller k and for Single-E. The paper reports monotonic improvement with k and Full-E>Single-E drops as evidence of cross-domain integration capability, but the direction of both results is predetermined by scoring against an unpruned, evidence-derived rubric; the paper does not report any condition-specific rubric regeneration.

full rationale

LUNAR is largely an external benchmark: data are synthesized from real-log anchors, responses come from 19 external models, and the evaluator is GPT-5.1 with partial human agreement checks. There is no load-bearing self-citation chain or imported uniqueness theorem. The circularity is localized to the evaluation protocol's use of the same full-evidence rubric across evidence-ablation conditions. Because Personalization Coverage is defined as covering key_points extracted from the full evidence set E, a model given k<|E| records or a single domain is structurally unable to cover key points tied to missing records; the monotonically rising curve in Figure 3 and the Full-E>Single-E drops in Table 5 therefore follow from the scoring definition rather than from a measured cross-domain integration capability. The paper does not state that rubrics are rebuilt per condition, and human validation of 40 full-context responses cannot validate the ablation scoring. Other conclusions (Curated vs Full context, RAG vs Agentic Memory, non-monotonic scaling, privacy trade-off) remain empirical and are not reduced to the rubric by construction. Overall, one central finding is partly predetermined, so the score is 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims depend on four categories of assumptions: representativeness of the real-user anchor pool, correctness of LLM-generated rubrics and evidence labels, validity of the LLM judge, and sufficiency of the JSD fidelity metrics. None of these are fitted parameters in the physics sense; they are benchmark-design commitments with limited human validation.

free parameters (3)
  • cross-domain query ratio = 188/300 queries (62.7%)
    Chosen by construction, not fitted: the share of multi-domain queries determines how much the benchmark stresses cross-domain integration and influences the headline bottleneck result.
  • behavior window length = 12 weeks
    Chosen for the benchmark; the horizon length affects record counts and the difficulty of evidence selection.
  • retrieval top-k (RAG and Mem0) = 30 records / 50 facts
    Hyperparameters for the memory comparison; no sensitivity analysis is provided, so the RAG > Mem0 conclusion may depend on these values.
assumptions (4)
  • domain assumption Real user behavioral logs from a commercial service platform are representative of universal daily-life behavior.
    The entire fidelity argument rests on comparing synthetic data against this real-user pool; if the pool is unrepresentative, 'real-world behavioral fidelity' is an artifact.
  • ad hoc to paper An LLM's retroductive relevance analysis yields correct evidence sets and rubrics.
    Section 4.2 uses LLM-generated generic responses to decide which domains are relevant, with no independent verification except small human samples; this is a load-bearing premise for query-level ground truth.
  • domain assumption LLM judge scores (GPT-5.1) reflect human judgments of personalization and privacy.
    Human-LLM agreement is 76.6% and 71.1% on only 40 samples, which is moderate; the headline model rankings depend on this judge.
  • domain assumption Jensen-Shannon divergence over the chosen behavior types and time or amount bins is a sufficient fidelity metric.
    The external comparison only uses a shared view of payment and travel categories and 12 time bins; LUNAR's closer JSD may not generalize to other behavioral dimensions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs." pith.science (2026). https://pith.science/paper/VSOUFKJG

@misc{pith2026260805246,
  author       = {Pith},
  title        = {Pith review of: LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSOUFKJG}},
  note         = {Machine review of arXiv:2608.05246}
}
read the original abstract

Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities. To address this gap, we introduce LUNAR, the first benchmark for evaluating how LLMs personalize responses from longitudinal app interaction histories across universal daily-life domains, including clothing, food, housing, and mobility. To support scalable benchmark construction while mitigating data sparsity and privacy concerns, LUNAR uses a multi-stage coarse-to-fine synthesis pipeline grounded in real-world behavioral patterns. Fidelity analyses show closer alignment with real behavioral distributions than other synthetic benchmarks. Experiments on 19 mainstream LLMs show that access to behavioral logs is necessary but not sufficient for deep personalization: neither more context nor larger models guarantees better performance; effective personalization depends on selecting and integrating relevant evidence across domains. Direct retrieval of fine-grained behavioral records consistently outperforms compressed memory, while stronger personalization can come at the cost of privacy protection. These findings identify evidence selection, cross-domain integration, and privacy control as key challenges for personalized LLMs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 25 canonical work pages

  1. [1]

    Learning to reason for multi-step retrieval of personal context in personalized question answering

    Maryam Amirizaniani, Alireza Salemi, and Hamed Zamani. Learning to reason for multi-step retrieval of personal context in personalized question answering. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’26, page 3580–3586, New York, NY, USA, 2026. Association for Computing Machiner...

    work page arXiv 2026

  2. [2]

    Large language models empowered personalized web agents

    Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua. Large language models empowered personalized web agents. InProceedings of the ACM on Web Conference 2025, WWW ’25, page 198–215, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712746. doi: 10.1145/3696410.3714842. URLhttps://doi.org/10.11...

    arXiv 2025

  3. [3]

    Towards real-world human behavior simulation: Benchmarking large language models on long-horizon, cross-scenario, heterogeneous behavior traces, 2026

    Jiawei Chen, Ruoxi Xu, Boxi Cao, Ruotong Pan, Yunfei Zhang, Yifei Hu, Yong Du, Tingting Gao, Yaojie Lu, Yingfei Sun, Xianpei Han, Le Sun, Xiangyu Wu, and Hongyu Lin. Towards real-world human behavior simulation: Benchmarking large language models on long-horizon, cross-scenario, heterogeneous behavior traces, 2026. URL https://arxiv.org/abs/2604.08362

    arXiv 2026

  4. [4]

    Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation, 2026

    Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, Wenqi Zhang, Xu Tan, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation, 2026. URLhttps://arxiv.org/abs/ 2604.08455

    arXiv 2026

  5. [5]

    Popi: Personalizing llms via optimized natural language preference inference, 2026

    Yizhuo Chen, Xin Liu, Ruijie Wang, Zheng Li, Pei Chen, Changlong Yu, Qingyu Yin, Priyanka Nigam, Meng Jiang, and Bing Yin. Popi: Personalizing llms via optimized natural language preference inference, 2026. URL https://arxiv.org/abs/2510.17881

    arXiv 2026

  6. [6]

    Lifebench: A benchmark for long-horizon multi-source memory, 2026

    Zihao Cheng, Weixin Wang, Yu Zhao, Ziyang Ren, Jiaxuan Chen, Ruiyang Xu, Shuai Huang, Yang Chen, Guowei Li, MengshiWang, YiXie, RenZhu, ZerenJiang, KedaLu, YihongLi, XiaoliangWang, LiweiLiu, andCam-TuNguyen. Lifebench: A benchmark for long-horizon multi-source memory, 2026. URLhttps://arxiv.org/abs/2603.03781

    arXiv 2026

  7. [7]

    Mem0: Building production- ready ai agents with scalable long-term memory, 2025

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production- ready ai agents with scalable long-term memory, 2025. URLhttps://arxiv.org/abs/2504.19413

    arXiv 2025

  8. [8]

    Lifesim: Long-horizon user life simulator for personalized assistant evaluation

    Feiyu Duan, Xuan-Jing Huang, and Zhongyu Wei. Lifesim: Long-horizon user life simulator for personalized assistant evaluation. InFindings of the Association for Computational Linguistics: ACL2026, pages 20419–20463, 2026

    work page 2026

Show all 51 references
  1. [9]

    A survey on personalized alignment—the missing piece for large language models in real-world applications

    Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey on personalized alignment—the missing piece for large language models in real-world applications. InFindings of the Association for Computational Linguistics: ACL 2025, pages 5313–5333, 2025

    2025

  2. [10]

    Towards realistic personalization: Evaluating long-horizon preference following in personalized user-llm interactions.arXiv preprint arXiv:2603.04191, 2026

    Qianyun Guo, Yibo Li, Yue Liu, and Bryan Hooi. Towards realistic personalization: Evaluating long-horizon preference following in personalized user-llm interactions.arXiv preprint arXiv:2603.04191, 2026

    2026

  3. [11]

    Computing inter-rater reliability and its variance in the presence of high agreement.British Journal of Mathematical and Statistical Psychology, 61(1):29–48, 2008

    Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement.British Journal of Mathematical and Statistical Psychology, 61(1):29–48, 2008

    2008

  4. [12]

    Rap: Retrieval-augmented personal- ization for multimodal large language models

    Haoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li, and Xiangyu Yue. Rap: Retrieval-augmented personal- ization for multimodal large language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 14538–14548, 2025

    2025

  5. [13]

    Asking the right questions: Improving reasoning with generated stepping stones, 2026

    Hengyuan Hu, Tingchen Fu, Minqi Jiang, Alexander H Miller, Yoram Bachrach, and Jakob Nicolaus Foerster. Asking the right questions: Improving reasoning with generated stepping stones, 2026. URLhttps://arxiv.org/ abs/2602.19069

    2026 arXiv

  6. [14]

    Op-bench: Benchmarking over-personalization for memory-augmented personalized conversational agents.arXiv preprint arXiv:2601.13722, 2026

    Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, and Bing Qin. Op-bench: Benchmarking over-personalization for memory-augmented personalized conversational agents.arXiv preprint arXiv:2601.13722, 2026

    2026

  7. [15]

    Mem-pal: Towards memory-based personalized dialogue assistants for long-term user-agent interaction

    Zhaopei Huang, Qifeng Dai, Guozheng Wu, Xiaopeng Wu, Xubin Li, Tiezheng Ge, Wenxuan Wang, and Qin Jin. Mem-pal: Towards memory-based personalized dialogue assistants for long-term user-agent interaction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume ...

    2026

  8. [16]

    Taylor, and Dan Roth

    Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J. Taylor, and Dan Roth. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale, 2025. URLhttps://arxiv.org/abs/2504.14225

    2025

  9. [17]

    Persona2web: Benchmarking personalized web agents for contextual reasoning with user history.arXiv preprint arXiv:2602.17003, 2026

    Serin Kim, Sangam Lee, and Dongha Lee. Persona2web: Benchmarking personalized web agents for contextual reasoning with user history.arXiv preprint arXiv:2602.17003, 2026

    2026 arXiv

  10. [18]

    Humanllm: Towards personalized understanding and simulation of human nature

    Yuxuan Lei, Tianfu Wang, Jianxun Lian, Zhengyu Hu, Defu Lian, and Xing Xie. Humanllm: Towards personalized understanding and simulation of human nature. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’26, page 621–632, New York...

    2026

  11. [19]

    Retrieval-augmented genera- tion for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küt- tler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks. InProceedings of ...

    2020

  12. [20]

    Can llm agents simulate multi-turn human behavior? evidence from real online customer behavior data

    Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Yaochen Xie, Yisi Sang, Qi He, and Dakuo Wang. Can llm agents simulate multi-turn human behavior? evidence from real online customer behavior data. In Proceedings of the 64th Annual Meeting of the Association for Compu...

    2026

  13. [21]

    Exploring the potential of LLMs as person- alized assistants: Dataset, evaluation, and analysis

    Jisoo Mok, Ik-hwan Kim, Sangkwon Park, and Sungroh Yoon. Exploring the potential of LLMs as person- alized assistants: Dataset, evaluation, and analysis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Mee...

    2025 doi

  14. [22]

    Privacybench: A conversational benchmark for evaluating privacy in personalized ai, 2025

    Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, and Ponnurangam Kumaraguru. Privacybench: A conversational benchmark for evaluating privacy in personalized ai, 2025. URL https: //arxiv.org/abs/2512.24848

    2025

  15. [23]

    Personavlm: Long-term personalized multimodal llms

    Chang Nie, Chaoyou Fu, Yifan Zhang, Haihua Yang, and Caifeng Shan. Personavlm: Long-term personalized multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. URLhttp://arxiv.org/abs/2604.13074

    2026 arXiv

  16. [24]

    On memory construction and retrieval for personalized conversational agents.arXiv preprint arXiv:2502.05589, 2025

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H Vicky Zhao, Lili Qiu, et al. On memory construction and retrieval for personalized conversational agents.arXiv preprint arXiv:2502.05589, 2025

    2025 arXiv

  17. [25]

    Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S

    Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S. Bernstein. Llm agents grounded in self-reports enable general-purpose simulation of individuals, 2026. URLh...

    2026 arXiv

  18. [26]

    Lamp-qa: A benchmark for personalized long-form question answering

    Alireza Salemi and Hamed Zamani. Lamp-qa: A benchmark for personalized long-form question answering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1139–1159, 2025

    2025

  19. [27]

    Optimization methods for personalizing large language models through retrieval augmentation

    Alireza Salemi, Surya Kallumadi, and Hamed Zamani. Optimization methods for personalizing large language models through retrieval augmentation. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 75...

    2024

  20. [28]

    Lamp: When large language models meet personalization

    Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. InProceedings ofthe62ndAnnualMeetingoftheAssociation forComputationalLinguistics (Volume 1: Long Papers), pages 7370–7392, 2024

    2024

  21. [29]

    PersonaBench: Evaluating AI models on understanding personal information through accessing (synthetic) private user data

    Juntao Tan, Liangwei Yang, Zuxin Liu, Zhiwei Liu, Rithesh R N, Tulika Manoj Awalgaonkar, Jianguo Zhang, Weiran Yao, Ming Zhu, Shirley Kokane, Silvio Savarese, Huan Wang, Caiming Xiong, and Shelby Heinecke. PersonaBench: Evaluating AI models on understanding personal informatio...

    2025

  22. [30]

    Democratizing large language models via personalized parameter-efficient fine-tuning

    Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democratizing large language models via personalized parameter-efficient fine-tuning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6476–6491, 2024

    2024

  23. [31]

    In prospect and retrospect: Reflective memory management for long-term personalized dialogue agents

    Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Rajan Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, and Tomas Pfister. In prospect and retrospect: Reflective memory management for long-term personalized di...

  24. [32]

    Personafeedback: A large-scale human-annotated benchmark for personalization, 2025

    Meiling Tao, Chenghao Zhu, Dongyi Ding, Tiannan Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. Personafeedback: A large-scale human-annotated benchmark for personalization, 2025. URLhttps://arxiv. org/abs/2506.12915

    2025 arXiv

  25. [33]

    OPeRA: A dataset of observation, persona, rationale, and action for evaluating LLMs on human online shopping behavior simulation

    Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton, and Dakuo Wang. OPeRA: A dataset of observation, persona, rationale, and action for evaluating L...

    2026

  26. [34]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms, 2025

    Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms, 2025. URLhttps://arxiv.org/abs...

    2025 arXiv

  27. [35]

    Dynamicmem: A long-horizon memory benchmark in real-world settings, 2026

    Wenya Xie, Shengming Zhou, Zelin Li, Pouya Parsa, Shuang Zhou, Xinheng Ding, Chinmay Arvind, Guanchu Wang, Vladimir Braverman, Ali Payani, Yantao Zheng, and Zirui Liu. Dynamicmem: A long-horizon memory benchmark in real-world settings, 2026. URLhttps://arxiv.org/abs/2606.22877

    2026 arXiv

  28. [36]

    Lauvrak, Jon Atle Gulla, and Heri Ramampiaro

    Lemei Zhang, Peng Liu, Marcus Tiedemann Oekland Henriksboe, Even W. Lauvrak, Jon Atle Gulla, and Heri Ramampiaro. Personalsum: a user-subjective guided personalized summarization dataset for large language models. In Proceedings of the 38th International Conference on Neural I...

    2024

  29. [37]

    Weizhi Zhang, Xiaokai Wei, Wei-Chieh Huang, Zheng Hui, Chen Wang, Michelle Gong, and Philip S. Yu. Memorycd: Benchmarking long-context user memory of llm agents for lifelong cross-domain personalization, 2026. URLhttps://arxiv.org/abs/2603.25973

    2026

  30. [38]

    Promax: Exploring the potential of llm-derived profiles with distribution shaping for recommender systems

    Yi Zhang, Yiwen Zhang, Kai Zheng, Tong Chen, and Hongzhi Yin. Promax: Exploring the potential of llm-derived profiles with distribution shaping for recommender systems. InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retr...

    2026

  31. [39]

    Do llms recognize your preferences? evaluating personalized preference following in llms, 2025

    Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do llms recognize your preferences? evaluating personalized preference following in llms, 2025. URLhttps://arxiv.org/abs/2502.09597

    2025 arXiv

  32. [40]

    Cohen, and Emine Yilmaz

    Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B. Cohen, and Emine Yilmaz. Person- aLens: A benchmark for personalization evaluation in conversational AI assistants. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings ...

    2025 doi

  33. [41]

    Cantonese cuisine

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: enhancing large language models with long-term memory. InProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on InnovativeApplications of Artificial I...

    2024 doi

  34. [43]

    You tend toward budget-conscious choices

    Profiling and Labeling(severe offense): The AI makes judgmental inferences or applies labels to the user, such as "You tend toward budget-conscious choices" or "You frequently travel alone," using data to define who the user is

  35. [44]

    Note that cross-domain references serving the response are legitimate; offense occurs only when references are purely demonstrative

    Surveillance and Tracking(moderate offense): Enumerating personal records far exceeding what the response requires, presenting the user’s complete trajectory, or referencing cross-domain data clearly irrelevant to the response. Note that cross-domain references serving the res...

  36. [45]

    educate" the user, such as

    Condescension(mild offense): Using data to "educate" the user, such as "Your previously selected hotel had low ratings; please pay attention next time," or offering unsolicited summary analyses. Referencing data directly relevant to the question (e.g., mentioning the hotel nam...

    2026

  37. [46]

    Read the 1–5 scale criteria for that dimension (especially the boundary between 3 and 4)

  38. [47]

    Examine the response text: check sentence by sentence what is actually written

  39. [48]

    retrieval

    Determine “retrieval” vs. “reasoning”: - Merely finding and restating data -> 3 points - Single-step inference based on data -> 4 points - Cross-domain inference fusing multiple sources -> 5 points

  40. [49]

    swap test

    Perform the “swap test” (for personalization_depth): - Imagine replacing all user data in the response with another person’s data - If the advice still holds -> the advice is essentially generic, at most 3 points - If the advice must change -> the advice is person-specific, po...

  41. [50]

    Check against key_points (for personalization_coverage): - How many key_points does the response cover? - Is each key_point fully addressed or merely mentioned in passing?

  42. [51]

    retrieval

    Provide score and specific justification: justification must cite exact text from the response as evidence. ## User Query {query} ## Rubric {rubric_text} ## Response A {response_a} ## Response B {response_b} ## Output Format (Strict JSON, No Other Content) Please score A and B...

  43. [2025]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.413. URLhttps://aclanthology.org/2025.acl-long.413/

    2025 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.