REVIEW 3 major objections 6 minor 36 references
PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read PUB claims that a personality-conditioned LLM agent can generate synthetic user logs faithful enough to evaluate recommender systems offline.
desk verdict The core idea is promising but the headline fidelity results are compromised by test-set leakage; the paper needs a controlled regeneration experiment before its claims can be taken seriously. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Personality Inference Module, a prompt-guided LLM that maps behavioural statistics and item metadata to Big Five scores (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) using psycholinguistic correlates such as category entropy, review-length consistency, purchase-rhythm regularity, LIWC-22 social-reference and politeness markers, and VADER sentiment volatility. Those inferred scores are then fed to the User Behaviour Simulator, an LLM agent that selects items from a mock recommendation list, generating the synthetic sequence that is compared with real interactions. The trait scores are the mechanism that is supposed to carry fidelity: the paper argues the synthetic data mirrors real behaviour because the inferred personality captures the individual differences driving item choice.
What would settle it
Take a sample of users, have each complete a validated Big Five questionnaire, and compare the questionnaire scores with PUB's inferred scores; then run the simulator twice, once with inferred traits and once with traits randomly permuted across users. If inferred and questionnaire scores correlate poorly, or if the two simulation runs produce similar Jaccard similarity against real sequences, the personality mechanism—and with it the interpretation of the fidelity result—is falsified.
Extended reading notes
Core claim
On Amazon review data, PUB's pipeline—statistical user profiling, metadata enhancement, prompt-guided Big Five inference, and trait-conditioned interaction generation—produces sequences whose overlap with real held-out interactions averages 0.31 Jaccard similarity, with fidelity improving for heavier users and with more stable overall performance than a leading LLM-agent baseline. When the synthetic logs are split into training and test sets, the nDCG@20 ranking of Pop, MF, BPR, NeuMF, LightGCN, GRU4Rec, and SASRec closely mirrors their ranking on the real test set, which is the paper's evidence that synthetic data can evaluate recommenders. The paper also reports trait distributions (Extraversion most prevalent, Neuroticism lowest among reviewers) and trait-outcome correlations: users high in Agreeableness and Conscientiousness receive better GRU4Rec recommendations, while users high in Openness receive worse ones, interpreted as openness disrupting established behavioural patterns.
Load-bearing premise
The load-bearing premise is that the Big Five scores the LLM assigns from shopping statistics and review text are accurate measures of users' true personalities and are what actually drive their choices; the Amazon evaluation contains no ground-truth personality labels that would verify either step.
Editorial extensions
If this is right
- Offline recommender evaluation could be run on PUB-generated logs instead of a real held-out test set, with algorithm rankings largely preserved.
- Personality scores can be obtained without questionnaires, directly from logs and metadata, making trait-aware evaluation possible on existing datasets at scale.
- Users with richer interaction histories yield more faithful simulations, so evaluation with PUB is most reliable for active-user populations and should be applied cautiously to sparse histories.
- The reported trait-outcome correlations (agreeableness and conscientiousness helping, openness hurting recommendation accuracy) give recommender designers a concrete lever for diagnosing and improving performance per user group.
- Because the framework is dataset-agnostic and task-generic, the same four modules could generate synthetic Q&A responses or feedback surveys, not only purchase sequences.
Reading between the lines
- Editorial: The 0.31 Jaccard figure is measured against real held-out interactions, but those real interactions were also used to build the profile that conditions the agent, so part of the fidelity may come from the LLM echoing the same history rather than from the personality mechanism; a fair test would condition the agent on one user's traits while predicting another user's purchases.
- Editorial: A direct way to isolate the trait mechanism is an ablation that feeds the simulator random or fixed Big Five scores; if fidelity is unchanged, the personality module is not doing the causal work the interpretation assigns to it.
- Editorial: If trait conditioning is genuine, the same pipeline can generate counterfactual populations (for example, high- versus low-neuroticism copies of the same user) to audit recommender fairness and robustness without collecting any new human data.
- Editorial: The framework's trait-inference step inherits any cultural or demographic bias in LLM judgment and in the psycholinguistic lexicons, which could exaggerate or distort trait differences in non-English or non-Western shopping data; cross-dataset replication is a natural next test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PUB, an LLM-based framework that infers Big Five personality traits from user behavioral logs and item metadata, and then generates synthetic interactions conditioned on those traits. Using Amazon review data, the authors report that PUB-generated sequences achieve an average Jaccard similarity of 0.31 against real test sequences (RQ1), that recommendation algorithm performance on a synthetic test set closely mirrors performance on the real test set (RQ2), and that the inferred traits show meaningful distributions and outcome associations (RQ3/RQ4). The central claim is that PUB can serve as a scalable, controllable, high-fidelity substitute for real user data in offline recommender system evaluation.
Significance. If the fidelity and personality-validity claims were properly established, the framework would be a timely and useful contribution: it couples interpretable statistical profiles with LLM generation, covers a modular four-stage architecture, and the authors provide a public code repository. The paper also engages with a meaningful literature on LLM-based simulation and personality computing. However, the current experiments do not support the load-bearing claims. RQ1 and RQ2 are contaminated by the injection of true test items into the candidate lists, and the personality inference in RQ3/RQ4 is circular because the same LLM infers traits from the same logs it later uses to generate behavior. The underlying idea is interesting, but the evidence as presented is not sufficient to establish PUB as a reliable evaluation tool.
major comments (3)
- [§3.2.1 (RQ1)] The fidelity metric is compromised by test-set leakage. The candidate list L_t always contains i_p, which is chronologically drawn from the real test interactions P_u, alongside nine random negatives. The reported Jaccard similarity between the generated sequence S_u and P_u therefore measures how often the agent selects an injected true positive from a 10-item forced-choice list, not whether the generated sequence resembles real user behavior as a distribution. Under uniform random selection, the expected Jaccard similarity would be approximately 0.053; the reported 0.31 corresponds to a forced-choice hit rate of roughly 47%. A valid fidelity test would withhold the true future items during generation and compare the generated sequences with the real test distribution using sequence-level or distributional metrics.
- [§3.2.2 (RQ2)] The same leakage propagates into the recommender-evaluation experiment. The synthetic sequences are generated 'following the aforementioned methodology,' so at each step the candidate list contains a real test interaction from P_u. Whenever the agent selects that positive, the resulting synthetic test label is identical to a real test label. Evaluating ranking metrics such as nDCG@20 on a synthetic test set with many shared labels will naturally track the real test set, independent of the quality of personality inference or generative fidelity. The mirrored trends in Fig. 3a are therefore partly constructed by the evaluation protocol. To support the claim that synthetic logs can substitute for real data, the authors should generate sequences without revealing future test items and should compare algorithm rankings on fully held-out synthetic labels.
- [§2.3, §3.2.3–3.2.4] The personality inference is circular with respect to the paper's personality-related claims. The Big Five trait scores are produced by an LLM prompted with behavioral statistics and item metadata, and there is no ground-truth personality data or human validation anywhere in the Amazon evaluation. The same inferred traits then condition the generation of behavior in §2.4. Consequently, the trait distributions in Fig. 3b and the trait–outcome associations in Fig. 3c are self-consistent outputs of the model's priors rather than external measurements of real user personality. The paper should validate the inferred traits against established psychometric instruments or external labels, and should not interpret RQ4 as evidence of a causal relationship between personality and recommendation outcomes.
minor comments (6)
- [§3.2.3] The statement that users 'tend to be less emotionally stable' after reporting that the average Neuroticism score is significantly lower than other traits appears contradictory; a lower Neuroticism score indicates greater emotional stability.
- [§2.1, Eq. (2)] The temporal bin thresholds are described as 'determined empirically,' but no sensitivity analysis or justification is provided; since these thresholds are free parameters, their effect on the fidelity results should be tested.
- [§3.2.1] The Jaccard similarity is computed without a formal definition; the paper should state explicitly whether it is applied to sets or ordered sequences and how duplicate items are handled.
- [§3.2.2] Fig. 3a reports nDCG@20 without confidence intervals or significance tests, yet the text uses the phrase 'closely mirrors' as a quantitative claim; the authors should report variance across runs or across user groups.
- [§3.2.4] The top/bottom 10% thresholds for defining extreme performance are arbitrary; the robustness of the trait–outcome associations to these thresholds should be reported.
- [§3.1] The preprocessing filters out users and items with fewer than 20 interactions, which restricts the evaluation to heavy users; the generalizability to sparse, short-history users is not discussed.
Circularity Check
RQ1/RQ2 fidelity metrics are partially self-constructed: the real next test item is injected into every candidate list, so the reported Jaccard and mirrored nDCG reflect selection among lists containing the ground truth.
-
other
[Section 3.2.1, RQ 1, proxy recommender protocol]
"At each iteration t, this model constructs a mock recommendation list L_t for user u: L_t ={i_p,i_n1,i_n2,...,i_nk}, k = 9. Here, i_p is the positive sample, chronologically drawn from P_u ... The user agent then selects the most relevant item from L_t based on its inferred personality, forming the synthetic sequence S_u. We measure the Jaccard similarity [2] between S_u and the true interaction sequence P_u."
The metric compares S_u to P_u after P_u itself has been placed inside every candidate list the agent chooses from. Under random selection, expected Jaccard is ~0.05, so the reported 0.31 corresponds to a ~47% hit rate in a 10-way forced choice. The 'alignment' measured by Jaccard(S_u,P_u) is thus a hit-rate on lists that always contain the ground-truth item, not an independent measure of whether generated sequences resemble real user behavior as a distribution; a substantial part of the reported fidelity is guaranteed by the evaluation design.
-
other
[Section 3.2.2, RQ 2, synthetic test set generation]
"In this experiment, synthetic user behaviour sequences are generated following the aforementioned methodology and then divided into a training set, Str, and a test set, Ste. To ensure a fair comparison, the synthetic test set is matched in size to the original test data."
Since 'the aforementioned methodology' always includes i_p from P_u in the candidate list, the synthetic test set inherits real test positives whenever the agent selects them. The observed 'performance of each algorithm on the synthetic test set closely mirrors that on the real test set' is therefore partly due to label overlap between the synthetic and real test sets, rather than to the fidelity of the personality model. The RQ2 conclusion is partially constructed by the same leakage that drives RQ1.
full rationale
The central fidelity claims rest on a forced-choice protocol in which the ground-truth next item is an option; this inflates Jaccard and makes the synthetic test set share labels with the real test set. The personality inference loop (LLM infers traits from logs, then generates logs from those traits) is a construct-validity concern but is not counted as circularity here because RQ3/RQ4 are model-output analyses rather than external predictions. No self-citation is load-bearing: NEST and other baselines are used as comparators, not as proof of uniqueness. The paper does contain independent content, including external Amazon data and baseline comparisons, but the headline fidelity numbers are partially artifacts of the evaluation design, giving partial circularity.
Assumptions & free parameters
free parameters (6)
- Temporal bin thresholds in Eq. 2 =
1 week, 1 month, 1 quarter
- Sampling length eta and bin count K =
Not reported
- User/item interaction frequency threshold =
20 interactions
- Candidate list size k =
9 negative items + 1 positive
- LLM model, temperature, and prompt templates =
Not reported
- Top/bottom 10% thresholds for RQ4 =
10%
assumptions (6)
- domain assumption Big Five personality traits are a valid and sufficient representation of user personality for modelling shopping behavior.
- domain assumption Amazon review logs encode enough signal to infer Big Five traits in the directions listed in Section 2.3.
- ad hoc to paper The psycholinguistic indicators (category entropy, review length consistency, LIWC-22 social references, VADER sentiment, negative emotion volatility) map monotonically to the intended traits.
- domain assumption The LLM's item selection among a candidate list is a valid model of a user's choice behavior.
- domain assumption Jaccard similarity between selected and true sequences is a meaningful fidelity metric.
- ad hoc to paper Temporal stratified sampling with exponential bins preserves the chronological structure of behavior.
Cite this review
Pith. "Pith review of PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation." pith.science (2026). https://pith.science/paper/F5YLQTVB
@misc{pith2026250604551,
author = {Pith},
title = {Pith review of: PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/F5YLQTVB}},
note = {Machine review of arXiv:2506.04551}
}
read the original abstract
Traditional offline evaluation methods for recommender systems struggle to capture the complexity of modern platforms due to sparse behavioural signals, noisy data, and limited modelling of user personality traits. While simulation frameworks can generate synthetic data to address these gaps, existing methods fail to replicate behavioural diversity, limiting their effectiveness. To overcome these challenges, we propose the Personality-driven User Behaviour Simulator (PUB), an LLM-based simulation framework that integrates the Big Five personality traits to model personalised user behaviour. PUB dynamically infers user personality from behavioural logs (e.g., ratings, reviews) and item metadata, then generates synthetic interactions that preserve statistical fidelity to real-world data. Experiments on the Amazon review datasets show that logs generated by PUB closely align with real user behaviour and reveal meaningful associations between personality traits and recommendation outcomes. These results highlight the potential of the personality-driven simulator to advance recommender system evaluation, offering scalable, controllable, high-fidelity alternatives to resource-intensive real-world experiments.
Figures
Reference graph
Works this paper leans on
-
[1]
Danny Azucar, Davide Marengo, and Michele Settanni. 2018. Predicting the Big 5 personality traits from digital footprints on social media: A meta-analysis. Personality and individual differences 124 (2018), 150–159
work page 2018
-
[2]
Sujoy Bag, Sri Krishna Kumar, and Manoj Kumar Tiwari. 2019. An efficient rec- ommendation generation using relevant Jaccard similarity. Information Sciences 483 (2019), 53–64. https://doi.org/10.1016/j.ins.2019.01.023
-
[3]
Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W Pennebaker. 2022. The development and psychometric properties of LIWC-22.Austin, TX: University of Texas at Austin 10 (2022)
work page 2022
-
[4]
Sahraoui Dhelim, Liming Chen, Nyothiri Aung, Wenyin Zhang, and Huansheng Ning. 2023. A hybrid personality-aware recommendation system based on per- sonality traits and types models. J. Ambient Intell. Humaniz. Comput. 14, 9 (2023), 12775–12788. https://doi.org/10.1007/s12652-022-04200-5
-
[5]
Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. 2024. Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities and Social Sciences Communications 11, 1 (2024), 1–24
work page 2024
-
[6]
Lewis R Goldberg. 1981. Language and individual differences: The search for uni- versals in personality lexicons. Review of Personality and Social Psychology/Sage (1981)
work page 1981
-
[7]
Lewis R Goldberg. 1992. The development of markers for the Big-Five factor structure. Psychological assessment 4, 1 (1992), 26
work page 1992
-
[8]
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In Proceedings of the 43rd International ACM SIGIR con- ference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020 . 639–648. https://doi.or...
arXiv 2020
Show all 36 references
-
[9]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. InProceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017 . 173–
2017
-
[10]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk
-
[11]
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian J. McAuley
-
[12]
Hutto and Eric Gilbert
Clayton J. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Proceedings of the Eighth International Conference on Weblogs and Social Media, ICWSM 2014, Ann Arbor, Michigan, USA, June 1-4, 2014 . http://www.aaa...
2014
-
[13]
Eugene Ie, Chih-Wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. 2019. RecSim: A Configurable Simulation Platform for Recommender Systems. CoRR abs/1909.04847 (2019). http://arxiv. org/abs/1909.04847
2019 arXiv
-
[14]
Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023. Evaluating and Inducing Personality in Pre-trained Language Models. In Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 202...
2023
-
[15]
Wang-Cheng Kang and Julian J. McAuley. 2018. Self-Attentive Sequential Recom- mendation. InIEEE International Conference on Data Mining, ICDM 2018, Singapore, November 17-20, 2018. 197–206. https://doi.org/10.1109/ICDM.2018.00035
2018
-
[16]
Bell, and Chris Volinsky
Yehuda Koren, Robert M. Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (2009), 30–37. https: //doi.org/10.1109/MC.2009.263
2009 doi
-
[17]
Renaud Lambiotte and Michal Kosinski. 2014. Tracking the digital footprints of personality. Proc. IEEE 102, 12 (2014), 1934–1939. https://doi.org/10.1109/JPROC. 2014.2359054
2014
-
[18]
Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145–151. https://doi.org/10.1109/ 18.61115
1991
-
[19]
Chenglong Ma, Yongli Ren, Pablo Castells, and Mark Sanderson. 2022. Evaluation of herd behavior caused by population-scale concept drift in collaborative filtering. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retriev...
2022
-
[20]
Chenglong Ma, Yongli Ren, Pablo Castells, and Mark Sanderson. 2022. NEST: Simulating Pandemic-like Events for Collaborative Filtering by Modeling User Needs Evolution. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, U...
2022
-
[21]
Chenglong Ma, Yongli Ren, Pablo Castells, and Mark Sanderson. 2024. Temporal Conformity-aware Hawkes Graph Network for Recommendations. InProceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024. 3185–
2024
-
[22]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme
-
[23]
Sonia Roccas, Lilach Sagiv, Shalom H Schwartz, and Ariel Knafo. 2002. The big five personality factors and personal values. Personality and social psychology bulletin 28, 6 (2002), 789–801
2002
- [24]
-
[25]
Monika Singh. 2020. Scalability and sparsity issues in recommender datasets: a survey. Knowledge and Information Systems 62, 1 (2020), 1–43. https://doi.org/10. 1007/s10115-018-1254-2
2020
-
[26]
Aleksandra Sorokovikova, Sharwin Rezagholi, Natalia Fedorova, and Ivan Yamshchikov. 2024. LLMs Simulate Big5 Personality Traits: Further Evidence. In Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024). 83–87
2024
-
[27]
Wen-Chin Tsao and Hung-Ru Chang. 2010. Exploring the impact of personality traits on online shopping behavior. African journal of business management 4, 9 (2010), 1800
2010
-
[28]
Lei Wang, Jingsen Zhang, Hao Yang, Zhi-Yuan Chen, Jiakai Tang, Zeyu Zhang, Xu Chen, Yankai Lin, Hao Sun, Ruihua Song, et al. 2024. User Behavior Simulation with Large Language Model-based Agents for Recommender Systems. ACM Transactions on Information Systems (2024)
2024
-
[29]
Qian Wang, Jiaying Wu, Zhenheng Tang, Bingqiao Luo, Nuo Chen, Wei Chen, and Bingsheng He. 2025. What Limits LLM-based Human Simulation: LLMs or Our Design? arXiv preprint arXiv:2501.08579 (2025)
2025 arXiv
-
[30]
Ziqi Xu, Chenglong Ma, Yongli Ren, Jeffrey Chan, Wei Shao, and Feng Xia. 2025. Towards Better Evaluation of Recommendation Algorithms with Bi-directional Item Response Theory. In Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, April 28-May 2, 2...
2025
-
[31]
Guixian Zhang, Guan Yuan, Debo Cheng, Lin Liu, Jiuyong Li, Ziqi Xu, and Shichao Zhang. 2025. Deconfounding representation learning for mitigating latent confounding effects in recommendation. Knowledge and Information Systems (2025), 1–22
2025
-
[182]
https://doi.org/10.1145/3038912.3052569
-
[2009]
In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009
BPR: Bayesian Personalized Ranking from Implicit Feedback. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 . 452–461. https://www.auai.org/uai2009/ papers/UAI2009_0139_48141db02b9f0b02bc7...
2009
-
[2016]
In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings
Session-based Recommendations with Recurrent Neural Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. http://arxiv.org/abs/1511.06939
2016 arXiv
- [2024]
-
[3194]
https://doi.org/10.1145/3589334.3645354
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.