Pith. sign in

REVIEW 3 major objections 3 minor 68 references

User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that measuring an LLM by its ability to predict individual user behavior, rather than by its score on fixed benchmarks, gives a contamination-resistant and theoretically grounded estimate of generalization, and it…

desk verdict Clever framing, invalid empirics: the model distribution is reconstructed from the target, so the reported entropy curves measure rank overlap with aggregate popularity, not predictive distribution. read the letter →

arxiv 2507.05266 v1 pith:ZG4YTXHA submitted 2025-06-30 cs.CL cs.AI

Sougata Saha , Monojit Choudhury This is my paper
classification cs.CLcs.AI
keywords LLMgeneralizationuserbehaviorpredictionpersonalizationentropycross-entropyinflectionpointdatacontaminationrecommendationevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard benchmark tasks such as knowledge retrieval and reasoning cannot reliably measure LLM generalization because models may have memorized the test data. It proposes user behavior prediction—predicting what a person or group will watch, listen to, or do next—as a contamination-resistant evaluation strategy. The authors build a statistical framework in which a model's generalization capacity is read off from the gap between its predicted behavior distribution and the true behavior distribution, and they test it on movie and music recommendation datasets. The results match the framework's prediction: GPT-4o shows a lower inflection point than GPT-4o-mini and Llama, meaning it generalizes better, while Llama barely beats a random baseline. If correct, this gives the field a cheap, scalable, and hard-to-contaminate way to compare models.

What carries the argument

The machinery is a pair of entropies defined over behavior distributions. For a proxy $\delta_j$ (a demographic group or a set of past interactions), the true entropy $H(B^T_{\delta_j})$ measures how unpredictable the group's next behavior is, while the cross-entropy $\hat{H}(B^T_{\delta_j})$ measures how well the model's predicted distribution approximates the true one. The generalization score is the expected gap $\hat{H}-H$, and the model's limit is located at the inflection point of the $H$ vs.\ $\hat{H}$ curve: the entropy level below which the cross-entropy rises instead of tracking the true entropy. A lower inflection point means the model continues to generalize to finer-grained user groups, which the paper equates with stronger generalization.

What would settle it

Run the same prompts but ask the model to output a confidence score or log-probability for every candidate, or sample many times to estimate the empirical distribution; if the relative inflection points of GPT-4o and GPT-4o-mini change, the reported ranking is an artifact of the imposed ground distribution.

Watch

Extended reading notes

Core claim

The paper's central claim is that the ability of an LLM to predict individual and group user behavior is a theoretically principled measure of its generalization, and that this measure can be operationalized as an entropy comparison. For any proxy that defines a user group, one can compute the true entropy $H(B^T_{\delta_j})$ of the behavior distribution and the model's cross-entropy $\hat{H}(B^T_{\delta_j})$ against the same distribution; the smaller the expected gap, the better the model generalizes. Plotting $\hat{H}$ against $H$ for proxies of decreasing specificity, the curve follows the ideal line for large entropy and then bends at an inflection point where the model can no longer capture individual preferences. The paper reports that GPT-4o has the lowest inflection point on both movie and music tasks, GPT-4o-mini is close behind, and Llama-3.1-8B-Instruct is only slightly better than random, which the authors take as evidence that the framework ranks models consistently with their practical capabilities.

Load-bearing premise

The evaluation assumes that a model's top-10 ranked list fully encodes its probability distribution over the 50 candidates, because it imposes the ground distribution onto the list to compute cross-entropy; if that imposed distribution does not reflect the model's real confidence, the entropy curves and inflection points are artifacts.

Editorial extensions

If this is right

  • If correct, any deployed LLM's generalization can be compared without worrying about data contamination, because user preferences shift over time and are inherently dynamic.
  • The framework repurposes existing recommendation datasets, so evaluation cost drops to roughly tens of dollars (about 192 USD for the GPT experiments here) instead of building new benchmark suites.
  • The framework predicts that a model's cross-entropy will increase as proxies become more specific, and that combining demography with history lowers the inflection point relative to history alone; both are observed in the data.
  • Generalization rankings produced this way are consistent with the practical quality ordering: GPT-4o outperforms GPT-4o-mini, which outperforms Llama, on both movie and music tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The entropy-difference measure could be applied to any domain with logged user interactions, not just movies and music; a natural extension is web search or news recommendation, where 'behavior' is more diverse and the proxies are implicit (e.g., device, time-of-day).
  • The inflection point might serve as a single-number summary of a model's personalization capacity, allowing apples-to-apples comparisons across models with different architectures and training data.
  • Because the evaluation is cheap, it could be run continuously on live traffic, giving a contamination-resistant monitoring signal that would catch degradation from fine-tuning or alignment updates.
  • The framework's reliance on demographic proxies raises the question of whether models that perform well on broad groups but poorly on small groups are truly generalizing or just encoding population-level stereotypes; the paper warns against this reading, but the entropy measure itself cannot distinguish the two.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper argues that LLM generalization should be evaluated through user behavior prediction (personalization) rather than task-centric benchmarks, because human-generated training data makes LLMs implicit behavior predictors and task benchmarks are vulnerable to contamination. It formalizes a framework in which generalization capacity is inversely proportional to the expected difference between the cross-entropy of a model's predictions and the true entropy of behavior, and hypothesizes that a lower inflection point on a target-entropy vs. cross-entropy curve indicates better generalization. The authors test the framework on MovieLens and last.fm datasets with GPT-4o, GPT-4o-mini, Llama-3.1-8B-Instruct, and a random baseline, using prompts that combine demographic and history information. They report that GPT-4o has the lowest inflection points, followed by GPT-4o-mini, Llama, and the random baseline, and interpret this as evidence for the framework.

Significance. The proposal is original and potentially valuable: if properly validated, it could provide a contamination-resistant, dynamic, and low-cost evaluation method that leverages existing datasets. The paper's explicit formalization (Eqs. 1-2), candidate sampling algorithm, and cost transparency are strengths. However, the current empirical support is invalid because the model's distribution is imposed from the ground truth rather than measured, and the inflection-point comparisons lack statistical grounding; as a result, the manuscript's central empirical claim is not yet supported.

major comments (3)
  1. [§4.2 Evaluation] The approximation described in the Evaluation paragraph of Section 4.2 constructs the model's prediction distribution q by taking the target distribution p, sorting its probabilities in descending order, and assigning the top 10 values to the model's returned list. This makes q a permutation of p, so the cross-entropy \(\hat H\) in Eq. (2) measures only how well the model's ranking aligns with the aggregate frequency distribution; the model's actual confidence or log-probabilities are discarded. The paper's claim that the experiments validate the framework and the Section 3.3 hypothesis is therefore not supported by Figures 2-7: any ranked model would produce such curves, and the inflection points are functions of p and the rank overlap, not of the model's predictive distribution. The 'optimistic estimate' caveat does not resolve this, because the resulting quantity is not an estimate of the model's distribution at all. To salvage the central claim, the authors need to measure or elicit the model's actual probabilities for the 50 candidates, or explicitly reposition the contribution as a ranking-alignment evaluation rather than a validation of the entropy-based generalization measure.
  2. [§4.2 Results and Observations] The inflection-point analysis has no uncertainty quantification. The curves are produced by binning target entropy into 200 bins, applying a rolling average with window 30, and fitting an order-4 polynomial; the reported differences in inflection points across models are given without confidence intervals, bootstrap resampling, or sensitivity checks over these hyperparameters. Several settings in Table 1 have very few examples (e.g., Movie All history 20 has 2 prompts, Movie Occupation history 20 has 2, Music No Proxy history 20 has 11), so the smoothed entropy estimates are likely noisy. Without error bars or stability analysis, the claimed ordering GPT-4o < GPT-4o-mini < Llama ≈ random is not established quantitatively.
  3. [§3.3 Hypothesis] The paper does not establish that the inflection point is the right statistic for the proposed measure of generalization. Section 3.2 defines generalization capacity in terms of the expected difference \(\hat H - H\), but the empirical analysis instead fits smoothed curves and extracts the point at which the curve deviates from the X=Y line; no derivation connects the inflection point to the quantity in Eq. (2), and no independent criterion validates the inflection-point ranking. Because the same curve is used both to define the phenomenon and to test the hypothesis, the 'prediction' that better models have lower inflection points is not a falsifiable test of the framework; the authors should either derive the inflection point from the definition or validate it against a held-out measure of generalization.
minor comments (3)
  1. [§2.2] In the last paragraph, 'Prior to that lets introduce a formal description' should be 'let's introduce', and the phrasing 'Since most available and high-quality LLM training data are human-generated' is a sentence fragment; the authors should proofread for grammatical issues.
  2. [§Limitations] The Limitations section acknowledges possible noise in entropy estimates but does not mention the most consequential threat: the imposed distribution described in Section 4.2. This should be disclosed as a key limitation.
  3. [Throughout] The paper alternates between 'GPT-4o', 'GPT4-o', and 'GPT-4o-mini'; please standardize model names.

Circularity Check

1 steps flagged · score 6.0 of 10

Empirical test of Eq. (2) is circular: §4.2 builds the model's 'prediction distribution' by imposing the ground distribution, so the cross-entropy curves and inflection points measure rank overlap with aggregate popularity, not the model's predictive distribution.

  1. fitted input called prediction [Section 4.2, Evaluation (paragraph beginning 'Since we only prompt the model...')]
    "Since we only prompt the model to generate a ranked list of 10 items, we approximate the prediction distribution over 50 items by imposing the ground distribution. We sort the target probabilities in descending order and assign the top 10 probability scores to the model’s prediction. The remaining 40 items, which are not in the model’s prediction, are sorted in descending order of their target probabilities and assigned the remainder of the target probabilities. Thus providing an optimistic estimate of the model predictions."

    In Eq. (2), the cross-entropy Ĥ(B^T_{δ_j}) = -Σ p(b_i|δ_j) log p(b_i|δ_j,θ) requires the model's predicted distribution p(·|δ_j,θ). The Evaluation paragraph replaces that distribution with a permutation of the target p: the largest p values are assigned to the model's top-10 items and the remaining p values to the rest in target order. Hence the computed cross-entropy is H(p) + KL(p || perm(p)), a rank-alignment score with aggregate popularity, not a measurement of the model's predictive distribution. Figures 2–3 and the fitted fourth-order polynomial 'inflection points' are therefore functions of p and the model's ranking; any ranked model yields some curve, and the Section 3.3 'prediction' is not an independent measurement of generalization.

full rationale

The paper's theoretical definition (Eqs. 1–2) is not itself circular: it defines generalization in terms of a cross-entropy gap, and the hypothesis about inflection points is a plausible additional claim. The circularity enters in the operationalization. §4.2 constructs the model's prediction distribution q by re-imposing the target distribution p, so the reported H(p,q) is a deterministic function of p and the model's top-10 list. Consequently, the entropy curves, rolling averages, and polynomial inflection points are descriptive statistics of rank overlap with aggregate popularity, not measurements of the model's predictive distribution. The abstract's statement that 'results align with our framework's predictions' is therefore a summary of a score that was partly built from the target distribution itself. The self-citations (Adilazuarda et al. 2024; Mukherjee et al. 2024) are not load-bearing, and no uniqueness theorem is invoked, so this is not a self-citation circularity. Because the model rankings themselves are genuine model outputs, there is some independent signal (e.g., GPT-4o's ranking aligns better with aggregate popularity than Llama's), so the paper is not wholly circular. However, the central empirical validation of the claimed framework reduces by construction, warranting a score of 6.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The framework rests on a few domain assumptions about stochastic user behavior and meaningful demographic proxies, plus an ad hoc definition of generalization in terms of cross-entropy. Several experimental smoothing parameters are free choices. No new entities are introduced.

free parameters (6)
  • Smoothing window length = 30
    Rolling average window for cross-entropy curves; hand-chosen without sensitivity analysis.
  • Target entropy bin count = 200
    Number of bins for averaging cross-entropy; arbitrary choice affecting curve shape.
  • Polynomial regression order = 4
    Order of polynomial fit used to visualize and extract the inflection point; no justification or robustness check.
  • Candidate list size (K) = 50
    Number of candidate items in each prompt; chosen without analysis of sensitivity.
  • Recommendation list size (N) = 10
    Number of items the model must recommend; fixed experimental setting.
  • History overlap threshold = 60%
    Threshold for defining user subgroup in Setup B; no justification for this specific value.
assumptions (5)
  • domain assumption User behavior is stochastic and can be represented as probability distributions over behaviors.
    Stated in Limitations and Ethical Implications; necessary for defining true entropy and cross-entropy.
  • domain assumption Demographic proxies are meaningful latent variables that predict user behavior.
    Used throughout the experiments; the average case of generalization depends on this assumption.
  • domain assumption LLMs are effectively trained on user behavior prediction because their pretraining data is human-generated behavior data.
    Section 2.1 argues this to justify evaluating LLMs on behavior prediction tasks.
  • domain assumption The plug-in frequency estimate from the dataset approximates the true behavior distribution.
    True entropy is computed from observed frequencies; the paper notes this may have large noise.
  • ad hoc to paper The cross-entropy gap between model and true distributions is a valid measure of generalization.
    Section 3.2 defines generalization in these terms without formal derivation or external validation against existing generalization measures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs." pith.science (2026). https://pith.science/paper/ZG4YTXHA

@misc{pith2026250705266,
  author       = {Pith},
  title        = {Pith review of: User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZG4YTXHA}},
  note         = {Machine review of arXiv:2507.05266}
}
read the original abstract

Measuring the generalization ability of Large Language Models (LLMs) is challenging due to data contamination. As models grow and computation becomes cheaper, ensuring tasks and test cases are unseen during training phases will become nearly impossible. We argue that knowledge-retrieval and reasoning tasks are not ideal for measuring generalization, as LLMs are not trained for specific tasks. Instead, we propose user behavior prediction, also a key aspect of personalization, as a theoretically sound, scalable, and robust alternative. We introduce a novel framework for this approach and test it on movie and music recommendation datasets for GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. Results align with our framework's predictions, showing GPT-4o outperforms GPT-4o-mini and Llama, though all models have much room for improvement, especially Llama.

Figures

Figures reproduced from arXiv: 2507.05266 by the authors.

Figure 1
Figure 1. (Left) Hypothesized Behavior. (Right) Hy [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Movie Entropy Trend. Setup A: Left, B: Middle, C: Right. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Music Entropy Trend. Setup A: Left, B: Middle, C: Right. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: GPT-4o Demographic Proxy and History-wise distributions for Movie (Left 2) and Music (Right 2) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: GPT-4o-mini Demographic Proxy and History-wise distributions for Movie (Left 2) and Music (Right 2) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Llama Demographic Proxy and History-wise distributions for Movie (Left 2) and Music (Right 2) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Random Demographic Proxy and History-wise distributions for Movie (Left 2) and Music (Right 2) [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: GPT-4o detailed plot for Movie [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: GPT-4o-mini detailed plot for Movie [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Llama detailed plot for Movie [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Random detailed plot for Movie [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: GPT-4o detailed plot for Music [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: GPT-4o-mini detailed plot for Music [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Llama detailed plot for Music [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Random detailed plot for Music [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 39 canonical work pages

  1. [1]

    Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B Divya. 2025. Agentic ai: Autonomous intelligence for complex goals--a comprehensive survey. IEEE Access

    2025

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O ' Neill, Ashutosh Modi, and Monojit Choudhury. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.882 Towards measuring and modeling culture in LLM s: A survey . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    doi:10.18653/v1/2024.emnlp-main.882 2024

  4. [4]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. https://doi.org/10.18653/v1/2024.acl-long.671 Investigating cultural alignment of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404--12422, Bangkok, Thailand. Association for Compu...

    doi:10.18653/v1/2024.acl-long.671 2024

  5. [5]

    Mary Bazire and Patrick Br \'e zillon. 2005. Understanding context before using it. In Modeling and Using Context: 5thInternational and Interdisciplinary Conference CONTEXT 2005, Paris, France, July 5-8, 2005. Proceedings 5, pages 29--40. Springer

    work page 2005

  6. [6]

    Christopher M Bishop and Nasser M Nasrabadi. 2006. Pattern recognition and machine learning, volume 4. Springer

    work page 2006

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  8. [8]

    Erion C ano, Maurizio Morisio, et al. 2017. Music mood dataset creation based on last. fm tags. In 2017 International Conference on Artificial Intelligence and Applications, Vienna, Austria, pages 15--26

    work page 2017

Show all 68 references
  1. [9]

    Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.7 Assessing cross-cultural alignment between C hat GPT and human societies: An empirical study . In Proceedings of the First Workshop on Cross-Cultur...

    2023 doi

  2. [10]

    O. Celma. 2010. Music Recommendation and Discovery in the Long Tail . Springer

    2010

  3. [11]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  4. [12]

    Ya-Xi Chen, Sebastian Boring, and Andreas Butz. 2010. How last. fm illustrates the musical world: user behavior and relevant user-generated content. In Proceedings of the international workshop on Visual Interfaces to the Social and Semantic Web, pages 1203--1204

    2010

  5. [13]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025. https://arxiv.org/abs/2501.17161 Sft memorizes, rl generalizes: A comparative study of foundation model post-training . Preprint, arXiv:2501.17161

    2025 arXiv

  6. [14]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . ...

    2021 arXiv

  7. [15]

    Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering chatgpt’s capabilities in recommender systems. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1126--1132

    2023

  8. [16]

    Ravi Dhar and Klaus Wertenbroch. 2000. Consumer choice between hedonic and utilitarian goods. Journal of marketing research, 37(1):60--71

    2000

  9. [17]

    Kevin Droe. 2006. Music preference and music education: A review of literature. Update: Applications of Research in Music Education, 24(2):23--32

    2006

  10. [18]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  11. [19]

    Haiyan Fan and Marshall Scott Poole. 2006. What is personalization? perspectives on the design and implementation of personalization in information systems. Journal of Organizational Computing and Electronic Commerce, 16(3-4):179--202

    2006

  12. [20]

    Ian Goodfellow. 2016. Deep learning, volume 196. MIT press

    2016

  13. [21]

    Mahesh Goyani and Neha Chaurasiya. 2020. A review of movie recommendation system: Limitations, survey and challenges. ELCVIA: electronic letters on computer vision and image analysis, 19(3):0018--37

    2020

  14. [22]

    Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736

    2023 arXiv

  15. [23]

    Gregory P Hanley, Brian A Iwata, and Eileen M Roscoe. 2006. Some determinants of changes in preference over time. Journal of Applied Behavior Analysis, 39(2):189--202

    2006

  16. [24]

    F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1--19

    2015

  17. [25]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. https://doi.org/10.18653/v1/2024.acl-long.211 O lympiad B ench: A challenging benchmark for promoting AG...

    2024 doi

  18. [26]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  19. [27]

    EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023. Aligning language models to user opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5906--5919

    2023

  20. [28]

    Akshita Jha, Aida Mostafazadeh Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. https://doi.org/10.18653/v1/2023.acl-long.548 S ee GULL : A stereotype benchmark with broad geo-cultural coverage leveraging generative models . In Proceedings of...

    2023 doi

  21. [29]

    Won Kim. 2002. Personalization: Definition, status, and challenges ahead. Journal of object technology, 1(1):29--40

    2002

  22. [30]

    Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi. 2022. A survey of recommendation systems: recommendation models, techniques, and application fields. Electronics, 11(1):141

    2022

  23. [31]

    Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. https://arxiv.org/abs/2307.07870 Large language models as superpositions of cultural perspectives . Preprint, arXiv:2307.07870

    2023 arXiv

  24. [32]

    Changmao Li and Jeffrey Flanigan. 2024. Task contamination: Language models may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18471--18480

    2024

  25. [33]

    Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024 a . https://openreview.net/forum?id=DbsLm2KAqP CULTURE - GEN : Revealing global cultural perception in language models through natural language prompting . In First Conference on Language Modeling

    2024

  26. [34]

    Huihan Li, Liwei Jiang, Jena D Hwang, Hyunwoo Kim, Sebastin Santy, Taylor Sorensen, Bill Yuchen Lin, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024 b . Culture-gen: Revealing global cultural perception in language models through natural language prompting. arXiv preprint arXiv:2...

    2024 arXiv

  27. [35]

    Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, et al. 2023. How can recommender systems benefit from large language models: A survey. ACM Transactions on Information Systems

    2023

  28. [36]

    Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, and Yan Zhang. 2023. Is chatgpt a good recommender? a preliminary study. arXiv preprint arXiv:2304.10149

    2023 arXiv

  29. [37]

    Celia Lury and Sophie Day. 2019. Algorithmic personalization as a mode of individuation. Theory, Culture & Society, 36(2):17--37

    2019

  30. [38]

    Sagnik Mukherjee, Muhammad Farid Adilazuarda, Sunayana Sitaram, Kalika Bali, Alham Fikri Aji, and Monojit Choudhury. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.884 Cultural conditioning or placebo? on the effectiveness of socio-demographic prompting . In Proceedings of ...

    2024 doi

  31. [39]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://doi.org/10.18653/v1/2021.acl-long.416 S tereo S et: Measuring stereotypical bias in pretrained language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inte...

    2021 doi

  32. [40]

    Paritosh Nagarnaik and A Thomas. 2015. Survey on recommendation system methods. In 2015 2nd international conference on electronics and communication systems (ICECS), pages 1603--1608. IEEE

    2015

  33. [41]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  34. [42]

    Johannes Putzke, Kai Fischbach, Detlef Schoder, and Peter A Gloor. 2014. Cross-cultural gender differences in the adoption and usage of social media platforms--an exploratory study of last. fm. Computer Networks, 75:519--530

    2014

  35. [43]

    Abhinav Sukumar Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.892 Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in LLM s . In Findings of the Ass...

    2023 doi

  36. [44]

    Everett M Rogers, Arvind Singhal, and Margaret M Quinlan. 2014. Diffusion of innovations. In An integrated approach to communication theory and research, pages 432--448. Routledge

    2014

  37. [45]

    Markus Schedl. 2016. The lfm-1b dataset for music retrieval and recommendation. In Proceedings of the 2016 ACM on international conference on multimedia retrieval, pages 103--110

    2016

  38. [46]

    Bernd Schmitt. 1999. Experiential marketing. Journal of marketing management, 15(1-3):53--67

    1999

  39. [47]

    Practices for governing agentic ai systems

    Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. Practices for governing agentic ai systems

  40. [48]

    Chenkai Sun, Jinning Li, Hou Pong Chan, ChengXiang Zhai, and Heng Ji. 2023. Measuring the effect of influential messages on varying personas. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 554--562

    2023

  41. [49]

    S Shyam Sundar and Sampada S Marathe. 2010. Personalization versus customization: The importance of agency, privacy, and power usage. Human communication research, 36(3):298--322

    2010

  42. [50]

    Hamidreza Tahmasbi, Mehrdad Jalali, and Hassan Shakeri. 2018. Modeling temporal dynamics of user preferences in movie recommendation. In 2018 8th international conference on computer and knowledge engineering (ICCKE), pages 194--199. IEEE

    2018

  43. [51]

    Kumar Tanmay, Aditi Khandelwal, Utkarsh Agarwal, and Monojit Choudhury. 2023. https://arxiv.org/abs/2309.13356 Probing the moral development of large language models through defining issues test . Preprint, arXiv:2309.13356

    2023 arXiv

  44. [52]

    B Thompson, SG Roberts, and G Lupyan. 2020. Cultural influences on word meanings revealed through large-scale semantic alignment. nature human behaviour, 4 (10), 1029--1038

    2020

  45. [53]

    Alan Turing. 1948. Intelligent machinery (1948). B. Jack Copeland, page 395

    1948

  46. [54]

    Alan Turing. 1950. Machinery and intelligence. Mind: A Quarterly Review of Psychology and Philosophy, 59(236):433--460

    1950

  47. [55]

    Jari Vesanen. 2007. What is personalization? a conceptual framework. European Journal of Marketing, 41(5/6):409--418

    2007

  48. [56]

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: will we run out of data? limits of llm scaling based on human-generated data. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  49. [57]

    Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.04325, 1

    2022 arXiv

  50. [58]

    Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, and Kai-Wei Chang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.648 Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems . In Findings of the Association for Computational Li...

    2023 doi

  51. [59]

    Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  52. [60]

    Brooke Wilken, Yuri Miyamoto, and Yukiko Uchida. 2011. Cultural influences on preference consistency: Consistency at the individual and collective levels. Journal of Consumer Psychology, 21(3):346--353

    2011

  53. [61]

    Ludwig Wittgenstein. 1953. Philosophical Investigations. Basil Blackwell, Oxford

    1953

  54. [62]

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. 2024. A survey on large language models for recommendation. World Wide Web, 27(5):60

    2024

  55. [63]

    Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. 2024. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027

    2024 arXiv

  56. [64]

    Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, et al. 2024. Recommender systems in the era of large language models (llms). IEEE Transactions on Knowledge and Data Engineering

    2024

  57. [65]

    Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. https://api.semanticscholar.org/CorpusID:265019021 Don't make your llm an evaluation benchmark cheater . ArXiv, abs/2311.01964

    2023 arXiv

  58. [66]

    Andreas Zimmermann, Andreas Lorenz, and Reinhard Oppermann. 2007. An operational definition of context. In Modeling and Using Context: 6th International and Interdisciplinary Conference, CONTEXT 2007, Roskilde, Denmark, August 20-24, 2007. Proceedings 6, pages 558--571. Springer

    2007

  59. [67]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  60. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.