Pith. sign in

REVIEW 6 major objections 5 minor 62 references

RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models

T0 review · 6 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A large language model that role-plays a user and compares two recommendation lists can rank recommender systems consistently with AUC while separating models whose offline metrics are nearly identical.

desk verdict Pairwise LLM evaluation for recommender systems is a promising idea, but the 'better discrimination' claim rests on unvalidated separation and a 4-point correlation with the very offline metrics the paper argues are insufficient. read the letter →

arxiv 2412.11068 v1 pith:OCVSUFPT submitted 2024-12-15 cs.IR cs.AI

Zhuo Wu , Qinglin Jia , Chuhan Wu , Zhaocheng Du , Shuai Wang , Zan Wang , Zhenhua Dong This is my paper · ORCID
classification cs.IRcs.AI
keywords recommendersystemsevaluationlargelanguagemodelspairwisecomparisonLLM-as-judgeusersimulationoffline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RecSys Arena proposes a practical offline evaluation method: for each user, a large language model receives a profile description and viewing history, sees the recommendation lists produced by two recommender systems, and judges which list better serves that user. The paper claims two things: the aggregated pairwise verdicts track the ranking given by AUC, and the method can distinguish recommender systems whose AUC and nDCG are too close to call, while also scoring subjective aspects such as inspiration, transparency, and impact. This is relevant because standard offline metrics are cheap but coarse and can disagree with online A/B tests, while online tests are slow and expensive. If the claim holds, RecSys Arena gives developers a scalable way to compare candidate algorithms before committing to an online experiment.

What carries the argument

The machinery is a prompt-centred pairwise comparison protocol. The prompt combines a constructed user profile (demographic attributes plus viewing history), the two recommendation lists, definitions of six evaluation dimensions, and a chain-of-thought instruction, and asks the LLM to produce qualitative reasoning and a win/tie/lose verdict for each dimension and overall. The design choices that carry the argument are: role-play to personalise, pairwise presentation to give the model contrast, and the quantile $Q$ to aggregate verdicts into a scalar ranking comparable to AUC. The six dimensions — Accuracy, Satisfaction, Inspiration, Content Quality, Transparency, and Impact on users — are what allow the method to claim coverage of subjective quality that offline metrics cannot express.

What would settle it

Take the pairs of recommender systems the LLM ranks apart, for instance LightGCN versus SASRec on MovieLens, run a real-user A/B test on the same recommendation lists, and check whether the system the LLM prefers is also the one real users engage with more. If the LLM's finer discriminations do not match real user behavior, the method is measuring the LLM's priors rather than user preference.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLM-based pairwise evaluation is both reliable and more discriminative than traditional offline metrics. In experiments on MovieLens and MIND comparing five recommender models, GPT-4o and DeepSeek-V2.5 produced pairwise rankings whose quantile scores $Q=(N_{\mathrm{win}}+N_{\mathrm{tie}})/(N_{\mathrm{lose}}+N_{\mathrm{tie}})$ correlated strongly with AUC, and the correlation was stronger for larger models; Llama3.1-8B agreed less and declared ties more often. For pairs of models with very similar AUC, such as LightGCN and SASRec, the pairwise judgments still separated them, while absolute LLM scoring did not. The paper also reports that inspiration judgments align with the diversity metric URD and that the LLM can identify when a system ignores user features such as age and occupation, which is the evidence for its transparency sub-dimension.

Load-bearing premise

The load-bearing premise is that a user profile built from demographic fields and a short viewing history lets the LLM accurately simulate that user's preferences; if the role-play does not track real user preferences, the pairwise rankings may reflect the LLM's own biases rather than user experience.

Editorial extensions

If this is right

  • Developers can rank candidate recommender systems offline using LLM pairwise judging, without waiting for an online A/B test, because the overall rankings agree with AUC trends.
  • For two systems with near-equal AUC and nDCG, the pairwise quantile $Q$ can still pick a winner, giving an offline signal about which candidate is more likely to win online.
  • Subjective dimensions of recommendation quality that lack user labels, such as inspiration and transparency, can be evaluated by proxy and produce rankings consistent with measurable diversity and feature usage.
  • Larger LLMs are more reliable as judges: in the paper's comparisons GPT-4o and DeepSeek-V2.5 track AUC, while Llama3.1-8B shows weaker agreement and more ties.
  • Because the evaluation dimensions are defined in the prompt, the same framework can be adapted to new aspects of quality by editing the instructions, not retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, the decisive test is whether LLM pairwise verdicts predict real user choices: running the same model pairs in an A/B test and checking agreement with the LLM's preferred system would directly validate the claimed finer discrimination.
  • The role-play mechanism implies that the LLM's prior assumptions about a demographic profile do some of the work; a natural follow-up is to perturb user profiles or mask the viewing history and measure how much the pairwise ranking shifts, isolating the signal that comes from genuine user information.
  • The pairwise comparison format could be extended to active sampling — choosing which model pairs to compare based on uncertainty in the current $Q$ estimates — which would make the method cheaper at scale while preserving its discriminative power.
  • A further extension is to use the LLM's qualitative justifications per dimension to generate diagnostic reports about why one system wins, turning the evaluation output into actionable feedback for system designers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes RecSys Arena, an LLM-based pairwise evaluation framework that compares two recommender systems by having an LLM role-play a user and judge their recommendation lists across six aspects. Using MovieLens and MIND, five recommender models, and three LLMs, the authors report that LLM judgements correlate with offline metrics (AUC) and claim that pairwise evaluation provides finer discrimination than AUC/nDCG for models with similar accuracy. They also assess sub-dimensions such as inspiration and transparency with indirect strategies. The paper concludes that LLM-based pairwise evaluation is a promising offline proxy for user preference.

Significance. The problem is important: scalable, fine-grained evaluation without online experiments would be valuable. The paper contributes a concrete prompt design, a public implementation, and a comparison of relative vs absolute evaluation. The pairwise design is well motivated by prior work on LLM pairwise ranking. However, the central claims rest on weak validation: the only external anchor is correlation with AUC (n=4), and the claimed better discrimination lacks any ground truth about user preference. If validated with human or online judgments, the method could be significant; in its current form, the evidence is insufficient for the strong conclusions drawn.

major comments (6)
  1. [Section 4.1, Table 3] The Pearson correlations between Q and AUC are computed over only four model-level points per LLM/dataset. With n=4, the p-values are 0.1027, 0.0998, 0.0563, 0.0469, 0.7338, and 0.2556; only one of six cells is below 0.05. The text nonetheless calls the correlations 'strong' and uses them to conclude that LLM evaluations are 'consistent with offline metrics.' This is statistically fragile and should be supplemented with confidence intervals, permutation tests, or more model pairs.
  2. [Section 4.1, Answer to RQ1 and Figures 3, 4, 6] The central claim that 'LLM-based pair-wise evaluations provide better discrimination' is not supported by any external ground truth. Visual separation of Q values for models with similar AUC/nDCG does not demonstrate that the LLM identifies which model is better for users; it could reflect LLM priors, popularity, or genre biases. To support this claim, the authors need to compare LLM pair-wise choices with online A/B test results, human preference judgments, or held-out behavioral outcomes. Without such a signal, the better-discrimination conclusion is unverified.
  3. [Section 3.1 and 4.2] The paper explicitly acknowledges the absence of subjective user labels for sub-dimensions. The indirect validations are inadequate: for inspiration (Table 4), the URD values used as reference differ only in the fourth decimal place (0.1968, 0.1962, 0.1963, 0.1954), and no test shows these differences are meaningful; for transparency, the evidence consists of two anecdotal examples (Figure 5). The claim that the six aspects reflect user experience (Section 2.2) therefore rests on an admitted missing-support point.
  4. [Section 4.3, Figure 6] The comparison of relative vs absolute evaluation uses only GPT-4o on MovieLens, without statistical tests or normalized-score details. The claim that relative evaluation 'provides better differentiation' is based on visual spread. Additionally, the Q values and absolute scores are normalized arbitrarily; without explaining the normalization and reporting confidence intervals, this comparison cannot be evaluated.
  5. [Section 2.4, Eq. (2)] The Q metric treats ties as balancing wins and losses, but tie rates vary widely across LLMs (e.g., Llama3.1-8B has tie percentages up to 40.7% in Table 2). Because Q shrinks toward 1 when ties dominate, the rankings of Llama3.1-8B may be an artifact of its tie propensity rather than a meaningful comparison. The authors should analyze tie rates and consider a tie-aware metric or a sensitivity analysis.
  6. [Introduction and Section 4.1] The paper's motivation is that offline metrics such as AUC do not capture user preference, but the only validation of LLM Q against an external signal is correlation with AUC. If AUC is insufficient to measure user preference, then correlation with AUC cannot establish that the LLM captures what AUC misses. A positive correlation only shows partial alignment with an offline proxy. The paper needs a non-offline anchor (e.g., online experiments, human annotations) to break this circularity.
minor comments (5)
  1. [Throughout] The text contains typos and formatting issues, including 'langage' (Section 3.4), 'reaults' (Figure 6 caption), 'confindence' (Section 5.1), and a missing division sign in Eq. (2) and the URD formula in Section 3.2.
  2. [Table 3] Table 3's interpretation is misleading: the text says 'P-values are both greater than 0.01' and calls the correlation 'not significant,' but DeepSeek-V2.5 on MIND has p=0.0469, which is significant at the 0.05 level; the negative correlation for Llama3.1-8B on MovieLens is described as 'moderate' but it is negative and insignificant.
  3. [Section 4.1] The paper does not describe how the Win/Tie/Lose labels are extracted from the LLM's free-text output (e.g., parsing, error handling, prompts for structured output). This reproducibility detail is missing.
  4. [Section 3.3] Section 3.3 lists five recommender systems, but Table 2 only reports pair-wise comparisons of the four non-FM models with FM; it would be clearer to also report the direct comparisons among all pairs, especially for the 'similar AUC' experiments, where the baseline model pairs are not explicitly listed.
  5. [Section 4] The paper should report the number of test users/sessions used in the evaluation; currently, no sample size is given, which prevents assessing the stability of the Q estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction: Q is computed from LLM-based win/tie/lose counts, not fit to offline metrics; weak external validation is a correctness concern, not circularity.

full rationale

The paper's derived quantity Q (Eq. 2) is defined directly from raw LLM win/tie/lose counts, Q = (N_win + N_tie) / (N_lose + N_tie), and is not fitted to AUC, nDCG, or URD. Table 3 correlates Q with AUC after the fact, but the correlation is not a term in the construction of Q; hence the alignment result is not circular by construction. Similarly, the inspiration-dimension check against URD (Table 4) is a post-hoc comparison, not an input used to define the LLM judgments. The only self-citations are to the MIND dataset [50], the NRMS model [49], and an earlier news-recommendation model [3]; these supply experimental data and baseline models, not the paper's validation criterion, so they are not load-bearing circular support. The paper itself acknowledges the key missing ground truth: 'Due to the absence of subjective user labels ... we are unable to directly assess the effectiveness of LLMs in evaluating sub-dimensions' (Section 4.2), and Table 3's p-values are weak or non-significant (e.g., GPT-4o p=0.1027/0.0998; Llama3.1-8B p=0.7338/0.2556). These are external-validity and statistical-inference weaknesses, not circularity: the strongest claim that LLM pairwise evaluation 'can more effectively identify subtle differences in performance that offline metrics might not capture' (Section 4.1) is under-supported because separation alone is not identification, but the claim does not reduce by definition or by fitted equivalence to the paper's input metrics. No formal circular step can be exhibited from the paper's equations or citation chain, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an unvalidated role-play assumption and on using the very offline metrics the paper argues are insufficient as validation ground truth. No new physical entities are introduced. The only hand-chosen number that materially shapes the evaluation is the set of six evaluation dimensions.

free parameters (1)
  • Number of evaluation aspects (6) = 6
    The six subjective dimensions (Accuracy, Satisfaction, Inspiration, Content Quality, Transparency, Impact on users) are chosen by the authors by hand. Different dimensions would likely change LLM judgments and Q, and no sensitivity analysis is provided.
assumptions (4)
  • domain assumption LLM role-play based on a short user profile and viewing history can faithfully simulate the target user's preferences
    Used throughout prompt construction in Section 2.3; if false, all pairwise evaluations are unanchored to the real user.
  • domain assumption AUC is a valid reference standard for judging whether LLM evaluation is correct
    RQ1 in Section 3.1 and Table 3 use correlation with AUC as the main validation, despite the paper arguing AUC is insufficient.
  • domain assumption URD is a valid reference standard for the inspiration dimension
    Section 4.2.1 validates the inspiration dimension against the offline diversity metric URD.
  • domain assumption Pairwise comparison improves LLM evaluation accuracy over pointwise evaluation
    Borrowed from cited work [14]; the paper relies on this to justify the entire pairwise design in Section 2.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models." pith.science (2026). https://pith.science/paper/OCVSUFPT

@misc{pith2026241211068,
  author       = {Pith},
  title        = {Pith review of: RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCVSUFPT}},
  note         = {Machine review of arXiv:2412.11068}
}
read the original abstract

Evaluating the quality of recommender systems is critical for algorithm design and optimization. Most evaluation methods are computed based on offline metrics for quick algorithm evolution, since online experiments are usually risky and time-consuming. However, offline evaluation usually cannot fully reflect users' preference for the outcome of different recommendation algorithms, and the results may not be consistent with online A/B test. Moreover, many offline metrics such as AUC do not offer sufficient information for comparing the subtle differences between two competitive recommender systems in different aspects, which may lead to substantial performance differences in long-term online serving. Fortunately, due to the strong commonsense knowledge and role-play capability of large language models (LLMs), it is possible to obtain simulated user feedback on offline recommendation results. Motivated by the idea of LLM Chatbot Arena, in this paper we present the idea of RecSys Arena, where the recommendation results given by two different recommender systems in each session are evaluated by an LLM judger to obtain fine-grained evaluation feedback. More specifically, for each sample we use LLM to generate a user profile description based on user behavior history or off-the-shelf profile features, which is used to guide LLM to play the role of this user and evaluate the relative preference for two recommendation results generated by different models. Through extensive experiments on two recommendation datasets in different scenarios, we demonstrate that many different LLMs not only provide general evaluation results that are highly consistent with canonical offline metrics, but also provide rich insight in many subjective aspects. Moreover, it can better distinguish different algorithms with comparable performance in terms of AUC and nDCG.

Figures

Figures reproduced from arXiv: 2412.11068 by the authors.

Figure 1
Figure 1. The differences among traditional offline evaluation, real-world user evaluation, and LLM-based pair-wise evaluation [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The outline of evaluation prompt template applied [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. LLM-based pair-wise evaluation results for recom [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Two showcases of LLM-based pair-wise evaluation [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The differences among offline metric (AUC), LLM [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 39 canonical work pages

  1. [1]

    Hello GPT-4o

    2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/

    2024

  2. [2]

    Introducing Meta Llama 3: The most capable openly available LLM to date

    2024. Introducing Meta Llama 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/

    work page 2024

  3. [3]

    Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long-and short-term user repre- sentations. In Annual Meeting of the Association for Computational Linguistics . 336–345

    work page 2019

  4. [4]

    Iman Avazpour, Teerat Pitakrat, Lars Grunske, and John Grundy. 2014. Dimen- sions and metrics for evaluating recommendation systems. In Recommendation Systems in Software Engineering . Springer, 245–273

    work page 2014

  5. [5]

    Christine Bauer, Eva Zangerle, and Alan Said. 2024. Exploring the landscape of recommender systems evaluation: Practices and perspectives. ACM Transactions on Recommender Systems 2, 1 (2024), 1–31

    2024

  6. [6]

    Joeran Beel, Stefan Langer, Marcel Genzmehr, Bela Gipp, Corinna Breitinger, and Andreas Nürnberger. 2013. Research paper recommender system evaluation: a quantitative literature survey. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation . 15–22

    work page 2013

  7. [7]

    Dimitrios Bountouridis, Jaron Harambam, Mykola Makhortykh, Mónica Marrero, Nava Tintarev, and Claudia Hauff. 2019. Siren: A simulation framework for understanding the effects of recommender systems in online news environments. In Proceedings of the Conference on Fairness, Accountability, and Transparency . 150–159

    work page 2019

  8. [8]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

    arXiv 2020

Show all 62 references
  1. [9]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    2024

  2. [10]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 (2024)

    2024 arXiv

  3. [11]

    Charles LA Clarke, Gianluca Demartini, Laura Dietz, Guglielmo Faggioli, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. 4.2 HMC: A Spectrum of Human–Machine-Collaborative Relevance Judgment Frameworks. Frontiers of In...

    2023

  4. [12]

    Charles LA Clarke, Chengxi Luo, and Mark D Smucker. 2021. Evaluation mea- sures based on preference graphs. In International Conference on Research on Development in Information Retrieval . 1534–1543

    2021

  5. [13]

    Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jing- dong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise Reduction in Speech Processing (2009), 1–4

    2009

  6. [14]

    Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongx- iang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering chatgpt’s capabilities in recommender systems. In ACM Conference on Recommender Systems . 1126–1132

    2023

  7. [15]

    DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture- of-Experts Language Model. arXiv:2405.04434 [cs.CL]

    2024 arXiv

  8. [16]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  9. [17]

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475 (2024)

    2024 arXiv

  10. [18]

    Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on large language models for relevance judgment. In International Conference on Rese...

    2023

  11. [19]

    Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline a/b testing for recommender systems. In ACM International Conference on Web Search and Data Mining . 198–206

    2018

  12. [20]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction.arXiv preprint arXiv:1703.04247 (2017)

    2017 arXiv

  13. [21]

    Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems 30 (2017)

    2017

  14. [22]

    F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems 5, 4 (2015), 1–19

    2015

  15. [23]

    Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. In International Conference on Research on Development in Information Retrieval. 639–648

    2020

  16. [24]

    Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20, 4 (2002), 422–446

    2002

  17. [25]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In International Conference on Data Mining . 197–206

    2018

  18. [26]

    Jaekyeong Kim, Ilyoung Choi, and Qinglong Li. 2021. Customer satisfaction of recommender system: Examining accuracy and diversity in several types of recommendation approaches. Sustainability 13, 11 (2021), 6165

    2021

  19. [27]

    Tom Kocmi and Christian Federmann. 2023. Large language models are state-of- the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520 (2023)

    2023 arXiv

  20. [28]

    Walid Krichene and Steffen Rendle. 2020. On sampled metrics for item recom- mendation. In ACM Knowledge Discovery and Data Mining . 1748–1757

    2020

  21. [29]

    Dong Li, Ruoming Jin, Jing Gao, and Zhi Liu. 2020. On sampling top-k recommen- dation evaluation. In ACM Knowledge Discovery and Data Mining . 2114–2124

    2020

  22. [30]

    Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. 2022. Improving graph collaborative filtering with neighborhood-enriched contrastive learning. In Proceedings of the ACM web conference 2022 . 2320–2329

    2022

  23. [31]

    Charles X Ling, Jin Huang, Harry Zhang, et al. 2003. AUC: a statistically consistent and more discriminating measure than accuracy. InInternational Joint Conference on Artificial Intelligence, Vol. 3. 519–524

    2003

  24. [32]

    Hongyu Lu, Weizhi Ma, Min Zhang, Maarten de Rijke, Yiqun Liu, and Shaoping Ma. 2021. Standing in Your Shoes: External Assessments for Personalized Rec- ommender Systems. In International Conference on Research on Development in Information Retrieval. 1523–1533

    2021

  25. [33]

    Xufang Luo, Zheng Liu, Shitao Xiao, Xing Xie, and Dongsheng Li. 2022. MIND- Sim: User Simulator for News Recommenders. In International World Wide Web Conferences. 2067–2077

    2022

  26. [34]

    Sean MacAvaney and Luca Soldaini. 2023. One-shot labeling for automatic relevance estimation. In International Conference on Research on Development in Information Retrieval. 2230–2235

    2023

  27. [35]

    Andrii Maksai, Florent Garcin, and Boi Faltings. 2015. Predicting online perfor- mance of news recommender systems through richer evaluation metrics. InACM Conference on Recommender Systems . 179–186

    2015

  28. [36]

    Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He

  29. [37]

    James McInerney, Brian Brost, Praveen Chandar, Rishabh Mehrotra, and Benjamin Carterette. 2020. Counterfactual evaluation of slate recommendations with sequential reward interactions. In ACM Knowledge Discovery and Data Mining . 1779–1788

    2020

  30. [38]

    Harrie Oosterhuis, Rolf Jagerman, Zhen Qin, Xuanhui Wang, and Michael Ben- dersky. 2024. Reliable confidence intervals for information retrieval evaluation using generative ai. In ACM Knowledge Discovery and Data Mining . 2307–2317

    2024

  31. [39]

    Lijing Qin and Xiaoyan Zhu. 2013. Promoting diversity in recommendation by entropy regularizer. In International Joint Conference on Artificial Intelligence . Citeseer

    2013

  32. [40]

    Alec Radford. 2018. Improving language understanding by generative pre- training. (2018)

    2018

  33. [41]

    Steffen Rendle. 2010. Factorization machines. In International Conference on Data Mining. 995–1000

    2010

  34. [42]

    Jan Šafařík, Vojtěch Vančura, and Pavel Kordík. 2022. RepSys: Framework for Interactive Evaluation of Recommender Systems. In ACM Conference on Recom- mender Systems. 636–639

    2022

  35. [43]

    Thiago Silveira, Min Zhang, Xiao Lin, Yiqun Liu, and Shaoping Ma. 2019. How good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learning and Cybernetics 10, 5 (2019), 813–831

    2019

  36. [44]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In International Conference on Research on Development in Information Retrieval . 1930–1940

    2024

  37. [45]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345

    2024

  38. [46]

    Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In International Conference on Research on Development in Information Retrieval . 165–174

    2019

  39. [47]

    Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen

  40. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837

    2022

  41. [49]

    Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie

  42. [50]

    Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al . 2020. Mind: A large- scale dataset for news recommendation. In Annual Meeting of the Association for Computational Linguistics. 3597–3606

    2020

  43. [51]

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al . 2024. A survey on large language models for recommendation. World Wide Web 27, 5 (2024), 60

    2024

  44. [52]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023)

    2023 arXiv

  45. [53]

    Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. 2022. Contrastive learning for sequential recommendation. In IEEE International Conference on Data Engineering . 1259–1273

    2022

  46. [54]

    Eva Zangerle and Christine Bauer. 2022. Evaluating recommender systems: survey and framework. Comput. Surveys 55, 8 (2022), 1–38

    2022

  47. [55]

    Shuo Zhang and Krisztian Balog. 2020. Evaluating conversational recommender systems via user simulation. In ACM Knowledge Discovery and Data Mining . 1512–1520

    2020

  48. [56]

    Xiaoyu Zhang, Yishan Li, Jiayin Wang, Bowen Sun, Weizhi Ma, Peijie Sun, and Min Zhang. 2024. Large Language Models as Evaluators for Recommendation Explanations. arXiv preprint arXiv:2406.03248 (2024)

    2024 arXiv

  49. [57]

    Wayne Xin Zhao, Junhua Chen, Pengfei Wang, Qi Gu, and Ji-Rong Wen. 2020. Revisiting Alternative Experimental Settings for Evaluating Top-N Item Rec- ommendation Algorithms. In ACM International Conference on Information and Knowledge Management. 2329–2332

    2020

  50. [58]

    Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma, Jinyang Liu, Guohao Cai, Xi Xiao, and Rui Zhang. 2022. Bars: Towards open benchmarking for recommender systems. In International Conference on Research on Development in Information Retrieval. 2912–2923

    2022

  51. [59]

    Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen. 2005. Improving recommendation lists through topic diversification. In International World Wide Web Conferences. 22–32

    2005

  52. [2019]

    InConference on Empirical Methods in Natural Language Processing

    Neural news recommendation with multi-head self-attention. InConference on Empirical Methods in Natural Language Processing . 6389–6394

  53. [2021]

    In ACM International Conference on Information and Knowledge Management

    UltraGCN: ultra simplification of graph convolutional networks for rec- ommendation. In ACM International Conference on Information and Knowledge Management. 1253–1262. Conference’17, July 2017, Washington, DC, USA Wu et al

    2017

  54. [2023]

    arXiv preprint arXiv:2305.13112 (2023)

    Rethinking the evaluation for conversational recommendation in the era of large language models. arXiv preprint arXiv:2305.13112 (2023)

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.