REVIEW 4 major objections 6 minor 21 references
CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper introduces CRS Arena, a crowdsourced platform where users converse with two anonymous conversational recommender systems and vote a winner, and reports that open and closed crowdsourcing produce strongly correlated system…
desk verdict Useful resource paper: the released dataset is the real contribution, but the robustness claim needs more stats before it is treated as established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the side-by-side battle: a user converses with two anonymous CRSs one after the other, gives each an explicit satisfaction or frustration signal, and then votes for a winner or a draw. A battle manager pairs systems so that the two with the fewest recorded conversations face each other, keeping comparisons balanced. Battle outcomes are aggregated into an Elo rating system, a standard way of converting pairwise wins and losses into a scalar strength score, here initialized at 1000 with a K-factor of 16. The platform's open architecture lets new systems join by implementing a single response-generation method, so the same machinery can grow into a community leaderboard. The accompanying dataset, CRSArena-Dial, stores the 474 conversations together with user feedback, making the pairwise preferences and conversation text the reusable evidence for later analysis.
What would settle it
Split the closed-environment battles into two random halves and compute Elo ratings for each half; if the two resulting rankings correlate weakly or reorder substantially, the assumption that 185 battles give stable ratings is contradicted. Alternatively, double the battle count and check whether the Spearman correlation between open and closed rankings stays near 0.700.
Extended reading notes
Core claim
The discovery the paper reports is that human pairwise preference data collected through an arena-style platform can rank conversational recommender systems in a way that is consistent across different crowdsourcing environments and yet diverges from conventional recall-based evaluation. With nine CRSs, the platform collected 474 conversations (254 open, 220 closed) with satisfaction/frustration feedback and 185 battle outcomes. Elo ratings computed with initial rating 1000 and K-factor 16 agree strongly with users' expressed satisfaction (Spearman $\rho=0.917$) and correlate strongly between open and closed setups (Spearman $\rho=0.700$ for Elo, $\rho=0.726$ for satisfaction), while correlating negatively with the previously reported recall@10 (Spearman $\rho=-0.238$). The conversation statistics, including utterances per dialogue, words per utterance, and Distinct-2 diversity, are also similar across both environments. The paper reads this as evidence that CRS Arena is robust to the choice of crowdsourcing setup and that holistic, user-facing evaluation captures a different picture of system quality than offline recommendation metrics.
Load-bearing premise
The ranking is computed from only 185 pairwise battles across nine systems, about twenty battles per system, so the central robustness claim depends on these few battles producing Elo ratings stable enough for the observed open-closed correlations to mean anything.
Editorial extensions
If this is right
- If CRS Arena is robust across open and closed crowdsourcing, then closed, paid crowd sessions can be used for fast evaluation of new systems, since a few hours of concentrated data collection gives rankings comparable to weeks of open traffic.
- If the Elo ranking is a truer reflection of user experience than recall@10, then systems optimized purely for recommendation accuracy may be overestimated by offline evaluation.
- The low satisfaction rates across all nine systems imply that current conversational recommenders, at least in this arena setting, do not yet reliably meet users' information needs.
- Release of CRSArena-Dial gives other researchers a real-user conversation corpus, distinct from Wizard-of-Oz-style corpora, for studying natural preference expression and system failures.
Reading between the lines
- We infer that if the open-closed correlation holds as battle counts grow, the most economical benchmark design would be a hybrid: open volunteer traffic for routine monitoring, and closed crowdsourcing windows when a new system needs a quick, controlled comparison.
- We infer that the negative correlation with recall@10 might shrink or shift once more battles accumulate, since with about 20 battles per system the Elo estimates are only a first snapshot rather than a settled leaderboard.
- We infer that a natural extension the authors do not develop is using the pairwise preference data to train a reward model or ranker that predicts which system a new user would prefer, turning the arena into a continuous learning loop.
- We infer that one testable consequence of the robustness claim is that the same ranking should reappear if the closed pool is drawn from a different region or language group, which would tell us whether the result is tied to English-speaking crowd workers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CRS Arena, a crowdsourced web platform for pairwise evaluation of conversational recommender systems (CRSs). Users interact with two anonymous CRSs in sequence, provide satisfaction/frustration feedback for each, and then vote for a winner or declare a draw. The authors ran the platform in an open setting and a closed Prolific-based setting, collected 474 conversations across nine CRSs, released the CRSArena-Dial dataset, and computed a preliminary Elo ranking from 185 pairwise battles. They report high correlations between open and closed environments and a negative rank correlation between the Elo ranking and the recall-based ranking reported in prior work, concluding that real-user interactive evaluation is important.
Significance. CRS Arena addresses a real evaluation gap: most CRS benchmarks are offline or non-interactive, and existing user-study frameworks are often small and rubric-based. Releasing the platform and the CRSArena-Dial dataset is a concrete contribution to the community, and the use of external human feedback avoids circularity in the evaluation. The authors are appropriately cautious in calling the ranking preliminary. The main limitation is that the analytical claims about ranking robustness and correlation are built on a very small number of battles and are reported without uncertainty quantification, so their strength is not yet established. If the robustness claim can be supported with appropriate statistical analysis, this would be a useful methodological result for CRS evaluation.
major comments (4)
- [Section 4, Table 1] The central robustness claim—that open and closed environments produce highly correlated rankings—rests on Elo ratings computed from only 185 pairwise battles (81 open and 104 closed), with no confidence intervals, significance tests, or sensitivity analysis for the Elo settings (initial rating 1000, K=16). Adjacent systems in the table differ by as little as 2–17 Elo points (e.g., BARCOR_OpenDialKG 968 vs. KBRD_OpenDialKG 966 in the open environment), so the point estimates are not distinguishable. The reported Spearman rho=0.700 between open and closed rankings could be unstable; the authors should provide bootstrap confidence intervals, a Bayesian rating model, or another uncertainty-aware analysis before claiming that the environments produce 'highly correlated rankings.'
- [Section 4, Table 1] The comparison between Elo ratings and the R@10 column mixes rankings obtained under very different protocols: Elo captures end-to-end conversational experience with noisy real users in the arena, while R@10 is an offline metric reported in Wang et al. [18] under different conditions. The reported Spearman rho=-0.238 is computed across only nine systems and is not accompanied by a significance test, so it cannot alone support the claim that good recommendation performance does not align with user satisfaction. The authors should report p-values or confidence intervals for this correlation and explicitly discuss the protocol differences that make the comparison, at best, suggestive.
- [Section 3.2] The matchmaking algorithm selects the two CRSs with the fewest recorded conversations and resolves ties randomly, making the battle order at least partly deterministic. Elo updates are order-dependent, and the reported ratings are therefore one trajectory rather than a stable estimate. The paper should quantify how the ratings and the subsequent open/closed correlations vary under different matchmaking orders or use a rating method that is invariant to battle order; otherwise the robustness claim is not supported.
- [Section 4] The open and closed setups differ not only in user population but also in instructions and collection protocol: closed workers were explicitly told to interact at least 5 times, and the collection durations differ (10 days vs. 7 hours). These differences confound the comparison of conversation characteristics in Table 2. The claim that both environments produce conversations with similar characteristics would be strengthened by repeating the analysis on length-matched subsets or by controlling for the number of user turns.
minor comments (6)
- [Table 1] The column heading 'Elo (rank)' and the duplicated rank 1 for ChatGPT_OpenDialKG and ChatGPT_ReDial in the 'Closed & open crowdsourcing' columns should be clarified with a tie-breaking rule.
- [Section 4] The paper does not report the number of unique users in the open environment or whether individual users could participate in multiple battles; repeated judgments by the same user would create dependencies that should be discussed.
- [Section 4] The recruitment channel for the open environment is not described; stating how users found the platform and whether any screening was applied would help readers assess the data quality.
- [Footnote 1] The note that Safari is not supported is presented as a footnote; since it affects the usability of the public platform, it would be better placed in the main text or in the platform's documentation.
- [Section 4] The sentence describing the R@10 values should specify the dataset and hyperparameters used in Wang et al. [18], because the arena conversations are not guaranteed to be comparable to that evaluation setup.
- [Section 3.2] There is a minor typo in the sentence introducing battles: 'battle. ' contains an extra space before the period.
Circularity Check
No significant circularity: the central ranking and robustness claims are empirical findings from collected human feedback, not fitted or self-referential derivations.
full rationale
CRS Arena's central claims are empirical. The Elo ratings are computed from 185 collected pairwise battle outcomes using stated settings (initial rating 1000, K-factor 16), and are then compared with satisfaction feedback, with the R@10 values from Wang et al. [18], and across open/closed environments. None of these quantities is defined in terms of another: battle outcomes, satisfaction buttons, and offline recall are measured independently, so the reported correlations (e.g., open/closed Spearman rho = 0.700) are observed comparisons rather than fitted parameters renamed as predictions. The only self-citations are methodological or background: Joko et al. [10] is cited for the Distinct-2 diversity measure, and Balog & Zhai [1] for user simulation. Neither is load-bearing; Distinct-2 is a standard n-gram diversity metric and the conversation-similarity claim does not rest on the citation itself. The sparse-battle concern (185 judgments, no confidence intervals) is a statistical robustness issue, not circularity. Therefore the derivation chain is self-contained and no circular step is present.
Assumptions & free parameters
free parameters (2)
- Elo initial rating =
1000
- Elo K-factor =
16
assumptions (2)
- domain assumption Pairwise preference battles can be aggregated into a transitive rating (Elo model assumptions).
- domain assumption Crowdsourced users' satisfaction feedback reflects meaningful CRS quality differences.
Cite this review
Pith. "Pith review of CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems." pith.science (2026). https://pith.science/paper/IPJ4GR7Y
@misc{pith2026241210514,
author = {Pith},
title = {Pith review of: CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/IPJ4GR7Y}},
note = {Machine review of arXiv:2412.10514}
}
read the original abstract
We introduce CRS Arena, a research platform for scalable benchmarking of Conversational Recommender Systems (CRS) based on human feedback. The platform displays pairwise battles between anonymous conversational recommender systems, where users interact with the systems one after the other before declaring either a winner or a draw. CRS Arena collects conversations and user feedback, providing a foundation for reliable evaluation and ranking of CRSs. We conduct experiments with CRS Arena on both open and closed crowdsourcing platforms, confirming that both setups produce highly correlated rankings of CRSs and conversations with similar characteristics. We release CRSArena-Dial, a dataset of 474 conversations and their corresponding user feedback, along with a preliminary ranking of the systems based on the Elo rating system. The platform is accessible at https://iai-group-crsarena.hf.space/.
Figures
Reference graph
Works this paper leans on
-
[18]
Xiaolei Wang, Xinyu Tang, Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. 2023. Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models. In Proc. of EMNLP ’23 . 10052–10065
work page 2023
-
[1]
Krisztian Balog and ChengXiang Zhai. 2024. User Simulation for Evaluating Information Access Systems. Found. Trends Inf. Retr. 18, 1-2 (2024), 1–261
work page 2024
-
[2]
Bennett, David Maxwell Chickering, and Susan T
Ben Carterette, Paul N. Bennett, David Maxwell Chickering, and Susan T. Dumais
-
[3]
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. Towards Knowledge-Based Recommender Dialog System. In Proc. of EMNLP-IJCNLP ’19 . 1803–1813
work page 2019
-
[4]
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Proc. of ICML ’24
work page 2024
- [5]
-
[6]
Arpad E Elo. 1967. The proposed uscf rating system, its development, theory, and applications. Chess life 22, 8 (1967), 242–247
work page 1967
-
[7]
Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey. AI Open 2 (2021), 100–126
2021
Show all 21 references
-
[8]
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A Survey on Conversational Recommender Systems. ACM Comput. Surv. 54, 5 (2021), 1–36
2021
-
[9]
Yucheng Jin, Li Chen, Wanling Cai, and Xianglin Zhao. 2024. CRS-Que: A User- centric Evaluation Framework for Conversational Recommender Systems. ACM Trans. Recomm. Syst. 2, 1 (2024)
2024
-
[10]
de Vries, Jeff Dalton, and Faegheh Hasibi
Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, and Faegheh Hasibi. 2024. Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search. In Proc. of SIGIR ’24. 796–806
2024
-
[11]
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019. Impor- tance of Search and Evaluation Strategies in Neural Dialogue Modeling. In Proc. of INLG ’19’. 76–87
2019
-
[12]
Raymond Li, Samira Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards Deep Conversational Recommendations. In Proc. of NIPS ’18. 9748–9758
2018
-
[13]
Ahtsham Manzoor and Dietmar Jannach. 2021. Generation-based vs. Retrieval- based Conversational Recommendation: A User-Centric Comparison. In Proc. of RecSys ’21. 515–520
2021
-
[14]
Ahtsham Manzoor and Dietmar Jannach. 2022. INFACT: An Online Human Evaluation Framework for Conversational Recommendation. In Proc. of KARS ’22. 6–11
2022
-
[15]
Ahtsham Manzoor and Dietmar Jannach. 2022. Towards retrieval-based conver- sational recommendation. Inf. Sys. 109 (2022), 102083
2022
-
[16]
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. OpenDi- alKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs. In Proc. of ACL ’19’. 845–854
2019
-
[17]
Ting-Chun Wang, Shang-Yu Su, and Yun-Nung Chen. 2022. BARCOR: To- wards A Unified Framework for Conversational Recommendation Systems. arXiv:2203.14257 [cs.CL]
2022 arXiv
-
[19]
Xiaolei Wang, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022. Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning. In Proc. of KDD ’22 . 1929–1937
2022
-
[20]
Kun Zhou, Xiaolei Wang, Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. 2021. CRSLab: An Open-Source Toolkit for Building Conversational Recommender System. In Proc. of ACL- IJCNLP ’21. 185–193
2021
-
[2008]
Here or there: preference judgments for relevance. In Proc. of ECIR ’08 . 16–27
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.