{"id":"040053bf-2d0e-47f9-89d1-5e5e4328358a","arxiv_id":"2412.10514","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper presents CRS Arena, a pairwise-battle crowdsourcing platform for benchmarking conversational recommender systems, releases a 474-conversation dataset, and reports that user-based rankings diverge from recall-based rankings.","lead":"CRS Arena is a crowdsourced platform that pits conversational recommender systems against each other in anonymous pairwise battles, where real users interact with both systems and vote for a winner. The authors release 474 real-user conversations and a preliminary Elo ranking, and report that open and closed crowdsourcing setups yield similar results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Elo ratings from only 185 battles are too sparse to support the open/closed robustness claim without uncertainty quantification; the reported rho=0.700 may not be stable.","rationale":"The paper's main contribution is a platform and dataset, which are real and released, but its analytical claim of robustness across crowdsourcing environments depends on correlation coefficients between sparse Elo rankings. The reader's weakest assumption identifies exactly this vulnerability: with 185 battles, 9 systems, and no confidence intervals, the observed rho=0.700 cannot be interpreted as strong evidence. I considered other possible concerns, such as the mismatch between 474 conversations and 185 battle outcomes, or the comparison with R@10 for a system with missing values, but those are secondary to the robustness claim. The most load-bearing issue remains the lack of uncertainty quantification on the Elo ratings themselves. A bootstrap or permutation test would directly address this and is a feasible, concrete check. Since the reader already returned a CONDITIONAL verdict, my stress-test does not change the recommendation; it reinforces the need for the stated condition. I found no internally inconsistent or fraudulent element; the paper is honest about the preliminary nature of its ranking, but the robustness conclusion overreaches relative to the statistical support. The proposed test is the minimal step that would settle whether the central claim holds.","tokens_in":6394,"tokens_out":7136,"duration_ms":70931,"concrete_test":"Perform a bootstrap sensitivity analysis: resample the 185 battle outcomes (or the 81 open and 104 closed battles separately) with replacement, recompute Elo for each resample using the same initial rating and K=16, and compute the Spearman correlation between open and closed Elo rankings. Repeat with randomly shuffled battle order within each resample to account for Elo's order dependence. Report the 95% confidence interval for rho and the rank-flip frequency; if the interval includes values near zero or negative, the 'highly correlated rankings' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central robustness claim—that open and closed crowdsourcing produce highly correlated rankings—rests on Elo ratings computed from 81 open and 104 closed pairwise judgments across 9 systems (Section 4, Table 1). With roughly 18 and 23 battles per system respectively, the Elo point estimates have large sampling error; adjacent systems differ by as little as 2–17 points (e.g., open Elo: BARCOR_OpenDialKG 968 vs. KBRD_OpenDialKG 966; combined: UniCRS_OpenDialKG 937 vs. CRB-CRS_ReDial 930). The paper reports Spearman rho=0.700 between open and closed Elo rankings but gives no confidence intervals, significance tests, or sensitivity analyses for the Elo settings (initial rating 1000, K=16). A conventional Spearman p-value treats the Elo values as observed without error, but they are themselves noisy estimates from sparse pairwise data; the true correlation between environments could be much weaker. Additionally, Elo updates are order-dependent, and the matchmaking algorithm (Section 3.2) determines a deterministic battle order, so the reported ratings are one draw from a wide distribution. Without quantifying that uncertainty, the claim that both environments produce 'highly correlated rankings' is not established. The paper itself calls the ranking 'preliminary,' which is consistent with this concern. The suggested conditional verdict is therefore appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CRS Arena, a crowdsourced web platform for pairwise evaluation of conversational recommender systems (CRSs). Users interact with two anonymous CRSs in sequence, provide satisfaction/frustration feedback for each, and then vote for a winner or declare a draw. The authors ran the platform in an open setting and a closed Prolific-based setting, collected 474 conversations across nine CRSs, released the CRSArena-Dial dataset, and computed a preliminary Elo ranking from 185 pairwise battles. They report high correlations between open and closed environments and a negative rank correlation between the Elo ranking and the recall-based ranking reported in prior work, concluding that real-user interactive evaluation is important.","tokens_in":6664,"tokens_out":3258,"duration_ms":31082,"significance":"CRS Arena addresses a real evaluation gap: most CRS benchmarks are offline or non-interactive, and existing user-study frameworks are often small and rubric-based. Releasing the platform and the CRSArena-Dial dataset is a concrete contribution to the community, and the use of external human feedback avoids circularity in the evaluation. The authors are appropriately cautious in calling the ranking preliminary. The main limitation is that the analytical claims about ranking robustness and correlation are built on a very small number of battles and are reported without uncertainty quantification, so their strength is not yet established. If the robustness claim can be supported with appropriate statistical analysis, this would be a useful methodological result for CRS evaluation.","major_comments":[{"comment":"The central robustness claim—that open and closed environments produce highly correlated rankings—rests on Elo ratings computed from only 185 pairwise battles (81 open and 104 closed), with no confidence intervals, significance tests, or sensitivity analysis for the Elo settings (initial rating 1000, K=16). Adjacent systems in the table differ by as little as 2–17 Elo points (e.g., BARCOR_OpenDialKG 968 vs. KBRD_OpenDialKG 966 in the open environment), so the point estimates are not distinguishable. The reported Spearman rho=0.700 between open and closed rankings could be unstable; the authors should provide bootstrap confidence intervals, a Bayesian rating model, or another uncertainty-aware analysis before claiming that the environments produce 'highly correlated rankings.'","section":"Section 4, Table 1"},{"comment":"The comparison between Elo ratings and the R@10 column mixes rankings obtained under very different protocols: Elo captures end-to-end conversational experience with noisy real users in the arena, while R@10 is an offline metric reported in Wang et al. [18] under different conditions. The reported Spearman rho=-0.238 is computed across only nine systems and is not accompanied by a significance test, so it cannot alone support the claim that good recommendation performance does not align with user satisfaction. The authors should report p-values or confidence intervals for this correlation and explicitly discuss the protocol differences that make the comparison, at best, suggestive.","section":"Section 4, Table 1"},{"comment":"The matchmaking algorithm selects the two CRSs with the fewest recorded conversations and resolves ties randomly, making the battle order at least partly deterministic. Elo updates are order-dependent, and the reported ratings are therefore one trajectory rather than a stable estimate. The paper should quantify how the ratings and the subsequent open/closed correlations vary under different matchmaking orders or use a rating method that is invariant to battle order; otherwise the robustness claim is not supported.","section":"Section 3.2"},{"comment":"The open and closed setups differ not only in user population but also in instructions and collection protocol: closed workers were explicitly told to interact at least 5 times, and the collection durations differ (10 days vs. 7 hours). These differences confound the comparison of conversation characteristics in Table 2. The claim that both environments produce conversations with similar characteristics would be strengthened by repeating the analysis on length-matched subsets or by controlling for the number of user turns.","section":"Section 4"}],"minor_comments":[{"comment":"The column heading 'Elo (rank)' and the duplicated rank 1 for ChatGPT_OpenDialKG and ChatGPT_ReDial in the 'Closed & open crowdsourcing' columns should be clarified with a tie-breaking rule.","section":"Table 1"},{"comment":"The paper does not report the number of unique users in the open environment or whether individual users could participate in multiple battles; repeated judgments by the same user would create dependencies that should be discussed.","section":"Section 4"},{"comment":"The recruitment channel for the open environment is not described; stating how users found the platform and whether any screening was applied would help readers assess the data quality.","section":"Section 4"},{"comment":"The note that Safari is not supported is presented as a footnote; since it affects the usability of the public platform, it would be better placed in the main text or in the platform's documentation.","section":"Footnote 1"},{"comment":"The sentence describing the R@10 values should specify the dataset and hyperparameters used in Wang et al. [18], because the arena conversations are not guaranteed to be comparable to that evaluation setup.","section":"Section 4"},{"comment":"There is a minor typo in the sentence introducing battles: 'battle. ' contains an extra space before the period.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the main weakness is the small number of battles and the absence of uncertainty quantification on the Elo-based correlations. The robustness claim is the load-bearing analytical result of the paper, and it is not yet established. The descriptive contribution—the platform and dataset—is valuable and clearly within the scope of WSDM, and the requested analyses (bootstrap, sensitivity checks, significance tests) are feasible within the manuscript's format. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful resource paper. The real contribution is CRSArena-Dial, 474 real user–CRS conversations with pairwise feedback, plus an open-source platform that applies the Chatbot Arena battle format to conversational recommenders. That dataset is genuinely new and fills a gap—most CRS corpora are Wizard-of-Oz or simulated. The paper is honest about noise and calls the ranking preliminary. The analysis is short and the statistical underpinnings are thin: 185 battles across 9 systems, Elo computed with arbitrary settings, no confidence intervals or significance tests. The open/closed robustness claim (Spearman 0.700) is the weakest link; with roughly 20 battles per system and adjacent Elo differences of a few points, that correlation could shift a lot. The stress-test note is right that Elo is order-dependent and matchmaking is deterministic, so the reported ratings are one draw from a wide distribution. The paper's own 'preliminary' caveat is consistent with that. Still, the negative correlation with R@10 and the low overall satisfaction numbers are suggestive, not decisive. The self-citation concern is minor: Distinct-2 comes from Joko et al. and iEvaLM usage is acknowledged; there is no circularity for the ranking claim. Who is this for? Anyone building or evaluating CRSs, and the IR community more broadly, because it offers a low-cost way to get human judgments at scale. The dataset alone makes it worth a look. It deserves peer review; a serious referee would ask for uncertainty quantification and significance tests on the correlations, and a clearer statement that the robustness conclusion is provisional. I'd take it.","headline":"Useful resource paper: the released dataset is the real contribution, but the robustness claim needs more stats before it is treated as established.","tokens_in":7168,"tokens_out":2011,"would_cite":true,"duration_ms":16620,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces CRS Arena, a crowdsourced platform where users converse with two anonymous conversational recommender systems and vote a winner, and reports that open and closed crowdsourcing produce strongly correlated system…","keywords":["conversational recommender systems","crowdsourced benchmarking","pairwise comparison","Elo rating","human evaluation","conversational dataset","user satisfaction"],"falsifier":"Split the closed-environment battles into two random halves and compute Elo ratings for each half; if the two resulting rankings correlate weakly or reorder substantially, the assumption that 185 battles give stable ratings is contradicted. Alternatively, double the battle count and check whether the Spearman correlation between open and closed rankings stays near 0.700.","tokens_in":6216,"feed_emoji":"⚔️","tokens_out":6412,"duration_ms":53997,"temperature":0.7,"pith_summary":"CRS Arena is a web platform that stages pairwise battles between anonymous conversational recommender systems, letting real users chat with two systems in sequence and declare a winner, a loss, or a draw. The paper's central claim is that this crowdsourced setup gives a scalable, reliable way to benchmark conversational recommenders, and it supports the claim by showing that open public traffic and a restricted closed crowdsourcing pool produce strongly correlated system rankings and similarly shaped conversations. Across 474 collected conversations, user satisfaction is low: even the best systems satisfy users only about half the time. The Elo ranking built from 185 pairwise judgments disagrees with the recall-based ranking reported for the same systems, which the authors take as evidence that offline recommendation metrics miss what real interactive use reveals. The paper also releases CRSArena-Dial, a dataset of real user-system conversations with first-party feedback, as a community resource.","feed_headline":"Real-user battles overturn recall-based ranking of recommenders","feed_subtitle":"Open and closed crowdsourcing agree on the new ranking, while satisfaction with every system stays low.","key_machinery":"The central mechanism is the side-by-side battle: a user converses with two anonymous CRSs one after the other, gives each an explicit satisfaction or frustration signal, and then votes for a winner or a draw. A battle manager pairs systems so that the two with the fewest recorded conversations face each other, keeping comparisons balanced. Battle outcomes are aggregated into an Elo rating system, a standard way of converting pairwise wins and losses into a scalar strength score, here initialized at 1000 with a K-factor of 16. The platform's open architecture lets new systems join by implementing a single response-generation method, so the same machinery can grow into a community leaderboard. The accompanying dataset, CRSArena-Dial, stores the 474 conversations together with user feedback, making the pairwise preferences and conversation text the reusable evidence for later analysis.","core_discovery":"The discovery the paper reports is that human pairwise preference data collected through an arena-style platform can rank conversational recommender systems in a way that is consistent across different crowdsourcing environments and yet diverges from conventional recall-based evaluation. With nine CRSs, the platform collected 474 conversations (254 open, 220 closed) with satisfaction/frustration feedback and 185 battle outcomes. Elo ratings computed with initial rating 1000 and K-factor 16 agree strongly with users' expressed satisfaction (Spearman $\\rho=0.917$) and correlate strongly between open and closed setups (Spearman $\\rho=0.700$ for Elo, $\\rho=0.726$ for satisfaction), while correlating negatively with the previously reported recall@10 (Spearman $\\rho=-0.238$). The conversation statistics, including utterances per dialogue, words per utterance, and Distinct-2 diversity, are also similar across both environments. The paper reads this as evidence that CRS Arena is robust to the choice of crowdsourcing setup and that holistic, user-facing evaluation captures a different picture of system quality than offline recommendation metrics.","pith_inferences":["We infer that if the open-closed correlation holds as battle counts grow, the most economical benchmark design would be a hybrid: open volunteer traffic for routine monitoring, and closed crowdsourcing windows when a new system needs a quick, controlled comparison.","We infer that the negative correlation with recall@10 might shrink or shift once more battles accumulate, since with about 20 battles per system the Elo estimates are only a first snapshot rather than a settled leaderboard.","We infer that a natural extension the authors do not develop is using the pairwise preference data to train a reward model or ranker that predicts which system a new user would prefer, turning the arena into a continuous learning loop.","We infer that one testable consequence of the robustness claim is that the same ranking should reappear if the closed pool is drawn from a different region or language group, which would tell us whether the result is tied to English-speaking crowd workers."],"forward_implications":["If CRS Arena is robust across open and closed crowdsourcing, then closed, paid crowd sessions can be used for fast evaluation of new systems, since a few hours of concentrated data collection gives rankings comparable to weeks of open traffic.","If the Elo ranking is a truer reflection of user experience than recall@10, then systems optimized purely for recommendation accuracy may be overestimated by offline evaluation.","The low satisfaction rates across all nine systems imply that current conversational recommenders, at least in this arena setting, do not yet reliably meet users' information needs.","Release of CRSArena-Dial gives other researchers a real-user conversation corpus, distinct from Wizard-of-Oz-style corpora, for studying natural preference expression and system failures."],"supporting_citations":[{"why":"Provides the pairwise battle-arena design that CRS Arena adapts from LLM benchmarking to conversational recommender systems.","marker":"[4]"},{"why":"Supplies the Elo rating method used to convert pairwise battle outcomes into the system ranking.","marker":"[6]"},{"why":"Supplies the underlying evaluation framework used to run the nine CRSs and the recall@10 values that CRS Arena's ranking is compared against.","marker":"[18]"},{"why":"Defines the Distinct-2 metric used to measure response diversity across open and closed environments.","marker":"[10]"},{"why":"Provides the ReDial dialogue corpus that several participating CRSs are trained on.","marker":"[12]"}],"fun_headline_variants":["Crowdsourced arena battles upend recall-based recommender rankings","Pairwise human battles rank chatbot recommenders differently than recall","Arena-style feedback reveals recommender rankings offline metrics miss","User battles align crowdsourced rankings that recall metrics contradict","Human pairwise feedback ranks recommenders, not recall at 10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking is computed from only 185 pairwise battles across nine systems, about twenty battles per system, so the central robustness claim depends on these few battles producing Elo ratings stable enough for the observed open-closed correlations to mean anything.","fun_headline_variants_meta":{"raw":{"variants":["Crowdsourced arena battles upend recall-based recommender rankings","Pairwise human battles rank chatbot recommenders differently than recall","Arena-style feedback reveals recommender rankings offline metrics miss","User battles align crowdsourced rankings that recall metrics contradict","Human pairwise feedback ranks recommenders, not recall at 10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4137,"prompt_tokens":899,"completion_tokens":3238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":3155}},"tokens_in":515,"tokens_out":3238,"duration_ms":22133,"temperature":1.0,"reasoning_tokens":3155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:52:35.839625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split the closed-environment battles into two random halves and compute Elo ratings for each half; if the two resulting rankings correlate weakly or reorder substantially, the assumption that 185 battles give stable ratings is contradicted. Alternatively, double the battle count and check whether the Spearman correlation between open and closed rankings stays near 0.700.","supporting_citations":[{"cited_title":"Gonzalez, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Provides the pairwise battle-arena design that CRS Arena adapts from LLM benchmarking to conversational recommender systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Elo rating method used to convert pairwise battle outcomes into the system ranking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the underlying evaluation framework used to run the nine CRSs and the recall@10 values that CRS Arena's ranking is compared against."},{"cited_title":"de Vries, Jeff Dalton, and Faegheh Hasibi","cited_arxiv_id":null,"evidence_quote":"Defines the Distinct-2 metric used to measure response diversity across open and closed environments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ReDial dialogue corpus that several participating CRSs are trained on."}],"review_version":1}