{"id":"a4eebb2d-4dc2-46f8-b012-396fe81d406d","arxiv_id":"2412.11068","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM judges role-playing as users rank two recommender systems pairwise, producing orderings that track AUC and, the authors argue, sharper separation between closely matched systems.","lead":"This paper tests whether large language models can act as simulated users, comparing two recommendation lists and judging which one better matches a specific person's profile and history. The goal is a cheap, fast way to evaluate recommender systems before committing to slow and expensive live user tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'better discrimination' claim lacks any user-preference ground truth; LLM separation alone could be an artifact of model priors rather than a measure of real user preference.","rationale":"I read the paper's strongest claim as: LLM pairwise evaluation is not merely consistent with offline metrics, but is more discriminative and captures user-relevant differences that AUC/nDCG miss. For this to be true, the LLM's win/lose judgment must be a valid proxy for real user preference. The paper never tests this directly. The validation in Table 3 correlates LLM outputs with AUC, the very metric the paper argues is insufficient, making the discriminative claim partly circular. Figures 3-4 show only that the LLM separates models, not that the separation matches user preference; an LLM with strong genre or popularity priors would produce the same kind of separation. The four-point Pearson correlations with mostly p > 0.05 make the alignment evidence weak even on its own terms. I also note an internal inconsistency in Table 2: the DeepSeek-V2.5 MovieLens row for DeepFM lists the same Win/Tie/Lose percentages as NRMS (64.7/12.6/22.7) but a different Q value (1.7680 vs. 2.1898), which suggests a data-entry error that should be corrected, though it is not the main load-bearing issue. The reader's verdict of CONDITIONAL is appropriate: the method is plausible and the alignment evidence is suggestive, but the central discriminative claim requires external user-preference ground truth. I would not raise or lower the verdict; the paper should be accepted only conditionally on providing such validation or substantially softening the claim.","tokens_in":14759,"tokens_out":3153,"duration_ms":31920,"concrete_test":"Collect human pairwise preference judgments on the same user-item pairs used in Figures 3-4. Stratify by model pairs with near-identical AUC/nDCG (e.g., LightGCN vs. SASRec on MovieLens), sample at least 200 pairs per near-tied model pair, have each pair judged by at least 3 human users, and use majority vote as ground truth. Compute agreement (accuracy and Cohen's kappa) between GPT-4o's pairwise choice and human majority choice, separately for near-tied and non-tied model pairs. If the 95% confidence interval for agreement on near-tied pairs does not exclude chance (0.5), the 'better discrimination' claim is unsupported. If agreement is high and significantly above chance, the claim survives despite the missing ground truth in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim, stated in Section 4.1 (Answer to RQ1), is that LLM-based pair-wise evaluations can 'more effectively identify subtle differences in performance that offline metrics might not capture.' The evidence for this is that LLM win rates and Q values (Figures 3-4) separate models with very similar AUC/nDCG values. But separation is not identification: a judge that relies on item popularity, title fluency, genre priors, or demographic stereotypes would also produce clean separations, and those separations would say nothing about actual user experience. The only validation against an external signal is Table 3, which correlates LLM Q-values with AUC. That is the very metric the paper argues is insufficient, so it cannot establish that the new signal is user-relevant; it only shows that LLM judgments are partially aligned with an offline proxy. Moreover, Table 3 uses only four model-pair data points, and the p-values are above 0.05 in four of the six cells, so even the alignment claim is statistically fragile. The load-bearing condition for the central claim is therefore unverified: LLM pairwise preferences must track real user preferences. Nothing in the paper demonstrates this, and the 'better discrimination' conclusion would collapse if the LLM's pairwise choices are driven by priors unrelated to the target user's actual preferences.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RecSys Arena, an LLM-based pairwise evaluation framework that compares two recommender systems by having an LLM role-play a user and judge their recommendation lists across six aspects. Using MovieLens and MIND, five recommender models, and three LLMs, the authors report that LLM judgements correlate with offline metrics (AUC) and claim that pairwise evaluation provides finer discrimination than AUC/nDCG for models with similar accuracy. They also assess sub-dimensions such as inspiration and transparency with indirect strategies. The paper concludes that LLM-based pairwise evaluation is a promising offline proxy for user preference.","tokens_in":14924,"tokens_out":6345,"duration_ms":50019,"significance":"The problem is important: scalable, fine-grained evaluation without online experiments would be valuable. The paper contributes a concrete prompt design, a public implementation, and a comparison of relative vs absolute evaluation. The pairwise design is well motivated by prior work on LLM pairwise ranking. However, the central claims rest on weak validation: the only external anchor is correlation with AUC (n=4), and the claimed better discrimination lacks any ground truth about user preference. If validated with human or online judgments, the method could be significant; in its current form, the evidence is insufficient for the strong conclusions drawn.","major_comments":[{"comment":"The Pearson correlations between Q and AUC are computed over only four model-level points per LLM/dataset. With n=4, the p-values are 0.1027, 0.0998, 0.0563, 0.0469, 0.7338, and 0.2556; only one of six cells is below 0.05. The text nonetheless calls the correlations 'strong' and uses them to conclude that LLM evaluations are 'consistent with offline metrics.' This is statistically fragile and should be supplemented with confidence intervals, permutation tests, or more model pairs.","section":"Section 4.1, Table 3"},{"comment":"The central claim that 'LLM-based pair-wise evaluations provide better discrimination' is not supported by any external ground truth. Visual separation of Q values for models with similar AUC/nDCG does not demonstrate that the LLM identifies which model is better for users; it could reflect LLM priors, popularity, or genre biases. To support this claim, the authors need to compare LLM pair-wise choices with online A/B test results, human preference judgments, or held-out behavioral outcomes. Without such a signal, the better-discrimination conclusion is unverified.","section":"Section 4.1, Answer to RQ1 and Figures 3, 4, 6"},{"comment":"The paper explicitly acknowledges the absence of subjective user labels for sub-dimensions. The indirect validations are inadequate: for inspiration (Table 4), the URD values used as reference differ only in the fourth decimal place (0.1968, 0.1962, 0.1963, 0.1954), and no test shows these differences are meaningful; for transparency, the evidence consists of two anecdotal examples (Figure 5). The claim that the six aspects reflect user experience (Section 2.2) therefore rests on an admitted missing-support point.","section":"Section 3.1 and 4.2"},{"comment":"The comparison of relative vs absolute evaluation uses only GPT-4o on MovieLens, without statistical tests or normalized-score details. The claim that relative evaluation 'provides better differentiation' is based on visual spread. Additionally, the Q values and absolute scores are normalized arbitrarily; without explaining the normalization and reporting confidence intervals, this comparison cannot be evaluated.","section":"Section 4.3, Figure 6"},{"comment":"The Q metric treats ties as balancing wins and losses, but tie rates vary widely across LLMs (e.g., Llama3.1-8B has tie percentages up to 40.7% in Table 2). Because Q shrinks toward 1 when ties dominate, the rankings of Llama3.1-8B may be an artifact of its tie propensity rather than a meaningful comparison. The authors should analyze tie rates and consider a tie-aware metric or a sensitivity analysis.","section":"Section 2.4, Eq. (2)"},{"comment":"The paper's motivation is that offline metrics such as AUC do not capture user preference, but the only validation of LLM Q against an external signal is correlation with AUC. If AUC is insufficient to measure user preference, then correlation with AUC cannot establish that the LLM captures what AUC misses. A positive correlation only shows partial alignment with an offline proxy. The paper needs a non-offline anchor (e.g., online experiments, human annotations) to break this circularity.","section":"Introduction and Section 4.1"}],"minor_comments":[{"comment":"The text contains typos and formatting issues, including 'langage' (Section 3.4), 'reaults' (Figure 6 caption), 'confindence' (Section 5.1), and a missing division sign in Eq. (2) and the URD formula in Section 3.2.","section":"Throughout"},{"comment":"Table 3's interpretation is misleading: the text says 'P-values are both greater than 0.01' and calls the correlation 'not significant,' but DeepSeek-V2.5 on MIND has p=0.0469, which is significant at the 0.05 level; the negative correlation for Llama3.1-8B on MovieLens is described as 'moderate' but it is negative and insignificant.","section":"Table 3"},{"comment":"The paper does not describe how the Win/Tie/Lose labels are extracted from the LLM's free-text output (e.g., parsing, error handling, prompts for structured output). This reproducibility detail is missing.","section":"Section 4.1"},{"comment":"Section 3.3 lists five recommender systems, but Table 2 only reports pair-wise comparisons of the four non-FM models with FM; it would be clearer to also report the direct comparisons among all pairs, especially for the 'similar AUC' experiments, where the baseline model pairs are not explicitly listed.","section":"Section 3.3"},{"comment":"The paper should report the number of test users/sessions used in the evaluation; currently, no sample size is given, which prevents assessing the stability of the Q estimates.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central validation issue is the absence of any ground truth for user preference. The correlation with AUC is insufficient given the authors' own premise that offline metrics are limited. I recommend major revision, with the expectation that the authors either add a human/online validation study, or substantially weaken the better-discrimination claim and reframe the paper as a proposal with preliminary evidence. The paper may be better suited for a workshop if the validation issue cannot be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper: it applies the Chatbot Arena pairwise comparison idea to offline recommender evaluation, with an LLM role-playing each user and judging which of two recommendation lists better fits that user, across six subjective dimensions. The Q metric is a neat win/tie/lose ratio. The idea is genuinely useful: it could give cheap, scalable, user-perspective signals that offline metrics like AUC and nDCG miss.\n\nWhat's actually new here is the pairing of role-play user simulation with relative pairwise judgment for offline recsys evaluation, not just absolute scoring or conversational settings. The experiments cover two datasets, five recommender models, and three LLMs of very different sizes, and the finding that larger LLMs align better with AUC is plausible. The qualitative transparency examples are suggestive.\n\nThe soft spots are not minor. The headline validation is a Pearson correlation between Q and AUC over only four model-level points, and most p-values are above 0.05. The text calls that \"strong correlation\" and \"consistent with trends,\" which is over-reading. The more serious problem is that the paper's central claim—that pairwise LLM evaluation \"more effectively identify subtle differences in performance that offline metrics might not capture\"—is validated only by visual separation in Figures 3, 4, and 6, with no ground truth about real user preference. Separation by an LLM judge could just be the model's priors about item popularity, genre, or demographic stereotypes. The RQ2 validation for \"inspiration\" correlates with URD, another offline metric, so it inherits the same circularity. The transparency check is two anecdotes. The paper is honest that no subjective user labels exist, but that means the load-bearing claim is unverified.\n\nThere is also a concrete error: in Table 2, the DeepSeek-V2.5 DeepFM row shows the same Win/Tie/Lose percentages as NRMS (64.7/12.6/22.7) but a different Q (1.7680 vs 2.1898). That's internally inconsistent. The code link is an anonymous repo, so reproducibility is unconfirmed.\n\nWho is this for? Researchers working on LLM-based evaluation or recsys offline evaluation would find the framework useful as a baseline and a prompt-design reference. It deserves a serious referee, but as a conditional accept at best: the authors need to validate the \"better discrimination\" claim against real user judgments or at least an A/B-style signal, increase the number of model data points, run significance tests, fix the table error, and release code.\n\nRecommendation: send it to peer review with a request for major revision; the core idea is worth the field's time, but the current evidence does not support the headline claims.","headline":"Pairwise LLM evaluation for recommender systems is a promising idea, but the 'better discrimination' claim rests on unvalidated separation and a 4-point correlation with the very offline metrics the paper argues are insufficient.","tokens_in":15519,"tokens_out":2813,"would_cite":false,"duration_ms":23137,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A large language model that role-plays a user and compares two recommendation lists can rank recommender systems consistently with AUC while separating models whose offline metrics are nearly identical.","keywords":["recommender systems","evaluation","large language models","pairwise comparison","LLM-as-judge","user simulation","offline evaluation"],"falsifier":"Take the pairs of recommender systems the LLM ranks apart, for instance LightGCN versus SASRec on MovieLens, run a real-user A/B test on the same recommendation lists, and check whether the system the LLM prefers is also the one real users engage with more. If the LLM's finer discriminations do not match real user behavior, the method is measuring the LLM's priors rather than user preference.","tokens_in":14494,"feed_emoji":"⚖️","tokens_out":9350,"duration_ms":68447,"temperature":0.7,"pith_summary":"RecSys Arena proposes a practical offline evaluation method: for each user, a large language model receives a profile description and viewing history, sees the recommendation lists produced by two recommender systems, and judges which list better serves that user. The paper claims two things: the aggregated pairwise verdicts track the ranking given by AUC, and the method can distinguish recommender systems whose AUC and nDCG are too close to call, while also scoring subjective aspects such as inspiration, transparency, and impact. This is relevant because standard offline metrics are cheap but coarse and can disagree with online A/B tests, while online tests are slow and expensive. If the claim holds, RecSys Arena gives developers a scalable way to compare candidate algorithms before committing to an online experiment.","feed_headline":"LLM judges spot ranking differences that AUC cannot","feed_subtitle":"Role-played users judge win, tie, or lose, tracking AUC while splitting near-equal models.","key_machinery":"The machinery is a prompt-centred pairwise comparison protocol. The prompt combines a constructed user profile (demographic attributes plus viewing history), the two recommendation lists, definitions of six evaluation dimensions, and a chain-of-thought instruction, and asks the LLM to produce qualitative reasoning and a win/tie/lose verdict for each dimension and overall. The design choices that carry the argument are: role-play to personalise, pairwise presentation to give the model contrast, and the quantile $Q$ to aggregate verdicts into a scalar ranking comparable to AUC. The six dimensions — Accuracy, Satisfaction, Inspiration, Content Quality, Transparency, and Impact on users — are what allow the method to claim coverage of subjective quality that offline metrics cannot express.","core_discovery":"On the paper's own terms, the discovery is that LLM-based pairwise evaluation is both reliable and more discriminative than traditional offline metrics. In experiments on MovieLens and MIND comparing five recommender models, GPT-4o and DeepSeek-V2.5 produced pairwise rankings whose quantile scores $Q=(N_{\\mathrm{win}}+N_{\\mathrm{tie}})/(N_{\\mathrm{lose}}+N_{\\mathrm{tie}})$ correlated strongly with AUC, and the correlation was stronger for larger models; Llama3.1-8B agreed less and declared ties more often. For pairs of models with very similar AUC, such as LightGCN and SASRec, the pairwise judgments still separated them, while absolute LLM scoring did not. The paper also reports that inspiration judgments align with the diversity metric URD and that the LLM can identify when a system ignores user features such as age and occupation, which is the evidence for its transparency sub-dimension.","pith_inferences":["Beyond the paper's experiments, the decisive test is whether LLM pairwise verdicts predict real user choices: running the same model pairs in an A/B test and checking agreement with the LLM's preferred system would directly validate the claimed finer discrimination.","The role-play mechanism implies that the LLM's prior assumptions about a demographic profile do some of the work; a natural follow-up is to perturb user profiles or mask the viewing history and measure how much the pairwise ranking shifts, isolating the signal that comes from genuine user information.","The pairwise comparison format could be extended to active sampling — choosing which model pairs to compare based on uncertainty in the current $Q$ estimates — which would make the method cheaper at scale while preserving its discriminative power.","A further extension is to use the LLM's qualitative justifications per dimension to generate diagnostic reports about why one system wins, turning the evaluation output into actionable feedback for system designers."],"forward_implications":["Developers can rank candidate recommender systems offline using LLM pairwise judging, without waiting for an online A/B test, because the overall rankings agree with AUC trends.","For two systems with near-equal AUC and nDCG, the pairwise quantile $Q$ can still pick a winner, giving an offline signal about which candidate is more likely to win online.","Subjective dimensions of recommendation quality that lack user labels, such as inspiration and transparency, can be evaluated by proxy and produce rankings consistent with measurable diversity and feature usage.","Larger LLMs are more reliable as judges: in the paper's comparisons GPT-4o and DeepSeek-V2.5 track AUC, while Llama3.1-8B shows weaker agreement and more ties.","Because the evaluation dimensions are defined in the prompt, the same framework can be adapted to new aspects of quality by editing the instructions, not retraining."],"supporting_citations":[{"why":"Supplies the Chatbot Arena paradigm of pairwise relative LLM judgment that RecSys Arena adapts to recommender evaluation.","marker":"[10]"},{"why":"Provides evidence that LLMs are better at pairwise than pointwise ranking in recommendation, motivating the relative design.","marker":"[14]"},{"why":"Establishes using LLM role-play as user simulators, the mechanism RecSys Arena builds on.","marker":"[47]"},{"why":"Represents prior LLM-based absolute evaluation of recommendation explanations, the approach RecSys Arena contrasts with.","marker":"[56]"},{"why":"Supplies chain-of-thought prompting used to elicit reasoning in the evaluation prompt.","marker":"[48]"},{"why":"Provides the MovieLens dataset used in experiments.","marker":"[22]"},{"why":"Provides the MIND news dataset used in experiments.","marker":"[50]"},{"why":"Defines AUC, the offline accuracy metric against which LLM pairwise rankings are measured.","marker":"[31]"},{"why":"Defines URD diversity, used as an indirect check on the inspiration sub-dimension.","marker":"[39]"}],"fun_headline_variants":["LLM judges rank recommendations better than AUC","Pairwise LLM feedback beats AUC for similar recommenders","Role-played users in LLM arena split near-equal models","Simulated user feedback via LLM outshines AUC for subtle differences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a user profile built from demographic fields and a short viewing history lets the LLM accurately simulate that user's preferences; if the role-play does not track real user preferences, the pairwise rankings may reflect the LLM's own biases rather than user experience.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges rank recommendations better than AUC","Pairwise LLM feedback beats AUC for similar recommenders","Role-played users in LLM arena split near-equal models","Simulated user feedback via LLM outshines AUC for subtle differences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000297,"raw_usage":{"total_tokens":1762,"prompt_tokens":1026,"completion_tokens":736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":667}},"tokens_in":642,"tokens_out":736,"duration_ms":6103,"temperature":1.0,"reasoning_tokens":667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:19:25.588434+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the pairs of recommender systems the LLM ranks apart, for instance LightGCN versus SASRec on MovieLens, run a real-user A/B test on the same recommendation lists, and check whether the system the LLM prefers is also the one real users engage with more. If the LLM's finer discriminations do not match real user behavior, the method is measuring the LLM's priors rather than user preference.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that LLMs are better at pairwise than pointwise ranking in recommendation, motivating the relative design."},{"cited_title":"Large Language Models as Evaluators for Recommendation Explanations","cited_arxiv_id":"2406.03248","evidence_quote":"Represents prior LLM-based absolute evaluation of recommendation explanations, the approach RecSys Arena contrasts with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MovieLens dataset used in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MIND news dataset used in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines AUC, the offline accuracy metric against which LLM pairwise rankings are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines URD diversity, used as an indirect check on the inspiration sub-dimension."}],"review_version":1}