{"id":"b9769499-4410-42b2-b1a5-485fc3c0652c","arxiv_id":"2507.17290","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Basic and multi-model LLM prompts can evaluate recommendation serendipity as well as or better than standard proxy formulas, reaching 21.5% Pearson correlation with user-study ratings.","lead":"This paper tests whether large language models can rate how surprising and useful a recommended item feels, comparing their scores to real user ratings from shopping and movie datasets. It finds that even simple LLM prompts match or beat traditional recommendation formulas, and that combining models with extra user information improves agreement to 21.5%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Movie-domain gold standard is an unexpectedness composite, not direct serendipity; the meta-evaluation may measure alignment with a proxy, undermining the central claim.","rationale":"Both the gold-standard validity and the test-set selection threaten the headline numbers, but the gold-standard concern is the more fundamental one because it affects the interpretation of every result in the paper, not just the optimal configuration. If the movie-domain target measures unexpectedness rather than serendipity, then the parity claim in the abstract is about the wrong construct. The selection issue, by contrast, only inflates the reported 21.5%; the general finding that zero-shot LLMs match or beat proxy metrics is based on many individual comparisons in Table 2 and would likely survive a corrected selection procedure. The paper's own disclosure in Section 2.2 makes the gold-standard issue concrete: the target is an explicit composite of three unexpectedness variables, and the original dataset had eight serendipity-related statements. A robustness check with alternative composites is the minimal experiment needed to establish that the results are not an artifact of this particular operationalization.","tokens_in":13370,"tokens_out":7006,"duration_ms":70261,"concrete_test":"Reconstruct the Serendipity-2018 ground truth from all eight 5-point Likert statements (or a factor-analytic composite) rather than the three unexpectedness variables, and re-run the SerenEva comparison in Tables 2 and 5. If the Pearson correlations and the relative ordering of LLMs versus proxy metrics change materially (e.g., the optimal configuration no longer exceeds SOG), the reported results are an artifact of the chosen composite. Additionally, for the Taobao dataset, test the single-item measure's reliability (e.g., via a test-retest or split-half) or use a multi-item composite if available to bound the noise in the gold standard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 states that the Serendipity-2018 dataset 'lacks direct user ratings on serendipity' and that the authors instead 'calculated the score by averaging three unexpectedness-related variables, as validated in [31]'. This means the movie-domain gold standard is itself a proxy constructed from unexpectedness items, not a direct user serendipity judgment. The Taobao gold standard is a single 'pleasant surprise' Likert item, which is noisy. The LLM prompts explicitly ask for a serendipity rating (defined as 'pleasant surprise'), so for movies the target construct is mismatched: the LLM is asked about serendipity while the target measures only unexpectedness. If the chosen composite is a poor operationalization of serendipity, then every reported Pearson correlation, and the abstract's headline '21.5% when compared to the results of the user study', is alignment with a flawed target. Consequently, the relative ranking of LLMs versus proxy metrics loses its meaning: both may simply be predicting unexpectedness. This is a construct-validity threat to the central claim that LLMs can evaluate serendipity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SerenEva, a meta-evaluation framework for assessing whether large language models (LLMs) can serve as serendipity evaluators in recommender systems. The authors compare zero-shot and few-shot LLM ratings against two user-study-derived gold standards (Taobao e-commerce and MovieLens/Serendipity-2018), benchmark them against four conventional proxy metrics (SOG, SNPR, PURS, DESR), investigate the effect of auxiliary user, item, and interaction data, and evaluate multi-LLM ensembles with score averaging. The main reported findings are that even basic zero-shot LLMs match or surpass proxy metrics, that auxiliary data improve performance in a domain-dependent manner, and that the best multi-LLM configuration reaches Pearson correlations of 20.23% (Taobao) and 21.51% (Serendipity-2018) with the user-study ground truth. The paper provides its code and uses publicly available models.","tokens_in":13595,"tokens_out":4700,"duration_ms":43658,"significance":"The study addresses a timely and practical question: whether LLMs can be inexpensive, reproducible evaluators of a subjective quantity like serendipity. The strengths of the manuscript are its clear meta-evaluation protocol, the use of real user-study data, the inclusion of several LLM families and sizes, and the public release of code. The zero-shot parity claim is supported by the results in Table 2, and the analysis of auxiliary data is a useful mapping of when such data help. However, the headline 'optimal' result is based on test-set selection, and the movie-domain gold standard is an unexpectedness composite rather than direct serendipity; both issues temper the strength of the central claims as currently presented.","major_comments":[{"comment":"The reported headline Pearson correlations (20.23% and 21.51%) are the maximum over a grid search of auxiliary data types and multi-LLM ensemble compositions evaluated on the same test set used to compute the correlations. No held-out validation or multiple-testing correction is described; this makes the optimal performance estimate optimistically biased. Please separate configuration selection from final evaluation (e.g., with a validation split) and temper the abstract and conclusions accordingly.","section":"Section 5, Table 5"},{"comment":"The Serendipity-2018 gold standard is not a direct user serendipity rating; the paper states that the dataset 'lacks direct user ratings on serendipity' and that the score was computed by averaging three unexpectedness-related variables. Since the LLM prompt asks for 'pleasant surprise', the correlations on this dataset measure alignment with an unexpectedness composite, which is a construct mismatch. Please justify this operationalization, discuss its validity, or provide sensitivity analyses with alternative gold-standard definitions.","section":"Section 2.2"},{"comment":"The paper reports only results that are statistically significant at p < 0.05 in a two-sided t-test, without any correction for the many models, prompts, auxiliary-data conditions, and ensemble configurations tested. Selective reporting of significant results inflates effect sizes and false-positive rates. Please report the full result matrix or apply a multiple-comparison correction, and distinguish exploratory from confirmatory findings.","section":"Section 2.5"}],"minor_comments":[{"comment":"The third panel caption contains a typo: 'Nummber' should be 'Number'.","section":"Figure 5 caption"},{"comment":"The sentence 'Because user studies provide real user feedback, making them the closest approximation to actual user perceptions. Therefore, we regard these results as the gold standard' is a fused sentence; consider splitting or rephrasing.","section":"Section 2.2"},{"comment":"The min-max normalization uses the distribution of outputs over the full evaluation set; this is acceptable for correlation but should be clarified for MAE/RMSE, since the normalized scores depend on the test set's min and max.","section":"Section 2.5, Eq. (6)"},{"comment":"The paper states that Big-Five traits, age, and gender 'did not help enhance performance and even reduced it'; providing the actual numbers (even in an appendix) would strengthen this negative result.","section":"Section 4.1"},{"comment":"The claim that 'multi-LLM techniques show performance improvements as the number of LLMs increases' is based on a limited set of ensembles; a direct comparison of ensemble sizes with error bars would clarify the trend.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for RecSys, and the code availability is a plus. The authors are transparent about the composite nature of the movie gold standard and about the selective reporting of significant results, but the latter is nevertheless a serious methodological concern. With careful revisions to the evaluation protocol, this paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful systematic benchmark. The paper does something new: it grids zero-shot, few-shot, auxiliary data, and multi-LLM ensembles for serendipity evaluation across two user-study datasets, and releases code. The zero-shot parity finding is solid: Qwen2.5-72B and GPT-4 with a simple prompt beat the best proxy metric (SOG) on both datasets in Pearson correlation. That is a clean, reproducible result, and the authors give prior work its due.\n\nThe soft spots are real but not fatal. The construct-validity concern from the stress test lands: the Serendipity-2018 gold standard is not a direct serendipity rating. The authors averaged three unexpectedness-related variables, while the LLM prompt asks for 'pleasant surprise.' So in the movie domain the evaluation measures alignment with an unexpectedness proxy, not with serendipity as defined. The paper discloses this in Section 2.2, but the abstract and conclusions treat both datasets symmetrically, which is an overstatement. The Taobao dataset has a direct 'pleasant surprise' Likert item, so half the paper is on solid ground.\n\nSecond, the headline 21.5% is the best result over a grid search on the test set. There is no held-out validation, no multiple-testing correction, and no confidence intervals. The zero-shot and few-shot comparisons are much more trustworthy. The direction of the auxiliary-data findings is plausible, but the exact optimal configuration is likely optimistic.\n\nThird, the authors report only statistically significant improvements, which is a form of selective reporting. They state this at the outset, so it is not hidden, but the full landscape of negative results would help the reader calibrate.\n\nWho should read this: anyone working on recommendation evaluation, especially serendipity. It is a useful benchmark, and the open code is a plus. It deserves a serious referee. My recommendation: send it to review; expect a revision that validates the selected configuration on held-out data and rewrites the abstract to acknowledge the construct mismatch in the movie dataset.","headline":"Useful systematic benchmark of LLM serendipity evaluators, but the headline 21.5% is test-set selection and the movie gold standard is an unexpectedness proxy.","tokens_in":14114,"tokens_out":3575,"would_cite":true,"duration_ms":33839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Plain prompted LLMs match or beat proxy metrics for judging recommendation serendipity.","keywords":["recommender systems","serendipity","large language models","LLM evaluation","meta-evaluation","proxy metrics","user studies","multi-LLM ensemble"],"falsifier":"Run the same SerenEva evaluation on a dataset where users directly rate serendipity, or where the gold standard is independently validated by a second user study; if LLM correlations against that direct gold standard fall below the proxy metrics, or below the 21.5% reported here, the paper's central claim fails. A simpler check is to recompute the Serendipity-2018 gold standard from all eight original survey statements instead of the three unexpectedness variables and see whether the LLM-versus-proxy ranking survives.","tokens_in":13183,"feed_emoji":"🤖","tokens_out":7882,"duration_ms":71385,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models can evaluate recommendation serendipity—whether a suggested item is an unexpected but pleasant surprise—as well as or better than the formula-based proxy metrics now used in place of user studies. The authors compare LLM ratings, produced by prompting models with a user's behavior history, against ratings from two real user studies and against four conventional proxy metrics. They report that even zero-shot LLMs match or surpass the best proxy metrics, and that adding auxiliary data and averaging scores across several LLMs raises the correlation with user-study ratings to about 21.5%. If correct, this gives researchers a cheap, reproducible way to measure serendipity without running costly user studies.","feed_headline":"Even simple LLMs beat proxy metrics at judging serendipity","feed_subtitle":"Zero-shot prompts align with user-study ratings; ensembling pushes correlation past 21 percent.","key_machinery":"The central object is SerenEva, a meta-evaluation framework that measures how well any candidate evaluator aligns with user-study ground truth. SerenEva feeds a constrained prompt—user behavior history, item information, and a 5-point Likert question about serendipity—to an LLM, then compares the LLM's numeric ratings with real user ratings using Pearson correlation, MAE, and RMSE; proxy metrics are first rescaled to the same 5-point Likert format. The framework also varies the auxiliary data injected into the prompt (user demographics and psychology, item popularity and similarity, interaction length and type) and supports multi-LLM ensembles that average scores across models. The load-bearing comparisons are the four proxy metrics (SOG, PURS, DESR, SNPR) and the two user-study datasets treated as gold standard.","core_discovery":"On the paper's own terms, the discovery is that serendipity evaluation can be performed by LLMs with no task-specific training. In a meta-evaluation across an e-commerce dataset (Taobao) and a movie dataset (Serendipity-2018), the authors find that a zero-shot GPT-4 or Qwen2.5-72B prompt reaches Pearson correlations of roughly 10.5–10.9% against user-study ratings, about double the 4.6–5.7% of the strongest proxy metric (SOG). Few-shot prompting improves most models further, and combining auxiliary data with a score-averaging ensemble of Qwen2.5-14B, Qwen2.5-72B, and GPT-4 yields Pearson correlations of 20.23% (Taobao) and 21.51% (Serendipity-2018), with correspondingly lower MAE and RMSE. The paper interprets these results as evidence that LLMs can combine the accuracy of user studies with the efficiency of proxy metrics.","pith_inferences":["The reported correlations are modest in absolute terms (around 0.2), so the practical claim is not that LLMs capture serendipity fully, but that they capture it as well as any existing cheap proxy; a reader should interpret 'match or surpass' relative to weak baselines.","Because the gold standard itself contains noise—the Taobao label is a single 'pleasant surprise' item and Serendipity-2018 averages three unexpectedness variables—the true human-alignment ceiling may be higher or lower than 21.5%; a direct serendipity-rating study would be needed to know.","The same SerenEva protocol could be applied to other subjective evaluation targets—novelty, diversity, trust, explanation quality—where proxy metrics are known to diverge from user perception.","The finding that multi-LLM averaging helps suggests that ensemble diversity, not raw model size, drives alignment; future work could test whether a small set of deliberately diverse open-weight models matches the GPT-4-inclusive ensemble."],"forward_implications":["Zero-shot or few-shot LLM prompts can serve as a drop-in replacement for proxy metrics when user studies are infeasible, giving similar or better alignment with human judgments.","Including a small amount of user-study data as few-shot examples lets smaller models like Qwen2.5-7B approach the accuracy of much larger models, lowering cost.","Auxiliary data helps only when matched to the domain: curiosity data matters for e-commerce, item popularity and long-term profiles matter for movies, and indiscriminate additions can hurt.","Multi-LLM ensembles with simple score averaging improve alignment with human ratings, so evaluation systems can trade API cost for accuracy by selecting diverse models.","LLM-based serendipity evaluation could make serendipity research more reproducible, since prompts and models are fixed and rerunnable, unlike one-off user studies."],"supporting_citations":[{"why":"Supplies the Taobao dataset and the 'pleasant surprise' definition of serendipity that serves as one gold standard.","marker":"[5]"},{"why":"Supplies the Serendipity-2018 movie dataset from a real user study, the second gold standard.","marker":"[14]"},{"why":"Defines the SOG proxy metric, the strongest conventional baseline the LLMs are compared against.","marker":"[17]"},{"why":"Defines the PURS proxy metric benchmarked as a conventional baseline.","marker":"[20]"},{"why":"Defines the DESR proxy metric benchmarked as a conventional baseline.","marker":"[21]"},{"why":"Defines the SNPR proxy metric benchmarked as a conventional baseline.","marker":"[36]"},{"why":"Validates averaging three unexpectedness variables for the Serendipity-2018 label and motivates the auxiliary-data categories.","marker":"[31]"},{"why":"Provides the Qwen2.5 models that yield the highest LLM evaluation correlations.","marker":"[35]"},{"why":"Provides the GPT-4 model used in the best multi-LLM ensembles.","marker":"[1]"}],"fun_headline_variants":["Zero-shot LLMs match or beat serendipity proxies","LLM ensembles lift serendipity correlation past 21%","Serendipity judges: simple LLMs rival user studies","No training needed: LLMs assess serendipity well","LLMs turn serendipity scoring into a zero-shot task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the user-study ratings used as gold standard genuinely measure serendipity; in the Taobao dataset that means a single 'pleasant surprise' Likert response, and in Serendipity-2018 an average of three unexpectedness-related survey variables rather than a direct serendipity rating.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLMs match or beat serendipity proxies","LLM ensembles lift serendipity correlation past 21%","Serendipity judges: simple LLMs rival user studies","No training needed: LLMs assess serendipity well","LLMs turn serendipity scoring into a zero-shot task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2404,"prompt_tokens":1008,"completion_tokens":1396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":1308}},"tokens_in":624,"tokens_out":1396,"duration_ms":10365,"temperature":1.0,"reasoning_tokens":1308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:51:41.452334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SerenEva evaluation on a dataset where users directly rate serendipity, or where the gold standard is independently validated by a second user study; if LLM correlations against that direct gold standard fall below the proxy metrics, or below the 21.5% reported here, the paper's central claim fails. A simpler check is to recompute the Serendipity-2018 gold standard from all eight original survey statements instead of the three unexpectedness variables and see whether the LLM-versus-proxy ranking survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Taobao dataset and the 'pleasant surprise' definition of serendipity that serves as one gold standard."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Serendipity-2018 movie dataset from a real user study, the second gold standard."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SOG proxy metric, the strongest conventional baseline the LLMs are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PURS proxy metric benchmarked as a conventional baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DESR proxy metric benchmarked as a conventional baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SNPR proxy metric benchmarked as a conventional baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Validates averaging three unexpectedness variables for the Serendipity-2018 label and motivates the auxiliary-data categories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPT-4 model used in the best multi-LLM ensembles."}],"review_version":1}