Pith. sign in

REVIEW 3 major objections 7 minor 35 references

Sparse offline recommender evaluations often rank models differently from dense user feedback, and no single design choice is best across datasets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 01:26 UTC pith:BXZIGK44

load-bearing objection Solid measurement paper: dense feedback as external reference shows sparse offline rankings can be weakly or negatively aligned, with no uniformly best design—but MovieLens “dense” is a 51-user pool. the 3 major comments →

arxiv 2607.25097 v1 pith:BXZIGK44 submitted 2026-07-27 cs.IR

On the Convergent Validity of Offline Evaluation Designs for Recommender Systems

Sushobhan Parajuli , Samira Vaez Barenji , Michael D. Ekstrand This is my paper
classification cs.IR
keywords recommender systemsoffline evaluationmeasurementvalidityimplicit feedbacksparsitysystem rankingconvergent validity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Offline recommender evaluation almost always uses sparse historical logs that are incomplete and biased, so it is unclear whether the model rankings those setups produce actually track true user preferences. This paper tests that question by training many recommenders and comparing the ranking each sparse evaluation design gives them against the ranking obtained from denser ground-truth feedback collected under more complete exposure. Agreement is measured by rank correlation. On one dataset the correlations are weakly positive and change with candidate-set and threshold choices; on another they are consistently negative, so sparse evaluation inverts the dense ranking. The practical upshot is that common differences between algorithms may be artifacts of the evaluation protocol as much as of model quality, and there is no uniformly reliable offline design.

Core claim

The convergent validity of sparse top-N offline evaluation—how well its model rankings agree with rankings from dense ground-truth feedback—depends on the dataset and on the particular dense target, and no combination of candidate-set construction, list length, or relevance threshold is uniformly best.

What carries the argument

Kendall’s τ between the full ordering of models under a sparse train–test NDCG setup and the ordering of the same models under dense ground-truth NDCG; that correlation is treated as the measure of convergent validity.

Load-bearing premise

The dense feedback sets (pooled interest judgments from a small recruited panel, and a near-fully observed watch matrix) measure the same notion of effectiveness that sparse top-N metrics are meant to capture.

What would settle it

Re-run the same model suite on a third dataset that also supplies both sparse logs and an independently collected dense preference matrix; if every sparse design still fails to produce positive rank correlation with the dense ranking, or if one design suddenly dominates on all three, the claim that validity is dataset- and target-dependent with no uniform winner is settled one way or the other.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Model leaderboards built only on sparse logs can reverse the true preference ordering of algorithms on some domains.
  • Full-candidate binary evaluation can raise agreement with dense targets on some data, but popularity-weighted sampling can be better for short-list targets on others.
  • Explicit-feedback models that look weak under sparse evaluation can be among the strongest under dense evaluation, so sparse-only selection can discard useful systems.
  • Researchers cannot treat a single offline protocol as a trustworthy proxy for user preference across datasets.
  • Differences reported between recommenders may be driven as much by evaluation design as by genuine effectiveness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If dense panels remain rare, the field may need cheap proxies (e.g., simulated complete exposure or multi-arm logging) whose rank agreement with existing dense sets can be pre-validated the same way.
  • The KuaiRec inversion suggests that treating continuous watch-ratio signals as explicit ratings systematically mis-ranks models; separate evaluation tracks for pure implicit versus hybrid feedback may be required.
  • Pooling bias and k-core filtering in the MovieLens extension could themselves inflate sparse–dense agreement; ablating those construction steps would test how much of the positive correlation is an artifact of the dense set’s construction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies the convergent validity of sparse offline evaluation for top-N recommender systems. Using a fixed zoo of 45 models (97 configurations on MovieLens-32M, 123 on KuaiRec), the authors rank model configurations under a variety of sparse evaluation designs (candidate-set construction, relevance threshold, list cutoff k) and under dense "ground-truth" evaluation (pooled interest.qrels from 51 recruited users for MovieLens; the 99.6%-dense small matrix for KuaiRec), then measure agreement via Kendall's τ with bootstrapped 95% CIs. On MovieLens, agreement is weakly positive (τ up to 0.43), with full candidate sets and binary thresholds generally best; on KuaiRec, agreement is strongly negative (τ from −0.19 to −0.57) under every design. The authors conclude that the validity of sparse evaluation is dataset- and target-dependent and that no offline evaluation design is uniformly best.

Significance. If the results hold, this is a useful empirical contribution to the ongoing methodological debate about offline recommender evaluation (Krichene & Rendle, Cañamares & Castells, Ihemelandu & Ekstrand). It improves on prior sampled-metrics work by using an external, denser reference instrument rather than treating full-set sparse evaluation as correct by fiat, and the KuaiRec ranking inversion—if it is a property of the evaluation designs rather than an artifact—is a striking and practically important finding. The study is reproducible (code deposited at a Zenodo DOI), reports bootstrap confidence intervals throughout, and is appropriately candid about limitations in §5. Its main weaknesses are that the most surprising result (KuaiRec inversion) is reported but not diagnosed, and that both dense targets have construct-validity caveats that bear directly on interpreting τ as "validity."

major comments (3)
  1. [§4.2, Tables 4–5, Figure 4] The headline KuaiRec result—sparse evaluation inverts the dense ranking under all designs (τ down to −0.57)—is not diagnosed, and there is a concrete alternative explanation that the current analysis cannot rule out. Table 5 shows the top dense models (BiasedSVD-d32/d64, Bias) scoring 0.0–0.001 NDCG under the best-correlated sparse design, i.e., near-degenerate metric values. Since all model configurations (explicit and implicit, and all hyperparameter variants) are pooled into a single τ, the inversion could be driven largely by explicit-feedback models treated on watch-ratio-as-ratings, rather than by a property of the evaluation design per se. Figures 2–4 already color by feedback type, but no τ is reported stratified by feedback type or model family. Please report τ separately for implicit-only and explicit-only configuration sets (or per family), and state how ties in sparse NDCG ar
  2. [§3.1, §5] The validity interpretation requires each dense target to be a higher-quality measurement of the same effectiveness construct over a comparable population, but both targets have construct gaps that are load-bearing for the paper's framing. For MovieLens, interest.qrels come from 51 recruited active users judging IR-style pools; items outside the pool are implicitly nonrelevant, so τ may partly measure agreement with the pool's model-family and popularity coverage rather than with true preferences—and the 51-user panel need not represent the 200k-user sparse test population. For KuaiRec, the small matrix was collected under controlled exposure (users shown a large portion of the catalog); preferences under forced exposure are arguably a different construct from preferences revealed in organic watching, so negative τ could indicate that the dense target, not the sparse design, is the outli
  3. [§3.2, §4, Tables 2 and 4] The unit of analysis for τ and for the bootstrap is unclear in a way that affects the reported confidence intervals. The 97/123 'model configurations' include many correlated hyperparameter variants of the same model family (e.g., 12 UserKNN/ItemKNN variants, 4-dim variants of each matrix factorization). If the bootstrap resamples configurations as if independent, the CIs in Tables 2 and 4 will be anti-conservative because within-family variants share both training behavior and evaluation outcomes. Please state the resampling unit explicitly and consider a family-level sensitivity analysis (e.g., one representative configuration per family, or cluster bootstrap by model family). This matters because several of the design-choice conclusions in §4.1/§4.2 (e.g., P-1000 vs full candidate sets, threshold orderings) rest on τ differences of 0.03–0.08 whose intervals already overlap.
minor comments (7)
  1. [§3.2] The P-1000 sampling scheme is described only as 'popularity-weighted'; please specify the weighting (raw counts, exponentiated popularity as in Ihemelandu & Ekstrand, etc.) and whether sampling is with replacement and fixed across models.
  2. [§3.1–3.2] For KuaiRec implicit training, watch ratios can exceed 2.0 substantially; please state whether values are clipped or transformed before being used as ALS/BPR confidence weights, since this choice interacts with the training-threshold factor that defines the 123 configurations.
  3. [§3.1] The MovieLens 0–7 interest scale is collapsed to 0–4 'by merging the top ranking levels'; please specify exactly which levels are merged, and note that the binary thresholds r≥2/r≥3 are then defined on the collapsed scale.
  4. [§3.1, Table 4] The graded KuaiRec gain (watch_ratio − 0.2, clipped at 0) is a free parameter of the study; a brief sensitivity note (or a justification from the watch-ratio distribution) would help, since the Graded columns show the most extreme negative τ values in Table 4.
  5. [§3.2] The decision to drop dense k=20 is reasonable, but the sparse tables retain k=20 while the text says it 'added little additional information'—consider stating once, early, which cutoff pairs are compared and why, to avoid the asymmetry confusing readers.
  6. [§4.1] In §4.1, 'BiasedSVD models perform significantly better on dense evaluation but are misranked on sparse evaluation' uses 'significantly' informally; either give a test or rephrase. Also, the phrase 'a lot higher' is informal for a venue of this type.
  7. [§4] Please report the number of users contributing to each dense metric explicitly in the results section (51 for MovieLens interest.qrels; the small-matrix user count for KuaiRec), since the precision of the dense rankings themselves depends on it.

Circularity Check

0 steps flagged

Empirical rank-correlation study; sparse vs dense metrics are distinct instruments, and agreement is measured rather than assumed or fitted.

full rationale

The paper’s central claim is an empirical finding: Kendall’s τ between model rankings under sparse train–test NDCG and under dense ground-truth NDCG varies by dataset and target, with no uniformly best sparse design. Sparse and dense scores are computed on different feedback instruments (historical logs vs. pooled interest.qrels / KuaiRec small matrix); τ is an observed association, not a quantity forced by definition or by a fitted parameter. Dense targets come from external collection protocols (Smucker & Chamani; Gao et al.), not from the authors’ own prior uniqueness theorems. Self-citations (LensKit, prior candidate-sampling work) supply tooling and related-work context and do not load-bear the τ results. Concerns about pooling bias or population mismatch in the MovieLens dense target are assumption/correctness issues, not circular reductions of outputs to inputs. No self-definitional loop, fitted-input-as-prediction, or ansatz-via-self-citation step is present. Score 0 with empty steps is the appropriate honest finding.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The work is empirical measurement, not a derivation from axioms. Load-bearing commitments are domain assumptions about what dense feedback represents, standard ranking correlation as the agreement measure, and conventional RS evaluation mechanics (NDCG, candidate sampling, relevance thresholds).

free parameters (5)
  • Binary relevance thresholds (MovieLens r≥2/3/4; KuaiRec watch ratio ≥0.85 and ≥2.0) = MovieLens interest ≥2 and ≥3 (primary); KuaiRec ≥0.85 and ≥2.0
    Hand-chosen cutoffs that define which interactions count as relevant in both sparse and dense binary metrics; they materially change τ.
  • Candidate sample size (1000) and sampling schemes (uniform vs popularity-weighted) = U-1000 and P-1000
    Design knobs fixed by the experimenters; primary factor of variation in MovieLens results.
  • Ranking cutoffs k ∈ {10,20,100} = sparse k=10,20,100; dense k=10,100
    Chosen list lengths for NDCG/DCG; dense side drops k=20 as redundant.
  • Implicit-training feedback inclusion cutoffs = 3 cutoffs MovieLens; 4 KuaiRec (values not fully enumerated in text)
    Multiple minimum feedback values expand the model configuration set (97 ML / 123 KuaiRec) and enter the ranked lists used for τ.
  • Graded gain transforms (interest 0–4; watch_ratio−0.2 clipped at 0) = ML collapse 0–7→0–4; KuaiRec subtract 0.2, clip ≥0
    Ad hoc mappings from raw dense/sparse signals to graded relevance gains.
axioms (5)
  • domain assumption Convergent validity: higher rank agreement (Kendall’s τ) between two measurement procedures implies greater validity w.r.t. the shared construct of model effectiveness / user preference.
    Stated in §1–§2 via measurement-theory citations; underpins interpreting τ as validity rather than mere association.
  • domain assumption Dense experimental feedback (pooled interest judgments; nearly fully observed watch matrix) is a higher-quality reference for true preferences than sparse logs.
    Core of the experimental design in §3; without it, dense rankings are not a validity anchor.
  • domain assumption Missing interactions in RS logs are MNAR; sparse offline metrics can therefore be internally invalid.
    Background from Steck and Marlin et al., §1–§2; motivates the study but is not re-proven.
  • ad hoc to paper Kendall’s τ on a fixed zoo of model configurations is an adequate summary of evaluation-design agreement.
    Chosen primary endpoint in §3–§4; alternatives (score correlation, top-k set overlap, online A/B) are not primary.
  • domain assumption Standard top-N metrics (NDCG/DCG) and common model families implemented in LensKit are representative enough for conclusions about offline evaluation validity.
    §3.2 model table and metric choices; limits external generality if other metrics/models behave differently.

pith-pipeline@v1.2.0-grok45-kimik3 · 22577 in / 3356 out tokens · 84005 ms · 2026-07-31T01:26:31.868638+00:00 · methodology

0 comments
read the original abstract

Offline evaluation on historical interaction logs is the most common evaluation methodology for recommender systems. However, such evaluations depend on sparse, incomplete, or biased data, which raises concerns about whether commonly used evaluation setups reliably reflect true user preferences. In this work, we study how offline evaluation design choices affect the validity of recommender system comparisons. We evaluate a set of recommendation models across several evaluation setups that vary key factors such as data filtering thresholds and candidate set construction. To assess the validity of these configurations, we measure the correlation between model rankings obtained from conventional train-test splits on sparse interaction data and rankings from evaluations based on dense ground-truth user feedback. We use this agreement as an indication of their validity with respect to true user preferences. Our results show that the validity of sparse evaluation depends on the dataset and the specific dense evaluation targets, and that there is no uniformly best offline evaluation design.

Figures

Figures reproduced from arXiv: 2607.25097 by Michael D. Ekstrand, Samira Vaez Barenji, Sushobhan Parajuli.

Figure 1
Figure 1. Figure 1: Diagram of experimental design for a single dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scatter plot of NDCG scores under sparse evaluation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Scatter plot of NDCG scores under sparse evaluation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Scatter plot of NDCG scores under sparse evaluation [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    Joeran Beel and Victor Brunel. 2019. Data Pruning in Recommender Systems Research: Best-Practice or Malpractice?. InProceedings of the 13th ACM Conference on Recommender Systems (RecSys)

    2019

  2. [2]

    Joeran Beel and Stefan Langer. 2015. A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems. InResearch and Advanced Technology for Digital Libraries, Sarantos Kapi- dakis, Cezary Mazurek, and Marcin Werla (Eds.). Vol. 9316. Springer International Publishing, Cham, 153–168. doi:10.100...

    work page doi:10.1007/978-3-319-24592-8_12 2015

  3. [3]

    Alejandro Bellogín, Pablo Castells, and Iván Cantador. 2017. Statistical Biases in Information Retrieval Metrics for Recommender Systems.Information Retrieval Journal20, 6 (Dec. 2017), 606–634. doi:10.1007/s10791-017-9312-z

    doi:10.1007/s10791-017-9312-z 2017

  4. [4]

    Rocío Cañamares and Pablo Castells. 2020. On Target Item Sampling in Offline Recommender System Evaluation. InRecSys ’20. ACM, New York, NY, USA, 259–268. doi:10.1145/3383313.3412259

    arXiv 2020

  5. [5]

    Diego Carraro and Derek Bridge. 2022. A Sampling Approach to Debiasing the Offline Evaluation of Recommender Systems.Journal of Intelligent Information Systems58, 2 (April 2022), 311–336. doi:10.1007/s10844-021-00651-y

    doi:10.1007/s10844-021-00651-y 2022

  6. [6]

    Cook and Thomas J

    David A. Cook and Thomas J. Beckman. 2006. Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application.The American Journal of Medicine119, 2 (Feb. 2006), 166.e7–166.e16. doi:10.1016/j.amjmed.2005. 10.036

    work page doi:10.1016/j.amjmed.2005 2006

  7. [7]

    Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of Recommender Algorithms on Top-N Recommendation Tasks. InProceedings of the Fourth ACM Conference on Recommender Systems. ACM, Barcelona, Spain, 39–46. doi:10.1145/1864708.1864721

    arXiv 2010

  8. [8]

    Michael D Ekstrand. 2020. LensKit for Python: Next-generation software for recommender systems experiments. InProceedings of the 29th ACM international conference on information & knowledge management. 2999–3006

    2020

  9. [9]

    Michael D Ekstrand and Vaibhav Mahant. 2017. Sturgeon and the Cool Kids: Problems with Top-N Recommender Evaluation. InProceedings of the 30th Florida Artificial Intelligence Research Society Conference (FLAIRS 30). AAAI Press

    2017

  10. [10]

    Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. 2022. KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems. InProceedings of the 31st ACM International Conference on Information & Knowledge Management. ACM, Atlanta, GA, USA, 540–550. doi:10.1145/3511808.3557220

    arXiv 2022

  11. [11]

    Danil Gusak, Anna Volodkevich, Anton Klenitskiy, Alexey Vasilev, and Evgeny Frolov. 2025. Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders. InProceedings of the Nineteenth ACM Conference on Recommender Systems. ACM, Prague, Czech Republic, 874–883. doi:10.1145/3705328.3748164

    arXiv 2025

  12. [12]

    Maxwell Harper and Joseph A

    F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context.ACM Trans. Interact. Intell. Syst.5, 4 (Dec. 2015), 19:1–19:19. doi:10.1145/2827872

    doi:10.1145/2827872 2015

  13. [13]

    Balázs Hidasi and Ádám Tibor Czapp. 2023. Widespread Flaws in Offline Eval- uation of Recommender Systems. InProceedings of the 17th ACM Conference on Recommender Systems. ACM, Singapore, Singapore, 848–855. doi:10.1145/ 3604915.3608839

    arXiv 2023

  14. [14]

    Ekstrand

    Ngozi Ihemelandu and Michael D. Ekstrand. 2023. Candidate Set Sampling for Evaluating Top-N Recommendation. InProceedings of the 22nd IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology. 88–94. arXiv:2309.11723 [cs.IR] doi:10.1109/WI-IAT59888.2023.00018

    Pith/arXiv arXiv 2023

  15. [15]

    Jadidinejad, Craig Macdonald, and Iadh Ounis

    Amir H. Jadidinejad, Craig Macdonald, and Iadh Ounis. 2022. The Simpson’s Paradox in the Offline Evaluation of Recommendation Systems.ACM Transactions on Information Systems40, 1 (Jan. 2022), 1–22. doi:10.1145/3458509

    doi:10.1145/3458509 2022

  16. [16]

    Olivier Jeunen. 2019. Revisiting Offline Evaluation for Implicit-Feedback Rec- ommender Systems. InProceedings of the 13th ACM Conference on Recommender Systems. ACM, Copenhagen, Denmark, 596–600. doi:10.1145/3298689.3347069

    arXiv 2019

  17. [17]

    Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2023. A Critical Study on Data Leakage in Recommender System Offline Evaluation.ACM Transactions on Information Systems41, 3 (July 2023), 1–27. doi:10.1145/3569930

    doi:10.1145/3569930 2023

  18. [18]

    Kimberlin and Almut G

    Carole L. Kimberlin and Almut G. Winterstein. 2008. Validity and Reliability of Measurement Instruments Used in Research.American Journal of Health-System Pharmacy65, 23 (Dec. 2008), 2276–2284. doi:10.2146/ajhp070364

    work page doi:10.2146/ajhp070364 2008

  19. [19]

    Yehuda Koren. 2008. Factorization Meets the Neighborhood: A Multifaceted Col- laborative Filtering Model. InProceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Las Vegas, Nevada, USA, 426–434. doi:10.1145/1401890.1401944

    arXiv 2008

  20. [20]

    Walid Krichene and Steffen Rendle. 2020. On Sampled Metrics for Item Recom- mendation. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, USA, 1748–1757. doi:10.1145/3394486.3403226

    arXiv 2020

  21. [21]

    Shane Culpepper

    Xiaolu Lu, Alistair Moffat, and J. Shane Culpepper. 2016. The Effect of Pooling and Evaluation Depth on IR Metrics.Information Retrieval Journal19, 4 (Aug. 2016), 416–445. doi:10.1007/s10791-016-9282-6

    work page doi:10.1007/s10791-016-9282-6 2016

  22. [22]

    Collaborative Filtering and the Missing at Random Assumption

    Benjamin Marlin, Richard S. Zemel, Sam Roweis, and Malcolm Slaney. 2012. Collaborative Filtering and the Missing at Random Assumption. doi:10.48550/ arXiv.1206.5267

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1206.5267 2012

  23. [23]

    Marlin and Richard S

    Benjamin M. Marlin and Richard S. Zemel. 2009. Collaborative Prediction and Ranking with Non-Random Missing Data. InProceedings of the Third ACM Conference on Recommender Systems. ACM, New York, New York, USA, 5–12. doi:10.1145/1639714.1639717

    arXiv 2009

  24. [24]

    Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2020. Ex- ploring Data Splitting Strategies for the Evaluation of Recommendation Models. InFourteenth ACM Conference on Recommender Systems. ACM, Virtual Event, Brazil, 681–686. doi:10.1145/3383313.3418479

    arXiv 2020

  25. [25]

    Pereira, Alan Said, and Rodrygo L

    Bruno L. Pereira, Alan Said, and Rodrygo L. T. Santos. 2025. On the Reliability of Sampling Strategies in Offline Recommender Evaluation. InProceedings of the Nineteenth ACM Conference on Recommender Systems (RecSys ’25). ACM, New York, NY, USA, 360–369. doi:10.1145/3705328.3748086

    arXiv 2025

  26. [26]

    Marco Rossetti, Fabio Stella, and Markus Zanker. 2016. Contrasting Offline and Online Results When Evaluating Recommendation Algorithms. InProceedings of the 10th ACM Conference on Recommender Systems. ACM, Boston, Massachusetts, USA, 31–34. doi:10.1145/2959100.2959176

    arXiv 2016

  27. [27]

    Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. 2025. Measurement to Meaning: A Validity-Centered Framework for AI Evaluation. doi:10.48550/arXiv.2505.10573

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.10573 2025

  28. [28]

    Smucker and Houmaan Chamani

    Mark D. Smucker and Houmaan Chamani. 2025. Extending MovieLens-32M to Provide New Evaluation Objectives. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Padua, Italy, 3520–3529. doi:10.1145/3726302.3730328

    arXiv 2025

  29. [29]

    Harald Steck. 2010. Training and Testing of Recommender Systems on Data Missing Not at Random. InProceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Washington DC USA, 713–722. doi:10.1145/1835804.1835895

    arXiv 2010

  30. [30]

    Aixin Sun. 2023. Take a Fresh Look at Recommender Systems from an Evaluation Standpoint. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, Taipei Taiwan, 2629–

    2023

  31. [31]

    Robin Verachtert, Lien Michiels, and Bart Goethals. 2022. Are We Forgetting Something? Correctly Evaluate a Recommender System with an Optimal Training Window. InProceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2022

    2022

  32. [32]

    2005.TREC: Experiment and evaluation in information retrieval

    Ellen M Voorhees, Donna K Harman, et al. 2005.TREC: Experiment and evaluation in information retrieval. Vol. 63. MIT press Cambridge. On the Convergent Validity of Offline Evaluation Designs for Recommender Systems RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA

    2005

  33. [33]

    Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Chouldechova, Emily Corvi, P

    Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Chouldechova, Emily Corvi, P. Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nicholas Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vo- gel, Hannah Washington, and Abigail Z. Jacobs. 2025. Position: Ev...

    Pith/arXiv arXiv 2025

  34. [34]

    Xiting Wang, Liming Jiang, José Hernández-Orallo, David Stillwell, Shiqiang Chen, Luning Sun, Fang Luo, and Xing Xie. 2026. Evaluating General-Purpose AI with Psychometrics.Commun. ACM(April 2026), 3769688. doi:10.1145/3769688

    work page doi:10.1145/3769688 2026

  35. [2638]

    doi:10.1145/3539618.3591931

    arXiv