Pith. sign in

REVIEW 4 major objections 6 minor 79 references

Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The choice of data splitting strategy for sequential recommenders can significantly change evaluation metrics and model rankings, so offline comparisons should be read with the split protocol in mind.

desk verdict A well-executed empirical study whose central descriptive claim holds; its practical recommendations are conditional on an unvalidated 'Successive as realistic' reference, but the paper deserves review. read the letter →

arxiv 2507.16289 v1 pith:PZ7VXQ4G submitted 2025-07-22 cs.IR cs.LG

Danil Gusak , Anna Volodkevich , Anton Klenitskiy , Alexey Vasilev , Evgeny Frolov This is my paper
classification cs.IRcs.LG
keywords sequentialrecommendersystemsofflineevaluationdatasplittingleave-one-outsplitglobaltemporalnext-itempredictionmodelrankingprotocol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the way interaction data is split into training and test sets can change which sequential recommender system looks best in offline evaluation. The authors define a global temporal split tailored to next-item prediction, with five target-selection options (last, first, random, successive, and all held-out items) and three validation-set constructions (global temporal, last training item, and user-based). Comparing these against the widely used leave-one-out split on eight datasets and mainstream baselines, they find that leave-one-out correlates only moderately with the successive scheme they treat as closest to production, and that both absolute metrics and final model rankings shift across split choices. They conclude that global temporal splitting with a last-item or random-item target is a practical, reproducible alternative to the more expensive successive evaluation. If correct, the results imply that published rankings obtained under leave-one-out should not be assumed to hold in deployment.

What carries the argument

The central object is the global temporal split for sequential recommenders, specified by a global cutoff time, a target-selection rule for each holdout sequence, and a validation-set construction; the reference protocol is the Successive target, where every interaction in a holdout sequence is scored as a separate next-item prediction with the preceding history as input, and metrics are averaged per user. The paper's measuring instrument is a systematic correlation analysis (Kendall and Spearman coefficients computed across models and hyperparameter configurations) between each split variant and this Reference, supplemented by an analysis of the time-gap distribution between target interactions and their predecessors, which explains why the First target is biased. This combination lets the authors identify which split components carry the most realistic signal and which are artifacts.

What would settle it

Serve models selected under different splits (for example, the best model chosen by LOO versus by GTS-Successive) in a live recommender with logged user responses, or on a large withheld future time window with known next interactions, and compare actual next-item match or engagement. If the model selected by LOO matches or beats the model selected by GTS-Successive, then the Successive reference is not a reliable proxy and the paper's central comparison loses its anchor.

Watch

Extended reading notes

Core claim

Using GTS with the Successive target (each held-out interaction treated as a separate next-item prediction, averaged by user) as the reference for 'close to production' evaluation, the paper measures how other splits agree with it. Averaged across datasets, GTS Random has the highest Kendall/Spearman agreement (about 0.91/0.98 for NDCG@10), GTS Last is next (about 0.83/0.94), leave-one-out is lower (about 0.71/0.87), GTS First is similar to LOO or worse on some datasets due to large time gaps after the cutoff, and GTS All has very low correlation (about 0.57/0.53), exposing its mismatch with the next-item objective. Validation and retraining experiments show that global temporal validation aligns best with test metrics and that retraining the selected model on combined training-plus-validation data improves final test performance for global-temporal and user-based validation. The paper's central claim is that data-splitting strategy for sequential recommenders is not a neutral experimental detail: it can materially affect model rankings and therefore deployment decisions.

Load-bearing premise

The paper states in Section 4.3 that the global temporal split with Successive target is treated as the most realistic and closest to production use, and all other splits are scored by how well they agree with that reference; there is no online or logged production data validating the reference itself. If Successive is not in fact the right proxy for real-world performance, the paper's recommendations about which targets and validation schemes to prefer could change.

Editorial extensions

If this is right

  • Published model rankings obtained with leave-one-out splits are less likely to transfer to temporal, production-like evaluation; meta-analyses that pool results across different split protocols will carry hidden ranking noise.
  • GTS with Last or Random target can substitute for the computationally expensive Successive evaluation in most offline studies, with Random requiring stored seeds or multi-seed averaging for reproducibility.
  • Validation sets for GTS should be built with a matching global temporal split, and final results should be reported both with and without retraining on the combined training and validation data, since retraining materially changes test scores.
  • Papers using GTS should report the cutoff quantile, target-selection rule, and input-sequence construction; without these details, results cannot be compared or reproduced.
  • Datasets with small holdout lengths or few active users after the cutoff may need a lower quantile to retain enough test users, trading off test-period duration against statistical power.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not fully pursue is a deterministic random target: sample one per-user target with a fixed stored seed and publish it, which should combine Random's high correlation with Last's reproducibility.
  • The median time-gap analysis suggests a screening criterion for sequential-recommendation datasets: if the gap between the cutoff and the First post-cutoff interaction is a clear session boundary, that target is structurally biased, and datasets lacking clean session boundaries might behave differently.
  • The roughly 0.7 Kendall agreement between LOO and the realistic reference implies that LOO-based leaderboards have a substantial 'split-noise' floor even when metrics are computed exactly, which could be quantified per dataset as a variance component.
  • For industrial deployment, the retraining results imply that validation-guided early stopping under global temporal validation leaves performance on the table unless the final model is retrained on train-plus-validation; reporting only the unretrained number understates what the deployed model would achieve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies how the choice of data splitting strategy affects offline evaluation of sequential recommender systems. It compares leave-one-out (LOO) splitting with several variants of global temporal splitting (GTS), differing in how target items are selected from the holdout sequence (Last, First, Random, Successive, All) and how the validation set is constructed (global temporal, user-based, last training item). Using eight datasets and three neural baselines (plus a non-neural kNN in the ranking analysis) with many hyperparameter configurations, the authors show that the choice of split changes absolute metric values and, in many cases, the relative ranking of models. They further analyze time gaps between target interactions and previous events, correlation of test metrics across splits, correlation between validation and test metrics, and the effect of retraining on combined training and validation data. The central descriptive claim—that split choice significantly impacts metrics and model rankings—is well supported. However, the paper's prescriptive conclusions, especially that GTS Last and GTS Random are practical alternatives to GTS Successive, rest on treating Successive as the 'most realistic' reference without external validation.

Significance. If the descriptive findings are taken at face value, the paper makes a useful contribution by documenting that common offline evaluation protocols for sequential recommenders are not interchangeable: model rankings change across splits, and LOO can diverge substantially from global temporal evaluation. The empirical scope is a clear strength: eight datasets, multiple models, hundreds of hyperparameter configurations, unsampled ranking metrics, and released code. The formalization of GTS target and validation options is also a practical contribution for the community, and the framework survey (Section 3.0.2) is informative. The main weakness is that the normative recommendations are derived from correlation with an author-defined reference (GTS Successive) that is asserted, not validated, to be the closest to production. The retraining analysis is also much thinner than the rest of the study. These issues are load-bearing for the paper's practical conclusions, and the claims of 'real-world alignment' would need either external evidence or a more careful framing as 'agreement with Successive.'

major comments (4)
  1. [Section 4.3] The paper states: 'We treat the GTS with Successive target as the most realistic and closest to production use, and compare all other splits against it.' All subsequent conclusions about which splits are 'suitable' or 'misaligned with real-world scenarios' are therefore correlations with this reference, not with observed production behavior. No online data, logged-bandit replay, or other external evidence is provided to validate Successive as the ground truth. If a different reference (e.g., an interaction-weighted protocol or a serving-based target) were chosen, the relative ranking of Last, Random, First, and All could change. This is a load-bearing assumption because the headline recommendation—that Last and Random are practical alternatives to Successive—depends entirely on the Successive reference. The authors should either provide external evidence for the Successive reference or explicitly reframe the conclusions as measuring agreement with Successive rather than closeness to production.
  2. [Section 4.2.1] The First target is characterized as biased because the first post-cutoff interaction follows a longer-than-typical inactivity gap, described as a 'session-boundary artifact.' However, in a deployed system the first user action after a model update is precisely an event the model must serve, and long inactivity gaps are a genuine serving condition. The analysis presumes that a realistic target should imitate the full-data inter-interaction-gap distribution, which is exactly the Successive assumption being tested. Thus, the time-gap evidence does not by itself establish that First is unrealistic; it only shows that First has a different gap distribution. The paper should acknowledge that this is an evaluative choice, not an empirical finding, and soften the claim that First is 'biased' without an external serving objective.
  3. [Section 4.6, Table 8] The retraining analysis is based on only two datasets (Diginetica and Amazon Beauty) and only one model (SASRec+). Yet the conclusion states that 'retraining on the combined training and validation data boosts final test performance for the reasonable validation options,' and the text generalizes about LTI and LOO experience 'more frequent performance drops.' This evidence is too thin for such general claims. The authors should either add more datasets and models to the retraining study, or restrict the conclusions to the two datasets and model actually examined.
  4. [Section 4.3 and Section 4.0.2] GTS Random is recommended as the highest-correlation alternative to Successive, but the analysis uses a single random draw of the target for each user and offers no repeated-seed variance. The paper acknowledges the non-determinism and suggests running multiple seeds, but it does not report whether the high correlation is stable across seeds. Without this, a reader cannot tell whether the high correlation of Random is a robust property or specific to one draw. The authors should provide seed-averaged correlations with variances, or at least a small repeated-seed study, before recommending Random as a reliable evaluation protocol.
minor comments (6)
  1. [Section 4.4] The text says 'our correlation analysis (Section 4.2.1)' but the correlation analysis is in Section 4.3; Section 4.2.1 is about time gaps. This reference should be corrected.
  2. [Section 4.5.1] The text says 'we follow the same approach as in Section 4.2.1' but should refer to Section 4.3. Please fix the cross-reference.
  3. [Section 4.2.1] The sentence 'The gap for the First target is much larger than for other targets, and the time gap across the dataset, which makes this target biased' is grammatically garbled and should be rewritten, e.g., 'The gap for the First target is much larger than for other targets, and differs from the overall dataset gap distribution, which we interpret as a bias for this target.'
  4. [Section 4.0.2] The definition of the interaction quantile q0.9 is not fully specified. It should state explicitly whether the cutoff is the 90th percentile of interaction timestamps, user first-activity times, or some other choice, since the split properties depend on this definition.
  5. [Section 4.0.2] The statement 'For GTS with GT and UB validation, we use the Last target as a reasonable and deterministic choice' seems to conflict with Section 4.5, where GT validation is evaluated with First, Random, Successive, and All targets as well. Clarify that the statement applies to the main experiments (RQ3/RQ4) and not to the validation-strategy comparison.
  6. [Section 4.3.3] Table 6 reports averaged correlations across datasets, but the number of configurations per model is not uniform (108 for SASRec+/BERT4Rec, 104 for GRU4Rec). It would be helpful to state whether the averages are weighted by model or by configuration, though this is a minor point.

Circularity Check

1 steps flagged · score 4.0 of 10

Central empirical comparison is self-contained, but the normative recommendation that GTS Last/Random are close-to-reality alternatives reduces to an author-defined Successive reference.

  1. self definitional [Section 4.3 (Consistency between different splits) and Section 5 (Conclusion)]
    "We treat the GTS with Successive target as the most realistic and closest to production use, and compare all other splits against it, suggesting that an appropriate split for next-item prediction should exhibit high correlation with this reference."

    The paper defines an 'appropriate split' as one with high correlation to the GTS Successive target, whose realism is asserted ('most realistic and closest to production use') rather than validated against online, logged-bandit, or other external production data. All normative conclusions—LOO is misaligned, First/All are less appropriate, Last/Random are suitable alternatives—are then read off from correlations against this self-declared reference. The correlation values themselves are genuine measurements, but the conclusion that Last/Random are 'close-to-reality' is a definitional transfer: it inherits the asserted realism of Successive without independent evidence, so the practical recommendation is an artifact of the chosen definition rather than an externally validated prediction.

full rationale

The paper's core descriptive contribution—that the choice of splitting strategy changes absolute metrics and model rankings (RQ1–RQ6, Tables 3–8)—is a self-contained empirical comparison on eight datasets with three baselines and public code, so it does not reduce to any fitted parameter, imported uniqueness theorem, or load-bearing self-citation. The authors' use of their own RePlay framework for metric computation is code-reproduced and not circular in argument. However, the normative layer does contain a definitional circularity: Section 4.3 declares Successive 'the most realistic and closest to production use', then defines a suitable split as one with high correlation to this reference; Section 5 then converts measured correlations into claims about 'close-to-reality' alignment. Because the realism of Successive is assumed rather than established, the headline recommendation that Last/Random are realistic alternatives is conditional on the authors' own definition and would not be forced by an external production ground truth. This affects the practical recommendation but not the central empirical finding about split sensitivity; hence score 4. The absence of online or logged-bandit validation of the Successive reference is a correctness and validity risk, not an additional source of circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No fitted model parameters or invented entities. The claims are empirical and depend on protocol choices and domain assumptions listed above. The main caveat is the author-defined Successive reference and the sensitivity of GTS conclusions to the chosen quantile.

free parameters (2)
  • GTS quantile q=0.9 for main experiments
    Chosen to balance test user count, test duration, and training data; Table 4 shows sensitivity to this choice.
  • User-based validation sample size = 1024 users
    Selected for GTS with UB validation; affects validation-subset statistics and training data reduction.
assumptions (4)
  • domain assumption GTS with Successive target is the most realistic evaluation protocol
    Used as the reference in correlation analysis (Section 4.3); all split recommendations are derived from agreement with it.
  • domain assumption Preserving a global timeline is necessary for real-world relevance
    Section 3.0.1 states any interaction after T_test must be excluded from training; this premise motivates the entire comparison.
  • domain assumption Kendall and Spearman correlations over model hyperparameter configurations are valid measures of split alignment
    The paper uses up to 108 configurations per model as evaluation points; these are not independent samples, which could affect correlation estimates.
  • domain assumption Unsampled ranking metrics are appropriate for evaluation
    Section 4.0.2 chooses unsampled NDCG, MRR, and HR following best practices; this choice affects all metric correlations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders." pith.science (2026). https://pith.science/paper/PZ7VXQ4G

@misc{pith2026250716289,
  author       = {Pith},
  title        = {Pith review of: Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZ7VXQ4G}},
  note         = {Machine review of arXiv:2507.16289}
}
read the original abstract

Modern sequential recommender systems, ranging from lightweight transformer-based variants to large language models, have become increasingly prominent in academia and industry due to their strong performance in the next-item prediction task. Yet common evaluation protocols for sequential recommendations remain insufficiently developed: they often fail to reflect the corresponding recommendation task accurately, or are not aligned with real-world scenarios. Although the widely used leave-one-out split matches next-item prediction, it permits the overlap between training and test periods, which leads to temporal leakage and unrealistically long test horizon, limiting real-world relevance. Global temporal splitting addresses these issues by evaluating on distinct future periods. However, its applications to sequential recommendations remain loosely defined, particularly in terms of selecting target interactions and constructing a validation subset that provides necessary consistency between validation and test metrics. In this paper, we demonstrate that evaluation outcomes can vary significantly across splitting strategies, influencing model rankings and practical deployment decisions. To improve reproducibility in both academic and industrial settings, we systematically compare different splitting strategies for sequential recommendations across multiple datasets and established baselines. Our findings show that prevalent splits, such as leave-one-out, may be insufficiently aligned with more realistic evaluation strategies. Code: https://github.com/monkey0head/time-to-split

Figures

Figures reproduced from arXiv: 2507.16289 by the authors.

Figure 1
Figure 1. Data splitting and target selection strategies for sequential recommendations. (a) Leave-one-out split. (b) Global [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Validation split options for GTS (Fig. 1b): (a) each [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Scatterplots for NDCG@10 between GTS Sucv. target [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distribution of time gaps 𝛿 between all interactions (Full data), and between target interaction and previous one for the First (left) and all others (right) target options for GTS on Zvuk dataset. 0.01 0.02 0.03 0.00 0.01 0.02 0.03 LOO K: 0.64 S: 0.83 Sports SASRec GR…
Figure 6
Figure 6. Figure 6: Kendall correlation between test NDCG@K for GTS with Successive target and other options. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Model rankings based on test NDCG@10 for LOO [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Kendall correlation between test and validation NDCG@K for GTS Last split with different validation strategies. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 39 canonical work pages

  1. [1]

    Vito Walter Anelli, Alejandro Bellogin, Antonio Ferrara, Daniele Malitesta, Fe- lice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021. Elliot: A Comprehensive and Rigorous Framework for Repro- ducible Recommender Systems Evaluation. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in In...

    arXiv 2021

  2. [2]

    David Ben-Shimon, Alexander Tsikinovsky, Michael Friedmann, Bracha Shapira, Lior Rokach, and Johannes Hoerle. 2015. RecSys Challenge 2015 and the YOO- CHOOSE Dataset. 357–358. doi:10.1145/2792838.2798723

    work page arXiv 2015

  3. [3]

    Rocío Cañamares and Pablo Castells. 2020. On Target Item Sampling in Offline Recommender System Evaluation. 259–268. doi:10.1145/3383313.3412259

    arXiv 2020

  4. [4]

    Huiyuan Chen, Yusan Lin, Menghai Pan, Lan Wang, Chin-Chia Michael Yeh, Xiaoting Li, Yan Zheng, Fei Wang, and Hao Yang. 2022. Denoising Self-attentive Sequential Recommendation

    work page 2022

  5. [5]

    Junyi Chen, Lu Chi, Bingyue Peng, and Zehuan Yuan. 2024. Hllm: Enhancing sequential recommendations via hierarchical large language models for item and user modeling. arXiv preprint arXiv:2409.12740 (2024)

    arXiv 2024

  6. [6]

    Ziqiang Cui, Haolun Wu, Bowei He, Ji Cheng, and Chen Ma. 2024. Diffusion- based Contrastive Learning for Sequential Recommendation. arXiv preprint arXiv:2405.09369 (2024)

    work page Pith review arXiv 2024

  7. [7]

    Alexander Dallmann, Daniel Zoller, and Andreas Hotho. 2021. A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models. In Fifteenth ACM Conference on Recommender Systems (RecSys ’21) . ACM. doi:10.1145/3460231.3475943

    work page arXiv 2021

  8. [8]

    Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7, Jan (2006), 1–30

    work page 2006

Show all 79 references
  1. [9]

    Sheng, Yanchi Liu, Guanfeng Liu, and Lei Zhao

    Hanwen Du, Hui Shi, Pengpeng Zhao, Deqing Wang, Victor S. Sheng, Yanchi Liu, Guanfeng Liu, and Lei Zhao. 2022. Contrastive Learning with Bidirectional Transformers for Sequential Recommendation. arXiv:2208.03895 [cs.IR]

    2022 arXiv

  2. [10]

    Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Zhao, Xing Xie, and Ji-Rong Wen

  3. [12]

    Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommen- dation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (Copenhagen, Denmark) (RecSys ’19). Asso...

    2019

  4. [13]

    Hamed Firooz, Maziar Sanjabi, Adrian Englhardt, Aman Gupta, Ben Levine, Dre Olgiati, Gungor Polatkan, Iuliia Melnychuk, Karthik Ramgopal, Kirill Talanine, et al. 2025. 360brew: A decoder-only foundation model for personalized ranking and recommendation. arXiv preprint arXiv:25...

    2025

  5. [14]

    Evgeny Frolov, Tatyana Matveeva, Leyla Mirvakhabova, and Ivan Oseledets. 2024. Self-Attentive Sequential Recommendations with Hyperbolic Representations. (2024)

    2024

  6. [15]

    Scott Graham, Jun-Ki Min, and Tao Wu. 2019. Microsoft recommenders: tools to accelerate developing recommender systems. In Proceedings of the 13th ACM Conference on Recommender Systems (Copenhagen, Denmark) (Rec- Sys ’19). Association for Computing Machinery, New York, NY, USA...

    2019

  7. [16]

    Asela Gunawardana, Guy Shani, and Sivan Yogev. 2012. Evaluating recommender systems. In Recommender systems handbook. Springer, 547–601

    2012

  8. [17]

    Danil Gusak, Gleb Mezentsev, Ivan Oseledets, and Evgeny Frolov. 2024. RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders. Proceedings of 33rd ACM International Conference on Information and Knowledge Management (CIKM ’24). doi:10.1145/3627673.3679986

    2024

  9. [18]

    F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5, 4 (2015), 1–19

    2015

  10. [19]

    Jesse Harte, Wouter Zorgdrager, Panos Louridas, Asterios Katsifodimos, Diet- mar Jannach, and Marios Fragkoulis. 2023. Leveraging large language models for sequential recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems. 1096–1102

    2023

  11. [20]

    Ruining He, Wang-Cheng Kang, and Julian McAuley. 2017. Translation-based recommendation. In Proceedings of the eleventh ACM conference on recommender systems. 161–169

    2017

  12. [21]

    Ruining He and Julian McAuley. 2016. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th international conference on data mining (ICDM) . IEEE, 191–200

    2016

  13. [22]

    Balázs Hidasi and Ádám Tibor Czapp. 2023. The effect of third party imple- mentations on reproducibility. In Proceedings of the 17th ACM Conference on Recommender Systems. 272–282

    2023

  14. [23]

    Balázs Hidasi and Ádám Tibor Czapp. 2023. Widespread Flaws in Offline Evalu- ation of Recommender Systems. In Proceedings of the 17th ACM Conference on Recommender Systems. 848–855

    2023

  15. [24]

    Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk

  16. [26]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (Nov. 1997), 1735–1780. doi:10.1162/neco.1997.9.8.1735

    1997 doi

  17. [27]

    Dietmar Jannach, Massimo Quadrana, and Paolo Cremonesi. 2022. Session-Based Recommender Systems. Springer US, New York, NY, 301–334. doi:10.1007/978-1- 0716-2197-4_8

    2022 doi

  18. [28]

    Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2023. A critical study on data leakage in recommender system offline evaluation. ACM Transactions on Information Systems 41, 3 (2023), 1–27

    2023

  19. [29]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In 2018 IEEE international conference on data mining (ICDM) . IEEE, 197–206

    2018

  20. [30]

    Anton Klenitskiy and Alexey Vasilev. 2023. Turning dross into gold loss: is bert4rec really better than sasrec?. In Proceedings of the 17th ACM Conference on Recommender Systems. 1120–1125

    2023

  21. [31]

    Anton Klenitskiy, Anna Volodkevich, Anton Pembek, and Alexey Vasilev. 2024. Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations. In Proceedings of the 18th ACM Conference on Recommender Systems. 1067–1072

    2024

  22. [32]

    Walid Krichene and Steffen Rendle. 2020. On Sampled Metrics for Item Recom- mendation. In KDD 2020. https://dl.acm.org/doi/10.1145/3394486.3403226

    2020

  23. [33]

    Sara Latifi, Dietmar Jannach, and Andres Ferraro. 2022. Sequential Recom- mendation: A Study on Transformers, Nearest Neighbors and Sampled Metrics. Information Sciences 609 (07 2022). doi:10.1016/j.ins.2022.07.079

    2022 doi

  24. [34]

    Jiayu Li, Hanyu Li, Zhiyu He, Weizhi Ma, Peijie Sun, Min Zhang, and Shaoping Ma. 2024. Rechorus2. 0: A modular and task-flexible recommendation library. In Proceedings of the 18th ACM Conference on Recommender Systems . 454–464

    2024

  25. [35]

    Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time Interval Aware Self- Attention for Sequential Recommendation. 322–330. doi:10.1145/3336191.3371786

    2020

  26. [36]

    Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. 2023. GPT4Rec: A generative framework for personalized recommen- dation and user interests interpretation. arXiv preprint arXiv:2304.03879 (2023)

    2023 arXiv

  27. [37]

    Chang Liu, Xiaoguang Li, Guohao Cai, Zhenhua Dong, Hong Zhu, and Lifeng Shang. 2021. Non-invasive Self-attention for Side Information Fusion in Sequen- tial Recommendation. arXiv:2103.03578 [cs.IR]

    2021 arXiv

  28. [38]

    Malte Ludewig and Dietmar Jannach. 2018. Evaluation of session-based recom- mendation algorithms. User Modeling and User-Adapted Interaction 28 (2018), 331–390

    2018

  29. [39]

    Julian McAuley, Jure Leskovec, and Dan Jurafsky. 2012. Learning attitudes and attributes from multi-aspect reviews. In 2012 IEEE 12th International Conference on Data Mining. IEEE, 1020–1025

    2012

  30. [40]

    Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based Recommendations on Styles and Substitutes. arXiv:1506.04757 [cs.CV]

    2015 arXiv

  31. [41]

    Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2020. Ex- ploring data splitting strategies for the evaluation of recommendation models. In Proceedings of the 14th acm conference on recommender systems . 681–686

    2020

  32. [42]

    Gleb Mezentsev, Danil Gusak, Ivan Oseledets, and Evgeny Frolov. 2024. Scalable cross-entropy loss for sequential recommendations with large item catalogs. In Proceedings of the 18th ACM Conference on Recommender Systems . 475–485

    2024

  33. [43]

    Lien Michiels, Robin Verachtert, and Bart Goethals. 2022. RecPack: An(other) Experimentation Toolkit for Top-N Recommendation using Implicit Feedback Data. In Proceedings of the 16th ACM Conference on Recommender Systems (Seattle, WA, USA) (RecSys ’22). Association for Computi...

    2022

  34. [44]

    Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- former: Sequence modeling for user representation at pinterest. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining . 3702–3712

    2022

  35. [45]

    Aleksandr Petrov and Craig Macdonald. 2022. A systematic review and replica- bility study of bert4rec for sequential recommendation. In Proceedings of the 16th ACM Conference on Recommender Systems . 436–447

    2022

  36. [46]

    Aleksandr Vladimirovich Petrov and Craig Macdonald. 2023. gsasrec: Reducing overconfidence in sequential recommendation trained with negative sampling. In Proceedings of the 17th ACM Conference on Recommender Systems . 116–128

    2023

  37. [47]

    Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. 2022. Contrastive Learn- ing for Representation Degeneration Problem in Sequential Recommendation. 813–823. doi:10.1145/3488560.3498433

    2022

  38. [48]

    Mostafa Rahmani, James Caverlee, and Fei Wang. 2023. Incorporating time in sequential recommendation models. In Proceedings of the 17th ACM Conference on Recommender Systems. 784–790

    2023

  39. [49]

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al

  40. [50]

    Noveen Sachdeva and Julian McAuley. 2020. How Useful are Reviews for Recom- mendation? A Critical Review and Potential Improvements. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval (Virtual Event, China) (SIG...

    2020

  41. [51]

    Aghiles Salah, Quoc-Tuan Truong, and Hady W. Lauw. 2020. Cornac: a compara- tive framework for multimodal recommender systems. J. Mach. Learn. Res. 21, 1, Article 95 (Jan. 2020), 5 pages

    2020

  42. [52]

    Teresa Scheidt and Joeran Beel. 2021. Time-dependent Evaluation of Recom- mender Systems.. In Perspectives@ RecSys

    2021

  43. [53]

    Valeriy Shevchenko, Nikita Belousov, Alexey Vasilev, Vladimir Zholobov, Artyom Sosedka, Natalia Semenova, Anna Volodkevich, Andrey Savchenko, and Alexey Zaytsev. 2024. From variability to stability: Advancing RecSys benchmarking practices. In Proceedings of the 30th ACM SIGKDD...

    2024

  44. [54]

    Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, et al. 2024. Better generalization with semantic ids: A case study in ranking for recommendations. In Proceedings of the 18th AC...

    2024

  45. [55]

    Aixin Sun. 2023. Take a fresh look at recommender systems from an evaluation standpoint. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2629–2638

    2023

  46. [56]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang

  47. [57]

    Zhu Sun, Hui Fang, Jie Yang, Xinghua Qu, Hongyang Liu, Di Yu, Yew-Soon Ong, and Jie Zhang. 2023. DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation. IEEE Trans. Pattern Anal. Mach. Intell. 45, 7 (July 2023), 8206–8226. doi:10.1109/TPAMI.2022.3231891

    2023

  48. [58]

    Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng

  49. [59]

    Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommenda- tion via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining . 565–573

    2018

  50. [60]

    Xuewen Tao, Mingming Ha, Qiongxu Ma, Hongwei Cheng, Wenfang Lin, Xiaobo Guo, Linxun Cheng, and Bing Han. 2023. Task aware feature extraction frame- work for sequential dependence multi-task learning. In Proceedings of the 17th ACM Conference on Recommender Systems . 151–160

    2023

  51. [61]

    Alexey Vasilev, Anna Volodkevich, Denis Kulandin, Tatiana Bysheva, and Anton Klenitskiy. 2024. RePlay: a Recommendation Framework for Experimentation and Production Use. In Proceedings of the 18th ACM Conference on Recommender Systems (Bari, Italy) (RecSys ’24). Association fo...

    2024

  52. [62]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  53. [63]

    Robin Verachtert, Lien Michiels, and Bart Goethals. 2022. Are We Forgetting Something? Correctly Evaluate a Recommender System With an Optimal Train- ing Window.. In Perspectives@ RecSys

    2022

  54. [64]

    Anna Volodkevich, Danil Gusak, Anton Klenitskiy, and Alexey Vasilev. 2024. Autoregressive Generation Strategies for Top-K Sequential Recommendations. arXiv preprint arXiv:2409.17730 (2024)

    2024 arXiv

  55. [65]

    Lukas Wegmeth, Tobias Vente, Lennart Purucker, and Joeran Beel. 2023. The Effect of Random Seeds for Data Splitting on Recommendation Accuracy.. In Perspectives@ RecSys

    2023

  56. [66]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al

  57. [67]

    Wujiang Xu, Zujie Liang, Jiaojiao Han, Xuying Ning, Wenfang Lin, Linxun Chen, Feng Wei, and Yongfeng Zhang. 2024. Slmrec: empowering small language models for sequential recommendation. arXiv e-prints (2024), arXiv–2405

    2024

  58. [68]

    Yufei Ye, Wei Guo, Jin Yao Chin, Hao Wang, Hong Zhu, Xi Lin, Yuyang Ye, Yong Liu, Ruiming Tang, Defu Lian, et al. 2025. FuXi-alpha: Scaling Recommen- dation Model with Feature Interaction Enhanced Transformer. arXiv preprint arXiv:2502.03036 (2025)

    2025 arXiv

  59. [69]

    Xiaohan Yu, Li Zhang, Xin Zhao, and Yue Wang. 2024. Break the ID-Language Barrier: An Adaption Framework for Sequential Recommendation. arXiv preprint arXiv:2411.18262 (2024)

    2024

  60. [70]

    Huimin Zeng, Xiaojie Wang, Anoop Jain, Zhicheng Dou, and Dong Wang. 2025. A non-contrastive learning framework for sequential recommendation with preference preserving profile generation. (2025)

    2025

  61. [71]

    Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhao- jie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152 (2024)

    2024 arXiv

  62. [72]

    arXiv preprint arXiv:1910.03771 (2019)

    Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)

    2019 arXiv

  63. [73]

    Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2021. RecBole: Towards a Unified, Com...

    2021

  64. [74]

    Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. 2021. Open Benchmarking for Click-Through Rate Prediction. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021 , ...

    2021

  65. [78]

    Wayne Xin Zhao, Zihan Lin, Zhichao Feng, Pengfei Wang, and Ji-Rong Wen. 2022. A revisiting study of appropriate offline evaluation for top-N recommendation algorithms. ACM Transactions on Information Systems 41, 2 (2022), 1–41

    2022

  66. [2015]

    arXiv preprint arXiv:1511.06939 (2015)

    Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015)

    2015 arXiv

  67. [2019]

    In Proceedings of the 28th ACM international conference on information and knowledge management

    BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management . 1441–1450

  68. [2020]

    In Proceedings of the 14th ACM Confer- ence on Recommender Systems (Virtual Event, Brazil) (RecSys ’20)

    Are We Evaluating Rigorously? Benchmarking Recommendation for Repro- ducible Evaluation and Fair Comparison. In Proceedings of the 14th ACM Confer- ence on Recommender Systems (Virtual Event, Brazil) (RecSys ’20). Association for Computing Machinery, New York, NY, USA, 23–32. ...

  69. [2021]

    1733–1737

    Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation. 1733–1737. doi:10.1145/3404835.3462978

  70. [2023]

    Advances in Neural Information Processing Systems 36 (2023), 10299–10315

    Recommender systems with generative retrieval. Advances in Neural Information Processing Systems 36 (2023), 10299–10315

    2023

  71. [2769]

    doi:10.1145/3459637.3482486

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.