{"id":"7046031f-4d2b-41d4-accc-5cfa42a4db3b","arxiv_id":"2507.16289","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Global temporal splits with Last or Random target selection correlate strongly with realistic successive evaluation, while leave-one-out splits produce inconsistent model rankings across datasets.","lead":"A systematic study of offline data splitting for sequential recommenders shows that the choice of split changes model rankings and metric values. The authors recommend global temporal splits with Last or Random targets over the common leave-one-out split for more realistic evaluation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The practical recommendation that GTS Last/Random are close-to-reality alternatives to Successive rests on an unvalidated Successive reference; the paper's time-gap analysis treats a real serving condition (prediction after inactivity) as bias, so online or replay data is needed before adopting…","rationale":"The reader's weakest assumption identifies exactly the unvalidated Successive reference, and I agree that this is the load-bearing point for the paper's practical recommendations. The descriptive claim—that different splits yield different metrics and model rankings—is convincingly supported by the broad experiments, the correlation tables, and the ranking figures, so no objection is raised against that claim. The concern is specifically about the normative step: the paper repeatedly frames GTS Last and GTS Random as 'close-to-reality' or 'more realistic' alternatives, but this framing is operationalized as correlation with Successive, which is itself asserted to be the production-like protocol. The time-gap analysis in Section 4.2 makes the assumption visible: a long gap before the First target is called a bias, yet predicting after a long gap is a real event in production. Similarly, the decision to user-average Successive metrics encodes a value judgment about equal user weight that may not match live traffic. None of this is internally inconsistent, and the paper does acknowledge the reference is an assumption in Section 4.3, but the acknowledgment is brief and the conclusions in Sections 4.4 and 5 are stated more strongly than the evidence supports. A temporal replay or logged production comparison would settle whether Successive, and hence Last/Random, actually track online model rankings better than LOO or First. Until such a check is performed, the practical recommendations should be treated as conditional, which is consistent with the reader's CONDITIONAL verdict. The code repository is promised but not verifiable from the text, so confidence should remain moderate.","tokens_in":20480,"tokens_out":6187,"duration_ms":72387,"concrete_test":"Run a temporal replay evaluation on one or two datasets (e.g., Diginetica and Zvuk): after the GTS cutoff, for every user interaction timestamp, generate a recommendation list using only interactions strictly before that timestamp, and compute NDCG@10 over all served events under both interaction-weighted and user-averaged aggregation. Rank the same model configurations under this replay protocol and under LOO, GTS Last, GTS Random (multiple seeds), GTS Successive, and GTS First. If replay rankings correlate with Successive/GTS Last/GTS Random significantly better than with LOO/GTS First, the reference assumption is empirically supported; if not, the paper's practical recommendations require an external ground truth or a narrowed scope.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The descriptive claim that splitting strategy changes metrics and model rankings is well supported by the experiments. The load-bearing weakness is in the normative inference drawn from those experiments. Section 4.3 states: 'We treat the GTS with Successive target as the most realistic and closest to production use, and compare all other splits against it.' Every subsequent conclusion about which split is 'more realistic' or 'misaligned with real-world scenarios' is therefore a correlation with this author-defined reference, not with an observed production ground truth. Section 4.2.1 illustrates why this matters: the First target is dismissed as biased because the first post-cutoff interaction follows a longer-than-typical inactivity gap. But in actual deployment, the first user action after a model update is exactly an event the system must serve, and long inactivity gaps are a genuine serving condition. The paper's characterization of this as a 'session-boundary artifact' presumes that a realistic target should imitate the full-data inter-interaction-gap distribution—which is the same Successive assumption being tested. Additionally, Successive uses per-user averaging (Section 4.0.2), a choice that treats each user equally, whereas real production traffic is typically interaction-weighted. No online data, logged-bandit replay, or other external evidence is provided to validate Successive as the reference. Thus the paper's practical recommendations (Last/Random as alternatives to Successive) are conditional on an unverified assumption; the central descriptive finding is not undermined, but the deployment-oriented conclusions are weaker than the text suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the choice of data splitting strategy affects offline evaluation of sequential recommender systems. It compares leave-one-out (LOO) splitting with several variants of global temporal splitting (GTS), differing in how target items are selected from the holdout sequence (Last, First, Random, Successive, All) and how the validation set is constructed (global temporal, user-based, last training item). Using eight datasets and three neural baselines (plus a non-neural kNN in the ranking analysis) with many hyperparameter configurations, the authors show that the choice of split changes absolute metric values and, in many cases, the relative ranking of models. They further analyze time gaps between target interactions and previous events, correlation of test metrics across splits, correlation between validation and test metrics, and the effect of retraining on combined training and validation data. The central descriptive claim—that split choice significantly impacts metrics and model rankings—is well supported. However, the paper's prescriptive conclusions, especially that GTS Last and GTS Random are practical alternatives to GTS Successive, rest on treating Successive as the 'most realistic' reference without external validation.","tokens_in":20837,"tokens_out":4258,"duration_ms":45297,"significance":"If the descriptive findings are taken at face value, the paper makes a useful contribution by documenting that common offline evaluation protocols for sequential recommenders are not interchangeable: model rankings change across splits, and LOO can diverge substantially from global temporal evaluation. The empirical scope is a clear strength: eight datasets, multiple models, hundreds of hyperparameter configurations, unsampled ranking metrics, and released code. The formalization of GTS target and validation options is also a practical contribution for the community, and the framework survey (Section 3.0.2) is informative. The main weakness is that the normative recommendations are derived from correlation with an author-defined reference (GTS Successive) that is asserted, not validated, to be the closest to production. The retraining analysis is also much thinner than the rest of the study. These issues are load-bearing for the paper's practical conclusions, and the claims of 'real-world alignment' would need either external evidence or a more careful framing as 'agreement with Successive.'","major_comments":[{"comment":"The paper states: 'We treat the GTS with Successive target as the most realistic and closest to production use, and compare all other splits against it.' All subsequent conclusions about which splits are 'suitable' or 'misaligned with real-world scenarios' are therefore correlations with this reference, not with observed production behavior. No online data, logged-bandit replay, or other external evidence is provided to validate Successive as the ground truth. If a different reference (e.g., an interaction-weighted protocol or a serving-based target) were chosen, the relative ranking of Last, Random, First, and All could change. This is a load-bearing assumption because the headline recommendation—that Last and Random are practical alternatives to Successive—depends entirely on the Successive reference. The authors should either provide external evidence for the Successive reference or explicitly reframe the conclusions as measuring agreement with Successive rather than closeness to production.","section":"Section 4.3"},{"comment":"The First target is characterized as biased because the first post-cutoff interaction follows a longer-than-typical inactivity gap, described as a 'session-boundary artifact.' However, in a deployed system the first user action after a model update is precisely an event the model must serve, and long inactivity gaps are a genuine serving condition. The analysis presumes that a realistic target should imitate the full-data inter-interaction-gap distribution, which is exactly the Successive assumption being tested. Thus, the time-gap evidence does not by itself establish that First is unrealistic; it only shows that First has a different gap distribution. The paper should acknowledge that this is an evaluative choice, not an empirical finding, and soften the claim that First is 'biased' without an external serving objective.","section":"Section 4.2.1"},{"comment":"The retraining analysis is based on only two datasets (Diginetica and Amazon Beauty) and only one model (SASRec+). Yet the conclusion states that 'retraining on the combined training and validation data boosts final test performance for the reasonable validation options,' and the text generalizes about LTI and LOO experience 'more frequent performance drops.' This evidence is too thin for such general claims. The authors should either add more datasets and models to the retraining study, or restrict the conclusions to the two datasets and model actually examined.","section":"Section 4.6, Table 8"},{"comment":"GTS Random is recommended as the highest-correlation alternative to Successive, but the analysis uses a single random draw of the target for each user and offers no repeated-seed variance. The paper acknowledges the non-determinism and suggests running multiple seeds, but it does not report whether the high correlation is stable across seeds. Without this, a reader cannot tell whether the high correlation of Random is a robust property or specific to one draw. The authors should provide seed-averaged correlations with variances, or at least a small repeated-seed study, before recommending Random as a reliable evaluation protocol.","section":"Section 4.3 and Section 4.0.2"}],"minor_comments":[{"comment":"The text says 'our correlation analysis (Section 4.2.1)' but the correlation analysis is in Section 4.3; Section 4.2.1 is about time gaps. This reference should be corrected.","section":"Section 4.4"},{"comment":"The text says 'we follow the same approach as in Section 4.2.1' but should refer to Section 4.3. Please fix the cross-reference.","section":"Section 4.5.1"},{"comment":"The sentence 'The gap for the First target is much larger than for other targets, and the time gap across the dataset, which makes this target biased' is grammatically garbled and should be rewritten, e.g., 'The gap for the First target is much larger than for other targets, and differs from the overall dataset gap distribution, which we interpret as a bias for this target.'","section":"Section 4.2.1"},{"comment":"The definition of the interaction quantile q0.9 is not fully specified. It should state explicitly whether the cutoff is the 90th percentile of interaction timestamps, user first-activity times, or some other choice, since the split properties depend on this definition.","section":"Section 4.0.2"},{"comment":"The statement 'For GTS with GT and UB validation, we use the Last target as a reasonable and deterministic choice' seems to conflict with Section 4.5, where GT validation is evaluated with First, Random, Successive, and All targets as well. Clarify that the statement applies to the main experiments (RQ3/RQ4) and not to the validation-strategy comparison.","section":"Section 4.0.2"},{"comment":"Table 6 reports averaged correlations across datasets, but the number of configurations per model is not uniform (108 for SASRec+/BERT4Rec, 104 for GRU4Rec). It would be helpful to state whether the averages are weighted by model or by configuration, though this is a minor point.","section":"Section 4.3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a revised version of a RecSys 2025 paper, and the scope is an empirical study rather than a methods paper. The central descriptive claim is solid and likely useful to the community. The main risk is that the prescriptive claims, if adopted as guidance, would inherit the unvalidated Successive reference assumption. In a journal version, I would expect either external validation (even a small logged-data study) or a more careful epistemic framing of the recommendations. The retraining section is also noticeably thinner than the rest and should be expanded or explicitly labeled as preliminary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a workmanlike empirical study of data splitting for sequential recommenders. The central claim—split choice changes metrics and model rankings—is convincingly supported. Eight datasets, three neural baselines plus a kNN, unsampled metrics, and a sensible taxonomy of GTS target options (Last, First, Random, Successive, All) and validation schemes (GT, UB, LTI). That taxonomy plus the systematic correlation analysis is the real contribution. It will be useful to anyone who has to pick an evaluation protocol.\n\nCredit where due: the split-statistics tables (training data, test period, holdout length, time gaps) are informative, and the finding that LOO leaves less training data than GTS on short-sequence datasets is a nice counterintuitive result. The paper does not oversell its novelty; it builds on Meng et al. and Ji et al.\n\nThe soft spot is exactly where the reader's report and the stress-test put it. Section 4.3 declares GTS Successive \"the most realistic and closest to production use\" and then measures everything against it. That is an evaluative assumption, not a result. It is probably a reasonable assumption for next-item prediction, but the paper offers no online or replay data to back it, and the First-target analysis illustrates the hazard: the paper dismisses First because the first post-cutoff interaction follows a longer inactivity gap, but in production that first interaction is exactly what the system must serve. The \"session-boundary artifact\" framing presumes the Successive gap distribution is the right one. Also, Successive uses per-user averaging, while real traffic is typically interaction-weighted. So the practical recommendation—Last and Random are near-Successive alternatives—is conditional. That does not sink the paper; it just means the deployment-oriented conclusions are weaker than the prose suggests.\n\nTwo minor issues: the retraining analysis (RQ6) is thin—two datasets, one model—and GTS Random has no repeated-seed variance, even though the paper names randomness as its reproducibility drawback. The correlation analysis also pools hyperparameter configurations as independent points, which likely inflates significance, though the ranking results in Figure 7 do not depend on that.\n\nOverall, the descriptive findings are solid and the paper is honestly written. The recommendations should be adopted with explicit reporting of the reference protocol and split randomness. Code is promised; I could not verify the repo. I would send it to review, and I would cite it in my own evaluation-protocol work.","headline":"A well-executed empirical study whose central descriptive claim holds; its practical recommendations are conditional on an unvalidated 'Successive as realistic' reference, but the paper deserves review.","tokens_in":21324,"tokens_out":1718,"would_cite":true,"duration_ms":19295,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The choice of data splitting strategy for sequential recommenders can significantly change evaluation metrics and model rankings, so offline comparisons should be read with the split protocol in mind.","keywords":["sequential recommender systems","offline evaluation","data splitting","leave-one-out split","global temporal split","next-item prediction","model ranking","evaluation protocol"],"falsifier":"Serve models selected under different splits (for example, the best model chosen by LOO versus by GTS-Successive) in a live recommender with logged user responses, or on a large withheld future time window with known next interactions, and compare actual next-item match or engagement. If the model selected by LOO matches or beats the model selected by GTS-Successive, then the Successive reference is not a reliable proxy and the paper's central comparison loses its anchor.","tokens_in":20246,"feed_emoji":"🔀","tokens_out":8446,"duration_ms":77957,"temperature":0.7,"pith_summary":"The paper argues that the way interaction data is split into training and test sets can change which sequential recommender system looks best in offline evaluation. The authors define a global temporal split tailored to next-item prediction, with five target-selection options (last, first, random, successive, and all held-out items) and three validation-set constructions (global temporal, last training item, and user-based). Comparing these against the widely used leave-one-out split on eight datasets and mainstream baselines, they find that leave-one-out correlates only moderately with the successive scheme they treat as closest to production, and that both absolute metrics and final model rankings shift across split choices. They conclude that global temporal splitting with a last-item or random-item target is a practical, reproducible alternative to the more expensive successive evaluation. If correct, the results imply that published rankings obtained under leave-one-out should not be assumed to hold in deployment.","feed_headline":"Split choice reorders recommender rankings in offline tests","feed_subtitle":"Leave-one-out picks different winners than realistic temporal splits, so published rankings may not transfer to production.","key_machinery":"The central object is the global temporal split for sequential recommenders, specified by a global cutoff time, a target-selection rule for each holdout sequence, and a validation-set construction; the reference protocol is the Successive target, where every interaction in a holdout sequence is scored as a separate next-item prediction with the preceding history as input, and metrics are averaged per user. The paper's measuring instrument is a systematic correlation analysis (Kendall and Spearman coefficients computed across models and hyperparameter configurations) between each split variant and this Reference, supplemented by an analysis of the time-gap distribution between target interactions and their predecessors, which explains why the First target is biased. This combination lets the authors identify which split components carry the most realistic signal and which are artifacts.","core_discovery":"Using GTS with the Successive target (each held-out interaction treated as a separate next-item prediction, averaged by user) as the reference for 'close to production' evaluation, the paper measures how other splits agree with it. Averaged across datasets, GTS Random has the highest Kendall/Spearman agreement (about 0.91/0.98 for NDCG@10), GTS Last is next (about 0.83/0.94), leave-one-out is lower (about 0.71/0.87), GTS First is similar to LOO or worse on some datasets due to large time gaps after the cutoff, and GTS All has very low correlation (about 0.57/0.53), exposing its mismatch with the next-item objective. Validation and retraining experiments show that global temporal validation aligns best with test metrics and that retraining the selected model on combined training-plus-validation data improves final test performance for global-temporal and user-based validation. The paper's central claim is that data-splitting strategy for sequential recommenders is not a neutral experimental detail: it can materially affect model rankings and therefore deployment decisions.","pith_inferences":["A natural extension the paper does not fully pursue is a deterministic random target: sample one per-user target with a fixed stored seed and publish it, which should combine Random's high correlation with Last's reproducibility.","The median time-gap analysis suggests a screening criterion for sequential-recommendation datasets: if the gap between the cutoff and the First post-cutoff interaction is a clear session boundary, that target is structurally biased, and datasets lacking clean session boundaries might behave differently.","The roughly 0.7 Kendall agreement between LOO and the realistic reference implies that LOO-based leaderboards have a substantial 'split-noise' floor even when metrics are computed exactly, which could be quantified per dataset as a variance component.","For industrial deployment, the retraining results imply that validation-guided early stopping under global temporal validation leaves performance on the table unless the final model is retrained on train-plus-validation; reporting only the unretrained number understates what the deployed model would achieve."],"forward_implications":["Published model rankings obtained with leave-one-out splits are less likely to transfer to temporal, production-like evaluation; meta-analyses that pool results across different split protocols will carry hidden ranking noise.","GTS with Last or Random target can substitute for the computationally expensive Successive evaluation in most offline studies, with Random requiring stored seeds or multi-seed averaging for reproducibility.","Validation sets for GTS should be built with a matching global temporal split, and final results should be reported both with and without retraining on the combined training and validation data, since retraining materially changes test scores.","Papers using GTS should report the cutoff quantile, target-selection rule, and input-sequence construction; without these details, results cannot be compared or reproduced.","Datasets with small holdout lengths or few active users after the cutoff may need a lower quantile to retain enough test users, trading off test-period duration against statistical power."],"supporting_citations":[{"why":"Documents temporal leakage in leave-one-out and motivates preserving a global timeline in offline evaluation.","marker":"[28]"},{"why":"Argues for evaluation on a global timeline and provides the successive-evaluation idea that anchors the paper's reference protocol.","marker":"[55]"},{"why":"Shows that the splitting strategy can change model rankings in top-N evaluation and introduces Kendall-correlation-based split agreement analysis.","marker":"[41]"},{"why":"Categorizes how experimental-protocol variations, including data leakage, produce inconsistent model rankings in recommender systems.","marker":"[23]"},{"why":"Supplies the SASRec+ baseline with full cross-entropy loss used in the experiments.","marker":"[30]"},{"why":"Provides dataset selection and analysis of sequential structure used to choose the eight datasets.","marker":"[31]"},{"why":"One of the recent works whose NIP-oriented target-selection practice (successive/last targets) is formalized by the paper.","marker":"[14]"},{"why":"Another recent work using last-item or successive targets for GTS-style evaluation of sequential recommenders.","marker":"[19]"},{"why":"Defines the common practice of treating all holdout items as a single target set, which the paper labels the All target and shows to be task-mismatched.","marker":"[16]"}],"fun_headline_variants":["Split strategy changes winner in recommender tests","Leave-one-out splits mislead sequential recommender rankings","Data split choice flips recommender model rankings","Temporal splits better mimic production for sequential recommenders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper states in Section 4.3 that the global temporal split with Successive target is treated as the most realistic and closest to production use, and all other splits are scored by how well they agree with that reference; there is no online or logged production data validating the reference itself. If Successive is not in fact the right proxy for real-world performance, the paper's recommendations about which targets and validation schemes to prefer could change.","fun_headline_variants_meta":{"raw":{"variants":["Split strategy changes winner in recommender tests","Leave-one-out splits mislead sequential recommender rankings","Data split choice flips recommender model rankings","Temporal splits better mimic production for sequential recommenders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1454,"prompt_tokens":994,"completion_tokens":460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":400}},"tokens_in":610,"tokens_out":460,"duration_ms":4754,"temperature":1.0,"reasoning_tokens":400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:12:52.106452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Serve models selected under different splits (for example, the best model chosen by LOO versus by GTS-Successive) in a live recommender with logged user responses, or on a large withheld future time window with known next interactions, and compare actual next-item match or engagement. If the model selected by LOO matches or beats the model selected by GTS-Successive, then the Successive reference is not a reliable proxy and the paper's central comparison loses its anchor.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents temporal leakage in leave-one-out and motivates preserving a global timeline in offline evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Argues for evaluation on a global timeline and provides the successive-evaluation idea that anchors the paper's reference protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that the splitting strategy can change model rankings in top-N evaluation and introduces Kendall-correlation-based split agreement analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Categorizes how experimental-protocol variations, including data leakage, produce inconsistent model rankings in recommender systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SASRec+ baseline with full cross-entropy loss used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides dataset selection and analysis of sequential structure used to choose the eight datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the recent works whose NIP-oriented target-selection practice (successive/last targets) is formalized by the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the common practice of treating all holdout items as a single target set, which the paper labels the All target and shows to be task-mismatched."}],"review_version":1}