{"id":"25e6b640-96fe-4309-80be-1eb3252c92b0","arxiv_id":"2506.14692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"With a common EasyRec implementation, BSARec consistently beats SASRec on MovieLens-1M and Foursquare-NYC, but the gains are smaller than the original BSARec paper reported.","lead":"This paper re-implements two sequential recommendation models, SASRec and BSARec, inside a shared library and compares them on two public datasets. It finds that BSARec still wins, but by smaller margins than originally claimed, which suggests implementation details influence reported performance gains.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies the evaluation protocol (split, candidate set, negative sampling), so the claimed smaller-than-original gains may be a protocol artifact; this must be settled before the central claim is trustworthy.","rationale":"In good faith, the paper is a comparative replicability study: its contribution is the claim that, with a common implementation, BSARec's frequency-enhancement bias still helps but less than originally claimed. The condition that must hold is that the comparison is internally fair and externally aligned with the original papers. The least secure part of that condition is the evaluation protocol: Section 3.1.1 omits the split, candidate set, and sampling procedure, and the paper's own motivating citations show these details are decisive. I also noticed the internal inconsistency that the reported best alpha=0.3 for fs-nyc is not in the stated grid [0.1, 0.5, 0.7, 0.9]; this is likely a typo but reinforces the need for configs. I did not base the verdict on that alone because it is fixable without changing the substantive claim. The absence of multiple seeds and significance testing is a secondary concern; even with a fixed protocol, significance would be needed, but the protocol issue is more fundamental because it affects what the numbers mean. Since the issue is addressable by releasing code and stating the protocol, the reader's CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":6260,"tokens_out":9426,"duration_ms":95022,"concrete_test":"Release the EasyRec configs and re-run Table 1 under the exact protocol of Shin et al. (same train/validation/test split, same sequence filtering, and the same candidate generation, e.g., leave-one-out with 100 sampled negatives per user). The key single check is to switch EasyRec's candidate generation from its default to the original 100-negative leave-one-out protocol and recompute the relative gains. If the reported 3.7-5.7% (ml-1m) and 10.3-14.3% (fs-nyc) gains change by more than a couple of points or reverse, the 'smaller gains' conclusion is protocol-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: BSARec beats SASRec, and the gains are smaller than Shin et al. reported. Both parts are only meaningful under a defined evaluation protocol, but Section 3.1.1 lists only datasets and metrics. It never states how the data were split into train/validation/test, how sequences were built or filtered, how the next-item candidate set was generated (full-item scoring vs. sampled negatives and how many), or how metrics were computed. The paper itself cites Klenitskiy & Vasilev and Petrov & Macdonald to argue that such choices can change both magnitude and sign of reported gains, so this omission is not cosmetic. Without code or a precise protocol statement, the head-to-head result and the comparison to the original results are not independently checkable; the paper therefore does not yet deliver the 'systematic replicability' study it announces, and the 'not as high as presented' conclusion may be an artifact of EasyRec's default protocol rather than a property of the models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a comparative study of two sequential recommender models, SASRec and BSARec, re-implemented within the EasyRec library so that both models share the same base PyTorch components. The authors claim that BSARec, with its frequency-enhancement bias terms, consistently outperforms SASRec on MovieLens-1m and Foursquare-nyc, but that the performance gains are smaller than those reported in the original BSARec paper (Shin et al., 2024). The study is motivated by concerns about inconsistent evaluation conditions in prior comparisons, and it emphasizes the importance of implementation details for fair model comparison.","tokens_in":6437,"tokens_out":2940,"duration_ms":29124,"significance":"If the central claim holds, the paper provides useful evidence about the magnitude of BSARec's improvement over SASRec under a shared implementation, which is a relevant contribution to the growing literature on fair and reproducible evaluation of sequential recommenders. The choice to use a common base architecture and PyTorch-native components is a genuine strength, as it addresses a real source of unfair comparisons. However, the significance is limited by the absence of a fully specified evaluation protocol and by the lack of any variance or significance analysis; as presented, the numerical results are not independently checkable and the comparison to the original BSARec paper is not calibrated.","major_comments":[{"comment":"The evaluation protocol is not specified: the paper does not state how the data were split into train/validation/test, how user sequences were built or filtered, whether the candidate set for evaluation was the full item set or a sampled negative set (and if sampled, how many negatives), or how the reported metrics were computed. This omission is load-bearing because the paper itself cites Klenitskiy and Vasilev (2023) and Petrov and Macdonald (2023) to argue that such protocol choices can change both the magnitude and the direction of reported gains. Without a precise protocol statement, or released code and configuration files, the central comparison and the 'smaller than original gains' conclusion cannot be independently verified.","section":"Section 3.1.1"},{"comment":"Every number in Table 1 comes from a single run, with no standard deviation, no multiple seeds, and no significance test. Several differences between BSARec and SASRec are small in absolute terms, for example NDCG@5 on ml-1m is 0.06172 versus 0.05854 and Precision@10 is 0.0134 versus 0.0127. Given this magnitude of difference, the claim that BSARec 'does indeed outperform' SASRec requires at least repeated runs with different random seeds and a statement of variance; otherwise the observed gaps could be within run-to-run noise.","section":"Table 1"},{"comment":"The hyperparameter tuning description is internally inconsistent. The stated grid for alpha is [0.1, 0.5, 0.7, 0.9], yet the best value reported for fs-nyc is alpha = 0.3, which is not in that grid. It is also unclear whether tuning was performed on a separate validation split, whether GridSearch or another procedure was used, and whether the other hyperparameters from the original papers were kept exactly. These details matter because the comparison to the original BSARec results depends on whether the same hyperparameter choices and selection criterion were used.","section":"Section 3.1.2"},{"comment":"The comparison with the original BSARec results is not calibrated. The paper states that the obtained scores are 'slightly lower than the original ones' and that gains are 'not as high as those presented by the authors,' but the original BSARec paper likely uses a different evaluation protocol (for example, a different negative sampling strategy or candidate set). Without matching the original protocol, or at least stating the protocol of both evaluations, the conclusion that implementation details rather than protocol differences explain the smaller gains is not supported. This is a correctness-risk concern that can be addressed concretely by reporting the exact protocol used by Shin et al. and by the present study.","section":"Section 4"}],"minor_comments":[{"comment":"The dataset description for MovieLens-1m reports 3,883 movies, but the standard MovieLens 1M dataset contains 3,706 movies; please verify the counted item nodes after preprocessing and state any filtering steps.","section":"Section 3.1.1"},{"comment":"The description of Foursquare-nyc mentions New York City and Tokyo, but the reported check-in count (227,428) refers only to NYC; please clarify what portion of the dataset was used.","section":"Section 3.1.1"},{"comment":"The terms BSALayer and BSARecLayer are used in overlapping ways; the paper should give a single consistent nomenclature for the layer that contains the frequency filter and for the combined encoder layer.","section":"Section 3"},{"comment":"The text says 'In Figure 4 the two best performing models are reported,' but Figures 2 and 3 appear to be separate plots and Figure 4 is captioned as containing both; please relabel or combine the figures for clarity.","section":"Section 4"},{"comment":"No version number, commit hash, or URL is given for the EasyRec library, which makes the claimed reproducibility incomplete; please provide the exact library version and, if possible, the configuration YAML files.","section":"Section 3"},{"comment":"The abstract uses 'underlying' where the intended word appears to be 'underlining'; please check throughout the manuscript for small wording errors that affect readability.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's reliance on EasyRec, which is co-developed by some of the authors, is worth editorial attention: the comparison itself is to an external model (SASRec), so the central claim is not circular, but the absence of a code release or protocol specification makes it impossible to rule out that EasyRec-specific defaults drive the reported result. The manuscript would be strengthened by a clear statement of the evaluation protocol and by repeated runs with variance reporting. The scope of the paper (a systematic comparison of two models) fits a venue that values reproducibility studies, provided the protocol gap is closed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper does something worth doing: it re-implements SASRec and BSARec in a shared EasyRec codebase and checks whether BSARec's claimed advantage survives. The directional answer is yes—BSARec beats SASRec on both datasets across all reported metrics. But the gains are smaller than in the original paper, and that second finding is where the paper gets shaky.\n\nCredit where it is due. The comparison is more controlled than most: same base implementation, PyTorch-native modules, hyperparameter tuning limited to the parameters that actually differ (alpha and c for BSARec, dropout for SASRec), and two datasets with multiple metrics. That is a legitimate community service, especially given the existing evidence that evaluation protocol choices can flip conclusions in sequential recommendation.\n\nThe soft spots are real and concentrated in the experimental protocol. Section 3.1.1 lists datasets and metrics but never states how data were split, how sequences were built, whether negative sampling was used, how many candidates were scored, or how the top-k metrics were computed. The paper itself cites Klenitskiy & Vasilev and Petrov & Macdonald to argue these choices matter. So the central claim—BSARec is better but less so than reported—sits on an unspecified protocol. The magnitude finding could be an artifact of EasyRec's defaults rather than a property of the models.\n\nSecond, every number in Table 1 is a single run. No seeds, no variance, no significance test. For a paper whose stated goal is systematic replicability, that is a notable gap. Third, no code or configuration files are released, so the replication cannot be inspected. Fourth, a minor internal inconsistency: the alpha grid is [0.1, 0.5, 0.7, 0.9] but the best fs-nyc alpha is reported as 0.3. Almost certainly a typo, but it needs fixing.\n\nOn the citation pattern: EasyRec is the authors' own library, which is fine here because the comparison target is external. The real issue is that the EasyRec configs are not provided, so self-citation cannot be checked.\n\nThe directional result is probably right—BSARec's inductive bias does help. But the paper's more interesting claim, that the original gains were inflated by implementation details, is unverified until the protocol is specified and multiple runs are reported. I would send this to serious peer review with a request for major revision: add code and configs, state the split and candidate generation, run several seeds, and correct the alpha grid. Desk rejection would be too harsh; the paper is useful as a cautionary data point for anyone comparing sequential recommenders.","headline":"A useful replication check showing BSARec does beat SASRec but by smaller margins than originally reported, undercut by missing protocol details and no code.","tokens_in":6963,"tokens_out":1816,"would_cite":true,"duration_ms":19210,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fair re-run finds BSARec outperforms SASRec, but less than claimed.","keywords":["sequential recommendation","BSARec","SASRec","replicability","frequency enhancement","self-attention","evaluation protocol","recommender systems"],"falsifier":"Re-run both models under the exact evaluation protocol of the original BSARec paper, with the same data splits, the same negative-sampling ratio or full-item scoring, and the same tuned hyperparameters, then compare the resulting ranking-metric gap to the gap reported here. If the gap returns to the original size, the smaller gains in this study are a protocol artifact; if it stays small, the original claims overstate the architectural benefit.","tokens_in":6044,"feed_emoji":"📊","tokens_out":7933,"duration_ms":68246,"temperature":0.7,"pith_summary":"This paper re-implements two sequential recommendation models, SASRec and its extension BSARec, using the same base code library so that any performance difference comes from the added frequency-enhancement module rather than from implementation details. The authors aim to test whether BSARec's inductive bias for high-frequency signals truly improves next-item recommendation over SASRec, as the original BSARec paper claims. Their results confirm that BSARec does outperform SASRec on both datasets examined, but the gains are smaller than those originally reported, ranging from about 3.7% to 5.7% on MovieLens-1m and from 10.3% to 14.3% on Foursquare-nyc. The study's broader point is that implementation alignment matters: performance comparisons between recommender systems can be misleading when models are built on different code bases.","feed_headline":"BSARec beats SASRec again, but by less than claimed","feed_subtitle":"A shared-implementation study finds frequency enhancement helps, just not as much as the original paper reported.","key_machinery":"The object that carries the argument is BSARec's BSALayer, a parallel branch added to the SASRec transformer block. It applies the Fast Fourier Transform to the attention signal, separates low- and high-frequency components according to a cutoff hyperparameter $c$, re-weights them with a learnable parameter $\\beta$, and recombines them via inverse FFT; the branch is blended into the self-attention output under a weight $\\alpha$. This frequency re-scaling is intended to counteract the low-pass filtering and oversmoothing of self-attention and to preserve short-term, high-frequency patterns. The comparison is made fair by re-implementing both SASRec and BSARec on the same base architecture, using native building blocks from the underlying deep-learning library, so the only systematic difference between the models is this added branch.","core_discovery":"BSARec, which augments SASRec's self-attention with a frequency-domain inductive bias that re-scales low- and high-frequency components via a learnable parameter and a cutoff frequency, provides a real but modest improvement over SASRec when both models share the same base implementation. Re-implemented on a common code base and evaluated on MovieLens-1m and Foursquare-nyc, BSARec achieves higher NDCG@5, NDCG@10, NDCG@20, Precision@10, and Recall@10 than SASRec on both datasets. The relative gains are between 3.7% and 5.7% on MovieLens-1m and between 10.3% and 14.3% on Foursquare-nyc, the latter likely reflecting the stronger temporal and high-frequency structure of check-in data. The authors conclude that the frequency-enhancement element works, but that the advantage over SASRec is smaller than the original BSARec paper reported, and that the discrepancy likely stems from implementation and evaluation differences.","pith_inferences":["Beyond the paper: if the original BSARec code is run under the evaluation protocol used here, the gap may shrink to the present size, which would imply that part of the originally reported advantage was an artifact of implementation rather than of the frequency-enhancement idea.","Beyond the paper: the fs-nyc results suggest that frequency-enhancing components should be tested on domains with strong burstiness, such as news, transactions, or social streams, where high-frequency signals are more pronounced.","Beyond the paper: because only three hyperparameters were tuned, a broader search over learning rate and embedding size might narrow or widen the gap; the claim of 'smaller gains' is conditional on the tuning budget used here."],"forward_implications":["Adding BSARec's frequency re-scaling branch to a shared SASRec base improves ranking metrics on both datasets, so the mechanism yields a genuine, if modest, gain beyond self-attention alone.","The larger relative gains on Foursquare-nyc than on MovieLens-1m indicate that the value of high-frequency enhancement depends on the temporal density and periodicity of the interaction data.","Performance comparisons between sequential recommender models should state and align the base implementation, because implementation differences can change the magnitude of reported gains.","The optimal cutoff frequency on both datasets is $c=1$, matching the original paper on ml-1m, which suggests that suppressing the lowest frequency band is a stable, transferable setting."],"supporting_citations":[{"why":"This citation supplies the baseline SASRec model that BSARec extends and that the comparison is built against.","marker":"Kang and McAuley (2018)"},{"why":"This citation defines BSARec and the original performance claims that the study replays and compares against.","marker":"Shin et al. (2024)"},{"why":"This citation provides the common base implementation library used to re-implement both models for a fair comparison.","marker":"Betello et al. (2024)"},{"why":"This citation documents how inconsistent evaluation conditions can distort performance comparisons, motivating the study's design.","marker":"Klenitskiy and Vasilev (2023)"},{"why":"This citation supports the concern that negative sampling choices can change reported model quality, framing the evaluation caveat.","marker":"Petrov and Macdonald (2023)"},{"why":"This citation supplies the MovieLens-1m dataset used for one of the two evaluation benchmarks.","marker":"Harper and Konstan (2015)"},{"why":"This citation supplies the Foursquare-nyc check-in dataset used for the second evaluation benchmark.","marker":"Yang et al. (2015)"}],"fun_headline_variants":["BSARec beats SASRec, but less than the original claim","Frequency boost beats SASRec, but not by much","Shared implementation shrinks BSARec's edge over SASRec","BSARec's gain over SASRec is real but modest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison depends on the assumption that the re-implementations faithfully reproduce the original SASRec and BSARec models, and that the data splits, candidate generation, and evaluation protocol match the conditions under which the original BSARec paper reported its larger gains.","fun_headline_variants_meta":{"raw":{"variants":["BSARec beats SASRec, but less than the original claim","Frequency boost beats SASRec, but not by much","Shared implementation shrinks BSARec's edge over SASRec","BSARec's gain over SASRec is real but modest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1333,"prompt_tokens":896,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":512,"tokens_out":437,"duration_ms":4552,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:48:07.615671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both models under the exact evaluation protocol of the original BSARec paper, with the same data splits, the same negative-sampling ratio or full-item scoring, and the same tuned hyperparameters, then compare the resulting ranking-metric gap to the gap reported here. If the gap returns to the original size, the smaller gains in this study are a protocol artifact; if it stays small, the original claims overstate the architectural benefit.","supporting_citations":[{"cited_title":"A reproducible analysis of sequential recommender systems","cited_arxiv_id":null,"evidence_quote":"This citation provides the common base implementation library used to re-implement both models for a fair comparison."}],"review_version":2}