REVIEW 4 major objections 6 minor 16 references
A Systematic Replicability and Comparative Study of BSARec and SASRec for Sequential Recommendation
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A fair re-run finds BSARec outperforms SASRec, but less than claimed.
desk verdict A useful replication check showing BSARec does beat SASRec but by smaller margins than originally reported, undercut by missing protocol details and no code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is BSARec's BSALayer, a parallel branch added to the SASRec transformer block. It applies the Fast Fourier Transform to the attention signal, separates low- and high-frequency components according to a cutoff hyperparameter $c$, re-weights them with a learnable parameter $\beta$, and recombines them via inverse FFT; the branch is blended into the self-attention output under a weight $\alpha$. This frequency re-scaling is intended to counteract the low-pass filtering and oversmoothing of self-attention and to preserve short-term, high-frequency patterns. The comparison is made fair by re-implementing both SASRec and BSARec on the same base architecture, using native building blocks from the underlying deep-learning library, so the only systematic difference between the models is this added branch.
What would settle it
Re-run both models under the exact evaluation protocol of the original BSARec paper, with the same data splits, the same negative-sampling ratio or full-item scoring, and the same tuned hyperparameters, then compare the resulting ranking-metric gap to the gap reported here. If the gap returns to the original size, the smaller gains in this study are a protocol artifact; if it stays small, the original claims overstate the architectural benefit.
Extended reading notes
Core claim
BSARec, which augments SASRec's self-attention with a frequency-domain inductive bias that re-scales low- and high-frequency components via a learnable parameter and a cutoff frequency, provides a real but modest improvement over SASRec when both models share the same base implementation. Re-implemented on a common code base and evaluated on MovieLens-1m and Foursquare-nyc, BSARec achieves higher NDCG@5, NDCG@10, NDCG@20, Precision@10, and Recall@10 than SASRec on both datasets. The relative gains are between 3.7% and 5.7% on MovieLens-1m and between 10.3% and 14.3% on Foursquare-nyc, the latter likely reflecting the stronger temporal and high-frequency structure of check-in data. The authors conclude that the frequency-enhancement element works, but that the advantage over SASRec is smaller than the original BSARec paper reported, and that the discrepancy likely stems from implementation and evaluation differences.
Load-bearing premise
The comparison depends on the assumption that the re-implementations faithfully reproduce the original SASRec and BSARec models, and that the data splits, candidate generation, and evaluation protocol match the conditions under which the original BSARec paper reported its larger gains.
Editorial extensions
If this is right
- Adding BSARec's frequency re-scaling branch to a shared SASRec base improves ranking metrics on both datasets, so the mechanism yields a genuine, if modest, gain beyond self-attention alone.
- The larger relative gains on Foursquare-nyc than on MovieLens-1m indicate that the value of high-frequency enhancement depends on the temporal density and periodicity of the interaction data.
- Performance comparisons between sequential recommender models should state and align the base implementation, because implementation differences can change the magnitude of reported gains.
- The optimal cutoff frequency on both datasets is $c=1$, matching the original paper on ml-1m, which suggests that suppressing the lowest frequency band is a stable, transferable setting.
Reading between the lines
- Beyond the paper: if the original BSARec code is run under the evaluation protocol used here, the gap may shrink to the present size, which would imply that part of the originally reported advantage was an artifact of implementation rather than of the frequency-enhancement idea.
- Beyond the paper: the fs-nyc results suggest that frequency-enhancing components should be tested on domains with strong burstiness, such as news, transactions, or social streams, where high-frequency signals are more pronounced.
- Beyond the paper: because only three hyperparameters were tuned, a broader search over learning rate and embedding size might narrow or widen the gap; the claim of 'smaller gains' is conditional on the tuning budget used here.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a comparative study of two sequential recommender models, SASRec and BSARec, re-implemented within the EasyRec library so that both models share the same base PyTorch components. The authors claim that BSARec, with its frequency-enhancement bias terms, consistently outperforms SASRec on MovieLens-1m and Foursquare-nyc, but that the performance gains are smaller than those reported in the original BSARec paper (Shin et al., 2024). The study is motivated by concerns about inconsistent evaluation conditions in prior comparisons, and it emphasizes the importance of implementation details for fair model comparison.
Significance. If the central claim holds, the paper provides useful evidence about the magnitude of BSARec's improvement over SASRec under a shared implementation, which is a relevant contribution to the growing literature on fair and reproducible evaluation of sequential recommenders. The choice to use a common base architecture and PyTorch-native components is a genuine strength, as it addresses a real source of unfair comparisons. However, the significance is limited by the absence of a fully specified evaluation protocol and by the lack of any variance or significance analysis; as presented, the numerical results are not independently checkable and the comparison to the original BSARec paper is not calibrated.
major comments (4)
- [Section 3.1.1] The evaluation protocol is not specified: the paper does not state how the data were split into train/validation/test, how user sequences were built or filtered, whether the candidate set for evaluation was the full item set or a sampled negative set (and if sampled, how many negatives), or how the reported metrics were computed. This omission is load-bearing because the paper itself cites Klenitskiy and Vasilev (2023) and Petrov and Macdonald (2023) to argue that such protocol choices can change both the magnitude and the direction of reported gains. Without a precise protocol statement, or released code and configuration files, the central comparison and the 'smaller than original gains' conclusion cannot be independently verified.
- [Table 1] Every number in Table 1 comes from a single run, with no standard deviation, no multiple seeds, and no significance test. Several differences between BSARec and SASRec are small in absolute terms, for example NDCG@5 on ml-1m is 0.06172 versus 0.05854 and Precision@10 is 0.0134 versus 0.0127. Given this magnitude of difference, the claim that BSARec 'does indeed outperform' SASRec requires at least repeated runs with different random seeds and a statement of variance; otherwise the observed gaps could be within run-to-run noise.
- [Section 3.1.2] The hyperparameter tuning description is internally inconsistent. The stated grid for alpha is [0.1, 0.5, 0.7, 0.9], yet the best value reported for fs-nyc is alpha = 0.3, which is not in that grid. It is also unclear whether tuning was performed on a separate validation split, whether GridSearch or another procedure was used, and whether the other hyperparameters from the original papers were kept exactly. These details matter because the comparison to the original BSARec results depends on whether the same hyperparameter choices and selection criterion were used.
- [Section 4] The comparison with the original BSARec results is not calibrated. The paper states that the obtained scores are 'slightly lower than the original ones' and that gains are 'not as high as those presented by the authors,' but the original BSARec paper likely uses a different evaluation protocol (for example, a different negative sampling strategy or candidate set). Without matching the original protocol, or at least stating the protocol of both evaluations, the conclusion that implementation details rather than protocol differences explain the smaller gains is not supported. This is a correctness-risk concern that can be addressed concretely by reporting the exact protocol used by Shin et al. and by the present study.
minor comments (6)
- [Section 3.1.1] The dataset description for MovieLens-1m reports 3,883 movies, but the standard MovieLens 1M dataset contains 3,706 movies; please verify the counted item nodes after preprocessing and state any filtering steps.
- [Section 3.1.1] The description of Foursquare-nyc mentions New York City and Tokyo, but the reported check-in count (227,428) refers only to NYC; please clarify what portion of the dataset was used.
- [Section 3] The terms BSALayer and BSARecLayer are used in overlapping ways; the paper should give a single consistent nomenclature for the layer that contains the frequency filter and for the combined encoder layer.
- [Section 4] The text says 'In Figure 4 the two best performing models are reported,' but Figures 2 and 3 appear to be separate plots and Figure 4 is captioned as containing both; please relabel or combine the figures for clarity.
- [Section 3] No version number, commit hash, or URL is given for the EasyRec library, which makes the claimed reproducibility incomplete; please provide the exact library version and, if possible, the configuration YAML files.
- [Abstract] The abstract uses 'underlying' where the intended word appears to be 'underlining'; please check throughout the manuscript for small wording errors that affect readability.
Circularity Check
No circularity: the paper reports an empirical comparison against external benchmarks; its self-citation of EasyRec is incidental and not load-bearing.
full rationale
The paper makes no derivational claim; its central assertion is empirical: that BSARec outperforms SASRec when both are re-implemented on a common EasyRec base, but with smaller gains than Shin et al. originally reported. Neither model is defined in terms of the measured outcome, and the comparison targets are external (the published BSARec architecture, public datasets, and standard ranking metrics). The only notable self-citation is EasyRec (Betello et al., 2024), whose author list overlaps with this paper's Federico Siciliano; but the EasyRec framework is described concretely in the methodology and the citation is not offered as a theorem that forces the result. The conclusion is therefore not equivalent to its inputs by construction. The reproducibility concern noted in the skeptic reading, namely that the paper omits split, candidate set, and negative-sampling details, is a real evidence-quality issue but not a circularity issue: omitted protocol information cannot make the measured outcome an input to the derivation, because no derivation is present. For the same reason, the claim that gains are smaller than originally reported is an empirical comparison to an external paper, not a renamed restatement of the authors' own implementation choices.
Assumptions & free parameters
free parameters (3)
- alpha (BSARec inductive-bias weight) =
0.7 (ml-1m), 0.3 (fs-nyc)
- c (frequency cutoff) =
1 for both datasets
- dropout rate for SASRec =
0.0005
assumptions (3)
- domain assumption The EasyRec re-implementations of SASRec and BSARec faithfully match the original architectures described in Kang and McAuley (2018) and Shin et al. (2024).
- domain assumption The evaluation metrics (NDCG, Precision, Recall) with the chosen (unstated) candidate set and splits capture recommendation quality equivalently for both models.
- domain assumption Self-attention acts as a low-pass filter, and frequency re-scaling is the appropriate remedy for oversmoothing.
Cite this review
Pith. "Pith review of A Systematic Replicability and Comparative Study of BSARec and SASRec for Sequential Recommendation." pith.science (2026). https://pith.science/paper/F2OG4Z6R
@misc{pith2026250614692,
author = {Pith},
title = {Pith review of: A Systematic Replicability and Comparative Study of BSARec and SASRec for Sequential Recommendation},
year = {2026},
howpublished = {\url{https://pith.science/paper/F2OG4Z6R}},
note = {Machine review of arXiv:2506.14692}
}
read the original abstract
This study aims at comparing two sequential recommender systems: Self-Attention based Sequential Recommendation (SASRec), and Beyond Self-Attention based Sequential Recommendation (BSARec) in order to check the improvement frequency enhancement - the added element in BSARec - has on recommendations. The models in the study, have been re-implemented with a common base-structure from EasyRec, with the aim of obtaining a fair and reproducible comparison. The results obtained displayed how BSARec, by including bias terms for frequency enhancement, does indeed outperform SASRec, although the increases in performance obtained, are not as high as those presented by the authors. This work aims at offering an overview on existing methods, and most importantly at underlying the importance of implementation details for performance comparison.
Figures
Reference graph
Works this paper leans on
-
[1]
A reproducible analysis of sequential recommender systems
Filippo Betello, Antonio Purificato, Federico Siciliano, Giovanni Trappolini, Andrea Bacciu, Nicola Tonellotto, and Fabrizio Silvestri. A reproducible analysis of sequential recommender systems. IEEE Access, 2024
work page 2024
-
[2]
A survey of sequential recommendation systems: Techniques, evaluation, and future directions
Tesfaye Fenta Boka, Zhendong Niu, and Rama Bastola Neupane. A survey of sequential recommendation systems: Techniques, evaluation, and future directions. Information Systems, 125: 0 102427, 2024. ISSN 0306-4379. doi:https://doi.org/10.1016/j.is.2024.102427. URL https://www.sciencedirect.com/science/article/pii/S0306437924000851
arXiv 2024
-
[3]
Xinyu Du, Huanhuan Yuan, Pengpeng Zhao, Jianfeng Qu, Fuzhen Zhuang, Guanfeng Liu, and Victor S. Sheng. Frequency enhanced hybrid attention network for sequential recommendation, 2023. URL https://arxiv.org/abs/2304.09184
arXiv 2023
-
[4]
The movielens datasets: History and context
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5 0 (4): 0 1--19, 2015
2015
-
[5]
Fusing similarity models with markov chains for sparse sequential recommendation
Ruining He and Julian McAuley. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th international conference on data mining (ICDM), pages 191--200. IEEE, 2016
work page 2016
-
[6]
Session-based recommendations with recurrent neural networks, 2016
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks, 2016. URL https://arxiv.org/abs/1511.06939
arXiv 2016
-
[7]
Self-attentive sequential recommendation, 2018
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation, 2018. URL https://arxiv.org/abs/1808.09781
arXiv 2018
-
[8]
Anton Klenitskiy and Alexey Vasilev. Turning dross into gold loss: is bert4rec really better than sasrec? In Proceedings of the 17th ACM Conference on Recommender Systems, RecSys '23, page 1120–1125, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702419. doi:10.1145/3604915.3610644. URL https://doi.org/10.1145/3604915.3610644
arXiv 2023
Show all 16 references
-
[9]
Bengio, and Geoffrey Hinton
Yann LeCun, Y. Bengio, and Geoffrey Hinton. Deep learning. Nature, 521: 0 436--44, 05 2015. doi:10.1038/nature14539
2015 doi
-
[10]
Pytorch: An imperative style, high-performance deep learning library, 2019
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...
2019 arXiv
-
[11]
gsasrec: Reducing overconfidence in sequential recommendation trained with negative sampling
Aleksandr Vladimirovich Petrov and Craig Macdonald. gsasrec: Reducing overconfidence in sequential recommendation trained with negative sampling. In Proceedings of the 17th ACM Conference on Recommender Systems, RecSys ’23, page 116–128. ACM, September 2023. doi:10.1145/360491...
2023
-
[12]
Contrastive learning for representation degeneration problem in sequential recommendation
Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. Contrastive learning for representation degeneration problem in sequential recommendation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM ’22, page 813–823. ACM, February 202...
2022
-
[13]
An attentive inductive bias for sequential recommendation beyond the self-attention, 2024
Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. An attentive inductive bias for sequential recommendation beyond the self-attention, 2024. URL https://arxiv.org/abs/2312.10325
2024 arXiv
-
[14]
Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer, 2019
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer, 2019. URL https://arxiv.org/abs/1904.06690
2019 arXiv
-
[15]
Gomez, ukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000–6010, Red Ho...
2017
-
[16]
Modeling user activity preference by leveraging user spatial temporal characteristics in lbsns
Dingqi Yang, Daqing Zhang, Vincent Zheng, and Zhiyong Yu. Modeling user activity preference by leveraging user spatial temporal characteristics in lbsns. Systems, Man, and Cybernetics: Systems, IEEE Transactions on, 45: 0 129--142, 01 2015. doi:10.1109/TSMC.2014.2327053
2015
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.