REVIEW 3 major objections 5 minor 1 cited by
Assessing the Impact of a User-Item Collaborative Attack on Class of Users
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that the user class most hurt by a user-item shilling attack reverses with dataset sparsity: on Yelp, highly-active users suffer more; on MovieLens, slightly-active users do.
desk verdict A plausible, genuinely new class-level reversal in shilling-attack impact, but the Yelp side depends on one unseeded subsample, so treat it as provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the user-class attack impact ratio $r = HR_{HA}/HR_{SA}$, where $HR@k$ is the fraction of attacked user-item pairs for which the pushed item enters the top-$k$ list. $r=1$ means equal impact, $r>1$ means highly-active users are more affected, and $r<1$ means slightly-active users are more affected. Around this ratio the paper organizes two crafted attack strategies (User-and-Model aware UMA, which learns from recommender feedback on a seed profile, and User-Neighbor aware UNA, which fills profiles with popular items from the victim's neighborhood), four recommendation models (User-kNN, Item-kNN, BPR-SLIM, BPR-MF), and a user-class split at the median rating count.
What would settle it
Run the same small-size UMA and UNA attacks on the full Yelp dataset and compute $\bar r$; if it falls below 1 rather than the reported 2.393 for UMA, the Yelp contrast is a subsampling artifact. A complementary check downsamples MovieLens to Yelp-like density and observes whether $r$ crosses from below 1 to above 1.
Extended reading notes
Core claim
The paper's claim is that user-class vulnerability to a user-item shilling attack is dataset-dependent and that the dependency tracks sparsity. Using the ratio $r = HR_{HA}/HR_{SA}$ of hit ratios for highly-active over slightly-active users, the experiments report $r>1$ for Yelp and $r<1$ for MovieLens across both attack types and most models, with the strongest contrast for small-size attacks (e.g., Yelp UMA $\bar r = 2.393$ vs MovieLens $\bar r = 0.658$). The proposed mechanism is that in a sparse dataset a slightly-active profile is too thin for the attacker to mimic, while in a dense dataset highly-active users' recommendations are protected by rich neighbor profiles, so pushing their items requires a very large injection. The paper presents this as an empirical contrast plus a sparsity conjecture, not as a proven universal law.
Load-bearing premise
The Yelp-side reversal rests on one random 5K-user/5K-item subsample of the 25K/25K Yelp data, with no reported seed, no repeated sampling, and no check that the subsample preserves the full dataset's attack dynamics.
Editorial extensions
If this is right
- Robustness comparisons between collaborative-filtering models need to be conditioned on user class and dataset density, since aggregate hit ratios can hide a reversal of which users are harmed.
- On sparse platforms an attacker who wants to damage a specific user should target highly-active users; on dense platforms the same attack is more effective against slightly-active users.
- Shilling-defense evaluations should report class-level hit ratios, not only mean accuracy, so that an attack is not declared harmless when it actually harms one user class.
- Dataset density becomes a controllable factor in attack evaluation: downsampling a dense dataset can change which user class appears more vulnerable.
Reading between the lines
- If sparsity is the true driver, then continuously varying density (for example by subsampling MovieLens down toward Yelp's 0.1% density) should move $r$ from below 1 to above 1; this is a direct test the paper does not run.
- Because the Yelp result comes from a single random 5K/5K subsample with no reported seed, a stratified repeated-sampling check could change the direction of the contrast; the sparsity explanation would be on firmer ground if the full 25K/25K dataset reproduced $r>1$.
- The median split into slightly- and highly-active users is arbitrary; moving the threshold would change class composition and could shift $r$, so the reversal should be checked across quartile-based definitions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the impact of user-item profile-injection (shilling) attacks on two classes of users, defined by their activity level: highly-active (HA) and slightly-active (SA). The authors define a metric r = HR_HA/HR_SA, where HR@10 is the hit ratio of a push attack over a set of user-item pairs, and compute r for two attack strategies (UMA and UNA), four collaborative-filtering models (User-kNN, Item-kNN, BPR-SLIM, BPR-MF), and two attack-size regimes (small and large) on the Yelp and MovieLens-1M datasets. The main empirical claim is that the user-class impact reverses across datasets: on Yelp, HA users are more vulnerable (r > 1), while on ML-1M, SA users are more vulnerable (r < 1). The paper also reports a fine-grained correlation analysis across CF models and conjectures that dataset sparsity drives the reversal.
Significance. If the reported reversal is robust, it is a useful and non-obvious observation for the recommender-systems robustness literature: it suggests that the density of the user-item matrix, not just attack size or attacker knowledge, controls which user class is most affected by profile-injection attacks. The paper's experimental setup is clearly described and uses publicly available datasets and a public library (MyMediaLite), which aids reproducibility. However, the empirical basis for the central claim is thin: the Yelp results come from a single unseeded subsample, no confidence intervals or significance tests are provided for the r values, and the sparsity explanation is a post-hoc conjecture rather than a tested hypothesis. These issues currently limit the strength of the conclusion.
major comments (3)
- [Section 2.2, Table 2] The Yelp side of the central reversal rests entirely on a single random 5K-user/5K-item subsample of the original 25K-user/25K-item dataset, yet no seed, number of repetitions, or representativeness check is reported. The paper itself notes that the subsample density (0.110%) differs from the full data density (0.093%), so the sampling changes the data distribution and could alter the HA/SA split and the attack dynamics. Without evidence that the class-conditional attack impact is preserved in the subsample, the claimed Yelp-versus-MovieLens reversal is not reliably established.
- [Section 3.1, Table 2] The claim that Yelp has r > 1 and ML-1M has r < 1 is supported only by averaged point estimates, with no confidence intervals or significance tests. Several individual cells are close to 1, for example Yelp UNA small r = 1.093 and large r = 1.005, and ML-1M UMA large r = 0.909 and UNA large r = 0.858. Given that HR@10 is a proportion over a finite set of attacked pairs and that small-size attacks average only 2, 10, 20, and 50 injected profiles, the reported ratios may not be statistically distinguishable from 1, undermining the dataset-contrast conclusion.
- [Section 2.1.4, Table 2] The number of attacked user-item pairs |S| and the procedure for selecting target items are not reported. The standard error of HR@10 depends directly on |S|, so the reader cannot assess whether observed differences between HA and SA users, or between datasets, are within sampling noise. Reporting |S| and ideally confidence intervals for the r values is essential for interpreting the results, especially for the small-size attack regime.
minor comments (5)
- [Abstract and Section 2.1.3] The abstract refers to "cold v.s. warm user" while the body consistently uses "slightly-active" and "highly-active"; the terminology should be aligned throughout.
- [Section 1] "reserach questions" is a typo for "research questions".
- [Section 2.1.1] The phrase "in order in order" is duplicated in the description of the UNA attack; also, the sentence "The number of items in a shilling profile is close to the mean value of the number of rating in the dataset" is vague and should specify the actual profile length used in the experiments.
- [Figure 1] The heat-maps are described by the text as showing pairwise Pearson correlation coefficients, but the figures do not include a color scale or printed values, making it impossible to verify the quoted values (e.g., ρ = 0.960, 0.993) from the figures alone.
- [Section 3.1] The phrase "the impact of attack on highly-active users is relatively higher in comparison with slightly-active users" is ambiguous; consider rephrasing as "the attack has a higher impact on highly-active users than on slightly-active users".
Circularity Check
No circularity: HR@10 and r are direct measurements, and the sparsity explanation is an explicitly labeled post-hoc conjecture rather than a fitted input.
full rationale
The paper's central claim is purely empirical. HR@10 is defined in Definition 1 as the fraction of successful attacks on a set of user-item pairs, and r is then defined in Section 3.1 as the ratio HR_HA/HR_SA of two measured values. No parameter is fitted to a subset and then used to predict the same subset, and no normalization or constraint in the experimental protocol enforces the reported reversal between Yelp (r > 1) and MovieLens (r < 1). The sparsity interpretation is explicitly presented as a conjecture: 'We conjecture that the above contrasting behaviors are directly linked with the characteristics of the datasets such as their sparsity.' This is post-hoc explanation, not circular derivation. The authors' self-citations [9, 10] appear only in the future-work paragraph and are not load-bearing for any result. The Yelp 5K-user/5K-item subsample, while a legitimate validity threat regarding representativeness and seed dependence, does not make the argument circular: the reported HR values and ratios remain direct measurements on that sample, and no equation reduces the conclusion to an input assumption. Therefore, no specific circular step can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- HA/SA split threshold =
second quartile of per-user rating counts
- UNA neighborhood size k =
50
- CF model neighbor count =
20
- Attack profile length =
close to mean per-user rating count
assumptions (4)
- domain assumption Shilling profiles are injected and not removed by any detection mechanism
- domain assumption Recommenders are retrained on the poisoned data before HR@10 is measured
- domain assumption The 5K/5K Yelp subsample preserves the full dataset's attack behavior
- domain assumption HR@10 is a meaningful measure of attack success
Cite this review
Pith. "Pith review of Assessing the Impact of a User-Item Collaborative Attack on Class of Users." pith.science (2026). https://pith.science/paper/S53ALOAI
@misc{pith2026190807968,
author = {Pith},
title = {Pith review of: Assessing the Impact of a User-Item Collaborative Attack on Class of Users},
year = {2026},
howpublished = {\url{https://pith.science/paper/S53ALOAI}},
note = {Machine review of arXiv:1908.07968}
}
read the original abstract
Collaborative Filtering (CF) models lie at the core of most recommendation systems due to their state-of-the-art accuracy. They are commonly adopted in e-commerce and online services for their impact on sales volume and/or diversity, and their impact on companies' outcome. However, CF models are only as good as the interaction data they work with. As these models rely on outside sources of information, counterfeit data such as user ratings or reviews can be injected by attackers to manipulate the underlying data and alter the impact of resulting recommendations, thus implementing a so-called shilling attack. While previous works have focused on evaluating shilling attack strategies from a global perspective paying particular attention to the effect of the size of attacks and attacker's knowledge, in this work we explore the effectiveness of shilling attacks under novel aspects. First, we investigate the effect of attack strategies crafted on a target user in order to push the recommendation of a low-ranking item to a higher position, referred to as user-item attack. Second, we evaluate the effectiveness of attacks in altering the impact of different CF models by contemplating the class of the target user, from the perspective of the richness of her profile (i.e., cold v.s. warm user). Finally, similar to previous work we contemplate the size of attack (i.e., the amount of fake profiles injected) in examining their success. The results of experiments on two widely used datasets in business and movie domains, namely Yelp and MovieLens, suggest that warm and cold users exhibit contrasting behaviors in datasets with different characteristics.
Figures
Forward citations
Cited by 1 Pith paper
-
LLM-Based User Simulation for Low-Knowledge Shilling Attacks on Recommender Systems
LLM-based agents can carry out stealthy shilling attacks that outperform traditional heuristic attacks by generating realistic fake profiles, ratings, and reviews.
Reference graph
Works this paper leans on
- [1]
-
[2]
Charu C. Aggarwal. 2016. Recommender Systems - The Textbook . Springer. https://doi.org/10.1007/978-3-319-29659-3
-
[3]
Xavier Amatriain and Justin Basilico. 2015. Recommender Systems in Industry: A Netflix Case Study. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 385–419. https://doi.org/10.1007/ 978-1-4899-7637-6_11
work page 2015
-
[4]
Runa Bhaumik, Chad Williams, Bamshad Mobasher, and Robin Burke. 2006. Securing collaborative filtering against malicious attacks through anomaly de- tection. In Proceedings of the 4th Workshop on Intelligent Techniques for Web Personalization (ITWP’06), Boston , Vol. 6. 10
work page 2006
-
[5]
Breese, David Heckerman, and Carl Myers Kadie
John S. Breese, David Heckerman, and Carl Myers Kadie. 1998. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In UAI ’98: Pro- ceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, University of Wisconsin Business School, Madison, Wisconsin, USA, July 24-26, 1998, Gregory F. Cooper and Serafín Moral (Eds....
work page 1998
-
[6]
Robin Burke, Bamshad Mobasher, Roman Zabicki, and Runa Bhaumik. 2005. Identifying attack models for secure recommendation. Beyond Personalization 2005 (2005)
work page 2005
-
[7]
Zunping Cheng and Neil Hurley. 2010. Robust Collaborative Recommendation by Least Trimmed Squares Matrix Factorization. In 22nd IEEE International Con- ference on Tools with Artificial Intelligence, ICTAI 2010, Arras, France, 27-29 October 2010 - Volume 2. IEEE Computer Society, 105–112. https://doi.org/10.1109/ICTAI. 2010.90
doi:10.1109/ictai 2010
-
[8]
Paul-Alexandru Chirita, Wolfgang Nejdl, and Cristian Zamfir. 2005. Preventing shilling attacks in online recommender systems. In Seventh ACM International Workshop on Web Information and Data Management (WIDM 2005), Bremen, Germany, November 4, 2005 , Angela Bonifati and Dongwon Lee (Eds.). ACM, 67–74. https://doi.org/10.1145/1097047.1097061
Show all 32 references
-
[9]
Yashar Deldjoo, Maurizio Ferrari Dacrema, Mihai Gabriel Constantin, Hamid Eghbal-zadeh, Stefano Cereda, Markus Schedl, Bogdan Ionescu, and Paolo Cre- monesi. 2019. Movie genome: alleviating new item cold start in movie rec- ommendation. User Model. User-Adapt. Interact. 29, 2 ...
2019 doi
-
[10]
Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, and Gabriella Pasi. 2018. Content-Based Multimedia Recommendation Systems: Definition and Appli- cation Domains. In Proceedings of the 9th Italian Information Retrieval Work- shop, Rome, Italy, May, 28-30, 2018. (CEUR Workshop Pr...
2018
-
[11]
Ihsan Gunes, Cihan Kaleli, Alper Bilge, and Huseyin Polat. 2014. Shilling attacks against recommender systems: a comprehensive survey. Artif. Intell. Rev. 42, 4 (2014), 767–799. https://doi.org/10.1007/s10462-012-9364-9
2014 doi
-
[12]
Maxwell Harper and Joseph A
F. Maxwell Harper and Joseph A. Konstan. 2016. The MovieLens Datasets: History and Context. TiiS 5, 4 (2016), 19:1–19:19. https://doi.org/10.1145/2827872
2016 doi
-
[13]
Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. 2018. Adversarial Personalized Ranking for Recommendation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, Kevyn Collins-...
2018
-
[14]
Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016. Fast Matrix Factorization for Online Recommendation with Implicit Feedback. In Proceedings of the 39th International ACM SIGIR conference on Research and Devel- opment in Information Retrieval, SIGIR 2016, Pisa,...
2016
-
[15]
Lam and John Riedl
Shyong K. Lam and John Riedl. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th international conference on World Wide Web, WWW 2004, New York, NY, USA, May 17-20, 2004, Stuart I. Feldman, Mike Uretsky, Marc Najork, and Craig E. Wills (Eds.). AC...
2004
-
[16]
Greg Linden, Brent Smith, and Jeremy York. 2003. Industry Report: Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Distributed Systems Online 4, 1 (2003)
2003
-
[17]
Bhaskar Mehta and Wolfgang Nejdl. 2008. Attack resistant collaborative filtering. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, Singapore, July 20-24, 2008 , Sung-Hyon Myaeng, Douglas W. O...
2008
-
[18]
Bhaskar Mehta and Wolfgang Nejdl. 2009. Unsupervised strategies for shilling detection and robust collaborative filtering. User Model. User-Adapt. Interact. 19, 1-2 (2009), 65–97. https://doi.org/10.1007/s11257-008-9050-4
2009 doi
-
[19]
Bamshad Mobasher, Robin Burke, Runa Bhaumik, and Chad Williams. 2005. Effective attack models for shilling item-based collaborative filtering systems. Citeseer
2005
-
[20]
Bamshad Mobasher, Robin Burke, and Jeff J Sandvig. 2006. Model-based collab- orative filtering as a defense against profile injection attacks. In AAAI, Vol. 6. 1388
2006
-
[21]
Burke, Runa Bhaumik, and Jeff J
Bamshad Mobasher, Robin D. Burke, Runa Bhaumik, and Jeff J. Sandvig. 2007. Attacks and Remedies in Collaborative Recommendation.IEEE Intelligent Systems 22, 3 (2007), 56–63. https://doi.org/10.1109/MIS.2007.45
2007 doi
-
[22]
Burke, Runa Bhaumik, and Chad Williams
Bamshad Mobasher, Robin D. Burke, Runa Bhaumik, and Chad Williams. 2007. Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness. ACM Trans. Internet Techn. 7, 4 (2007), 23. https://doi.org/ 10.1145/1278366.1278372
2007
-
[23]
Xia Ning and George Karypis. 2011. SLIM: Sparse Linear Methods for Top-N Recommender Systems. In 11th IEEE International Conference on Data Mining, ICDM 2011, Vancouver, BC, Canada, December 11-14, 2011 , Diane J. Cook, Jian Pei, Wei Wang, Osmar R. Zaïane, and Xindong Wu (Eds....
2011 doi
-
[24]
O’Mahony, Neil J
Michael P. O’Mahony, Neil J. Hurley, Nicholas Kushmerick, and Guenole C. M. Silvestre. 2004. Collaborative recommendation: A robustness analysis.ACM Trans. Internet Techn. 4, 4 (2004), 344–377. https://doi.org/10.1145/1031114.1031116
2004
-
[25]
O’Mahony, Neil J
Michael P. O’Mahony, Neil J. Hurley, and Guenole C. M. Silvestre. 2005. Rec- ommender Systems: Attack Types and Strategies. In Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Appli- cations of Artificial Intelligence Con...
2005
-
[26]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme
-
[27]
Konstan, and John Riedl
Badrul Munir Sarwar, George Karypis, Joseph A. Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the Tenth International World Wide Web Conference, WWW 10, Hong Kong, China, May 1-5, 2001 , Vincent Y. Shen, Nobuo Sai...
2001
-
[28]
Yue Shi, Martha Larson, and Alan Hanjalic. 2014. Collaborative filtering beyond the user-item matrix: A survey of the state of the art and future challenges. ACM Computing Surveys (CSUR) 47, 1 (2014), 3
2014
-
[29]
Zhihai Yang and Zhongmin Cai. 2017. Detecting abnormal profiles in collabo- rative filtering recommender systems. J. Intell. Inf. Syst. 48, 3 (2017), 499–518. https://doi.org/10.1007/s10844-016-0424-5
2017 doi
-
[30]
Mi Zhang, Jie Tang, Xuchen Zhang, and Xiangyang Xue. 2014. Addressing cold start in recommender systems: A semi-supervised co-training algorithm. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. ACM, 73–82. 5
2014
-
[52]
https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_ id=231&proceeding_id=14
-
[2009]
In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 , Jeff A
BPR: Bayesian Personalized Ranking from Implicit Feedback. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 , Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452–461. https://dslpitt.o...
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.