Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Assessing the Impact of a User-Item Collaborative Attack on Class of Users

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read This paper claims that the user class most hurt by a user-item shilling attack reverses with dataset sparsity: on Yelp, highly-active users suffer more; on MovieLens, slightly-active users do.

desk verdict A plausible, genuinely new class-level reversal in shilling-attack impact, but the Yelp side depends on one unseeded subsample, so treat it as provisional. read the letter →

arxiv 1908.07968 v1 pith:S53ALOAI submitted 2019-08-21 cs.IR cs.CRcs.LG

classification cs.IRcs.CRcs.LG
keywords shillingattackscollaborativefilteringuser-itemattackuserclassesdatasetsparsityrecommendersecurityhitratioprofileinjection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a shilling attack that pushes one item into a target user's top-k list hurts users differently depending on how active they are. It splits users into slightly-active and highly-active groups at the median rating count and measures hit ratio at $k=10$ under two profile-injection strategies across four collaborative-filtering models. The central finding is a reversal: on the sparse Yelp data the attack hits highly-active users harder ($r>1$), while on the dense MovieLens data it hits slightly-active users harder ($r<1$). The authors attribute the contrast to the roughly 40-fold difference in dataset density and argue that data characteristics, not just attack size and attacker knowledge, shape attack impact.

What carries the argument

The carrying object is the user-class attack impact ratio $r = HR_{HA}/HR_{SA}$, where $HR@k$ is the fraction of attacked user-item pairs for which the pushed item enters the top-$k$ list. $r=1$ means equal impact, $r>1$ means highly-active users are more affected, and $r<1$ means slightly-active users are more affected. Around this ratio the paper organizes two crafted attack strategies (User-and-Model aware UMA, which learns from recommender feedback on a seed profile, and User-Neighbor aware UNA, which fills profiles with popular items from the victim's neighborhood), four recommendation models (User-kNN, Item-kNN, BPR-SLIM, BPR-MF), and a user-class split at the median rating count.

What would settle it

Run the same small-size UMA and UNA attacks on the full Yelp dataset and compute $\bar r$; if it falls below 1 rather than the reported 2.393 for UMA, the Yelp contrast is a subsampling artifact. A complementary check downsamples MovieLens to Yelp-like density and observes whether $r$ crosses from below 1 to above 1.

Watch

Extended reading notes

Core claim

The paper's claim is that user-class vulnerability to a user-item shilling attack is dataset-dependent and that the dependency tracks sparsity. Using the ratio $r = HR_{HA}/HR_{SA}$ of hit ratios for highly-active over slightly-active users, the experiments report $r>1$ for Yelp and $r<1$ for MovieLens across both attack types and most models, with the strongest contrast for small-size attacks (e.g., Yelp UMA $\bar r = 2.393$ vs MovieLens $\bar r = 0.658$). The proposed mechanism is that in a sparse dataset a slightly-active profile is too thin for the attacker to mimic, while in a dense dataset highly-active users' recommendations are protected by rich neighbor profiles, so pushing their items requires a very large injection. The paper presents this as an empirical contrast plus a sparsity conjecture, not as a proven universal law.

Load-bearing premise

The Yelp-side reversal rests on one random 5K-user/5K-item subsample of the 25K/25K Yelp data, with no reported seed, no repeated sampling, and no check that the subsample preserves the full dataset's attack dynamics.

Editorial extensions

If this is right

  • Robustness comparisons between collaborative-filtering models need to be conditioned on user class and dataset density, since aggregate hit ratios can hide a reversal of which users are harmed.
  • On sparse platforms an attacker who wants to damage a specific user should target highly-active users; on dense platforms the same attack is more effective against slightly-active users.
  • Shilling-defense evaluations should report class-level hit ratios, not only mean accuracy, so that an attack is not declared harmless when it actually harms one user class.
  • Dataset density becomes a controllable factor in attack evaluation: downsampling a dense dataset can change which user class appears more vulnerable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If sparsity is the true driver, then continuously varying density (for example by subsampling MovieLens down toward Yelp's 0.1% density) should move $r$ from below 1 to above 1; this is a direct test the paper does not run.
  • Because the Yelp result comes from a single random 5K/5K subsample with no reported seed, a stratified repeated-sampling check could change the direction of the contrast; the sparsity explanation would be on firmer ground if the full 25K/25K dataset reproduced $r>1$.
  • The median split into slightly- and highly-active users is arbitrary; moving the threshold would change class composition and could shift $r$, so the reversal should be checked across quartile-based definitions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the impact of user-item profile-injection (shilling) attacks on two classes of users, defined by their activity level: highly-active (HA) and slightly-active (SA). The authors define a metric r = HR_HA/HR_SA, where HR@10 is the hit ratio of a push attack over a set of user-item pairs, and compute r for two attack strategies (UMA and UNA), four collaborative-filtering models (User-kNN, Item-kNN, BPR-SLIM, BPR-MF), and two attack-size regimes (small and large) on the Yelp and MovieLens-1M datasets. The main empirical claim is that the user-class impact reverses across datasets: on Yelp, HA users are more vulnerable (r > 1), while on ML-1M, SA users are more vulnerable (r < 1). The paper also reports a fine-grained correlation analysis across CF models and conjectures that dataset sparsity drives the reversal.

Significance. If the reported reversal is robust, it is a useful and non-obvious observation for the recommender-systems robustness literature: it suggests that the density of the user-item matrix, not just attack size or attacker knowledge, controls which user class is most affected by profile-injection attacks. The paper's experimental setup is clearly described and uses publicly available datasets and a public library (MyMediaLite), which aids reproducibility. However, the empirical basis for the central claim is thin: the Yelp results come from a single unseeded subsample, no confidence intervals or significance tests are provided for the r values, and the sparsity explanation is a post-hoc conjecture rather than a tested hypothesis. These issues currently limit the strength of the conclusion.

major comments (3)
  1. [Section 2.2, Table 2] The Yelp side of the central reversal rests entirely on a single random 5K-user/5K-item subsample of the original 25K-user/25K-item dataset, yet no seed, number of repetitions, or representativeness check is reported. The paper itself notes that the subsample density (0.110%) differs from the full data density (0.093%), so the sampling changes the data distribution and could alter the HA/SA split and the attack dynamics. Without evidence that the class-conditional attack impact is preserved in the subsample, the claimed Yelp-versus-MovieLens reversal is not reliably established.
  2. [Section 3.1, Table 2] The claim that Yelp has r > 1 and ML-1M has r < 1 is supported only by averaged point estimates, with no confidence intervals or significance tests. Several individual cells are close to 1, for example Yelp UNA small r = 1.093 and large r = 1.005, and ML-1M UMA large r = 0.909 and UNA large r = 0.858. Given that HR@10 is a proportion over a finite set of attacked pairs and that small-size attacks average only 2, 10, 20, and 50 injected profiles, the reported ratios may not be statistically distinguishable from 1, undermining the dataset-contrast conclusion.
  3. [Section 2.1.4, Table 2] The number of attacked user-item pairs |S| and the procedure for selecting target items are not reported. The standard error of HR@10 depends directly on |S|, so the reader cannot assess whether observed differences between HA and SA users, or between datasets, are within sampling noise. Reporting |S| and ideally confidence intervals for the r values is essential for interpreting the results, especially for the small-size attack regime.
minor comments (5)
  1. [Abstract and Section 2.1.3] The abstract refers to "cold v.s. warm user" while the body consistently uses "slightly-active" and "highly-active"; the terminology should be aligned throughout.
  2. [Section 1] "reserach questions" is a typo for "research questions".
  3. [Section 2.1.1] The phrase "in order in order" is duplicated in the description of the UNA attack; also, the sentence "The number of items in a shilling profile is close to the mean value of the number of rating in the dataset" is vague and should specify the actual profile length used in the experiments.
  4. [Figure 1] The heat-maps are described by the text as showing pairwise Pearson correlation coefficients, but the figures do not include a color scale or printed values, making it impossible to verify the quoted values (e.g., ρ = 0.960, 0.993) from the figures alone.
  5. [Section 3.1] The phrase "the impact of attack on highly-active users is relatively higher in comparison with slightly-active users" is ambiguous; consider rephrasing as "the attack has a higher impact on highly-active users than on slightly-active users".

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: HR@10 and r are direct measurements, and the sparsity explanation is an explicitly labeled post-hoc conjecture rather than a fitted input.

full rationale

The paper's central claim is purely empirical. HR@10 is defined in Definition 1 as the fraction of successful attacks on a set of user-item pairs, and r is then defined in Section 3.1 as the ratio HR_HA/HR_SA of two measured values. No parameter is fitted to a subset and then used to predict the same subset, and no normalization or constraint in the experimental protocol enforces the reported reversal between Yelp (r > 1) and MovieLens (r < 1). The sparsity interpretation is explicitly presented as a conjecture: 'We conjecture that the above contrasting behaviors are directly linked with the characteristics of the datasets such as their sparsity.' This is post-hoc explanation, not circular derivation. The authors' self-citations [9, 10] appear only in the future-work paragraph and are not load-bearing for any result. The Yelp 5K-user/5K-item subsample, while a legitimate validity threat regarding representativeness and seed dependence, does not make the argument circular: the reported HR values and ratios remain direct measurements on that sample, and no equation reduces the conclusion to an input assumption. Therefore, no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical observation, so the ledger lists the hand-chosen experimental settings that condition the result, the domain assumptions about the evaluation setup, and no invented entities. No numbers are fitted to force the outcome.

free parameters (4)
  • HA/SA split threshold = second quartile of per-user rating counts
    Users are split at the median rating count; the ratio r would change under a different quantile.
  • UNA neighborhood size k = 50
    Set in footnote 1 for finding similar users; no sensitivity analysis, affects which items fill the attack profile.
  • CF model neighbor count = 20
    Used for User-kNN and Item-kNN per [19]; affects recommendation stability under attack.
  • Attack profile length = close to mean per-user rating count
    Specified vaguely in Section 2.1.1; profile length controls how much fake data is injected.
assumptions (4)
  • domain assumption Shilling profiles are injected and not removed by any detection mechanism
    The evaluation assumes no defense filters the fake accounts; standard in shilling attack studies but optimistic.
  • domain assumption Recommenders are retrained on the poisoned data before HR@10 is measured
    Required for the attack to shift recommendations; implied by the protocol but not stated.
  • domain assumption The 5K/5K Yelp subsample preserves the full dataset's attack behavior
    The subsample reduces to 3.4% of ratings; representativeness is assumed without evidence.
  • domain assumption HR@10 is a meaningful measure of attack success
    Modified from Hit-Ratio [17]; other metrics like prediction shift might rank attacks differently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing the Impact of a User-Item Collaborative Attack on Class of Users." pith.science (2026). https://pith.science/paper/S53ALOAI

@misc{pith2026190807968,
  author       = {Pith},
  title        = {Pith review of: Assessing the Impact of a User-Item Collaborative Attack on Class of Users},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S53ALOAI}},
  note         = {Machine review of arXiv:1908.07968}
}
read the original abstract

Collaborative Filtering (CF) models lie at the core of most recommendation systems due to their state-of-the-art accuracy. They are commonly adopted in e-commerce and online services for their impact on sales volume and/or diversity, and their impact on companies' outcome. However, CF models are only as good as the interaction data they work with. As these models rely on outside sources of information, counterfeit data such as user ratings or reviews can be injected by attackers to manipulate the underlying data and alter the impact of resulting recommendations, thus implementing a so-called shilling attack. While previous works have focused on evaluating shilling attack strategies from a global perspective paying particular attention to the effect of the size of attacks and attacker's knowledge, in this work we explore the effectiveness of shilling attacks under novel aspects. First, we investigate the effect of attack strategies crafted on a target user in order to push the recommendation of a low-ranking item to a higher position, referred to as user-item attack. Second, we evaluate the effectiveness of attacks in altering the impact of different CF models by contemplating the class of the target user, from the perspective of the richness of her profile (i.e., cold v.s. warm user). Finally, similar to previous work we contemplate the size of attack (i.e., the amount of fake profiles injected) in examining their success. The results of experiments on two widely used datasets in business and movie domains, namely Yelp and MovieLens, suggest that warm and cold users exhibit contrasting behaviors in datasets with different characteristics.

Figures

Figures reproduced from arXiv: 1908.07968 by the authors.

Figure 1
Figure 1. Heat-map of Correlation Coefficient (ρ) of different mea￾sures between CF models for small-size attacks: (a) HR@10 on Slightly-active Users, (b) HR@10 on Highly-active Users. support size of the user profile (i.e., the user profile is not rich enough for the attacker to be able to mimic it in a crafted way). In contrast, highly-active users are more immune to attack in ML￾1M with higher density, because their recomm… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM-Based User Simulation for Low-Knowledge Shilling Attacks on Recommender Systems

    cs.IR 2025-05 conditional novelty 6.0 of 10

    LLM-based agents can carry out stealthy shilling attacks that outperform traditional heuristic attacks by generating realistic fake profiles, ratings, and reviews.

Reference graph

Works this paper leans on

32 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gediminas Adomavicius and Jingjing Zhang. 2015. Improving Stability of Rec- ommender Systems: A Meta-Algorithmic Approach. IEEE Trans. Knowl. Data Eng. 27, 6 (2015), 1573–1587. https://doi.org/10.1109/TKDE.2014.2384502

  2. [2]

    Aggarwal

    Charu C. Aggarwal. 2016. Recommender Systems - The Textbook . Springer. https://doi.org/10.1007/978-3-319-29659-3

  3. [3]

    Xavier Amatriain and Justin Basilico. 2015. Recommender Systems in Industry: A Netflix Case Study. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 385–419. https://doi.org/10.1007/ 978-1-4899-7637-6_11

  4. [4]

    Runa Bhaumik, Chad Williams, Bamshad Mobasher, and Robin Burke. 2006. Securing collaborative filtering against malicious attacks through anomaly de- tection. In Proceedings of the 4th Workshop on Intelligent Techniques for Web Personalization (ITWP’06), Boston , Vol. 6. 10

  5. [5]

    Breese, David Heckerman, and Carl Myers Kadie

    John S. Breese, David Heckerman, and Carl Myers Kadie. 1998. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In UAI ’98: Pro- ceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, University of Wisconsin Business School, Madison, Wisconsin, USA, July 24-26, 1998, Gregory F. Cooper and Serafín Moral (Eds....

  6. [6]

    Robin Burke, Bamshad Mobasher, Roman Zabicki, and Runa Bhaumik. 2005. Identifying attack models for secure recommendation. Beyond Personalization 2005 (2005)

  7. [7]

    Zunping Cheng and Neil Hurley. 2010. Robust Collaborative Recommendation by Least Trimmed Squares Matrix Factorization. In 22nd IEEE International Con- ference on Tools with Artificial Intelligence, ICTAI 2010, Arras, France, 27-29 October 2010 - Volume 2. IEEE Computer Society, 105–112. https://doi.org/10.1109/ICTAI. 2010.90

  8. [8]

    Paul-Alexandru Chirita, Wolfgang Nejdl, and Cristian Zamfir. 2005. Preventing shilling attacks in online recommender systems. In Seventh ACM International Workshop on Web Information and Data Management (WIDM 2005), Bremen, Germany, November 4, 2005 , Angela Bonifati and Dongwon Lee (Eds.). ACM, 67–74. https://doi.org/10.1145/1097047.1097061

Show all 32 references
  1. [9]

    Yashar Deldjoo, Maurizio Ferrari Dacrema, Mihai Gabriel Constantin, Hamid Eghbal-zadeh, Stefano Cereda, Markus Schedl, Bogdan Ionescu, and Paolo Cre- monesi. 2019. Movie genome: alleviating new item cold start in movie rec- ommendation. User Model. User-Adapt. Interact. 29, 2 ...

  2. [10]

    Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, and Gabriella Pasi. 2018. Content-Based Multimedia Recommendation Systems: Definition and Appli- cation Domains. In Proceedings of the 9th Italian Information Retrieval Work- shop, Rome, Italy, May, 28-30, 2018. (CEUR Workshop Pr...

  3. [11]

    Ihsan Gunes, Cihan Kaleli, Alper Bilge, and Huseyin Polat. 2014. Shilling attacks against recommender systems: a comprehensive survey. Artif. Intell. Rev. 42, 4 (2014), 767–799. https://doi.org/10.1007/s10462-012-9364-9

  4. [12]

    Maxwell Harper and Joseph A

    F. Maxwell Harper and Joseph A. Konstan. 2016. The MovieLens Datasets: History and Context. TiiS 5, 4 (2016), 19:1–19:19. https://doi.org/10.1145/2827872

  5. [13]

    Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. 2018. Adversarial Personalized Ranking for Recommendation. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, Kevyn Collins-...

  6. [14]

    Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016. Fast Matrix Factorization for Online Recommendation with Implicit Feedback. In Proceedings of the 39th International ACM SIGIR conference on Research and Devel- opment in Information Retrieval, SIGIR 2016, Pisa,...

  7. [15]

    Lam and John Riedl

    Shyong K. Lam and John Riedl. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th international conference on World Wide Web, WWW 2004, New York, NY, USA, May 17-20, 2004, Stuart I. Feldman, Mike Uretsky, Marc Najork, and Craig E. Wills (Eds.). AC...

  8. [16]

    Greg Linden, Brent Smith, and Jeremy York. 2003. Industry Report: Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Distributed Systems Online 4, 1 (2003)

  9. [17]

    Bhaskar Mehta and Wolfgang Nejdl. 2008. Attack resistant collaborative filtering. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, Singapore, July 20-24, 2008 , Sung-Hyon Myaeng, Douglas W. O...

  10. [18]

    Bhaskar Mehta and Wolfgang Nejdl. 2009. Unsupervised strategies for shilling detection and robust collaborative filtering. User Model. User-Adapt. Interact. 19, 1-2 (2009), 65–97. https://doi.org/10.1007/s11257-008-9050-4

  11. [19]

    Bamshad Mobasher, Robin Burke, Runa Bhaumik, and Chad Williams. 2005. Effective attack models for shilling item-based collaborative filtering systems. Citeseer

  12. [20]

    Bamshad Mobasher, Robin Burke, and Jeff J Sandvig. 2006. Model-based collab- orative filtering as a defense against profile injection attacks. In AAAI, Vol. 6. 1388

  13. [21]

    Burke, Runa Bhaumik, and Jeff J

    Bamshad Mobasher, Robin D. Burke, Runa Bhaumik, and Jeff J. Sandvig. 2007. Attacks and Remedies in Collaborative Recommendation.IEEE Intelligent Systems 22, 3 (2007), 56–63. https://doi.org/10.1109/MIS.2007.45

  14. [22]

    Burke, Runa Bhaumik, and Chad Williams

    Bamshad Mobasher, Robin D. Burke, Runa Bhaumik, and Chad Williams. 2007. Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness. ACM Trans. Internet Techn. 7, 4 (2007), 23. https://doi.org/ 10.1145/1278366.1278372

  15. [23]

    Xia Ning and George Karypis. 2011. SLIM: Sparse Linear Methods for Top-N Recommender Systems. In 11th IEEE International Conference on Data Mining, ICDM 2011, Vancouver, BC, Canada, December 11-14, 2011 , Diane J. Cook, Jian Pei, Wei Wang, Osmar R. Zaïane, and Xindong Wu (Eds....

  16. [24]

    O’Mahony, Neil J

    Michael P. O’Mahony, Neil J. Hurley, Nicholas Kushmerick, and Guenole C. M. Silvestre. 2004. Collaborative recommendation: A robustness analysis.ACM Trans. Internet Techn. 4, 4 (2004), 344–377. https://doi.org/10.1145/1031114.1031116

  17. [25]

    O’Mahony, Neil J

    Michael P. O’Mahony, Neil J. Hurley, and Guenole C. M. Silvestre. 2005. Rec- ommender Systems: Attack Types and Strategies. In Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Appli- cations of Artificial Intelligence Con...

  18. [26]

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme

  19. [27]

    Konstan, and John Riedl

    Badrul Munir Sarwar, George Karypis, Joseph A. Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the Tenth International World Wide Web Conference, WWW 10, Hong Kong, China, May 1-5, 2001 , Vincent Y. Shen, Nobuo Sai...

  20. [28]

    Yue Shi, Martha Larson, and Alan Hanjalic. 2014. Collaborative filtering beyond the user-item matrix: A survey of the state of the art and future challenges. ACM Computing Surveys (CSUR) 47, 1 (2014), 3

  21. [29]

    Zhihai Yang and Zhongmin Cai. 2017. Detecting abnormal profiles in collabo- rative filtering recommender systems. J. Intell. Inf. Syst. 48, 3 (2017), 499–518. https://doi.org/10.1007/s10844-016-0424-5

  22. [30]

    Mi Zhang, Jie Tang, Xuchen Zhang, and Xiangyang Xue. 2014. Addressing cold start in recommender systems: A semi-supervised co-training algorithm. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. ACM, 73–82. 5

  23. [52]

    https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_ id=231&proceeding_id=14

  24. [2009]

    In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 , Jeff A

    BPR: Bayesian Personalized Ranking from Implicit Feedback. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009 , Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452–461. https://dslpitt.o...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.