How Reliable Are FOSS Popularity Metrics? Analyzing the Effort Required for Spoofing Common Software Popularity Metrics
Pith reviewed 2026-05-22 16:26 UTC · model grok-4.3
The pith
Many FOSS popularity metrics can be spoofed with low to moderate effort
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The analysis finds that many metric categories, especially commit data, issue-tracker activity, downloads, repository contents, and dependency relations, are manipulable with low to moderate effort, and it identifies a sybil attack comprising more than 70,000 spam packages on npm.
What carries the argument
Decomposition of combined metrics into atomic metric categories analyzed for spoofing effort under a maintainer-centered threat model
If this is right
- Quantitative FOSS metrics should be used with much greater caution in software engineering research and practice.
- Particular caution is needed when metrics are used for ranking, dataset construction, or any allocation process.
- The real-world sybil attack on npm shows that manipulation is not just theoretical but has occurred at large scale.
- Metrics that become optimization targets are especially susceptible to gaming.
Where Pith is reading between the lines
- If these metrics are easily spoofed, funding or award programs relying on them may need independent verification methods.
- Future metric design could focus on signals that are harder for maintainers to fabricate, such as external usage data.
- Platform operators might implement better detection for coordinated spam packages to protect metric integrity.
Load-bearing premise
The spoofing effort estimates assume a threat model where the attacker can act as or control the project maintainer and directly influence signals like commits or repository contents.
What would settle it
Demonstrating that a specific metric category requires high effort to spoof even when the attacker controls the maintainer, or that the identified npm sybil attack is unrelated to metric manipulation.
Figures
read the original abstract
Quantitative metrics derived from software repositories and package ecosystems are widely used to assess the impact, popularity, maintenance, and criticality of free and open source software (FOSS) projects. However, these metrics are often assumed to be reliable despite their potential susceptibility to manipulation. Prior empirical software engineering and security research deployed these in a variety of ways which assume they indeed capture project impact and popularity. Yet, the extent to which these underlying signals can be spoofed in practice, and the consequences this has for downstream uses of the metrics, has received little focused attention. To address this gap, the paper decomposes existing combined metrics into atomic metric categories, analyzes their spoofing effort under a maintainer-centered threat model, and investigates a real-world sybil attack on npm connected to an impact-based reward mechanism. The analysis finds that many metric categories, especially commit data, issue-tracker activity, downloads, repository contents, and dependency relations, are manipulable with low to moderate effort, and it identifies a sybil attack comprising more than 70,000 spam packages on npm. These results imply that quantitative FOSS metrics should be used with much greater caution in software engineering research and practice, particularly for ranking, dataset construction, and any allocation or evaluation process that turns metrics into optimization targets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper decomposes existing combined FOSS popularity metrics into atomic metric categories, analyzes their spoofing effort under a maintainer-centered threat model, and investigates a real-world sybil attack on npm connected to an impact-based reward mechanism. The analysis finds that many metric categories, especially commit data, issue-tracker activity, downloads, repository contents, and dependency relations, are manipulable with low to moderate effort, and it identifies a sybil attack comprising more than 70,000 spam packages on npm. These results imply that quantitative FOSS metrics should be used with much greater caution in software engineering research and practice, particularly for ranking, dataset construction, and any allocation or evaluation process that turns metrics into optimization targets.
Significance. If the results hold, this work contributes by showing that widely used quantitative metrics for FOSS impact and popularity are susceptible to manipulation, with direct implications for empirical software engineering and security research that relies on them. The systematic decomposition of metrics into atomic categories and the grounding in a concrete real-world npm sybil attack provide a useful framework for assessing metric reliability. The paper explicitly credits the empirical observation of the large-scale attack as supporting evidence for the effort analysis.
major comments (2)
- [Analysis of spoofing effort for atomic metric categories] The central claim that commit data, issue-tracker activity, downloads, repository contents, and dependency relations are manipulable with low to moderate effort rests on qualitative judgments derived from the maintainer-centered threat model and metric decomposition. No section reports actual execution of the spoofing procedures (e.g., creating and pushing commits, uploading packages to trigger download counters, or fabricating dependency edges) together with wall-clock time, API-call counts, or monetary cost. Consequently the 'low to moderate' classification remains an author judgment rather than an observed quantity. This is load-bearing for the main finding.
- [Investigation of the npm sybil attack] The identification of a sybil attack comprising more than 70,000 spam packages on npm is presented as supporting evidence for the manipulability claims, yet the detection heuristic, confirmation that these packages were created specifically to game an impact-based reward, and any raw data or reproducibility package are not provided. This limits independent verification of the count and attribution.
minor comments (1)
- [Threat model] The threat model is described as maintainer-centered, but a brief explicit statement of its scope and any assumptions about attacker capabilities would improve clarity for readers.
Simulated Author's Rebuttal
We thank the referee for their constructive and detailed review. The comments raise valid points about strengthening the empirical basis of our effort analysis and improving the verifiability of the npm sybil attack findings. We address each major comment below and have revised the manuscript to provide additional transparency and detail while preserving the core contributions.
read point-by-point responses
-
Referee: [Analysis of spoofing effort for atomic metric categories] The central claim that commit data, issue-tracker activity, downloads, repository contents, and dependency relations are manipulable with low to moderate effort rests on qualitative judgments derived from the maintainer-centered threat model and metric decomposition. No section reports actual execution of the spoofing procedures (e.g., creating and pushing commits, uploading packages to trigger download counters, or fabricating dependency edges) together with wall-clock time, API-call counts, or monetary cost. Consequently the 'low to moderate' classification remains an author judgment rather than an observed quantity. This is load-bearing for the main finding.
Authors: We acknowledge that our spoofing effort analysis is qualitative and derived from reasoning under the maintainer-centered threat model rather than from direct experimental execution. Performing actual spoofing would require creating artificial activity on production platforms, which poses ethical issues and would violate terms of service. Instead, the classification rests on a systematic decomposition of each metric into the minimal sequence of actions an adversary with maintainer access would need, cross-referenced against documented real-world incidents and platform APIs. To address the concern, we have added an appendix with expanded step-by-step procedure descriptions for each atomic category, including counts of required operations (e.g., git commands, API endpoints) and rough cost estimates drawn from public documentation and prior literature. These additions make the basis for the 'low to moderate' labels more explicit without altering the original analysis. revision: partial
-
Referee: [Investigation of the npm sybil attack] The identification of a sybil attack comprising more than 70,000 spam packages on npm is presented as supporting evidence for the manipulability claims, yet the detection heuristic, confirmation that these packages were created specifically to game an impact-based reward, and any raw data or reproducibility package are not provided. This limits independent verification of the count and attribution.
Authors: We agree that explicit details on the detection method and supporting artifacts would aid verification. In the revised manuscript we have expanded the section on the npm sybil attack to include the precise heuristic (name-pattern clustering, absence of functional code, temporal burst patterns, and linkage to impact-based reward programs) together with examples of confirmation evidence from package metadata and maintainer behavior. We have also added a public reproducibility repository containing the heuristic implementation, aggregate statistics, and a curated sample of 1,000 annotated packages. The full set of 70,000+ packages is too large for direct inclusion, but we now state that the complete dataset is available to researchers upon request under a data-use agreement that respects platform policies and privacy considerations. revision: yes
Circularity Check
No circularity: analysis rests on metric decomposition and external npm observation under explicit threat model
full rationale
The paper decomposes FOSS metrics into atomic categories and evaluates spoofing effort via a maintainer-centered threat model, then supports the low-to-moderate effort claim with direct observation of an existing >70k-package npm sybil attack. No equations, fitted parameters, or self-citations are used to derive the central findings; the effort classification follows from the stated assumptions rather than reducing to those assumptions by construction. The derivation chain is therefore self-contained against external benchmarks and does not exhibit any of the enumerated circularity patterns.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Quantitative metrics derived from software repositories and package ecosystems are widely used to assess the impact, popularity, maintenance, and criticality of FOSS projects.
Reference graph
Works this paper leans on
-
[1]
Arya, A., Brown, C., Pike, R., The Open Source Security Foundation: Open Source Project Criticality Score, https://github.com/ossf/criticality_score
-
[2]
University of Westminster Press
Birkinbine, B.J.: Incorporating the Digital Commons: Corporate Involvement in Free and Open Source Software. University of Westminster Press. https://doi.org/ 10.16997/book39
-
[3]
Empirical Evidence and Further Developments of the Research
Bonaccorsi, A., Lorenzi, D., Merito, M., Rossi, C.: Business Firms’ Engagement in Community Projects. Empirical Evidence and Further Developments of the Research. In: First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS’07: ICSE Workshops 2007). IEEE. https://doi.org/10. 1109/floss.2007.3
work page 2007
-
[4]
Brackett, S.A., Meyers, J.S., Scott, S.: O$$ security: Does more money for open source software mean better security? A proof of concept
-
[5]
Capiluppi, A., Stol, K.J., Boldyreff, C.: Exploring the Role of Commercial Stake- holders in Open Source Software Evolution, pp. 178–200. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-33442-9_12
-
[6]
Journal of Systems and Software84(1), 144–161 (2011)
Capra, E., Francalanci, C., Merlo, F., Rossi-Lamastra, C.: Firms’ involvement in open source projects: A trade-off between software structural quality and popularity. Journal of Systems and Software84(1), 144–161 (2011). https://doi.org/https: //doi.org/10.1016/j.jss.2010.09.004, information Networking and Software Services 20 B. Swierzy et al
-
[7]
Chao, J., Tao, S., Ribbink, A.: Evaluating the evaluators: On package scores and their underlying metrics
-
[8]
CHAOSS: Funding Impact Measurement Working Group, https://github.com/ chaoss/wg-funding-impact
-
[9]
CHAOSS: Project Popularity, https://chaoss.community/kb/ metric-project-popularity/
-
[10]
In: Proceeding of the 2005 ACM SIGCOMM workshop on Economics of peer-to-peer systems - P2PECON ’05
Cheng, A., Friedman, E.: Sybilproof reputation mechanisms. In: Proceeding of the 2005 ACM SIGCOMM workshop on Economics of peer-to-peer systems - P2PECON ’05. p. 128. P2PECON ’05, ACM Press. https://doi.org/10.1145/1080192.1080202
-
[11]
In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
Coelho, J., Valente, M.T.: Why modern open source projects fail. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. pp. 186–196. ESEC/FSE’17, ACM. https://doi.org/10.1145/3106237.3106246
-
[12]
https://doi.org/10.1016/J.INFSOF.2020.106274, https://doi.org/10.1016/j.infsof
Coelho, J., Valente, M.T., Milen, L., Silva, L.L.: Is this GitHub project maintained? Measuring the level of maintenance activity of open-source projects122, 106274. https://doi.org/10.1016/J.INFSOF.2020.106274, https://doi.org/10.1016/j.infsof. 2020.106274
-
[13]
Coelho, J., Valente, M.T., Silva, L.L., Hora, A.: Why we engage in FLOSS: answers from core developers. In: Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering. ICSE ’18, ACM. https: //doi.org/10.1145/3195836.3195848
-
[14]
Eghbal, N.: Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure, https://www.fordfoundation.org/work/learning/research-reports/ roads-and-bridges-the-unseen-labor-behind-our-digital-infrastructure/
-
[15]
Eldeeb, Y., Sikora, A.: How Much Are GitHub Stars Worth to You?, https:// the-guild.dev/blog/judging-open-source-by-github-stars
-
[16]
Traceability transformed: Generating more accurate links with pre-trained BERT models
Ferreira, G., Jia, L., Sunshine, J., Kastner, C.: Containing malicious package updates in npm with a lightweight permission system. In: IEEE/ACM 43rd International Conference on Software Engineering. pp. 1334–1346. IEEE. https://doi.org/10. 1109/icse43902.2021.00121
-
[17]
Geer, D., Sieniawski, G.P.: Who Will Pay the Piper for Open Source Software Maintenance? Can We Increase Reliability as We Increase Reliance?45(2), https: //www.usenix.org/publications/login/summer2020/geer
-
[18]
https://doi.org/10.4018/ ijossp.2016070102
Gerlach, J.H., Wu, C.G., Cunningham, L.F., Young, C.E.: An Exploratory Study of Conflict over Paying Debian Developers7(3), 20–38. https://doi.org/10.4018/ ijossp.2016070102
-
[19]
https://doi.org/10.1109/tifs.2014
Gong, N.Z., Frank, M., Mittal, P.: Sybilbelief: A semi-supervised learning approach for structure-based sybil detection9(6), 976–987. https://doi.org/10.1109/tifs.2014. 2316975
-
[20]
https://doi.org/10.1109/ms.2013.95
Gonzalez-Barahona, J.M., Izquierdo-Cortazar, D., Maffulli, S., Robles, G.: Under- standing How Companies Interact with Free Software Communities30(5), 38–45. https://doi.org/10.1109/ms.2013.95
-
[21]
https://doi.org/10.1109/MSP.2010.85
Halderman, J.A.: To Strengthen Security, Change Developers’ Incentives8(2), 79–82. https://doi.org/10.1109/MSP.2010.85
-
[22]
In: Proceedings of the 2nd International Workshop on Software Health
Iaffaldano, G., Steinmacher, I., Calefato, F., Gerosa, M., Lanubile, F.: Why do developers take breaks from contributing to OSS projects? a preliminary analysis. In: Proceedings of the 2nd International Workshop on Software Health. p. 9–16. SoHeal ’19, IEEE Press. https://doi.org/10.1109/SoHeal.2019.00009, https://doi. org/10.1109/SoHeal.2019.00009
-
[23]
Joslyn, H.: Is crypto the solution to paying open source developers?, https:// thenewstack.io/is-crypto-the-solution-to-paying-open-source-developers/ Susceptibility to Fraud of Monetary Incentive Mechanisms for FOSS 21
-
[24]
Kabir, M.M.A., Wang, Y., Yao, D., Meng, N.: How do developers follow security- relevantbestpracticeswhenusingnpmpackages?In:2022IEEESecureDevelopment Conference (SecDev). IEEE. https://doi.org/10.1109/secdev53368.2022.00027
-
[25]
https://doi.org/10.1007/s10664-015-9393-5
Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian, D.: An in-depth study of the promises and perils of mining GitHub21(5), 2035–2071. https://doi.org/10.1007/s10664-015-9393-5
-
[26]
https: //doi.org/10.1145/3690632
Li, X., Zhang, Y., Osborne, C., Zhou, M., Jin, Z., Liu, H.: Systematic Literature Review of Commercial Participation in Open Source Software34(2), 1–31. https: //doi.org/10.1145/3690632
-
[27]
Linåker, J., Link, G., Lumbard, K.: Sustaining Maintenance Labor for Healthy Open Source Software Projects through Human Infrastructure: A Maintainer Perspective. In: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. pp. 37–48. ESEM ’24, ACM. https://doi.org/10.1145/3674805.3686667
-
[28]
https://doi.org/10.1109/jproc.2024.3379855
Longtchi, T.T., Rodriguez, R.M., Al-Shawaf, L., Atyabi, A., Xu, S.: Internet-based social engineering psychology, attacks, and defenses: A survey112(3), 210–246. https://doi.org/10.1109/jproc.2024.3379855
-
[29]
https://doi.org/10.1109/tem.2021.3122012, http://dx.doi.org/10.1109/ TEM.2021.3122012
Mujahid, S., Costa, D.E., Abdalkareem, R., Shihab, E., Saied, M.A., Adams, B.: Toward Using Package Centrality Trend to Identify Packages in Decline69(6), 3618–3632. https://doi.org/10.1109/tem.2021.3122012, http://dx.doi.org/10.1109/ TEM.2021.3122012
-
[30]
Müller, W., Plötz, H., Redlich, J.P., Shiraki, T.: Sybil proof anonymous reputation management. In: Proceedings of the 4th international conference on Security and privacy in communication netowrks. pp. 1–10. Securecomm08, ACM. https://doi. org/10.1145/1460877.1460887
-
[31]
Open Source Initiative: The open source definition, https://opensource.org/osd
-
[32]
Open Source Technology Improvement Fund: Open source technology improvement fund, https://ostif.org
-
[33]
Open Technology Fund: Localization labfree and open source software sustainability fund, https://www.opentech.fund/funds/ free-and-open-source-software-sustainability-fund/
-
[34]
In: 2024 Conference on Computer-Supported Cooperative Work and Social Computing
Osborne, C.: Open Source Software Developers’ Views on Public and Private Funding: A Case Study on scikit-learn. In: 2024 Conference on Computer-Supported Cooperative Work and Social Computing. pp. 154–161. CSCW ’24, ACM. https: //doi.org/10.1145/3678884.3681844
-
[35]
Osborne, C.: Why Companies "Democratise" Artificial Intelligence: The Case of Open Source Software Donations
-
[36]
Osborne, C., Sharratt, P., Foster, D., Boehm, M.: A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development
-
[37]
In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering
Overney, C., Meinicke, J., Kästner, C., Vasilescu, B.: How to not get rich: an empirical study of donations in open source. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. ICSE ’20, ACM. https: //doi.org/10.1145/3377811.3380410
-
[38]
Pohl, T., Ohm, M., Boes, F., Meier, M.: You can run but you can’t hide: Runtime protection against malicious package updates for node.js. In: Sicherheit 2024. pp. 231–241. Gesellschaft für Informatik e.V., Bonn (2024). https://doi.org/10.18420/ sicherheit2024_015
work page 2024
-
[39]
https: //doi.org/10.1109/mc.2010.24 22 B
Riehle, D.: The Economic Case for Open Source Foundations43(1), 86–90. https: //doi.org/10.1109/mc.2010.24 22 B. Swierzy et al
-
[40]
Riehle, D., Riemer, P., Kolassa, C., Schmidt, M.: Paid vs. Volunteer Work in Open Source. In: 47th Hawaii International Conference on System Sciences. IEEE. https://doi.org/10.1109/hicss.2014.407
-
[41]
Ruohonen, J., Choudhary, G., Alami, A.: An Overview of Cyber Security Funding for Open Source Software
- [42]
-
[43]
Shpota, S.: Github activity generator, https://github.com/Shpota/ github-activity-generator
-
[44]
Sovereign Tech Agency: Sovereign tech fund, https://www.sovereign.tech/programs/ fund
-
[45]
Sovereign Tech Agency: Technologist, https://www.sovereign.tech/jobs/technologist
-
[46]
https://doi.org/10.1002/(sici)1234-981x(199707)5:3<305::aid-euro184>3
Strathern, M.: ‘improving ratings’: audit in the british university system5(3), 305–321. https://doi.org/10.1002/(sici)1234-981x(199707)5:3<305::aid-euro184>3. 0.co;2-4, https://doi.org/10.1002/(sici)1234-981x(199707)5:3<305::aid-euro184>3. 0.co;2-4
-
[47]
tea Association: Tea documentation, https://docs.tea.xyz/tea
-
[48]
tea Association: What is Proof of Contribution? (technical), https://docs.tea.xyz/tea/i-want-to.../learn-about-proof-of-contribution/ what-is-proof-of-contribution-technical
-
[49]
The Linux Foundation: Alpha-omega, https://alpha-omega.dev
-
[50]
Tumbleson, C.: The disappointing tea.xyz, https://connortumbleson.com/2024/02/ 26/the-disappointing-tea-xyz/
work page 2024
-
[51]
https://doi.org/10.1145/3387111
Wang, Z., Feng, Y., Wang, Y., Jones, J.A., Redmiles, D.: Unveiling Elite Developers’ Activities in Open Source Projects29(3), 1–35. https://doi.org/10.1145/3387111
-
[52]
Warren, E.: Foss sustainability fund 2024: the grant proposal is declined, https: //codeberg.org/forgejo/sustainability/pulls/48
work page 2024
-
[53]
https://doi.org/10.1111/j.1467-9310.2006
West,J.,Gallagher,S.:Challengesofopeninnovation:theparadoxoffirminvestment in open-source software36(3), 319–331. https://doi.org/10.1111/j.1467-9310.2006. 00436.x
-
[54]
In: Proceedings of the 14th International Workshop on Principles of Software Evolution
Yamashita, K., McIntosh, S., Kamei, Y., Hassan, A.E., Ubayashi, N.: Revisiting the applicability of the pareto principle to core development teams in open source software projects. In: Proceedings of the 14th International Workshop on Principles of Software Evolution. ESEC/FSE’15, ACM. https://doi.org/10.1145/2804360. 2804366
-
[55]
https://doi.org/10.1109/access.2024.3403197
Zaoui, M., Yousra, B., Yassine, S., Yassine, M., Karim, O.: A comprehensive taxonomy of social engineering attacks and defense mechanisms: Toward effective mitigation strategies12, 72224–72241. https://doi.org/10.1109/access.2024.3403197
-
[56]
In: CHI Conference on Human Factors in Computing Systems
Zhang, X., Wang, T., Yu, Y., Zeng, Q., Li, Z., Wang, H.: Who, what, why and how? towards the monetary incentive in crowd collaboration: A case study of github’s sponsor mechanism. In: CHI Conference on Human Factors in Computing Systems. pp. 1–18. CHI ’22, ACM. https://doi.org/10.1145/3491102.3501822
-
[57]
https://doi.org/10.1145/3510849
Zhang, Y., Liu, H., Tan, X., Zhou, M., Jin, Z., Zhu, J.: Turnover of Companies in OpenStack: Prevalence and Rationale31(4), 1–24. https://doi.org/10.1145/3510849
-
[58]
In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Zhang, Y., Qin, M., Stol, K.J., Zhou, M., Liu, H.: How Are Paid and Volunteer Open Source Developers Different? A Study of the Rust Project. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. pp. 1–13. ICSE ’24, ACM. https://doi.org/10.1145/3597503.3639197 Susceptibility to Fraud of Monetary Incentive Mechanisms for FOSS 23
-
[59]
Zhang, Y., Stol, K.J., Liu, H., Zhou, M.: Corporate dominance in open source ecosystems: a case study of OpenStack. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 1048–1060. ESEC/FSE ’22, ACM. https://doi.org/10. 1145/3540250.3549117
-
[60]
https://doi.org/10.1007/s10664-021-10060-y
Zhou, J., Wang, S., Kamei, Y., Hassan, A.E., Ubayashi, N.: Studying donations and their expenses in open source projects: a case study of GitHub projects collecting do- nations through open collectives27(1). https://doi.org/10.1007/s10664-021-10060-y
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.