Pith. sign in

REVIEW 4 major objections 6 minor 105 references

"I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most security reports in npm repositories are never tagged as security issues.

desk verdict Solid qualitative work and a useful bot survey, but the 14.8% security prevalence is likely inflated by mixing pull requests into the issue dataset and by weak classifier validation. read the letter →

arxiv 2506.07728 v1 pith:G4IJ7OB5 submitted 2025-06-09 cs.CR

Rajdeep Ghosh , Shiladitya De , Mainack Mondal This is my paper
classification cs.CR
keywords npmsecurityissuesGitHubbotsmachinelearningclassificationvulnerabilityreportingopensourceissuetriage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies security-related issue reporting in GitHub repositories of npm packages, collecting 10,907,467 issues from 37,278 repositories tied to 45,466 sampled packages. It finds that only 0.13% of issues carry security-related tags, but a manually validated machine-learning classifier identifies 1,617,738 untagged issues (14.8% of all issues) as security-related, along with 4,461,934 comments on them. The paper argues that current bots are neither detecting nor assisting with most of these issues, and that user-reported security issues are frequently closed stale without a valid reason. A correlation analysis indicates that issues with CVE references or proposed fixes are resolved faster, while reproducible issues without such references tend to languish. If correct, the paper shows that tag-based security triage in npm repositories systematically hides the majority of security reports from tooling and maintainers.

What carries the argument

The carrying mechanism is a two-stage detection pipeline: a Word2Vec similarity search over developer-defined issue tags to find security-related tags, followed by a fine-tuned RoBERTa text classifier trained on 2,000 manually labeled issues that classifies all untagged issues as security-related or not. A second fine-tuned RoBERTa model, trained on 1,638 quotes coded with a four-level thematic hierarchy, labels each issue's creation, discussion, and resolution phases with interaction themes. Generalized linear mixed models then correlate issue resolution time, staleness, and successful resolution with factors such as CVE mention, reproducibility, bot involvement, and maintainer count.

What would settle it

Re-collect the data from the same 37,278 repositories using the GitHub Issues API while explicitly filtering out records that contain a pull_request field, then re-run the published classifier; if the fraction of security-related issues drops markedly below 14.8%, the paper's headline estimate is inflated by pull requests. A lighter check is to sample the 1,617,738 flagged issues and count how many contain a pull request link or a 'pull request' marker.

Watch

Extended reading notes

Core claim

The central discovery is that security-related issue reporting in npm repositories is vast but almost invisible to existing tagging practices. Using a hierarchical qualitative coding of a sample of user-reported security issues, the authors built a fine-tuned RoBERTa classifier that flags 14.8% of all collected issues as security-related, 114 times more than the 0.13% that repository owners tagged. They further find that bots, which create most of the tagged security issues (87.6% by Dependabot), are largely rule-based and rarely assist with user-reported issues, and that reproducible security issues without CVE mentions are disproportionately closed stale without resolution. The paper presents this as a lower bound, since any missed or unreported security issues would only increase the counts.

Load-bearing premise

The load-bearing data premise is that the 10,907,467 records collected from the GitHub Issues endpoint are actual issue reports and not pull requests, since the endpoint returns both and the paper does not state that pull_request entries were filtered out.

Editorial extensions

If this is right

  • Project maintainers cannot rely on GitHub labels to gauge how many security reports their repositories receive; the untagged fraction dwarfs the tagged one.
  • Bots that only manage dependencies and stale issues will not close the security-reporting gap; security-focused, ideally free, AI-assisted bots are needed.
  • Reproducibility alone does not trigger prompt fixes; CVE references or attached pull requests are the strong correlates of resolution.
  • A large share of security reports (more than 23% per the paper) receive no comments at all, suggesting community engagement is sparse.
  • The paper's data and models are released, so other researchers can reproduce or extend the 14.8% estimate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct re-run of the pipeline that explicitly excludes pull requests from the collected records would test whether the 14.8% figure is inflated; the paper does not report such a filter, and several examples it presents as issues are pull requests.
  • If the under-tagging result holds, security scanners that rely on GitHub labels, CVE databases, or Dependabot-style alerts will systematically miss most real-world security reports; a practical extension would be to run the released classifier as a triage pre-filter in npm repositories.
  • The CVE-resolution correlation suggests maintainers use external vulnerability databases as a priority signal; a controlled study of maintainer response times to CVE-linked vs non-CVE reproducible reports could test this mechanism directly.
  • The finding that bot involvement correlates with staleness and low resolution could mean bots are closing issues that humans would otherwise fix, so bot policies should be audited for their effect on security issues specifically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a large-scale empirical study of security-related issue reporting in GitHub repositories of npm packages. The authors collect 10,907,467 records via the GitHub Issues API, identify 13,835 security-tagged records (0.13%), train a RoBERTa classifier on a manually labeled subset, and apply it to 9,131,800 'untagged user-reported issues,' yielding 1,617,738 predicted security issues (reported as 14.8% of all issues). They then analyze bot involvement, user-developer interactions through qualitative coding plus a theme classifier, and resolution correlates via GLMMs. The main claims are that security issues are largely untagged, bots are insufficient, and reproducible issues without CVE references tend to close stale.

Significance. If the central estimate were supported, the paper would provide a valuable evidence base for security tooling in the npm ecosystem. Strengths include the large stratified sample, the mixed-method qualitative coding with reported inter-rater agreement, the staged ML pipeline with performance tables, and the release of models and code. However, the central 14.8% estimate is not currently supported due to data-collection and validation problems described below; the manuscript's main conclusions currently rest on these unsupported numbers.

major comments (4)
  1. [§3.2, §5.1, §6] The GitHub Issues REST endpoint returns pull requests as well as issues, and the manuscript never states that records with a `pull_request` field were removed. The paper's own examples confirm PRs are in the data: reference [6] is storybookjs/storybook pull request #19885, reference [13] is apache/superset pull request #30200, and reference [1] is expressjs/express pull request #2748; Section 8.1 even describes 'pull requests made by users' as part of the issue creation phase. Consequently, the 10,907,467 record base, the 0.13% tag rate, the 1,617,738 classifier positives, and the 7,731 Dependabot 'issues' in Section 7.1 all mix code-change proposals with user-reported issues. This undermines RQ1 and RQ2.
  2. [§5.1] The denominator applied to the classifier is unexplained and arithmetically inconsistent with the headline percentage. The paper states the model was applied to '9,131,800 user-reported issues which were not tagged with any security-related tags,' but 10,907,467 minus the 13,835 security-tagged records is 10,893,632, not 9,131,800, and no other exclusion is described. Moreover, 1,617,738/9,131,800 = 17.7%, whereas the abstract reports 14.8% of all issues (1,617,738/10,907,467 = 14.8%). The 14.8% claim therefore does not follow from the stated application set.
  3. [§5.1, Table 6] The classifier validation does not establish performance on the target distribution of untagged issues. The confusion matrix in Table 6 is based on 100 issues with only 9 true security positives, giving a wide confidence interval for the claimed 96% accuracy, and the sampling frame for those 100 issues is not described. Since the training positives come from the already security-tagged set (the paper notes 43% of classifier-flagged samples contain the literal string 'security'), the model may be capitalizing on shallow lexical cues; without a validation sample drawn from the untagged population, the 1,617,738 count lacks empirical support. The limitations paragraph in Section 1 does not mention this threat to validity.
  4. [§3.4, §7.1] The bot analysis conflates pull requests with issues. Dependabot opens pull requests, not issues, yet Section 7.1 states that Dependabot created 87.6% of the 7,731 bot-reported security 'issues.' The subsequent conclusion that bots are insufficient for detecting or addressing security issues is therefore based on PR activity rather than issue reports, and the counts of 7,731 bot-created records and 93 unique bot accounts are not interpretable as issue-reporting behavior.
minor comments (6)
  1. [§3.3, Table 3] Table 3 lists 'bug' twice (3,796 and 141 in the two columns), and the second occurrence appears to be a formatting error; also clarify whether the 23,356 unique tags are counted per repository or per package.
  2. [§9.4] The Pearson correlation R=0.9918 between the number of active maintainers and the number of active maintainers who comment is close to a tautology and does not support the claim that few maintainers engage; report the actual distributions and a more direct measure.
  3. [§9, Tables 31-33] The GLMM tables report odds ratios for 'time to close,' but time to close is a continuous duration; specify the model family, link function, and whether the outcome was dichotomized or modeled as a survival outcome.
  4. [§4, §5.1] The qualitative coding section reports Cohen's kappa = 0.85 for open coding and 0.74 for gold-standard labeling, but the gold-standard labeling in Section 5.1 appears to have used the same coders; state whether the coders were independent for the 2,000-issue labeling and report per-class breakdowns.
  5. [References] Reference [20] duplicates reference [19] (same Strapi issue #12320); one should be corrected.
  6. [§5.1] The statement 'out of 1,000 classifier-flagged randomly sampled security-related issues, 57% didn't contain the string security' implies 43% did; reporting the full contingency would help assess reliance on the lexical signal.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 14.8% security-issue estimate is a model prediction grounded in independent human labels, not a fitted input or tag-based tautology.

full rationale

The paper's central claim—1,617,738 untagged security-related issues (14.8%)—comes from a supervised classifier, not from re-describing its own training labels. In Section 5.1, ground truth is created by sampling 1,000 security-tagged user issues and 1,000 untagged issues and having two coders independently label them (Cohen's kappa 0.74), yielding 1,042 security and 1,058 non-security labels; the labels are therefore human judgments, not the tag filter itself. The RoBERTa model is then applied to the 9,131,800 untagged issues, an out-of-sample extrapolation whose 14.8% output is not fixed by the 50/50 training balance. The paper also reports a held-out 100-issue manual confusion matrix (Table 6, 96% agreement), providing independent evidence. The Word2Vec tag-similarity step (Section 3.3) is used only to find candidate tags; the later classifier explicitly uses full title and body, and the limitations section acknowledges tag-based coverage limits. There is no load-bearing self-citation: the only same-author citation (Raj et al. [83]) is a coding-quantity precedent (100 of ~1,747 quotes), not a uniqueness theorem or a premise that forces the results. The skeptic's PR/issue contamination concern about the GitHub API collection in Section 3.2 is a data-validity risk that could bias counts, but it is not circular: including pull requests inflates denominators and numerators without making the ML output equal to its inputs by construction. No derivation step reduces to its own inputs, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a data-collection pipeline that assumes GitHub API issues exclude pull requests, and on ML models whose validation is not matched to the target population. The paper introduces no new physical or conceptual entities, but the operational construct 'security-related issue' is defined by the classifiers and their thresholds.

free parameters (2)
  • Word2Vec tag similarity threshold = 0.8
    Section 3.3 selects tags with similarity above 0.8 because "0.8 gave the most relevant output." This threshold determines the 0.13% tagged baseline and the positive training set for the downstream classifier.
  • RoBERTa classification decision threshold = not reported
    The paper does not report the probability threshold used to label 1,617,738 issues as security-related. With a balanced training set, the default 0.5 threshold likely over-predicts in an imbalanced population, and this choice directly controls the headline count.
assumptions (4)
  • domain assumption GitHub API 'issues' responses represent issue reports only, not pull requests.
    Section 3.2 says it collected "all the issues data" via the GitHub API, but the standard Issues REST endpoint returns pull requests as well. The paper does not state that pull_request entries were filtered, and several cited examples are pull requests.
  • domain assumption RoBERTa validation performance transfers from a balanced 2,000-issue sample to the imbalanced 9.1M untagged population.
    Section 5.1 validates the classifier on a 100-issue sample with only 9 security positives, then applies it to all untagged issues without calibration or population-level precision estimates.
  • domain assumption The theme hierarchy derived from 6,104 tagged security issues is representative of themes in the 1.6M predicted untagged issues.
    Section 5.2 trains a multi-label theme classifier on hand-coded quotes from the tagged set and applies it to the extended set, assuming no distribution shift between tagged and untagged security reports.
  • domain assumption Accounts with 'bot' in the username are bots.
    Sections 3.4 and 7.2 use username matching to count bot-created issues and comments. The paper itself notes that some accounts named 'bot' may not be real bots and that bots without 'bot' in the name are missed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of "I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages." pith.science (2026). https://pith.science/paper/G4IJ7OB5

@misc{pith2026250607728,
  author       = {Pith},
  title        = {Pith review of: "I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4IJ7OB5}},
  note         = {Machine review of arXiv:2506.07728}
}
read the original abstract

The npm (Node Package Manager) ecosystem is the most important package manager for JavaScript development with millions of users. Consequently, a plethora of earlier work investigated how vulnerability reporting, patch propagation, and in general detection as well as resolution of security issues in such ecosystems can be facilitated. However, understanding the ground reality of security-related issue reporting by users (and bots) in npm-along with the associated challenges has been relatively less explored at scale. In this work, we bridge this gap by collecting 10,907,467 issues reported across GitHub repositories of 45,466 diverse npm packages. We found that the tags associated with these issues indicate the existence of only 0.13% security-related issues. However, our approach of manual analysis followed by developing high accuracy machine learning models identify 1,617,738 security-related issues which are not tagged as security-related (14.8% of all issues) as well as 4,461,934 comments made on these issues. We found that the bots which are in wide use today might not be sufficient for either detecting or offering assistance. Furthermore, our analysis of user-developer interaction data hints that many user-reported security issues might not be addressed by developers-they are not tagged as security-related issues and might be closed without valid justification. Consequently, a correlation analysis hints that the developers quickly handle security issues with known solutions (e.g., corresponding to CVE). However, security issues without such known solutions (even with reproducible code) might not be resolved. Our findings offer actionable insights for improving security management in open-source ecosystems, highlighting the need for smarter tools and better collaboration. The data and code for this work is available at https://doi.org/10.5281/zenodo.15614029

Figures

Figures reproduced from arXiv: 2506.07728 by the authors.

Figure 1
Figure 1. Pipeline for filtering security issues and identifying [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of repositories across different tag [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 3
Figure 3. Wordcloud of most influential tokens identified [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

105 extracted references · 70 canonical work pages

  1. [6]

    https://github

    Bump mdx2-csf dependency by shilman · pull request #19885 · storybookjs/storybook. https://github. com/storybookjs/storybook/pull/19885

  2. [13]

    https://github.com/apache/superset/pull/ 30200

    fix(embedded): Dashboard screenshot should use gues- tuser by geido · pull request #30200 · apache/superset. https://github.com/apache/superset/pull/ 30200

  3. [1]

    https://github.com/ expressjs/express/pull/2748

    add request.local,inside,outside getters to test where a request is coming from by blitmap · pull request #2748 · expressjs/express. https://github.com/ expressjs/express/pull/2748

  4. [2]

    https://github.com/ MetaMask/snaps/issues/1018

    Additionally harden common endowments · issue #1018 · metamask/snaps. https://github.com/ MetaMask/snaps/issues/1018

  5. [3]

    https:// github.com/netlify/build/issues/193

    Allow plugins to access specific secret environment variables · issue #193 · netlify/build. https:// github.com/netlify/build/issues/193

  6. [4]

    https://github.com/ joystream/joystream/issues/2969

    Are unsettled english auctions safe? · issue #2969 · joystream/joystream. https://github.com/ joystream/joystream/issues/2969

  7. [5]

    https://github

    Bug with ‘addon-docs‘ and “ html element · issue #15810 · storybookjs/storybook. https://github. com/storybookjs/storybook/issues/15810

  8. [7]

    https://github.com/ apollographql/apollo-client/issues/3592

    cache leaks secrets · issue #3592 · apollographql/apollo-client. https://github.com/ apollographql/apollo-client/issues/3592

Show all 105 references
  1. [8]

    https://github.com/ SAP/spartacus/issues/4725

    Cannot remove coupon if the coupon code contains “.” · issue #4725 · sap/spartacus. https://github.com/ SAP/spartacus/issues/4725. 15

  2. [9]

    https://github

    Cannot run provider verification against https end- point that has ca, if called from a http endpoint · is- sue #281 · pact-foundation/pact-js. https://github. com/pact-foundation/pact-js/issues/281

  3. [10]

    https://github.com/ SonarSource/SonarJS/issues/4467

    Deprecate rule s5743 · issue #4467 · sonar- source/sonarjs. https://github.com/ SonarSource/SonarJS/issues/4467

  4. [11]

    https://github.com/angular/angular/issues/ 37442

    Docker best practices · issue #37442 · angular/angular. https://github.com/angular/angular/issues/ 37442

  5. [12]

    https://github.com/dotnet/aspnetcore/ issues/54412#issuecomment-1992596067

    Dynamically updating authentionschemes on autho- rizationpolicies · issue #54412 · dotnet/aspnetcore. https://github.com/dotnet/aspnetcore/ issues/54412#issuecomment-1992596067

  6. [14]

    · issue #98 · ollionorg/flow- core

    Found leaked credentials. · issue #98 · ollionorg/flow- core. https://github.com/ollionorg/ flow-core/issues/98

  7. [15]

    https://docs

    GitHub REST API documentation. https://docs. github.com/en/rest

  8. [16]

    https://github.com/pulumi/ pulumi-azure-native/issues/2408

    liststaticsitesecrets results are not marked as secrets · issue #2408 · pulumi/pulumi- azure-native. https://github.com/pulumi/ pulumi-azure-native/issues/2408

  9. [17]

    https: //github.com/grafana/grafana/issues/15591

    Move out server admin to a separate menu item on sidemenu · issue #15591 · grafana/grafana. https: //github.com/grafana/grafana/issues/15591

  10. [18]

    nonetype

    Opensea testnets - unsupported operand type(s) for /: “nonetype” and “nonetype” · issue #122 · projectopensea/opensea-js. https://github.com/ ProjectOpenSea/opensea-js/issues/122

  11. [20]

    https://github.com/ strapi/strapi/issues/12320

    Passwords are not validated upon user creation · is- sue #12320 · strapi/strapi. https://github.com/ strapi/strapi/issues/12320

  12. [21]

    https://github.com/ dotnet/aspnetcore/issues/56656

    Policyevaluator doesn’t have correct logic · issue #56656 · dotnet/aspnetcore. https://github.com/ dotnet/aspnetcore/issues/56656

  13. [22]

    https://github.com/pulumi/pulumi/issues/ 8946#issuecomment-1005987646

    Randomid still includes unencrypted key in state file · issue #8946 · pulumi/pulumi. https://github.com/pulumi/pulumi/issues/ 8946#issuecomment-1005987646

  14. [23]

    https://docs.npmjs.com/ cli/v8/using-npm/registry

    registry - npm docs. https://docs.npmjs.com/ cli/v8/using-npm/registry

  15. [24]

    https://github.com/expo/expo/ issues/19018

    Removing then re-installing an apk causes the se- curestore read to throw exceptions · issue #19018 · expo/expo. https://github.com/expo/expo/ issues/19018

  16. [25]

    https://github.com/grafana/ grafana/issues/1619

    Secure phantomjs png rendering · issue #1619 · grafana/grafana. https://github.com/grafana/ grafana/issues/1619

  17. [26]

    https://github.com/ Automattic/wp-calypso/issues/7017

    Security: Strings in list of access permissions of con- nected applications are not localized · issue #7017 · automattic/wp-calypso. https://github.com/ Automattic/wp-calypso/issues/7017

  18. [27]

    @storybook/addon-info > marksy > marked

    [security] vulnerability of low severity in “@storybook/addon-info > marksy > marked” · issue #7842 · storybookjs/storybook. https: //github.com/storybookjs/storybook/issues/ 7842#issuecomment-641845722

  19. [28]

    https://github

    self.url method causes trouble with ie11 (routing) · issue #16825 · angular/angular.js. https://github. com/angular/angular.js/issues/16825

  20. [29]

    https://github

    Setting admin_pass shows password in log · issue #4806 · rocketchat/rocket.chat. https://github. com/RocketChat/Rocket.Chat/issues/4806

  21. [30]

    https:// github.com/Agoric/agoric-sdk/issues/6071

    Test that offer snoozing works for both sides of the psm contract · issue #6071 · agoric/agoric-sdk. https:// github.com/Agoric/agoric-sdk/issues/6071

  22. [31]

    https://github

    Warn at install about path lookup issues in global in- stalls · issue #7255 · npm/npm. https://github. com/npm/npm/issues/7255

  23. [32]

    IEEE Press, 2017

    ASE ’17: Proceedings of the 32nd IEEE/ACM Interna- tional Conference on Automated Software Engineering. IEEE Press, 2017

    2017

  24. [33]

    Asso- ciation for Computing Machinery

    ICSE-SEIP ’18: Proceedings of the 40th International Conference on Software Engineering: Software Engi- neering in Practice, New York, NY , USA, 2018. Asso- ciation for Computing Machinery

    2018

  25. [34]

    Gerosa, and Emad Shihab

    Ahmad Abdellatif, Mairieli Wessel, Igor Steinmacher, Marco A. Gerosa, and Emad Shihab. Bothunter: an approach to detect software bots in github. InProceed- ings of the 19th International Conference on Mining Software Repositories, MSR ’22, page 6–17, New York, NY , USA, 2022. ...

    2022

  26. [35]

    Understanding the [cls] token in bert: A com- prehensive guide, 2021

    Aditya. Understanding the [cls] token in bert: A com- prehensive guide, 2021. 16

    2021

  27. [36]

    Optuna: A next- generation hyperparameter optimization framework, 2019

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework, 2019

    2019

  28. [37]

    On the discoverability of npm vul- nerabilities in node.js projects.ACM Trans

    Mahmoud Alfadel, Diego Elias Costa, Emad Shihab, and Bram Adams. On the discoverability of npm vul- nerabilities in node.js projects.ACM Trans. Softw. Eng. Methodol., 32(4), May 2023

    2023

  29. [38]

    Qwen tech- nical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Si...

    2023 arXiv

  30. [39]

    Buck and Devon F

    Amber M. Buck and Devon F. Ralston. I didn’t sign up for your research study: The ethics of using “public” data.Computers and Composition, 61:102655, 2021. Rhetorics of Data: Collection, Consent, & Critical Dig- ital Literacies

    2021

  31. [40]

    Noah Bühlmann and Mohammad Ghafari. How do de- velopers deal with security issue reports on github? In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, SAC ’22, page 1580–1589, New York, NY , USA, 2022. Association for Computing Ma- chinery

    2022

  32. [41]

    Exploring the use of labels to categorize issues in open-source software projects

    Jordi Cabot, Javier Luis Cánovas Izquierdo, Valerio Cosentino, and Belén Rolandi. Exploring the use of labels to categorize issues in open-source software projects. In2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pages 550–...

    2015

  33. [42]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Rob...

    2022

  34. [43]

    Ali Babar

    Roland Croft, Dominic Newlands, Ziyu Chen, and M. Ali Babar. An empirical study of rule-based and learning-based approaches for static application secu- rity testing. InProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engi- neering and Measureme...

    2021

  35. [44]

    An em- pirical study of blockchain repositories in github

    Ajoy Das, Gias Uddin, and Guenther Ruhe. An em- pirical study of blockchain repositories in github. In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering, EASE ’22, page 211–220, New York, NY , USA, 2022. Association for Compu...

    2022

  36. [45]

    On the impact of security vulnerabilities in the npm package dependency network

    Alexandre Decan, Tom Mens, and Eleni Constanti- nou. On the impact of security vulnerabilities in the npm package dependency network. InProceedings of the 15th International Conference on Mining Software Repositories, MSR ’18, page 181–191, New York, NY , USA, 2018. Associatio...

    2018

  37. [46]

    An empirical comparison of dependency network evo- lution in seven software packaging ecosystems.Em- pirical Software Engineering, 24:381–416, 2019

    Alexandre Decan, Tom Mens, and Philippe Grosjean. An empirical comparison of dependency network evo- lution in seven software packaging ecosystems.Em- pirical Software Engineering, 24:381–416, 2019. Pub- lished: 10 February 2018, Issue Date: 15 February 2019

    2019

  38. [47]

    On the use of github actions in software development repositories

    Alexandre Decan, Tom Mens, Pooya Rostami Mazrae, and Mehdi Golzadeh. On the use of github actions in software development repositories. In2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 235–245, 2022

    2022

  39. [48]

    On measuring affects of github issues’ commenters

    Giuseppe Destefanis, Marco Ortu, David Bowes, Michele Marchesi, and Roberto Tonelli. On measuring affects of github issues’ commenters. InProceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering, SEmotion ’18, page 14–19, New York, NY , USA, 2...

    2018

  40. [49]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin. Bert: Pre-training of deep bidirec- tional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  41. [50]

    Detecting and characterizing bots that com- mit code

    Tapajit Dey, Sara Mousavi, Eduardo Ponce, Tanner Fry, Bogdan Vasilescu, Anna Filippova, and Audris Mockus. Detecting and characterizing bots that com- mit code. InProceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, page 209–219, New York,...

    2020

  42. [51]

    Diniz, Daniel Cruz, Fabio Ferreira, Cleiton Tavares, and Eduardo Figueiredo

    Joao P. Diniz, Daniel Cruz, Fabio Ferreira, Cleiton Tavares, and Eduardo Figueiredo. Github label em- beddings. In2020 IEEE 20th International Working Conference on Source Code Analysis and Manipula- tion (SCAM), pages 249–253, 2020

    2020

  43. [52]

    Managing labels

    GitHub Docs. Managing labels. https://docs.github.com/en/issues/ using-labels-and-milestones-to-track-work/ managing-labels, 2023

    2023

  44. [53]

    Ethical issues in qualitative research on internet communities.BMJ, 323(7321):1103–1105, 2001

    Gunther Eysenbach and James E Till. Ethical issues in qualitative research on internet communities.BMJ, 323(7321):1103–1105, 2001

    2001

  45. [54]

    Codebert: A pre-trained model for programming and natural languages.arXiv preprint arXiv:2002.08155, 2020

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages.arXiv preprint arXiv:2002.08155, 2020

    2002 arXiv

  46. [55]

    Containing malicious package up- dates in npm with a lightweight permission system

    Gabriel Ferreira, Limin Jia, Joshua Sunshine, and Christian Kästner. Containing malicious package up- dates in npm with a lightweight permission system. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 1334–1346, 2021

    2021

  47. [56]

    Codeql: Semantic code analysis engine

    GitHub. Codeql: Semantic code analysis engine. https://codeql.github.com/

  48. [57]

    Dependabot: Keep your dependencies up to date.https://github.com/dependabot

    GitHub. Dependabot: Keep your dependencies up to date.https://github.com/dependabot

  49. [58]

    Github marketplace

    GitHub. Github marketplace. https://github.com/ marketplace, 2017

    2017

  50. [59]

    Privately reporting a security vulnerability

    GitHub. Privately reporting a security vulnerability. https://docs.github.com/en/code-security/ security-advisories, 2023

    2023

  51. [60]

    A ground-truth dataset and clas- sification model for detecting bots in github issue and pr comments.Journal of Systems and Software, 175:110911, 2021

    Mehdi Golzadeh, Alexandre Decan, Damien Legay, and Tom Mens. A ground-truth dataset and clas- sification model for detecting bots in github issue and pr comments.Journal of Systems and Software, 175:110911, 2021

    2021

  52. [61]

    Recogniz- ing bot activity in collaborative software development

    Mehdi Golzadeh, Tom Mens, Alexandre Decan, Eleni Constantinou, and Natarajan Chidambaram. Recogniz- ing bot activity in collaborative software development. IEEE Softw., 39(5):56–61, September 2022

    2022

  53. [62]

    Outside the comfort zone: Analysing llm capabilities in software vulnerability detection

    Yuejun Guo, Constantinos Patsakis, Qiang Hu, Qiang Tang, and Fran Casino. Outside the comfort zone: Analysing llm capabilities in software vulnerability detection. InComputer Security – ESORICS 2024: 29th European Symposium on Research in Computer Security, Bydgoszcz, Poland, ...

    2024

  54. [63]

    Gunnar Harboe and Elaine M. Huang. Real-world affinity diagramming practices: Bridging the paper- digital gap. InProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, page 95–104, New York, NY , USA, 2015. Association for Computing Machinery

    2015

  55. [64]

    Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020

    2006 arXiv

  56. [65]

    Empirical analysis of vul- nerabilities life cycle in golang ecosystem

    Jinchang Hu, Lyuye Zhang, Chengwei Liu, Sen Yang, Song Huang, and Yang Liu. Empirical analysis of vul- nerabilities life cycle in golang ecosystem. InProceed- ings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY , USA, 2024. Associ...

    2024

  57. [66]

    Spiderscan: Practical detection of malicious npm packages based on graph-based behav- ior modeling and matching

    Yiheng Huang, Ruisi Wang, Wen Zheng, Zhuotong Zhou, Susheng Wu, Shulin Ke, Bihuan Chen, Shan Gao, and Xin Peng. Spiderscan: Practical detection of malicious npm packages based on graph-based behav- ior modeling and matching. InProceedings of the 39th IEEE/ACM International Con...

    2024

  58. [67]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    arXiv

  59. [68]

    Recommending tags for pull requests in github.Infor- mation and Software Technology, 129:106394, 2021

    Jing Jiang, Qiudi Wu, Jin Cao, Xin Xia, and Li Zhang. Recommending tags for pull requests in github.Infor- mation and Software Technology, 129:106394, 2021

    2021

  60. [69]

    What is a minimal reproducible example (mre)? https://label.dev/ articles/minimal-reproducible-example/ #what-is-a-mre, 2023

    Label.dev. What is a minimal reproducible example (mre)? https://label.dev/ articles/minimal-reproducible-example/ #what-is-a-mre, 2023

    2023

  61. [70]

    On the feasibility of cross-language detection of malicious packages in npm and pypi

    Piergiorgio Ladisa, Serena Elisa Ponta, Nicola Ron- zoni, Matias Martinez, and Olivier Barais. On the feasibility of cross-language detection of malicious packages in npm and pypi. InProceedings of the 39th Annual Computer Security Applications Conference, ACSAC ’23, page 71–8...

    2023

  62. [71]

    Software bots.IEEE Software, 35(1):18–23, 2018

    Carlene Lebeuf, Margaret-Anne Storey, and Alexey Zagalsky. Software bots.IEEE Software, 35(1):18–23, 2018. 18

    2018

  63. [72]

    Patchfinder: A two-phase approach to security patch tracing for disclosed vulnerabilities in open-source software

    Kaixuan Li, Jian Zhang, Sen Chen, Han Liu, Yang Liu, and Yixiang Chen. Patchfinder: A two-phase approach to security patch tracing for disclosed vulnerabilities in open-source software. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analy...

    2024

  64. [73]

    Code of conduct conversations in open source software projects on github.Proc

    Renee Li, Pavitthra Pandurangan, Hana Frluckaj, and Laura Dabbish. Code of conduct conversations in open source software projects on github.Proc. ACM Hum.- Comput. Interact., 5(CSCW1), April 2021

    2021

  65. [74]

    Demystifying the vulnera- bility propagation and its evolution via dependency trees in the npm ecosystem

    Chengwei Liu, Sen Chen, Lingling Fan, Bihuan Chen, Yang Liu, and Xin Peng. Demystifying the vulnera- bility propagation and its evolution via dependency trees in the npm ecosystem. InProceedings of the 44th International Conference on Software Engineer- ing, ICSE ’22, page 672...

  66. [75]

    Roberta: A robustly optimized bert pre- training approach.arXiv preprint arXiv:1907.11692, 364, 2019

    Yinhan Liu. Roberta: A robustly optimized bert pre- training approach.arXiv preprint arXiv:1907.11692, 364, 2019

    1907 arXiv

  67. [76]

    Sane github labels

    Dave Lunny. Sane github labels. https://medium.com/@dave_lunny/ sane-github-labels-c5d2e6004b63, 2016

    2016

  68. [77]

    Zeyang Ma, Shouvick Mondal, Tse-Hsun Peter Chen, Haoxiang Zhang, and Ahmed E. Hassan. Vulnet: Towards improving vulnerability management in the maven ecosystem.Empirical Software Engineering, 29, 06 2024

    2024

  69. [78]

    Samim Mirhosseini and Chris Parnin. Can automated pull requests encourage software developers to upgrade out-of-date dependencies? InProceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE ’17, page 84–94. IEEE Press, 2017

    2017

  70. [79]

    npm: The package manager for javascript

    npm, Inc. npm: The package manager for javascript. https://www.npmjs.com/, 2010

    2010

  71. [80]

    Backstabber’s knife collection: A review of open source software supply chain attacks

    Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. Backstabber’s knife collection: A review of open source software supply chain attacks. In Clé- mentine Maurice, Leyla Bilge, Gianluca Stringhini, and Nuno Neves, editors,Detection of Intrusions and Malware, and Vulnera...

    2020

  72. [81]

    Oliveira, Ana Flávia C

    Gabriel P. Oliveira, Ana Flávia C. Moura, Natér- cia A. Batista, Michele A. Brandão, Andre Hora, and Mirella M. Moro. How do developers collaborate? investigating github heterogeneous networks.Software Quality Journal, 31(1):211–241, September 2022

    2022

  73. [82]

    Santosa, Asankhaya Sharma, and David Lo

    Gede Artha Azriadi Prana, Abhishek Sharma, Lwin Khin Shar, Darius Foo, Andrew E. Santosa, Asankhaya Sharma, and David Lo. Out of sight, out of mind? how vulnerable dependencies affect open-source projects.Empirical Softw. Engg., 26(4), July 2021

    2021

  74. [83]

    i just hated it and i want my money back

    Rohit Raj, Mridul Newar, and Mainack Mondal. "i just hated it and i want my money back": Data-driven understanding of mobile VPN service switching prefer- ences in the wild. In33rd USENIX Security Symposium (USENIX Security 24), pages 6021–6037, Philadelphia, PA, August 2024. ...

    2024

  75. [84]

    Minimal bug reports

    Matthew Rocklin. Minimal bug reports. https: //matthewrocklin.com/minimal-bug-reports. html, 2018

    2018

  76. [85]

    SAGE Publications, London, England, 3rd edition, 2015

    Johnny Saldana.The Coding Manual for Qualitative Researchers. SAGE Publications, London, England, 3rd edition, 2015

    2015

  77. [86]

    Saturation in qualitative research: exploring its conceptualization and opera- tionalization.Quality & quantity, 52:1893–1907, 2018

    Benjamin Saunders, Julius Sim, Tom Kingstone, Shula Baker, Jackie Waterfield, Bernadette Bartlam, Heather Burroughs, and Clare Jinks. Saturation in qualitative research: exploring its conceptualization and opera- tionalization.Quality & quantity, 52:1893–1907, 2018

    1907

  78. [87]

    Practical automated detection of malicious npm packages

    Adriana Sejfia and Max Schäfer. Practical automated detection of malicious npm packages. InProceedings of the 44th International Conference on Software En- gineering, ICSE ’22, page 1681–1692, New York, NY , USA, 2022. Association for Computing Machinery

    2022

  79. [88]

    Do users change their settings? https://archive.uie.com/brainsparks/2011/ 09/14/do-users-change-their-settings/ , 2011

    Jared Spool. Do users change their settings? https://archive.uie.com/brainsparks/2011/ 09/14/do-users-change-their-settings/ , 2011

    2011

  80. [89]

    Dis- rupting developer productivity one bot at a time

    Margaret-Anne Storey and Alexey Zagalsky. Dis- rupting developer productivity one bot at a time. In Proceedings of the 2016 24th ACM SIGSOFT Inter- national Symposium on Foundations of Software En- gineering, FSE 2016, page 928–931, New York, NY , USA, 2016. Association for Co...

    2016

  81. [90]

    A first look at good first issues on github

    Xin Tan, Minghui Zhou, and Zeyu Sun. A first look at good first issues on github. InProceedings of the 28th ACM Joint Meeting on European Software Engineer- ing Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, page 398–409, New York, NY , USA...

    2020

  82. [91]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  83. [92]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  84. [93]

    Github’s dependabot is caus- ing a ton of “spam” in our frontend (angular) reposit

    Hacker News User. Github’s dependabot is caus- ing a ton of “spam” in our frontend (angular) reposit... https://news.ycombinator.com/item? id=27929596, 2022

    2022

  85. [94]

    Honestly dependabot is so bad i’m surprised anything other than the smallest pro

    Hacker News User. Honestly dependabot is so bad i’m surprised anything other than the smallest pro... https://news.ycombinator.com/item?id= 31963643, 2022

    2022

  86. [95]

    Mapping user preference to privacy default settings.ACM Trans

    Jason Watson, Heather Richter Lipford, and Andrew Besmer. Mapping user preference to privacy default settings.ACM Trans. Comput.-Hum. Interact., 22(6), November 2015

    2015

  87. [97]

    Wiese, Ivanilton Polato, Ana Paula Chaves, and Marco A

    Mairieli Wessel, Bruno Mendes de Souza, Igor Stein- macher, Igor S. Wiese, Ivanilton Polato, Ana Paula Chaves, and Marco A. Gerosa. The power of bots: Characterizing and understanding bots in oss projects.Proc. ACM Hum.-Comput. Interact., 2(CSCW), November 2018

    2018

  88. [98]

    The inconve- nient side of software bots on pull requests

    Mairieli Wessel and Igor Steinmacher. The inconve- nient side of software bots on pull requests. InProceed- ings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW’20, page 51–55, New York, NY , USA, 2020. Association for Computing Machinery

    2020

  89. [99]

    Identifying affected libraries and their ecosystems for open source software vulnerabil- ities

    Susheng Wu, Wenyan Song, Kaifeng Huang, Bihuan Chen, and Xin Peng. Identifying affected libraries and their ecosystems for open source software vulnerabil- ities. InProceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering, ICSE ’24, New York, NY , US...

    2024

  90. [100]

    Understanding the threats of up- stream vulnerabilities to downstream projects in the maven ecosystem

    Yulun Wu, Zeliang Yu, Ming Wen, Qiang Li, Deqing Zou, and Hai Jin. Understanding the threats of up- stream vulnerabilities to downstream projects in the maven ecosystem. In2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE), pages 1046–1058, 2023

    2023

  91. [101]

    Maltracker: A fine-grained npm malware tracker copi- loted by llm-enhanced dataset

    Zeliang Yu, Ming Wen, Xiaochen Guo, and Hai Jin. Maltracker: A fine-grained npm malware tracker copi- loted by llm-enhanced dataset. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1759–1771, New York, NY , USA, ...

    2024

  92. [102]

    On the impact of security vulnerabilities in the npm and rubygems dependency networks.Empirical Software Engineering, 27(107), 2022

    Abdelrahman Zerouali, Tom Mens, Alexandre Decan, et al. On the impact of security vulnerabilities in the npm and rubygems dependency networks.Empirical Software Engineering, 27(107), 2022

    2022

  93. [103]

    Killing two birds with one stone: Malicious package detection in npm and pypi using a single model of malicious behavior sequence.ACM Trans

    Junan Zhang, Kaifeng Huang, Yiheng Huang, Bihuan Chen, Ruisi Wang, Chong Wang, and Xin Peng. Killing two birds with one stone: Malicious package detection in npm and pypi using a single model of malicious behavior sequence.ACM Trans. Softw. Eng. Methodol., November 2024. Just Accepted

    2024

  94. [104]

    Mitigating persistence of open-source vulnerabilities in maven ecosystem

    Lyuye Zhang, Chengwei Liu, Sen Chen, Zhengzi Xu, Lingling Fan, Lida Zhao, Yiran Zhang, and Yang Liu. Mitigating persistence of open-source vulnerabilities in maven ecosystem. In2023 38th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE), pages 191–203, 2023

    2023

  95. [105]

    Successfully resolved

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019. A Details of npm packages In this section, we provide additional details related to the dataset used in our study...

    1904 arXiv

  96. [107]

    role": "user

    Also the five most frequently mentioned CVE IDs (strictly CVE-IDs are mentioned here) are in the Table 28. E Details related to bots An overview of various activities performed by bots in user- reported issues, along with their frequency in security-related issues, is provided...

    2023

  97. [2022]

    Association for Computing Machinery

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.