REVIEW 3 major objections 3 minor 25 references
Policy-driven Software Bill of Materials on GitHub: An Empirical Study
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Policy-driven SBOMs are rare on GitHub—0.56% of popular repositories—and even those list thousands of known vulnerabilities.
desk verdict A useful, honestly-reported SBOM prevalence baseline that needs a recall audit before the 0.56% figure can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a taxonomy-based filtering pipeline: SBOMs are first split into In-the-Lab (research-generated) and In-the-Wild, then In-the-Wild is further divided into dependency management, security scanning, licensing, test data, third-party, and self-license categories. A set of 29 exclusion regexes, refined iteratively and validated by a manually checked sample of 100 out of 620 files, isolates the 'policy-driven' subset. This classification is what gives the 0.56% prevalence figure its meaning, so the taxonomy and filter carry the argument's weight.
What would settle it
Re-run the repository collection and have three independent reviewers manually classify all 620 kept files and a random sample of the excluded ones; if the misclassification rate exceeds the paper's stated 5% margin, the prevalence figure and the derived vulnerability and license counts would need revision.
Extended reading notes
Core claim
The paper establishes that policy-driven SBOMs—SBOMs created to support security risk assessment, supply-chain transparency, or license compliance, as opposed to test data, third-party artifacts, or self-license files—are almost absent from popular open-source projects. Out of 26,823 repositories processed, only 152 (0.56%) contain such an SBOM. Across the 620 policy-driven SBOM files, the paper counts 25,430 unique dependencies that carry 2,202 unique known vulnerabilities reported in 19,225 CVEs, and notes that 22.41% of the files list no licensing information at all. It also reports an average SBOM quality score of 7.07 on a 0–10 scale, with the most common vulnerabilities tied to weak in
Load-bearing premise
The filtering step that decides which SBOM files count as policy-driven—29 exclusion regexes plus a manual check of only 100 of the 620 files—is accurate enough that the 0.56% prevalence and the 620-file dataset are not distorted by misclassification.
Editorial extensions
If this is right
- Researchers mining SBOMs from public repositories should distinguish policy-driven files from test or research artifacts, or their findings about real-world practice may be skewed.
- The published dataset of 620 policy-driven SBOMs can serve as a benchmark for evaluating SBOM quality tools and vulnerability scanners on real artifacts rather than synthetic ones.
- Open-source maintainers and software vendors integrating open-source dependencies should expect that SBOM-driven vulnerability and license checks will reveal unpatched known vulnerabilities and missing license metadata at scale.
- The low prevalence suggests that upcoming regulatory mandates will require substantial tooling and process changes before most projects can produce compliant SBOMs.
- The 22.41% of policy-driven SBOMs without license information points to a gap that license-compliance automation will need to handle explicitly.
Reading between the lines
- Because SourceGraph only indexes files smaller than 1MB and only the main branch, the true prevalence of policy-driven SBOMs could be lower (missed large or branched files) or higher (some excluded files may still be policy-driven), so 0.56% is best read as a lower-bound estimate for mainstream repositories.
- The concentration of unpatched, high-severity vulnerabilities in declared dependencies implies that SBOMs alone do little without a routine scanning and patching loop; regulations will likely need to pair SBOM production with mandatory update timelines to be effective.
- A natural extension is a time-to-patch analysis: correlating when a vulnerability is disclosed with when (or whether) the project updates its SBOM-declared dependency version would turn the static dataset into a measure of supply-chain responsiveness.
- The finding that 22.41% of policy-driven SBOMs lack license fields suggests that companies consuming open-source software may need to supplement SBOM data with external license databases, since the SBOM format alone does not guarantee license coverage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a mining study of GitHub repositories to identify "policy-driven" SBOM files, defined as SBOMs created for security or compliance purposes rather than as test fixtures, third-party artifacts, or research-generated files. The authors retrieve 26,823 repositories (top-starred GitHub repos per language plus 198 SourceGraph-only matches), apply 29 exclusion regexes plus manual checks, retain 620 SBOM files from 152 repositories (0.56%), and analyze them with sbomqs and osv-scanner. They report 181,283 dependency entries (25,430 unique), an average quality score of 7.07/10, 19,225 CVEs mapping to 2,202 unique vulnerabilities, and 22.41% of SBOMs without license information. The paper contributes a taxonomy, a dataset, and a replication package.
Significance. The value of this study lies in shifting SBOM research from synthetic artifacts to publicly available, policy-oriented SBOMs and in quantifying the current state of practice against upcoming regulatory mandates. The paper is transparent: it provides the dataset, the regexes, tool versions, and it acknowledges several tooling and search limitations. If the prevalence and vulnerability numbers are taken at face value, they are useful baseline evidence for the SBOM research community. However, the headline prevalence depends on two methodological choices that are not yet fully validated: the recall of the exclusion regexes and the composition of the denominator. These affect the central claim and need to be addressed in revision.
major comments (3)
- [Section 3.2 and Table 1] The filtering pipeline is validated only for precision, not recall. The manual audit of 100 retained SBOMs can only bound the false-positive rate in the retained set; it says nothing about files removed by the 29 regexes. Several patterns in Table 1 are over-broad for a policy-driven classification (e.g., 'dependenc*', 'lib*', 'package*', 'test'), so a legitimate file such as dependencies.spdx.json or test/sbom.json could be excluded. Because the headline result is 152 policy-SBOM repositories, even a small number of such false negatives changes the reported 0.56% materially. The paper needs a recall audit on a sample of excluded files, or a sensitivity analysis with the regexes relaxed, and must not state that manual inspection showed 'no false positives' (with zero failures in 100, the upper 95% confidence bound is about 3%).
- [Section 3.1 and Section 4.1] The denominator is not a pure popular-repository cohort. The 26,823 repositories include 198 SourceGraph-only repositories that were added because the SBOM search regexes matched their contents. These repositories are therefore selected on the outcome (presence of SBOM-like files), which biases the prevalence estimate for 'popular open-source projects'. Please report the prevalence computed on the 26,625 top-starred repositories alone, and separately report how many of the 152 policy-SBOM repositories come from the SourceGraph-only subset. Without this, RQ1's answer as stated is not fully supported.
- [Section 5.1 and Section 4.3] The tooling failures are acknowledged but not reflected in the result tables. The osv-scanner failed for 22% of SBOMs, and sbomqs could not parse 66% of SPDX tag-value files; yet Section 4.3 reports aggregate totals such as 19,225 CVEs and 2,202 unique vulnerabilities without stating the effective number of successfully scanned SBOMs or the distribution of failures by format/version. This makes the vulnerability and licensing statistics incomplete in an unquantified way. Please report the effective N for each analysis, test whether the failures are correlated with format or generation tool, and characterize the resulting direction of bias (e.g., whether the totals are underestimates).
minor comments (3)
- [Section 3.1] The star threshold is lowered from 100 to 35 for some languages. Please report the per-language thresholds or include them in the replication package, since the denominator's composition depends on them.
- [Section 5.1 and typography] Typographical issues: 'OW ASP SCVS' should be 'OWASP SCVS'; Section 4.1 and Table 4 contain 'Y AML' for YAML; Table 3 contains 'V ersion'; the introduction contains 'polices' for 'policies'; Section 5.2 contains 'When investing RQ3', likely meaning 'In answering RQ3'.
- [Section 4.2 and Table 4] Table 4 reports quality scores for SPDX tag-value despite the 66% parse failure noted in Section 5.1. Clarify which subset of tag-value files is represented and whether the median/max values are based only on the successfully parsed files.
Circularity Check
No circularity: the study is an empirical measurement pipeline with no fitted parameter reused as a prediction and no load-bearing self-citation.
full rationale
The derivation chain is: define a taxonomy of SBOM uses (policy-driven vs. in-the-lab/test/3rd-party/self-license); mine GitHub/SourceGraph; apply exclusion regexes; manually audit 100/620 files; then count prevalence, qualities via sbomqs, vulnerabilities via osv-scanner. The central result (0.56% of 26,823 repositories contain policy-driven SBOM) is a measurement over the authors' operationalized definition, not a quantity derived from that definition by algebra or by fitting. No equation equates the output to an input: the quality scores come from an external tool, the vulnerability counts from osv-scanner's advisory databases, and the license data from the SBOM contents. The paper contains no self-citation that is load-bearing; previous work (Torres-Arias et al., Nocera et al., Soeiro et al.) is used for context and comparison, not to establish the current paper's claims. The acknowledged limitations (regex false negatives/positives, SourceGraph indexing, tool parse failures) concern measurement validity, not circularity. A recall audit would improve robustness, but that is a correctness/threat-to-validity concern, not circularity. Therefore score 0.
Assumptions & free parameters
free parameters (2)
- min_stars_threshold =
100 (lowered to 35 for some languages)
- artifact_size_limit =
100MB
assumptions (5)
- domain assumption SourceGraph's index (main branch, files under 1MB, repos under 10GB) is a representative window on GitHub SBOMs.
- domain assumption SPDX and CycloneDX are the only relevant SBOM formats, and the required attributes (SPDXID, bomFormat) are sufficient to identify SBOM files.
- domain assumption Manual inspection of 100 randomly sampled files validates the policy-driven label for all 620 files.
- domain assumption osv-scanner and its 28 advisory databases provide a complete-enough vulnerability picture of the declared dependencies.
- domain assumption Popularity can be operationalized by GitHub star counts with thresholds of 100 or 35.
Cite this review
Pith. "Pith review of Policy-driven Software Bill of Materials on GitHub: An Empirical Study." pith.science (2026). https://pith.science/paper/WEBJBG2K
@misc{pith2026250901255,
author = {Pith},
title = {Pith review of: Policy-driven Software Bill of Materials on GitHub: An Empirical Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/WEBJBG2K}},
note = {Machine review of arXiv:2509.01255}
}
read the original abstract
Background. The Software Bill of Materials (SBOM) is a machine-readable list of all the software dependencies included in a software. SBOM emerged as way to assist securing the software supply chain. However, despite mandates from governments to use SBOM, research on this artifact is still in its early stages. Aims. We want to understand the current state of SBOM in open-source projects, focusing specifically on policy-driven SBOMs, i.e., SBOM created to achieve security goals, such as enhancing project transparency and ensuring compliance, rather than being used as fixtures for tools or artificially generated for benchmarking or academic research purposes. Method. We performed a mining software repository study to collect and carefully select SBOM files hosted on GitHub. We analyzed the information reported in policy-driven SBOMs and the vulnerabilities associated with the declared dependencies by means of descriptive statistics. Results. We show that only 0.56% of popular GitHub repositories contain policy-driven SBOM. The declared dependencies contain 2,202 unique vulnerabilities, while 22% of them do not report licensing information. Conclusion. Our findings provide insights for SBOM usage to support security assessment and licensing.
Figures
Reference graph
Works this paper leans on
-
[1]
Defense-in-depth security strategy in log4j vulnerability analysis
Sylvia Feng and Muharman Lubis. Defense-in-depth security strategy in log4j vulnerability analysis. In 2022 International Conference Advancement in Data Science, E-learning and Information Systems (ICADEIS), pages 01–04. IEEE, 2022. 10 A PREPRINT - SEPTEMBER 9, 2025
work page 2022
-
[2]
Navigating the solarwinds supply chain attack.The Procurement Lawyer, 56(2), 2021
Evan D Wolff, KM Growley, MG Gruden, et al. Navigating the solarwinds supply chain attack.The Procurement Lawyer, 56(2), 2021
work page 2021
-
[3]
A systematic analysis of the event- stream incident
Iosif Arvanitis, Grigoris Ntousakis, Sotiris Ioannidis, and Nikos Vasilakis. A systematic analysis of the event- stream incident. In Proceedings of the 15th European Workshop on Systems Security, pages 22–28, 2022
work page 2022
-
[4]
Log4shell: Redefining the web attack surface
Douglas Everson, Long Cheng, and Zhenkai Zhang. Log4shell: Redefining the web attack surface. In Workshop on Measurements, Attacks, and Defenses for the Web (MADWeb) 2022, 2022
work page 2022
-
[5]
Software bills of materials are required
Nusrat Zahan, Elizabeth Lin, Mahzabin Tamanna, William Enck, and Laurie Williams. Software bills of materials are required. are we there yet? IEEE Security & Privacy, 21(2):82–88, 2023
work page 2023
-
[6]
The state of software bill of materials (sbom) and cybersecurity readiness
Stephen Hendrick and J Zemlin. The state of software bill of materials (sbom) and cybersecurity readiness. Technical report, The Linux Foundation, 2022
work page 2022
-
[7]
Charting the path to sbom adoption: A business stakeholder-centric approach
Berend Kloeg, Aaron Yi Ding, Sjoerd Pellegrom, and Yury Zhauniarovich. Charting the path to sbom adoption: A business stakeholder-centric approach. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, pages 1770–1783, 2024
work page 2024
-
[8]
Trevor Stalnaker, Nathan Wintersgill, Oscar Chaparro, Massimiliano Di Penta, Daniel M German, and Denys Poshyvanyk. Boms away! inside the minds of stakeholders: A comprehensive study of bills of materials for software systems. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , pages 1–13, 2024
work page 2024
Show all 25 references
-
[9]
An empirical study on software bill of materials: Where we stand and the road ahead
Boming Xia, Tingting Bi, Zhenchang Xing, Qinghua Lu, and Liming Zhu. An empirical study on software bill of materials: Where we stand and the road ahead. arXiv, 2023
2023
-
[10]
A viewpoint on knowing software: Bill of materials quality when you see it
Santiago Torres-Arias, Dan Geer, and John Speed Meyers. A viewpoint on knowing software: Bill of materials quality when you see it. IEEE Security & Privacy, 21(6):50–54, 2023
2023
-
[11]
A landscape study of open source and proprietary tools for software bill of materials (sbom)
Mehdi Mirakhorli, Derek Garcia, Schuyler Dillon, Kevin Laporte, Matthew Morrison, Henry Lu, Viktoria Koscinski, and Christopher Enoch. A landscape study of open source and proprietary tools for software bill of materials (sbom). arXiv preprint arXiv:2402.11151, 2024
2024 arXiv
-
[12]
Assessing security risks of software supply chains using software bill of materials
Eric O’Donoghue, Ann Marie Reinhold, and Clemente Izurieta. Assessing security risks of software supply chains using software bill of materials. In 2nd International Workshop on Mining Software Repositories for Privacy and Security, MSR4P&S,(SANER 2024), Rovaniemi, Finland, 2024
2024
-
[13]
Challenges of producing software bill of materials for java
Musard Balliu, Benoit Baudry, Sofia Bobadilla, Mathias Ekstedt, Martin Monperrus, Javier Ron, Aman Sharma, Gabriel Skoglund, C´esar Soto-Valero, and Martin Wittlinger. Challenges of producing software bill of materials for java. IEEE Security & Privacy, 2023
2023
-
[14]
On the correctness of metadata-based sbom generation: A differential analysis approach
Sheng Yu, Wei Song, Xunchao Hu, and Heng Yin. On the correctness of metadata-based sbom generation: A differential analysis approach. In2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 29–36. IEEE, 2024
2024
-
[15]
Sbom generation tools under microscope: A focus on the npm ecosystem
Md Fazle Rabbi, Arifa Islam Champa, Costain Nachuma, and Minhaz Fahim Zibran. Sbom generation tools under microscope: A focus on the npm ecosystem. In Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, pages 1233–1241, 2024
2024
-
[16]
License usage and changes: a large-scale study on github
Christopher Vendome, Gabriele Bavota, Massimiliano Di Penta, Mario Linares-V ´asquez, Daniel German, and Denys Poshyvanyk. License usage and changes: a large-scale study on github. Empirical Software Engineering, 22:1537–1577, 2017
2017
-
[17]
Types of software bill of materials (sbom)
Kate Stewart and Melissa Rhodes. Types of software bill of materials (sbom). https://web. archive.org/web/20240907164647/https://www.cisa.gov/sites/default/files/2023-04/ sbom-types-document-508c.pdf
2023
-
[18]
Software bill of materials adoption: a mining study from github
Sabato Nocera, Simone Romano, Massimiliano Di Penta, Rita Francese, and Giuseppe Scanniello. Software bill of materials adoption: a mining study from github. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 39–49. IEEE, 2023
2023
-
[19]
Wild sboms: a large-scale dataset of software bills of materials from public code, 2025
Luıs Soeiro, Thomas Robert, and Stefano Zacchiroli. Wild sboms: a large-scale dataset of software bills of materials from public code, 2025
2025
-
[20]
Mining software repositories to assist developers and support managers
Ahmed E Hassan. Mining software repositories to assist developers and support managers. In 2006 22nd IEEE International Conference on Software Maintenance, pages 339–342. IEEE, 2006
2006
-
[21]
Determining sample size; how to calculate survey sample size
Hamed Taherdoost. Determining sample size; how to calculate survey sample size. International Journal of Economics and Management Systems, 2, 2017. 11 A PREPRINT - SEPTEMBER 9, 2025
2017
-
[22]
On the way to SBOMs: Investigating design issues and solutions in practice
Tingting Bi, Boming Xia, Zhenchang Xing, Qinghua Lu, and Liming Zhu. On the way to SBOMs: Investigating design issues and solutions in practice. ACM Transactions on Software Engineering and Methodology, page 0, 2024
2024
-
[23]
Memory error detection in security testing
Nasif Imtiaz and Laurie Williams. Memory error detection in security testing. arXiv, 2021
2021
-
[24]
Open source solutions for vulnerability assessment: A comparative analysis
Dinis Barroqueiro Cruz, Jo ˜ao Rafael Almeida, and Jos ´e Lu´ıs Oliveira. Open source solutions for vulnerability assessment: A comparative analysis. IEEE Access, 11:100234–100255, 2023
2023
-
[25]
Lidetector: License incompatibility detection for open source software
Sihan Xu, Ya Gao, Lingling Fan, Zheli Liu, Yang Liu, and Hua Ji. Lidetector: License incompatibility detection for open source software. ACM Transactions on Software Engineering and Methodology, 32(1):1–28, 2023. 12
2023
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.