Pith. sign in

REVIEW 3 major objections 3 minor 25 references

Policy-driven Software Bill of Materials on GitHub: An Empirical Study

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read Policy-driven SBOMs are rare on GitHub—0.56% of popular repositories—and even those list thousands of known vulnerabilities.

desk verdict A useful, honestly-reported SBOM prevalence baseline that needs a recall audit before the 0.56% figure can be taken at face value. read the letter →

arxiv 2509.01255 v1 pith:WEBJBG2K submitted 2025-09-01 cs.SE

classification cs.SE
keywords softwarebillofmaterialsSBOMsupplychainsecurityopen-sourceGitHubminingvulnerabilitymanagementlicensingempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish how often open-source projects create Software Bills of Materials (SBOMs) for real security and compliance reasons, rather than as research artifacts or test fixtures. By mining 26,823 popular GitHub repositories, it finds only 152 contain what it calls policy-driven SBOMs, a 0.56% prevalence. From the 620 policy-driven SBOMs collected, it reports that the declared dependencies include 2,202 unique known vulnerabilities (19,225 CVE instances) and that 22.41% of the SBOMs provide no licensing information. If these numbers hold, they give the first empirical measure of how far the open-source world is from the SBOM expectations of new regulations like the EU Cyber Resilience Act.

What carries the argument

The central mechanism is a taxonomy-based filtering pipeline: SBOMs are first split into In-the-Lab (research-generated) and In-the-Wild, then In-the-Wild is further divided into dependency management, security scanning, licensing, test data, third-party, and self-license categories. A set of 29 exclusion regexes, refined iteratively and validated by a manually checked sample of 100 out of 620 files, isolates the 'policy-driven' subset. This classification is what gives the 0.56% prevalence figure its meaning, so the taxonomy and filter carry the argument's weight.

What would settle it

Re-run the repository collection and have three independent reviewers manually classify all 620 kept files and a random sample of the excluded ones; if the misclassification rate exceeds the paper's stated 5% margin, the prevalence figure and the derived vulnerability and license counts would need revision.

Watch

Extended reading notes

Core claim

The paper establishes that policy-driven SBOMs—SBOMs created to support security risk assessment, supply-chain transparency, or license compliance, as opposed to test data, third-party artifacts, or self-license files—are almost absent from popular open-source projects. Out of 26,823 repositories processed, only 152 (0.56%) contain such an SBOM. Across the 620 policy-driven SBOM files, the paper counts 25,430 unique dependencies that carry 2,202 unique known vulnerabilities reported in 19,225 CVEs, and notes that 22.41% of the files list no licensing information at all. It also reports an average SBOM quality score of 7.07 on a 0–10 scale, with the most common vulnerabilities tied to weak in

Load-bearing premise

The filtering step that decides which SBOM files count as policy-driven—29 exclusion regexes plus a manual check of only 100 of the 620 files—is accurate enough that the 0.56% prevalence and the 620-file dataset are not distorted by misclassification.

Editorial extensions

If this is right

  • Researchers mining SBOMs from public repositories should distinguish policy-driven files from test or research artifacts, or their findings about real-world practice may be skewed.
  • The published dataset of 620 policy-driven SBOMs can serve as a benchmark for evaluating SBOM quality tools and vulnerability scanners on real artifacts rather than synthetic ones.
  • Open-source maintainers and software vendors integrating open-source dependencies should expect that SBOM-driven vulnerability and license checks will reveal unpatched known vulnerabilities and missing license metadata at scale.
  • The low prevalence suggests that upcoming regulatory mandates will require substantial tooling and process changes before most projects can produce compliant SBOMs.
  • The 22.41% of policy-driven SBOMs without license information points to a gap that license-compliance automation will need to handle explicitly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because SourceGraph only indexes files smaller than 1MB and only the main branch, the true prevalence of policy-driven SBOMs could be lower (missed large or branched files) or higher (some excluded files may still be policy-driven), so 0.56% is best read as a lower-bound estimate for mainstream repositories.
  • The concentration of unpatched, high-severity vulnerabilities in declared dependencies implies that SBOMs alone do little without a routine scanning and patching loop; regulations will likely need to pair SBOM production with mandatory update timelines to be effective.
  • A natural extension is a time-to-patch analysis: correlating when a vulnerability is disclosed with when (or whether) the project updates its SBOM-declared dependency version would turn the static dataset into a measure of supply-chain responsiveness.
  • The finding that 22.41% of policy-driven SBOMs lack license fields suggests that companies consuming open-source software may need to supplement SBOM data with external license databases, since the SBOM format alone does not guarantee license coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper reports a mining study of GitHub repositories to identify "policy-driven" SBOM files, defined as SBOMs created for security or compliance purposes rather than as test fixtures, third-party artifacts, or research-generated files. The authors retrieve 26,823 repositories (top-starred GitHub repos per language plus 198 SourceGraph-only matches), apply 29 exclusion regexes plus manual checks, retain 620 SBOM files from 152 repositories (0.56%), and analyze them with sbomqs and osv-scanner. They report 181,283 dependency entries (25,430 unique), an average quality score of 7.07/10, 19,225 CVEs mapping to 2,202 unique vulnerabilities, and 22.41% of SBOMs without license information. The paper contributes a taxonomy, a dataset, and a replication package.

Significance. The value of this study lies in shifting SBOM research from synthetic artifacts to publicly available, policy-oriented SBOMs and in quantifying the current state of practice against upcoming regulatory mandates. The paper is transparent: it provides the dataset, the regexes, tool versions, and it acknowledges several tooling and search limitations. If the prevalence and vulnerability numbers are taken at face value, they are useful baseline evidence for the SBOM research community. However, the headline prevalence depends on two methodological choices that are not yet fully validated: the recall of the exclusion regexes and the composition of the denominator. These affect the central claim and need to be addressed in revision.

major comments (3)
  1. [Section 3.2 and Table 1] The filtering pipeline is validated only for precision, not recall. The manual audit of 100 retained SBOMs can only bound the false-positive rate in the retained set; it says nothing about files removed by the 29 regexes. Several patterns in Table 1 are over-broad for a policy-driven classification (e.g., 'dependenc*', 'lib*', 'package*', 'test'), so a legitimate file such as dependencies.spdx.json or test/sbom.json could be excluded. Because the headline result is 152 policy-SBOM repositories, even a small number of such false negatives changes the reported 0.56% materially. The paper needs a recall audit on a sample of excluded files, or a sensitivity analysis with the regexes relaxed, and must not state that manual inspection showed 'no false positives' (with zero failures in 100, the upper 95% confidence bound is about 3%).
  2. [Section 3.1 and Section 4.1] The denominator is not a pure popular-repository cohort. The 26,823 repositories include 198 SourceGraph-only repositories that were added because the SBOM search regexes matched their contents. These repositories are therefore selected on the outcome (presence of SBOM-like files), which biases the prevalence estimate for 'popular open-source projects'. Please report the prevalence computed on the 26,625 top-starred repositories alone, and separately report how many of the 152 policy-SBOM repositories come from the SourceGraph-only subset. Without this, RQ1's answer as stated is not fully supported.
  3. [Section 5.1 and Section 4.3] The tooling failures are acknowledged but not reflected in the result tables. The osv-scanner failed for 22% of SBOMs, and sbomqs could not parse 66% of SPDX tag-value files; yet Section 4.3 reports aggregate totals such as 19,225 CVEs and 2,202 unique vulnerabilities without stating the effective number of successfully scanned SBOMs or the distribution of failures by format/version. This makes the vulnerability and licensing statistics incomplete in an unquantified way. Please report the effective N for each analysis, test whether the failures are correlated with format or generation tool, and characterize the resulting direction of bias (e.g., whether the totals are underestimates).
minor comments (3)
  1. [Section 3.1] The star threshold is lowered from 100 to 35 for some languages. Please report the per-language thresholds or include them in the replication package, since the denominator's composition depends on them.
  2. [Section 5.1 and typography] Typographical issues: 'OW ASP SCVS' should be 'OWASP SCVS'; Section 4.1 and Table 4 contain 'Y AML' for YAML; Table 3 contains 'V ersion'; the introduction contains 'polices' for 'policies'; Section 5.2 contains 'When investing RQ3', likely meaning 'In answering RQ3'.
  3. [Section 4.2 and Table 4] Table 4 reports quality scores for SPDX tag-value despite the 66% parse failure noted in Section 5.1. Clarify which subset of tag-value files is represented and whether the median/max values are based only on the successfully parsed files.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical measurement pipeline with no fitted parameter reused as a prediction and no load-bearing self-citation.

full rationale

The derivation chain is: define a taxonomy of SBOM uses (policy-driven vs. in-the-lab/test/3rd-party/self-license); mine GitHub/SourceGraph; apply exclusion regexes; manually audit 100/620 files; then count prevalence, qualities via sbomqs, vulnerabilities via osv-scanner. The central result (0.56% of 26,823 repositories contain policy-driven SBOM) is a measurement over the authors' operationalized definition, not a quantity derived from that definition by algebra or by fitting. No equation equates the output to an input: the quality scores come from an external tool, the vulnerability counts from osv-scanner's advisory databases, and the license data from the SBOM contents. The paper contains no self-citation that is load-bearing; previous work (Torres-Arias et al., Nocera et al., Soeiro et al.) is used for context and comparison, not to establish the current paper's claims. The acknowledged limitations (regex false negatives/positives, SourceGraph indexing, tool parse failures) concern measurement validity, not circularity. A recall audit would improve robustness, but that is a correctness/threat-to-validity concern, not circularity. Therefore score 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters in the fit-a-model sense; the listed parameters are hand-chosen sampling thresholds that influence the prevalence denominator. The axioms are the coverage and classification assumptions behind the mining pipeline. No new physical or conceptual entities are postulated.

free parameters (2)
  • min_stars_threshold = 100 (lowered to 35 for some languages)
    Hand-chosen threshold for 'popular' repositories in Section 3.1; directly shapes the denominator for the 0.56% prevalence claim.
  • artifact_size_limit = 100MB
    GitHub API limit for downloading release artifacts (Section 3.1); SBOMs in larger archives are missed, affecting recall.
assumptions (5)
  • domain assumption SourceGraph's index (main branch, files under 1MB, repos under 10GB) is a representative window on GitHub SBOMs.
    Limits stated in Sections 3.1 and 5.1; the prevalence estimate is conditional on this coverage.
  • domain assumption SPDX and CycloneDX are the only relevant SBOM formats, and the required attributes (SPDXID, bomFormat) are sufficient to identify SBOM files.
    Detection regexes in Section 3.1 rely on these markers; acknowledged false positives and negatives in Section 5.1.
  • domain assumption Manual inspection of 100 randomly sampled files validates the policy-driven label for all 620 files.
    Section 3.2 uses a 96-file sample size for 95% confidence and 5% margin; applies that to the whole filtered set.
  • domain assumption osv-scanner and its 28 advisory databases provide a complete-enough vulnerability picture of the declared dependencies.
    Section 3.3 and 5.1; the tool failed for 22% of SBOMs, so vulnerability counts are lower bounds.
  • domain assumption Popularity can be operationalized by GitHub star counts with thresholds of 100 or 35.
    Section 3.1 defines the sampling frame; the 0.56% result only applies to this popularity proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Policy-driven Software Bill of Materials on GitHub: An Empirical Study." pith.science (2026). https://pith.science/paper/WEBJBG2K

@misc{pith2026250901255,
  author       = {Pith},
  title        = {Pith review of: Policy-driven Software Bill of Materials on GitHub: An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEBJBG2K}},
  note         = {Machine review of arXiv:2509.01255}
}
read the original abstract

Background. The Software Bill of Materials (SBOM) is a machine-readable list of all the software dependencies included in a software. SBOM emerged as way to assist securing the software supply chain. However, despite mandates from governments to use SBOM, research on this artifact is still in its early stages. Aims. We want to understand the current state of SBOM in open-source projects, focusing specifically on policy-driven SBOMs, i.e., SBOM created to achieve security goals, such as enhancing project transparency and ensuring compliance, rather than being used as fixtures for tools or artificially generated for benchmarking or academic research purposes. Method. We performed a mining software repository study to collect and carefully select SBOM files hosted on GitHub. We analyzed the information reported in policy-driven SBOMs and the vulnerabilities associated with the declared dependencies by means of descriptive statistics. Results. We show that only 0.56% of popular GitHub repositories contain policy-driven SBOM. The declared dependencies contain 2,202 unique vulnerabilities, while 22% of them do not report licensing information. Conclusion. Our findings provide insights for SBOM usage to support security assessment and licensing.

Figures

Figures reproduced from arXiv: 2509.01255 by the authors.

Figure 1
Figure 1. Taxonomy of SBOM types (green considered in our study, red excluded) with examples. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the steps for gathering and analyzing SBOM files. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of dependencies declared in SBOM. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: SBOM vulnerability and licensing information. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 24 canonical work pages

  1. [1]

    Defense-in-depth security strategy in log4j vulnerability analysis

    Sylvia Feng and Muharman Lubis. Defense-in-depth security strategy in log4j vulnerability analysis. In 2022 International Conference Advancement in Data Science, E-learning and Information Systems (ICADEIS), pages 01–04. IEEE, 2022. 10 A PREPRINT - SEPTEMBER 9, 2025

  2. [2]

    Navigating the solarwinds supply chain attack.The Procurement Lawyer, 56(2), 2021

    Evan D Wolff, KM Growley, MG Gruden, et al. Navigating the solarwinds supply chain attack.The Procurement Lawyer, 56(2), 2021

  3. [3]

    A systematic analysis of the event- stream incident

    Iosif Arvanitis, Grigoris Ntousakis, Sotiris Ioannidis, and Nikos Vasilakis. A systematic analysis of the event- stream incident. In Proceedings of the 15th European Workshop on Systems Security, pages 22–28, 2022

  4. [4]

    Log4shell: Redefining the web attack surface

    Douglas Everson, Long Cheng, and Zhenkai Zhang. Log4shell: Redefining the web attack surface. In Workshop on Measurements, Attacks, and Defenses for the Web (MADWeb) 2022, 2022

  5. [5]

    Software bills of materials are required

    Nusrat Zahan, Elizabeth Lin, Mahzabin Tamanna, William Enck, and Laurie Williams. Software bills of materials are required. are we there yet? IEEE Security & Privacy, 21(2):82–88, 2023

  6. [6]

    The state of software bill of materials (sbom) and cybersecurity readiness

    Stephen Hendrick and J Zemlin. The state of software bill of materials (sbom) and cybersecurity readiness. Technical report, The Linux Foundation, 2022

  7. [7]

    Charting the path to sbom adoption: A business stakeholder-centric approach

    Berend Kloeg, Aaron Yi Ding, Sjoerd Pellegrom, and Yury Zhauniarovich. Charting the path to sbom adoption: A business stakeholder-centric approach. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, pages 1770–1783, 2024

  8. [8]

    Boms away! inside the minds of stakeholders: A comprehensive study of bills of materials for software systems

    Trevor Stalnaker, Nathan Wintersgill, Oscar Chaparro, Massimiliano Di Penta, Daniel M German, and Denys Poshyvanyk. Boms away! inside the minds of stakeholders: A comprehensive study of bills of materials for software systems. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , pages 1–13, 2024

Show all 25 references
  1. [9]

    An empirical study on software bill of materials: Where we stand and the road ahead

    Boming Xia, Tingting Bi, Zhenchang Xing, Qinghua Lu, and Liming Zhu. An empirical study on software bill of materials: Where we stand and the road ahead. arXiv, 2023

  2. [10]

    A viewpoint on knowing software: Bill of materials quality when you see it

    Santiago Torres-Arias, Dan Geer, and John Speed Meyers. A viewpoint on knowing software: Bill of materials quality when you see it. IEEE Security & Privacy, 21(6):50–54, 2023

  3. [11]

    A landscape study of open source and proprietary tools for software bill of materials (sbom)

    Mehdi Mirakhorli, Derek Garcia, Schuyler Dillon, Kevin Laporte, Matthew Morrison, Henry Lu, Viktoria Koscinski, and Christopher Enoch. A landscape study of open source and proprietary tools for software bill of materials (sbom). arXiv preprint arXiv:2402.11151, 2024

  4. [12]

    Assessing security risks of software supply chains using software bill of materials

    Eric O’Donoghue, Ann Marie Reinhold, and Clemente Izurieta. Assessing security risks of software supply chains using software bill of materials. In 2nd International Workshop on Mining Software Repositories for Privacy and Security, MSR4P&S,(SANER 2024), Rovaniemi, Finland, 2024

  5. [13]

    Challenges of producing software bill of materials for java

    Musard Balliu, Benoit Baudry, Sofia Bobadilla, Mathias Ekstedt, Martin Monperrus, Javier Ron, Aman Sharma, Gabriel Skoglund, C´esar Soto-Valero, and Martin Wittlinger. Challenges of producing software bill of materials for java. IEEE Security & Privacy, 2023

  6. [14]

    On the correctness of metadata-based sbom generation: A differential analysis approach

    Sheng Yu, Wei Song, Xunchao Hu, and Heng Yin. On the correctness of metadata-based sbom generation: A differential analysis approach. In2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 29–36. IEEE, 2024

  7. [15]

    Sbom generation tools under microscope: A focus on the npm ecosystem

    Md Fazle Rabbi, Arifa Islam Champa, Costain Nachuma, and Minhaz Fahim Zibran. Sbom generation tools under microscope: A focus on the npm ecosystem. In Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, pages 1233–1241, 2024

  8. [16]

    License usage and changes: a large-scale study on github

    Christopher Vendome, Gabriele Bavota, Massimiliano Di Penta, Mario Linares-V ´asquez, Daniel German, and Denys Poshyvanyk. License usage and changes: a large-scale study on github. Empirical Software Engineering, 22:1537–1577, 2017

  9. [17]

    Types of software bill of materials (sbom)

    Kate Stewart and Melissa Rhodes. Types of software bill of materials (sbom). https://web. archive.org/web/20240907164647/https://www.cisa.gov/sites/default/files/2023-04/ sbom-types-document-508c.pdf

  10. [18]

    Software bill of materials adoption: a mining study from github

    Sabato Nocera, Simone Romano, Massimiliano Di Penta, Rita Francese, and Giuseppe Scanniello. Software bill of materials adoption: a mining study from github. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 39–49. IEEE, 2023

  11. [19]

    Wild sboms: a large-scale dataset of software bills of materials from public code, 2025

    Luıs Soeiro, Thomas Robert, and Stefano Zacchiroli. Wild sboms: a large-scale dataset of software bills of materials from public code, 2025

  12. [20]

    Mining software repositories to assist developers and support managers

    Ahmed E Hassan. Mining software repositories to assist developers and support managers. In 2006 22nd IEEE International Conference on Software Maintenance, pages 339–342. IEEE, 2006

  13. [21]

    Determining sample size; how to calculate survey sample size

    Hamed Taherdoost. Determining sample size; how to calculate survey sample size. International Journal of Economics and Management Systems, 2, 2017. 11 A PREPRINT - SEPTEMBER 9, 2025

  14. [22]

    On the way to SBOMs: Investigating design issues and solutions in practice

    Tingting Bi, Boming Xia, Zhenchang Xing, Qinghua Lu, and Liming Zhu. On the way to SBOMs: Investigating design issues and solutions in practice. ACM Transactions on Software Engineering and Methodology, page 0, 2024

  15. [23]

    Memory error detection in security testing

    Nasif Imtiaz and Laurie Williams. Memory error detection in security testing. arXiv, 2021

  16. [24]

    Open source solutions for vulnerability assessment: A comparative analysis

    Dinis Barroqueiro Cruz, Jo ˜ao Rafael Almeida, and Jos ´e Lu´ıs Oliveira. Open source solutions for vulnerability assessment: A comparative analysis. IEEE Access, 11:100234–100255, 2023

  17. [25]

    Lidetector: License incompatibility detection for open source software

    Sihan Xu, Ya Gao, Lingling Fan, Zheli Liu, Yang Liu, and Hua Ji. Lidetector: License incompatibility detection for open source software. ACM Transactions on Software Engineering and Methodology, 32(1):1–28, 2023. 12

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.