{"id":"83d800c1-fbd1-42e6-94b4-94a35b910470","arxiv_id":"2509.01255","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Only 0.56% of popular GitHub repositories contain policy-driven SBOMs, and the dependencies they list include 2,202 unique known vulnerabilities with 22% of SBOMs lacking license information.","lead":"The paper mined GitHub for software bill of materials (SBOM) files and found only 0.56% of popular repositories keep them as a policy tool. It offers a dataset of 620 such files and an analysis of the vulnerabilities and licenses they record.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Filtering pipeline's 100-file audit checks precision but not recall; over-broad regexes may exclude true policy-driven SBOMs, so the 0.56% prevalence is not yet robust.","rationale":"The reader's conditional verdict is appropriate. The paper is a well-structured empirical study with an auditable pipeline, a replication package, and a clear taxonomy; those are genuine strengths. However, the central claim—only 0.56% of popular GitHub repositories contain policy-driven SBOM—depends on the filtering pipeline's ability to separate policy-driven SBOMs from test data, third-party files, and self-license files. The manual audit of 100 retained files validates precision only. It does not estimate recall, because excluded files are never re-examined. Several exclusion regexes are broad enough to remove plausible policy-driven SBOMs, e.g., filenames containing 'dependencies', 'lib', 'package', or 'test'. Even a small false-negative rate among the 26,823-repo denominator would materially change 0.56%. The denominator also includes 198 SourceGraph-only repositories added for matching SBOM patterns, which makes the point estimate a mix of a popularity sample and an SBOM-enriched set; this is not a fatal flaw but further weakens the claim's external precision. A targeted recall audit would settle the matter. Because the paper already discloses several related limitations and the reader's CONDITIONAL verdict reflects that uncertainty, my read does not change the verdict.","tokens_in":10532,"tokens_out":6482,"duration_ms":80569,"concrete_test":"Using the replication package, re-run the initial SPDX/CycloneDX search over the 26,823-repo set without applying the 29 exclusion regexes. Randomly sample 200 files that the pipeline would exclude and 100 repositories from the 26,625 cohort that produced no match, then have two independent annotators classify them under the paper's taxonomy (policy-driven vs. test/3rd-party/self-license/research). Count policy-driven cases among excluded files and no-match repos, and recompute prevalence as corrected_policy_repos/26,823. If the corrected prevalence remains near 0.56%, the filtering concern is settled; if it exceeds roughly 1%, the central claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §3.2, the 620 retained SBOMs are produced by applying 29 exclusion regexes to filenames (Test Data, 3rd Party) and content (Self-License). The manual check of 100 of 620 retained files can only upper-bound false positives in the retained set; it says nothing about false negatives, because files removed by the regexes are not inspected. Several patterns are over-broad for a policy-driven classification: 'dependenc*', 'lib*', 'package*', 'contrib*', and 'test' can match legitimate project-owned SBOMs, e.g., dependencies.spdx.json, lib/sbom.json, or test/sbom.json. Since the headline 0.56% is 152 policy-SBOM repositories out of 26,823, even a modest false-negative rate among SBOM-producing repositories changes the reported prevalence materially. The 100-file audit cannot bound that rate. Separately, the denominator is not a pure 'popular repositories' cohort: 26,823 includes 198 SourceGraph-only repos added because they matched SBOM regexes, so the prevalence estimate mixes a popularity sample with an SBOM-enriched convenience set. The paper discloses some tooling and indexing limits, but not this recall gap; a recall audit is needed before the central claim is accepted at face value.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a mining study of GitHub repositories to identify \"policy-driven\" SBOM files, defined as SBOMs created for security or compliance purposes rather than as test fixtures, third-party artifacts, or research-generated files. The authors retrieve 26,823 repositories (top-starred GitHub repos per language plus 198 SourceGraph-only matches), apply 29 exclusion regexes plus manual checks, retain 620 SBOM files from 152 repositories (0.56%), and analyze them with sbomqs and osv-scanner. They report 181,283 dependency entries (25,430 unique), an average quality score of 7.07/10, 19,225 CVEs mapping to 2,202 unique vulnerabilities, and 22.41% of SBOMs without license information. The paper contributes a taxonomy, a dataset, and a replication package.","tokens_in":10824,"tokens_out":7630,"duration_ms":82994,"significance":"The value of this study lies in shifting SBOM research from synthetic artifacts to publicly available, policy-oriented SBOMs and in quantifying the current state of practice against upcoming regulatory mandates. The paper is transparent: it provides the dataset, the regexes, tool versions, and it acknowledges several tooling and search limitations. If the prevalence and vulnerability numbers are taken at face value, they are useful baseline evidence for the SBOM research community. However, the headline prevalence depends on two methodological choices that are not yet fully validated: the recall of the exclusion regexes and the composition of the denominator. These affect the central claim and need to be addressed in revision.","major_comments":[{"comment":"The filtering pipeline is validated only for precision, not recall. The manual audit of 100 retained SBOMs can only bound the false-positive rate in the retained set; it says nothing about files removed by the 29 regexes. Several patterns in Table 1 are over-broad for a policy-driven classification (e.g., 'dependenc*', 'lib*', 'package*', 'test'), so a legitimate file such as dependencies.spdx.json or test/sbom.json could be excluded. Because the headline result is 152 policy-SBOM repositories, even a small number of such false negatives changes the reported 0.56% materially. The paper needs a recall audit on a sample of excluded files, or a sensitivity analysis with the regexes relaxed, and must not state that manual inspection showed 'no false positives' (with zero failures in 100, the upper 95% confidence bound is about 3%).","section":"Section 3.2 and Table 1"},{"comment":"The denominator is not a pure popular-repository cohort. The 26,823 repositories include 198 SourceGraph-only repositories that were added because the SBOM search regexes matched their contents. These repositories are therefore selected on the outcome (presence of SBOM-like files), which biases the prevalence estimate for 'popular open-source projects'. Please report the prevalence computed on the 26,625 top-starred repositories alone, and separately report how many of the 152 policy-SBOM repositories come from the SourceGraph-only subset. Without this, RQ1's answer as stated is not fully supported.","section":"Section 3.1 and Section 4.1"},{"comment":"The tooling failures are acknowledged but not reflected in the result tables. The osv-scanner failed for 22% of SBOMs, and sbomqs could not parse 66% of SPDX tag-value files; yet Section 4.3 reports aggregate totals such as 19,225 CVEs and 2,202 unique vulnerabilities without stating the effective number of successfully scanned SBOMs or the distribution of failures by format/version. This makes the vulnerability and licensing statistics incomplete in an unquantified way. Please report the effective N for each analysis, test whether the failures are correlated with format or generation tool, and characterize the resulting direction of bias (e.g., whether the totals are underestimates).","section":"Section 5.1 and Section 4.3"}],"minor_comments":[{"comment":"The star threshold is lowered from 100 to 35 for some languages. Please report the per-language thresholds or include them in the replication package, since the denominator's composition depends on them.","section":"Section 3.1"},{"comment":"Typographical issues: 'OW ASP SCVS' should be 'OWASP SCVS'; Section 4.1 and Table 4 contain 'Y AML' for YAML; Table 3 contains 'V ersion'; the introduction contains 'polices' for 'policies'; Section 5.2 contains 'When investing RQ3', likely meaning 'In answering RQ3'.","section":"Section 5.1 and typography"},{"comment":"Table 4 reports quality scores for SPDX tag-value despite the 66% parse failure noted in Section 5.1. Clarify which subset of tag-value files is represented and whether the median/max values are based only on the successfully parsed files.","section":"Section 4.2 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about recall is legitimate and affects the headline prevalence claim. The paper is otherwise methodical and the dataset is a useful contribution; I would be willing to review a revised version that addresses the denominator issue and adds a recall audit or sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick summary: this is a mining study that filters real GitHub SBOMs to isolate 'policy-driven' ones and analyzes their quality, vulnerabilities, and licenses. The main contribution is a new dataset of 620 files from 152 repos, plus the observation that only 0.56% of the sampled repos contain such SBOMs. That number is likely the paper's headline, and it's the softest part.\n\nWhat's genuinely new: prior datasets (Torres-Arias, Nocera, Soeiro) either had smaller samples, didn't focus on policy-driven SBOMs, or didn't pair vulnerabilities with licensing. The taxonomy in Figure 1 is simple but useful. The methodology is described in enough detail to audit, and the replication package is available. The tooling failures (66% of SPDX tag-value, 22% of scans) are disclosed, which is honest.\n\nThe soft spots are real but not fatal. First, the filtering pipeline uses 29 exclusion regexes plus a manual check of only 100 of 620 retained files. That audit bounds false positives in the retained set, not false negatives from the regexes. Several patterns ('dependenc*', 'lib*', 'package*', 'test') could match legitimate project-owned SBOMs. So the 0.56% prevalence could be an underestimate, and the paper doesn't quantify the possible range. Second, the denominator mixes the GitHub popularity sample (26,625 repos) with 198 SourceGraph-only repos that were added because they matched SBOM patterns. That makes the prevalence figure less clean than the abstract implies. The paper acknowledges the first issue in limitations but doesn't treat it as load-bearing, and it doesn't mention the denominator mixing at all.\n\nNone of this invalidates the dataset or the vulnerability/license statistics. The 2,202 unique vulnerabilities among 25,430 unique dependencies is a solid descriptive result. The 22% no-license figure is useful. If the authors added a recall audit (e.g., manually labeling a sample of excluded SBOMs) and reported prevalence separately for the GitHub and SourceGraph subsets, the paper would be much stronger.\n\nBottom line: this is a useful empirical contribution for SBOM researchers, especially those thinking about EU CRA compliance. It deserves peer review. I'd send it out, but I'd ask the authors to address the recall gap and the denominator issue before it's accepted.","headline":"A useful, honestly-reported SBOM prevalence baseline that needs a recall audit before the 0.56% figure can be taken at face value.","tokens_in":11339,"tokens_out":3057,"would_cite":true,"duration_ms":32482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Policy-driven SBOMs are rare on GitHub—0.56% of popular repositories—and even those list thousands of known vulnerabilities.","keywords":["software bill of materials","SBOM","supply chain security","open-source","GitHub mining","vulnerability management","licensing","empirical study"],"falsifier":"Re-run the repository collection and have three independent reviewers manually classify all 620 kept files and a random sample of the excluded ones; if the misclassification rate exceeds the paper's stated 5% margin, the prevalence figure and the derived vulnerability and license counts would need revision.","tokens_in":10425,"feed_emoji":"📦","tokens_out":3717,"duration_ms":42919,"temperature":0.7,"pith_summary":"This paper tries to establish how often open-source projects create Software Bills of Materials (SBOMs) for real security and compliance reasons, rather than as research artifacts or test fixtures. By mining 26,823 popular GitHub repositories, it finds only 152 contain what it calls policy-driven SBOMs, a 0.56% prevalence. From the 620 policy-driven SBOMs collected, it reports that the declared dependencies include 2,202 unique known vulnerabilities (19,225 CVE instances) and that 22.41% of the SBOMs provide no licensing information. If these numbers hold, they give the first empirical measure of how far the open-source world is from the SBOM expectations of new regulations like the EU Cyber Resilience Act.","feed_headline":"Only 0.56% of popular GitHub repos carry policy-driven SBOMs","feed_subtitle":"A scan of 620 real SBOM files finds 2,202 unpatched vulnerabilities and sparse license data.","key_machinery":"The central mechanism is a taxonomy-based filtering pipeline: SBOMs are first split into In-the-Lab (research-generated) and In-the-Wild, then In-the-Wild is further divided into dependency management, security scanning, licensing, test data, third-party, and self-license categories. A set of 29 exclusion regexes, refined iteratively and validated by a manually checked sample of 100 out of 620 files, isolates the 'policy-driven' subset. This classification is what gives the 0.56% prevalence figure its meaning, so the taxonomy and filter carry the argument's weight.","core_discovery":"The paper establishes that policy-driven SBOMs—SBOMs created to support security risk assessment, supply-chain transparency, or license compliance, as opposed to test data, third-party artifacts, or self-license files—are almost absent from popular open-source projects. Out of 26,823 repositories processed, only 152 (0.56%) contain such an SBOM. Across the 620 policy-driven SBOM files, the paper counts 25,430 unique dependencies that carry 2,202 unique known vulnerabilities reported in 19,225 CVEs, and notes that 22.41% of the files list no licensing information at all. It also reports an average SBOM quality score of 7.07 on a 0–10 scale, with the most common vulnerabilities tied to weak in","pith_inferences":["Because SourceGraph only indexes files smaller than 1MB and only the main branch, the true prevalence of policy-driven SBOMs could be lower (missed large or branched files) or higher (some excluded files may still be policy-driven), so 0.56% is best read as a lower-bound estimate for mainstream repositories.","The concentration of unpatched, high-severity vulnerabilities in declared dependencies implies that SBOMs alone do little without a routine scanning and patching loop; regulations will likely need to pair SBOM production with mandatory update timelines to be effective.","A natural extension is a time-to-patch analysis: correlating when a vulnerability is disclosed with when (or whether) the project updates its SBOM-declared dependency version would turn the static dataset into a measure of supply-chain responsiveness.","The finding that 22.41% of policy-driven SBOMs lack license fields suggests that companies consuming open-source software may need to supplement SBOM data with external license databases, since the SBOM format alone does not guarantee license coverage."],"forward_implications":["Researchers mining SBOMs from public repositories should distinguish policy-driven files from test or research artifacts, or their findings about real-world practice may be skewed.","The published dataset of 620 policy-driven SBOMs can serve as a benchmark for evaluating SBOM quality tools and vulnerability scanners on real artifacts rather than synthetic ones.","Open-source maintainers and software vendors integrating open-source dependencies should expect that SBOM-driven vulnerability and license checks will reveal unpatched known vulnerabilities and missing license metadata at scale.","The low prevalence suggests that upcoming regulatory mandates will require substantial tooling and process changes before most projects can produce compliant SBOMs.","The 22.41% of policy-driven SBOMs without license information points to a gap that license-compliance automation will need to handle explicitly."],"supporting_citations":[{"why":"Provides the In-the-Lab vs. In-the-Wild taxonomy and the quality-assessment research agenda the paper builds on.","marker":"[10]"},{"why":"Landscape study of SBOM tools that justifies the choice of sbomqs and the interpretation of quality scores.","marker":"[11]"},{"why":"Prior work assessing vulnerabilities in SBOM-declared dependencies; the paper extends it from synthetic to policy-driven datasets.","marker":"[12]"},{"why":"Mining study of SBOM adoption on GitHub that the paper compares against for method and scale.","marker":"[18]"},{"why":"Large-scale wild-SBOM dataset from Software Heritage that provides an alternative collection context and quality-measurement baseline.","marker":"[19]"},{"why":"Sample-size formula used to justify the manual check of 100 out of 620 files for filtering accuracy.","marker":"[21]"}],"fun_headline_variants":["Only 1 in 178 GitHub repos has a policy-driven SBOM","620 SBOM files carry 2,202 unique vulnerabilities","GitHub SBOM adoption under 1%, licenses often missing","Policy-driven SBOMs: rare, vulnerable, and incomplete","2,202 vulnerabilities found in scarce GitHub SBOMs"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The filtering step that decides which SBOM files count as policy-driven—29 exclusion regexes plus a manual check of only 100 of the 620 files—is accurate enough that the 0.56% prevalence and the 620-file dataset are not distorted by misclassification.","fun_headline_variants_meta":{"raw":{"variants":["Only 1 in 178 GitHub repos has a policy-driven SBOM","620 SBOM files carry 2,202 unique vulnerabilities","GitHub SBOM adoption under 1%, licenses often missing","Policy-driven SBOMs: rare, vulnerable, and incomplete","2,202 vulnerabilities found in scarce GitHub SBOMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1088,"prompt_tokens":757,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":245}},"tokens_in":501,"tokens_out":331,"duration_ms":4214,"temperature":1.0,"reasoning_tokens":245,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:41:28.490112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the repository collection and have three independent reviewers manually classify all 620 kept files and a random sample of the excluded ones; if the misclassification rate exceeds the paper's stated 5% margin, the prevalence figure and the derived vulnerability and license counts would need revision.","supporting_citations":[{"cited_title":"A viewpoint on knowing software: Bill of materials quality when you see it","cited_arxiv_id":null,"evidence_quote":"Provides the In-the-Lab vs. In-the-Wild taxonomy and the quality-assessment research agenda the paper builds on."},{"cited_title":"Assessing security risks of software supply chains using software bill of materials","cited_arxiv_id":null,"evidence_quote":"Prior work assessing vulnerabilities in SBOM-declared dependencies; the paper extends it from synthetic to policy-driven datasets."},{"cited_title":"Software bill of materials adoption: a mining study from github","cited_arxiv_id":null,"evidence_quote":"Mining study of SBOM adoption on GitHub that the paper compares against for method and scale."},{"cited_title":"Wild sboms: a large-scale dataset of software bills of materials from public code, 2025","cited_arxiv_id":null,"evidence_quote":"Large-scale wild-SBOM dataset from Software Heritage that provides an alternative collection context and quality-measurement baseline."},{"cited_title":"Determining sample size; how to calculate survey sample size","cited_arxiv_id":null,"evidence_quote":"Sample-size formula used to justify the manual check of 100 out of 620 files for filtering accuracy."}],"review_version":1}