{"id":"12e99eaa-031d-4c1d-9d1e-e8e2c853f0ef","arxiv_id":"2504.20763","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The LLM open-source supply chain is dominated by a few hub packages, and known vulnerabilities can reach hundreds of downstream packages within three dependency layers, though this impact measure counts all dependents rather than confirmed exploitable paths.","lead":"This study maps the open-source package ecosystem that supports large language models, covering 15,725 packages from PyPI and NPM and 180 known security vulnerabilities. It offers a structured quantitative picture of the LLM tool supply chain, but the dataset is not yet public and several headline numbers conflict.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph-size inconsistencies (13,486/15,725/61,965 nodes) mean the structural and propagation statistics may come from different graphs; the paper must state which graph feeds each table and release the construction pipeline.","rationale":"I read the paper as an empirical mapping of the LLMSC, and the central claim requires that the structural and propagation statistics be computed on a consistent, well-defined graph. The manuscript contains at least three different graph sizes and does not explicitly state which graph is used for each research question. This is a correctness and reproducibility risk rather than a novelty dispute. The authors' Section VI caveat about vulnerability reachability is honest and real, but it does not address whether the dependency edges themselves were resolved and counted consistently. I therefore keep the reader's CONDITIONAL verdict: the concern can be resolved by releasing the dataset and reconciling the graph definitions, but until then the headline numbers are not independently checkable.","tokens_in":17887,"tokens_out":7746,"duration_ms":75717,"concrete_test":"Publish the exact node/edge lists and construction script, then run the pipeline end-to-end and recompute Tables III and VII from a single declared graph. Specifically, verify that the 15,725-package graph yields 404 dependency trees with average size 37.34, maximum 3,094, and ten-largest share 77.66%, and check whether the abstract's 13,486-node/28,704-edge graph corresponds to any pipeline stage. If the numbers do not reproduce, the central topology and propagation claims should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing concern: the RQ1/RQ3 statistics are not tied to a single declared graph or version-resolution rule. The submission header abstract gives 13,486 nodes and 28,704 edges; the full-text abstract and Section II-A give 15,725 nodes and 10,402 edges; Section II-B/Table II then reports a V1 graph of 61,965 nodes and 26,736 edges and a V2 graph with 15,725 package nodes, without stating which one feeds Sections III and V. Section III-B's 404 dependency trees with average size 37.34 imply roughly 15,000 nodes, so the structure results appear to use V2, yet the vulnerability propagation numbers in Table VII (e.g., layer-2 maximum 1,043) depend on version-specific nodes and could differ materially in V1. Because the ten-largest-tree share of 77.66% and the 142.1/237.8 propagation averages are headline claims, the absence of an explicit mapping from graph definition to result table makes the central claims unreproducible. This is distinct from the acknowledged reachability limitation in Section VI; it is a prior question of which graph was measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the first empirical study of the large language model supply chain (LLMSC), defined as the open-source packages, libraries, and tools used to develop, deploy, and maintain LLMs. The authors collect metadata and READMEs for 44,303 PyPI and NPM packages, filter them with a keyword search plus a two-stage heuristic classification using DeepSeek-V3, and obtain 15,725 LLM-related packages across 14 functional domains. They then construct a directed dependency graph, attach 180 curated vulnerability records from huntr, GitHub Advisory, and MITRE, and examine three research questions: structural topology, domain distribution and evolution, and vulnerability propagation. The main findings are that the graph is locally dense but globally sparse, with 79.7% of dependency trees containing fewer than 5 nodes, that ten large trees account for 77.66% of nodes, and that critical vulnerabilities propagate to an average of 142.1 nodes at layer 2 and 237.8 at layer 3 of dependency trees. The paper also analyzes domain transitions and identifies hub packages such as transformers as high-risk bridges.","tokens_in":18143,"tokens_out":2968,"duration_ms":30920,"significance":"If the results are reproducible, this is a useful and timely empirical contribution to software supply chain security. The paper addresses a genuinely under-explored ecosystem, provides quantitative structural and vulnerability-propagation claims, and connects them to concrete security risks in widely used packages such as transformers and llama-index. The described pipeline, including the keyword-based collection, the version-aware graph construction, and the multi-source vulnerability curation, is a reasonable methodology for a first characterization study. The paper also makes appropriate connections to prior work on NPM, Maven, PyPI, and deep-learning supply chains. However, the absence of released artifacts and the unresolved inconsistencies in graph sizes across sections currently prevent independent verification of the headline numbers.","major_comments":[{"comment":"","section":"Abstract, §II-A, §II-B, Table II"},{"comment":"","section":"§V-B, Table VII, §VI"},{"comment":"","section":"§II-A, §II-B, Contributions"}],"minor_comments":[{"comment":"","section":"Abstract vs. Full Text"},{"comment":"","section":"Table IV"},{"comment":"","section":"Table VII"},{"comment":"","section":"Figure 7"},{"comment":"","section":"§III-B"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, this is a reasonable first empirical map of the LLM open-source ecosystem: the domain split (44% plugins/external tools) and the hub structure (openai, transformers, pandas at the top) are concrete, useful facts. Second, the paper is sloppier than it should be about its own graph: three different node/edge counts appear in the abstract and body, and it never says which of the V1/V2 graphs in Table II feeds RQ1 and RQ3. That makes the central numbers—404 trees, 37.34 average size, 142.1/237.8 propagation averages—unverifiable as written.\n\nWhat's actually new: applying standard dependency-tree and centrality analysis to LLM-specific packages is new, even if the method is not. The paper does a decent job of collecting and filtering 15,725 packages from PyPI and NPM, and the domain classification is a reasonable extension of earlier DL supply chain work. I credit them for explicitly stating in Section VI that vulnerable package presence does not guarantee invocation; the propagation numbers are upper bounds, not exploit estimates. The structural findings (locally dense, globally sparse; diamond/triangle dominance) are plausible and could be useful for ecosystem health analysis.\n\nSoft spots. The graph-count inconsistency is the big one. The header abstract says 13,486 nodes and 28,704 edges; the full abstract says 15,725 and 10,402; Section II-B reports 61,965 and 26,736 after dependency resolution; Table II gives V1 and V2 but no mapping. The RQ1 tree size and RQ3 propagation tables appear to be V2 (15,725 nodes), and the max out-degree of 1,891 matches Figure 5b, but the paper never says so. A reader cannot reproduce or even reconcile these numbers. Second, the 'first empirical study' claim sits badly next to refs [11] and [37], which already examine LLM supply chain composition and risks; the related work section doesn't clearly differentiate this paper's contribution. Third, the dataset is not released (only promised upon acceptance), so the measurements are not independently checkable. These are not fatal to the idea, but they are what a referee needs to fix.\n\nWho should read it: anyone working on LLM ecosystem security or software supply chain measurement. It's a good starting point, not a definitive study. I'd send it to peer review, but the authors should be told to expect major revision: reconcile the graph version, state which version feeds each table, release the data and code, and soften both the 'first' and 'cascading risk' claims. The underlying measurements are worth checking, not the inconsistencies.","headline":"Useful but sloppy first map of the LLM supply chain; graph-size inconsistencies and the overclaimed 'first' status need fixing before the numbers can be trusted.","tokens_in":18628,"tokens_out":5827,"would_cite":true,"duration_ms":50663,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The open-source LLM supply chain is a few giant hubs around many tiny trees, and its vulnerabilities peak at the third dependency layer.","keywords":["large language model supply chain","software supply chain security","dependency graph analysis","vulnerability propagation","ecosystem topology","PyPI","NPM","empirical software engineering"],"falsifier":"Choose a sample of the 180 reported vulnerabilities, build call graphs or trace execution to see whether each downstream package actually reaches the vulnerable function, and compare the reachable counts with the reported layer averages; if the reachable counts are far smaller, the propagation numbers are best read as structural exposure rather than realized risk.","tokens_in":17732,"feed_emoji":"🔗","tokens_out":9368,"duration_ms":88914,"temperature":0.7,"pith_summary":"This paper sets out to show that the supply chain of open-source packages used to build and run large language models has a measurable shape: a few very large dependency hubs surrounded by many tiny, mostly isolated trees. It maps 15,725 LLM-related packages from the two major open-source package registries into a directed dependency graph, classifies the packages into 14 functional domains, and overlays 180 known vulnerabilities. The central empirical claim is that the ecosystem is locally dense but globally sparse: 79.7% of dependency trees have fewer than five nodes, while the ten largest trees account for 77.66% of all nodes. It also finds that vulnerabilities spread most in the first three dependency layers, peaking at an average of 237.8 affected downstream nodes at layer three, with hub packages such as transformers concentrating that risk. A sympathetic reader would take the paper as establishing quantitative ground truth for an ecosystem whose structure had previously been largely unknown.","feed_headline":"Ten dependency trees hold 77.66% of LLM packages, study finds","feed_subtitle":"Vulnerabilities spread through those hubs to an average of 237.8 nodes by the third dependency layer.","key_machinery":"The operative object is the LLM supply chain dependency graph, a directed graph whose nodes are package-version pairs plus 180 vulnerability nodes and whose edges are dependency links and affects links. From this graph the paper extracts rooted dependency trees and classifies their internal wiring into named substructures, namely chains, triangles, diamonds, stars, and cycles, then measures vulnerability spread by counting how many downstream nodes sit one, two, or three layers away from a vulnerable root. It also uses composite centrality, defined as betweenness centrality divided by degree centrality, to flag bridge nodes that connect otherwise separate clusters. The layer counts are the mechanism that turns the static graph into a risk statement: propagation is concentrated in the first three transmissions.","core_discovery":"The paper's central discovery is quantitative: the LLM supply chain is not a flat crowd of packages but a hierarchy in which 404 dependency trees range from two nodes to 3,094 nodes, with 79.7% of trees below five nodes and the ten largest trees containing 77.66% of all nodes. Inside these trees the dominant substructures are diamonds and triangles, indicating heavy reuse of shared intermediate libraries, and a small set of high-out-degree hubs carries most of the downstream load. On the security side, the paper reports that the 180 curated vulnerabilities propagate from an average of 8.3 directly affected package nodes at layer 1 to 142.1 at layer 2 and 237.8 at layer 3 before declining in deeper layers. A single example, CVE-2023-6730 in transformers, reaches 1,336 downstream nodes. The paper also finds that Plugins/External Tools packages make up 44.26% of the ecosystem, and that most workflows pass through LLM inference and external-tool packages.","pith_inferences":["Editorial inference: the locally dense, globally sparse shape may be a general signature of fast-growing developer ecosystems, and re-running this construction on model hubs, agent frameworks, or vector-database ecosystems would test that.","Editorial inference: the layer-3 peak suggests a concrete patching heuristic, namely quarantine or update packages within three hops of a vulnerable core, but the paper does not test whether that policy reduces realized exploitation.","Editorial inference: because affected-node counts are computed from dependency presence rather than code reachability, re-analyzing the same vulnerabilities with call-graph or runtime tracing would probably yield lower numbers and would convert these upper bounds into risk estimates."],"forward_implications":["Securing the ten largest dependency trees protects most of the ecosystem, because those trees contain 77.66% of all nodes.","Patching effort should target the first three dependency layers, where the average affected-node count rises from 8.3 to 237.8 before falling off.","High-out-degree hubs such as transformers are single points of failure; a single vulnerability in such a package can touch more than 1,300 downstream packages.","The 14-domain classification shows where risk concentrates, with Plugins/External Tools dominating at 44.26% and driving most domain transitions.","The published dataset and graph construction pipeline allow the measurements to be replicated and tracked as the ecosystem grows."],"supporting_citations":[{"why":"Supplies the baseline deep-learning supply chain structure that the LLMSC findings are contrasted against.","marker":"[21]"},{"why":"Provides the domain-cluster and dependency classification approach adapted to the LLMSC dataset.","marker":"[22]"},{"why":"Supplies the dependency-tree vulnerability propagation method and layer-based impact analysis.","marker":"[26]"},{"why":"Frames upstream-to-downstream vulnerability propagation and contributes the reachability caveat used in the threats-to-validity discussion.","marker":"[27]"},{"why":"Source of PyPI package metadata and README content used to build the dataset.","marker":"[31]"},{"why":"Source of NPM package metadata and dependency declarations used to build the dataset.","marker":"[32]"},{"why":"The classifier used to filter packages as LLM-related and to assign them to functional domains.","marker":"[33]"},{"why":"Primary vulnerability data source, contributing 148 of the 180 curated vulnerabilities.","marker":"[36]"},{"why":"Supplies the LLM lifecycle framework used to define the 14 functional domains.","marker":"[37]"},{"why":"Motivates the layer-based propagation analysis and the granularity limitation on vulnerability reachability.","marker":"[39]"}],"fun_headline_variants":["Vulnerabilities in LLM packages cascade to 238 nodes by third layer","One transformers vulnerability reaches 1,336 downstream packages","72% of LLM dependency trees are smaller than 5 nodes","Study maps LLM software supply chain: 13,486 packages, 180 vulnerabilities","LLM supply chain is top-heavy: ten trees hold 77.66% of nodes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Vulnerability propagation is measured by counting every downstream package whose dependency tree contains a vulnerable package, without checking whether the vulnerable code is actually imported and invoked; the paper itself notes this, so the reported layer averages are upper bounds on reachability rather than confirmed exploit counts.","fun_headline_variants_meta":{"raw":{"variants":["Vulnerabilities in LLM packages cascade to 238 nodes by third layer","One transformers vulnerability reaches 1,336 downstream packages","72% of LLM dependency trees are smaller than 5 nodes","Study maps LLM software supply chain: 13,486 packages, 180 vulnerabilities","LLM supply chain is top-heavy: ten trees hold 77.66% of nodes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000973,"raw_usage":{"total_tokens":4213,"prompt_tokens":1099,"completion_tokens":3114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":3015}},"tokens_in":715,"tokens_out":3114,"duration_ms":22681,"temperature":1.0,"reasoning_tokens":3015,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:20:39.860523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Choose a sample of the 180 reported vulnerabilities, build call graphs or trace execution to see whether each downstream package actually reaches the vulnerable function, and compare the reachable counts with the reported layer averages; if the reachable counts are far smaller, the propagation numbers are best read as structural exposure rather than realized risk.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of PyPI package metadata and README content used to build the dataset."},{"cited_title":"Registry,","cited_arxiv_id":null,"evidence_quote":"Source of NPM package metadata and dependency declarations used to build the dataset."},{"cited_title":"Deepseek-v3,","cited_arxiv_id":null,"evidence_quote":"The classifier used to filter packages as LLM-related and to assign them to functional domains."},{"cited_title":"The world’s first bug bounty platform for ai/ml,","cited_arxiv_id":null,"evidence_quote":"Primary vulnerability data source, contributing 148 of the 180 curated vulnerabilities."},{"cited_title":"On the effect of transitivity and granularity on vulnerability propagation in the maven ecosystem,","cited_arxiv_id":null,"evidence_quote":"Motivates the layer-based propagation analysis and the granularity limitation on vulnerability reachability."}],"review_version":1}