{"id":"78747a03-cd0b-4f1f-8d41-1b35b00e900b","arxiv_id":"2412.16922","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-powered web-crawling system automatically builds supply chain maps and appears to cover emerging-economy firms better than Bloomberg SPLC in a semiconductor case study.","lead":"Researchers built an automated system that reads online articles and uses large language models to map which companies supply goods to which other companies. In a semiconductor case study, the system captured supply links involving mainland Chinese and other emerging-economy firms that the commercial Bloomberg dataset largely missed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 77% precision is measured globally, but the headline claim is about China; without country-stratified precision, the PRC network metrics in Table I cannot support 'greater applicability'.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper is not ready for acceptance as-is. My stress-test pass isolates one assumption as load-bearing: global precision is used to validate a country-specific claim. The paper reports only one precision number (77%, n=200) and applies it to the entire graph, including the PRC subgraph that drives Table I and the conclusion. Because the system's crawling is seeded by industry reports and web search, the composition of Chinese edges is likely different from US edges, and the paper acknowledges media bias. A 20-percentage-point precision gap would be enough to explain the PRC density/modularity differences. The original 200-sample almost surely contains few PRC relationships, since the full graph has ~20,000 companies across many countries, so the global estimate is uninformative for the claim. The proposed stratified evaluation is inexpensive because the labeling protocol already exists; it directly tests whether the claimed 'greater applicability' survives false-positive correction. If PRC precision is comparable to global, the claim is strengthened; if not, the network metrics are artifacts. This does not change the verdict: the paper remains CONDITIONAL pending this evidence.","tokens_in":8758,"tokens_out":6841,"duration_ms":67515,"concrete_test":"Re-run the human validation stratified by country/region: sample at least 100 system-extracted supply relationships involving mainland Chinese companies and 100 involving non-Chinese companies (using the same evidence-checking protocol as in Section IV), and compute per-stratum precision with 95% exact binomial CIs. Also report how many PRC relationships were in the original 200. If PRC precision is significantly below the global 77% (e.g., lower bound below 60%), or if removing predicted false positives from the PRC subgraph changes Table I's density/modularity pattern, the 'greater applicability' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—greater applicability for emerging economies—rests on Table I's PRC subgraph comparisons and the assertion that the system's China cluster is cohesive. The only accuracy evidence is a global precision of 77% from 200 randomly sampled relationships (Section IV). No country breakdown is reported. If precision on Chinese/emerging-economy relationships is materially lower than 77%, then the denser, more modular PRC structure in Table I could be an artifact of false positives, not evidence of true coverage. This risk is heightened by the system's own Discussion: online content availability is biased toward highly digitized countries and media-covered firms, so the extra Chinese edges may reflect web visibility rather than supply chain reality. The number of Chinese relationships in the 200-sample is unknown, and the confidence interval around 77% (n=200) is wide, making a regional deficit of 20 points entirely possible. Without stratified precision, the comparative network metrics are uninterpretable as evidence for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a Supply Chain Knowledge Graph Mining System that uses large language models (LLMs) and iterative web crawling to extract supply chain relationships from online content. The system is evaluated on a semiconductor supply chain case study, producing a graph of approximately 6,613 reliable supply relationships after a discriminative-model validation step, and compared with a Bloomberg SPLC benchmark (1,753 companies, 5,749 relationships). The authors report a global precision of 77% on 200 randomly sampled, human-labeled relationships and present graph-theoretic metrics (degree, density, modularity) for the full network and for the US and mainland China subgraphs. They conclude that the system 'provides greater applicability for emerging economies, such as mainland China,' complementing gaps in existing datasets.","tokens_in":8895,"tokens_out":6879,"duration_ms":59959,"significance":"The paper addresses a timely and important problem: supply chain transparency in emerging economies is limited by the poor coverage of commercial relationship datasets. The proposed pipeline—iterative seeding from industry reports, LLM-based joint extraction, synonym disambiguation, and LLM-based discriminative validation—is a reasonable design, and the resulting multi-relationship graph could be a useful resource. However, the current evidence is not yet sufficient to support the central comparative claim. The only accuracy statistic is a global precision estimate on a small sample, and the network comparisons in Table I are confounded by differing data construction processes and by the online-content bias that the authors themselves acknowledge in Section VI. The paper also does not provide code, data, or a detailed annotation protocol. If the authors add a stratified evaluation with confidence intervals and a more rigorous treatment of the Bloomberg comparison, the contribution would be substantially stronger and of interest to both information extraction and supply chain management communities.","major_comments":[{"comment":"The claim that the system reveals a cohesive mainland China supply chain ('average degree is even higher than that of the U.S.', 'companies from Mainland China exhibit tighter internal connections') is not supported by the reported evidence. Precision is given only as a global 77% on 200 sampled relationships, with no country-stratified estimates or confidence intervals. Because Section VI acknowledges that online content availability and media attention are biased toward certain companies, countries, and industries, the higher PRC degree and lower modularity in the system's data could be an artifact of web-visibility bias rather than a genuine difference in supply chain structure. Please provide region- and relationship-type-stratified precision and confidence intervals, or otherwise demonstrate that the PRC network metrics are not driven by extraction bias.","section":"Section V, Table I"},{"comment":"The network metrics are compared across two datasets that are built under fundamentally different constraints: the Bloomberg benchmark is truncated to the top 20 suppliers/customers per company, while the system's graph is seeded from industry reports, iteratively crawled, and then pruned to roughly 5,000 relationships to match Bloomberg's size. The authors themselves state that high modularity 'may result from data incompleteness that disrupts the network.' Consequently, the density and modularity differences between the two datasets—and between the PRC and US subgraphs—cannot be interpreted as differences in the underlying supply chain structure without controlling for truncation, sampling, and graph-size effects. Please add an analysis that separates measurement artifacts from genuine coverage differences, for example by computing metrics on comparable subsets or using degree-preserving random baselines.","section":"Section IV, Eqs. (2)-(3), and Table I"},{"comment":"The overlap statistics (1,020 of 1,753 Bloomberg companies covered; 173 of 5,749 relationships duplicated) are presented as evidence of complementarity, but the entity-resolution matching procedure is not documented, and the two extraction pipelines have different scopes: Bloomberg starts from the top 100 global semiconductor companies by market capitalization, whereas the system's crawl is seeded from industry research reports. Without a careful description of the matching (including how company aliases and relationship directions are aligned) and without recall-oriented coverage statistics in each direction, these numbers cannot establish 'greater applicability' for emerging economies. Please document the matching method and report coverage per country (e.g., the fraction of Bloomberg's PRC companies and relationships that the system finds, and vice versa).","section":"Section V"},{"comment":"The evaluation sample of 200 randomly sampled relationships is the sole accuracy evidence for the pipeline, yet the manuscript does not report a confidence interval for the 77% precision, annotation guidelines, or inter-annotator agreement. With n=200, the approximate 95% confidence interval for 77% is wide (roughly ±6 percentage points or more), and the number of Chinese or emerging-economy relationships in the sample is not stated. Please report the exact binomial confidence interval, the country/relationship-type composition of the 200 samples, and ideally a second-annotator agreement measure.","section":"Section IV"}],"minor_comments":[{"comment":"There are numerous typos and grammatical errors, including 'Mangement' in the affiliations, 'DESGIN' in the Section III heading, 'continent' in Section II (likely 'convenient'), 'Malasia' in Section V, and subject-verb disagreements such as 'the data ... does not show.' A professional language edit is needed.","section":"Throughout"},{"comment":"Reference [25] is described as an unpublished preprint, but no repository or identifier is given; the paper should either provide a verifiable link or remove the comparative claim with the parenthetical 'which we believe should be precision.'","section":"References"},{"comment":"The sentence 'Our work achieves a precision of 0.77 in supply chain relationship extraction' conflates extraction with the post-hoc discriminative validation; please clarify that the precision is for the full pipeline after validation.","section":"Section VI"},{"comment":"The pruning step is described as 'considering the size of our dataset should match that of the Bloomberg benchmark dataset's' but the exact number of final relationships used for Table I and the pruning criterion (e.g., random, confidence-based, or degree-based) are not stated. This information is needed for reproducibility.","section":"Section IV"},{"comment":"The manuscript does not state whether the code, prompts, or extracted graph will be released. For a system paper, a data/code availability statement would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript is an extended abstract-length report that presents a promising system but does not yet provide sufficient evidence for its headline claim. I recommend major revision focused on a stratified precision evaluation and a clearer, more honest comparison with Bloomberg SPLC. The paper's scope fits venues on applied NLP or information systems, but in its current form the central claim about emerging-economy applicability is not established. I also note that the reference to [25] as an 'unpublished' preprint without a public identifier should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here’s my take. The paper builds a working prototype: an iterative web-crawling pipeline that uses LLMs to extract, disambiguate, and verify supply chain relationships, with a semiconductor industry case study and a comparison to Bloomberg SPLC. That is genuinely new relative to earlier LLM-on-Wikipedia work and to news-article deep learning. The authors also do a decent job of owning the limitations—media bias, time-series fuzziness, missing monetary flows—which makes the paper easy to trust even where it is weak.\n\nThe weakness is the headline claim. They say the system has “greater applicability for emerging economies,” and point to a denser China subnet in Table I plus the presence of India and Malaysia companies. But the only precision number is 77% on 200 global random samples. There is no country-stratified accuracy. If precision on Chinese-language or smaller-firm relationships is lower, the cohesive China cluster could be a product of false positives or web visibility bias—something they acknowledge in the discussion. So the comparative network metrics are not interpretable as evidence for the claim yet.\n\nThere are also numeric inconsistencies: 9,226 supply relationships become 6,613 after validation, 4,484 unique, and then “around 5,000” is used for the graph. That needs to be reconciled. The Bloomberg baseline is itself truncated to top-20 suppliers/customers, which may hurt coverage comparisons. And the paper does not release code or data, so the results are not independently checkable.\n\nNone of this is fatal to the core idea. The pipeline is coherent, the related work is surveyed fairly, and the authors are clear that this is an early prototype. But the evidence is not yet strong enough for the emerging-economy claim. A stratified precision evaluation, released data/code, and a baseline comparison on a shared corpus would make the paper convincing.\n\nWho should read it: researchers in LLM-based information extraction, supply chain analytics, and knowledge graph construction. It’s a useful case study in evaluation pitfalls. I’d send it to peer review rather than desk-reject, but the revision needs to address the stratified evaluation and the count inconsistencies.\n\nRecommendation: engage with it, but expect major revision.","headline":"A useful LLM-based supply chain mining prototype whose emerging-economy claim overreaches the evidence.","tokens_in":9431,"tokens_out":3891,"would_cite":true,"duration_ms":35472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM-based web mining system can build supply chain knowledge graphs that cover emerging economies more fully than commercial datasets like Bloomberg SPLC, with a semiconductor case study as evidence.","keywords":["supply chain transparency","large language models","knowledge graph mining","supply chain relationship extraction","emerging economies","semiconductor supply chain","web crawling","network analysis"],"falsifier":"A concrete check would be to take random samples of 200-300 extracted relationships separately for China, India, Brazil, and the US, have them human-labeled, and compute precision by region and by firm size; if precision drops sharply in regions with thinner media coverage, or if the system's China-vs-Bloomberg network differences disappear after matching companies by identifier and time period, the central claim of better emerging-economy coverage would be undermined.","tokens_in":8507,"feed_emoji":"🔗","tokens_out":3170,"duration_ms":39960,"temperature":0.7,"pith_summary":"The paper sets out to show that an automated pipeline combining web crawling with large language models can construct supply chain relationship graphs that mainstream commercial datasets under-serve, especially in emerging economies. It tests this in the semiconductor industry, using mainland China as the focal emerging market and Bloomberg SPLC as the benchmark. If the claim holds, the system offers a low-marginal-cost path to supply chain transparency where regulatory disclosure is weak, filling gaps rather than replacing existing data. The central supporting evidence is a measured 77% precision on a human-labeled sample of extracted relationships, plus network statistics showing Chinese companies form a more cohesive cluster in the new graph than in Bloomberg.","feed_headline":"LLM web crawler finds supply-chain links Bloomberg misses","feed_subtitle":"A semiconductor test shows 77% precision and denser China coverage, filling gaps in emerging-economy data.","key_machinery":"The Supply Chain Knowledge Graph Mining System is the central mechanism: a loop that initializes a Company Library from industry reports, dynamically generates search keywords, crawls retrieved web pages, and uses LLMs to extract knowledge-graph triplets in JSON format with mandatory textual evidence. Synonym disambiguation runs through relation matching, embedding similarity, LLM testimony, and a human-checked synonym list, while a discriminative LLM model re-verifies each extracted relationship and raises precision by 8 percentage points. This machinery carries the argument by producing the graph whose network metrics are then compared against Bloomberg's.","core_discovery":"The central discovery is that an iterative system seeded with industry research reports can walk the web from company to company, using LLMs to jointly extract supply-chain triplets with textual evidence, and thereby assemble a graph of more than 20,000 company nodes and 6,613 reliable supply relationships after a discriminative-model validation step. In the semiconductor case, this graph reached 77% precision on a random sample of 200 human-validated relationships. Compared with the Bloomberg SPLC benchmark, the system's China subnet has a higher average degree and a lower modularity gap relative to the global network, while Bloomberg's China subnet appears sparse and peripheral. The paper interprets this as evidence that the system detects supply chain relationships in emerging economies that Bloomberg misses, and it notes additional coverage of Indian, Malaysian, and other emerging-market companies.","pith_inferences":["If media attention is uneven across firms and regions, the system's coverage may systematically favor well-covered companies, so the measured China advantage could partly reflect online-content availability rather than genuine supply-chain structure; this is a testable extension of the paper's own caveat about media bias.","A sharper comparison with Bloomberg would align company identifiers and time windows, then compute recall of Bloomberg edges and precision on non-Chinese emerging-market samples; such a check could quantify how much of the coverage gap reflects genuine missing data versus sampling artifacts.","The 77% precision result is based on one industry, one country focus, and one LLM pipeline configuration, so a natural extension is to replicate it in less-digitized industries or lower-resource languages, where LLM extraction performance may change."],"forward_implications":["The same pipeline can be extended to extract other relationship types, such as competition, partnership, shareholding, and technology links, at very little marginal cost.","The system can complement Bloomberg, FactSet, and similar datasets by adding coverage in countries where disclosure regulations leave most supply relationships undocumented.","Running the system for longer periods should densify the extracted graph, addressing the sparsity observed relative to Bloomberg.","Requiring LLMs to output textual evidence for each relationship enables fact traceability and supports human validation of extracted claims."],"supporting_citations":[{"why":"Defines the Bloomberg SPLC dataset that serves as the benchmark for comparison.","marker":"[4]"},{"why":"Provides the evidence on low supplier-name disclosure rates among Chinese listed companies, motivating the gap the system aims to fill.","marker":"[7]"},{"why":"Supplies the prior deep-learning baseline for supply chain relationship extraction that the paper compares against.","marker":"[18]"},{"why":"Reports earlier LLM-based supply chain relationship extraction performance that the paper extends and compares with its own precision.","marker":"[25]"},{"why":"Contributes the keyword-generation and web-search approach that the system's iterative discovery process builds on.","marker":"[15]"},{"why":"Provides an earlier machine-learning system for company relation extraction from web content, including precision benchmarks.","marker":"[17]"},{"why":"Supplies the modularity measure used to compare community structure between the system's graph and Bloomberg's subnet.","marker":"[28]"}],"fun_headline_variants":["LLM web crawler finds supply-chain links Bloomberg misses","AI supply chain graph hits 77% precision, beats Bloomberg","Semiconductor test: LLM mining reveals hidden supplier links","LLM system uncovers 6,600 supply links emerging economies","Web-crawling LLM exposes China supply chain data gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the 200 randomly sampled human-labeled relationships and the pruned graph of roughly 5,000 relationships are representative across countries, sources, and relationship types, so the observed China coverage differences reflect genuine gaps in Bloomberg rather than measurement artifacts.","fun_headline_variants_meta":{"raw":{"variants":["LLM web crawler finds supply-chain links Bloomberg misses","AI supply chain graph hits 77% precision, beats Bloomberg","Semiconductor test: LLM mining reveals hidden supplier links","LLM system uncovers 6,600 supply links emerging economies","Web-crawling LLM exposes China supply chain data gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1251,"prompt_tokens":919,"completion_tokens":332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":535,"tokens_out":332,"duration_ms":3843,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:58:39.431567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to take random samples of 200-300 extracted relationships separately for China, India, Brazil, and the US, have them human-labeled, and compute precision by region and by firm size; if precision drops sharply in regions with thinner media coverage, or if the system's China-vs-Bloomberg network differences disappear after matching companies by identifier and time period, the central claim of better emerging-economy coverage would be undermined.","supporting_citations":[{"cited_title":"Supply Chain on Bloomberg,","cited_arxiv_id":null,"evidence_quote":"Defines the Bloomberg SPLC dataset that serves as the benchmark for comparison."},{"cited_title":"Analysis of supply chain cooperation net works of Chinese listed companies from a social network perspective,","cited_arxiv_id":null,"evidence_quote":"Provides the evidence on low supplier-name disclosure rates among Chinese listed companies, motivating the gap the system aims to fill."},{"cited_title":"Extracting supply chain ma ps from news articles using deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior deep-learning baseline for supply chain relationship extraction that the paper compares against."},{"cited_title":"Enhancing Supply Chain Visibi lity with Knowledge Graphs and Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Reports earlier LLM-based supply chain relationship extraction performance that the paper extends and compares with its own precision."},{"cited_title":"Extracting Social Networks Among Various Entities on the Web","cited_arxiv_id":null,"evidence_quote":"Contributes the keyword-generation and web-search approach that the system's iterative discovery process builds on."},{"cited_title":"Company Relation Extraction from Web News Articles for Analyzing Industry Structure","cited_arxiv_id":null,"evidence_quote":"Provides an earlier machine-learning system for company relation extraction from web content, including precision benchmarks."},{"cited_title":"Fa st unfolding of communities in large networks","cited_arxiv_id":null,"evidence_quote":"Supplies the modularity measure used to compare community structure between the system's graph and Bloomberg's subnet."}],"review_version":1}