{"id":"b98848b8-f42c-4e97-920f-505cfaccd4ac","arxiv_id":"2411.11017","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Linux distributions mostly rely on reproducible builds and signing, and current open-source malware scanners perform poorly on a new benchmark of malicious Linux packages.","lead":"Interviews with Linux distribution maintainers and a new benchmark of malicious Linux packages show that most distributions rely on reproducible builds and signatures, and only Wolfi actively scans for malware. Existing open-source scanners, including VirusTotal, Malcontent, and OSS-Detect-Backdoor, produce many false positives and miss much of the malware, according to the benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central detection-rate claim contradicted by paper's own confusion matrices; VirusTotal recall exceeds 70% in Tables 5 and 7, and several matrix row sums do not match dataset sizes.","rationale":"The reader identified the benign-label assumption as the weakest point, but the authors explicitly acknowledge that limitation in Section 8.1 and argue it is unlikely to materially affect results. A more fundamental issue is that the central quantitative claim is contradicted by the paper's own confusion matrices. In Table 5, VirusTotal identifies 22 of 30 malicious samples (73.3%) in Dataset #1, exceeding the 70% threshold the conclusion denies; Table 7 shows the same for Dataset #2. Unless 'detection rate' is defined in a nonstandard way (which the paper does not do), the conclusion is false. This is not a subtle interpretation issue; it is a direct contradiction between the headline finding and the reported data. Furthermore, the confusion matrices have arithmetic inconsistencies: Bandit4Mal's 'Malicious' row in Table 5 sums to 10 rather than 30, suggesting missing or mislabeled entries. Such inconsistencies indicate the evaluation numbers are not trustworthy. Without the dataset or code, the reader cannot audit these results. The benign-label concern is secondary because even if all benign labels are correct, the confusion matrices still fail to support the 'none above 70%' claim. Thus the paper's primary empirical contribution is unsupported and internally inconsistent, warranting rejection or major revision. This moves the verdict from CONDITIONAL to REJECT.","tokens_in":15919,"tokens_out":5356,"duration_ms":147918,"concrete_test":"Recompute detection rates from the confusion matrices in Tables 5–9 using the standard recall formula TP/(TP+FN). For VirusTotal, Table 5 gives 22/30 = 73.3% and Table 7 gives 22/30 = 73.3%, both exceeding 70%. Check whether the authors define 'detection rate' differently; if so, they must state the metric and show how the conclusion follows. Also verify row sums: Table 5's Bandit4Mal 'Malicious' row sums to 10, but Dataset #1 contains 30 malicious samples; request the full per-sample results or the released code and dataset to reproduce the tables. If the confusion matrices are correct, the 'none above 70%' claim is false; if they are incorrect, the evaluation is unreliable.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central conclusion (Section 9) states that 'none of the tools achieved detection rates above 70%.' Yet the confusion matrices in Table 5 and Table 7 show VirusTotal with 22 true positives out of 30 malicious samples (22/30 = 73.3%), which is above 70%. Table 6 shows VirusTotal at exactly 70% (21/30). If 'detection rate' means recall, the central claim is empirically false based on the paper's own data. No alternative metric is defined, so the statement is either contradicted or unverifiable. Additionally, the confusion matrices contain internal inconsistencies: in Table 5, Bandit4Mal's 'Malicious' row sums to 10 instead of the 30 samples in Dataset #1; other rows similarly do not reconcile with reported dataset sizes. These arithmetic problems undermine the reliability of the quantitative evaluation. The benchmark code and dataset are not provided, so the numbers cannot be independently checked. The benign-sample assumption (Section 8.1) is a secondary concern; even if all benign labels are correct, the reported confusion matrices do not support the headline detection-rate claim. This is the most load-bearing weakness because the paper's main contribution is the empirical assessment of scanner inadequacy, and that assessment is internally inconsistent and contradicted by its own tables.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper examines malware prevention in Linux distribution package repositories through two studies: semi-structured interviews with seven maintainers from Alpine, Arch, Debian, Ubuntu, and Wolfi, and a benchmark evaluation of six open-source malware scanners (Bandit4Mal, Malcontent, ODB, Packj, VirusTotal, Capslock) on six newly constructed malicious datasets paired with benign Wolfi packages and repositories. The interviews find that most distributions rely on reproducible builds and cryptographic signing, with only Wolfi actively scanning packages using Malcontent. The scanner evaluation reports high false positive rates and generally low true positive rates, leading the authors to conclude that none of the evaluated tools achieved detection rates above 70% and that current tools are unlikely to be adopted by Linux distribution maintainers.","tokens_in":16234,"tokens_out":5897,"duration_ms":61992,"significance":"If the benchmark results were reliable, the paper would make a useful contribution: it is, to my knowledge, the first interview study of Linux distribution maintainers on proactive malware scanning, and it provides a Linux-package malware benchmark that spans source code and APK artifacts plus an over-time variant. The dataset design is thoughtful, and the authors explicitly document several threats to validity. However, the quantitative evaluation as currently presented is internally inconsistent and cannot support the headline conclusion; the paper therefore reads as a promising empirical study that currently needs substantial repair rather than as a completed result.","major_comments":[{"comment":"The central claim that 'none of the tools achieved detection rates above 70%' is contradicted by the paper's own confusion matrices: Table 5 reports VirusTotal with 22 true positives out of 30 malicious samples in Dataset #1 (73.3%) and ODB with 29 true positives (96.7%), and Table 7 reports VirusTotal with 22 true positives out of 30 in Dataset #2. Because no alternative definition of 'detection rate' is given, the conclusion is either false or unverifiable; the metric must be defined and reconciled with the tables.","section":"Section 9, Tables 5 and 7"},{"comment":"Several confusion matrices contain row sums that do not match the dataset sizes stated in Table 2. In Table 5, Bandit4Mal's malicious row sums to 10 rather than 30, and the benign rows sum to 891, 1856, 1775, 1183, and 1855 rather than the 1866 Wolfi upstream repositories. In Table 7, ODB's malicious row sums to 25 rather than 30 and its benign row sums to 1748 rather than 1866. In Table 8, ODB's malicious row sums to 28 rather than 30. These inconsistencies make the reported precision and recall values impossible to verify and suggest that failed analyses or mislabeled columns are being silently dropped.","section":"Tables 5, 7, and 8"},{"comment":"The detection threshold is not applied uniformly across tools: the stated default threshold is one alert, but for Malcontent only 'High' and 'Critical' alerts are considered while for the other scanners any alert appears to count. Because no severity threshold is reported for the other tools, the cross-tool comparisons in Tables 5-9 conflate raw alert counts with detection decisions and do not constitute a controlled comparison of detection capability.","section":"Section 3.4"},{"comment":"The benign ground truth is load-bearing for all false-positive claims, yet Section 3.3.1 assumes all Wolfi packages and upstream repositories are benign and Section 8.1 acknowledges that this assumption has not been thoroughly validated. Since high false-positive rates are central to the conclusion that the tools are unsuitable, the paper needs either substantially stronger validation of the benign labels or a sensitivity analysis showing that plausible labeling errors do not change the conclusion.","section":"Sections 3.3.1 and 8.1"},{"comment":"The paper introduces six datasets as a contribution, but no dataset repository, artifact link, or evaluation script is provided. Given the arithmetic inconsistencies in the reported tables, independent verification is impossible; the authors should release the benchmark data and evaluation code, or at minimum the per-sample classification counts underlying every table.","section":"Sections 3.2 and 3.3"}],"minor_comments":[{"comment":"Several table entries are incomplete, including ODB's mean being reported as 'value missing' and the Malcontent observation ending with '(statement incomplete)'; the table and surrounding text should be completed.","section":"Section 5, Table 4"},{"comment":"The text '16 000Y ARA detection rules' should read '16,000 YARA detection rules'.","section":"Section 3.4"},{"comment":"The phrase 'these findings shed light on the shed light on a fundamental gap' contains a duplicated clause and should be rewritten.","section":"Section 1"},{"comment":"The table heading says 'Dataset #4 vs Malicious Wolfi APKs' where the surrounding text and the identical benign columns indicate it should say 'vs Benign Wolfi APKs'.","section":"Table 8"},{"comment":"Reference [33] contains a stray 'wolfi os' in its title, and the URL formatting should be checked.","section":"References"},{"comment":"The closing sentence 'Here is to hoping' is informal for a journal conclusion; consider 'Here is hoping' or a more formal rephrasing.","section":"Section 9"},{"comment":"The caption does not define the 'combined dataset' used for the alert distribution; clarify whether it pools benign and malicious samples and justify that aggregation.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"Three authors are affiliated with Chainguard, which maintains Malcontent, one of the evaluated tools, and Wolfi, which is the source of all benign samples and the only distribution reported to perform active malware scanning. This is a clear conflict of interest that the paper discloses only indirectly in Section 3.1. I see no direct evidence of manipulated results, but the relationship should be stated explicitly and the benchmark should be independently audited. The paper also claims several 'firsts'; these novelty claims are plausible but should be sharpened against the cited prior work. If the tables, artifact, and central claim are corrected, this could become a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The interview study is the real contribution here, and it's worth reading. But the quantitative evaluation—the other half of the paper—doesn't hold together. The central conclusion in Section 9, that “none of the tools achieved detection rates above 70%”, is directly contradicted by Tables 5 and 7, where VirusTotal shows 22/30 true positives (73%). Unless “detection rate” means something other than recall, and that is never defined, the sentence is false on the paper's own evidence.\n\nWhat's good: RQ1 gives the first systematic interview evidence I know of on how maintainers at Alpine, Arch, Debian, Ubuntu, and Wolfi think about malware. The finding that most have leaned on reproducible builds and signing, and that only Wolfi actively scans, is plausible and well-supported. The benchmark design covers six dataset types, including source vs APK and an over-time version-diff setup, which is a thoughtful spread. The authors also disclose the Wolfi/Chainguard affiliation, which is more than many conflict-of-interest cases do.\n\nThe soft spots are real, and some are load-bearing. The confusion matrices don't reconcile: Bandit4Mal's malicious row in Table 5 sums to 10, not the 30 samples in Dataset #1, and benign row totals vary across tools without explanation. The 70% claim is the empirical core of the paper, and it's contradicted by the paper's own numbers. Also: the benign packages are assumed benign (acknowledged in Section 8.1, but not validated); the benchmark dataset and code are not released, so nothing can be independently checked; and there are production-quality problems—missing values in Table 4 and incomplete sentences in Section 5—that suggest the manuscript is not ready as-is. The conflict of interest in evaluating Malcontent and Wolfi packages isn't disqualifying on its own, but a sensitivity analysis and an independent check of thresholds would make the results much easier to trust.\n\nIf the quantitative part is fixed—corrected matrices, a clearly defined metric, and released artifacts—the paper's bottom line may still hold: most open scanners are too noisy for distro maintainers in practice. The interviews alone justify a serious look. So I'd send it to peer review, but with a strong request for major revision and full artifact release. Right now I'd cite it for the RQ1 findings, not for the RQ2 numbers.","headline":"The interview findings are genuinely useful, but the paper's headline claim that no scanner beats 70% is contradicted by its own tables and the quantitative evaluation needs a full re-do before I'd trust it.","tokens_in":16697,"tokens_out":3339,"would_cite":true,"duration_ms":37555,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current open-source malware scanners are inadequate for Linux distribution packages, with none achieving detection rates above 70% in a new benchmark built from Wolfi OS packages.","keywords":["Linux malware","software supply chain security","malware detection benchmark","Linux distributions","Wolfi OS","APK packages","malware scanners","XZ Utils backdoor"],"falsifier":"Audit a random sample of the 1,866 Wolfi upstream repositories and 1,652 APKs with independent manual reverse engineering or a dynamic sandbox; finding malware in a non-trivial fraction would invalidate the reported false positive rates. Alternatively, rerun the evaluation on a hand-verified benign corpus from another distribution, such as Debian or Alpine, and check whether the tools' false positive rates match the ones reported here.","tokens_in":15778,"feed_emoji":"🛡️","tokens_out":7370,"duration_ms":61591,"temperature":0.7,"pith_summary":"Existing open-source malware detection tools are not ready to protect Linux distribution package repositories, according to this study. Interviews with maintainers from five major distributions show that most projects rely on reproducible builds and cryptographic signing, and only Wolfi OS actively scans packages for malware. The authors built six benchmark datasets of malicious and benign Linux packages, covering both source code and APK binaries, and evaluated six open-source scanners against them. None of the tools exceeded a 70% true positive rate, and most produced so many false positives that volunteer maintainers would be overwhelmed. The stakes are concrete: without better scanners, a backdoor like the XZ Utils compromise could enter major distributions undetected.","feed_headline":"No open-source scanner detects even 70% of Linux package malware","feed_subtitle":"A new benchmark of malicious and benign Wolfi packages shows current tools drown maintainers in false positives.","key_machinery":"The load-bearing mechanism is the benchmark dataset suite constructed by the authors. Starting from the Wolfi OS package set, the authors assume every package and upstream repository is benign ground truth, then build six malicious datasets: known malicious samples from the Backstabber's Knife Collection; those samples compiled into Wolfi APKs with the melange build tool; benign source projects injected with known malicious code snippets; those injected projects packaged as APKs; real Linux malware source code converted into APKs; and Go projects with Coldfire-based malware inserted into the middle of five-version histories. Each sample is scanned, and a tool counts as detecting malware if it produces at least one alert or, for VirusTotal, a malicious verdict. This single-threshold protocol produces the confusion matrices across all six datasets, and those matrices are what support the claim that no tool exceeds 70% detection.","core_discovery":"The central discovery is that open-source malware scanners, as they exist today, cannot reliably distinguish malicious Linux packages from benign ones. The paper introduces the first benchmark suite for Linux package malware, consisting of six datasets: historical malicious source code from the Backstabber's Knife Collection; the same samples compiled into Wolfi APKs; synthetic source code with injected malicious snippets; synthetic malicious APKs; full-fledged Linux malware turned into APKs; and an over-time dataset of Go projects where only the middle version of each five-version history is malicious. These are paired with 1,866 benign Wolfi upstream repositories and 1,652 benign APKs. Scanning this collection with Bandit4Mal, Malcontent, ODB, Packj, VirusTotal, and Capslock yields a uniformly negative result: no tool reaches a 70% detection rate, most tools flag large fractions of benign packages as malicious, and differential version-diff analysis is noisy and low-precision. The paper concludes that current tools are unlikely to be adopted by distribution maintainers and need better rules, dynamic analysis, and tuning.","pith_inferences":["The benchmark's benign ground truth is the weakest link: if a non-trivial fraction of the 1,866 Wolfi packages are actually malicious, the reported false positive rates are inflated and the entire comparison shifts; a manual audit of a random sample would settle this.","The 'none above 70%' result likely underestimates real-world difficulty, because the malicious datasets are largely historical or synthetically injected and real malware can be obfuscated; a dynamic-analysis-based benchmark could be even harsher.","The authors' affiliation with Wolfi's maintainer raises a question the paper does not address: whether Malcontent would score differently if benchmarked by an independent team on a non-Wolfi benign corpus.","A natural extension is to evaluate commercial scanners against the same benchmark, since VirusTotal's aggregate beats all open-source tools and the paper's exclusion of closed-source scanners leaves that comparison open."],"forward_implications":["Distribution maintainers should not rely on current open-source scanners as a gate before accepting packages, since the false positive volumes would overwhelm volunteer review.","Wolfi's Malcontent-based scanning, while pioneering, will also miss most real malware until its detection rules improve substantially.","The benchmark suite can serve as a reusable baseline for future scanners, and any tool that exceeds roughly 70% true positive at a tolerable false positive rate on these datasets would be a meaningful advance.","Differential scanning that compares alert sets between package versions does not yet reduce noise enough to be practically useful.","VirusTotal's relatively balanced performance suggests that aggregating many detection engines is more promising than any single open-source tool."],"supporting_citations":[{"why":"Supplies the historical malicious open-source samples that seed Dataset #1 and, after compilation, Dataset #2.","marker":"[4]"},{"why":"Provides the injected malicious code snippets and the prior evidence of high false positive rates that motivate the benchmark design.","marker":"[23]"},{"why":"The melange build tool that turns malicious source code into Wolfi APK binaries for Datasets #2, #4, #5, and #6.","marker":"[31]"},{"why":"The Wolfi OS package set and its upstream repositories form the benign control group for all six datasets.","marker":"[28]"},{"why":"The Coldfire Go malware library whose samples are injected into version histories to construct Dataset #6.","marker":"[33]"},{"why":"Malcontent is the scanner Wolfi runs in production and one of the six tools benchmarked, including in the over-time differential analysis.","marker":"[37]"},{"why":"The Reproducible Builds project is the primary counter-malware practice interviewees cite, grounding the RQ1 findings.","marker":"[11]"}],"fun_headline_variants":["Linux malware scanners flunk new benchmark","Open-source scanners fail Linux malware test","Benchmark: no scanner catches most Linux malware","False positives plague Linux malware scanners","Only Wolfi scans Linux packages; tools underperform"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark labels every Wolfi OS package and its upstream repository as benign without verifying it, so any actually malicious package in that set would inflate the measured false positive rates and distort the ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Linux malware scanners flunk new benchmark","Open-source scanners fail Linux malware test","Benchmark: no scanner catches most Linux malware","False positives plague Linux malware scanners","Only Wolfi scans Linux packages; tools underperform"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1352,"prompt_tokens":998,"completion_tokens":354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":289}},"tokens_in":614,"tokens_out":354,"duration_ms":4668,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:00:17.239352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of the 1,866 Wolfi upstream repositories and 1,652 APKs with independent manual reverse engineering or a dynamic sandbox; finding malware in a non-trivial fraction would invalidate the reported false positive rates. Alternatively, rerun the evaluation on a hand-verified benign corpus from another distribution, such as Debian or Alpine, and check whether the tools' false positive rates match the ones reported here.","supporting_citations":[{"cited_title":"Bad snakes: Understanding and improving python package index malware scanning,","cited_arxiv_id":null,"evidence_quote":"Provides the injected malicious code snippets and the prior evidence of high false positive rates that motivate the benchmark design."},{"cited_title":"melange,","cited_arxiv_id":null,"evidence_quote":"The melange build tool that turns malicious source code into Wolfi APK binaries for Datasets #2, #4, #5, and #6."},{"cited_title":"Wolfi os,","cited_arxiv_id":null,"evidence_quote":"The Wolfi OS package set and its upstream repositories form the benign control group for all six datasets."},{"cited_title":"Coldfire: Golang malware development librarywolfi os,","cited_arxiv_id":null,"evidence_quote":"The Coldfire Go malware library whose samples are injected into version histories to construct Dataset #6."},{"cited_title":"Malcontent,","cited_arxiv_id":null,"evidence_quote":"Malcontent is the scanner Wolfi runs in production and one of the six tools benchmarked, including in the over-time differential analysis."},{"cited_title":"Reproducible builds: Increasing the integrity of software supply chains,","cited_arxiv_id":null,"evidence_quote":"The Reproducible Builds project is the primary counter-malware practice interviewees cite, grounding the RQ1 findings."}],"review_version":1}