{"id":"11bbe914-24db-4a5a-a4fe-e7ec944e3ed0","arxiv_id":"2412.05259","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A stacking ensemble classifier using static features, including a new vocabulary-based text feature, detects malicious PyPI packages with a claimed F1 of 94 percent, though the per-class malicious F1 is 88 percent.","lead":"This paper trains a machine learning model on package metadata, code, files, and text to flag malicious Python packages on PyPI, reporting high detection accuracy. It matters because package registries are flooded with new submissions and need automated triage to stop supply chain attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract and RQ1 report a macro-averaged F1 (94.2%) as the F1 for identifying malicious packages; per-class malicious F1 is 88% in training and ≈85% on the test set, so the headline claim overstates the detector's positive-class performance.","rationale":"The reader's conditional verdict is appropriate and remains unchanged, but I identify a different load-bearing concern than the reader's stated weakest assumption. The unverified benign labels on the 397 typical packages are a real validity threat; however, the macro-average F1 issue is more decisive because it is demonstrated inside the paper's own tables and changes the headline number by roughly 6–9 percentage points for the class of interest. The reader did note in the rationale that the abstract overstates the 94.2% F1 and that the malicious-class F1 is 88%, so there is partial agreement, but the reader's formal weakest_assumption focuses on label verification rather than on the macro-average reporting. The concrete test above settles the concern arithmetically and by re-running on the shared dataset. If the per-class malicious F1 is indeed ≈85% on the test set, the paper's central claim should be revised to accurately describe performance on the malicious class; the method may still be useful, but the current abstract and RQ1 wording are not supported. This does not change the conditional verdict, which already requires reporting corrections and additional validation, but it sharpens the required correction.","tokens_in":17569,"tokens_out":7494,"duration_ms":79153,"concrete_test":"Recompute the malicious-class F1 from the Table 5 confusion counts: with TP=105, FN=38, and FP=0 (as stated in Section 5.1), malicious-class F1 = 2*105 / (2*105 + 0 + 38) ≈ 84.7%. Independently verify this against the released Zenodo dataset by rerunning the trained stacking classifier on the test split and computing per-class precision, recall, and F1. If the positive-class F1 is ≈85%, the Abstract and RQ1 conclusions must be restated as macro-averaged F1, not as the F1 for detecting malicious packages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'F1-measure of 0.94 for identifying malicious packages'—is not supported by the paper's own per-class metrics. In Table 3, the best stacking configuration reports per-class F1 scores of 88% for malicious and 100% for benign packages; the 94.2% value is the macro average of these two classes, not the F1 for the class the detector is designed to find. The same issue appears in the test-set results in Table 5 and Section 5.1: with 105 true positives, 38 false negatives, and 0 false positives among the 397 typical packages, the malicious-class recall is 105/143 = 73.4%, precision is 100%, and F1 is ≈84.7%, whereas the reported 90% F1 is again the macro average (≈90.1%). Thus the abstract's headline figure is internally inconsistent with the confusion counts given in the paper; the model misses 26.6% of malicious test packages, a substantially weaker result than '94.2% F1 for identifying malicious packages' implies. This is a reporting/interpretation flaw in the paper's own data, not an external-threat argument, and it directly affects the strongest claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a package-level machine learning pipeline for detecting malicious PyPI packages using metadata, file, code, and text features. The authors train six classifiers on 5,193 popular PyPI packages and 138 known malicious packages, evaluate them with stratified 10-fold cross-validation, and test the best stacking ensemble on 397 unverified 'typical' packages and 143 new malicious packages. They report a training F1-score of 94.2% and a test F1-score of 90%, and compare their approach against Bandit, Packj, and the method of Ohm et al. in terms of usefulness and detection coverage. The paper also releases a dataset and identifies important features such as suspicious APIs, URLs, license mismatches, and author/homepage validity.","tokens_in":17780,"tokens_out":4016,"duration_ms":37148,"significance":"If the reported performance reflects per-class detection accuracy, the approach would be a practical, package-level vetting aid for PyPI, complementing function-level scanners, and the released dataset could support further research. The authors should be credited for evaluating on a separate test set and for comparing with existing tools. However, the headline F1 is a macro-average that overstates malicious-class detection, and the text feature is derived from Packj, which is also a baseline in the usefulness comparison. These issues directly affect how the central claims should be interpreted and require attention before publication.","major_comments":[{"comment":"The reported F1-scores of 94.2% (training) and 90% (test) are macro-averages over the benign and malicious classes, not the F1 for the malicious class that the paper claims to identify. In Table 3, the best stacking configuration shows a per-class malicious F1 of 88%. From the test confusion counts in Table 5 (105 true positives, 38 false negatives, 0 false positives among the 397 typical packages), the malicious-class precision is 100%, recall is 105/143 = 73.4%, and F1 is approximately 84.7%. Thus the abstract's statement 'F1-measure of 0.94 for identifying malicious packages' is inconsistent with the paper's own per-class numbers and overstates the detector's positive-class performance. The per-class metrics should be reported and used in the abstract and RQ1.","section":"Abstract and Section 5.1 (Tables 3 and 5)"},{"comment":"The text-related feature is constructed from Packj's static analysis report ('we rely on the generated static analysis report of the packj tool to construct a portion of text feature'), and Packj is one of the two baseline tools in the usefulness evaluation of RQ3. The model therefore ingests a key signal from the very tool it is claimed to outperform, which makes the RQ3 comparison circular. Section 7's threats-to-validity paragraph acknowledges reliance on Packj for feature extraction 'rather than drawing final conclusions' but does not address the baseline comparison. The authors should either re-run the RQ3 comparison with features that do not come from Packj, or explicitly frame the comparison as showing the added value of Packj-derived features rather than as a head-to-head against Packj.","section":"Section 4.2 and Section 5.3 (RQ3)"},{"comment":"The 397 'typical' packages treated as benign in the test set are not verified as non-malicious, although Section 4.1 states that 'we cannot assume that other recently published packages are benign.' The test row reports zero false positives among these packages; if any of them are actually malicious, the reported precision and F1 overestimate the model's generalization. The authors should verify these labels with an independent malicious-package scanner or at least report the results under an alternative assumption (e.g., a small percentage of the typical packages being malicious) to quantify the sensitivity of the conclusion.","section":"Section 4.1 and Table 5"}],"minor_comments":[{"comment":"The table reports test precision of 96% and recall of 87%, but these are macro-averages; the table should state this explicitly and include the per-class precision, recall, and F1 for both malignant and benign classes.","section":"Table 5"},{"comment":"The figure content appears to have been replaced by a garbled path string ('/uni00000053/uni00000044/...'), making the figure unreadable in the submitted version; the plot should be regenerated and its axes and legend described in the caption.","section":"Figure 3"},{"comment":"The sentence 'We found no correlation' after computing Pearson correlation does not report the correlation values or the significance threshold; including these values would improve reproducibility.","section":"Section 4.2"},{"comment":"The adaptation of Ohm et al.'s method to Python packages is not described in enough detail; specify which features were ported and how the intersection of the three classifiers was implemented on the PyPI dataset.","section":"Section 5.3"},{"comment":"There is a typo in 'For examaple' (should be 'For example') in the paragraph about libpesh-0.1.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The macro-F1 criticism raised in the stress-test note is well-supported by the manuscript's own tables: the 94.2% figure is a macro-average, and the malicious-class test F1 is about 85%. The circularity concern is also substantial because Packj supplies the text feature and is also a baseline; this should be fixed before publication. The paper has merit and the issues are addressable within its scope, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate incremental contribution—supervised ML with some new metadata/file features adapted to PyPI, plus a released dataset—but the way results are reported oversells the detector. The abstract's 94.2% F1 is a macro average; per-class malicious F1 is 88% in training and roughly 85% on the test set (105/143 true positives, 38 false negatives). That's still decent, but it's not the same claim.\n\nWhat's genuinely useful: the vocabulary/text feature, built from suspicious code lines, carries signal; the paper shows it beats the metadata-code-file features alone (89% vs 84% F1). The external test set of 143 newer malicious packages is a good check, and the manual misclassification analysis is honest. Sharing the dataset on Zenodo is a real plus.\n\nSoft spots, in order of severity:\n\n1. The headline number. The abstract and RQ1 repeatedly say 'F1-score of 0.94 for identifying malicious packages' when the per-class malicious F1 is lower. The test-set confusion counts in Table 5 don't match the reported precision/recall/F1 either—if 38 of 143 malicious are missed and all 397 typical are correctly accepted, malicious recall is 73%, not 87%. The macro averaging should be stated clearly.\n\n2. Test labels. The 397 'typical' packages are assumed benign without verification, and the paper itself says newly published packages can't be assumed benign. If some are malicious, the test precision/F1 are optimistic.\n\n3. Packj circularity. The text feature is built from Packj's static analysis output, and Packj is one of the two baselines in RQ3. The paper acknowledges using Packj for feature extraction but doesn't fully address the confound. This doesn't sink the approach, but the usefulness comparison to Packj is weaker than presented.\n\n4. No code. The dataset is there, but the pipeline and classifier configuration aren't. Pin tool versions and release the feature extraction code.\n\nOverall this paper deserves a serious referee. The core idea isn't groundbreaking but it's sound enough to warrant revision, not desk rejection. I'd ask the authors to correct the F1 reporting, verify or caveat the test labels, and release the pipeline. If they do that, the paper makes a modest, honest contribution.","headline":"A plausible PyPI malware detector with a useful shared dataset, but the headline F1 is a macro average and the Packj-based text feature confounds the baseline comparison.","tokens_in":18402,"tokens_out":3320,"would_cite":true,"duration_ms":30476,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A stacking ensemble classifier using package metadata, code, and text vocabulary is shown to detect malicious PyPI packages with an F1 of 94.2%, and 90% on unseen typical packages.","keywords":["malicious packages","supply chain security","PyPI","stacking ensemble","static analysis","vocabulary-based detection","typosquatting"],"falsifier":"Take the 397 'typical' packages from the test set and subject each to deep manual inspection plus dynamic sandbox execution to check for malicious behavior. If more than a small percentage of them show malicious or unwanted activity, the classifier's assumed ground truth is wrong, and its test precision and F1 overestimate real-world performance. Similarly, the 5,193 popular training packages could be spot-checked the same way.","tokens_in":17347,"feed_emoji":"🛡️","tokens_out":6724,"duration_ms":60156,"temperature":0.7,"pith_summary":"This paper sets out to show that a machine-learning classifier, built from static clues in a Python package's metadata, configuration files, source code, and the text of its suspicious code lines, can identify malicious PyPI packages as a whole package rather than as individual function calls. The authors report that a stacking ensemble of five base classifiers reaches an F1-score of 94.2 percent on a cross-validated set of 138 known malicious and 5,193 popular packages, and 90 percent on an unseen set of 143 new malicious and 397 typical packages. If these numbers hold, registry maintainers could run the classifier on every newly submitted package and triage the worst candidates for manual review, cutting down the manual workload that currently limits ecosystem vetting. The paper also claims its approach produces fewer false alerts than two widely used static-analysis tools, Bandit and Packj, on the sample it tested.","feed_headline":"Machine learning detects malicious PyPI packages with 94% F1","feed_subtitle":"Static clues like missing emails, license mismatches, and suspicious code text separate malware from benign libraries.","key_machinery":"The load-bearing mechanism is the vocabulary-based text feature. The authors use the Packj static-analysis report to extract lines of code that invoke sensitive APIs (read, open, connect, getattr, system, and others), enrich them with code from setup.py and any source files containing suspicious URLs, and feed the tokenized text to the classifier as a document. This lets the model learn the wording and combination of code rather than just the presence of boolean flags. Around this text feature, the model layers carefully crafted package-level indicators: a missing or invalid homepage, a missing or invalid author email, a setup.cfg that has been left at the minimal auto-generated configuration, mismatched license declarations across the classifier/field/file locations, the presence of install commands or suspicious URLs, and long strings typical of obfuscation. The stacking ensemble is the meta-classifier that combines Random Forest, Support Vector Machine, Decision Tree, Multilayer Perceptron, and Bernoulli Naive Bayes, and the paper reports that the synergy of all features plus text achieves the highest F1.","core_discovery":"The central claim is that a stacking ensemble classifier, using eight feature groups—two metadata features (valid homepage or repository, valid author email), two file-level features (minimum setup configuration, license mismatch), three code-level features (install command, suspicious URL, long string), and one text feature built from lines of code that call suspicious APIs such as getattr, connect, open, and read—can separate malicious PyPI packages from benign ones at package level. On the training set the stacking classifier achieves 98% precision and 91% recall (F1 94.2%), and on a held-out test set of 143 new malicious packages and 397 randomly selected typical packages it achieves 96% precision and 87% recall (F1 90%). The text vocabulary alone outperforms the other seven features combined (F1 89% versus 84%), and the full model recovers 105 of 143 malicious test packages while flagging no benign test packages. These results are presented as evidence that the approach generalizes from popular packages to the broader PyPI population.","pith_inferences":["I infer that the license-mismatch feature, which the paper finds in 77% of malicious but only 1% of benign packages, could be used as a standalone high-precision pre-filter, leaving the ML model to adjudicate ambiguous cases.","If the core claim is correct, a multi-tier vetting pipeline combining this static classifier with subsequent dynamic analysis of only flagged packages would be cheaper and more accurate than either method alone.","The paper's own analysis leaves open that 'typical' PyPI packages may sometimes be malicious, so a practical deployment should follow up on the classifier's 'benign' verdicts with a low-cost dynamic probe to catch false negatives.","A natural test of the vocabulary hypothesis is to retrain on a newer malicious package collection and see whether the same token patterns still discriminate, or whether attackers shift vocabulary to evade the model."],"forward_implications":["Registry maintainers could run the classifier on every new PyPI submission and send only the flagged packages to a human reviewer, making the weekly triage of roughly 1,800 versions feasible.","Because the model operates on static features, it can scale to the entire registry without the resource cost of dynamic analysis, and it alerts on whole packages, not just suspicious function calls.","The text vocabulary feature points reviewers to the specific code lines the model found suspicious, so a human can quickly verify the alert.","The approach's low false-positive rate on the tested benign packages suggests it could be deployed as an early filter before more expensive analysis methods are invoked.","The 38 erroneously classified malicious packages were mostly clones of popular packages with thin payloads, so augmenting the features with similarity or provenance signals could close that gap."],"supporting_citations":[{"why":"Supplies the curated malicious-package dataset (138 for training, 143 for testing) that defines the positive class.","marker":"[38]"},{"why":"Generates the static-analysis report of suspicious API calls that forms the basis for the text vocabulary feature.","marker":"[41]"},{"why":"Provides the closest prior method and feature set that is reused, extended, and compared against in RQ3.","marker":"[37]"},{"why":"One of the two baseline tools in the RQ3 comparison, used to measure false-alert rates.","marker":"[8]"},{"why":"Motivates the use of typical (non-popular) packages in the test set to show generalization.","marker":"[57]"},{"why":"Source of the top dependent-upon package list used to build the benign training set.","marker":"[3]"}],"fun_headline_variants":["ML stack ensemble spots malicious PyPI packages with 94% F1","Stacking classifier flags malicious PyPI packages at 94% F1","Malicious PyPI packages caught by ML static analysis: 94% F1","Static clues help ML find malicious PyPI packages: 94% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that all packages labeled 'benign' in both training and test sets genuinely contain no malicious code, even though it admits that newly published packages cannot be assumed benign; if any of the 397 typical test packages are actually malicious, the reported test precision and F1 of 90 percent are too optimistic.","fun_headline_variants_meta":{"raw":{"variants":["ML stack ensemble spots malicious PyPI packages with 94% F1","Stacking classifier flags malicious PyPI packages at 94% F1","Malicious PyPI packages caught by ML static analysis: 94% F1","Static clues help ML find malicious PyPI packages: 94% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1303,"prompt_tokens":977,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":593,"tokens_out":326,"duration_ms":3354,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:01.614749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 397 'typical' packages from the test set and subject each to deep manual inspection plus dynamic sandbox execution to check for malicious behavior. If more than a small percentage of them show malicious or unwanted activity, the classifier's assumed ground truth is wrong, and its test precision and F1 overestimate real-world performance. Similarly, the 5,193 popular training packages could be spot-checked the same way.","supporting_citations":[{"cited_title":"(accessed: 20.07.2024)","cited_arxiv_id":null,"evidence_quote":"Generates the static-analysis report of suspicious API calls that forms the basis for the text vocabulary feature."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the closest prior method and feature set that is reused, extended, and compared against in RQ3."},{"cited_title":"(accessed: 27.07.2024)","cited_arxiv_id":null,"evidence_quote":"One of the two baseline tools in the RQ3 comparison, used to measure false-alert rates."},{"cited_title":"Libraries- the open source discovery service","cited_arxiv_id":null,"evidence_quote":"Source of the top dependent-upon package list used to build the benign training set."}],"review_version":1}