PYPILINE: Malicious PyPI Package Detection via Suspicious API Knowledge and Agent Workflow
Pith reviewed 2026-06-26 20:29 UTC · model grok-4.3
The pith
PYPILINE detects malicious PyPI packages using a suspicious API knowledge base and an agent workflow for semantic analysis.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
PYPILINE first performs static analysis on known malicious packages to extract abstract syntax trees and generate API call graphs, from which a structured suspicious API knowledge base is built. In the agent workflow, RAG technology invokes this knowledge base to support in-depth semantic analysis of target packages, output structured evaluation reports, and automatically send them to a mail server.
What carries the argument
The suspicious API knowledge base extracted from call graphs of known malicious packages, which the agent workflow queries via RAG to enable semantic detection.
If this is right
- The method achieves 96.7% precision, 99.6% recall, and 98.1% F1 score.
- F1 score improves by 5.7 to 21.6 percentage points over baseline tools.
- A single package is detected in an average of 0.6 seconds when 30 threads run concurrently.
- A large-scale study of malware packages reveals common attack strategies and most frequently abused APIs.
Where Pith is reading between the lines
- The same static extraction plus agent workflow pattern could be tested on package repositories for other languages.
- Periodic rebuilding of the knowledge base from newly confirmed malware would be needed to sustain performance.
- The structured reports could feed into automated quarantine or review queues at PyPI itself.
Load-bearing premise
The suspicious API knowledge base extracted from known malicious packages captures patterns sufficient to identify new malicious packages accurately through semantic analysis without dynamic execution.
What would settle it
Evaluating PYPILINE on a set of malicious PyPI packages discovered after the knowledge base was constructed to check whether recall remains near 99.6%.
Figures
read the original abstract
Detecting malicious PyPI packages is crucial for maintaining the security of the open source software supply chain. Traditional static rule detection methods require continuous maintenance by experienced security personnel, resulting in high labor costs. Dynamic analysis methods require actual execution of the target package code, posing a risk of malicious code proliferation, and incurring significant runtime overhead and low detection efficiency. Machine learning and LLM methods iterate the detection kernel but cannot invoke multiple tools, resulting in insufficient automation.To address these issues, we propose a novel detection method called PYPILINE, which combines suspicious API knowledge and agent workflow. PYPILINE first performs static analysis on known malicious packages, extracting abstract syntax trees and generating API call graphs. From these graphs, a structured suspicious API knowledge base is extracted and constructed. In the agent workflow, PYPILINE uses RAG technology to invoke this knowledge base to enhance analytical capabilities, performing in-depth semantic analysis of the packages, outputting structured evaluation reports, and automatically sending the reports to a mail server.Experimental results show that PYPILINE achieves precision of 96.7\%, recall of 99.6\%, and F1 score of 98.1\%. F1 score is improved by 5.7 to 21.6 percentage points compared to baseline tools. When 30 threads execute concurrently, detecting a single package takes an average of only 0.6 seconds.Furthermore, we conducted a large scale empirical study of malware packages, systematically revealing common attack strategies and the most frequently abused APIs. PYPILINE provides an intelligent, efficient, and automated package detection solution, enhancing the security of the open source software ecosystem.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes PYPILINE, a detection method for malicious PyPI packages that first performs static analysis on known malware to extract ASTs and API call graphs, constructs a structured suspicious API knowledge base, and then employs an RAG-augmented agent workflow for semantic analysis, structured report generation, and automated email delivery. It claims precision of 96.7%, recall of 99.6%, F1 of 98.1% (improving 5.7–21.6 points over baselines), average detection time of 0.6 seconds per package under 30-thread concurrency, and additional insights from a large-scale empirical study of malware attack strategies and abused APIs.
Significance. If the performance claims are substantiated, the work would offer a practical, fully static and automated alternative to dynamic analysis (avoiding execution risks) and manual rule maintenance for PyPI supply-chain security. The integration of extracted knowledge with agent workflows for structured outputs is a notable engineering contribution, and the empirical study on common attack patterns could inform broader defenses. No code release or reproducible artifacts are mentioned.
major comments (2)
- [Abstract] Abstract: The central performance claims (precision 96.7%, recall 99.6%, F1 98.1% and 5.7–21.6 point gains) are presented with no information on dataset size, malicious/benign split, baseline implementations, statistical significance, error bars, or controls against post-hoc tuning. These details are load-bearing for validating the reported results and improvements.
- [Abstract] Abstract (method and evaluation description): The detection pipeline depends on the suspicious API knowledge base (extracted via static analysis and call graphs from known malware) being representative for unseen packages. No temporal split, out-of-distribution testing, or analysis of missed obfuscated/dynamic APIs is described, leaving the generalization assumption untested and directly affecting the recall claim of 99.6%.
minor comments (1)
- [Abstract] Abstract: 'large scale empirical study' should be written as 'large-scale empirical study'.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the abstract and evaluation. We address each major comment below and will revise the manuscript to improve transparency and acknowledge limitations where appropriate.
read point-by-point responses
-
Referee: [Abstract] Abstract: The central performance claims (precision 96.7%, recall 99.6%, F1 98.1% and 5.7–21.6 point gains) are presented with no information on dataset size, malicious/benign split, baseline implementations, statistical significance, error bars, or controls against post-hoc tuning. These details are load-bearing for validating the reported results and improvements.
Authors: We agree that the abstract would be strengthened by including key evaluation details. The full manuscript (Experiments section) specifies the dataset composition, split ratios, baseline re-implementations, and reports statistical tests with error bars. We will revise the abstract to concisely reference the dataset scale, split, and note that full statistical analysis appears in the body, while respecting length limits. revision: yes
-
Referee: [Abstract] Abstract (method and evaluation description): The detection pipeline depends on the suspicious API knowledge base (extracted via static analysis and call graphs from known malware) being representative for unseen packages. No temporal split, out-of-distribution testing, or analysis of missed obfuscated/dynamic APIs is described, leaving the generalization assumption untested and directly affecting the recall claim of 99.6%.
Authors: The referee correctly notes that the manuscript does not describe a temporal split, explicit OOD testing, or analysis of obfuscated/dynamic APIs. The knowledge base is derived from a diverse collection of known malware, but we did not perform these additional generalization checks. We will revise the manuscript to explicitly state this limitation in the evaluation and discussion sections and discuss its implications for the reported recall. revision: partial
Circularity Check
No circularity: empirical pipeline relies on external knowledge base and held-out evaluation
full rationale
The paper builds a suspicious API knowledge base via static analysis (ASTs and call graphs) of known malicious packages, then applies RAG-augmented agent workflow to analyze new packages and reports empirical metrics (precision 96.7%, recall 99.6%, F1 98.1%) from experimental runs. No equations, fitted parameters, or self-citations are shown to reduce the performance claims to the inputs by construction; the knowledge base is extracted from a separate set of known malware and the evaluation is presented as independent testing. The method is self-contained against external benchmarks with no load-bearing self-referential steps.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Static analysis of abstract syntax trees and API call graphs from known malicious packages yields a representative and generalizable suspicious API knowledge base.
- domain assumption RAG-enhanced agent workflow can perform in-depth semantic analysis and produce accurate structured reports without dynamic execution.
Reference graph
Works this paper leans on
-
[1]
2026 open source security and risk analysis report
“2026 open source security and risk analysis report.” 2026, https://www.blackduck.com/resources/analyst-reports/ open-source-security-risk-analysis.html
2026
-
[2]
Huawei company. pypi mirror of huawei company
“Huawei company. pypi mirror of huawei company.” 2024, https:// mirrors.huaweicloud.com/repository/pypi/simple/
2024
-
[3]
Tencent company. pypi mirror of tencent company
“Tencent company. pypi mirror of tencent company.” 2024, https:// mirrors.cloud.tencent.com/pypi/simple
2024
-
[4]
Alibaba company. pypi mirror of alibaba company
“Alibaba company. pypi mirror of alibaba company.” 2024, https:// mirrors.aliyun.com/pypi/simple/
2024
-
[5]
Pypi index
“Pypi index.” 2024, https://pypi.org/
2024
-
[6]
Pypi simple
“Pypi simple.” 2024, https://pypi.org/simple/
2024
-
[7]
Beyond typosquatting: an in-depth look at package confusion,
S. Neupane, G. Holmes, E. Wyss, D. Davidson, and L. De Carli, “Beyond typosquatting: an in-depth look at package confusion,” in32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 3439– 3456
2023
-
[9]
Continuous intrusion: Characterizing the security of continuous integration services,
Y . Gu, L. Ying, H. Chai, C. Qiao, H. Duan, and X. Gao, “Continuous intrusion: Characterizing the security of continuous integration services,” in2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 1561–1577
2023
-
[10]
Investigating package related security threats in software registries,
Y . Gu, L. Ying, Y . Pu, X. Hu, H. Chai, R. Wang, X. Gao, and H. Duan, “Investigating package related security threats in software registries,” in 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 1578–1595
2023
-
[11]
State of the software supply chain
“State of the software supply chain.” 2026, https://www.sonatype.com/ state-of-the-software-supply-chain/Introduction
2026
-
[12]
”tianwen
“”tianwen” review of malicious packages on pypi in q3 2023 (part 1).” 2023, https://tianwen.qianxin.com/blog/2023/10/17/PyPI-2023-Q3-1/
2023
-
[13]
”tianwen
“”tianwen” review of malicious packages on pypi in q3 2023 (part 2).” 2023, https://tianwen.qianxin.com/blog/2023/10/20/PyPI-2023-Q3-2/
2023
-
[14]
Pypi malicious package analysis: jsonconfig-utils built-in rat backdoor and multi-platform persistence
“Pypi malicious package analysis: jsonconfig-utils built-in rat backdoor and multi-platform persistence.” 2023, https://tianwen.qianxin.com/blog/ 2023/10/31/malware-nuget-halloween/
2023
-
[15]
”tianwen
“”tianwen” discovery of halloween pranks (malicious) packages in the nuget ecosystem.” 2023, https://tianwen.qianxin.com/blog/2023/10/31/ malware-nuget-halloween/
2023
-
[16]
”tianwen
“”tianwen” discovery of pypi malicious package poisoning attack tar- geting ci/cd.” 2026, https://research.qianxin.com/archives/3253
2026
-
[17]
Pypi inundated by malicious typosquatting campaign
“Pypi inundated by malicious typosquatting campaign.” 2026, https://blog.checkpoint.com/securing-the-cloud/ pypi-inundated-by-malicious-typosquatting-campaign/
2026
-
[18]
Guarddog is a cli tool to identify malicious pypi and npm packages
“Guarddog is a cli tool to identify malicious pypi and npm packages.” 2025, https://github.com/DataDog/guarddog
2025
-
[19]
Oss gadget is a collection of tools that can help analyze open source projects
“Oss gadget is a collection of tools that can help analyze open source projects.” 2023, https://github.com/microsoft/OSSGadget
2023
-
[20]
Towards measuring supply chain attacks on package managers for interpreted languages,
R. Duan, O. Alrawi, R. P. Kasturi, R. Elder, B. Saltaformaggio, and W. Lee, “Towards measuring supply chain attacks on package managers for interpreted languages,” in28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, February 21-25,
2021
-
[21]
The Internet Society, 2021
2021
-
[22]
Practical automated detection of malicious npm packages,
A. Sejfia and M. Sch ¨afer, “Practical automated detection of malicious npm packages,” inProceedings of the 44th international conference on software engineering, 2022, pp. 1681–1692
2022
-
[23]
Xgboost: A scalable tree boosting system,
T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” inProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794
2016
-
[24]
Lightgbm: A highly efficient gradient boosting decision tree,
G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.- Y . Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Advances in neural information processing systems, vol. 30, 2017
2017
-
[25]
{MalGuard}: Towards{Real-Time}, accurate, and actionable detection of malicious packages in{PyPI}ecosystem,
X. Gao, X. Sun, S. Cao, K. Huang, D. Wu, X. Liu, X. Lin, and Y . Xiang, “{MalGuard}: Towards{Real-Time}, accurate, and actionable detection of malicious packages in{PyPI}ecosystem,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 4741–4758
2025
-
[26]
An empirical study of malicious code in pypi ecosystem,
W. Guo, Z. Xu, C. Liu, C. Huang, Y . Fang, and Y . Liu, “An empirical study of malicious code in pypi ecosystem,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 166–177
2023
-
[27]
to do this bandit processes each file, builds an ast from it, and runs appropriate plugins against the ast nodes.” 2024, https://github.com/ PyCQA/bandit
“Bandit is a tool designed to find common security issues in python code. to do this bandit processes each file, builds an ast from it, and runs appropriate plugins against the ast nodes.” 2024, https://github.com/ PyCQA/bandit
2024
-
[28]
Mal- wukong: Towards fast, accurate, and multilingual detection of malicious code poisoning in oss supply chains,
N. Li, S. Wang, M. Feng, K. Wang, M. Wang, and H. Wang, “Mal- wukong: Towards fast, accurate, and multilingual detection of malicious code poisoning in oss supply chains,” in2023 38th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1993–2005
2023
-
[29]
1+ 1¿ 2: Inte- grating deep code behaviors with metadata features for malicious pypi package detection,
X. Sun, X. Gao, S. Cao, L. Bo, X. Wu, and K. Huang, “1+ 1¿ 2: Inte- grating deep code behaviors with metadata features for malicious pypi package detection,” inProceedings of the 39th IEEE/ACM international conference on automated software engineering, 2024, pp. 1159–1170
2024
-
[30]
Killing two birds with one stone: Malicious package detection in npm and pypi using a single model of malicious behavior sequence,
J. Zhang, K. Huang, Y . Huang, B. Chen, R. Wang, C. Wang, and X. Peng, “Killing two birds with one stone: Malicious package detection in npm and pypi using a single model of malicious behavior sequence,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 4, p. 1–28, 2025
2025
-
[31]
A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[32]
Kimi K2: Open Agentic Intelligence
K. Team, Y . Bai, Y . Bao, Y . Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chenet al., “Kimi k2: Open agentic intelligence,” arXiv preprint arXiv:2507.20534, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[33]
Doubao, an ai independently developed by bytedance
“Doubao, an ai independently developed by bytedance.” 2025, https: //www.doubao.com/chat/
2025
-
[34]
Glm: General language model pretraining with autoregressive blank infilling,
Z. Du, Y . Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model pretraining with autoregressive blank infilling,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 320– 335
2022
-
[35]
Maltracker: A fine-grained npm malware tracker copiloted by llm-enhanced dataset,
Z. Yu, M. Wen, X. Guo, and H. Jin, “Maltracker: A fine-grained npm malware tracker copiloted by llm-enhanced dataset,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1759–1771
2024
-
[36]
Understanding npm malicious package detection: A benchmark-driven empirical analysis,
W. Guo, Z. Chen, Z. Xu, C. Liu, M. Kang, S. Song, C. Liu, Y . Xu, W. Sun, and Y . Liu, “Understanding npm malicious package detection: A benchmark-driven empirical analysis,”arXiv preprint arXiv:2603.27549, 2026
-
[37]
A decision-theoretic generalization of on-line learning and an application to boosting,
Y . Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,”Journal of computer and system sciences, vol. 55, no. 1, pp. 119–139, 1997
1997
-
[38]
Random forests,
L. Breiman, “Random forests,”Machine learning, vol. 45, no. 1, pp. 5–32, 2001
2001
-
[39]
Leveraging large language models to detect npm malicious packages,
N. Zahan, P. Burckhardt, M. Lysenko, F. Aboukhadijeh, and L. Williams, “Leveraging large language models to detect npm malicious packages,” in2025 IEEE/ACM 47th International Conference on Software Engi- neering (ICSE). IEEE, 2025, pp. 2625–2637
2025
-
[40]
Spiderscan: Practical detection of malicious npm packages based on graph-based behavior modeling and matching,
Y . Huang, R. Wang, W. Zheng, Z. Zhou, S. Wu, S. Ke, B. Chen, S. Gao, and X. Peng, “Spiderscan: Practical detection of malicious npm packages based on graph-based behavior modeling and matching,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 1146–1158
2024
-
[41]
{DONAPI}: Malicious{NPM}packages detector using behavior sequence knowledge mapping,
C. Huang, N. Wang, Z. Wang, S. Sun, L. Li, J. Chen, Q. Zhao, J. Han, Z. Yang, and L. Shi, “{DONAPI}: Malicious{NPM}packages detector using behavior sequence knowledge mapping,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 3765–3782
2024
-
[42]
On the feasibility of cross-language detection of malicious packages in npm and pypi,
P. Ladisa, S. E. Ponta, N. Ronzoni, M. Martinez, and O. Barais, “On the feasibility of cross-language detection of malicious packages in npm and pypi,” inProceedings of the 39th annual computer security applications conference, 2023, pp. 71–82
2023
-
[43]
Exposing the hidden layer: Software repositories in the service of seo manipulation,
M. Wu, G. Hong, W. Mai, X. Wu, L. Zhang, Y . Pu, H. Chai, L. Ying, H. Duan, and M. Yang, “Exposing the hidden layer: Software repositories in the service of seo manipulation,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2025, pp. 684–684
2025
-
[44]
Understanding the response to open-source dependency abandonment in the npm ecosystem,
C. Miller, M. Jahanshahi, A. Mockus, B. Vasilescu, and C. Kastner, “Understanding the response to open-source dependency abandonment in the npm ecosystem,” 2025
2025
-
[45]
Alert: peacenotwar module sabotages npm developers in the node-ipc package to protest the invasion of ukraine
“Alert: peacenotwar module sabotages npm developers in the node-ipc package to protest the invasion of ukraine.” 2022, https://snyk.io/blog/ peacenotwar-malicious-npm-node-ipc-package-vulnerability/
2022
-
[46]
On the security of containers: Threat modeling, attack analysis, and mitigation strategies,
A. Y . Wong, E. G. Chekole, M. Ochoa, and J. Zhou, “On the security of containers: Threat modeling, attack analysis, and mitigation strategies,” Computers & Security, vol. 128, p. 103140, 2023
2023
-
[47]
What quality aspects influence the adoption of docker images?
G. Rosa, S. Scalabrino, G. Bavota, and R. Oliveto, “What quality aspects influence the adoption of docker images?”ACM Transactions on Software Engineering and Methodology, vol. 32, no. 6, pp. 1–30, 2023
2023
-
[48]
Shipwright: A human-in-the-loop system for dockerfile repair,
J. Henkel, D. Silva, L. Teixeira, M. d’Amorim, and T. Reps, “Shipwright: A human-in-the-loop system for dockerfile repair,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1148–1160
2021
-
[49]
Secrets revealed in container images: An internet-wide study on occurrence and impact,
M. Dahlmanns, C. Sander, R. Decker, K. Wehrle, J. Pennekamp, A. Belova, T. Bergs, M. Bodenbenner, A. B ¨uhrig-Polaczek, I. Kunze et al., “Secrets revealed in container images: An internet-wide study on occurrence and impact,” inACM Transactions on Internet Technology, no. 252-266. ACM, 2023, pp. 252–266
2023
-
[50]
On the relation between outdated docker containers, severity vulnerabilities, and bugs,
A. Zerouali, T. Mens, G. Robles, and J. M. Gonzalez-Barahona, “On the relation between outdated docker containers, severity vulnerabilities, and bugs,” in2019 ieee 26th international conference on software analysis, evolution and reengineering (saner). IEEE, 2019, pp. 491–501
2019
-
[51]
On the impact of outdated and vulnerable javascript pack- ages in docker images,
A. Zerouali, V . Cosentino, T. Mens, G. Robles, and J. M. Gonzalez- Barahona, “On the impact of outdated and vulnerable javascript pack- ages in docker images,” in2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2019, pp. 619–623
2019
-
[52]
Meta-maintanance for dockerfiles: Are we there yet?
T. Tanaka, H. Hata, B. Chinthanet, R. G. Kula, and K. Matsumoto, “Meta-maintanance for dockerfiles: Are we there yet?”arXiv preprint arXiv:2305.03251, 2023
-
[53]
Exploring the unchartered space of container registry typosquatting,
G. Liu, X. Gao, H. Wang, and K. Sun, “Exploring the unchartered space of container registry typosquatting,” in31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 35–51
2022
-
[54]
Understanding the security risks of docker hub,
P. Liu, S. Ji, L. Fu, K. Lu, X. Zhang, W.-H. Lee, T. Lu, W. Chen, and R. Beyah, “Understanding the security risks of docker hub,” inComputer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14–18, 2020, Proceedings, Part I 25. Springer, 2020, pp. 257–276
2020
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.