Pith. sign in

REVIEW 3 major objections 5 minor 26 references

OSPtrack: A Labeled Dataset Targeting Simulated Execution of Open-Source Software

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OSPtrack labels 1,962 malicious open-source packages from their simulated runtime behavior.

desk verdict Useful runtime-behavior dataset for OSS malware, but the benign labels are assigned by absence from two incomplete blacklists; fix that and the counts, and it's a solid resource. read the letter →

arxiv 2411.14829 v2 pith:D5UKRPVD submitted 2024-11-22 cs.CR

Zhuoran Tan , Christos Anagnosstopoulos , Jeremy Singer This is my paper · ORCID
classification cs.CR
keywords open-sourcesoftwaresupplychainmaliciouspackagedetectionruntimebehavioranalysisdynamicsandboxlabeleddatasetnpmPyPIpackage-analysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OSPtrack is a dataset built to show that malicious open-source packages leave detectable traces when they actually run, not just when their code is read. The authors simulated the execution of packages from npm, PyPI, crates.io, NuGet, and Packagist in isolated sandboxes and recorded what each package did: which files it touched, which sockets it opened, which commands it ran, and which DNS queries it made. The result is 9,461 package reports, 1,962 of them labeled malicious, with attack-type sub-labels such as command-and-control and root shell. A sympathetic reader would care because most prior datasets for supply-chain security annotate package metadata or source code, leaving runtime behavior unexplored; this dataset aims to fill that gap and to support detectors that work even when source code is unavailable.

What carries the argument

The engine is the open-source package-analysis sandbox, which takes an ecosystem, package name, and version, executes the package in an isolated environment, and emits static and dynamic observations. On top of this sandbox, the authors add a parallelized scheduler with timeout and health checks, a parsing stage that reduces each JSON or Parquet report to eight dimensions (files, sockets, commands, and DNS under both import and install), and Algorithm 1, which matches each report against the Backstabber's Knife Collection and the OpenSSF malicious-packages repository to assign binary labels and attack-type sub-labels. This pipeline converts raw execution logs into a form a classifier can train on.

What would settle it

Take a package independently confirmed malicious after the OpenSSF list snapshot used in the paper, run it through the same simulation and Algorithm 1 labeling, and check whether it is labeled benign; finding even one such package would show the benign-by-default rule mislabels unseen threats.

Watch

Extended reading notes

Core claim

The paper's central claim is that simulated execution of open-source packages, combined with label matching against two established malicious-package references, produces a labeled runtime-behavior dataset that is both reusable and methodologically sound. OSPtrack spans five ecosystems and provides eight extracted features—files, sockets, commands, and DNS records observed under both import and install—for each of 9,461 reports, of which 1,962 are malicious. The labels come in two layers: a binary benign/malicious label and a second layer giving the attack type (for example, C2 communication, root shell, or command execution). The authors argue this is the first dataset of its kind targeted specifically at open-source software and that it makes runtime indicators available for training detection models, for comparative ecosystem analysis, and for attack profiling.

Load-bearing premise

For the large benign portion of the dataset, a package is labeled benign exactly when it fails to match either reference list of known malicious packages; if a malicious package is missing from those lists, it is quietly counted as safe.

Editorial extensions

If this is right

  • Machine-learning models can be trained to distinguish benign from malicious packages using runtime traces alone, which matters when source code is unavailable or obfuscated.
  • The attack-type sub-labels allow detectors to be tuned to specific behaviors such as C2 communication or reverse shells rather than a generic malicious flag.
  • Because the same eight features are collected across five ecosystems, the dataset supports cross-ecosystem comparisons of how malware behaves.
  • The dataset is designed to be extended: the authors plan regular updates and additional simulated reports for under-represented ecosystems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The underlying assumption of a complete reference list means a malicious package absent from both BKC and the OpenSSF repository would be silently labeled benign in the large queried portion of the dataset; a careful consumer should test this false-negative rate before training on the labels.
  • Because the benign class is defined by absence of a match rather than verified cleanliness, downstream models may learn to treat 'unknown' as 'safe,' which is a different behavior from recognizing known-benign software.
  • The eight retained features are coarse—they do not capture process trees, file contents, or the order of actions—so the dataset may support detection of conspicuous malicious actions but not subtle or evasive malware.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OSPtrack, a labeled dataset of runtime behaviors of open-source packages, built with the package-analysis sandbox and the public BigQuery dataset produced by that tool. The authors collect malicious package references from the Backstabbers-Knife-Collection (BKC) and the OpenSSF malicious-packages repository, simulate a subset of those packages in a sandbox, parse the resulting JSON and Parquet reports into eight features (files, sockets, commands, DNS under import and install), and label each report as benign or malicious with optional attack-type sub-labels. The abstract reports 9,461 package reports, 1,962 malicious, spanning five ecosystems. The dataset and code are released publicly on Zenodo.

Significance. If the labels are reliable, OSPtrack would be a useful resource for training and evaluating runtime-behavior-based detectors of malicious open-source packages. Its claimed novelty is the combination of multi-ecosystem coverage with execution-level features and per-report labels, plus sub-labels for attack types. The authors also ship the extraction and labeling code, which aids reproducibility. However, the value of the dataset hinges on the correctness of the benign/malicious ground truth, and the current labeling procedure has a load-bearing completeness assumption that is not stated as a limitation. The inconsistencies in the reported counts also need to be resolved before the dataset can be used with confidence.

major comments (3)
  1. [Abstract and Table II] The abstract reports 9,461 total reports and 1,962 malicious reports, but Table II does not add up to these figures. Summing all rows in Table II gives 9,460 if the sub-label rows are counted separately, and 9,381 if the sub-label rows are subsets of the malicious rows. The malicious counts in Table II (1,882 with 'na' plus 79 sub-labeled rows) also do not match the abstract's 1,962. This inconsistency undermines the dataset's quantitative claims and must be reconciled in the table and/or the abstract.
  2. [Algorithm 1 and Section III-D] The labeling of D1 (the BigQuery-derived reports) is performed only by searching for a match in Rf; if no match is found, the label remains uninitialized and is implicitly benign. This treats the BKC and OpenSSF reference lists as complete and correct. BKC covers only npm, PyPI, and Ruby, while the OpenSSF list is a curated snapshot with inherent discovery lag. Any malicious package absent from both lists is therefore mislabeled benign. Since D1 constitutes the majority of the benign samples, this directly corrupts the negative class used to train runtime detectors. The paper should either justify the completeness assumption, add a limitation explicitly stating this risk, or use a more conservative labeling scheme (e.g., an 'unknown' class).
  3. [Introduction and Table II] The Introduction lists NuGet as one of the five ecosystems in the dataset, but Table II contains no NuGet rows and instead includes packagist (which has no malicious samples). This discrepancy must be resolved: either NuGet data exists but was omitted from the table, or the ecosystem list in the Introduction is wrong. The claim of 'five ecosystems' is also misleading if packagist contributes only benign reports and no malicious ones.
minor comments (5)
  1. [Section III-B] The text states that '4235 reports have been created, in which only 1961 is valid,' while Table II reports 1,962 malicious reports. The relationship between these numbers should be clarified.
  2. [Section VI] The Limitations section does not mention the assumption that the BKC and OpenSSF references are complete for labeling unmatched BigQuery reports as benign, which is a significant omission given the labeling method.
  3. [References] Reference [14] duplicates reference [6]; the duplicate should be removed or replaced with a distinct source.
  4. [References] There are minor typographical errors in the references: 'techiques' should be 'techniques' in reference [15], and the author initials in reference [9] appear as 'V Y' with inconsistent spacing.
  5. [Section III-D] In Algorithm 1, the initialization of L for D1 is missing, making the benign default implicit. Even if the default is intended, the pseudocode should state it explicitly, and the sub-label assignment for D1 records that do match should be described.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: labels are imported from external threat-intelligence datasets and matched by package identity, not derived from or fed back into the runtime features.

full rationale

OSPtrack is a dataset construction paper, not a derived prediction. The labeling pipeline (Algorithm 1) matches package identity (name, ecosystem, version) against the external Backstabbers-Knife-Collection [10] and OpenSSF malicious-package repository [7]; benign labels for BigQuery reports are the implicit default when no match is found. The runtime and static features (files, sockets, commands, DNS) are produced independently by the package-analysis sandbox and are not used to compute the labels, nor are labels used to construct features. The D2 simulation reports are selected from the same known-malicious sources and default to label 1; their sub-labels come from the same metric files, which is a transparency choice rather than a fitted prediction. The only author self-citation ([8]) is the dataset/code repository and is not load-bearing in any argument. The paper's main genuine weakness is external: the benign default for unmatched D1 records assumes BKC and OpenSSF are complete as of collection, so malicious packages absent from both lists are silently labeled benign. That is a ground-truth completeness/soundness limitation, noted only partially in Section VI (missing/deleted packages and timeouts, not reference incompleteness), but it does not make the derivation circular because the reference lists are independently published external data rather than outputs of this paper. No equation, parameter, or fitted quantity is recycled into the labels; there is no self-referential reduction. Score 0.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The paper does not introduce new theoretical entities or fitted parameters. Its reliance is on two domain assumptions: the fidelity of the sandbox and the correctness of the reference malicious-package lists. The two free parameters are design choices that affect the dataset's coverage and balance.

free parameters (2)
  • Simulation timeout = 10 minutes
    Manually chosen cutoff for the sandbox; packages exceeding it are excluded, which may remove slow-triggering malware and bias feature coverage.
  • BigQuery benign sample size = 7,500
    Arbitrary quota for the majority-benign portion, not justified by a formal sampling strategy. This affects class balance and may introduce selection bias.
assumptions (2)
  • domain assumption The package-analysis sandbox faithfully captures the execution behavior of the analyzed packages.
    All dynamic features are derived from the sandbox reports. If the sandbox misses anti-analysis or environment-dependent behavior, the dataset's features will be incomplete or misleading.
  • domain assumption The BKC and OpenSSF malicious package datasets are accurate and complete ground-truth references.
    Labels and sub-labels are matched against these lists. Errors or omissions in these references propagate directly into the dataset's label quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OSPtrack: A Labeled Dataset Targeting Simulated Execution of Open-Source Software." pith.science (2026). https://pith.science/paper/D5UKRPVD

@misc{pith2026241114829,
  author       = {Pith},
  title        = {Pith review of: OSPtrack: A Labeled Dataset Targeting Simulated Execution of Open-Source Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5UKRPVD}},
  note         = {Machine review of arXiv:2411.14829}
}
read the original abstract

Open-source software serves as a foundation for the internet and the cyber supply chain, but its exploitation is becoming increasingly prevalent. While advances in vulnerability detection for OSS have been significant, prior research has largely focused on static code analysis, often neglecting runtime indicators. To address this shortfall, we created a comprehensive dataset spanning five ecosystems, capturing features generated during the execution of packages and libraries in isolated environments. The dataset includes 9,461 package reports, of which 1,962 are identified as malicious, and encompasses both static and dynamic features such as files, sockets, commands, and DNS records. Each report is labeled with verified information and detailed sub-labels for attack types, facilitating the identification of malicious indicators when source code is unavailable. This dataset supports runtime detection, enhances detection model training, and enables efficient comparative analysis across ecosystems, contributing to the strengthening of supply chain security.

Figures

Figures reproduced from arXiv: 2411.14829 by the authors.

Figure 1
Figure 1. The Data Generation Framework. 1. Collect package information 1.a. Query analyzed results from BigQuery 2. Simulate packages with package [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 25 canonical work pages

  1. [1]

    Special report: Mandiant m-trends 2024,

    Mandiant, “Special report: Mandiant m-trends 2024,” 2024. [Online]. Available: https://services.google.com/fh/files/misc/m-trends-2024.pdf

    work page 2024

  2. [2]

    Towards the Detection of Malicious Java Packages,

    P. Ladisa, H. Plate, M. Martinez, O. Barais, and S. E. Ponta, “Towards the Detection of Malicious Java Packages,” in Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses . Los Angeles CA USA: ACM, 2022, pp. 63–72

    work page 2022

  3. [3]

    A source code vul- nerability detection method based on adaptive graph neural networks,

    C. Liang, Q. Wei, Z. Jiang, Y . Wang, and J. Du, “A source code vul- nerability detection method based on adaptive graph neural networks,” in Proceedings of the 39th IEEE/ACM International Conference on Au- tomated Software Engineering Workshops , ser. ASEW ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 187–196

    work page 2024

  4. [4]

    malicious-software-packages-dataset,

    D. S. Labs, “malicious-software-packages-dataset,” Mar

    work page

  5. [5]

    Backstabber’s knife collection: A review of open source software supply chain attacks,

    M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s knife collection: A review of open source software supply chain attacks,” in Detection of Intrusions and Malware, and Vulnerability Assessment: 17th International Conference, DIMVA 2020, Lisbon, Portugal, June 24–26, 2020, Proceedings . Berlin, Heidelberg: Springer-Verlag, 2020, p. 23–43

    work page 2020

  6. [7]

    Openssf malicious packages,

    O. S. S. Foundation, “Openssf malicious packages,” https://github.com/ ossf/malicious-packages/, 2024

    work page 2024

  7. [8]

    OSPtrack: A Labelled Dataset Targeting Simulated Open-Source Package Execution,

    Z. Tan, C. Anagnostopoulos, and J. Singer, “OSPtrack: A Labelled Dataset Targeting Simulated Open-Source Package Execution,” Nov

    work page

  8. [9]

    Perspectives on the solarwinds incident,

    S. Peisert, B. Schneier, H. Okhravi, F. Massacci, T. Benzel, C. Landwehr, M. Mannan, J. Mirkovic, A. Prakash, and J. B. Michael, “Perspectives on the solarwinds incident,” IEEE Security & Privacy , vol. 19, no. 2, pp. 7–13, 2021

    work page 2021

Show all 26 references
  1. [10]

    Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks,

    M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks,” 2020

    2020

  2. [11]

    Malicious Package Detection in NPM and PyPI using a Single Model of Malicious Behavior Sequence,

    J. Zhang, K. Huang, B. Chen, C. Wang, Z. Tian, and X. Peng, “Malicious Package Detection in NPM and PyPI using a Single Model of Malicious Behavior Sequence,” Sep. 2023, arXiv:2309.02637 [cs]

    2023 arXiv

  3. [12]

    An empirical study of malicious code in pypi ecosystem,

    W. Guo, Z. Xu, C. Liu, C. Huang, Y . Fang, and Y . Liu, “An empirical study of malicious code in pypi ecosystem,” 2023. [Online]. Available: https://arxiv.org/abs/2309.11021

    2023 arXiv

  4. [13]

    Securing Code with Context: Enhancing Vulnerability Detection through Contextualized Graph Representations,

    M. F. Rozi, T. Ban, S. Ozawa, A. Yamada, T. Takahashi, and D. In- oue, “Securing Code with Context: Enhancing Vulnerability Detection through Contextualized Graph Representations,” IEEE Access , pp. 1–1, 2024

    2024

  5. [14]

    DetectVul: A statement-level code vulnerability detection for Python,

    H.-C. Tran, A.-D. Tran, and K.-H. Le, “DetectVul: A statement-level code vulnerability detection for Python,” Future Generation Computer Systems, p. 107504, Sep. 2024

    2024

  6. [15]

    Ransomware detection: Leveraging sandbox, text mining techiques and machine learning,

    A. Parisot, L. M. S. Bento, and R. C. S. Machado, “Ransomware detection: Leveraging sandbox, text mining techiques and machine learning,” 2024 IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4.0 & IoT) , pp. 446–451, 2024

    2024

  7. [16]

    Dynamic feature dataset for ransomware detection using machine learning algorithms,

    J. A. H. Silva and M. H. ´Alvarez, “Dynamic feature dataset for ransomware detection using machine learning algorithms,” Sensors, vol. 23, no. 3, pp. 1053–1053, 2023

    2023

  8. [17]

    Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed,

    M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner, and A. Rauber, “Have it Your Way: Generating Customized Log Datasets With a Model-Driven Simulation Testbed,” IEEE Transactions on Reli- ability, vol. 70, no. 1, pp. 402–415, Mar. 2021

    2021

  9. [18]

    An information-theoretic perspective of tf–idf measures,

    A. Aizawa, “An information-theoretic perspective of tf–idf measures,” Information Processing & Management , vol. 39, no. 1, pp. 45–65, 2003

    2003

  10. [19]

    Poster: Toward automating the generation of malware analysis reports using the sandbox logs,

    B. Sun, A. Fujino, and T. Mori, “Poster: Toward automating the generation of malware analysis reports using the sandbox logs,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , ser. CCS ’16. New York, NY , USA: Association for Computing...

    2016

  11. [20]

    Redefining malware sandboxing: Enhancing analysis through sysmon and elk integration,

    R.-V . Mahmoud, M. Anagnostopoulos, S. Pastrana, and J. M. Pedersen, “Redefining malware sandboxing: Enhancing analysis through sysmon and elk integration,” IEEE Access , vol. 12, pp. 68 624–68 636, 2024

    2024

  12. [21]

    Benchmarking label dynamics of virustotal engines

    S. Zhu, Z. Zhang, L. Yang, L. Song, and G. Wang, “Benchmarking label dynamics of virustotal engines.” Association for Computing Machinery, 2020, pp. 2081–2083

    2020

  13. [22]

    Exorcist: Automated Differential Analysis to Detect Compromises in Closed- Source Software Supply Chains,

    F. Barr-Smith, T. Blazytko, R. Baker, and I. Martinovic, “Exorcist: Automated Differential Analysis to Detect Compromises in Closed- Source Software Supply Chains,” in Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses. Los ...

    2022

  14. [23]

    Advanced persistent threat intelligent profiling technique: A survey,

    B. Tang, J. Wang, Z. Yu, B. Chen, W. Ge, J. Yu, and T. Lu, “Advanced persistent threat intelligent profiling technique: A survey,” Computers and Electrical Engineering , vol. 103, p. 108261, 2022

    2022

  15. [24]

    Cskg4apt: A cy- bersecurity knowledge graph for advanced persistent threat organization attribution,

    Y . Ren, Y . Xiao, Y . Zhou, Z. Zhang, and Z. Tian, “Cskg4apt: A cy- bersecurity knowledge graph for advanced persistent threat organization attribution,” IEEE Transactions on Knowledge and Data Engineering , vol. 35, no. 6, pp. 5695–5709, 2023

    2023

  16. [25]

    Vul- lmgnns: Fusing language models and online-distilled graph neural net- works for code vulnerability detection,

    R. Liu, Y . Wang, H. Xu, J. Sun, F. Zhang, P. Li, and Z. Guo, “Vul- lmgnns: Fusing language models and online-distilled graph neural net- works for code vulnerability detection,” Information Fusion, p. 102748, 2024

    2024

  17. [2023]

    Available: https://github.com/datadog/ malicious-software-packages-dataset

    [Online]. Available: https://github.com/datadog/ malicious-software-packages-dataset

  18. [2024]

    Available: https://doi.org/10.5281/zenodo.14197378

    [Online]. Available: https://doi.org/10.5281/zenodo.14197378

    doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.