{"id":"bd9d8844-6abf-469a-a914-37fcd26a1d0e","arxiv_id":"2411.14829","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OSPtrack is a new labeled dataset of 9,461 runtime-behavior reports from simulated open-source package executions, of which 1,962 are malicious, spanning five ecosystems.","lead":"This paper presents OSPtrack, a labeled dataset of security-monitoring reports from simulated executions of open-source packages, covering five package ecosystems and including features like file, socket, command, and DNS activity. It aims to help researchers train systems that detect malicious software at runtime rather than only by static code review.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benign labels in the BigQuery portion are assigned by absence of a match to two incomplete malicious-package references; any malicious package not yet listed is silently mislabeled benign, corrupting ground truth.","rationale":"The reader's weakest assumption correctly identifies the benign-by-default labeling of BigQuery reports as the most load-bearing vulnerability in the paper's central claim. The dataset is presented as providing verified ground-truth labels, but the benign class is derived solely from non-membership in two finite, historically curated malicious-package lists. This is not merely a hypothetical completeness concern: the OpenSSF list explicitly tracks documented attacks up to a cutoff date, and the BKC list is limited to three ecosystems. A malicious package published, analyzed, and pulled from BigQuery before being discovered would be labeled benign. Since the stated purpose of the dataset is training detection models, such false negatives directly undermine the utility of the resource. The paper even omits this as a stated limitation in Section VI, which makes the assumption more dangerous. The proposed test is concrete and feasible: compare the released benign labels against the present-day OpenSSF and BKC lists. This is a meaningful falsification check, because any now-listed benign record is necessarily a false negative at the time of release. The test also complements the reader's other findings about count inconsistencies and the missing NuGet rows; those are errors that can be corrected, whereas the benign-labeling issue affects the validity of the dataset's core ground truth. The verdict remains conditional, matching the reader's assessment, because the dataset may still be useful if the labeling assumption is acknowledged and the benign labels are treated as 'not known malicious' rather than 'verified benign.'","tokens_in":6635,"tokens_out":4080,"duration_ms":46012,"concrete_test":"Re-query the current OpenSSF malicious-packages repository and BKC for every (ecosystem, package, version) labeled benign in OSPtrack, and count records that now appear in either list. Then manually inspect a random sample of 100 benign-labeled reports' extracted DNS, socket, command, and file features against known IOC patterns. If even one benign-labeled version now appears in a malicious reference list, or any sampled report shows a clear malicious indicator, the benign-by-default assumption has demonstrable false negatives and the dataset must be re-released with uncertain labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's central claim that every report is 'labeled with verified information' rests on Algorithm 1's first loop, which labels BigQuery-derived reports as benign unless they match a record in BKC or the OpenSSF malicious-packages list. The pseudocode never explicitly initializes L for D1, so the benign default is implicit, and Section VI lists other limitations but not this completeness assumption. This is load-bearing because both reference lists are necessarily incomplete: BKC covers only npm, PyPI, and Ruby, while the OpenSSF list is a curated snapshot of documented attacks as of November 2024, with inherent discovery lag. Any malicious package analyzed by package-analysis and pulled from BigQuery that is absent from both lists is assigned label 0. Since D1 is the 'majority of benign samples' and Table II's benign count (7,499) essentially equals the reported D1 size, unmatched reports are indeed the benign class. False benign labels corrupt the negative class exactly where the dataset promises reliable ground truth for training runtime detectors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OSPtrack, a labeled dataset of runtime behaviors of open-source packages, built with the package-analysis sandbox and the public BigQuery dataset produced by that tool. The authors collect malicious package references from the Backstabbers-Knife-Collection (BKC) and the OpenSSF malicious-packages repository, simulate a subset of those packages in a sandbox, parse the resulting JSON and Parquet reports into eight features (files, sockets, commands, DNS under import and install), and label each report as benign or malicious with optional attack-type sub-labels. The abstract reports 9,461 package reports, 1,962 malicious, spanning five ecosystems. The dataset and code are released publicly on Zenodo.","tokens_in":6773,"tokens_out":2980,"duration_ms":28720,"significance":"If the labels are reliable, OSPtrack would be a useful resource for training and evaluating runtime-behavior-based detectors of malicious open-source packages. Its claimed novelty is the combination of multi-ecosystem coverage with execution-level features and per-report labels, plus sub-labels for attack types. The authors also ship the extraction and labeling code, which aids reproducibility. However, the value of the dataset hinges on the correctness of the benign/malicious ground truth, and the current labeling procedure has a load-bearing completeness assumption that is not stated as a limitation. The inconsistencies in the reported counts also need to be resolved before the dataset can be used with confidence.","major_comments":[{"comment":"The abstract reports 9,461 total reports and 1,962 malicious reports, but Table II does not add up to these figures. Summing all rows in Table II gives 9,460 if the sub-label rows are counted separately, and 9,381 if the sub-label rows are subsets of the malicious rows. The malicious counts in Table II (1,882 with 'na' plus 79 sub-labeled rows) also do not match the abstract's 1,962. This inconsistency undermines the dataset's quantitative claims and must be reconciled in the table and/or the abstract.","section":"Abstract and Table II"},{"comment":"The labeling of D1 (the BigQuery-derived reports) is performed only by searching for a match in Rf; if no match is found, the label remains uninitialized and is implicitly benign. This treats the BKC and OpenSSF reference lists as complete and correct. BKC covers only npm, PyPI, and Ruby, while the OpenSSF list is a curated snapshot with inherent discovery lag. Any malicious package absent from both lists is therefore mislabeled benign. Since D1 constitutes the majority of the benign samples, this directly corrupts the negative class used to train runtime detectors. The paper should either justify the completeness assumption, add a limitation explicitly stating this risk, or use a more conservative labeling scheme (e.g., an 'unknown' class).","section":"Algorithm 1 and Section III-D"},{"comment":"The Introduction lists NuGet as one of the five ecosystems in the dataset, but Table II contains no NuGet rows and instead includes packagist (which has no malicious samples). This discrepancy must be resolved: either NuGet data exists but was omitted from the table, or the ecosystem list in the Introduction is wrong. The claim of 'five ecosystems' is also misleading if packagist contributes only benign reports and no malicious ones.","section":"Introduction and Table II"}],"minor_comments":[{"comment":"The text states that '4235 reports have been created, in which only 1961 is valid,' while Table II reports 1,962 malicious reports. The relationship between these numbers should be clarified.","section":"Section III-B"},{"comment":"The Limitations section does not mention the assumption that the BKC and OpenSSF references are complete for labeling unmatched BigQuery reports as benign, which is a significant omission given the labeling method.","section":"Section VI"},{"comment":"Reference [14] duplicates reference [6]; the duplicate should be removed or replaced with a distinct source.","section":"References"},{"comment":"There are minor typographical errors in the references: 'techiques' should be 'techniques' in reference [15], and the author initials in reference [9] appear as 'V Y' with inconsistent spacing.","section":"References"},{"comment":"In Algorithm 1, the initialization of L for D1 is missing, making the benign default implicit. Even if the default is intended, the pseudocode should state it explicitly, and the sub-label assignment for D1 records that do match should be described.","section":"Section III-D"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are commendable, but the label-integrity issue in Algorithm 1 is central to the dataset's value. The count inconsistencies are easy to fix but need to be addressed before publication. If the authors can clearly document the benign-default assumption and its risks, or better, provide additional validation for the benign labels, the contribution would be solid. I would not reject the paper, but the current version needs substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OSPtrack gives the community something that didn't exist before: a labeled dataset of runtime behavior for open-source packages, built by running the OSFF package-analysis sandbox and matching results against BKC and the OpenSSF malicious-packages list. The combination of dynamic features (files, sockets, commands, DNS) with attack-type sub-labels across several ecosystems is genuinely new. Prior collections are metadata-only or unlabeled, so this fills a real gap. The data and code are on Zenodo, which is the right move.\n\nThe paper is mostly straightforward about its limits: deleted packages, timeout exclusions, and partial coverage are all listed. But the stress-test concern lands squarely. Algorithm 1 labels BigQuery-derived reports as benign when they don't match BKC or OpenSSF. That default is implicit, and it assumes both references are complete and current. BKC only covers npm, PyPI, and Ruby; OpenSSF is a curated snapshot with discovery lag. Any malicious package that was analyzed by package-analysis but hasn't been documented yet is silently labeled benign — and those unmatched reports are the bulk of the negative class. The paper's claim that every report is 'labeled with verified information' is too strong for those benign labels. This doesn't kill the dataset, but it needs to be stated as a limitation, and ideally the unmatched reports should be flagged as 'unknown' or manually sampled rather than confidently labeled benign.\n\nThere are also smaller inconsistencies. The abstract says 9,461 reports, Table II sums to 9,460. The malicious count is 1,962 in the text, 1,961 from the simulation and from the table if the sub-label rows are additive. NuGet appears in the introduction but not in the ecosystem table, where RubyGems appears instead. Packagist shows only benign samples. None of these are fatal, but they need a cleanup pass.\n\nWho this is for: people training runtime-based malware detectors for OSS, or comparing malicious behavior across ecosystems. They'll get a convenient starting point and should treat the negative labels with caution until the authors address the labeling assumption. This deserves peer review — it's a real resource with fixable methodology flaws, not a desk reject. I'd send it to referees with the expectation of a revision that fixes the counts and makes the benign-label logic explicit.","headline":"Useful runtime-behavior dataset for OSS malware, but the benign labels are assigned by absence from two incomplete blacklists; fix that and the counts, and it's a solid resource.","tokens_in":7338,"tokens_out":4341,"would_cite":false,"duration_ms":40337,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OSPtrack labels 1,962 malicious open-source packages from their simulated runtime behavior.","keywords":["open-source software supply chain","malicious package detection","runtime behavior analysis","dynamic analysis sandbox","labeled dataset","npm","PyPI","package-analysis"],"falsifier":"Take a package independently confirmed malicious after the OpenSSF list snapshot used in the paper, run it through the same simulation and Algorithm 1 labeling, and check whether it is labeled benign; finding even one such package would show the benign-by-default rule mislabels unseen threats.","tokens_in":6389,"feed_emoji":"🛡️","tokens_out":4683,"duration_ms":45277,"temperature":0.7,"pith_summary":"OSPtrack is a dataset built to show that malicious open-source packages leave detectable traces when they actually run, not just when their code is read. The authors simulated the execution of packages from npm, PyPI, crates.io, NuGet, and Packagist in isolated sandboxes and recorded what each package did: which files it touched, which sockets it opened, which commands it ran, and which DNS queries it made. The result is 9,461 package reports, 1,962 of them labeled malicious, with attack-type sub-labels such as command-and-control and root shell. A sympathetic reader would care because most prior datasets for supply-chain security annotate package metadata or source code, leaving runtime behavior unexplored; this dataset aims to fill that gap and to support detectors that work even when source code is unavailable.","feed_headline":"Simulated runs reveal 1,962 malicious open-source packages","feed_subtitle":"A labeled dataset of runtime traces across five ecosystems aims to catch attacks that static code review misses.","key_machinery":"The engine is the open-source package-analysis sandbox, which takes an ecosystem, package name, and version, executes the package in an isolated environment, and emits static and dynamic observations. On top of this sandbox, the authors add a parallelized scheduler with timeout and health checks, a parsing stage that reduces each JSON or Parquet report to eight dimensions (files, sockets, commands, and DNS under both import and install), and Algorithm 1, which matches each report against the Backstabber's Knife Collection and the OpenSSF malicious-packages repository to assign binary labels and attack-type sub-labels. This pipeline converts raw execution logs into a form a classifier can train on.","core_discovery":"The paper's central claim is that simulated execution of open-source packages, combined with label matching against two established malicious-package references, produces a labeled runtime-behavior dataset that is both reusable and methodologically sound. OSPtrack spans five ecosystems and provides eight extracted features—files, sockets, commands, and DNS records observed under both import and install—for each of 9,461 reports, of which 1,962 are malicious. The labels come in two layers: a binary benign/malicious label and a second layer giving the attack type (for example, C2 communication, root shell, or command execution). The authors argue this is the first dataset of its kind targeted specifically at open-source software and that it makes runtime indicators available for training detection models, for comparative ecosystem analysis, and for attack profiling.","pith_inferences":["The underlying assumption of a complete reference list means a malicious package absent from both BKC and the OpenSSF repository would be silently labeled benign in the large queried portion of the dataset; a careful consumer should test this false-negative rate before training on the labels.","Because the benign class is defined by absence of a match rather than verified cleanliness, downstream models may learn to treat 'unknown' as 'safe,' which is a different behavior from recognizing known-benign software.","The eight retained features are coarse—they do not capture process trees, file contents, or the order of actions—so the dataset may support detection of conspicuous malicious actions but not subtle or evasive malware."],"forward_implications":["Machine-learning models can be trained to distinguish benign from malicious packages using runtime traces alone, which matters when source code is unavailable or obfuscated.","The attack-type sub-labels allow detectors to be tuned to specific behaviors such as C2 communication or reverse shells rather than a generic malicious flag.","Because the same eight features are collected across five ecosystems, the dataset supports cross-ecosystem comparisons of how malware behaves.","The dataset is designed to be extended: the authors plan regular updates and additional simulated reports for under-represented ecosystems."],"supporting_citations":[{"why":"The OpenSSF malicious-packages repository supplies the majority of known malicious package records and the metadata used to match labels and sub-labels.","marker":"[7]"},{"why":"The Backstabber's Knife Collection supplies the second malicious-package reference and the attack-type methodology used to assign sub-labels.","marker":"[10]"},{"why":"This categorization of install-time, import-time, and run-time trigger scenarios guides which package behaviors the simulation and feature extraction should capture.","marker":"[11]"}],"fun_headline_variants":["Runtime dataset flags 1,962 malicious open-source packages","Simulated runs uncover 1,962 malicious packages in OSS","OSPtrack: executing OSS to catch 1,962 malicious packages","Five ecosystems, 9,461 reports, 1,962 malicious packages","Dynamic traces of open-source packages reveal threats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the large benign portion of the dataset, a package is labeled benign exactly when it fails to match either reference list of known malicious packages; if a malicious package is missing from those lists, it is quietly counted as safe.","fun_headline_variants_meta":{"raw":{"variants":["Runtime dataset flags 1,962 malicious open-source packages","Simulated runs uncover 1,962 malicious packages in OSS","OSPtrack: executing OSS to catch 1,962 malicious packages","Five ecosystems, 9,461 reports, 1,962 malicious packages","Dynamic traces of open-source packages reveal threats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1318,"prompt_tokens":851,"completion_tokens":467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":467,"tokens_out":467,"duration_ms":5004,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:49:20.295648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a package independently confirmed malicious after the OpenSSF list snapshot used in the paper, run it through the same simulation and Algorithm 1 labeling, and check whether it is labeled benign; finding even one such package would show the benign-by-default rule mislabels unseen threats.","supporting_citations":[{"cited_title":"Openssf malicious packages,","cited_arxiv_id":null,"evidence_quote":"The OpenSSF malicious-packages repository supplies the majority of known malicious package records and the metadata used to match labels and sub-labels."},{"cited_title":"Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks,","cited_arxiv_id":null,"evidence_quote":"The Backstabber's Knife Collection supplies the second malicious-package reference and the attack-type methodology used to assign sub-labels."}],"review_version":1}