Pith. sign in

REVIEW 3 major objections 4 minor 28 references

This paper builds a public dataset of dependency resolution at historical release points for npm, PyPI, and crates.io packages, enriched with vulnerability data, so that dynamic metrics and longitudinal security studies become possible.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:07 UTC pith:HPAS7X2K

load-bearing objection Useful cross-ecosystem release-point dependency dataset, but PyPI/crates.io resolution rests on an un-auditable deps.dev black box and the validation is only a smoke test. the 3 major comments →

arxiv 2607.15315 v1 pith:HPAS7X2K submitted 2026-07-15 cs.SE cs.CR

CHRONO-RESOLUTION: A Dependency Resolution Dataset at Release Points for npm, PyPI, and crates.io Packages

Imranur Rahman , Jill Marley , Ranindya Paramitha , Laurie Williams This is my paper
classification cs.SE cs.CR
keywords software ecosystemsdependency resolutionrelease pointsnpmPyPIcrates.iosoftware supply chain securitydynamic metrics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to solve a missing capability: major package ecosystems do not let you ask what dependencies a package would have pulled at a specific time in the past, which blocks dynamic metrics like dependency freshness and update speed. The authors propose a methodology that slices each package–dependency relationship into time intervals bounded by release events, resolves the dependency version at each interval start, and enriches the result with vulnerability ranges. They apply this to the largest, oldest, and newest major ecosystems and release the resulting dataset (210,949 packages, millions of relationships, thousands of advisories). If the retroactive resolution is faithful, researchers can for the first time study ecosystem evolution and security exposure at release-point granularity across ecosystems.

Core claim

The central claim is that a static snapshot of package metadata can be turned into a temporal dataset of dependency resolution at release points by cutting time into intervals between release events and asking a modern resolver what it would install at the start of each interval. The authors add a Boolean 'remediated' flag by joining resolved versions against vulnerability range data, treating a vulnerable version as unremediated only when a fixed version already existed at that time. The released dataset covers 146,651 npm, 48,608 PyPI, and 15,690 crates.io packages, with 2.2M, 0.36M, and 0.16M unique direct relationships and 6,948 advisories, and is intended to be the reference for dynamic

What carries the argument

The core device is the release-point interval: every package–dependency pair is divided into contiguous intervals whose boundaries are the release times of either the package or the dependency, and the dependency version is resolved at the interval start. This converts an atemporal constraint (for example, '>=1.0.0') into a concrete, time-stamped version that a developer would have installed then, making dynamic metrics such as freshness, time-to-update, and time-to-remediate computable from the data alone.

Load-bearing premise

The dataset assumes that a modern resolver's retroactive answer at each release point equals what a developer would actually have installed then; PyPI data before late 2020 explicitly may not match historical behavior, and the manual check of 50 packages per ecosystem is a smoke test, not statistical evidence.

What would settle it

Run period-accurate package-manager binaries—npm with its time-travel flag and historical pip and Cargo versions—on a random sample of release points, and compare their resolved dependency versions against the rows in CHRONO-RESOLUTION; a non-negligible mismatch, especially for pre-2020 PyPI, would refute the historical-accuracy assumption.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dynamic metrics such as dependency freshness, update rhythm, mean-time-to-update, and mean-time-to-remediate can now be computed at release-point granularity for the three ecosystems.
  • Researchers can track, version by version, which dependencies enter or leave common use, enabling ecosystem evolution studies that static snapshots cannot support.
  • Supply-chain analyses can measure n-day exposure and distinguish fixable from unfixable vulnerabilities through the newly added fix-available annotation.
  • The released pipeline lets others rebuild or extend the dataset as APIs evolve or as new ecosystems are added.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the retroactive resolution holds up against period-accurate package managers, this dataset could become a standard benchmark for dynamic dependency metrics, effectively giving every major ecosystem the time-travel feature npm has natively.
  • The interval-splitting methodology generalizes beyond the three ecosystems; any registry with version metadata and a resolution service could be added without changing the core design.
  • The pre-2020 PyPI caveat is a real boundary: longitudinal studies across that cutoff should either restrict to post-2020 data or treat earlier rows as reconstructions, and a period-accurate pip reproduction would be a natural validation target.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents CHRONO-RESOLUTION, a dataset of dependency resolution at release points for npm, PyPI, and crates.io packages, enriched with vulnerability data from OSV. The methodology collects package metadata from the three registries, applies inclusion criteria (age, activity, at least one dependency), uses deps.dev to resolve dependency graphs at historical release points, and post-processes the result into interval-based RELATIONS and OSV-EXTENDED tables. The dataset comprises roughly 146k npm, 49k PyPI, and 16k crates.io packages, with direct-dependency relationships and vulnerability annotations. The authors also describe two prior studies that used this dataset to compute dependency-update and remediation metrics.

Significance. If the central claim is accepted, this is the first publicly released cross-ecosystem dataset offering dependency resolution at historical release points, filling a gap for dynamic metrics such as MTTU, MTTR, and n-day exposure. The paper's strengths include: a large, multi-ecosystem collection; integration of OSV vulnerability data with a fix-available flag; release of raw and filtered variants; and provision of construction scripts and usage examples for reproducibility. The dataset has clear potential value to the empirical software engineering and software supply chain security communities, and the paper is unusually transparent about its own limitations.

major comments (3)
  1. [VI (Construct Validity: Retroactive SemVer Resolution for PyPI)] The paper's title and abstract claim "dependency resolution at release points" without qualification. Yet Section VI states that for PyPI packages released before pip 20.3 (November 2020), "resolved versions represent what the modern resolver would have installed, which may differ from what developers actually experienced." Since PyPI dates from 2003 and a large fraction of its history is pre-2020, this concession directly undermines the claim that the dataset provides actual historical dependency states for these packages. The manuscript should either narrow the central claim (e.g., to "modern-resolver reconstructions at release points") or restrict the PyPI portion to post-2020 data, and the abstract/title should reflect this limitation.
  2. [VI (Validation Sample Size) and III-A3] The 50-package-per-ecosystem manual validation is explicitly called "a smoke test... rather than statistical evidence" and "cannot detect retroactive resolution errors." Because deps.dev provided the resolutions via special access and the dataset cannot be rebuilt via the public API (III-A3), the correctness of the key attribute—historical release-point resolution—is not independently verified for any ecosystem. This is a load-bearing gap: the dataset's unique value is precisely the historical resolution, and the provided validation only checks consistency with modern resolvers and live registries. The authors should add a stratified, statistically defensible validation using period-accurate tools (e.g., npm --before, historical pip binaries, old cargo) and report per-ecosystem agreement rates.
  3. [III-A2 and Table IV] There is an internal inconsistency in the reported package counts. Section III-A2 says the filtered dataset contains "163,207 (146,651 npm, 48,608 PyPI, and 15,690 crates.io) packages," but these numbers sum to 210,949. The abstract and Table IV also report the same component counts. The discrepancy likely stems from a typo in the total, but for a dataset paper, a correct package count is essential. Please reconcile the total and verify all aggregate statistics.
minor comments (4)
  1. [Title and Abstract] The package ecosystem names appear without proper spacing: "fornpm,PyPI, andcrates.iopackages" should be "for npm, PyPI, and crates.io packages."
  2. [III-A1] The statement "npm is the largest (with 3.67 million packages compared to Maven's 715k)" cites no source for the Maven number; consider adding a reference or clarifying that the comparison is illustrative.
  3. [III-C2] The definition of 'remediated = true' when no fixed version was available at interval start is a design choice, but it could be surprising to readers. A brief justification or pointer to the fix-available flag would improve clarity.
  4. [VI] The paper recommends limiting PyPI analyses to post-2020 data but does not state what fraction of the released PyPI rows corresponds to pre-2020 intervals. Reporting this fraction would help users decide whether to subset the data.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical dataset construction with candidly stated validation limits, not a derivation whose result equals its input.

full rationale

This paper constructs and releases an empirical dataset; it does not derive a prediction or first-principles result, so the circularity patterns do not apply. Dependency resolutions are obtained from the external deps.dev service (Section III-A3), and vulnerability data from OSV (Section III-B); neither is defined in terms of the authors' own claims, and no fitted parameter is renamed as a prediction. The self-citations to [22] and [25] describe earlier internal snapshots and metrics, and the paper explicitly states those snapshots were not public and that this release extends them (Section II, 'What is new in this release'), so the prior work is contextual, not load-bearing for the dataset's construction. The closest validation concern is the Section VI admission that the manual validation 'used live registry API and modern resolver behavior from deps.dev, so it cannot detect retroactive resolution errors,' and that PyPI pre-2020 data represents modern-resolver reconstruction rather than historical developer experience. This is a candid, explicitly stated data-provenance limitation, not a circular derivation: the paper does not use the validation as a derivation step, does not claim the validation proves historical accuracy beyond a 'smoke test,' and provides independent external anchors (live registry APIs, OSV advisories, SemVer checks) for the metadata components. Therefore there is no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The ledger is small because this is a dataset artifact rather than a derivation. The load-bearing items are external data sources treated as ground truth (deps.dev release-time resolution, OSV advisories), a SemVer ordering assumption, and design thresholds. No new physical or conceptual entities are introduced.

free parameters (2)
  • package age and activity thresholds = 2 years
    Inclusion requires package age >= 2 years, a release within the last 2 years, and at least one dependency (Section III-A2). Design choice, not fitted to data; raw unfiltered data also released.
  • manual validation sample = 50 packages per ecosystem (150 total)
    The pipeline was validated by hand on 150 packages; authors call this a smoke test rather than statistical evidence (Section VI).
axioms (4)
  • domain assumption Semantic Versioning (SemVer) provides a valid total order for version comparison and constraint resolution across npm, PyPI, and crates.io; non-conforming versions are dropped (Section III-C1).
    All updated/remediated flags and interval assignments depend on SemVer ordering; dropping non-SemVer rows could bias the dataset against packages using irregular versioning.
  • domain assumption deps.dev's special-access dependency resolution at release times is treated as ground truth for what developers would have installed (Section III-A3).
    PyPI/crates.io do not provide time-travel resolution; the authors obtained resolutions from deps.dev under special access, and their own manual validation cannot verify retroactive accuracy.
  • domain assumption OSV advisories collected 2024-09-12 capture the vulnerability ranges relevant to the included packages (Section III-B).
    OSV aggregates several feeds, but any vulnerability not in those feeds is absent; later disclosures are excluded by the cutoff date.
  • domain assumption No versions of the package or its dependency change during a defined time interval, so the resolved version at interval start persists until interval end (Section III-A3).
    This partitions release points into intervals; if the resolution process missed a release, the interval boundaries and 'updated' flags would be misassigned.

pith-pipeline@v1.3.0-alltime-deepseek · 8024 in / 16283 out tokens · 138323 ms · 2026-08-02T02:07:57.996541+00:00 · methodology

0 comments
read the original abstract

Dependency resolution at a specified point in time in the past can provide insight into software evolution in software ecosystems and facilitate the design of dynamic metrics (e.g., dependency freshness, dependency update rhythm). However, dependency resolution at specified points in time is not possible in major software ecosystems due to a lack of support from package management tools. The goal of this paper is to aid practitioners and researchers in analyzing the state of the ecosystem dependency graph at release points using an enriched dataset with dependency resolution at release points for npm, PyPI, and crates.io packages. In this paper, we present a methodology to construct dependency resolution at release points of packages in software ecosystems, which we enrich with vulnerability data from OSV. We apply our methodology to construct CHRONO-RESOLUTION, a dataset of dependency resolution at release points for npm, PyPI, and crates.io packages, and release it for future research.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 1 canonical work pages

  1. [1]

    https://github.com/advisories

    GitHub Advisory Database. https://github.com/advisories. Last accessed: July-2026

    2026

  2. [2]

    https://deps.dev/

    Open Source Insights: Understand your dependencies. https://deps.dev/. Last accessed: July-2026

    2026

  3. [3]

    https://ecosyste.ms/

    Open source intelligence for your project. https://ecosyste.ms/. Last accessed: July-2026

    2026

  4. [4]

    https: //osv.dev

    OSV .dev : A distributed vulnerability database for open source. https: //osv.dev. Last accessed: July-2026

    2026

  5. [5]

    https://semver.org/

    Semantic Versioning 2.0. https://semver.org/. Last accessed: July-2026

    2026

  6. [6]

    Why do developers use trivial packages? an empirical case study on npm

    ABDALKAREEM, R., NOURRY, O., WEHAIBI, S., MUJAHID, S.,AND SHIHAB, E. Why do developers use trivial packages? an empirical case study on npm. InProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering(New York, NY , USA, Aug. 2017), ESEC/FSE 2017, Association for Computing Machinery, pp. 385–395

    2017

  7. [7]

    R., RIETVELD, K.,ANDGADYATSKAYA, O

    AKHOUNDALI, J., NOURI, S. R., RIETVELD, K.,ANDGADYATSKAYA, O. MoreFixes: A Large-Scale Dataset of CVE Fix Commits Mined through Enhanced Repository Discovery. InProceedings of the 20th International Conference on Predictive Models and Data Analytics in Software Engineering(New York, NY , USA, July 2024), PROMISE 2024, Association for Computing Machinery, ...

    2024

  8. [8]

    COELHO, J.,ANDVALENTE, M. T. Why modern open source projects fail. InProceedings of the 2017 11th Joint meeting on foundations of software engineering(2017), pp. 186–196

    2017

  9. [9]

    Measur- ing Dependency Freshness in Software Systems

    COX, J., BOUWERS, E.,VANEEKELEN, M.,ANDVISSER, J. Measur- ing Dependency Freshness in Software Systems. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering(May 2015), vol. 2, pp. 109–118. ISSN: 1558-1225

    2015

  10. [10]

    A.,ANDKHOLOOSI, M

    CROFT, R., BABAR, M. A.,ANDKHOLOOSI, M. M. Data quality for software vulnerability datasets. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)(2023), pp. 121–133

    2023

  11. [11]

    OSV: Current data sources

    GOOGLE. OSV: Current data sources. https://google.github.io/osv.dev /data/#current-data-sources. Last accessed: July-2026

    2026

  12. [12]

    Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attacks

    HE, H., VASILESCU, B.,ANDK ¨ASTNER, C. Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attacks. InProceedings of the ACM on Software Engineering, Volume 2, Number FSE, Article FSE013 (July 2025)(Feb. 2025). arXiv:2502.06662 [cs]

    Pith/arXiv arXiv 2025

  13. [13]

    Empirical Analysis of Vulnerabilities Life Cycle in Golang Ecosystem

    HU, J., ZHANG, L., LIU, C., YANG, S., HUANG, S.,ANDLIU, Y. Empirical Analysis of Vulnerabilities Life Cycle in Golang Ecosystem. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(New York, NY , USA, Apr. 2024), ICSE ’24, Association for Computing Machinery, pp. 1–13

    2024

  14. [14]

    A Preliminary Study of Rhythm and Speed in the Maven Ecosystem

    JAIME, D., ELHADDAD, J.,ANDPOIZAT, P. A Preliminary Study of Rhythm and Speed in the Maven Ecosystem. In21st Belgium- Netherlands Software Evolution Workshop(Mons, Belgium, Sept. 2022)

    2022

  15. [15]

    E.,ANDPOIZAT, P

    JAIME, D., HADDAD, J. E.,ANDPOIZAT, P. Goblin: A Framework for Enriching and Querying the Maven Central Dependency Graph. In Proceedings of the 21st International Conference on Mining Software Repositories(New York, NY , USA, July 2024), MSR ’24, Association for Computing Machinery, pp. 37–41

    2024

  16. [16]

    Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java

    LI, K., CHEN, S., FAN, L., FENG, R., LIU, H., LIU, C., LIU, Y.,AND CHEN, Y. Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(New York, NY , USA, Nov. 2023), ESEC/FSE 2023, Association ...

    2023

  17. [17]

    Demystifying the Vulnerability Propagation and Its Evolution via De- pendency Trees in the NPM Ecosystem

    LIU, C., CHEN, S., FAN, L., CHEN, B., LIU, Y.,ANDPENG, X. Demystifying the Vulnerability Propagation and Its Evolution via De- pendency Trees in the NPM Ecosystem. InProceedings of the 44th International Conference on Software Engineering(May 2022), pp. 672–

    2022

  18. [18]

    World of code: Enabling a research workflow for mining and analyzing the universe of open source vcs data.Empirical Software Engineering 26, 2 (2021), 1–42

    MA, Y., DEY, T., BOGART, C., AMREEN, S., VALIEV, M., TUTKO, A., KENNARD, D., ZARETZKI, R.,ANDMOCKUS, A. World of code: Enabling a research workflow for mining and analyzing the universe of open source vcs data.Empirical Software Engineering 26, 2 (2021), 1–42

    2021

  19. [19]

    Understanding the Response to Open-Source Dependency Abandonment in the npm Ecosystem

    MILLER, C., JAHANSHAHI, M., MOCKUS, A., VASILESCU, B.,AND KASTNER, C. Understanding the Response to Open-Source Dependency Abandonment in the npm Ecosystem. InInternational Conference on Software Engineering(2025)

    2025

  20. [20]

    A Large Scale Analysis of Semantic Versioning in NPM

    PINCKNEY, D., CASSANO, F., GUHA, A.,ANDBELL, J. A Large Scale Analysis of Semantic Versioning in NPM. InProceedings of the 20th International Conference on Mining Software Repositories(2023)

    2023

  21. [21]

    npm-follower: A Complete Dataset Tracking the NPM Ecosystem

    PINCKNEY, D., CASSANO, F., GUHA, A.,ANDBELL, J. npm-follower: A Complete Dataset Tracking the NPM Ecosystem. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(San Francisco CA USA, Nov. 2023), ACM, pp. 2132–2136

    2023

  22. [22]

    RAHMAN, I., MARLEY, J., ENCK, W.,ANDWILLIAMS, L. Which is better for reducing outdated and vulnerable dependencies: Pinning or floating? In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)(2025), pp. 2795–2807

    2025

  23. [23]

    CHRONO-RESOLUTION: A dependency resolution dataset at release points for npm, pypi, and crates.io packages

    RAHMAN, I., MARLEY, J., PARAMITHA, R.,ANDWILLIAMS, L. CHRONO-RESOLUTION: A dependency resolution dataset at release points for npm, pypi, and crates.io packages. https://doi.org/10.5281/zenodo.17576373

    work page doi:10.5281/zenodo.17576373

  24. [24]

    What’s in a package? getting visibility into dependencies using security-sensitive api calls, 2025

    RAHMAN, I., PARAMITHA, R., PLATE, H., WERMKE, D.,AND WILLIAMS, L. What’s in a package? getting visibility into dependencies using security-sensitive api calls, 2025

    2025

  25. [25]

    How quickly do development teams update their vulnerable dependencies?, 2026

    RAHMAN, I., PARAMITHA, R., ZAHAN, N., ENCK, W.,AND WILLIAMS, L. How quickly do development teams update their vulnerable dependencies?, 2026. [26]THEORY. A postgres data type for the semantic version format with support for btree and hash indexing. https://pgxn.org/dist/semver/. Last accessed: July-2026

    2026

  26. [27]

    Libraries.io – The Open Source Discovery Service

    TIDELIFT, I. Libraries.io – The Open Source Discovery Service. https: //libraries.io/. Last accessed: July-2026

    2026

  27. [28]

    Research directions in software supply chain security.ACM Trans

    WILLIAMS, L., BENEDETTI, G., HAMER, S., PARAMITHA, R., RAH- MAN, I., TAMANNA, M., TYSTAHL, G., ZAHAN, N., MORRISON, P., ACAR, Y., CUKIER, M., K ¨ASTNER, C., KAPRAVELOS, A., WERMKE, D.,ANDENCK, W. Research directions in software supply chain security.ACM Trans. Softw. Eng. Methodol. 34, 5 (May 2025)

    2025

  28. [684]

    arXiv:2201.03981 [cs]

    Pith/arXiv arXiv