{"id":"3a21d04d-2c62-4ad8-84f9-9fc42bce99e1","arxiv_id":"2607.15315","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CHRONO-RESOLUTION provides per-release resolved dependency versions and OSV vulnerability annotations for active npm, PyPI, and crates.io packages.","lead":"The authors built and released a large dataset showing which dependency versions npm, PyPI, and crates.io packages would use at each historical release, joined with vulnerability advisories. It lets researchers compute dynamic supply-chain metrics like dependency freshness and update speed that static data cannot support.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deps.dev retroactive resolution is not validated against period-accurate package managers; paper admits pre-2020 PyPI differs and validation is a smoke test.","rationale":"The reader's weakest_assumption identifies exactly the same concern: deps.dev's retroactive resolution at release times may not equal what a developer would have installed then. This is the most load-bearing point because the dataset's entire novelty and utility rest on the accuracy of this reconstruction. The paper itself provides strong evidence that the assumption fails for a significant portion of PyPI data (pre-2020) and admits that the validation cannot detect retroactive errors. The manual validation, covering 150 packages total, is too small to substantiate the 163k-package dataset. An internal arithmetic inconsistency (163,207 vs. 210,949 packages) is a secondary red flag but does not directly threaten the resolution correctness; I do not elevate it to the primary concern. The proposed concrete test—comparing against period-accurate package manager binaries—would settle whether the concern lands. Given that the reader already assigned CONDITIONAL based on this same weakness, my analysis supports that verdict without change. I see no reason to strengthen to REJECT because the paper clearly scopes its limitations and provides a released artifact that could still be useful if the historical accuracy is confirmed or the dataset is relabeled as a modern-resolver reconstruction.","tokens_in":8470,"tokens_out":3987,"duration_ms":36428,"concrete_test":"Select a stratified random sample of, say, 100 package-release points per ecosystem, balanced across time periods (pre-2010, 2010-2015, 2015-2020, post-2020). For each sample, reconstruct the actual dependency resolution using period-accurate package manager binaries: pip versions corresponding to the release date (e.g., pip 19.x for 2019) in an isolated container with a snapshot of available package versions, npm's `--before` argument where applicable (noting its own uncertainties), and a historical crates.io index for cargo. Compare the resolved dependency versions and highest available versions against the dataset's columns. If the mismatch rate is above a small threshold (e.g., >5%) or systematic for any period, the dataset does not accurately represent historical resolution for that period, and the central claim must be scoped accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the dataset provides 'dependency resolution at release points' for npm, PyPI, and crates.io. This claim depends entirely on deps.dev's retroactive resolution, which the paper cannot independently audit. Section III-A3 states that PyPI and crates.io resolutions were provided by the deps.dev team via special access and that the dataset cannot be rebuilt via the public API. Section VI concedes that for PyPI versions released before pip 20.3 (November 2020), the resolved versions represent what a modern resolver would install, not what developers would have installed at the time. The manual validation of 50 packages per ecosystem is explicitly described as 'a smoke test... rather than statistical evidence' and uses live registry APIs and modern resolver behavior, so it cannot detect retroactive resolution errors even for npm and crates.io. Thus, for the large temporal portion of PyPI that predates 2020, and potentially for any historical release in any ecosystem, the released data are modern-resolver reconstructions rather than verified historical resolutions. This undermines the dataset's core value proposition for dynamic metrics like MTTU and MTTR, which require knowing the actual dependency state at release points.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CHRONO-RESOLUTION, a dataset of dependency resolution at release points for npm, PyPI, and crates.io packages, enriched with vulnerability data from OSV. The methodology collects package metadata from the three registries, applies inclusion criteria (age, activity, at least one dependency), uses deps.dev to resolve dependency graphs at historical release points, and post-processes the result into interval-based RELATIONS and OSV-EXTENDED tables. The dataset comprises roughly 146k npm, 49k PyPI, and 16k crates.io packages, with direct-dependency relationships and vulnerability annotations. The authors also describe two prior studies that used this dataset to compute dependency-update and remediation metrics.","tokens_in":8649,"tokens_out":3447,"duration_ms":33985,"significance":"If the central claim is accepted, this is the first publicly released cross-ecosystem dataset offering dependency resolution at historical release points, filling a gap for dynamic metrics such as MTTU, MTTR, and n-day exposure. The paper's strengths include: a large, multi-ecosystem collection; integration of OSV vulnerability data with a fix-available flag; release of raw and filtered variants; and provision of construction scripts and usage examples for reproducibility. The dataset has clear potential value to the empirical software engineering and software supply chain security communities, and the paper is unusually transparent about its own limitations.","major_comments":[{"comment":"The paper's title and abstract claim \"dependency resolution at release points\" without qualification. Yet Section VI states that for PyPI packages released before pip 20.3 (November 2020), \"resolved versions represent what the modern resolver would have installed, which may differ from what developers actually experienced.\" Since PyPI dates from 2003 and a large fraction of its history is pre-2020, this concession directly undermines the claim that the dataset provides actual historical dependency states for these packages. The manuscript should either narrow the central claim (e.g., to \"modern-resolver reconstructions at release points\") or restrict the PyPI portion to post-2020 data, and the abstract/title should reflect this limitation.","section":"VI (Construct Validity: Retroactive SemVer Resolution for PyPI)"},{"comment":"The 50-package-per-ecosystem manual validation is explicitly called \"a smoke test... rather than statistical evidence\" and \"cannot detect retroactive resolution errors.\" Because deps.dev provided the resolutions via special access and the dataset cannot be rebuilt via the public API (III-A3), the correctness of the key attribute—historical release-point resolution—is not independently verified for any ecosystem. This is a load-bearing gap: the dataset's unique value is precisely the historical resolution, and the provided validation only checks consistency with modern resolvers and live registries. The authors should add a stratified, statistically defensible validation using period-accurate tools (e.g., npm --before, historical pip binaries, old cargo) and report per-ecosystem agreement rates.","section":"VI (Validation Sample Size) and III-A3"},{"comment":"There is an internal inconsistency in the reported package counts. Section III-A2 says the filtered dataset contains \"163,207 (146,651 npm, 48,608 PyPI, and 15,690 crates.io) packages,\" but these numbers sum to 210,949. The abstract and Table IV also report the same component counts. The discrepancy likely stems from a typo in the total, but for a dataset paper, a correct package count is essential. Please reconcile the total and verify all aggregate statistics.","section":"III-A2 and Table IV"}],"minor_comments":[{"comment":"The package ecosystem names appear without proper spacing: \"fornpm,PyPI, andcrates.iopackages\" should be \"for npm, PyPI, and crates.io packages.\"","section":"Title and Abstract"},{"comment":"The statement \"npm is the largest (with 3.67 million packages compared to Maven's 715k)\" cites no source for the Maven number; consider adding a reference or clarifying that the comparison is illustrative.","section":"III-A1"},{"comment":"The definition of 'remediated = true' when no fixed version was available at interval start is a design choice, but it could be surprising to readers. A brief justification or pointer to the fix-available flag would improve clarity.","section":"III-C2"},{"comment":"The paper recommends limiting PyPI analyses to post-2020 data but does not state what fraction of the released PyPI rows corresponds to pre-2020 intervals. Reporting this fraction would help users decide whether to subset the data.","section":"VI"}],"recommendation":"major_revision","confidential_remarks":"The dataset is potentially valuable and the authors are transparent about limitations, but the central historical-resolution claim is overstated relative to the evidence presented. The manuscript should be revised to scope the claims, strengthen validation (or explicitly label the data as reconstructions), and fix the package-count error. These are fixable within the manuscript's scope without abandoning the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: the genuinely new artifact here is a public dataset that gives dependency resolution at release points for npm, PyPI, and crates.io, enriched with OSV advisories including unfixed ones. The npm part is incremental over npm-follower and the --before flag, but extending release-point resolution to PyPI and crates.io at this scale is new, and the OSV fix-available flag is a nice addition.\n\nWhat the paper does well: the construction pipeline is clearly described, with release-point intervals, constraint types, and updated/remediated flags. The raw and filtered variants are both released, so researchers with different inclusion criteria can work from the full collection. The Limitations section is unusually honest: it explicitly calls the validation a smoke test, admits pre-2020 PyPI rows are modern-resolver reconstructions, and states the dataset cannot be rebuilt via the public deps.dev API.\n\nThe soft spots are real but not fatal. The load-bearing assumption is that deps.dev's retroactive resolution equals what a developer would have installed at that time. For PyPI and crates.io, this cannot be independently audited, and the paper's own 50-package-per-ecosystem validation uses live APIs and modern resolver behavior, so it cannot detect retroactive errors. The PyPI pre-pip-20.3 caveat is acknowledged, but it means a large temporal portion of PyPI is reconstruction, which limits the historical-accuracy claim for dynamic metrics like MTTU and MTTR. There is also an arithmetic inconsistency: 146,651 + 48,608 + 15,690 = 210,949, not the stated 163,207, and the filtering percentages don't quite add up either. Minor, but sloppy in a dataset paper. The reproducibility limitation is more substantial: the dataset's core value depends on resolution outputs from a special-access deps.dev arrangement.\n\nWho this is for: researchers studying ecosystem evolution, dependency freshness, update rhythm, or vulnerability exposure over time, particularly if they want cross-ecosystem coverage. They will get value from the dataset and the methodology, provided they read the caveats. The paper deserves a serious referee. My recommendation: accept the dataset with major revisions, primarily to segment pre-2020 PyPI rows, fix the arithmetic, and push the authors to find a way to publish or independently validate the deps.dev resolution outputs. The core idea is sound; the execution is honest and the artifact is useful.","headline":"Useful cross-ecosystem release-point dependency dataset, but PyPI/crates.io resolution rests on an un-auditable deps.dev black box and the validation is only a smoke test.","tokens_in":9237,"tokens_out":2444,"would_cite":true,"duration_ms":24308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds a public dataset of dependency resolution at historical release points for npm, PyPI, and crates.io packages, enriched with vulnerability data, so that dynamic metrics and longitudinal security studies become possible.","keywords":["software ecosystems","dependency resolution","release points","npm","PyPI","crates.io","software supply chain security","dynamic metrics"],"falsifier":"Run period-accurate package-manager binaries—npm with its time-travel flag and historical pip and Cargo versions—on a random sample of release points, and compare their resolved dependency versions against the rows in CHRONO-RESOLUTION; a non-negligible mismatch, especially for pre-2020 PyPI, would refute the historical-accuracy assumption.","tokens_in":8278,"feed_emoji":"📦","tokens_out":4847,"duration_ms":44911,"temperature":0.7,"pith_summary":"The paper aims to solve a missing capability: major package ecosystems do not let you ask what dependencies a package would have pulled at a specific time in the past, which blocks dynamic metrics like dependency freshness and update speed. The authors propose a methodology that slices each package–dependency relationship into time intervals bounded by release events, resolves the dependency version at each interval start, and enriches the result with vulnerability ranges. They apply this to the largest, oldest, and newest major ecosystems and release the resulting dataset (210,949 packages, millions of relationships, thousands of advisories). If the retroactive resolution is faithful, researchers can for the first time study ecosystem evolution and security exposure at release-point granularity across ecosystems.","feed_headline":"210k packages: dependency graph rebuilt at every release point","feed_subtitle":"First public cross-ecosystem dataset lets researchers measure freshness, update speed, and n-day exposure over time.","key_machinery":"The core device is the release-point interval: every package–dependency pair is divided into contiguous intervals whose boundaries are the release times of either the package or the dependency, and the dependency version is resolved at the interval start. This converts an atemporal constraint (for example, '>=1.0.0') into a concrete, time-stamped version that a developer would have installed then, making dynamic metrics such as freshness, time-to-update, and time-to-remediate computable from the data alone.","core_discovery":"The central claim is that a static snapshot of package metadata can be turned into a temporal dataset of dependency resolution at release points by cutting time into intervals between release events and asking a modern resolver what it would install at the start of each interval. The authors add a Boolean 'remediated' flag by joining resolved versions against vulnerability range data, treating a vulnerable version as unremediated only when a fixed version already existed at that time. The released dataset covers 146,651 npm, 48,608 PyPI, and 15,690 crates.io packages, with 2.2M, 0.36M, and 0.16M unique direct relationships and 6,948 advisories, and is intended to be the reference for dynamic","pith_inferences":["If the retroactive resolution holds up against period-accurate package managers, this dataset could become a standard benchmark for dynamic dependency metrics, effectively giving every major ecosystem the time-travel feature npm has natively.","The interval-splitting methodology generalizes beyond the three ecosystems; any registry with version metadata and a resolution service could be added without changing the core design.","The pre-2020 PyPI caveat is a real boundary: longitudinal studies across that cutoff should either restrict to post-2020 data or treat earlier rows as reconstructions, and a period-accurate pip reproduction would be a natural validation target."],"forward_implications":["Dynamic metrics such as dependency freshness, update rhythm, mean-time-to-update, and mean-time-to-remediate can now be computed at release-point granularity for the three ecosystems.","Researchers can track, version by version, which dependencies enter or leave common use, enabling ecosystem evolution studies that static snapshots cannot support.","Supply-chain analyses can measure n-day exposure and distinguish fixable from unfixable vulnerabilities through the newly added fix-available annotation.","The released pipeline lets others rebuild or extend the dataset as APIs evolve or as new ecosystems are added."],"fun_headline_variants":["See any past state of npm, PyPI, crates.io dependencies at each release","Dataset rebuilds dependency graphs at every release point across 3 ecosystems","Time-resolved dependencies: 210k packages + vulnerabilities at each release","Back-in-time dependency resolution for npm, PyPI, and crates.io packages"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The dataset assumes that a modern resolver's retroactive answer at each release point equals what a developer would actually have installed then; PyPI data before late 2020 explicitly may not match historical behavior, and the manual check of 50 packages per ecosystem is a smoke test, not statistical evidence.","fun_headline_variants_meta":{"raw":{"variants":["See any past state of npm, PyPI, crates.io dependencies at each release","Dataset rebuilds dependency graphs at every release point across 3 ecosystems","Time-resolved dependencies: 210k packages + vulnerabilities at each release","Back-in-time dependency resolution for npm, PyPI, and crates.io packages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1209,"prompt_tokens":705,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":449,"tokens_out":504,"duration_ms":5061,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:07:57.996541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run period-accurate package-manager binaries—npm with its time-travel flag and historical pip and Cargo versions—on a random sample of release points, and compare their resolved dependency versions against the rows in CHRONO-RESOLUTION; a non-negligible mismatch, especially for pre-2020 PyPI, would refute the historical-accuracy assumption.","supporting_citations":[],"review_version":1}