{"id":"59fca847-39c7-424d-b225-7f6229130689","arxiv_id":"2505.15042","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"More than 12,000 archived software repository home pages were examined, showing that less than 5% of source files are typically archived and that deeper source files are archived much less often.","lead":"Researchers measured how much of the GitHub, GitLab, BitBucket, and Sourceforge software projects are preserved in the Internet Archive's Wayback Machine. They found that most repositories have very few archived source files and that archived copies of repository home pages are often damaged or incomplete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Archival percentages are computed by dividing mementos from any capture date by the current GitHub API tree, so the complexity/depth gradient may be an artifact of comparing historical archives to today's repository state.","rationale":"The reader's weakest assumption pointed at CDX completeness and GitHub tree mismatches; I agree. My read sharpens this: the mismatch is not just a possible source of noise, it is structural and specifically biases the depth/complexity comparison that the title asserts. The raw-file exclusion is also worth checking, but it would change the overall level, not the complexity gradient; the temporal denominator mismatch is what the causal claim depends on. The presentational-page findings in Section 5.1 are less affected because they assess damage of captured home pages rather than coverage of a changing tree. The paper deserves credit for clearly describing the CDX and GitHub API pipeline and for using 200-status mementos; the concern is not about data fabrication but about whether the denominator measures what the claim requires. No formal verification or released dataset exists, so an independent snapshot-matched recomputation is the appropriate check.","tokens_in":14044,"tokens_out":7760,"duration_ms":73339,"concrete_test":"Take a random sample of 200 GitHub repositories with non-zero archived source counts. For each, use the GitHub API commits/tree endpoint to obtain the repository tree at the timestamp of the most recent archived source-file memento (or, failing that, at the latest capture of the repository home page), then re-run the CDX matching against that snapshot tree. Recompute the overall archival percentage and the depth-stratified rates reported in Figure 9. If the root-vs-depth-one gradient (42% vs 6.11%) persists under snapshot-matched denominators, the complexity claim is supported; if it flattens or reverses, the reported gradient is an artifact of comparing today's tree to historical captures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the denominator construction in Section 4. To measure how much of a repository's source tree is archived, the authors take the repository's current source tree from the GitHub API and query the Wayback CDX API for mementos of each file's HTML page, using the most recent 200-status memento. But the CDX mementos are collected over the repository's entire lifetime, while the GitHub API, as used here, returns the live default branch at study time. The two are not temporally aligned. Files added to the default branch after the last crawl are in the denominator but cannot have mementos, which mechanically lowers the reported archival percentage; files deleted, renamed, or moved before the study are absent from the denominator, which raises it. The depth result is the most vulnerable: a file at depth two introduced in a recent commit will be counted as 'not archived' even though no crawler ever had a chance to archive it as part of that tree. Likewise, if the archived snapshot used branch 'master' while today's default branch is 'main', every file path queried under 'main' may fail to match older captures. The paper itself concedes in Section 6 that temporal coherence was not addressed; that limitation is not merely a nuance, it is the pivot on which the 'complexity leads to diminished availability' claim rests. The observed depth gradient could survive the correction, but it has not been demonstrated until the denominator is matched to the capture date.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates how well Web-hosted Git repositories are preserved in the Internet Archive's Wayback Machine. The authors use a sample of 12,110 repository home pages from GitHub, GitLab, BitBucket, and Sourceforge, applying the Memento Damage tool to quantify missing page resources; they report that 58.22% of archived home pages are undamaged, 29.85% have minor damage, and 1.6% have major damage. For the source-tree analysis on about 10,000 GitHub repositories, they combine the GitHub API's current source tree with Wayback CDX API queries to determine what fraction of source-file HTML pages have a 200-status memento, finding that on average fewer than 5% of source files are archived, about 65% of repositories have no archived source files, and the archival rate falls from about 42% for root-level files (among repositories with at least one archived source file) to about 6% at depth one and about 2% at depth two. The paper concludes that repository complexity, measured by file count and directory depth, leads to diminished web archive availability.","tokens_in":14335,"tokens_out":10608,"duration_ms":93303,"significance":"If the measurements are sound, this is a useful empirical contribution to digital preservation and scholarly reproducibility, quantifying the known intuition that page-per-file repository structures are poorly captured by conventional web crawlers. The paper's strengths include the large corpus, the use of public APIs, and the release of the crawler extension and collection scripts, which make the pipeline replicable. The distinction between presentational-page damage and source-tree completeness is appropriate. However, the title's causal claim is not supported by the paper's own weak correlations, and the source-tree denominator is constructed without temporal alignment between the live repository state and the archived captures, a gap the authors explicitly concede in Section 6. With a time-matched denominator, the headline depth gradient could survive, but it has not yet been demonstrated.","major_comments":[{"comment":"The denominator for the archival percentage is the repository's current source tree from the GitHub API, while the numerator is any 200-status memento found via CDX over the repository's entire lifetime. Files added to the default branch after the last crawl are counted as 'not archived' even though no crawler had the chance to capture them, and files deleted, renamed, or moved before the study are absent from the denominator. The depth gradient in Figure 9 is exactly what this temporal mismatch could produce if root-level files tend to be older and deeper files were added in recent commits. Section 6 acknowledges that 'temporal coherence' was not addressed, but this is not a peripheral caveat: the central quantitative claims depend on the denominator. The authors should either match the denominator to the capture date (for example, by reconstructing the tree at the memento's datetime using the GitHub API with a commit SHA, or by restricting the denominator to files known to exist before a relevant capture window) or provide a sensitivity analysis showing that the depth gradient is robust to such temporal restriction. The paper should also state how branch names in URI paths were normalized, since a default-branch change from 'master' to 'main' would make many CDX queries miss older captures.","section":"Section 4 and Section 6"},{"comment":"The reported correlations are internally inconsistent with the narrative. Section 5.2 gives Spearman's rho = 0.246 for file count and rho = 0.116 for structural depth, both positive, yet the text says 'more complex source trees generally had lower rates of availability' and Figure 9 shows a negative depth gradient. Either the sign is missing, the variable is actually the percentage of files not archived, or the interpretation is wrong; please correct and re-report. Regardless of sign, |rho| <= 0.246 is far too weak to support the title's causal 'leads to' claim, and the p-values only reflect the large sample size. To support causal language, the authors would need an identification strategy that controls for repository age, update recency, popularity, and external in-links, which they discuss qualitatively for outliers such as swift-evolution and Konect-Toolbox. As it stands, a descriptive claim such as 'is associated with' is the most the data support.","section":"Section 5.2 and title"},{"comment":"The depth analysis in Figure 9 is restricted to repositories with at least one archived source file, but the denominator at each depth is not defined. Is it the number of files at that depth in the current live tree, or the number of files at that depth in the archived snapshot? If it is the former, the temporal mismatch in the first major comment applies with extra force. In addition, the 'structural depth' used in the correlation analysis is never defined: is it maximum directory depth, mean depth, or the depth of each individual file? Without this definition, the reported rho = 0.116 cannot be interpreted or reproduced.","section":"Section 5.2, Figure 9"}],"minor_comments":[{"comment":"The BitBucket row does not sum to 100%: Not Archived 90.1% + Undamaged 0.09% + Minor 0.002% + Major 0% leaves about 9.8% of the 610 sampled repositories unaccounted for. The text says only six BitBucket pages archived before 2020 could be fully rendered, so the table needs a clearer breakdown that distinguishes 'not archived', 'archived but not rendered', and 'damaged'.","section":"Table 1"},{"comment":"The URI schema description states that 'type is usually either blob for directories and file for files', but on GitHub the path component is 'tree' for directory pages and 'blob' for file pages. If the implemented crawler uses this description literally, it would query incorrect URLs for GitHub source files.","section":"Section 4"},{"comment":"The abstract reports a 14.89% archival rate for files linked directly from repository home pages, but Section 5.2 reports 'just over 42%' for repositories with a non-zero amount of archived source files. Please state the denominator for each figure so the two numbers are not confusing.","section":"Abstract and Section 5.2"},{"comment":"The sentence 'three-quarters of the repositories had less than 15% archived' refers to Figure 7, which is an ECDF for repositories with a non-zero number of archived files. Make this conditional explicit in the text, since the 65% of repositories with zero archived files otherwise makes the sentence ambiguous.","section":"Section 5.2"},{"comment":"The paper should state the data collection dates and the GitHub API version used, since live repository trees and CDX results change over time and the study's replicability depends on knowing the observation window.","section":"Section 4 and Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The descriptive measurements are likely useful and the authors have been candid about the temporal-coherence limitation. If the authors re-run the source-tree analysis with a time-matched denominator and adjust the title to a descriptive claim, the paper could be publishable. If the time-matched analysis reverses the depth gradient, the central contribution collapses; this is why I could not recommend minor revision. The GitHub sample is drawn substantially from the authors' own prior citation dataset, which is appropriate but should be more explicitly acknowledged as a citation-biased convenience sample."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful measurement paper with a title that overpromises. The new thing here is the depth-resolved look at how much of a GitHub repository's source tree actually survives in the Wayback Machine—more than 12,000 repos, less than 5% of files archived on average, and a sharp drop after depth zero. That number and the depth curve are worth knowing. The presentational-page damage stats are a reasonable secondary contribution, and the paper is honest about its own limitations.\n\nThe soft spots are concentrated in the causal claim. The title says complexity 'leads to' diminished availability, but their own correlations are weak (Spearman 0.246 for file count, 0.116 for depth). Weak correlation by itself doesn't kill a behavioral claim, but here the mechanism is inferred, not tested. More importantly, the denominator in Section 4 is the current GitHub default-branch tree, while the mementos come from any capture date over the repo's lifetime. Files added after the last crawl are counted as 'not archived'; files renamed or moved are invisible. The paper concedes this in Section 6 when it says temporal coherence was not addressed. That's not a minor footnote: it directly affects the depth gradient, which is the paper's headline result. The gradient may well survive a proper temporal fix—shallow files really do get more crawler attention—but the paper hasn't shown that yet.\n\nOther issues are minor: the sample is partly drawn from trending/topic searches, so it's not a random slice of GitHub; the raw dataset isn't released; the damage thresholds (25% major, depth grouping >4) are reasonable but arbitrary. None of these are fatal.\n\nWho should read this: web archiving researchers and digital preservation folks, and anyone designing crawlers for code hosting sites. It belongs in the literature after revision—retitle or heavily soften the causal language, and either match the tree snapshot to the capture time or clearly frame the result as 'current tree vs. all historical mementos' with the caveat front and center. I'd send it to peer review; it's a solid empirical contribution in need of a more careful frame.","headline":"Useful depth-resolved measurement of how little GitHub source code survives in the Wayback Machine, but the causal title outruns the evidence.","tokens_in":14822,"tokens_out":1824,"would_cite":true,"duration_ms":17936,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Web archives preserve less than 5% of GitHub source files, and availability drops sharply with directory depth.","keywords":["Web archiving","GitHub","Git repositories","Digital preservation","Wayback Machine","Memento","Crawler behavior","Source code"],"falsifier":"Take a fresh random sample of GitHub repositories, enumerate every source file via the GitHub API, and for each file query the Wayback Machine's availability API for any snapshot. If the fraction of files with at least one snapshot were substantially higher than the paper's depth-stratified rates, especially at depth two or beyond, the claim would fail. The test is cheap and independent, since it does not rely on the paper's original sample or its index queries.","tokens_in":13864,"feed_emoji":"🗄️","tokens_out":8700,"duration_ms":68182,"temperature":0.7,"pith_summary":"This paper measures how well web-hosted software repositories are preserved in the Internet Archive's Wayback Machine, and finds the preservation is far too incomplete to rebuild a project from its archived pages. Using more than 10,000 archived GitHub repositories (plus GitLab, BitBucket, and Sourceforge samples), the authors compare each repository's live source tree, obtained from the GitHub API, against the set of archived file pages, obtained from the Wayback Machine's index. They report that, on average, less than 5% of a repository's source files have an archived page, that around 65% of sampled repositories have no archived source files at all, and that availability drops off steeply with directory depth. The finding matters because the Wayback Machine is a common last-resort source for recovering software whose original host has disappeared; if the paper is right, that fallback generally fails for source code.","feed_headline":"Only 5% of GitHub source files survive web archive capture","feed_subtitle":"Depth is the deciding factor: files linked from a repo's home page get archived; nested files mostly vanish.","key_machinery":"The central mechanism is the depth-stratified comparison of a live repository's file tree against its archived pages. The authors enumerate every source-file URL in a repository from the GitHub API, query the Wayback Machine's index for archived versions of those URLs, and then stratify the archival rate by the file's distance from the repository root. This depth stratification exposes the pattern that carries the argument: archival availability decays steeply with each additional directory level. A secondary mechanism is the page-damage measurement used on archived home pages, which weights missing page resources (images, stylesheets, scripts) by size and importance to produce a damage score between 0% and 100%.","core_discovery":"The paper's claim is that the HTML-page representation of a Git repository—one page per source file, with the repository home page linking only to top-level files—interacts badly with the way web archives crawl. The authors find that archived repository home pages are often present and readable, but the source tree is mostly absent: across all sampled GitHub repositories, only about 4.72% of source-file pages were archived, and among repositories with at least one archived source file the average was still only 13.39%. Availability is heavily depth-dependent: files linked directly from a repository's home page are archived at roughly 42% (among repositories with any archive presence), while files one directory deep drop to 6.11% and files two directories deep to 2.44%. They also measure home-page damage with a page-damage metric, finding 31.21% of GitHub home pages have minor damage and 1.43% have major damage, with recent JavaScript-loaded pages on GitLab and BitBucket often not replayable at all. The conclusion is that the archived source tree of a repository is typically too incomplete to be reconstructed.","pith_inferences":["The depth-decay pattern likely generalizes beyond GitHub to any site that exposes a hierarchical document tree as per-URL pages, such as documentation sites or package registries; the same crawl-depth bias should apply.","Because the sample skews toward popular and trending repositories, the true average archival rate across all of GitHub may be even lower than the reported 4.72%.","A testable extension would be to correlate the depth-decay curve with a repository's in-links from external sites; the paper's outlier examples suggest external links and popularity are stronger predictors than file count or structural depth, even though the sampled correlations were weak.","A practical design suggestion is that archive crawlers could be given the raw git bundle or archive download rather than the HTML page representation, sidestepping depth entirely."],"forward_implications":["A project that disappears from GitHub will usually not have its source code recoverable from the Wayback Machine.","Repositories with flat, small trees (especially under about 22 files) are the only ones likely to be fully archived.","Researchers citing a GitHub repository cannot treat a Wayback capture of the home page as evidence that the code itself is preserved.","Web archives that want to preserve source code need crawlers that deliberately traverse repository trees, or they need to integrate with source-code archives.","Linking source files directly from a home page or an external site substantially raises the chance that those files get archived."],"supporting_citations":[{"why":"Supplies the earlier finding that only 68.39% of analyzed repositories were archived in Software Heritage, creating the comparison point the present study extends.","marker":"[18]"},{"why":"Provided the majority of the GitHub repository URIs analyzed in this study.","marker":"[19]"},{"why":"Developed the page-damage assessment service used to measure archived home-page fidelity.","marker":"[41]"},{"why":"Defines the weighted page-damage metric that the damage assessment is built on.","marker":"[6]"},{"why":"Defines the memento concept used to identify archived versions of source-file pages.","marker":"[45]"},{"why":"Documents how JavaScript-heavy pages challenge archival crawlers, supporting the explanation for GitLab and BitBucket failures.","marker":"[7]"},{"why":"Shows replay failures in client-side-rendered pages, further explaining why some archived repository pages are unusable.","marker":"[47]"}],"fun_headline_variants":["GitHub source files mostly missing from web archives","Depth decides: only 5% of GitHub code archived","Web archive misses 95% of GitHub source files","Archived GitHub repos lose deep source trees"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measurement assumes the GitHub API's list of source files and the Wayback Machine's index of archived pages both accurately reflect what existed and what was captured; if either is incomplete, the reported archival percentages could be systematically wrong.","fun_headline_variants_meta":{"raw":{"variants":["GitHub source files mostly missing from web archives","Depth decides: only 5% of GitHub code archived","Web archive misses 95% of GitHub source files","Archived GitHub repos lose deep source trees"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3370,"prompt_tokens":985,"completion_tokens":2385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2324}},"tokens_in":601,"tokens_out":2385,"duration_ms":16316,"temperature":1.0,"reasoning_tokens":2324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:05.559435+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of GitHub repositories, enumerate every source file via the GitHub API, and for each file query the Wayback Machine's availability API for any snapshot. If the fraction of files with at least one snapshot were substantially higher than the paper's depth-stratified rates, especially at depth two or beyond, the claim would fail. The test is cheap and independent, since it does not rely on the paper's original sample or its index queries.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Developed the page-damage assessment service used to measure archived home-page fidelity."},{"cited_title":"Weigle, and Michael L","cited_arxiv_id":null,"evidence_quote":"Defines the weighted page-damage metric that the damage assessment is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the memento concept used to identify archived versions of source-file pages."}],"review_version":1}