Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Longitudinal Sampling of URLs From the Wayback Machine

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 26-year sample of 27.3 million archived URLs is built to measure how long web pages last.

desk verdict A candid and detailed dataset paper whose central claim holds up; the main work in review should be on reproducibility, not on whether the sample exists. read the letter →

arxiv 2507.14752 v1 pith:RGNPSAUC submitted 2025-07-19 cs.DL

Kritika Garg , Sawood Alam , Dietrich Ayala , Mark Graham , Michele C. Weigle , Michael L. Nelson This is my paper
classification cs.DL
keywords webarchivesURLsamplingWaybackMachineMementoprotocollongitudinaldatasetpagelifespanCDXindexTimeMaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper documents a large longitudinal sample of archived web pages: 27.3 million URLs with 3.8 billion archived captures, first archived between 1996 and 2021, drawn from the Wayback Machine. The authors' aim is to make a dataset that can revisit the question 'How long does a web page last?' at a scale far beyond earlier studies that tracked hundreds or millions of pages for short periods. They sample directly from the archive's own index rather than starting from an external URL list, and they walk through the sequence of filters and corrections that balance temporal and domain coverage. The paper is explicit that the result is a sample of the archived web, carrying the archive's own biases, not a representative sample of the live web.

What carries the argument

The central machinery is the ZipNum index: a compressed form of the Wayback Machine's CDX index containing every 6000th entry, which makes it possible to sample URLs without scanning hundreds of billions of records. Sampling from it makes a URL's inclusion probability proportional to its memento count, with URLs having more than 6,000 captures guaranteed inclusion; the paper then layers a sequence of corrections on top of that frame, including extension-based HTML filtering, CDX API MIME-type verification, root-URL upsampling for early years, and logarithmic downsampling per domain using $\min(N, \text{round}(K \cdot \log(N) + C))$. The unit of the final dataset is the Memento-protocol TimeMap, the full list of archived captures for one original URL, and collecting these TimeMaps for all 27.3 million URLs yields the 3.8 billion mementos.

What would settle it

Compute the per-URL inclusion probability under the full pipeline, multiplying the ZipNum every-6000th-line draw, the extension heuristic, the CDX MIME-type verification, the root-URL upsampling, and the log-domain downsampling; if those probabilities vary by more than an order of magnitude within a year and no weights are published with the dataset, then pooled lifespan estimates from the 27.3 million URLs cannot be taken as unbiased for the archived web.

Watch

Extended reading notes

Core claim

The central claim is that it is feasible to build a web-lifespan dataset of 27.3 million URLs and 3.8 billion mementos by taking every 6000th line of the Wayback Machine's CDX index, filtering for likely HTML by filename extension, confirming text/html MIME types through the CDX API, upsampling sparse early years with root URLs extracted from deep links, and log-downsampling domains so popular sites do not dominate. The final sample has roughly one million HTML URLs per year across 26 years and 7 million unique hosts, each with its full TimeMap. The authors present the resulting dataset as the foundation for estimating the size, nature, and prevalence of the publicly archivable web, and for re-examining page lifespan.

Load-bearing premise

The dataset can support lifespan estimates only if the biases introduced by the multi-stage sampling procedure, namely memento-count-weighted ZipNum selection, extension-based HTML filtering, early-year root-URL upsampling, and hand-tuned log downsampling, are small enough or documented enough to be corrected for.

Editorial extensions

If this is right

  • Lifespan and survival-curve estimates for HTML pages first archived in each year from 1996 to 2021 can be computed from a dataset of roughly one million URLs per year, something earlier studies could not support.
  • Because the dataset stores full TimeMaps, analyses can look beyond first and last capture to archiving intensity, gaps between captures, and reuse of the same content over time.
  • The logarithmic downsampling keeps the heavy-tailed distribution of URLs per domain while preventing a few platforms from dominating, so domain-level diversity is retained in each yearly bin.
  • The documented failure modes, including wildcard asterisk URLs, revisit records with missing status codes, session-ID aliases, and SURT canonicalization errors, give future archive-sampling projects a concrete checklist.
  • Researchers studying link rot and reference rot can use this sample as a comparison point for URL sets drawn from scholarly articles or news stories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not provide inclusion probabilities or sampling weights for the final 27.3 million URLs, so any pooled estimate of how long a web page lasts will need a separate design-based analysis or sensitivity study before it can be treated as a population quantity; this is my inference from the described procedure.
  • Because the extension-based HTML filter was validated only for precision and never for recall, the sample's coverage of extensionless or unusual HTML URLs is unknown; checking recall would require querying the CDX API for a random set of URLs that failed the filter.
  • A testable extension of the same pipeline would apply the sampling stages to non-HTML MIME types such as images, CSS, and JavaScript, letting the lifespan question extend to embedded resources that are part of the user experience.
  • Comparing survival curves from this archive-driven sample with curves from small manually revisited random samples would quantify how much archive selection changes lifespan estimates; this comparison is not performed in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes the construction of the Not Your Parents' Web (NYPW) dataset, a longitudinal sample of URLs drawn from the Internet Archive's Wayback Machine. The authors start from the ZipNum index (every 6000th CDX entry), filter to 'likely HTML' URLs by filename-extension heuristics, verify MIME types via CDX API queries, group URLs by year of first capture, upsample early years by extracting root URLs from deep links, and then apply long-tail reduction and logarithmic downsampling to balance domain representation. The final dataset is reported to contain TimeMaps for 27.3 million URLs and 3.8 billion mementos spanning 1996-2021. The paper also documents several practical issues encountered, including session-ID aliases, wildcard URLs, warc/revisit records, and a SURT canonicalization bug.

Significance. If the dataset is reproducible and its biases are adequately characterized, it would be a valuable public resource for studying web page longevity, complementing prior studies that start from external URL corpora. The authors provide GitHub links for much of the sampling code, a clear funnel diagram of the pipeline, and an honest discussion of several known limitations (e.g., no true negatives/false negatives for the HTML filter, and acknowledged archival selectivity). The Memento terminology and the detailed account of TimeMap collection issues (revisit-record rehydration, wildcard URLs, SURT bugs) are useful lessons for the web-archiving community. However, because the paper's central claim is the existence and composition of the dataset, the internal numerical inconsistencies, the absence of false-negative analysis, and the lack of design weights are significant gaps that must be addressed before the dataset can be used for reliable downstream inference.

major comments (4)
  1. [§4.5.1, §4.5.2, §4.8] The counts at the critical 107M-to-92M stage are internally inconsistent, which undermines the reproducibility of the central claim. Section 4.5.1 says 115 million URLs from popular domains were excluded, but Section 4.8 says 111 million; the text states that 92 million of the 107 million likely-HTML URLs were text/html, while Table 5 sums to 92.6 million; and the reported precision of 88.12% for the 40M sample does not match Table 4's totals (35.6M/40.3M ≈ 88.3%). The exact version of the extension regexes/heuristics and an auditable enumeration of the 107M URL set are not provided. Please correct the counts, specify the exact regex and code version, and describe how the 107M set was formed and stored so that the pipeline is independently reconstructible.
  2. [§4.5.2] The extension-based HTML filter has no false-negative (recall) analysis. The authors explicitly state that the evaluation only measured precision on URLs already predicted as HTML, so the number of HTML pages excluded by the heuristic is unknown. Because the dataset is intended to represent 'web pages,' a missing recall estimate means the filtering stage introduces an unquantified selection bias. Please add a validation experiment on a random sample of URLs that do not pass the extension heuristics, query their MIME types via the CDX API, and report the estimated false-negative rate, ideally stratified by year and TLD.
  3. [§4.7-§4.9] The multi-stage sample design (ZipNum probability proportional to memento count, popular-domain exclusion/reintroduction, root-URL upsampling, long-tail reduction, and per-year logarithmic downsampling with hand-tuned K and C) is not accompanied by inclusion probabilities or design weights. Without these, the final dataset cannot be treated as a probability sample for estimating quantities like 'how long does a web page last.' Moreover, the popular-domain reintroduction is described only for twitter.com, yet Table 8 shows many other popular domains in the final sample. Please provide per-URL inclusion weights (or the code and parameters sufficient to compute them), clarify the exact procedure used for all excluded popular domains, and state the target population and estimands the dataset is intended to support.
  4. [§4.9, Table 7, §5] The downsampling parameters K and C, as well as the 900,000-domain long-tail reduction threshold, were chosen post hoc by trial-and-error to hit target sample sizes. The resulting per-year sample sizes and the reported 'heavy-tailed' domain/memento distributions are therefore partly artifacts of the tuning procedure rather than empirical properties of the archive. The Discussion should more explicitly separate design-induced properties from observational findings, and a sensitivity analysis around K, C, and the long-tail threshold would help readers understand how much the final dataset's shape depends on these choices.
minor comments (6)
  1. [§4.9] In the paragraph following Equation (1), the text says 'The term log(N)+C sets a baseline...' but the equation is K·log(N)+C; please fix this to avoid confusion about whether C is inside the logarithm.
  2. [Figure 1 caption] The caption contains the typo 'response turnicated' and should read 'response truncated.'
  3. [§4.7] The sentence 'We decided to retain the root URLs that were first archived between 1996 and 2002' is unclear relative to Figure 9, which shows root-URL counts for all years; please clarify whether newly extracted root URLs outside 1996-2002 were discarded or included in subsequent downsampling steps.
  4. [§4.2, §4.5.1] The relationship between the 292 million ZipNum entries, 285 million unique SURTs, and the 'sampled 285 million URLs' phrasing in the abstract should be stated more precisely, since duplicate removal occurs before URL conversion.
  5. [Dataset access] The paper mentions GitHub code and the NYPW project but does not provide a persistent identifier or explicit access instructions for the final 27.3M-URL dataset and the 1.4TB of TimeMaps; please add a data-availability statement with a DOI or archive.org identifier.
  6. [§4.5.2, Table 5] The text says '92 million out of 107 million URLs' while Table 5 sums to 92.6 million; use a single consistent figure (and corresponding percentage) throughout the manuscript.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivational circularity; dataset statistics are measured outputs. A mild normalization-dependence exists: the reported per-year balance is produced by hand-tuned downsampling parameters rather than discovered.

  1. other [Section 4.9, Equation 1 and Table 7; restated in Section 4.13]
    "Our goal was to reduce the number of URLs while maintaining approximately 1 million URLs per yearly sample. To achieve this, we adjusted the parameters K and C in Equation 1, allowing us to fine-tune the downsampling process for each year's dataset."

    The per-year 'Reduced URLs' counts in Table 7 are generated by applying Equation 1 after K is tuned until the target of roughly 1 million URLs per year is reached (with C=1). The later statement in Section 4.13 that 'Each bin contains more than one million URLs' therefore restates the fitting target instead of reporting an independently measured property of the archive. This makes the reported temporal balance partly constructed by the pipeline. It is not a full circular derivation: the paper makes no prediction, and the headline quantities (27.3M URLs, 3.8B mementos, 7M domains) are measured outputs of the completed pipeline.

full rationale

This paper is a dataset-construction and methodology report rather than a derivation with testable predictions, so the classical circularity patterns do not arise: there is no definitional identity between an input and an output, no fitted value is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The ZipNum selection bias is established in the paper itself by Figure 5 and the stated 6000-line block structure, not imported from a load-bearing self-citation. The self-citations in the references (project blog, prior posters, TrendMachine) are contextual and do not carry the numerical claims. The only normalization-dependence signal is the hand-tuning of K and C in the log-downsampling step to hit the target of approximately 1 million URLs per year, which means statements about the resulting temporal balance are engineered rather than discovered. The paper acknowledges the tuning goal explicitly, so this is a mild caveat about constructedness, not a hidden circular derivation. Other limitations raised by the pipeline (the undocumented 107M-to-92M transition, missing inclusion probabilities, and reproducibility of the exact heuristic version) are completeness and bias concerns, not circularity, and therefore do not affect this score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The pipeline depends on several domain assumptions that are stated but not validated quantitatively. The most consequential are the use of first-archive date as a proxy for creation date and the acceptance of the ZipNum/CDX frame with its memento-count-proportional selection. The downsampling parameters are free choices. No new theoretical entities are introduced.

free parameters (5)
  • K (log-downsampling scale) = per-year values in Table 7 (1-5)
    Tuned per year via trial-and-error to hit target of ~1M URLs per year; directly controls final yearly sample sizes.
  • C (log-downsampling minimum) = 1
    Chosen constant; ensures at least 1 URL per domain retained.
  • Long-tail reduction fraction = 90%
    Reduced the long tail by 90% for years with >900,000 unique domains; threshold and fraction chosen through trial-and-error.
  • Target URLs per year = 1 million
    Goal set by authors, not derived from a research question; determines how K and C are tuned.
  • Popular domain exclusion count = 115 million (also stated as 111 million)
    URLs from popular domains were set aside for separate processing; the count differs across Sections 4.5.1 and 4.8.
assumptions (4)
  • domain assumption The earliest memento datetime is a reasonable proxy for when a web page was created.
    Section 4.6: 'we utilized the datetime of the earliest memento as an estimate for this purpose.' This is load-bearing for temporal sampling and for any downstream longevity estimation.
  • domain assumption HTML pages intended for humans are the object of study, and extension-based heuristics plus text/html MIME capture them adequately.
    Sections 4.5.1-4.5.2; the paper acknowledges no false-negative measurement and excludes extensionless or unusual HTML resources.
  • domain assumption The Internet Archive's ZipNum index (every 6000th CDX line) is an acceptable sampling frame for the archived web.
    Section 4.1; this frame is biased toward high-memento URLs, and the paper does not correct for the resulting inclusion probabilities.
  • domain assumption warc/revisit records can be rehydrated to their original status codes by matching content digests within an LRU cache.
    Section 4.11; this is necessary for accurate TimeMap analysis and rests on the assumption that the prior record with the same digest is present and correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Longitudinal Sampling of URLs From the Wayback Machine." pith.science (2026). https://pith.science/paper/RGNPSAUC

@misc{pith2026250714752,
  author       = {Pith},
  title        = {Pith review of: Longitudinal Sampling of URLs From the Wayback Machine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RGNPSAUC}},
  note         = {Machine review of arXiv:2507.14752}
}
read the original abstract

We document strategies and lessons learned from sampling the web by collecting 27.3 million URLs with 3.8 billion archived pages spanning 26 years (1996-2021) from the Internet Archive's (IA) Wayback Machine. Our goal is to revisit fundamental questions regarding the size, nature, and prevalence of the publicly archivable web, in particular, to reconsider the question: "How long does a web page last?" Addressing this question requires obtaining a sample of the web. We proposed several dimensions to sample URLs from the Wayback Machine's holdings: time of first archive, HTML vs. other MIME types, URL depth (top-level pages vs. deep links), and top-level domain (TLD). We sampled 285 million URLs from IA's ZipNum index file, which contains every 6000th line of the CDX index. These indexes also include URLs of embedded resources such as images, CSS, and JavaScript. To limit our sample to "web pages" (i.e., pages intended for human interaction), we filtered for likely HTML pages based on filename extension. We then queried IA's CDX API to determine the time of first capture and MIME type of each URL. We grouped 92 million text/html URLs based on year of first capture. Archiving speed and capacity have increased over time, so we found more URLs archived in later years. To counter this, we extracted top-level URLs from deep links to upsample earlier years. Our target was 1 million URLs per year, but due to sparseness during 1996-2021, we clustered those years, collecting 1.2 million URLs for that range. Popular domains like Yahoo and Twitter were over-represented, so we performed logarithmic-scale downsampling. Our final dataset contains TimeMaps of 27.3 million URLs, comprising 3.8 billion archived pages. We convey lessons learned from sampling the archived web to inform future studies.

Figures

Figures reproduced from arXiv: 2507.14752 by the authors.

Figure 1
Figure 1. A curl request to the IA’s Wayback Machine highlights for a specific memento of example.com. The response highlights [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. A snippet of the TimeMap for www.example.com from the Internet Archive using the CDX API. The first record in the CDX response contains information such as the first archived date of the web page and its MIME type. space-separated format [13], with each line representing a memento, i.e., a snapshot of a URL at a specific time. A line in the CDX file has the following structure: [“urlkey” ,“timestamp”, “original”, “M… view at source ↗
Figure 3
Figure 3. A snippet of IA’s ZipNum Index, showcasing its structured format with fields including SURT key, capture datetime, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The overview of NYPW sampling process illustrating the multi-step sampling methodology used to derive the NYPW [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The probability of a URL from the CDX index of the Wayback Machine getting sampled via the ZipNum index is [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of a ZipNum file creation from a CDX index. Popularly archived domains are more likely to be sampled [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Regex pattern used to detect URLs with session IDs [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: The first archive year of 92 million sampled HTML URLs. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The first archive year of 22.3M root URLs and the 89.9 million deep links. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Hydrating the ‘revisit’ record in the TimeMaps. In the upper TimeMap snippet, [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Number of root URLs and deeplinks for each year in our final sample. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Log-log plot of the Complementary Cumulative Distribution Function (CCDF) for the number of HTML URLs per [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Log-log plot of the CCDF representing the number of mementos per URL. The x-axis indicates the number of [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not Here, Go There: Analyzing Redirection Patterns on the Web

    cs.DL 2025-07 conditional novelty 6.0 of 10

    Following 11 million redirecting web addresses shows that half fail, most canonical redirects are HTTP-to-HTTPS, and 62,000 custom 404 pages include many soft 404s.

Reference graph

Works this paper leans on

72 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    Dumais, and Jonathan L

    Eytan Adar, Jaime Teevan, Susan T. Dumais, and Jonathan L. Elsas. 2009. The web changes everything: Understanding the dynamics of web content. In Proceedings of the Second ACM International Conference on Web Search and Data Mining (Barcelona, Spain) (WSDM ’09). Association for Computing Machinery, New York, NY, USA, 282–291. https://doi.org/10.1145/149875...

    work page arXiv 2009

  2. [2]

    Teru Agata, Yosuke Miyata, Emi Ishita, Atsushi Ikeuchi, and Shuichi Ueda. 2014. Life Span of Web Pages: A Survey of 10 Million Pages Collected in 2001. In Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2014) . IEEE, London, UK, 463–464. https://doi.org/10.1109/JCDL.2014.6970226

    arXiv 2014

  3. [3]

    Ainsworth, Ahmed Alsum, Hany SalahEldeen, Michele C

    Scott G. Ainsworth, Ahmed Alsum, Hany SalahEldeen, Michele C. Weigle, and Michael L. Nelson. 2011. How much of the web is archived?. In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries (Ottawa, Ontario, Canada) (JCDL ’11). Association for Computing Machinery, New York, NY, USA, 133–136. https://doi.org/10.1145/19...

    work page arXiv 2011

  4. [4]

    Sawood Alam. 2020. MementoMap: A Web Archive Profiling Framework for Efficient Memento Routing . PhD dissertation. Old Dominion University, Norfolk, VA, USA. https://doi.org/10.25777/5vnk-s536

    work page doi:10.25777/5vnk-s536 2020

  5. [5]

    Sawood Alam. 2024. stream_sampler.py — Reservoir sampling utility. GitHub. https://github.com/ibnesayeed/utils/blob/main/textfiles/ stream_sampler.py

    work page 2024

  6. [7]

    Weigle, Michael L

    Sawood Alam, Mark Graham, Kritika Garg, Michele C. Weigle, Michael L. Nelson, and Dietrich Ayala. 2023. Not Your Parents’ Web. DWeb Camp 2023. https://dwebcamp2023.sched.com/event/1NnA6/not-your-parents-web

    work page 2023

  7. [8]

    Weigle, Michael L

    Sawood Alam, Mark Graham, Kritika Garg, Michele C. Weigle, Michael L. Nelson, and Dietrich Ayala. 2024. Some URLs Are Immortal, Most Are Ephemeral. DWeb Camp 2024. https://dwebcamp2024.sched.com/event/1hX2l/some-urls-are-immortal-most-are-ephemeral

    work page 2024

  8. [10]

    Sawood Alam, Michele Weigle, Michael Nelson, Fernando Melo, Daniel Bicho, and Daniel Gomes. 2019. MementoMap Framework for Flexible and Adaptive Web Archive Profiling. In Proceedings of the 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL) . IEEE, 172–181. https://doi.org/10.1109/JCDL.2019.00033

    arXiv 2019

Show all 72 references
  1. [11]

    Alkwai, Michael L

    Lulwah M. Alkwai, Michael L. Nelson, and Michele C. Weigle. 2017. Comparing the Archival Rate of Arabic, English, Danish, and Korean Language Web Pages. ACM Transactions on Information Systems 36, 1, Article 1 (June 2017), 34 pages. https://doi.org/10.1145/3041656 Longitudinal...

    2017 doi

  2. [12]

    Weigle, Michael L

    Ahmed AlSum, Michele C. Weigle, Michael L. Nelson, and Herbert Van de Sompel. 2014. Profiling Web Archive Coverage for Top-Level Domain and Content Language. International Journal on Digital Libraries 14, 3-4 (August 2014), 149–166. https://doi.org/10.1007/s00799- 014-0118-y

    2014 doi

  3. [13]

    Internet Archive. 2024. CDX File Format Specification. https://archive.org/web/researcher/cdx_file_format.php

    2024

  4. [14]

    Internet Archive. 2024. SURT (Sort-friendly URI Reordering Transform). https://github.com/internetarchive/surt

    2024

  5. [15]

    Internet Archive. 2024. Wayback CDX Server API Documentation. https://github.com/internetarchive/wayback/blob/master/wayback- cdx-server/README.md

    2024

  6. [16]

    Vangelis Banos and Yannis Manolopoulos. 2015. A Quantitative Approach to Evaluate Website Archivability Using the CLEAR+ Method. International Journal on Digital Libraries (2015), 1–23. https://doi.org/10.1007/s00799-015-0144-4

    2015 doi

  7. [17]

    Ziv Bar-Yossef, Idit Keidar, and Uri Schonfeld. 2009. Do not crawl in the DUST: Different URLs with similar text. ACM Transactions on the Web 3, 1, Article 3 (Jan. 2009), 31 pages. https://doi.org/10.1145/1462148.1462151

    2009

  8. [18]

    Tim Berners-Lee, Robert Cailliau, Ari Luotonen, Henrik Frystyk Nielsen, and Arthur Secret. 1994. The World-Wide Web. Commun. ACM 37, 8 (August 1994), 76–82. https://doi.org/10.1145/179606.179671

    1994

  9. [19]

    Fielding, and Larry Masinter

    Tim Berners-Lee, Roy T. Fielding, and Larry Masinter. 2005. RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax. Internet Engineering Task Force (IETF). https://www.rfc-editor.org/rfc/rfc3986

    2005

  10. [20]

    Brewington and George Cybenko

    Brian E. Brewington and George Cybenko. 2000. How dynamic is the Web? Computer Networks 33, 1 (2000), 257–276. https: //doi.org/10.1016/S1389-1286(00)00045-1

    2000 doi

  11. [21]

    Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems 30, 1-7 (1998), 107–117. https://doi.org/10.1016/S0169-7552(98)00110-X

    1998 doi

  12. [22]

    Brunelle, Mat Kelly, Hany SalahEldeen, Michele C

    Justin F. Brunelle, Mat Kelly, Hany SalahEldeen, Michele C. Weigle, and Michael L. Nelson. 2014. Not All Mementos Are Created Equal: Measuring the Impact of Missing Resources. In Proceedings of the IEEE/ACM Joint Conference on Digital Libraries (JCDL) . IEEE, 321–330. https://...

    2014

  13. [23]

    Athena Chapekis, Samuel Bestvater, Emma Remy, and Gonzalo Rivero. 2024. When Online Content Disappears. https://www. pewresearch.org/data-labs/2024/05/17/when-online-content-disappears/

    2024

  14. [24]

    Junghoo Cho and Hector Garcia-Molina. 2000. The Evolution of the Web and Implications for an Incremental Crawler. In Proceedings of the 26th International Conference on Very Large Data Bases (VLDB ’00) . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 200–209

    2000

  15. [25]

    Wikipedia contributors. 2024. Cache replacement policies – Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/wiki/Cache_ replacement_policies

    2024

  16. [26]

    Nelson, Robert Sanderson, Lyudmila Balakireva, Scott Ainsworth, and Harihar Shankar

    Herbert Van de Sompel, Michael L. Nelson, Robert Sanderson, Lyudmila Balakireva, Scott Ainsworth, and Harihar Shankar. 2013. RFC 7089 - HTTP Framework for Time-Based Access to Resource States – Memento. Internet Engineering Task Force (IETF). https://tools.ietf.org/html/rfc7089

    2013

  17. [27]

    Donald Eastlake and Al Panitz. 1999. RFC 2606 - Reserved Top Level DNS Names. Internet Engineering Task Force (IETF). https: //www.rfc-editor.org/rfc/rfc2606

    1999

  18. [28]

    Dennis Fetterly, Mark Manasse, Marc Najork, and Janet Wiener. 2003. A large-scale study of the evolution of web pages. In Proceedings of the 12th International Conference on World Wide Web (Budapest, Hungary) (WWW ’03). Association for Computing Machinery, New York, USA, 669–6...

    2003

  19. [29]

    Not Your Parents’ Web

    Filecoin Foundation. 2024. The Web Isn’t Forever: New Research Findings from “Not Your Parents’ Web” Project. https://fil.org/blog/the- web-isn-t-forever-new-research-findings-from-not-your-parents-web-project

    2024

  20. [30]

    Weigle, and Michael L

    Kritika Garg, Sawood Alam, Dietrich Ayala, Mark Graham, Michele C. Weigle, and Michael L. Nelson. 2024. Some URLs Are Immortal, Most Are Ephemeral. Poster, iPRES 2024; published via Zenodo. https://doi.org/10.5281/zenodo.13687116

    2024 doi

  21. [31]

    Weigle, and Michael L

    Kritika Garg, Sawood Alam, Dietrich Ayala, Michele C. Weigle, and Michael L. Nelson. 2024. Some URLs Are Immortal, Most Are Ephemeral (Poster #203). Poster session at the IIPC GA & Web Archiving Conference 2024. https://netpreserve.org/ga2024/abstracts/ #poster_203 Poster #203

    2024

  22. [32]

    Weigle, and Michael L

    Kritika Garg, Sawood Alam, Dietrich Ayala, Michele C. Weigle, and Michael L. Nelson. 2025. Not Here, Go There: Analyzing Redirection Patterns on the Web. In Proceedings of the 17th ACM Web Science Conference (WebSci ’25) . Association for Computing Machinery, New York, NY, USA...

    2025

  23. [33]

    Kritika Garg, Sawood Alam, Michele Weigle, Michael Nelson, Corentin Barreau, and Mark Graham. 2023. Lessons Learned From the Longitudinal Sampling of a Large Web Archive. UNT Digital Library. https://digital.library.unt.edu/ark:/67531/metadc2143930/

    2023

  24. [34]

    Jayanetti, Sawood Alam, Michele C

    Kritika Garg, Himarsha R. Jayanetti, Sawood Alam, Michele C. Weigle, and Michael L. Nelson. 2021. Replaying Archived Twitter: When your bird is broken, will it bring you down?. In Proceedings of the 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) . 160–169. https://...

    2021

  25. [35]

    Jayanetti, Sawood Alam, Michele C

    Kritika Garg, Himarsha R. Jayanetti, Sawood Alam, Michele C. Weigle, and Michael L. Nelson. 2024. Challenges in Replaying Archived Twitter Pages. International Journal on Digital Libraries 25, 2 (2024), 217–236. https://doi.org/10.1007/s00799-023-00379-w

    2024 doi

  26. [36]

    Google. n.d.. Cloaking - Search Essentials. https://developers.google.com/search/docs/essentials/spam-policies#cloaking 28 • Garg et al

  27. [37]

    Mark Graham. 2019. The Wayback Machine’s Save Page Now is New and Improved! https://blog.archive.org/2019/10/23/the-wayback- machines-save-page-now-is-new-and-improved/

    2019

  28. [38]

    Wendy Hall and Thanassis Tiropanis. 2012. Web evolution and Web Science. Computer Networks 56, 18 (2012), 3859–3865. https: //doi.org/10.1016/j.comnet.2012.10.004

    2012 doi

  29. [39]

    Hussam Hallak. 2018. Why We Need Private Web Archives: Almost Two-Thirds of Web Traffic IS NOT Publicly Archivable. https://ws- dl.blogspot.com/2018/07/2018-07-18-why-we-need-private-web.html

    2018

  30. [40]

    Helge Holzmann, Wolfgang Nejdl, and Avishek Anand. 2016. The Dawn of Today’s Popular Domains: A Study of the Archived German Web over 18 Years. InProceedings of the 16th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2016) . ACM, New Jersey, Newark, USA, 73–82. https:...

    2016

  31. [41]

    IIPC. 2024. WARC 1.1 Specification - Revisit Record. https://iipc.github.io/warc-specifications/specifications/warc-format/warc- 1.1/#revisit

    2024

  32. [42]

    Ian Jacobs and Norman Walsh. 2004. Architecture of the World Wide Web, Volume One. World Wide Web Consortium (W3C). https://www.w3.org/TR/webarch/

    2004

  33. [43]

    Nwala, Gavindya Jayawardena, Jian Wu, Sampath Jayarathna, Michael L

    Yasith Jayawardana, Alexander C. Nwala, Gavindya Jayawardena, Jian Wu, Sampath Jayarathna, Michael L. Nelson, and C. Lee Giles

  34. [44]

    Jones, Herbert Van de Sompel, Harihar Shankar, Martin Klein, Richard Tobin, and Claire Grover

    Shawn M. Jones, Herbert Van de Sompel, Harihar Shankar, Martin Klein, Richard Tobin, and Claire Grover. 2016. Scholarly Context Adrift: Three out of Four URI References Lead to Changed Content. PLOS ONE 11, 12 (2016), e0167475. https://doi.org/10.1371/journal. pone.0167475

    2016 doi

  35. [45]

    Jones, Martin Klein, Herbert Van de Sompel, Michael L

    Shawn M. Jones, Martin Klein, Herbert Van de Sompel, Michael L. Nelson, and Michele C. Weigle. 2021. Interoperability for Accessing Versions of Web Resources with the Memento Protocol. In The Past Web: Exploring Web Archives. Springer International Publishing

    2021

  36. [46]

    Brewster Kahle. 2019. The Internet Archive is working to provide free and open access to all the world’s knowledge. Twitter. https://x.com/brewster_kahle/status/1118172506777509890 Accessed: 2024-10-09

    2019

  37. [47]

    Alkwai, Sawood Alam, Michael L

    Mat Kelly, Lulwah M. Alkwai, Sawood Alam, Michael L. Nelson, Michele C. Weigle, and Herbert Van de Sompel. 2017. Impact of URI Canonicalization on Memento Count. In Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries (JCDL) (Toronto, Ontario, Canada) (JCDL ’...

    2017

  38. [48]

    Martin Klein and Lyudmila Balakireva. 2020. On the Persistence of Persistent Identifiers of the Scholarly Web. In Digital Libraries for Open Knowledge. Springer International Publishing, Cham, 102–115. https://doi.org/10.1007/978-3-030-54956-5_8

    2020 doi

  39. [49]

    Martin Klein, Herbert Van de Sompel, Robert Sanderson, Harihar Shankar, Lyudmila Balakireva, Ke Zhou, and Richard Tobin. 2014. Scholarly Context Not Found: One in Five Articles Suffers from Reference Rot. PLOS ONE 9, 12 (2014), e115253. https://doi.org/10.1371/ journal.pone.0115253

    2014

  40. [50]

    Wallace Koehler. 1999. An Analysis of Web Page and Web Site Constancy and Permanence. Journal of the American Society for Information Science 50, 2 (1999), 162–180. https://doi.org/10.1002/(SICI)1097-4571(1999)50:2<162::AID-ASI7>3.0.CO;2-B

    1999 doi

  41. [51]

    Wallace Koehler. 2002. Web Page Change and Persistence: A Four-Year Longitudinal Study. Journal of the American Society for Information Science and Technology 53, 2 (2002), 162–171. https://doi.org/10.1002/asi.10018

    2002 doi

  42. [52]

    Ilya Kreymer. 2024. Indexing in PyWB. https://pywb.readthedocs.io/en/latest/manual/indexing.html

    2024

  43. [53]

    Ilya Kreymer. 2024. Web Archive Indexing Tools. https://github.com/ikreymer/webarchive-indexing

    2024

  44. [54]

    Daniela Major. 2021. The Problem of Web Ephemera. In The Past Web: Exploring Web Archives. Springer International Publishing, Cham, 5–10. https://doi.org/10.1007/978-3-030-63291-5_1

    2021 doi

  45. [55]

    Daniela Major and Daniel Gomes. 2021. Web Archives Preserve Our Digital Collective Memory. InThe Past Web: Exploring Web Archives. Springer International Publishing, Cham, 11–19. https://doi.org/10.1007/978-3-030-63291-5_2

    2021 doi

  46. [56]

    Luca Messarra, Chris Freeland, and Juliya Ziskina (Eds.). 2024. Vanishing Culture: A Report on Our Fragile Cultural Record . Internet Archive. https://archive.org/details/vanishing-culture-report

    2024

  47. [57]

    Michael L. Nelson. 2021. Not Your Parents’ Web: The Scope and Archiving of the Modern Web. https://ws-dl.blogspot.com/2021/10/2021- 10-20-not-your-parents-web-scope.html

    2021

  48. [58]

    Nelson and Herbert Van de Sompel

    Michael L. Nelson and Herbert Van de Sompel. 2019. Adding the Dimension of Time to HTTP. In SAGE Handbook of Web History. SAGE Publishing

    2019

  49. [59]

    Alexandros Ntoulas, Junghoo Cho, and Christopher Olston. 2004. What's New on the Web? The Evolution of the Web from a Search Engine Perspective. In Proceedings of the 13th International World Wide Web Conference (WWW) . ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/988...

    2004

  50. [60]

    V. Pareto. 1896. Cours d’Economie Politique Professe a l’Universite de Lausanne . Number v. 1. F. Rouge. https://books.google.com/books? id=KjnhnQAACAAJ

  51. [61]

    SalahEldeen and Michael L

    Hany M. SalahEldeen and Michael L. Nelson. 2012. Losing My Revolution: How Many Resources Shared on Social Media Have Been Lost?. In Proceedings of the 16th International Conference on Theory and Practice of Digital Libraries (TPDL) (TPDL ’12) . Springer, 125–137. https://doi....

    2012 doi

  52. [62]

    SalahEldeen and Michael L

    Hany M. SalahEldeen and Michael L. Nelson. 2013. Carbon dating the web: estimating the age of web resources. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13 Companion). Association for Computing Machinery, New York, NY, U...

    2013

  53. [63]

    Mohammed Nauman Siddique and Sawood Alam. 2019. TweetedAt: Finding Tweet Timestamps for Pre and Post Snowflake Tweet IDs. https://ws-dl.blogspot.com/2019/08/2019-08-03-tweetedat-finding-tweet.html

    2019

  54. [64]

    Aaron Swartz. 2013. Zipnum and CDX Cluster Merging. https://web.archive.org/web/20160804001009/http://aaron.blog.archive.org/ 2013/05/28/zipnum-and-cdx-cluster-merging/

    2013

  55. [65]

    Nelson, Robert Sanderson, Lyudmila L

    Herbert Van de Sompel, Michael L. Nelson, Robert Sanderson, Lyudmila L. Balakireva, Scott Ainsworth, and Harihar Shankar. 2009. Memento: Time Travel for the Web . Technical Report arXiv:0911.1112. arXiv

    2009 arXiv

  56. [66]

    Jeffrey S. Vitter. 1985. Random Sampling with a Reservoir. ACM Trans. Math. Software 11, 1 (1985), 37–57. https://doi.org/10.1145/3147. 3165

    1985 doi

  57. [67]

    Eveline Vlassenroot, Sally Chambers, Emmanuel Di Pretoro, Friedel Geeraert, Gerald Haesendonck, Alejandra Michel, and Peter Mechant. 2019. Web archives as a data resource for digital scholars. International Journal of Digital Humanities 1 (2019), 85–111. https://doi.org/10.100...

    2019 doi

  58. [68]

    Peter Webster. 2021. Digital Archaeology in the Web of Links: Reconstructing a Late-1990s Web Sphere. In The Past Web: Exploring Web Archives. Springer International Publishing, Cham, 155–164. https://doi.org/10.1007/978-3-030-63291-5_12

    2021 doi

  59. [69]

    Michele C. Weigle. 2024. Some URLs are Immortal, Most are Not. https://ws-dl.blogspot.com/2024/09/2024-09-20-some-urls-are- immortal-most.html

    2024

  60. [70]

    Weigle, Michael L

    Michele C. Weigle, Michael L. Nelson, Sawood Alam, and Mark Graham. 2024. Right HTML, Wrong JSON: Challenges in Replaying Archived Webpages Built with Client-Side Rendering. In Proceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL) (Santa Fe, New Mexico,...

    2024

  61. [71]

    WS-DL Research Group. 2024. Not Your Parents Web Project. https://github.com/oduwsdl/nypw

    2024

  62. [72]

    Baoning Wu and Brian D. Davison. 2005. Cloaking and Redirection: A Preliminary Study. In AIRWeb 2005, First International Workshop on Adversarial Information Retrieval on the Web, co-located with the WWW conference, Chiba, Japan, May 2005 . 7–16. http://airweb.cse. lehigh.edu/...

    2005

  63. [73]

    Zittrain, John Bowers, and Clare Stanton

    Jonathan L. Zittrain, John Bowers, and Clare Stanton. 2021. The Paper of Record Meets an Ephemeral Web: An Examination of Linkrot and Content Drift within The New York Times. SSRN Electronic Journal (2021), 1–13. https://doi.org/10.2139/ssrn.3833133

    2021 doi

  64. [2020]

    In 2020 IEEE International Conference on Big Data (Big Data)

    Modeling Updates of Scholarly Webpages Using Archived Data. In 2020 IEEE International Conference on Big Data (Big Data) . IEEE, 1868–1877. https://doi.org/10.1109/BigData50022.2020.9377796

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.