Pith. sign in

REVIEW 4 major objections 4 minor 34 references

Making Recommendations from Web Archives for "Lost" Web Pages

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Web archives can replace a bare 404 with ranked recommendations built from the requested URI alone.

desk verdict Honest, clearly written systems paper with a useful URI diagnostic, but the central recommendation claim is untested and the headline F1 is filtered. read the letter →

arxiv 1908.02819 v1 pith:WOTHW6PG submitted 2019-08-07 cs.DL cs.IR

Lulwah M. Alkwai , Michael L. Nelson , Michele C. Weigle This is my paper · ORCID
classification cs.DLcs.IR
keywords webarchivesURIclassificationDMOZrecommendationsystemWaybackMachineMemento-Damagedeeplostpages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to change how web archives answer when a requested page is missing. Instead of returning only "not archived," the archive can treat the requested URI as a query: classify it into DMOZ categories using machine learning on the URI text, collect other archived URIs in those categories, and rank them by temporal closeness, popularity, URI similarity, and archival quality. The headline measured result is that first-level category assignment reaches F1 = 0.59 when the URI is reduced to character n-grams after removing numerals and the top-level domain; deep-level classification is much weaker (micro-F1 = 0.30 at level two, 0.15 at level three). The paper also shows that real Wayback requests are mostly short, depth-0 URIs with few dictionary words, which is exactly the regime where URI-only classification struggles. If the approach holds, archives could enrich both 404 and 200 responses with recommendations the user did not know existed.

What carries the argument

The load-bearing object is the URI string itself, treated as a query. Tokenization uses the all-grams method: after removing the top-level domain and numerals, the URI is decomposed into overlapping 4- to 8-character grams; these grams feed a Naive Bayes classifier for first-level DMOZ category assignment. Deep classification follows a search-and-prune hierarchy: DMOZ entries are indexed by category, cosine similarity scores each category against the request, the top 10 candidate categories form a tree that is pruned with ancestor assistance, and a classifier selects the final path. Candidate ranking is a weighted sum $score = w_t t + w_p p + w_s s + w_q q$ over temporal similarity, popularity, URI similarity, and archival quality. The whole design assumes that co-membership in a DMOZ category, predicted from the URI alone, is a workable proxy for "similar to the lost page."

What would settle it

A controlled study in which users who request unarchived URIs rate the top-ranked same-DMOZ-category recommendations against shuffled archived pages; if the system's recommendations are not rated significantly more relevant than the shuffled baseline, the central claim is unsupported. A simpler computational check would compare recommended candidates' page content similarity to the lost page's content, where obtainable, and show whether same-category URIs really do have related content.

Watch

Extended reading notes

Core claim

The central claim is that a "lost" web page—one absent from both the live web and the archive—can be matched to archived near-neighbors using only its URI. The authors' pipeline first checks DMOZ and Wikipedia for an existing category; for unclassified URIs, a Naive Bayes classifier trained on DMOZ assigns a top-level topic from all character grams of the URI with TLD and numerals removed (best F1 = 0.59), then a cosine-similarity and tree-pruning procedure picks a deep category (micro-F1 = 0.30 at level two, 0.15 at level three). Candidate URIs from the predicted category are filtered for archival presence and scored by the weighted sum of temporal similarity, Alexa-based popularity combined with archive capture count, Jaccard URI similarity, and Memento-Damage-derived quality. The authors argue this can enhance both HTTP 404 and HTTP 200 archive responses.

Load-bearing premise

The load-bearing premise is that two pages in the same DMOZ category are what a user would accept as a substitute for the missing page; the paper does not test this with users, so if category co-membership does not track user relevance, the recommendations fail.

Editorial extensions

If this is right

  • Archives can surface related material without building a content index, since only the URI and a category directory are needed.
  • For the 13% of sampled Wayback requests already present in DMOZ or Wikipedia, recommendations can bypass fragile URI classification entirely.
  • Because most real requests are depth-0 URIs, domain-level signals such as long strings and dictionary words in the hostname will dominate in practice, and path-level signals will rarely help.
  • The ranking weights in $score = w_t t + w_p p + w_s s + w_q q$ are tunable, so archives can favor near-in-time captures, popular domains, URI similarity, or cleanly archived pages depending on user needs.
  • The weak deep-level F1 of 0.30 means first-level category matches will often be the practical ceiling until richer signals are added.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same URI-only pipeline could attach to live-web 404 responses as well as archive responses, since it never requires page content.
  • A further extension: the archive's own request logs could supply the popularity term and even explicit relevance feedback, allowing the ranking weights to be learned instead of hand-set.
  • The dictionary-poor nature of real requests suggests that further gains in URI classification will come from character-level or subword models rather than from word lexicons.
  • If deep-level F1 remains near 0.30, fine-grained same-subcategory recommendations will likely need content surrogates such as link neighborhoods or titles, leaving URI-only methods suited to broad-topic suggestions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a pipeline for augmenting web-archive 404/200 responses with recommendations of archived URIs related to a 'lost' requested URI. After checking whether the URI is already categorized in DMOZ or Wikipedia, the pipeline classifies the URI string with a Naïve Bayes first-level classifier, then performs deep classification using cosine similarity and a pruned category tree, collects candidate URIs from the predicted category, filters them to those present in the archive, and ranks the survivors by a weighted combination of temporal similarity, popularity, URI similarity, and archival quality. The evaluation reports a first-level F1 of 0.59 for all-grams from the URI after removing the TLD and numbers, deep-level micro-average F1 values of 0.30 at level 2 and 0.15 at level 3, and compares URI pattern statistics of correctly classified DMOZ URIs with those of real Wayback Machine access-log requests.

Significance. If the pipeline worked as intended, it would address a real gap in web-archive access by turning binary availability responses into useful alternative-page recommendations without needing page content. The authors should be credited for careful dataset construction, for concretely specifying the ontology lookups, and for explicitly quantifying the mismatch between DMOZ URIs and real archive requests. However, the headline classification numbers are obtained under a favorable filtered test set, and no experiment measures whether the recommended URIs are actually useful to users. The current evidence supports the components only weakly and does not yet establish the system's value proposition.

major comments (4)
  1. [Section 4.2.1 / Table 6] The reported first-level F1=0.59 is computed only on test URIs that survive the filter 'URIs that contain tokens not seen in the training set'; the size and composition of the excluded fraction are not reported. A deployed classifier cannot refuse requests with novel tokens, and the access-log analysis in Section 6 suggests such requests are common. Please report the F1 on the unfiltered test set and on a sample of Wayback access-log URIs.
  2. [Section 4.3 / Figure 5] Deep classification, which is the step that actually determines the candidate category, achieves only Mi-F1=0.30 at level 2 and 0.15 at level 3, and the paper itself concludes that URI-only evidence is 'not enough' for deep classification. Because Step 2 collects candidates from the predicted deep category, errors at this stage produce candidate sets from the wrong topic area, and no downstream component can repair that. The manuscript needs an analysis of how deep-classification accuracy propagates to candidate quality, or an argument that first-level categories suffice.
  3. [Section 6 / Section 4.4] No experiment measures end-to-end recommendation quality. The only illustration is the single worked example in Section 5, and there are no relevance judgments, user study, or comparison with baselines such as domain-only or lexical-signature recommendations. The assumption that co-membership in a predicted DMOZ category implies user relevance is untested. At minimum, a manual relevance assessment on a sample of real requests and a baseline comparison are needed to support the claim that the recommendations are useful.
  4. [Section 4.4.1 / Eq. (2)] As written, Eq. (2) defines a temporal distance (|rd - cd|/(ud - ed)), not a similarity: a candidate whose Memento-Datetime exactly equals the request datetime receives t=0 and is penalized in Eq. (1) if the weight wt is positive. If the implementation actually uses 1 - t, an inverse, or a negated distance, the equation and the variable names must be corrected; as it stands, the ranking feature is inconsistent with its stated purpose.
minor comments (4)
  1. [Section 4.4 / example] The phrase 'Using equal weights (wt = wp = wq)' omits ws; if ws was intentionally zero or also equal, that should be stated.
  2. [Table 6] The 'F1 score' column would be clearer if its averaging procedure were stated; the separate Micro and Macro columns suggest the first column is a third averaging, but the text does not define it.
  3. [Table 8] The percentages in Table 8 are computed over correctly classified URIs only; the denominator (2,996) appears later in the text, but it should be stated next to the table.
  4. [Table 1 / Section 3.1] The category 'Kids_ands_Teens' in Table 1 appears to be a typo for 'Kids_and_Teens'; also, the text says those categories were filtered out, but Table 1 still lists them, which is confusing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the classification results are benchmarked on held-out DMOZ data and the ranking formula is defined with fixed features rather than fitted to the outcome.

full rationale

The paper's derivation chain is a concrete pipeline: check DMOZ/Wikipedia, classify the requested URI with a Naive Bayes classifier trained on DMOZ, gather same-category DMOZ candidates, filter by archive presence, and rank with a fixed weighted sum of temporal similarity, popularity, URI similarity, and archival quality. The headline F1=0.59 is measured on held-out DMOZ entries: 'we took the DMOZ dataset and created a 10-fold cross-validation set, using 90% for training and 10% for testing.' Thus the reported classifier performance is not equivalent to its training input. Similarly, the deep-level Mi-F1 values (0.30 at level 2, 0.15 at level 3) are computed against held-out DMOZ URIs and are frankly reported as low. The ranking equation, score = wt*t + wp*p + ws*s + wq*q, is defined a priori with no parameters fit to user judgments or to the evaluation set, and no fitted value is later renamed as a prediction. Citations to the authors' prior work support datasets and utility tools (e.g., access-log selection, Memento-Damage) but are not the load-bearing justification for the classification claim, which rests on an independent held-out benchmark. The absence of an end-to-end user-relevance evaluation is a validity gap rather than circularity: the recommendation value proposition is asserted, not derived from a fitted quantity. Under the stated rules, this is a non-finding with score 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result depends on the DMOZ ontology and the 'lost URI' premise rather than on invented physical or formal entities. The main free parameters are the ranking weights, which are not learned or justified, plus popularity-normalization constants set by hand. No new entities are introduced.

free parameters (5)
  • Ranking weights w_t, w_p, w_s, w_q = unspecified; example uses equal weights
    Equation 1 defines the final score as a weighted sum, but the weights are never learned or justified. Different weight choices would change the ranked output, and no evaluation determines good values.
  • Alexa lowest ranking x = 30,000,000
    Set by hand in Section 4.4.2 as the current lowest Alexa ranking; used to normalize the popularity feature.
  • Maximum archive count m = 538,300
    Set to the number of times google.com has been archived; used to normalize log n in the popularity equation.
  • Top-N candidate categories N = 10
    Adopted from Xue et al. [32]; the number of candidate categories selected in deep-level classification.
  • Long-string threshold = 10 contiguous letters
    Used to define URI patterns in Section 3.1 and Section 6; based on the average English word length of 5 letters.
assumptions (4)
  • domain assumption The requested page is 'lost': not available on the live web or in the archive, so only the URI is available as evidence.
    Stated in the Introduction and Section 4; this premise justifies the URI-only classification design.
  • domain assumption DMOZ categories are a valid ontology for topical relevance, and same-category co-membership is a good basis for recommendations.
    Steps 2 and 4 collect and rank candidates from the predicted DMOZ category; no relevance judgments validate this mapping.
  • domain assumption DMOZ entries are likely to be archived, so filtering by archival presence still leaves usable candidates.
    Section 3.1 motivates DMOZ as a candidate source by citing prior estimates of archival coverage.
  • domain assumption A Naive Bayes classifier trained on DMOZ URIs transfers to real user requests to the Wayback Machine.
    Section 6 shows real Wayback URIs contain fewer dictionary words and have different depth distributions, which the authors acknowledge may reduce classification accuracy in practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Making Recommendations from Web Archives for "Lost" Web Pages." pith.science (2026). https://pith.science/paper/WOTHW6PG

@misc{pith2026190802819,
  author       = {Pith},
  title        = {Pith review of: Making Recommendations from Web Archives for "Lost" Web Pages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WOTHW6PG}},
  note         = {Machine review of arXiv:1908.02819}
}
read the original abstract

When a user requests a web page from a web archive, the user will typically either get an HTTP 200 if the page is available, or an HTTP 404 if the web page has not been archived. This is because web archives are typically accessed by URI lookup, and the response is binary: the archive either has the page or it does not, and the user will not know of other archived web pages that exist and are potentially similar to the requested web page. In this paper, we propose augmenting these binary responses with a model for selecting and ranking recommended web pages in a Web archive. This is to enhance both HTTP 404 responses and HTTP 200 responses by surfacing web pages in the archive that the user may not know existed. First, we check if the URI is already classified in DMOZ or Wikipedia. If the requested URI is not found, we use ML to classify the URI using DMOZ as our ontology and collect candidate URIs to recommended to the user. Next, we filter the candidates based on if they are present in the archive. Finally, we rank candidates based on several features, such as archival quality, web page popularity, temporal similarity, and URI similarity. We calculated the F1 score for different methods of classifying the requested web page at the first level. We found that using all-grams from the URI after removing numerals and the TLD produced the best result with F1=0.59. For second-level classification, the micro-average F1=0.30. We found that 44.89% of the correctly classified URIs contained at least one word that exists in a dictionary and 50.07% of the correctly classified URIs contained long strings in the domain. In comparison with the URIs from our Wayback access logs, only 5.39% of those URIs contained only words from a dictionary, and 26.74% contained at least one word from a dictionary. These percentages are low and may affect the ability for the requested URI to be correctly classified.

Figures

Figures reproduced from arXiv: 1908.02819 by the authors.

Figure 1
Figure 1. The actual response to the requested URI [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. ODU main page found in DMOZ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Searching for the request http://odu.edu in Wikipedia resulted in finding the Wikipedia web page https://en.wikipedia. org/wiki/Old_Dominion_University that contains the requested URI as the official website in the external link section. We use other web pages in the same categories (at the end of the page) as candidate web pages [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The process of pruning a hierarchical tree using [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Performance on classifying to different levels us [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 33 canonical work pages

  1. [1]

    Ainsworth, Ahmed Alsum, Hany M

    Scott G. Ainsworth, Ahmed Alsum, Hany M. SalahEldeen, Michele C. Weigle, and Michael L. Nelson. 2011. How Much of the Web is Archived?. In Proceedings of the 11th IEEE/ACM Joint Conference on Digital Libraries (JCDL) . 133–136

    work page 2011

  2. [2]

    Sawood Alam and Michael L Nelson. 2016. MemGator-A portable concurrent memento aggregator: Cross-platform CLI and server binaries in Go. InProceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries . ACM, 243–244

    work page 2016

  3. [3]

    Alkwai, Michael L

    Lulwah M. Alkwai, Michael L. Nelson, and Michele C. Weigle. 2017. Comparing the Archival Rate of Arabic, English, Danish, and Korean Language Web Pages. ACM Transactions on Information Systems (TOIS) 36, 1 (2017), 1:1–1:34

    work page 2017

  4. [4]

    Yasmin AlNoamany. 2016. Using Web Archives to Enrich the Live Web Experience Through Storytelling. Ph.D. Dissertation. Old Dominion University

    work page 2016

  5. [5]

    Weigle, and Michael L

    Yasmin AlNoamany, Ahmed AlSum, Michele C. Weigle, and Michael L. Nelson

    work page

  6. [6]

    Weigle, and Michael L

    Yasmin AlNoamany, Michele C. Weigle, and Michael L. Nelson. 2013. Access Patterns for Robots and Humans in Web Archives. In Proceedings of the 13th IEEE/ACM Joint Conference on Digital Libraries (JCDL) . 339–348

    work page 2013

  7. [7]

    Ahmed AlSum. 2014. Web Archive Services Framework for Tighter Integration Between the Past and Present Web . Ph.D. Dissertation. Old Dominion University

    work page 2014

  8. [8]

    Eda Baykan, Monika Henzinger, Ludmila Marian, and Ingmar Weber. 2009. Purely URL-based Topic Classification. In Proceedings of the 18th International conference on World Wide Web (WWW). 1109–1110

    work page 2009

Show all 34 references
  1. [9]

    Eda Baykan, Monika Henzinger, Ludmila Marian, and Ingmar Weber. 2011. A Comprehensive Study of Features and Algorithms for URL-based Topic Classifi- cation. ACM Transactions on the Web (TWEB) 5, 3 (2011), 15

    2011

  2. [10]

    Brunelle, Mat Kelly, Hany SalahEldeen, Michele C

    Justin F. Brunelle, Mat Kelly, Hany SalahEldeen, Michele C. Weigle, and Michael L. Nelson. 2015. Not all mementos are created equal: Measuring the impact of missing resources. International Journal on Digital Libraries (IJDL) 16, 3-4 (2015), 283–301

    2015

  3. [11]

    M Indra Devi, R Rajaram, and K Selvakuberan. 2007. Machine learning tech- niques for automated web page classification using URL features. In Proceedings of the International Conference on Computational Intelligence and Multimedia Applications (ICCIMA), Vol. 2. 116–120

    2007

  4. [12]

    Jonathan Goldsmith. 2016. A Pythonic wrapper for the Wikipedia API. https: //github.com/goldsmith/Wikipedia. (2016)

    2016

  5. [13]

    Huurdeman, Anat Ben-David, Jaap Kamps, Thaer Samar, and Arjen P

    Hugo C. Huurdeman, Anat Ben-David, Jaap Kamps, Thaer Samar, and Arjen P. de Vries. 2014. Finding pages on the unarchived web. In Proceedings of the 14th IEEE/ACM Joint Conference on Digital Libraries (JCDL) . 331–340

    2014

  6. [14]

    Huurdeman, Jaap Kamps, Thaer Samar, Arjen P

    Hugo C. Huurdeman, Jaap Kamps, Thaer Samar, Arjen P. de Vries, Anat Ben- David, and Richard A. Rogers. 2015. Lost but not Forgotten: Finding Pages on the Unarchived Web. International Journal on Digital Libraries (IJDL) 16, 3-4 (2015), 247–265

    2015

  7. [15]

    Shawn M. Jones. 2018. A Preview of MementoEmbed: Embeddable Surrogates for Archived Web Pages. https://ws-dl.blogspot.com/2018/08/ 2018-08-01-preview-of-mementoembed.html. (2018)

    2018

  8. [16]

    Brewster Kahle. 2019. 703,726,890,000 URL’s now in the @waybackmachine by the @internetarchive ! (703 billion) Over a billion more added each week. The Web is a grand experiment in sharing and giving. Loving it! http://web.archive.org/. https: //twitter.com/brewster_kahle/stat...

    2019

  9. [17]

    Min-Yen Kan. 2004. Web Page Classification Without the Web Page. InProceedings of the 13th International World Wide Web conference on Alternate Track Papers and Posters. 262–263

    2004

  10. [18]

    Min-Yen Kan and Hoang Oanh Nguyen Thi. 2005. Fast Webpage Classification Using URL Features. In Proceedings of the 14th ACM International Conference on Information and Knowledge Management (CKIM) . 325–326

    2005

  11. [19]

    Nattiya Kanhabua, Philipp Kemkes, Wolfgang Nejdl, Tu Ngoc Nguyen, Felipe Reis, and Nam Khanh Tran. 2016. How to search the Internet Archive without indexing it. In Proceedings of the International conference on Theory and Practice of Digital Libraries (TPDL) . 147–160

    2016

  12. [20]

    Martin Klein and Michael L. Nelson. 2014. Moved but not Gone: An Evaluation of Real-time Methods for Discovering Replacement Web Pages. International Journal on Digital Libraries (IJDL) 14, 1-2 (2014), 17–38

    2014

  13. [21]

    Martin Majlis. 2019. Python wrapper for Wikipedia. https://github.com/ martin-majlis/Wikipedia-API. (2019)

    2019

  14. [22]

    David D Palmer. 1997. A trainable rule-based algorithm for word segmentation. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics. Association ...

    1997

  15. [23]

    John R Pierce. 2012. An introduction to information theory: symbols, signals and noise. Courier Corporation

    2012

  16. [24]

    Rajalakshmi and Chandrabose Aravindan

    R. Rajalakshmi and Chandrabose Aravindan. 2011. Naive bayes approach for website classification. In Proceedings of the Information Technology and Mobile Communication. Communications in Computer and Information Science . Vol. 147

    2011

  17. [25]

    R Rajalakshmi and Chandrabose Aravindan. 2013. Web Page Classification Using N-gram Based URL Features. In Proceedings of the 5th International Conference on Advanced Computing (ICoAC). 15–21

    2013

  18. [26]

    Erika Siregar. 2017. Deploying the Memento-Damage Service. https://ws-dl. blogspot.com/2017/11/2017-11-22-deploying-memento-damage.html. (2017)

    2017

  19. [27]

    Aixin Sun and Ee-Peng Lim. 2001. Hierarchical text classification and evaluation. In Data Mining, 2001. ICDM 2001, Proceedings IEEE International Conference on . IEEE, 521–528

    2001

  20. [28]

    Brad Tofel. 2007. Wayback for Accessing Web Archives. In 7th International Web Archiving Workshop (IW A W’07)

    2007

  21. [29]

    Nelson, and Robert Sanderson

    Herbert Van de Sompel, Michael L. Nelson, and Robert Sanderson. 2013. HTTP framework for time-based access to resource states – Memento, Internet RFC

    2013

  22. [30]

    Wikipedia. [n. d.]. History of Wikipedia. https://en.wikipedia.org/wiki/History_ of_Wikipedia. ([n. d.])

  23. [31]

    Wikipedia. [n. d.]. List of Wikipedias. https://en.wikipedia.org/wiki/List_of_ Wikipedias. ([n. d.])

  24. [32]

    Gui-Rong Xue, Dikan Xing, Qiang Yang, and Yong Yu. 2008. Deep classification in large-scale text hierarchies. InProceedings of the 31st annual International ACM SIGIR conference on Research and Development in Information Retrieval . 619–626

    2008

  25. [2014]

    International Journal on Digital Libraries (IJDL) 14, 3-4 (2014), 101–115

    Who and What Links to the Internet Archive. International Journal on Digital Libraries (IJDL) 14, 3-4 (2014), 101–115

    2014

  26. [7089]

    http://tools.ietf.org/html/rfc7089. (2013)

    2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.