{"id":"f1679aa7-4362-4c7b-b2ed-5224fa24eb33","arxiv_id":"1908.02819","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A URI-only classifier and ranker recommends archived pages for lost web addresses, reaching 0.59 first-level F1, but lacks an end-to-end evaluation of the recommendations themselves.","lead":"When a web archive is missing a requested page, this paper proposes recommending other archived pages that may be similar, using only the web address. The authors test a URI classifier on a large web directory and reach F1=0.59 for first-level topic categories, but the final recommendation list is not evaluated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"End-to-end recommendation quality is never measured, and the reported F1 comes from filtered DMOZ test data, so the central recommendation claim is unsupported.","rationale":"The reader's weakest assumption (same-category co-membership implies relevance) is real, but the more immediately load-bearing problem is that the quantitative evidence never reaches the recommendation claim. The system's only measured component, first-level classification, is evaluated on a DMOZ holdout after discarding test URIs with unseen tokens, and the paper's own access-log analysis shows the target URIs have different lexical properties. Deep classification, which actually drives candidate collection for URIs not already in DMOZ or Wikipedia, has very low Mi-F1 and is not evaluated on access-log URIs. Since ranking operates only on candidates collected from a predicted category, wrong classification cannot be repaired downstream. No end-to-end or user-facing evaluation exists. This does not prove the approach is wrong; it means the central claim is currently unverified, which is exactly the conditional situation the reader described. I set verdict_should_be to UNCHANGED because the concern reinforces the CONDITIONAL verdict rather than overturning it. Agreement is partial: the reader focused on the relevance proxy, while I focus on the transferability of the classification evaluation; both point to the same missing end-to-end validation.","tokens_in":13887,"tokens_out":6215,"duration_ms":70066,"concrete_test":"Run the full pipeline on a random sample of 200 URIs from the filtered Feb 8, 2012 Wayback access-log set, produce top-5 recommendations per URI, and have at least two independent annotators judge whether each recommendation is a relevant substitute for the requested URI. Report precision@5 against a random-DMOZ-category control; this single end-to-end test determines whether the recommendation claim has empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that augmenting archive responses with recommended URIs is useful. For that claim to hold, the URI classifier must place real \"lost\" requests into DMOZ categories containing relevant pages, and the ranker must surface them. Two linked gaps undermine this. First, the headline F1=0.59 (Section 4.2.1, Table 6) is measured after \"filtered out URIs that contain tokens not seen in the training set,\" which removes the hardest test instances; a deployable classifier cannot refuse unseen-token requests. Second, the evaluation is performed only on DMOZ URIs, not on the target population: the Wayback access-log analysis in Section 6 shows only 26.74% of real request URIs contain at least one dictionary word versus 44.89% of correctly classified DMOZ URIs, and only 3.24% have long strings in the path versus 13.45%. The paper itself concedes these differences \"may affect the ability for the requested URI to be correctly classified.\" Because Step 2's deep classification (Mi-F1=0.30 at level 2, 0.15 at level 3) is both weak and unvalidated on access-log URIs, candidate collection for the roughly 87% of requests not already in DMOZ/Wikipedia (Section 4.1) can retrieve pages from the wrong category; ranking (Section 4.4) cannot repair a wrong candidate set. No experiment measures end-to-end recommendation quality or user relevance, so the value proposition is unsupported by the presented evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for augmenting web-archive 404/200 responses with recommendations of archived URIs related to a 'lost' requested URI. After checking whether the URI is already categorized in DMOZ or Wikipedia, the pipeline classifies the URI string with a Naïve Bayes first-level classifier, then performs deep classification using cosine similarity and a pruned category tree, collects candidate URIs from the predicted category, filters them to those present in the archive, and ranks the survivors by a weighted combination of temporal similarity, popularity, URI similarity, and archival quality. The evaluation reports a first-level F1 of 0.59 for all-grams from the URI after removing the TLD and numbers, deep-level micro-average F1 values of 0.30 at level 2 and 0.15 at level 3, and compares URI pattern statistics of correctly classified DMOZ URIs with those of real Wayback Machine access-log requests.","tokens_in":14184,"tokens_out":4946,"duration_ms":51374,"significance":"If the pipeline worked as intended, it would address a real gap in web-archive access by turning binary availability responses into useful alternative-page recommendations without needing page content. The authors should be credited for careful dataset construction, for concretely specifying the ontology lookups, and for explicitly quantifying the mismatch between DMOZ URIs and real archive requests. However, the headline classification numbers are obtained under a favorable filtered test set, and no experiment measures whether the recommended URIs are actually useful to users. The current evidence supports the components only weakly and does not yet establish the system's value proposition.","major_comments":[{"comment":"The reported first-level F1=0.59 is computed only on test URIs that survive the filter 'URIs that contain tokens not seen in the training set'; the size and composition of the excluded fraction are not reported. A deployed classifier cannot refuse requests with novel tokens, and the access-log analysis in Section 6 suggests such requests are common. Please report the F1 on the unfiltered test set and on a sample of Wayback access-log URIs.","section":"Section 4.2.1 / Table 6"},{"comment":"Deep classification, which is the step that actually determines the candidate category, achieves only Mi-F1=0.30 at level 2 and 0.15 at level 3, and the paper itself concludes that URI-only evidence is 'not enough' for deep classification. Because Step 2 collects candidates from the predicted deep category, errors at this stage produce candidate sets from the wrong topic area, and no downstream component can repair that. The manuscript needs an analysis of how deep-classification accuracy propagates to candidate quality, or an argument that first-level categories suffice.","section":"Section 4.3 / Figure 5"},{"comment":"No experiment measures end-to-end recommendation quality. The only illustration is the single worked example in Section 5, and there are no relevance judgments, user study, or comparison with baselines such as domain-only or lexical-signature recommendations. The assumption that co-membership in a predicted DMOZ category implies user relevance is untested. At minimum, a manual relevance assessment on a sample of real requests and a baseline comparison are needed to support the claim that the recommendations are useful.","section":"Section 6 / Section 4.4"},{"comment":"As written, Eq. (2) defines a temporal distance (|rd - cd|/(ud - ed)), not a similarity: a candidate whose Memento-Datetime exactly equals the request datetime receives t=0 and is penalized in Eq. (1) if the weight wt is positive. If the implementation actually uses 1 - t, an inverse, or a negated distance, the equation and the variable names must be corrected; as it stands, the ranking feature is inconsistent with its stated purpose.","section":"Section 4.4.1 / Eq. (2)"}],"minor_comments":[{"comment":"The phrase 'Using equal weights (wt = wp = wq)' omits ws; if ws was intentionally zero or also equal, that should be stated.","section":"Section 4.4 / example"},{"comment":"The 'F1 score' column would be clearer if its averaging procedure were stated; the separate Micro and Macro columns suggest the first column is a third averaging, but the text does not define it.","section":"Table 6"},{"comment":"The percentages in Table 8 are computed over correctly classified URIs only; the denominator (2,996) appears later in the text, but it should be stated next to the table.","section":"Table 8"},{"comment":"The category 'Kids_ands_Teens' in Table 1 appears to be a typo for 'Kids_and_Teens'; also, the text says those categories were filtered out, but Table 1 still lists them, which is confusing.","section":"Table 1 / Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an early-version report of an ongoing project. The dataset construction is careful and the limitations are honestly stated, but the central recommendation claim is currently unsupported because the evaluation stops at component-level classification accuracy. I do not see a load-bearing error that forces rejection; the gaps are addressable with additional experiments, so major revision seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid workshop-style paper that overreaches a bit in its framing. The genuinely new thing is the end-to-end pipeline: classify a lost URI into a DMOZ/Wikipedia category using only the URI string, then filter and rank archived candidates. Nobody in the cited prior work put those pieces together for archive 404/200 augmentation. The paper also earns credit for honesty: it reports the deep classification numbers (Mi-F1 0.30 at level 2, 0.15 at level 3) without spin, and it openly documents the mismatch between DMOZ URIs and real Wayback requests (26.74% vs 44.89% dictionary words, 3.24% vs 13.45% long path strings). That diagnostic analysis of URI patterns is genuinely useful and could inform future work.\n\nThe soft spots are in proportion to how central they are. The headline F1=0.59 is computed after filtering out test URIs with tokens unseen in training. That is selection on the hardest instances, and a deployed classifier cannot refuse those requests. The paper follows Baykan et al. in doing this, but the excluded fraction is not reported, and the F1 is the paper's main quantitative support. Second, the ranking step—where the user-facing value lives—has no evaluation at all. The four features and weights in Equation 1 are plausible but never tested against user relevance or any proxy. Third, the deep classification is weak enough that candidate collection for the 87% of requests not already in DMOZ/Wikipedia will often pull from the wrong category, and ranking cannot repair a wrong candidate set. The paper concedes most of this, so I don't think it is deceptive. It is an honest, preliminary system description.\n\nWho is this for? People building archive recommendation or missing-page recovery systems; the URI pattern statistics and pipeline description have real value. It is not a breakthrough and the main claim is unproven. But it deserves a serious referee rather than desk rejection: the methods are replicable, the writing is clear, and the limitations are stated. I would recommend conditional acceptance with the end-to-end claim reframed as a proposal, or a major revision that adds even a small user or relevance-judgment study.","headline":"Honest, clearly written systems paper with a useful URI diagnostic, but the central recommendation claim is untested and the headline F1 is filtered.","tokens_in":14754,"tokens_out":2001,"would_cite":true,"duration_ms":21901,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Web archives can replace a bare 404 with ranked recommendations built from the requested URI alone.","keywords":["web archives","URI classification","DMOZ","recommendation system","Wayback Machine","Memento-Damage","deep classification","lost web pages"],"falsifier":"A controlled study in which users who request unarchived URIs rate the top-ranked same-DMOZ-category recommendations against shuffled archived pages; if the system's recommendations are not rated significantly more relevant than the shuffled baseline, the central claim is unsupported. A simpler computational check would compare recommended candidates' page content similarity to the lost page's content, where obtainable, and show whether same-category URIs really do have related content.","tokens_in":13662,"feed_emoji":"🔗","tokens_out":7446,"duration_ms":76984,"temperature":0.7,"pith_summary":"The paper tries to change how web archives answer when a requested page is missing. Instead of returning only \"not archived,\" the archive can treat the requested URI as a query: classify it into DMOZ categories using machine learning on the URI text, collect other archived URIs in those categories, and rank them by temporal closeness, popularity, URI similarity, and archival quality. The headline measured result is that first-level category assignment reaches F1 = 0.59 when the URI is reduced to character n-grams after removing numerals and the top-level domain; deep-level classification is much weaker (micro-F1 = 0.30 at level two, 0.15 at level three). The paper also shows that real Wayback requests are mostly short, depth-0 URIs with few dictionary words, which is exactly the regime where URI-only classification struggles. If the approach holds, archives could enrich both 404 and 200 responses with recommendations the user did not know existed.","feed_headline":"A missing page's URL alone can drive archive recommendations","feed_subtitle":"URI-only classification reaches F1 0.59 for topic, then ranks archived candidates by time, popularity, and quality.","key_machinery":"The load-bearing object is the URI string itself, treated as a query. Tokenization uses the all-grams method: after removing the top-level domain and numerals, the URI is decomposed into overlapping 4- to 8-character grams; these grams feed a Naive Bayes classifier for first-level DMOZ category assignment. Deep classification follows a search-and-prune hierarchy: DMOZ entries are indexed by category, cosine similarity scores each category against the request, the top 10 candidate categories form a tree that is pruned with ancestor assistance, and a classifier selects the final path. Candidate ranking is a weighted sum $score = w_t t + w_p p + w_s s + w_q q$ over temporal similarity, popularity, URI similarity, and archival quality. The whole design assumes that co-membership in a DMOZ category, predicted from the URI alone, is a workable proxy for \"similar to the lost page.\"","core_discovery":"The central claim is that a \"lost\" web page—one absent from both the live web and the archive—can be matched to archived near-neighbors using only its URI. The authors' pipeline first checks DMOZ and Wikipedia for an existing category; for unclassified URIs, a Naive Bayes classifier trained on DMOZ assigns a top-level topic from all character grams of the URI with TLD and numerals removed (best F1 = 0.59), then a cosine-similarity and tree-pruning procedure picks a deep category (micro-F1 = 0.30 at level two, 0.15 at level three). Candidate URIs from the predicted category are filtered for archival presence and scored by the weighted sum of temporal similarity, Alexa-based popularity combined with archive capture count, Jaccard URI similarity, and Memento-Damage-derived quality. The authors argue this can enhance both HTTP 404 and HTTP 200 archive responses.","pith_inferences":["Beyond the paper's claims, the same URI-only pipeline could attach to live-web 404 responses as well as archive responses, since it never requires page content.","A further extension: the archive's own request logs could supply the popularity term and even explicit relevance feedback, allowing the ranking weights to be learned instead of hand-set.","The dictionary-poor nature of real requests suggests that further gains in URI classification will come from character-level or subword models rather than from word lexicons.","If deep-level F1 remains near 0.30, fine-grained same-subcategory recommendations will likely need content surrogates such as link neighborhoods or titles, leaving URI-only methods suited to broad-topic suggestions."],"forward_implications":["Archives can surface related material without building a content index, since only the URI and a category directory are needed.","For the 13% of sampled Wayback requests already present in DMOZ or Wikipedia, recommendations can bypass fragile URI classification entirely.","Because most real requests are depth-0 URIs, domain-level signals such as long strings and dictionary words in the hostname will dominate in practice, and path-level signals will rarely help.","The ranking weights in $score = w_t t + w_p p + w_s s + w_q q$ are tunable, so archives can favor near-in-time captures, popular domains, URI similarity, or cleanly archived pages depending on user needs.","The weak deep-level F1 of 0.30 means first-level category matches will often be the practical ceiling until richer signals are added."],"supporting_citations":[{"why":"Supplies the premise that many pages are unarchived and that DMOZ-listed pages are likely archived.","marker":"[1]"},{"why":"MemGator is the tool used to filter candidate URIs by whether a memento exists in the archive.","marker":"[2]"},{"why":"Provides the Wayback Machine access-log sample used to characterize real requests and evaluate URI properties.","marker":"[6]"},{"why":"Defines the three tokenization methods (tokens, all-grams from tokens, all-grams from URI) adopted for URI classification.","marker":"[9]"},{"why":"Provides the Memento-Damage algorithm that measures archival quality, one of the four ranking features.","marker":"[10]"},{"why":"Establishes the N-gram URL classification baseline and comparison point for the F1 results.","marker":"[25]"},{"why":"Defines Memento-Datetime, the temporal distance measure used in the ranking score.","marker":"[29]"},{"why":"Supplies the deep hierarchical classification method with category search, tree construction, and ancestor-assistance pruning that Step 2 adapts.","marker":"[32]"}],"fun_headline_variants":["Lost page? Its URL can reveal archived stand-ins","URI alone finds archive alternatives for missing pages","F1 0.59: URI text alone can guide archive recommendations","Missing page? Its URI picks archived twins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two pages in the same DMOZ category are what a user would accept as a substitute for the missing page; the paper does not test this with users, so if category co-membership does not track user relevance, the recommendations fail.","fun_headline_variants_meta":{"raw":{"variants":["Lost page? Its URL can reveal archived stand-ins","URI alone finds archive alternatives for missing pages","F1 0.59: URI text alone can guide archive recommendations","Missing page? Its URI picks archived twins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001171,"raw_usage":{"total_tokens":4927,"prompt_tokens":1116,"completion_tokens":3811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":3748}},"tokens_in":732,"tokens_out":3811,"duration_ms":28941,"temperature":1.0,"reasoning_tokens":3748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:32:53.364956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study in which users who request unarchived URIs rate the top-ranked same-DMOZ-category recommendations against shuffled archived pages; if the system's recommendations are not rated significantly more relevant than the shuffled baseline, the central claim is unsupported. A simpler computational check would compare recommended candidates' page content similarity to the lost page's content, where obtainable, and show whether same-category URIs really do have related content.","supporting_citations":[{"cited_title":"Ainsworth, Ahmed Alsum, Hany M","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that many pages are unarchived and that DMOZ-listed pages are likely archived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MemGator is the tool used to filter candidate URIs by whether a memento exists in the archive."},{"cited_title":"Weigle, and Michael L","cited_arxiv_id":null,"evidence_quote":"Provides the Wayback Machine access-log sample used to characterize real requests and evaluate URI properties."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the three tokenization methods (tokens, all-grams from tokens, all-grams from URI) adopted for URI classification."},{"cited_title":"Brunelle, Mat Kelly, Hany SalahEldeen, Michele C","cited_arxiv_id":null,"evidence_quote":"Provides the Memento-Damage algorithm that measures archival quality, one of the four ranking features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the N-gram URL classification baseline and comparison point for the F1 results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the deep hierarchical classification method with category search, tree construction, and ancestor-assistance pruning that Step 2 adapts."}],"review_version":1}