{"id":"f1b42bee-e144-404b-9ad6-6cb0e1ab4c2d","arxiv_id":"2505.24091","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors extend the EDGI 2016-2020 collection back to 2008 using many archive sources, producing 1,220 triplets and reporting that 87% of pages with Trump-era term deletions had those terms added under Obama.","lead":"This paper builds a 2008, 2016, and 2020 archive snapshot dataset of 1,220 US federal environmental webpages by stitching together captures from many archives. It reports that 87% of pages with terms deleted under Trump had those terms added under Obama, and it offers a methodology for extending web archive collections backwards in time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 87% Obama-addition claim is not reproducible: Section 4.2's own category counts sum to 702, not 740, and the method for classifying 'added during Obama' is never specified, so the headline number may be an artifact of the classification.","rationale":"The central claim is the 87% figure, which is a statement about the dataset itself. Before worrying about whether the dataset generalizes, the number must be internally consistent and reproducible from a defined measurement. Section 4.2 fails on both counts: the category counts do not sum to the stated denominator, and the procedure for classifying when a deleted term was added is never specified. The temporal attribution is particularly fragile because a single 2008 snapshot cannot distinguish 'absent in January 2008' from 'not yet added'; a term added in mid-2008 under Bush would be labeled Obama. The proposed test directly checks both the arithmetic and the attribution bias. The reader's sampling concern is real but secondary: if the 87% is not reproducible even within the constructed dataset, representativeness does not yet matter. Hence the verdict should remain REJECT.","tokens_in":15947,"tokens_out":6645,"duration_ms":66777,"concrete_test":"Ask the authors to release the per-page classification table for the 1,220 triplets, with the 2016 and 2020 term presences and the 2008 snapshots used. Then (a) recompute the three category counts and reconcile them with the stated 740; and (b) for a random sample of pages classified as 'Obama-only additions,' check whether the deleted term appears in any other 2008 capture from End of Term or Common Crawl (which occurred later in 2008). If such captures exist for more than a small fraction, the 'Obama addition' attribution is systematically contaminated by late-Bush additions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 reports that 740 of 1,220 pages had terms deleted during the Trump administration, then breaks those pages down as 373 with deleted terms added in both the Obama and Bush administrations, 274 with deleted terms only from Obama, and 55 only from Bush. These three categories total 702 pages, leaving 38 pages of the stated 740 unexplained. The abstract's 87% (647/740) cannot be recovered from the reported counts: if the stated denominator is correct, 38 pages are missing; if the three categories are exhaustive, the denominator should be 702 and the percentage would be 92%. The manuscript never describes how 'added during the Obama administration' was determined: no term-extraction algorithm, no treatment of captures with partial or mis-rendered content, and no handling of the fact that the 2008 memento is a single snapshot. A term absent from a January 2008 capture but present in 2016 would be labeled Obama-added even if it was placed on the page later in 2008, during the Bush administration. Because the 87% figure is the central empirical claim, this missing description and unresolved count discrepancy are load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodology for temporally extending existing web archive collections by aggregating captures from multiple sources, and it applies this methodology to extend the EDGI collection of US federal environmental webpages back to 2008. The authors construct a dataset of 1,220 archived triplets (2008, 2016, 2020) across 30 agencies and use it to ask whether terms deleted from these pages during the Trump administration had been added during the Obama administration. They report that 81% of the pages changed between 2008 and 2020 and that 87% of pages with terms deleted under Trump had at least one deleted term added during the Obama administration. The paper also analyzes the provenance of the 2008 mementos, identifies change trends by agency, and examines whether deleted terms were restored by 2024.","tokens_in":16207,"tokens_out":11138,"duration_ms":103155,"significance":"If the empirical result were reliable, the paper would be a useful contribution to web-archiving methodology and to the political-science question of whether Trump-era deletions represented a rollback of Obama-era content. The dataset-assembly work is substantial: the authors document the need to aggregate captures from 16 organizations, quantify the limited contribution of the End of Term archive, and demonstrate that a single crawl source is insufficient for this period. The provenance analysis is a strength, as is the honest reporting of the manual augmentation steps. However, the headline longitudinal claim is currently not reproducible: the reported category counts do not sum to the stated denominator, the method for classifying when a term was added is not specified, and the temporal boundaries used do not align with the actual presidential administrations. These issues are load-bearing because the 87% figure is the central empirical claim in the abstract.","major_comments":[{"comment":"The three reported categories for the 740 pages with terms deleted during the Trump administration—373 with deleted terms added in both the Obama and Bush administrations, 274 with deleted terms only from Obama, and 55 with deleted terms only from Bush—sum to 702, not 740. The abstract's 87% figure is 647/740, but the reported counts yield 647/702 = 92.2%. The 38-page discrepancy is never explained. If the three categories are intended to be exhaustive, the denominator should be 702; if they are not exhaustive, the text must state how the remaining 38 pages were classified and why they are excluded. Without this, the paper's central empirical claim is not reproducible.","section":"Section 4.2, Figure 11"},{"comment":"The manuscript never specifies how a deleted term was determined to have been \"added during the Obama administration.\" It cites prior work [9] for change-text search, but it does not describe the term-extraction algorithm, the criteria for \"fully deleted,\" the handling of partially rendered or mis-captured mementos, or the exact capture dates used in the comparison. Because the dataset contains only a single 2008 memento per page, a term absent from that memento but present in a later 2008 memento would be classified as \"Obama-added\" even if it was introduced during the Bush administration. The stated classification window—\"between July 2008 and June 2016\"—also does not match the Obama administration, which began in January 2009. These omissions make the 87% headline result unverifiable.","section":"Section 4.2"},{"comment":"The analysis treats terms deleted \"between July 2016 and July 2020\" as deletions during the Trump administration, but the Trump administration did not begin until January 2017. Changes observed between July 2016 and January 2017 occurred while Obama was still president. The paper does not report the actual capture dates of the 2016 mementos, so the reader cannot determine how much of the observed deletion predates the Trump administration. This conflation directly affects the interpretation of the 87% result as evidence about Trump-era deletions.","section":"Section 4.2"},{"comment":"The 87% result is computed on a non-random, quota-based convenience sample. Section 3 describes a target of 15 high and 15 deep links per agency, stopping criteria once quotas are met, and iterative manual augmentation, and the final dataset is heavily skewed toward domains with many archived captures (e.g., noaa.gov, ferc.gov, osha.gov). Because the percentages in Section 4.2 are unweighted, they describe this curated sample rather than the population of US federal environmental webpages. If the authors intend the result to answer the general research question about the Trump administration, the paper needs a representativeness analysis or a clearly hedged framing. As written, the conclusion in Section 5 (\"show that the Republican president Trump deleted terms mostly added during the previous Democrat President Obama's administration\") overstates what the dataset can support.","section":"Section 3, Section 5"}],"minor_comments":[{"comment":"The first sentence of Section 4 says the final dataset contains \"1,200\" triplets, while the rest of the paper (including the abstract, Figure 11, and Section 4.2) says \"1,220.\" This should be harmonized.","section":"Section 4"},{"comment":"The provenance counts in Table 5 sum to 1,211, not 1,220. In addition, several agency rows in Table 5 do not match the corresponding totals in Table 4 (for example, justice.gov shows 19 mementos in Table 5 but 24 triplets in Table 4, and nasa.gov shows 32 vs. 30). These discrepancies need to be resolved or explicitly explained.","section":"Section 4.1.1, Table 5"},{"comment":"The sentence \"Web crawlers like Heritrix CITE are available for crawling the live web\" contains a dangling \"CITE\" that appears to be a placeholder for a citation.","section":"Section 3.3"},{"comment":"The sentence \"The web interface gives the earliest and most recent archival dates of each page, , as shown in Figure 4\" contains a doubled comma that should be removed.","section":"Section 3.6"},{"comment":"The word \"surpising\" should be \"surprising.\"","section":"Section 4.2.1"},{"comment":"The text says that one-third of the 33 exemplars persisted until October 2024, which implies 11 pages, but the subsequent breakdown (6 pages with restored terms and 4 pages with no restored terms) sums to 10. This should be clarified.","section":"Section 4.2.3"}],"recommendation":"reject","confidential_remarks":"The paper has a useful methodological contribution and a potentially valuable dataset, but the central longitudinal analysis is not reproducible and the temporal classification is not valid as presented. The count discrepancy in Section 4.2 and the missing method for classifying when terms were added are load-bearing errors. A revised paper that focuses on the dataset construction and provenance methodology, or that re-does the change analysis with properly aligned dates and a fully specified classification, could be reconsidered in the future."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Paul—\n\nTwo things to know about arXiv:2505.24091. First, the paper's real contribution is the retroactive extension workflow for web archive collections: using MemGator/CDX to mine the 2008 holdings, past-web crawling with a sticky time policy, full-domain CDX queries for small domains, filtering the End of Term data for crawler traps, and manual augmentation. That part is genuinely useful, and the provenance analysis (Alexa, Common Crawl, Archive-It, EoT) is a nice piece of empirical work. Second, the headline empirical claim — 87% of Trump-era deleted terms were Obama-era additions — is not currently supportable, and that is the paper's central answer to its own research question.\n\nThe problem is not the back-of-the-envelope nature of the count; it's that the counts don't add up. Section 4.2 reports 740 pages with Trump-era deletions, then breaks those into 373 (Obama+Bush), 274 (Obama only), and 55 (Bush only). That sums to 702. The missing 38 pages are never explained. If the denominator should be 702, the Obama-share is 92%, not 87%. Either way, the abstract's headline number is unreproducible from the paper itself.\n\nCompounding that, the method for deciding when a term was 'added' is never described. The paper references the authors' earlier change-detection work but does not say how a term absent from a single January 2008 capture but present in 2016 gets assigned to the Obama administration rather than to late-2008 Bush-era edits. That is a load-bearing step for the central claim, and it is invisible. Also, the dataset itself is not released, so there is no way to check the classification.\n\nThe selection-bias concern is real but not fatal: the dataset is a quota-based convenience sample of 1,220 pages that happen to have archived captures in all three years. The paper acknowledges this. That limits the inferential power, but it does not invalidate the methodology.\n\nBottom line: if you read this for the method, you get a solid, citable workflow. If you read it for the 87% answer, you get a number that the paper itself cannot reproduce. The right outcome, in my view, is to send it to referees and require them to fix the arithmetic, describe and validate the term-classification, and release the dataset. It's not a desk reject; it's a major-revision paper.","headline":"The extension methodology is a real contribution, but the headline 87% finding is not reproducible from the paper's own counts or its unspecified term-classification method.","tokens_in":16713,"tokens_out":3326,"would_cite":true,"duration_ms":27297,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most Trump-era website deletions removed Obama-era additions","keywords":["web archives","longitudinal analysis","US federal environmental websites","presidential administrations","term deletions","memento aggregation","past web crawling","End of Term Web Archive"],"falsifier":"Independently draw a random sample of the 15.8 million 2008 End of Term URLs from the 30 agencies, extend each forward to 2016 and 2020 without the paper's manual augmentation, and compute the share of Trump-deleted terms that were Obama-added; if that share falls far below 87 percent, the result is an artifact of the curation process rather than a property of the federal environmental web.","tokens_in":15757,"feed_emoji":"🗄️","tokens_out":3690,"duration_ms":35136,"temperature":0.7,"pith_summary":"The paper tries to establish that an existing web archive collection can be extended backward in time after the fact, and that doing so answers a concrete political question: were website terms deleted by the Trump administration added during the Obama administration? By assembling 1,220 archived page triplets from 2008, 2016, and 2020 across US federal environmental agencies, the authors found that 81 percent of the pages changed over the period and that 87 percent of pages with Trump-era deletions contained terms added under Obama. The method combines existing crawl data, past-web crawling, full-domain archive lookups, and manual augmentation, because no single archive held enough 2008 captures. If the finding holds, the high-profile climate and regulation deletions were largely removals of content introduced by the previous Democratic administration rather than longstanding Republican-era content.","feed_headline":"Most Trump-era website deletions removed Obama-era additions","feed_subtitle":"Extending federal web archives back to 2008 shows 87 percent of Trump-era deleted terms were added under Obama.","key_machinery":"The load-bearing mechanism is a temporal-extension methodology for web archive collections, built around the 2008–2016–2020 archived triplet as the unit of analysis. The method starts with the EDGI 2016–2020 collection, checks each page for 2008 mementos through TimeMap aggregation with MemGator, verifies archived HTTP statuses through CDX API lookups, crawls the past web with a sticky time policy to find pages that existed in 2008 and persisted to 2020, performs full-domain CDX queries for small poorly covered domains, combines the End of Term 2008 crawl, and finishes with iterative manual augmentation using the Wayback Machine's URLs tool and manual link replay. The methodology is what makes the 87 percent finding possible, because no single archive, crawl, or automated brute-force pass supplied enough 2008 captures.","core_discovery":"The central claim is that the Trump administration's deletions from federal environmental websites were predominantly removals of terms first added during the Obama administration, and that this fact becomes visible only when a collection designed for 2016–2020 is temporally extended back to 2008. The authors contribute a dataset of 1,220 archived triplets, each with successful captures in 2008, 2016, and 2020, and report that 990 pages changed between 2008 and 2020 while 740 pages had Trump-era term deletions. Of those 740 pages, 87 percent had at least one deleted term added under Obama, whereas only 55 pages had deleted terms exclusively from the Bush era. They also find that repeated deletion patterns, called change trends, concentrated in agencies such as OSHA, NIH, and NOAA, and that among the 56 tracked terms, regulation-related terms were deleted more often than climate-related terms.","pith_inferences":["Beyond the paper: because the dataset excludes pages fully deleted by 2020 and pages with no 2008 capture, the 87 percent figure may understate the true share of Trump-era deletions that removed Obama-era additions, since the most aggressively changed pages are exactly the ones most likely to be missing from such a sample.","Beyond the paper: two-thirds of the 2008 mementos came from Alexa crawls, so the 2008 snapshot reflects the interests and coverage of one commercial crawler; a different 2008 archive source could yield a different content mix and potentially a different deletion attribution.","Beyond the paper: the same temporal-extension method could be applied to the 2020–2024 transition to test whether the pattern reverses, and the paper's own exemplar analysis already shows some Obama-era terms restored by 2024.","Beyond the paper: the finding that manual augmentation was essential suggests that this method has a human-in-the-loop cost that will scale poorly to much larger collections, so automated approaches alone may not reproduce the result on broader domains."],"forward_implications":["The 1,220-triplet dataset provides a reusable longitudinal collection covering three US presidential administrations, and the method behind it can extend other existing collections backward in time.","The 81 percent change rate between 2008 and 2020 shows that a URL persisting over twelve years does not imply that its content persisted.","The 87 percent figure indicates that Trump-era deletions were largely removals of Obama-era additions, with only a small number of pages showing deletions exclusively from Bush-era content.","Agency-level deletion patterns vary sharply: OSHA, NIH, and NOAA showed many repeated term deletions, while 17 of the 30 agencies showed no change trends at all.","Among the 56 tracked terms, regulation-related deletions outnumbered climate-related deletions, suggesting the administration's rollback was broader than climate content alone."],"supporting_citations":[{"why":"Supplies the original EDGI 2016–2020 collection of 40,000 federal environmental webpages that the paper extends backward to 2008.","marker":"[26]"},{"why":"MemGator is the tool used to aggregate TimeMaps across web archives to find 2008 mementos for the EDGI candidate pages.","marker":"[4]"},{"why":"Provides the past-web crawling approach with a sticky time policy that the paper adapts to crawl the 2008 web for missing pages.","marker":"[20]"},{"why":"Supplies the End of Term Web Archive dataset used to probe for additional longitudinal candidates and to estimate domain-level 2008 holdings.","marker":"[31]"},{"why":"Establishes that archived webpage collections are assembled from many provenance sources, the premise behind the paper's aggregation strategy.","marker":"[5]"},{"why":"Documents the End of Term Web Archive partnership that produced the 2008 crawl contributing 65 of the 1,220 mementos.","marker":"[34]"},{"why":"Provides the redirect categories used in the exemplar case study to classify whether pages that decayed after 2020 moved to new URLs.","marker":"[11]"},{"why":"Prior work by the same authors that collected TimeMaps for 30,000 EDGI pages and informed the candidate extraction process.","marker":"[9]"}],"fun_headline_variants":["87% of Trump-era web deletions were Obama-era additions","Obama terms swept up in 87% of Trump-era website deletions","Archive extension links 87% of Trump deletions to Obama additions","Temporal archive extension shows Trump deletions hit Obama terms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 1,220 pages with successful captures in 2008, 2016, and 2020 fairly represent the federal environmental pages whose content changed across administrations, even though pages deleted by 2020 and pages without 2008 captures are excluded by the way the sample was built.","fun_headline_variants_meta":{"raw":{"variants":["87% of Trump-era web deletions were Obama-era additions","Obama terms swept up in 87% of Trump-era website deletions","Archive extension links 87% of Trump deletions to Obama additions","Temporal archive extension shows Trump deletions hit Obama terms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001122,"raw_usage":{"total_tokens":4697,"prompt_tokens":1006,"completion_tokens":3691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":3633}},"tokens_in":622,"tokens_out":3691,"duration_ms":23659,"temperature":1.0,"reasoning_tokens":3633,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:35:44.985223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently draw a random sample of the 15.8 million 2008 End of Term URLs from the 30 agencies, extend each forward to 2016 and 2020 without the paper's manual augmentation, and compute the share of Trump-deleted terms that were Obama-added; if that share falls far below 87 percent, the result is an artifact of the curation process rather than a property of the federal environmental web.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original EDGI 2016–2020 collection of 40,000 federal environmental webpages that the paper extends backward to 2008."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MemGator is the tool used to aggregate TimeMaps across web archives to find 2008 mementos for the EDGI candidate pages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the past-web crawling approach with a sticky time policy that the paper adapts to crawl the 2008 web for missing pages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that archived webpage collections are assembled from many provenance sources, the premise behind the paper's aggregation strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the End of Term Web Archive partnership that produced the 2008 crawl contributing 65 of the 1,220 mementos."}],"review_version":1}