Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

Devashish Gosain; Ha Dao; Nicolas Steinacker-Olsztyn

Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

T0 review · 2 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper establishes that 60.0% of reputable news sites disallow at least one AI crawler in robots.txt, versus 9.1% of misinformation sites, a gap that has widened from 23% since September 2023 and may be skewing the content available to A

desk verdict Solid descriptive measurement of a real and growing asymmetry, but the headline 'misinformation is more open' is one plausible interpretation among several; worth refereeing with a request for popularity-controlled analyses and released artifacts. read the letter →

arxiv 2510.10315 v4 pith:CLU7EPMC submitted 2025-10-11 cs.CY

Nicolas Steinacker-Olsztyn , Devashish Gosain , Ha Dao This is my paper · ORCID

classification cs.CY

keywords robots.txtAIcrawlersmisinformationLLMtrainingdatawebcontentaccesscontrolgatekeepinglongitudinalanalysiscredibilityclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the robots.txt files that govern AI web crawlers treat reputable news and misinformation sites differently. It finds a stark asymmetry: as of May 2025, 60.0% of reputable news sites disallow at least one AI agent, versus 9.1% of misinformation sites, and the gap has grown from 23% to nearly 60% since September 2023. Reputable sites block an average of 15.5 AI user agents; misinformation sites block fewer than one. The authors also measure active blocking via HTTP User-agent headers, finding both types use it, but reputable sites' behavior matches their declared robots.txt rules far more closely. If this pattern holds, the content most available to AI training crawlers is increasingly low-credibility.

What carries the argument

The central mechanism is the Robots Exclusion Protocol (robots.txt): a voluntary file at a domain root listing User-agent directives and Disallow rules. The paper operationalizes 'AI gatekeeping' as the presence of a DisallowAll rule for any of a curated list of 63 AI user agents (such as GPTBot, CCBot, and Google-Extended), and complements this with active-blocking measurements that vary only the HTTP User-Agent header to see if the server returns non-200 statuses. The longitudinal component uses six archived snapshots at four-month intervals to trace changes in these directives over time.

What would settle it

Take a matched sample of reputable and misinformation sites with similar popularity rankings, content platforms, and languages; if, after adjustment, the difference in DisallowAll-for-AI rates between the two groups is no longer statistically significant, the claim that misinformation per se drives openness is falsified. Alternatively, comparing the same sites' robots.txt files over the claimed timeline and finding that the reputable sites' disallow rates did not rise from roughly 23% in September 2023 to roughly 60% by May 2025 would falsify the longitudinal trend.

Watch

Extended reading notes

Core claim

The paper's central discovery is that content-access signaling on the web is credibility-stratified: sites rated reputable by the credibility classification the authors adopt are far more likely to declare blanket bans on AI crawlers in their robots.txt files, and to list far more AI agents, than sites rated as misinformation. The largest specific number: 60.0% vs 9.1% for DisallowAll of at least one AI agent; average blocked agents 15.5 vs 0.77. This gap is not explained by popularity alone, and it has widened sharply over the observation period (September 2023 to May 2025). The authors further show that active blocking—refusing HTTP requests when the User-Agent header names an AI crawler—i

Load-bearing premise

The entire reputable-versus-misinformation comparison rests on the credibility ratings the authors adopt; if those labels are noisy or track political alignment, technical sophistication, or language rather than factual reliability, the robots.txt gap may be about those correlated attributes instead of misinformation itself.

Editorial extensions

If this is right

LLM training corpora assembled from publicly accessible web content will contain a disproportionately large share of misinformation-site content, because those sites rarely forbid AI crawlers.
A self-reinforcing loop can emerge: permissive misinformation sites are scraped repeatedly, and their content reappears in AI-generated text, amplifying its reach.
Reputable news sites' growing blocklists (median over 25 agents by early 2025) mean the common pool of freely crawlable text is increasingly low-credibility or non-news material.
Active blocking is an imperfect substitute for robots.txt because it is all-or-nothing per agent and can interfere with legitimate indexing; the divergence between declared and actual blocking on misinformation sites suggests a patchwork rather than coherent policy.
The widening gap indicates that awareness of AI crawling is itself a differentiator: misinformation operators appear largely unresponsive to AI crawler announcements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A controlled comparison matching reputable and misinformation sites on popularity rank, content management system, and language would likely attenuate the 60%-vs-9% gap, because the paper's own popularity check is an unadjusted cumulative distribution; the misinformation effect may partly be an under-resourced-site effect.
The credibility labels may be confounded with political orientation and English-language coverage; the robots.txt differences could track editorial professionalism or legal exposure rather than factual reliability per se.
A testable extension would measure whether per-site inclusion rates in public web crawl corpora correlate with robots.txt AI-blocking, directly linking declared gatekeeping to actual training-data exposure.
Because the paper measures declarations and one pair of active-blocking agents, not the behavior of the full crawler ecosystem, the true training-data skew could be smaller (if crawlers ignore robots.txt) or larger (if crawlers over-comply) than the declared asymmetry suggests.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something genuinely new: it segments AI-related robots.txt restrictions by credibility class and tracks the gap over time. Prior work measured the overall rise of AI blocking (Longpre, Dinzinger, Liu); this is the first to compare reputable news sites against MBFC-labeled misinformation sites longitudinally. The core descriptive finding—60.0% of reputable sites disallow at least one AI agent versus 9.1% of misinformation sites, with the gap widening from 23% to ~60% between Sep 2023 and May 2025—is internally consistent and survives the paper's stated method. The active-blocking measurement is a nice addition, and the authors are careful not to overclaim causality to training data. Credit where due: the crawler methodology is reasonable, the longitudinal use of Internet Archive snapshots is appropriate, and the paper is clearly written.

The soft spots are real but not fatal. The big one is the popularity/technical-maturity confound. Reputable and misinformation sites differ enormously in size, resources, and likely webmaster sophistication, and Fig. 3 does not actually control for Tranco rank—it just plots ECDFs of sites that already have a DisallowAll rule. The stress-test note is right that this is the load-bearing alternative explanation. The paper acknowledges in §5.2 that missing directives 'may reflect limited awareness, technical capacity, or willingness,' but it never attempts to separate these. That said, the manual inspection of top-10k misinformation sites (10 of 14 impose no AI restrictions) is a partial and credible response; it's not a full control but it shows the effect is not purely a Top-10k popularity story. The MBFC labels are subjective and English-centric, but the authors acknowledge this too, and prior work commonly uses MBFC. The active-blocking probes use only two UAs from one location, so that section is suggestive rather than definitive. No released data or code, which hampers replication but is not a methodological flaw.

The paper is a solid descriptive contribution, not a paradigm shift. The interpretation as 'misinformation is more open' is plausible and important, but it should be framed as a hypothesis about what drives the gap, not a demonstrated fact. The central descriptive result holds; what needs work is the identification.

Who should read this: anyone working on AI training data sourcing, web measurement, or content moderation. It deserves a serious referee—an editor should send it out, with a request for a rank-matched or regression-controlled analysis and data release. I'd bring it to a reading group and would cite it if I were writing about AI data access asymmetries.

Referee Report

2 major / 6 minor

Summary. The paper compares robots.txt files of 3,369 reputable news websites and 710 misinformation websites, as classified by Media Bias/Fact Check, to measure how often each group restricts AI crawlers. It reports that 60.0% of reputable sites with a robots.txt disallow at least one AI agent, versus 9.1% of misinformation sites; the average number of disallowed agents is 15.5 vs 0.77. It also measures active blocking via HTTP requests with AI User-Agent strings, finding both groups engage in it, and uses Internet Archive snapshots to show the gap grew from 23% to ~60% between September 2023 and May 2025. The authors conclude that the accessible web is increasingly skewed toward lower-credibility content.

Significance. This is the first study, to my knowledge, that compares robots.txt AI-exclusion practices across credibility categories. The dataset is large, the crawling is multi-vantage, the AI agent list is carefully curated, and the longitudinal analysis using six IA snapshots is a valuable contribution. The paper is transparent about its methodology and limitations. If the descriptive gap is robust, it raises important and timely questions about AI training data composition and the incentives of low-credibility sites to remain permissive. The paper does not overclaim causality regarding training data, explicitly stating that it does not have visibility into proprietary datasets. Its main weakness is the limited control for popularity and other correlated site characteristics when interpreting the gap as a property of 'misinformation'.

major comments (2)

[§4.1, Figure 3] The claim that 'popularity alone does not determine the robots.txt practices' is not supported by the evidence presented. The ECDF in Fig. 3 plots the Tranco rank distribution only among sites that already have a DisallowAll rule; it does not estimate the probability of having such a rule conditional on rank. Because the reputable set has vastly higher robots.txt adoption (96.4% vs 73.8%) and is skewed to higher popularity (Fig. 8), the raw gap in Table 2 could be driven entirely by differences in site scale or management capacity. A stratified comparison within Tranco bands or a logistic regression with rank as a covariate is needed to support the interpretation that misinformation status, rather than correlated attributes, explains the gap. As written, the paper's own limitation in §5.2 is more accurate than the §4.1 claim.
[§4.3] The definition of active blocking is underspecified. The text states that the crawler recorded 'the returned status code and response size in bytes' and filters out non-200 responses, but it never states the exact decision rule for classifying a site as actively blocking. Was a site classified as blocking if a 200 response in the control crawl became a 4xx/5xx in the AI-agent crawl? If response-size shrinkage alone was used, what threshold? Without this criterion, the headline figures (e.g., 16.9% of reputable sites blocking both agents) cannot be reproduced or interpreted. The use of only two AI UAs and a single vantage point is acknowledged, but the binary classification rule is essential and should be stated explicitly.

minor comments (6)

[Throughout] 'DisallowAll' is not a standard REP term; define it (e.g., 'disallow:/') or use 'fully disallowed'.
[Figure 2] The two numbers '70,410 robots.txt Retrieved' and '3,672 robots.txt Retrieved' are inconsistent with Table 2 (3,179 + 493 = 3,672). If 70,410 is the total across all vantage points/snapshots, clarify.
[§3.2] Typo: 'we conduct alongi-tudinal analysis' should be 'conduct a longitudinal analysis'.
[§4.3] The control crawl excludes 18 sites that 'returned 200 responses but were no longer operational'; how was non-operational status determined? Unclear.
[§5.2] The limitations section does not mention the active-blocking methodological choices (only two AI UAs, single vantage point, single request per UA); consider adding.
[§3.2/§4.4] For the longitudinal analysis, the paper selects the most recent IA snapshot before each date but does not report per-snapshot coverage or whether sites with missing snapshots differ systematically. A brief coverage table would strengthen confidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported quantities are direct measurements of crawled robots.txt files, with categories taken from an external credibility classification.

full rationale

The paper's central claims are measured, not derived from fitted parameters or self-referential definitions. Section 3.1 states that the reputable/misinformation split 'relies on MBFC to construct two domain sets,' an external human classification of factual reporting; it is not defined in terms of robots.txt behavior. Section 3.2 describes direct retrieval: 'we issued an HTTP request to the standard root location of the robots.txt file.' The headline result, '60.0% of reputable sites include a DisallowAll directive for at least one AI agent, compared to just 9.1% among misinformation sites' (Table 2), is an aggregate of parsed DisallowAll rules from those retrieved files. The AI-agent list is compiled from Dark Visitors, Cloudflare, Liu et al., and ai.robots.txt (§3.3), none of which encode the paper's outcome. The longitudinal analysis uses Internet Archive snapshots (§3.2, §4.4), again direct observation. No parameter is fitted to one subset and then 'predicted' on a closely related quantity, and no self-citation is load-bearing: Liu et al. [31] and MBFC [35] are external sources, and the paper does not invoke any theorem or result by its own authors. The manuscript's own limitation statements (§5.2: labels 'may carry inherent subjectivity,' 'we do not verify crawler behavior,' 'we cannot establish a direct or causal relationship') and the popularity discussion (§4.1, Figure 3) identify a potential confound and interpretive caution, but they do not show that any result is equivalent by construction to its inputs. The derivation chain is self-contained as a measurement study.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central quantities — percentages of sites disallowing AI agents — are measured directly, with no fitted parameters. However, the comparison's validity depends on external labels (MBFC), a curated agent list, interpretation of HTTP responses as blocking, and the representativeness of Internet Archive snapshots.

assumptions (5)

domain assumption MBFC's human classifications correctly separate misinformation websites from reputable news websites.
The entire grouping (§3.1) uses MBFC labels; if these are inaccurate or biased, the central comparison is misattributed.
domain assumption The 63 AI user agents compiled from external catalogs are a representative proxy for AI crawler traffic.
The analysis counts only these agents (§3.3); the results would change if major AI crawlers are missing or if non-AI agents are included.
domain assumption Robots.txt directives are a meaningful signal of intended access, and AI crawlers generally respect them.
The paper relies on this to interpret declared rules as shaping training data; it cites prior compliance evidence [31] but does not verify crawler behavior itself.
domain assumption A non-200 HTTP response to a request carrying an AI user-agent string, after control crawls, constitutes active blocking.
Used in §4.3 to measure active blocking; transient failures and bot-detection heuristics could produce false positives, though control crawls mitigate some.
domain assumption Internet Archive snapshots accurately represent each site's robots.txt over time and have comparable coverage across groups.
The longitudinal trend (§4.4) depends on snapshot availability and timing; coverage differences between reputable and misinformation sites could bias the trend.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web." pith.science (2026). https://pith.science/paper/CLU7EPMC

@misc{pith2026251010315,
  author       = {Pith},
  title        = {Pith review of: Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLU7EPMC}},
  note         = {Machine review of arXiv:2510.10315}
}

read the original abstract

Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark contrast: 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files. Reputable sites forbid an average of 15.5 AI user agents, while misinformation sites prohibit fewer than one. We then measure active blocking behavior, where websites refuse to return content when HTTP requests include AI crawler user agents, and reveal that both categories of websites utilize it. Notably, the behavior of reputable news websites in this regard aligns more closely with their declared robots.txt directive than that of misinformation websites. Finally, our longitudinal analysis reveals that this gap has widened over time, with AI-blocking by reputable sites rising from 23% in September 2023 to nearly 60% by May 2025. Our findings highlight a growing asymmetry in content accessibility, as reflected in robots.txt directives, that may shape the training data available to LLMs, raising essential questions for web transparency, data ethics, and the future of AI training practices.