Pith. sign in

REVIEW 2 major objections 6 minor 52 references

Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

T0 review · 2 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that 60.0% of reputable news sites disallow at least one AI crawler in robots.txt, versus 9.1% of misinformation sites, a gap that has widened from 23% since September 2023 and may be skewing the content available to A

desk verdict Solid descriptive measurement of a real and growing asymmetry, but the headline 'misinformation is more open' is one plausible interpretation among several; worth refereeing with a request for popularity-controlled analyses and released artifacts. read the letter →

arxiv 2510.10315 v3 pith:CLU7EPMC submitted 2025-10-11 cs.CY

Nicolas Steinacker-Olsztyn , Devashish Gosain , Ha Dao This is my paper · ORCID
classification cs.CY
keywords robots.txtAIcrawlersmisinformationLLMtrainingdatawebcontentaccesscontrolgatekeepinglongitudinalanalysiscredibilityclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the robots.txt files that govern AI web crawlers treat reputable news and misinformation sites differently. It finds a stark asymmetry: as of May 2025, 60.0% of reputable news sites disallow at least one AI agent, versus 9.1% of misinformation sites, and the gap has grown from 23% to nearly 60% since September 2023. Reputable sites block an average of 15.5 AI user agents; misinformation sites block fewer than one. The authors also measure active blocking via HTTP User-agent headers, finding both types use it, but reputable sites' behavior matches their declared robots.txt rules far more closely. If this pattern holds, the content most available to AI training crawlers is increasingly low-credibility.

What carries the argument

The central mechanism is the Robots Exclusion Protocol (robots.txt): a voluntary file at a domain root listing User-agent directives and Disallow rules. The paper operationalizes 'AI gatekeeping' as the presence of a DisallowAll rule for any of a curated list of 63 AI user agents (such as GPTBot, CCBot, and Google-Extended), and complements this with active-blocking measurements that vary only the HTTP User-Agent header to see if the server returns non-200 statuses. The longitudinal component uses six archived snapshots at four-month intervals to trace changes in these directives over time.

What would settle it

Take a matched sample of reputable and misinformation sites with similar popularity rankings, content platforms, and languages; if, after adjustment, the difference in DisallowAll-for-AI rates between the two groups is no longer statistically significant, the claim that misinformation per se drives openness is falsified. Alternatively, comparing the same sites' robots.txt files over the claimed timeline and finding that the reputable sites' disallow rates did not rise from roughly 23% in September 2023 to roughly 60% by May 2025 would falsify the longitudinal trend.

Watch

Extended reading notes

Core claim

The paper's central discovery is that content-access signaling on the web is credibility-stratified: sites rated reputable by the credibility classification the authors adopt are far more likely to declare blanket bans on AI crawlers in their robots.txt files, and to list far more AI agents, than sites rated as misinformation. The largest specific number: 60.0% vs 9.1% for DisallowAll of at least one AI agent; average blocked agents 15.5 vs 0.77. This gap is not explained by popularity alone, and it has widened sharply over the observation period (September 2023 to May 2025). The authors further show that active blocking—refusing HTTP requests when the User-Agent header names an AI crawler—i

Load-bearing premise

The entire reputable-versus-misinformation comparison rests on the credibility ratings the authors adopt; if those labels are noisy or track political alignment, technical sophistication, or language rather than factual reliability, the robots.txt gap may be about those correlated attributes instead of misinformation itself.

Editorial extensions

If this is right

  • LLM training corpora assembled from publicly accessible web content will contain a disproportionately large share of misinformation-site content, because those sites rarely forbid AI crawlers.
  • A self-reinforcing loop can emerge: permissive misinformation sites are scraped repeatedly, and their content reappears in AI-generated text, amplifying its reach.
  • Reputable news sites' growing blocklists (median over 25 agents by early 2025) mean the common pool of freely crawlable text is increasingly low-credibility or non-news material.
  • Active blocking is an imperfect substitute for robots.txt because it is all-or-nothing per agent and can interfere with legitimate indexing; the divergence between declared and actual blocking on misinformation sites suggests a patchwork rather than coherent policy.
  • The widening gap indicates that awareness of AI crawling is itself a differentiator: misinformation operators appear largely unresponsive to AI crawler announcements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled comparison matching reputable and misinformation sites on popularity rank, content management system, and language would likely attenuate the 60%-vs-9% gap, because the paper's own popularity check is an unadjusted cumulative distribution; the misinformation effect may partly be an under-resourced-site effect.
  • The credibility labels may be confounded with political orientation and English-language coverage; the robots.txt differences could track editorial professionalism or legal exposure rather than factual reliability per se.
  • A testable extension would measure whether per-site inclusion rates in public web crawl corpora correlate with robots.txt AI-blocking, directly linking declared gatekeeping to actual training-data exposure.
  • Because the paper measures declarations and one pair of active-blocking agents, not the behavior of the full crawler ecosystem, the true training-data skew could be smaller (if crawlers ignore robots.txt) or larger (if crawlers over-comply) than the declared asymmetry suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper compares robots.txt files of 3,369 reputable news websites and 710 misinformation websites, as classified by Media Bias/Fact Check, to measure how often each group restricts AI crawlers. It reports that 60.0% of reputable sites with a robots.txt disallow at least one AI agent, versus 9.1% of misinformation sites; the average number of disallowed agents is 15.5 vs 0.77. It also measures active blocking via HTTP requests with AI User-Agent strings, finding both groups engage in it, and uses Internet Archive snapshots to show the gap grew from 23% to ~60% between September 2023 and May 2025. The authors conclude that the accessible web is increasingly skewed toward lower-credibility content.

Significance. This is the first study, to my knowledge, that compares robots.txt AI-exclusion practices across credibility categories. The dataset is large, the crawling is multi-vantage, the AI agent list is carefully curated, and the longitudinal analysis using six IA snapshots is a valuable contribution. The paper is transparent about its methodology and limitations. If the descriptive gap is robust, it raises important and timely questions about AI training data composition and the incentives of low-credibility sites to remain permissive. The paper does not overclaim causality regarding training data, explicitly stating that it does not have visibility into proprietary datasets. Its main weakness is the limited control for popularity and other correlated site characteristics when interpreting the gap as a property of 'misinformation'.

major comments (2)
  1. [§4.1, Figure 3] The claim that 'popularity alone does not determine the robots.txt practices' is not supported by the evidence presented. The ECDF in Fig. 3 plots the Tranco rank distribution only among sites that already have a DisallowAll rule; it does not estimate the probability of having such a rule conditional on rank. Because the reputable set has vastly higher robots.txt adoption (96.4% vs 73.8%) and is skewed to higher popularity (Fig. 8), the raw gap in Table 2 could be driven entirely by differences in site scale or management capacity. A stratified comparison within Tranco bands or a logistic regression with rank as a covariate is needed to support the interpretation that misinformation status, rather than correlated attributes, explains the gap. As written, the paper's own limitation in §5.2 is more accurate than the §4.1 claim.
  2. [§4.3] The definition of active blocking is underspecified. The text states that the crawler recorded 'the returned status code and response size in bytes' and filters out non-200 responses, but it never states the exact decision rule for classifying a site as actively blocking. Was a site classified as blocking if a 200 response in the control crawl became a 4xx/5xx in the AI-agent crawl? If response-size shrinkage alone was used, what threshold? Without this criterion, the headline figures (e.g., 16.9% of reputable sites blocking both agents) cannot be reproduced or interpreted. The use of only two AI UAs and a single vantage point is acknowledged, but the binary classification rule is essential and should be stated explicitly.
minor comments (6)
  1. [Throughout] 'DisallowAll' is not a standard REP term; define it (e.g., 'disallow:/') or use 'fully disallowed'.
  2. [Figure 2] The two numbers '70,410 robots.txt Retrieved' and '3,672 robots.txt Retrieved' are inconsistent with Table 2 (3,179 + 493 = 3,672). If 70,410 is the total across all vantage points/snapshots, clarify.
  3. [§3.2] Typo: 'we conduct alongi-tudinal analysis' should be 'conduct a longitudinal analysis'.
  4. [§4.3] The control crawl excludes 18 sites that 'returned 200 responses but were no longer operational'; how was non-operational status determined? Unclear.
  5. [§5.2] The limitations section does not mention the active-blocking methodological choices (only two AI UAs, single vantage point, single request per UA); consider adding.
  6. [§3.2/§4.4] For the longitudinal analysis, the paper selects the most recent IA snapshot before each date but does not report per-snapshot coverage or whether sites with missing snapshots differ systematically. A brief coverage table would strengthen confidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported quantities are direct measurements of crawled robots.txt files, with categories taken from an external credibility classification.

full rationale

The paper's central claims are measured, not derived from fitted parameters or self-referential definitions. Section 3.1 states that the reputable/misinformation split 'relies on MBFC to construct two domain sets,' an external human classification of factual reporting; it is not defined in terms of robots.txt behavior. Section 3.2 describes direct retrieval: 'we issued an HTTP request to the standard root location of the robots.txt file.' The headline result, '60.0% of reputable sites include a DisallowAll directive for at least one AI agent, compared to just 9.1% among misinformation sites' (Table 2), is an aggregate of parsed DisallowAll rules from those retrieved files. The AI-agent list is compiled from Dark Visitors, Cloudflare, Liu et al., and ai.robots.txt (§3.3), none of which encode the paper's outcome. The longitudinal analysis uses Internet Archive snapshots (§3.2, §4.4), again direct observation. No parameter is fitted to one subset and then 'predicted' on a closely related quantity, and no self-citation is load-bearing: Liu et al. [31] and MBFC [35] are external sources, and the paper does not invoke any theorem or result by its own authors. The manuscript's own limitation statements (§5.2: labels 'may carry inherent subjectivity,' 'we do not verify crawler behavior,' 'we cannot establish a direct or causal relationship') and the popularity discussion (§4.1, Figure 3) identify a potential confound and interpretive caution, but they do not show that any result is equivalent by construction to its inputs. The derivation chain is self-contained as a measurement study.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central quantities — percentages of sites disallowing AI agents — are measured directly, with no fitted parameters. However, the comparison's validity depends on external labels (MBFC), a curated agent list, interpretation of HTTP responses as blocking, and the representativeness of Internet Archive snapshots.

assumptions (5)
  • domain assumption MBFC's human classifications correctly separate misinformation websites from reputable news websites.
    The entire grouping (§3.1) uses MBFC labels; if these are inaccurate or biased, the central comparison is misattributed.
  • domain assumption The 63 AI user agents compiled from external catalogs are a representative proxy for AI crawler traffic.
    The analysis counts only these agents (§3.3); the results would change if major AI crawlers are missing or if non-AI agents are included.
  • domain assumption Robots.txt directives are a meaningful signal of intended access, and AI crawlers generally respect them.
    The paper relies on this to interpret declared rules as shaping training data; it cites prior compliance evidence [31] but does not verify crawler behavior itself.
  • domain assumption A non-200 HTTP response to a request carrying an AI user-agent string, after control crawls, constitutes active blocking.
    Used in §4.3 to measure active blocking; transient failures and bot-detection heuristics could produce false positives, though control crawls mitigate some.
  • domain assumption Internet Archive snapshots accurately represent each site's robots.txt over time and have comparable coverage across groups.
    The longitudinal trend (§4.4) depends on snapshot availability and timing; coverage differences between reputable and misinformation sites could bias the trend.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web." pith.science (2026). https://pith.science/paper/CLU7EPMC

@misc{pith2026251010315,
  author       = {Pith},
  title        = {Pith review of: Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLU7EPMC}},
  note         = {Machine review of arXiv:2510.10315}
}
read the original abstract

Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark contrast: 60.0% of reputable sites disallow at least one AI crawler, compared to just 9.1% of misinformation sites in their robots.txt files. Reputable sites forbid an average of 15.5 AI user agents, while misinformation sites prohibit fewer than one. We then measure active blocking behavior, where websites refuse to return content when HTTP requests include AI crawler user agents, and reveal that both categories of websites utilize it. Notably, the behavior of reputable news websites in this regard aligns more closely with their declared robots.txt directive than that of misinformation websites. Finally, our longitudinal analysis reveals that this gap has widened over time, with AI-blocking by reputable sites rising from 23% in September 2023 to nearly 60% by May 2025. Our findings highlight a growing asymmetry in content accessibility that may shape the training data available to LLMs, raising essential questions for web transparency, data ethics, and the future of AI training practices.

Figures

Figures reproduced from arXiv: 2510.10315 by the authors.

Figure 1
Figure 1. Example robots.txt file. the protocol’s limitations for expressing data preferences in mod￾ern AI contexts. Liu et al. [31] found that while adoption of newer signals like NoAI remains low, major AI crawlers generally respect robots.txt. They also studied network level crawling blocks pro￾vided by Cloudflare, and the prevalance of active blocking based off of the declared User-agent header. Dinzinger et al. [13] tra… view at source ↗
Figure 2
Figure 2. Methodology overview. any URL redirections observed during the retrieval process. In addi￾tion, to improve measurement robustness and account for potential geo-blocking or region-specific bot responses, we deployed AWS cloud instances across seven geographic regions: one vantage point in Germany, one in Sweden, one in the United States, one in Brazil, and one each in Africa, Asia, and Australia. Each site was crawle… view at source ↗
Figure 3
Figure 3. Tranco popularity for websites that have a [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Number of AI agents disallowed per website in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Percentage of sites that DisallowAll at least one AI agent in their robots.txt over time. 09-2023 01-2024 05-2024 09-2024 01-2025 05-2025 0 10 20 30 40 50 60 Number of AI Agents Reputable News Websites Misinformation Websites [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Distribution of rank of news websites based on [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 5 linked inside Pith

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally On Your Phone.arXiv preprint arXiv:2404.14219(2024)

    arXiv 2024

  2. [2]

    Marco T Bastos and Dan Mercea. 2019. The Brexit Botnet and User-generated Hyperpartisan News.Social Science Computer Review37, 1 (2019), 38–54

    2019

  3. [3]

    Bogle, Ariel. 2023. New York Times, CNN and Australia’s ABC Block OpenAI’s GPTBot Web Crawler From Accessing Content. https://www.theguardian.com/technology/2023/aug/25/new-york-times- cnn-and-abc-block-openais-gptbot-web-crawler-from-scraping-content Accessed: 2025-06-03

    2023

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models Are Few-shot Learners.Advances in Neural Information Processing Systems33 (2020), 1877–1901

    2020

  5. [5]

    Chien-yi Chang and Xin He. 2025. The Liabilities of robots.txt. arXiv:2503.06035 [cs.CY] https://arxiv.org/abs/2503.06035

    arXiv 2025

  6. [6]

    Zhouhan Chen and Juliana Freire. 2020. Proactive Discovery of Fake News Domains From Real-time Social Media Feeds. InCompanion Proceedings of the Web Conference 2020. 584–592

    2020

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2023. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research24, 240 (2023), 1–113

    2023

  8. [8]

    Rajdipa Chowdhury, Sriram Srinivasan, and Lise Getoor. 2020. Joint Estimation of User and Publisher Credibility for Fake News Detection. InProceedings of the CIKM. 1993–1996

    2020

Show all 52 references
  1. [9]

    Cloudflare. 2025. Verified Bots. https://radar.cloudflare.com/bots#verified-bots Accessed: 2025-06-03

    2025

  2. [10]

    Common Crawl. 2025. CCBot. https://commoncrawl.org/ccbot Accessed: 2025- 06-03

    2025

  3. [11]

    Dark Visitors. 2025. Agents. https://darkvisitors.com/agents Accessed: 2025-06- 03

    2025

  4. [12]

    Dark Visitors. 2025. ai.robots.txt. https://github.com/ai-robots-txt/ai.robots.txt Accessed: 2025-06-03

    2025

  5. [13]

    Michael Dinzinger and Michael Granitzer. 2024. A Longitudinal Study of Content Control Mechanisms. InCompanion Proceedings of the ACM Web Conference 2024. ACM, 1382–1387

    2024

  6. [14]

    Directorate-General for Communication. 2024. AI Act Enters Into Force. https:// commission.europa.eu/news/ai-act-enters-force-2024-08-01_en Accessed: 2025- 06-04

    2024

  7. [15]

    European Parliament, Committee on Legal Affairs. 2025. Study: Gen- erative AI from a Copyright Perspective. https://www.europarl. europa.eu/meetdocs/2024_2029/plmrep/COMMITTEES/JURI/DV/2025/05- 12/2025.05.12_item6_Study_GenAIfromacopyrightperspective_EN.pdf Accessed: 2025-05-30

    2025

  8. [16]

    Bilal Ghanem, Simone Paolo Ponzetto, Paolo Rosso, and Francisco Rangel. 2021. FakeFlow: Fake News Detection by Modeling the Flow of Affective Information. InProceedings of the EACL. 679–689

    2021

  9. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models.arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  10. [18]

    Matthew Gray. 1995. Measuring the Growth of the Web. https://www.mit.edu/ people/mkgray/growth/ Accessed: 2025-05-30

    1995

  11. [19]

    Maurício Gruppi, Benjamin D Horne, and Sibel Adali. 2021. NELA-GT-2020: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles.CoRR(2021)

    2021

  12. [20]

    Internet Archive. 1996. About the Internet Archive. https://archive.org/about/ Accessed: 2021-04-15

    1996

  13. [21]

    Irwin, Kate. 2024. New York Times Sends Perplexity Cease-and-Desist Over AI Scraping. https://www.wired.com/story/aws-perplexity-bot-scraping- investigation/ Accessed: 2025-06-04

    2024

  14. [22]

    Jaime Jiménez and Jari Arkko. 2024. AI, robots.txt. (2024). https://www.ietf.org/ slides/slides-aicontrolws-ai-robotstxt-00.pdf Accessed: 2025-06-03

    2024

  15. [23]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361(2020)

    2020 arXiv

  16. [24]

    Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, and Emily Wenger. 2025. Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-scale Empirical Study.arXiv preprint arXiv:2505.21733(2025)

    2025

  17. [25]

    Santanu Kolay, Paolo D’Alberto, Ali Dasdan, and Arnab Bhattacharjee. 2008. A larger scale study of robots.txt. InProceedings of the WWW. 1171–1172

    2008

  18. [26]

    Martijn Koster. 1994. Important: Spiders, Robots and Web Wanderers. http: //1997.webhistory.org/www.lists/www-talk.1994q1/0717.html Accessed: 2025- 05-31

    1994

  19. [27]

    Martijn Koster, Gary Illyes, Eric Kuan, and Remy Burger. 2022. RFC 9309: Robots Exclusion Protocol. https://www.rfc-editor.org/rfc/rfc9309.html Accessed: 2025- 05-31

    2022

  20. [28]

    David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Penny- cook, David Rothschild, et al. 2018. The Science of Fake News.Science(2018), 1094–1096

    2018

  21. [29]

    Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Ko- rczyński, and Wouter Joosen. 2019. Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation. https://tranco-list.eu/. Accessed: 2025-05-13

    2019

  22. [30]

    Christopher Lehane. 2025. [OpenAI Response] OSTP/NSF RFI: Notice Request for Information on the Development of an Artificial Intelligence (AI) Action Plan. https://cdn.openai.com/global-affairs/ostprfi/ec680b75-d539-4653-b297- 8bcf6e5f7686/openai-response-ostpnsf-rfi-notice-re...

    2025

  23. [31]

    Enze Liu, Elisa Luo, Shawn Shan, Geoffrey Voelker, Ben Zhao, and Stefan Savage

  24. [32]

    Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. 2024. Consent in Crisis: The Rapid Decline of the AI Data Commons. Advances in Neural Information Processing Systems3...

    2024

  25. [33]

    2025.NewsGuard Monthly AI Misinforma- tion Monitor of Leading AI Chatbots

    Maitland, Eva and Sadeghi, McKenzie. 2025.NewsGuard Monthly AI Misinforma- tion Monitor of Leading AI Chatbots. Technical Report. NewsGuard

    2025

  26. [34]

    Media Bias/Fact Check. 2015. Methodology. https://mediabiasfactcheck.com/ methodology/ Accessed: 2025-05-30

    2015

  27. [35]

    Media Bias/Fact Check. 2015. Search and Learn the Bias of News Media. https: //mediabiasfactcheck.com/ Accessed: 2025-05-30

    2015

  28. [36]

    Mehrotra, Dhruv and Couts, Andrew. 2024. Amazon Is Investigating Perplexity Over Claims of Scraping Abuse. https://uk.pcmag.com/ai/154890/new-york- times-sends-perplexity-cease-and-desist-over-ai-scraping Accessed: 2025-06- 03

    2024

  29. [37]

    OpenAI. 2025. Overview of OpenAI Crawlers. https://platform.openai.com/ docs/gptbot Accessed: 2025-06-03

    2025

  30. [38]

    Organisation for Economic Co-operation and Development. 2025. Intel- lectual Property Issues in Artificial Intelligence Trained on Scraped Data. https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/02/ intellectual-property-issues-in-artificial-intelligence-train...

    2025

  31. [39]

    Markatos, and Nicolas Kourtellis

    Emmanouil Papadogiannakis, Panagiotis Papadopoulos, Evangelos P. Markatos, and Nicolas Kourtellis. 2023. Who Funds Misinformation? A Systematic Analysis of the Ad-related Profit Routines of Fake News Sites. InProceedings of the WWW. 2765–2776

    2023

  32. [40]

    Kien Pham, Aécio Santos, and Juliana Freire. 2016. Understanding Website Behavior Based on User Agent. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval(Pisa, Italy)(SIGIR ’16). Association for Computing Machinery...

    2016

  33. [41]

    Jef John Roberts. 2017. Hoax Over ‘Dead’ Ethereum Founder Spurs $4 Billion Wipe Out. https://fortune.com/2017/06/26/vitalik-death/ Accessed: 2025-05-30

    2017

  34. [42]

    Romain, Danielle. 2023. An Update on Web Publisher Controls. https://blog. google/technology/ai/an-update-on-web-publisher-controls/ Accessed: 2025-06- 04

    2023

  35. [43]

    Romain, Danielle. 2025. Does Anthropic Crawl Data From the Web, and How Can Site Owners Block the Crawler? https://support.anthropic.com/en/ articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site- owners-block-the-crawler Accessed: 2025-06-04

    2025

  36. [44]

    Sadeghi, McKenzie and Dimitriadis, Dimitris, and Arvanitis, Lorenzo and Padovese, Virginia and Pozzi, Giulia, and Badilini, Sara and Vercellone, Chiara, and Wang, Macrina and Huet, Natalie, and Fishman, Zack and Pfaller, Leonie and Adams, Natalie and Wollen, Miranda. 2025. Tra...

    2025

  37. [45]

    Schaul, Kevin and Chen, Szu Yu and Tiku, Nitasha. 2023. Inside the Se- cret List of Websites That Make AI Like ChatGPT Sound Smart. https:// www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/ Ac- cessed: 2025-06-03

    2023

  38. [46]

    Yang Sun, Ziming Zhuang, Isaac G Councill, and C Lee Giles. 2007. Determining Bias to Search Engines From robots.txt. InProceedings of the WI. IEEE, 149–155

    2007

  39. [47]

    Yang Sun, Ziming Zhuang, and C Lee Giles. 2007. A Large-scale Study of robots.txt. InProceedings of the WWW. 1123–1124

    2007

  40. [48]

    Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. 2024. Jamba- 1.5: Hybrid Transformer-mamba Models at Scale.arXiv preprint arXiv:2408.12570 (2024)

    2024 arXiv

  41. [49]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and Efficient Foundation Language Models.arXiv preprint arXiv:2302.13971(2023)

    2023 arXiv

  42. [50]

    World Health Organization. 2021. Fighting Misinformation in the Time of COVID- 19, One Click at a Time. https://www.who.int/news-room/feature-stories/detail/ fighting-misinformation-in-the-time-of-covid-19-one-click-at-a-time Accessed: 2025-05-30

    2021

  43. [51]

    Xinyi Zhou, Apurva Mulay, Emilio Ferrara, and Reza Zafarani. 2020. Recovery: A Multimodal Repository for Covid-19 News Credibility Research. InProceedings of the CIKM. 3205–3212. A Ranking of reputable news and misinformation websites Figure 8 shows the distribution of website...

    2020

  44. [2025]

    InProceedings of the IMC

    Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. InProceedings of the IMC

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.