Under controlled tests, several major AI assistants (Gemini, Grok, DeepSeek, Qwen) accessed robots.txt-disallowed web pages without ever requesting the robots.txt file, while Claude and Mistral generally complied.
Carl , title =
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CY 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
Under controlled tests, several major AI assistants (Gemini, Grok, DeepSeek, Qwen) accessed robots.txt-disallowed web pages without ever requesting the robots.txt file, while Claude and Mistral generally complied.