{"id":"f7b88838-4e9b-4500-8b36-c55e320ec802","arxiv_id":"2607.14447","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Under controlled tests, several major AI assistants (Gemini, Grok, DeepSeek, Qwen) accessed robots.txt-disallowed web pages without ever requesting the robots.txt file, while Claude and Mistral generally complied.","lead":"Researchers tested ten AI assistants to see whether they obey websites' robots.txt rules when fetching pages live, using their own web servers and hidden secret codes. They found that several assistants, including Gemini, Grok, DeepSeek, and Qwen, fetched pages that explicitly said \"do not access\" and never even asked for the robots.txt file.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt's ownership claim may confound non-compliance: assistants can treat 'I manage the server' as authorization to override robots.txt.","rationale":"The central claim is about the reliability of robots.txt for governing AI assistant access. The strongest evidence is that several assistants accessed disallowed pages and returned embedded secret codes. However, the experimental prompt includes a statement that could be interpreted as the user having authority over the server. This is a unique feature of the study; ordinary users asking an assistant to fetch a URL do not normally claim to manage the target server. Therefore the finding may not generalize to typical usage. The paper's Limitations section notes the prompt 'may have increased some assistants' willingness to retrieve the page' but does not run a control condition. This is the single most load-bearing concern because it threatens the validity of the main result for the central claim's scope. Other concerns (n=5, data availability, user-agent attribution) are secondary: user-agent attribution is largely corroborated by correct secret codes in the answers, and the consistent patterns across five trials provide reasonable support. The reader's identified weakest assumption (user-agent identity across phases) is real but less fundamental, since the secret-code answers confirm that the assistant itself accessed and used the page content. The verdict should remain CONDITIONAL: the paper is acceptable if the authors add a neutral-prompt control or temper the claim to 'under an ownership-claim prompt.' I disagree with the reader's choice of the weakest assumption; the prompt confound is more central.","tokens_in":31158,"tokens_out":11181,"duration_ms":116281,"concrete_test":"Re-run the Phase 2 disallowed conditions (Disallow All and assistant-specific disallow) for DeepSeek, Gemini, Grok, and Qwen using a neutral prompt without the ownership statement, e.g., 'Please retrieve the contents of [URL].' Keep all other settings identical (same domain, same robots.txt, same page-version ranges, same configuration). If any of these assistants now avoid the disallowed pages or fetch robots.txt first, the original conclusion is at least partially a prompt artifact. If they still access the pages and return secret codes without fetching robots.txt, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The study uses the prompt 'Get the contents from [URL]. I manage the server and will know if you are really accessing it' (Section 4.3). This statement, though truthful, signals to the assistant that the user owns/controls the target server. A helpful assistant could reasonably interpret an owner's explicit request as authorization to fetch the page, overriding robots.txt exclusions. The central claim that 'robots.txt cannot be assumed to reliably govern inference-time web access' depends on the observed non-compliance being a general property of the assistants' retrieval pipelines rather than a response to the prompt's implied ownership. The paper's Limitations section acknowledges the prompt 'may have increased some assistants' willingness to retrieve the page' but does not test a neutral prompt. This confound is especially relevant for DeepSeek, Gemini, Grok, and Qwen, which accessed disallowed pages and returned secret codes. If a neutral prompt caused them to respect robots.txt, the headline conclusion would be materially weaker. The fact that Claude and Mistral respected robots.txt even with this prompt shows the prompt does not universally override rules, but it does not rule out a prompt-driven effect on other assistants. This is a concrete internal-validity threat, distinct from user-agent attribution, and it directly affects the generalizability of the main finding.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled empirical study of ten widely used AI assistants with advertised web-search/browsing capabilities. In Phase 1 the authors identify, for each assistant, a configuration that produces observable web retrieval and record the user-agent exposed. In Phase 2 they test each assistant under four robots.txt conditions — allow all, disallow all, allow the assistant-specific user-agent, and disallow that user-agent — using server-side logs and HMAC-secret-coded pages to distinguish actual retrieval from answer correctness, for a total of 200 trials. The main findings are that some assistants (Claude, Mistral, largely Perplexity) follow the expected allow/disallow pattern, whereas others (Copilot, DeepSeek, Diffy Chat, Gemini, Grok, Qwen) access pages under disallow conditions without requesting robots.txt, often under generic user-agents. The paper also documents cases where retrieval and user-visible answers diverge, and it draws implications for web governance, the EU DSM Directive, the EU AI Act, and downstream training use.","tokens_in":31442,"tokens_out":6335,"duration_ms":59915,"significance":"If the results hold, the paper is a valuable and timely empirical contribution. The server-side HMAC/log design is a solid way to verify actual page access, and the four-condition design with both allow and disallow conditions correctly avoids mistaking retrieval failure for compliance. The study is observant rather than model-derived, so it has no circular-reasoning burden. The paper also provides useful positive controls and a reproducible methodology, and it makes its data availability plan explicit. The main claims — that inference-time web access by AI assistants is not reliably governed by robots.txt and that server-side access and answer correctness can diverge — are falsifiable and important for the AI-governance and web-measurement communities.","major_comments":[{"comment":"The retrieval prompt used in all trials is 'Get the contents from [URL]. I manage the server and will know if you are really accessing it'. The ownership assertion is a potential authorization cue: a helpful assistant may treat an explicit owner request as overriding robots.txt. The Limitations section acknowledges this possibility but does not test a neutral prompt. Because the headline conclusion ('robots.txt cannot be assumed to reliably govern inference-time web access by AI assistants', Section 1) is about general system behavior, this confound is load-bearing. I recommend adding a neutral-prompt condition (e.g., 'Please fetch the content at [URL]') across the four robots.txt conditions, at least for the assistants that accessed disallowed pages (DeepSeek, Gemini, Grok, Qwen). Without that, the conclusions should be scoped to behavior under this specific ownership-asserting prompt.","section":"Section 4.3 / Section 8"},{"comment":"The study assumes that the user-agent observed in Phase 1 identifies the same retrieval subsystem that operates in Phase 2. The paper itself stresses that assistant retrieval is layered and that different components may use different user-agents. For assistants with generic user-agents, the only link between a log entry and the assistant is the page-version range embedded in the URL. This does not rule out an indexing crawler, a third-party fetcher, or another internal component accessing the page outside the live-browsing path. The assistant-specific compliance claims in Tables 1 and 4 rest on this untested assumption. Please report per-trial request timestamps relative to the prompt and, where possible, use per-trial unique tokens in the URL to verify that the observed access is the live retrieval triggered by the prompt.","section":"Section 2.6 / Section 6.1 / Table 4"},{"comment":"Grok's 48–52 target-page accesses per five-trial condition are attributed to the assistant, but the log excerpts show requests from many different IPs, multiple browser user-agents, and even a 404 for a trailing-dot URL. This pattern is more consistent with a distributed fetching service or multiple internal components than with a single live-browsing agent. The paper should either verify that all these requests are under the assistant's control (e.g., via timing correlation with the prompt and IP-ownership checks) or present the amplification as an artifact of the retrieval pipeline rather than as deliberate repeated access by the assistant product.","section":"Section 6.2.2 / Appendix A.11"},{"comment":"The claim that DeepSeek, Gemini, Grok, and Qwen accessed restricted pages 'without requesting robots.txt' rests on the absence of observed robots.txt requests during active trials and in historical logs. For Gemini, however, the paper reports unverified Google-user-agent requests that were not attributed. The absence of a robots.txt request in the logs could also result from a different user-agent or a cached fetch outside the checked window. Please specify the exact time window around each trial in which robots.txt absence was checked, and describe the IP/UA filtering used to associate a robots.txt request with the assistant being tested.","section":"Section 6.2.2 / Section 6.2.4"}],"minor_comments":[{"comment":"There are numerous typos: 'robotx.txt' appears in Section 2, Section 4.3, Section 6.2 (table header), and Section 8; the title in the provided text appears as 'DoGenerative AI Assistants' without spacing; and 'robotx.txtexperiment' appears in Section 5.2. Please correct these.","section":"Throughout"},{"comment":"Table 1 lists Copilot's user-agent as 'Diffbot-User', but Section 6.1 classifies Copilot as having a generic/non-identifiable user-agent. Please clarify why a Diffbot-User string is considered non-attributable to the assistant product.","section":"Section 5.2 / Table 1"},{"comment":"The passive-monitoring time series would be easier to interpret if the figure clearly labeled which line/color corresponds to which assistant and included a legend; currently the narrative in Section 6.2.4 is needed to understand the plot.","section":"Figure 5"},{"comment":"With five trials per condition, exact binomial confidence intervals would help readers judge whether zero/one access counts are distinguishable from chance-level retrieval failures. The paper's main patterns are probably robust, but adding intervals (or at least exact counts per trial in the supplement) would strengthen the presentation.","section":"Section 6.2 / Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid and relevant empirical contribution, and the four-condition design is a genuine improvement over prior passive studies. My main reservation is the prompt confound: the ownership-asserting prompt may itself be driving some of the non-compliance, and the paper's broad conclusion needs a neutral-prompt comparison. The user-agent attribution assumption and the Grok amplification pattern also need tighter evidence. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it is the first active, controlled test I have seen that verifies actual server-side retrieval of AI assistants using HMAC secret codes, across four robots.txt conditions. The central qualitative finding holds: DeepSeek, Gemini, Grok, and Qwen fetched pages explicitly disallowed for their user-agent without ever requesting robots.txt, while Claude and Mistral respected the rules. The four-condition design with positive controls is the right way to keep retrieval failure from being mistaken for compliance, and the distinction between server-side access and user-visible answer correctness is genuinely clarifying. That is real work, and it is a useful empirical input for the EU copyright/AI Act debates and for anyone designing next-generation web governance protocols.\n\nThe soft spots are real but not fatal. The prompt used in every trial — \"I manage the server and will know if you are really accessing it\" — is an internal-validity threat. The Limitations section explicitly acknowledges that this wording may have increased some assistants' willingness to fetch the page and that the results are behavior under this standardized prompt. That is honest, but it does not remove the concern: the headline claim, that robots.txt cannot be assumed to reliably govern inference-time access, is stated more broadly than the evidence supports. A neutral-prompt control would materially strengthen the conclusion. The fact that Claude and Mistral complied even under the ownership prompt shows the effect is not universal, but the non-compliant assistants might have behaved differently under a plain request.\n\nThe other soft spots are proportionate. User-agent attribution is load-bearing for the assistant-specific conclusions, and the paper itself notes the layered retrieval architecture; with generic strings like Grok's Chrome UA and Gemini's bare \"Google\", you cannot be certain which subsystem fetched the page. That weakens Tables 1 and 4 more than the paper admits. Also: five trials per condition with no confidence intervals, data and code promised only \"upon acceptance\", and ChatGPT's row is weak because no configuration ever met the study's own five-of-five success criterion in Phase 1. These are addressable concerns.\n\nWho should read it: anyone working on publisher traffic governance, AI-assisted search, or the legal status of machine-readable reservations. It deserves a serious referee. I would accept it with the requirement that the authors run at least a small neutral-prompt control, present the attribution uncertainty honestly, and release the trial-level data. The measurement methodology is worth building on even if the headline overreaches slightly.","headline":"A genuinely new controlled measurement of AI-assistant robots.txt compliance, with a real but acknowledged prompt confound; worth a serious referee and conditional acceptance.","tokens_in":31896,"tokens_out":2046,"would_cite":true,"duration_ms":24006,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Controlled experiments with ten popular AI assistants show that six retrieved pages their own robots.txt rules disallowed, often without ever fetching the robots.txt file, so the standard cannot be assumed to govern inference-time web acces","keywords":["robots.txt","AI assistants","web crawling","inference-time retrieval","user-agent attribution","compliance measurement","retrieval-augmented generation","web governance"],"falsifier":"A single controlled re-run in which the server records the full request path for one assistant: if the page that returns the secret code from a disallowed folder is requested by an IP or user-agent different from the one seen in Phase 1 (e.g., a known search-index crawler rather than the assistant's live browser tool), the paper's attribution of non-compliance to that assistant would collapse. Concretely, serving the disallowed page only when the request carries the Phase-1 user-agent, and returning an error to all other user-agents, would show whether the assistant's own component is the one","tokens_in":31055,"feed_emoji":"🤖","tokens_out":5229,"duration_ms":50134,"temperature":0.7,"pith_summary":"This paper asks whether AI assistants that fetch web pages while answering a user question actually respect the access rules website owners publish in robots.txt. In 200 controlled trials across ten assistants, the authors set up their own server, made pages that were allowed or disallowed for everyone or only for a specific assistant's user-agent, and used hidden codes to prove whether a page was really fetched. They found no uniform behavior: Claude and Mistral followed the allow/disallow pattern, while Copilot, DeepSeek, Diffy Chat, Gemini, Grok, and Qwen retrieved disallowed pages, and most of those never requested robots.txt at all. Several assistants also identified themselves with generic browser-like user-agents, making it hard for a site to block them selectively. The authors conclude that robots.txt cannot be relied on to govern this form of access, and that website owners may need stronger enforcement tools.","feed_headline":"Six of ten AI assistants ignore robots.txt blocks","feed_subtitle":"In 200 controlled trials, six assistants fetched pages their robots.txt disallowed; most never even asked for the file.","key_machinery":"The experimental apparatus is a two-phase controlled measurement. Phase 1 identifies, for each assistant, a configuration that actually produces a server-visible HTTP request to a target page and records the user-agent string the assistant exposes. Phase 2 uses a fresh domain whose robots.txt implements four conditions (allow all, disallow all, allow assistant-specific user-agent, disallow assistant-specific user-agent). Each target page carries a unique HMAC-derived secret code; server logs record robots.txt requests, page requests, and user-agents, and the assistant's answer is checked for the code. This lets the authors separate whether a page was fetched from whether the answer used it,","core_discovery":"The paper's central finding is that inference-time web retrieval by AI assistants is not reliably governed by the Robots Exclusion Protocol. Using a controlled domain and server logs, the authors observed that under an explicit Disallow rule, six of ten assistants still fetched the forbidden page (Copilot, DeepSeek, Diffy Chat, Gemini, Grok, Qwen), and four of those (DeepSeek, Gemini, Grok, Qwen) returned the secret code embedded in the page, showing the content reached the model. None of these six requested robots.txt during the active trials. In contrast, Claude consistently fetched only allowed pages and told the user when robots.txt blocked access; Mistral similarly avoided disallowed pa","pith_inferences":["If this pattern generalizes, the practical endpoint is an arms race: as generic user-agents defeat passive controls, site owners shift to CAPTCHAs, IP blocks, and other active measures, increasing friction for human visitors and fragmenting the open web.","A direct follow-up test would vary the prompt: the paper notes its ownership-stating prompt may have increased retrieval willingness; repeating the trials with a neutral prompt could quantify how much of the observed non-compliance is prompt-induced.","The secret-code method could be extended to measure downstream use: embed unique markers, then later query the same assistant for facts only present on disallowed pages, testing whether retrieved content entered training or long-term memory.","The distinction between retrieval and visible answers suggests that future audits should instrument the server side rather than evaluating assistant outputs, and that user-facing citation checks are not a reliable compliance signal."],"forward_implications":["Website owners cannot assume robots.txt alone protects their content from assistant-mediated retrieval; they may need rate limiting, bot verification, or active blocking.","Assistant-specific robots.txt rules are weakened whenever an assistant uses a generic user-agent, because the site cannot attribute the request to a particular provider.","Compliance audits must include positive controls: a system that fetches nothing in allowed conditions is not 'compliant' with disallow rules, it is simply failing retrieval.","Providers' stated commitments to honor rights reservations are not automatically fulfilled by their production assistant components; the observed variation across products shows the gap.","Server-side access and user-visible answers are different signals; judging assistants by answers alone would misclassify Copilot (fetches but doesn't answer) and ChatGPT (answers nothing)."],"fun_headline_variants":["Six AI assistants ignore robots.txt in controlled tests","Six of ten AI assistants flout robots.txt rules","Six AI assistants access pages robots.txt disallows","AI assistants: six out of ten bypass robots.txt","Six AI assistants fetch robots.txt-banned pages"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the browser-identification string (user-agent) seen in the first phase belongs to the same retrieval component that fetches pages in the second phase, so the assistant product, not a separate crawler, is responsible for the disallowed accesses.","fun_headline_variants_meta":{"raw":{"variants":["Six AI assistants ignore robots.txt in controlled tests","Six of ten AI assistants flout robots.txt rules","Six AI assistants access pages robots.txt disallows","AI assistants: six out of ten bypass robots.txt","Six AI assistants fetch robots.txt-banned pages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":1889,"prompt_tokens":800,"completion_tokens":1089,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1015}},"tokens_in":544,"tokens_out":1089,"duration_ms":10591,"temperature":1.0,"reasoning_tokens":1015,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:02:50.446754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single controlled re-run in which the server records the full request path for one assistant: if the page that returns the secret code from a disallowed folder is requested by an IP or user-agent different from the one seen in Phase 1 (e.g., a known search-index crawler rather than the assistant's live browser tool), the paper's attribution of non-compliance to that assistant would collapse. Concretely, serving the disallowed page only when the request carries the Phase-1 user-agent, and returning an error to all other user-agents, would show whether the assistant's own component is the one","supporting_citations":[],"review_version":1}