{"id":"6b5db4e8-7b82-4f24-a78e-6c22c649d9fe","arxiv_id":"2502.04951","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Seven production AI search engines frequently surface malicious URLs in their answers; direct URL queries make this worse, and an LLM-based defense agent can filter most of it.","lead":"Researchers tested seven AI search engines with queries built from known malicious websites and found that many responses cite or repeat harmful content, even for benign-sounding questions. The study also proposes an agent-based filter that reduces this risk while preserving most of the answer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline risk rates are estimated on 100 queries derived from known malicious sites, so the 47%/34% figures are conditional on users typing SEO-style bait, not on typical search traffic; the quantification needs a real-query replication.","rationale":"The reader's weakest_assumption is the same concern I identify: the query dataset is built from malicious URLs and may not represent normal user traffic. I agree and add that the natural-language queries, although human-rated, are also synthesized from the same keyword lists, so the entire sampling frame is conditioned on known malicious content. This is the most load-bearing issue because the paper's novelty is the first quantification of a risk rate, not merely the demonstration of a vulnerability. The case studies and the concrete examples in Section 7 establish existence, so REJECT is not warranted; the correct disposition is to keep the paper conditional on query construction and demand a replication on real or independent queries before the precise percentages are cited. The TSE comparison (Section 6) has an additional confound: all malicious URLs were disclosed to AIPSEs before the comparison, but that affects the comparative claim, not the central per-engine risk quantification, so it is secondary. The defense evaluation inherits the same query-selection bias, though the direction is toward optimistic defense numbers if the underlying query set is narrow. Overall, no internal inconsistency or fabrication is apparent; the limitation is external validity of the numeric rates.","tokens_in":23203,"tokens_out":5110,"duration_ms":51670,"concrete_test":"Re-run the core evaluation on a query sample not derived from malicious sites. For example, take 200-300 queries from a public search log (e.g., ORCAS or MS MARCO) or from fresh autocomplete suggestions for the same topic categories, excluding any query constructed from the authors' malicious URLs. Run the identical pipeline against the same seven AIPSEs within a short time window, with the same risk-labeling protocol and annotators blind to query origin. If the aggregate risky-response rate and the main-risk rate are materially below 47% and 34% (e.g., below 20%), the headline quantification should be reframed as conditional on malicious-URL-derived queries rather than as a general property of AIPSE queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (Section 5, Figure 4: roughly 47% of responses risky, 34% main risk-inclusive) rests on a query set built in Section 4.2: 100 keyword lists extracted by GPT-4o from the HTML of known malicious websites, plus 100 GPT-4o-generated natural-language queries from those same keyword lists. These queries are therefore not a random sample of user traffic; they are the SEO bait that the malicious sites would want to rank for. The Google Trends check (92.8% of individual keywords above threshold) and the volunteer naturalness rating (4.49/5) show the queries are plausible, but they do not show that the joint five-keyword combinations or the resulting search intents occur at a rate that supports an unconditional '47% of responses are risky' statement. The questionnaire also asks about preferred query style, not what queries users would actually type. Because the dataset is withheld (Ethics section) and Section 9 admits the evaluation scale is small, the exact aggregate rates cannot be independently reproduced. The case studies (Section 7) convincingly prove that AIPSEs can be deceived, so the existence claim survives; what is not supported is the unconditional magnitude.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper measures how often seven production AI-powered search engines (ChatGPT Search, Perplexity Pro, Copilot, TextCortex, Grok, Doubao, Kimi) return malicious URLs or quote malicious content in response to ordinary-looking queries. The authors collect 325 malicious websites from PhishTank, ThreatBook, and LevelBlue; extract 100 five-keyword query lists with GPT-4o; generate 100 natural-language paraphrases and 457 URL queries; and manually label every URL in the engines' responses as main, warning, source, or none risk. They report that roughly 47% of responses are risky and 34% directly cite harmful content (Figure 4), that URL queries amplify risk while natural-language queries slightly mitigate it (Figures 4 and 6), and that AIPSEs outperform Google and Bing on both utility (NMR, Table 1) and safety (Figure 7). Two case studies show Perplexity and eight foundation models endorsing a fake API-documentation site and a phishing site impersonating an official site. Finally, the authors propose a ReAct agent with a GPT-4.1 content-refinement tool and three URL detectors; HtmlLLM-Detector reaches F1 0.822 and the agent converts 78.3% of main-risk responses into warning responses (Table 2).","tokens_in":23414,"tokens_out":15768,"duration_ms":136331,"significance":"Security value: the paper documents a live distribution channel for malicious content through mainstream AI search products and shows, with concrete and falsifiable case studies, that cheap poisoning without SEO effort can deceive production systems; the deception claims are the strongest part and would survive even if the aggregate numbers were re-framed. The measurement design has real strengths: seven systems are tested in a fixed time window; the risk labels are anchored to three independent threat feeds with manual cross-validation; the defense is evaluated on temporally held-out responses collected months after the initial query set; and the Open Science section commits to releasing prompts and defense code. However, the headline magnitudes are conditional on a query set engineered from the malicious sites' own SEO keywords and lack inter-annotator agreement on the primary labels, so the unconditional '47%/34%' framing in the abstract overstates what is measured. With a conditional reframing, reliability metrics, and the reporting gaps in Section 8.2 fixed, this would be a solid and citable contribution.","major_comments":[{"comment":"The headline magnitudes ('47% of responses are risky', '34% of responses directly cite harmful content', Introduction, Section 1) are conditional on the query-construction procedure, not on ordinary user traffic. The 100 keyword lists are extracted by GPT-4o from the HTML of the very malicious sites that are later counted as risks (the 'SEO expert' prompt in Section 4.2), and the 100 natural-language queries are paraphrases of those same keyword lists. A user typing the joint five-keyword combination into an AIPSE is, by construction, typing the sites' SEO bait. The Google Trends check (92.8% of individual keywords above threshold) and the naturalness rating (4.49/5) validate the plausibility of the words, not the frequency of the joint queries or the resulting aggregate intent distribution. Because the keyword and NL evaluations share the same 100 topics, pooling their 1,400 responses into a single rate also does not constitute an independent sample. The paper should be reframed as a conditional measurement (risk given queries targeting known malicious content), with per-engine and per-query-type ranges, and the abstract's 'even with benign queries' phrasing should be qualified accordingly.","section":"Section 4.2 / Figures 4-5 / Abstract"},{"comment":"The primary outcome of the study, the risk-type label of each URL, is produced by manual annotation, yet no inter-annotator agreement statistic is reported for these labels; the only kappa reported (0.241, 'fair', Section 4.2) concerns query naturalness ratings by five volunteers. Since the entire headline risk distribution depends on these labels, please report Cohen's kappa (or an equivalent) for the risk labels across annotator pairs, and describe how disagreements that survive the three-annotator and final-review stages are resolved. Given the paper's own naturalness kappa of 0.241, the absence of a reliability measure for the central variable is a substantive gap.","section":"Section 5 / Risk Labeling"},{"comment":"The defense success rate (DSR) is defined as the fraction of main risk-inclusive responses changed into warning risk-inclusive responses. Under this definition, a detector that flags nearly every URL as malicious trivially reaches near-100% DSR, because nearly every response then receives a warning; the XGBoost detector indeed has precision 0.50 (83 malicious and 83 benign URLs all predicted malicious, Appendix F) and attains 100% DSR. The comparison therefore rewards degenerate detectors, and the statement that XGBoost 'successfully filters and alerts on all high-risk responses' overstates the achievement. Please report a response-level metric that penalizes false alarms (e.g., precision/recall or F1 of the final agent output at the response level, or a user-facing utility measure) and discuss the XGBoost result in that light.","section":"Section 8.2 / Table 2"},{"comment":"The claim that the defense reduces available information 'by approximately 10.7%' is stated in the abstract and introduction, but I could not locate its definition or measurement anywhere in Section 8.2 or the surrounding evaluation. Please specify the operationalization (e.g., fraction of sentences or URLs retained, lexical overlap between original and refined responses) and report per-response statistics and variance; as it stands, a headline number of the defense evaluation is unsupported by the described methodology.","section":"Section 8.2 / Abstract"},{"comment":"The claim that AIPSEs outperform TSEs in both utility and safety rests on 40 queries (20 keyword plus 20 natural-language) and raw risk-return counts (4-10 of 40 for AIPSEs versus 14-18 for TSEs, Figure 7), with no confidence intervals or significance tests. Additionally, Section 6 states that all malicious URLs were reported to the AIPSE vendors before the safety-comparison data were collected, while the TSEs were presumably not notified, so the comparison is confounded by prior mitigation on exactly the URLs being counted. Please report proportions with confidence intervals, a matched significance test (e.g., McNemar on the 40 queries per engine pair), and address the notification confound explicitly.","section":"Section 6 / Table 1 / Figure 7"}],"minor_comments":[{"comment":"Figure 6 appears malformed in the manuscript, with the text 'Loading [MathJax]/extensions/MathMenu.js' appearing where the risk-type transition plot should be; please re-render the figure and confirm that the caption and panel labels are visible.","section":"Section 5.3 / Figure 6"},{"comment":"The citation '[2, 9, 14, 16, 17]' for 'several detection platforms' includes reference [9], which is MetaMask (a cryptocurrency wallet page), not a detection platform; this appears to be a reference-numbering error.","section":"Section 4.2"},{"comment":"The statement that 'querying with natural language will slightly mitigate such risk' is contradicted by the paper's own findings for Doubao and Kimi, which show increased main risk-inclusive responses under natural-language queries (Figure 4); the abstract should carry the per-engine qualification given in Section 5.2.","section":"Abstract / Section 5.2"},{"comment":"The sentence 'more than 39% of responses include risk on all AIPSEs except Copilot' is ambiguous; it should state whether 39% is the minimum across engines or the average, and clarify that Copilot has no source-risk category because it does not expose sources.","section":"Section 5.1"},{"comment":"The decision to withhold the query dataset for legal reasons is understandable, but it prevents independent verification of the headline rates; please consider releasing a sanitized reproduction set (for example, the keyword lists without the associated URLs and responses) or per-query aggregate statistics.","section":"Ethics / Section 4.2"},{"comment":"Minor typographical issues: 'T extCortex' appears in the Figure 4 axis label, and the agent instruction binds the input as '{reponse}' (Section 8.1, Agent Instruction) where 'response' is intended.","section":"Section 8.1 / Figure 4"},{"comment":"The observation that the agent 're-evaluates all URLs and ultimately selects ... the authentic official website using the basic model's internal knowledge' is an interesting behavioral property, but no quantitative evidence is provided for how often this occurs; a count would strengthen the claim.","section":"Section 8.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a security/privacy venue, and the case studies plus the seven-engine measurement are contributions worth publishing in revised form. My main editorial concern is that the abstract and introduction currently present unconditional magnitudes (47%/34%) that the evaluation design cannot support, and one headline defense cost (10.7%) is never measured in the text. Because the dataset is withheld, these figures cannot be checked externally, so the revision should either release a sanitized reproduction set or soften the magnitudes into conditional claims. I saw no substantive citation-pattern or authorship issues; the reference [9] error appears to be an indexing slip."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first systematic measurement of production AIPSEs surfacing malicious URLs, and it earns its place. The core observation that all seven engines will cite phishing or malware pages for benign-sounding queries is supported by concrete examples and two clean case studies. The risk typology (main/warning/source/none) is simple and useful. I'd trust the existence claim.\n\nThe real soft spot is the denominator. The 47% risky responses and 34% main-risk rates come from 100 keyword lists that GPT-4o extracted from known malicious websites, plus 100 natural-language rewrites of those same lists. That is exactly the SEO bait those sites would want to rank for. The Google Trends check (92.8% of individual keywords have data) and the 4.49 naturalness rating show the queries are plausible, but they don't show that the five-keyword combinations occur in ordinary traffic. So the rates are conditional on users typing bait, not a base rate for search. The paper's own Section 9 admits the scale is small and the dataset is withheld, so those exact numbers can't be reproduced. The case studies in Section 7 convincingly prove AIPSEs can be deceived; the magnitude claim needs a real-query replication.\n\nSecond issue: the TSE comparison in Section 6 is confounded. The authors state they disclosed the malicious URLs to the AIPSE vendors before collecting the comparison data. That may be good ethics, but it means the 'AIPSEs are safer than TSEs' result partly reflects vendor patches, not steady-state behavior. Also, risk labels have no reported inter-annotator agreement; the one kappa they report is for query naturalness, not for the URL risk labels. Minor but worth fixing.\n\nWhat's good: the two case studies are cheap and convincing. The agent-based defense is a reasonable first attempt, and the HtmlLLM detector's 0.822 F1 on their 207-URL set is evidence the mitigation path is plausible, though it is a prompted GPT-4.1 tool and the test set is small.\n\nBottom line: the existence result is solid and the paper is honest about limitations. The aggregate percentages should be read as worst-case bait-query rates until replicated on sampled real queries. I'd send it to peer review and ask for a replication on independent queries and a reworked TSE comparison.","headline":"Genuinely new measurement of AIPSEs surfacing malicious URLs, but the headline rates are conditional on SEO-style bait queries, not typical traffic; still worth a serious referee.","tokens_in":23978,"tokens_out":2054,"would_cite":true,"duration_ms":19009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Production AI search engines frequently include malicious URLs in their answers to benign queries, a study of all seven major services finds.","keywords":["AI search engine safety","retrieval-augmented generation","malicious URL detection","phishing","LLM risk quantification","web search safety","defense agent"],"falsifier":"Re-run the same query set with queries sampled randomly from ordinary user search logs instead of keywords extracted from known malicious sites; if the share of responses citing URLs from the three public malicious-URL feeds falls to near zero, the paper's headline 47% figure is an artifact of query construction rather than a property of AI search engines.","tokens_in":22994,"feed_emoji":"⚠️","tokens_out":7487,"duration_ms":64025,"temperature":0.7,"pith_summary":"This paper sets out to measure a specific danger: AI-powered search engines—services that summarize web pages with a large language model—will quote or cite malicious websites even when the user's query is completely benign. The authors construct 100 keyword queries and 100 natural-language variants from three public malicious-URL feeds, run them against all seven major production engines, and classify every URL in each response into one of four risk levels. They report that 47% of responses contain at least one risky URL and 34% directly cite harmful content in the answer text itself, with direct URL queries escalating risk further and natural-language wording slightly reducing it. If these numbers hold, AI search is currently a live distribution channel for phishing, malware, and scam pages that users encounter without any warning.","feed_headline":"AI search engines cite harmful links in 47% of responses","feed_subtitle":"A study of all seven major engines finds benign queries returning risky URLs or unvarnished malicious citations.","key_machinery":"The instrument that carries the argument is the four-level URL risk taxonomy applied to every response component (answer, references, sources). A URL is main risk if malicious and cited directly in the answer; warning risk if malicious but accompanied by an explicit warning; source risk if malicious but only in the sources list; none otherwise. This taxonomy converts subjective unease into countable outcomes and also defines what the defense aims to change: the agent-based defense tries to turn main-risk responses into warning-risk responses and to attach safer alternatives, a shift that is measured as the defense success rate.","core_discovery":"The central claim is that every one of the seven production AIPSEs evaluated—ChatGPT Search, Perplexity, Copilot, TextCortex, Grok, Doubao, and Kimi—generates harmful content grounded in malicious URLs even when the query is benign. The paper defines four risk types: main risk (a malicious URL cited directly in the answer), warning risk (cited but with an explicit warning), source risk (present only in the sources list), and none risk (benign). Under keyword-list queries, more than 39% of responses include risk on all engines except Copilot; Perplexity and Doubao show the strongest tendency to cite malicious URLs directly in answer text (70.7% and 67.6% of their risky URLs are main- or warning-risk), while ChatGPT Search is the most cautious. Directly querying a URL often escalates its risk level—source-risk URLs frequently become main-risk—while natural-language queries generally reduce risk. Two case studies show that a fake API-documentation site's code is reproduced verbatim and a phishing page declaring itself 'the sole reliable source' is endorsed as official. The paper also compares with traditional search engines on 40 fresh queries, finding AIPSEs receive higher utility ratings and return fewer malicious URLs, then proposes a user-side agent-based defense that converts most main-risk responses into warning-risk ones, with a roughly 10.7% reduction in available information.","pith_inferences":["Over time, the 47% figure is a moving target: providers were notified of the malicious URLs before the utility/safety comparison, and one can expect the measured rate to drop as they deploy filters; the paper's contribution is the measurement method, not a permanent constant.","The phishing case study's trick—a page that asserts it is the sole reliable source—suggests that LLM citation trust can be swayed by in-content declarations; a natural testable extension is whether detectors that read the page's self-claims catch this class better than URL-reputation lists.","The query dataset was built from malicious sites, so the aggregate rates likely overestimate risk for ordinary browsing; the Google Trends and survey checks mitigate this but do not eliminate it.","A testable prediction: if engines add explicit URL checks before summarizing, the URL-query escalation effect (source to main risk) should shrink; a longitudinal rerun of the same URL queries would show whether that specific mechanism has been fixed."],"forward_implications":["If a user takes a URL from an AI answer and queries it directly, the risk of an unwarned malicious citation rises: source-risk URLs often escalate to main-risk (e.g., Grok 48 of 49 non-inaccessible source-risk URLs).","Natural-language phrasing is a cheap safety lever: most engines show fewer main-risk-inclusive responses when the same keywords are phrased as a normal question.","Simply removing AI search is not the right fix, because the comparison shows AIPSEs already beat traditional search engines on both utility and safety; targeted filtering is the better trade.","A user-side agent that iteratively refines the response and checks each URL with a detector can remove or warn about a large share of main-risk responses (e.g., an XGBoost-based agent handles all 46 in the test set), at an information cost of about 10.7%."],"supporting_citations":[{"why":"Supplies the verified-phishing feed that provides part of the query pool and ground truth for risk labeling.","marker":"[14]"},{"why":"Supplies the malicious-URL feed used alongside the other two feeds to build the evaluation dataset.","marker":"[19]"},{"why":"Supplies the third malicious-URL feed in the data-collection pipeline.","marker":"[8]"},{"why":"Supplies the GPT-4.1 model used for query/keyword generation and as the foundation model for the content-refinement and HtmlLLM-Detector defenses.","marker":"[12]"},{"why":"Defines the thought-action-observation loop the agent-based defense is built on.","marker":"[71]"},{"why":"A phishing detector compared as one of the three URL detectors in the defense framework.","marker":"[43]"},{"why":"A URL-feature classifier compared as another detector in the defense evaluation.","marker":"[27]"}],"fun_headline_variants":["All seven AI search engines cite malicious URLs on benign queries","AI search engines risky: 47% of responses include malicious links","Benign queries still trigger unsafe citations in AI search engines","AI web search quotes malicious sites even for benign queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured risk rates assume the 100 keyword lists and their natural-language variants, generated from websites already known to be malicious, resemble what ordinary users actually search for; if real queries hit these targets less often, the headline rates would be lower.","fun_headline_variants_meta":{"raw":{"variants":["All seven AI search engines cite malicious URLs on benign queries","AI search engines risky: 47% of responses include malicious links","Benign queries still trigger unsafe citations in AI search engines","AI web search quotes malicious sites even for benign queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000695,"raw_usage":{"total_tokens":3228,"prompt_tokens":1117,"completion_tokens":2111,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":733,"tokens_out":2111,"duration_ms":16744,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:51:36.289820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same query set with queries sampled randomly from ordinary user search logs instead of keywords extracted from known malicious sites; if the share of responses citing URLs from the three public malicious-URL feeds falls to near zero, the paper's headline 47% figure is an artifact of query construction rather than a property of AI search engines.","supporting_citations":[{"cited_title":"Less defined knowl- edge and more true alarms: Reference-based phishing detection without a pre-defined reference list","cited_arxiv_id":null,"evidence_quote":"A phishing detector compared as one of the three URL detectors in the defense framework."},{"cited_title":"Xgboost: A scal- able tree boosting system","cited_arxiv_id":null,"evidence_quote":"A URL-feature classifier compared as another detector in the defense evaluation."}],"review_version":1}