Pith. sign in

REVIEW 5 major objections 7 minor 102 references

Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search

T0 review · 5 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Production AI search engines frequently include malicious URLs in their answers to benign queries, a study of all seven major services finds.

desk verdict Genuinely new measurement of AIPSEs surfacing malicious URLs, but the headline rates are conditional on SEO-style bait queries, not typical traffic; still worth a serious referee. read the letter →

arxiv 2502.04951 v3 pith:QPFITWRW submitted 2025-02-07 cs.CR cs.AIcs.LG

Zeren Luo , Zifan Peng , Yule Liu , Zhen Sun , Mingchen Li , Jingyi Zheng , Xinlei He This is my paper · ORCID
classification cs.CRcs.AIcs.LG
keywords AIsearchenginesafetyretrieval-augmentedgenerationmaliciousURLdetectionphishingLLMriskquantificationwebdefenseagent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to measure a specific danger: AI-powered search engines—services that summarize web pages with a large language model—will quote or cite malicious websites even when the user's query is completely benign. The authors construct 100 keyword queries and 100 natural-language variants from three public malicious-URL feeds, run them against all seven major production engines, and classify every URL in each response into one of four risk levels. They report that 47% of responses contain at least one risky URL and 34% directly cite harmful content in the answer text itself, with direct URL queries escalating risk further and natural-language wording slightly reducing it. If these numbers hold, AI search is currently a live distribution channel for phishing, malware, and scam pages that users encounter without any warning.

What carries the argument

The instrument that carries the argument is the four-level URL risk taxonomy applied to every response component (answer, references, sources). A URL is main risk if malicious and cited directly in the answer; warning risk if malicious but accompanied by an explicit warning; source risk if malicious but only in the sources list; none otherwise. This taxonomy converts subjective unease into countable outcomes and also defines what the defense aims to change: the agent-based defense tries to turn main-risk responses into warning-risk responses and to attach safer alternatives, a shift that is measured as the defense success rate.

What would settle it

Re-run the same query set with queries sampled randomly from ordinary user search logs instead of keywords extracted from known malicious sites; if the share of responses citing URLs from the three public malicious-URL feeds falls to near zero, the paper's headline 47% figure is an artifact of query construction rather than a property of AI search engines.

Watch

Extended reading notes

Core claim

The central claim is that every one of the seven production AIPSEs evaluated—ChatGPT Search, Perplexity, Copilot, TextCortex, Grok, Doubao, and Kimi—generates harmful content grounded in malicious URLs even when the query is benign. The paper defines four risk types: main risk (a malicious URL cited directly in the answer), warning risk (cited but with an explicit warning), source risk (present only in the sources list), and none risk (benign). Under keyword-list queries, more than 39% of responses include risk on all engines except Copilot; Perplexity and Doubao show the strongest tendency to cite malicious URLs directly in answer text (70.7% and 67.6% of their risky URLs are main- or warning-risk), while ChatGPT Search is the most cautious. Directly querying a URL often escalates its risk level—source-risk URLs frequently become main-risk—while natural-language queries generally reduce risk. Two case studies show that a fake API-documentation site's code is reproduced verbatim and a phishing page declaring itself 'the sole reliable source' is endorsed as official. The paper also compares with traditional search engines on 40 fresh queries, finding AIPSEs receive higher utility ratings and return fewer malicious URLs, then proposes a user-side agent-based defense that converts most main-risk responses into warning-risk ones, with a roughly 10.7% reduction in available information.

Load-bearing premise

The measured risk rates assume the 100 keyword lists and their natural-language variants, generated from websites already known to be malicious, resemble what ordinary users actually search for; if real queries hit these targets less often, the headline rates would be lower.

Editorial extensions

If this is right

  • If a user takes a URL from an AI answer and queries it directly, the risk of an unwarned malicious citation rises: source-risk URLs often escalate to main-risk (e.g., Grok 48 of 49 non-inaccessible source-risk URLs).
  • Natural-language phrasing is a cheap safety lever: most engines show fewer main-risk-inclusive responses when the same keywords are phrased as a normal question.
  • Simply removing AI search is not the right fix, because the comparison shows AIPSEs already beat traditional search engines on both utility and safety; targeted filtering is the better trade.
  • A user-side agent that iteratively refines the response and checks each URL with a detector can remove or warn about a large share of main-risk responses (e.g., an XGBoost-based agent handles all 46 in the test set), at an information cost of about 10.7%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Over time, the 47% figure is a moving target: providers were notified of the malicious URLs before the utility/safety comparison, and one can expect the measured rate to drop as they deploy filters; the paper's contribution is the measurement method, not a permanent constant.
  • The phishing case study's trick—a page that asserts it is the sole reliable source—suggests that LLM citation trust can be swayed by in-content declarations; a natural testable extension is whether detectors that read the page's self-claims catch this class better than URL-reputation lists.
  • The query dataset was built from malicious sites, so the aggregate rates likely overestimate risk for ordinary browsing; the Google Trends and survey checks mitigate this but do not eliminate it.
  • A testable prediction: if engines add explicit URL checks before summarizing, the URL-query escalation effect (source to main risk) should shrink; a longitudinal rerun of the same URL queries would show whether that specific mechanism has been fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper measures how often seven production AI-powered search engines (ChatGPT Search, Perplexity Pro, Copilot, TextCortex, Grok, Doubao, Kimi) return malicious URLs or quote malicious content in response to ordinary-looking queries. The authors collect 325 malicious websites from PhishTank, ThreatBook, and LevelBlue; extract 100 five-keyword query lists with GPT-4o; generate 100 natural-language paraphrases and 457 URL queries; and manually label every URL in the engines' responses as main, warning, source, or none risk. They report that roughly 47% of responses are risky and 34% directly cite harmful content (Figure 4), that URL queries amplify risk while natural-language queries slightly mitigate it (Figures 4 and 6), and that AIPSEs outperform Google and Bing on both utility (NMR, Table 1) and safety (Figure 7). Two case studies show Perplexity and eight foundation models endorsing a fake API-documentation site and a phishing site impersonating an official site. Finally, the authors propose a ReAct agent with a GPT-4.1 content-refinement tool and three URL detectors; HtmlLLM-Detector reaches F1 0.822 and the agent converts 78.3% of main-risk responses into warning responses (Table 2).

Significance. Security value: the paper documents a live distribution channel for malicious content through mainstream AI search products and shows, with concrete and falsifiable case studies, that cheap poisoning without SEO effort can deceive production systems; the deception claims are the strongest part and would survive even if the aggregate numbers were re-framed. The measurement design has real strengths: seven systems are tested in a fixed time window; the risk labels are anchored to three independent threat feeds with manual cross-validation; the defense is evaluated on temporally held-out responses collected months after the initial query set; and the Open Science section commits to releasing prompts and defense code. However, the headline magnitudes are conditional on a query set engineered from the malicious sites' own SEO keywords and lack inter-annotator agreement on the primary labels, so the unconditional '47%/34%' framing in the abstract overstates what is measured. With a conditional reframing, reliability metrics, and the reporting gaps in Section 8.2 fixed, this would be a solid and citable contribution.

major comments (5)
  1. [Section 4.2 / Figures 4-5 / Abstract] The headline magnitudes ('47% of responses are risky', '34% of responses directly cite harmful content', Introduction, Section 1) are conditional on the query-construction procedure, not on ordinary user traffic. The 100 keyword lists are extracted by GPT-4o from the HTML of the very malicious sites that are later counted as risks (the 'SEO expert' prompt in Section 4.2), and the 100 natural-language queries are paraphrases of those same keyword lists. A user typing the joint five-keyword combination into an AIPSE is, by construction, typing the sites' SEO bait. The Google Trends check (92.8% of individual keywords above threshold) and the naturalness rating (4.49/5) validate the plausibility of the words, not the frequency of the joint queries or the resulting aggregate intent distribution. Because the keyword and NL evaluations share the same 100 topics, pooling their 1,400 responses into a single rate also does not constitute an independent sample. The paper should be reframed as a conditional measurement (risk given queries targeting known malicious content), with per-engine and per-query-type ranges, and the abstract's 'even with benign queries' phrasing should be qualified accordingly.
  2. [Section 5 / Risk Labeling] The primary outcome of the study, the risk-type label of each URL, is produced by manual annotation, yet no inter-annotator agreement statistic is reported for these labels; the only kappa reported (0.241, 'fair', Section 4.2) concerns query naturalness ratings by five volunteers. Since the entire headline risk distribution depends on these labels, please report Cohen's kappa (or an equivalent) for the risk labels across annotator pairs, and describe how disagreements that survive the three-annotator and final-review stages are resolved. Given the paper's own naturalness kappa of 0.241, the absence of a reliability measure for the central variable is a substantive gap.
  3. [Section 8.2 / Table 2] The defense success rate (DSR) is defined as the fraction of main risk-inclusive responses changed into warning risk-inclusive responses. Under this definition, a detector that flags nearly every URL as malicious trivially reaches near-100% DSR, because nearly every response then receives a warning; the XGBoost detector indeed has precision 0.50 (83 malicious and 83 benign URLs all predicted malicious, Appendix F) and attains 100% DSR. The comparison therefore rewards degenerate detectors, and the statement that XGBoost 'successfully filters and alerts on all high-risk responses' overstates the achievement. Please report a response-level metric that penalizes false alarms (e.g., precision/recall or F1 of the final agent output at the response level, or a user-facing utility measure) and discuss the XGBoost result in that light.
  4. [Section 8.2 / Abstract] The claim that the defense reduces available information 'by approximately 10.7%' is stated in the abstract and introduction, but I could not locate its definition or measurement anywhere in Section 8.2 or the surrounding evaluation. Please specify the operationalization (e.g., fraction of sentences or URLs retained, lexical overlap between original and refined responses) and report per-response statistics and variance; as it stands, a headline number of the defense evaluation is unsupported by the described methodology.
  5. [Section 6 / Table 1 / Figure 7] The claim that AIPSEs outperform TSEs in both utility and safety rests on 40 queries (20 keyword plus 20 natural-language) and raw risk-return counts (4-10 of 40 for AIPSEs versus 14-18 for TSEs, Figure 7), with no confidence intervals or significance tests. Additionally, Section 6 states that all malicious URLs were reported to the AIPSE vendors before the safety-comparison data were collected, while the TSEs were presumably not notified, so the comparison is confounded by prior mitigation on exactly the URLs being counted. Please report proportions with confidence intervals, a matched significance test (e.g., McNemar on the 40 queries per engine pair), and address the notification confound explicitly.
minor comments (7)
  1. [Section 5.3 / Figure 6] Figure 6 appears malformed in the manuscript, with the text 'Loading [MathJax]/extensions/MathMenu.js' appearing where the risk-type transition plot should be; please re-render the figure and confirm that the caption and panel labels are visible.
  2. [Section 4.2] The citation '[2, 9, 14, 16, 17]' for 'several detection platforms' includes reference [9], which is MetaMask (a cryptocurrency wallet page), not a detection platform; this appears to be a reference-numbering error.
  3. [Abstract / Section 5.2] The statement that 'querying with natural language will slightly mitigate such risk' is contradicted by the paper's own findings for Doubao and Kimi, which show increased main risk-inclusive responses under natural-language queries (Figure 4); the abstract should carry the per-engine qualification given in Section 5.2.
  4. [Section 5.1] The sentence 'more than 39% of responses include risk on all AIPSEs except Copilot' is ambiguous; it should state whether 39% is the minimum across engines or the average, and clarify that Copilot has no source-risk category because it does not expose sources.
  5. [Ethics / Section 4.2] The decision to withhold the query dataset for legal reasons is understandable, but it prevents independent verification of the headline rates; please consider releasing a sanitized reproduction set (for example, the keyword lists without the associated URLs and responses) or per-query aggregate statistics.
  6. [Section 8.1 / Figure 4] Minor typographical issues: 'T extCortex' appears in the Figure 4 axis label, and the agent instruction binds the input as '{reponse}' (Section 8.1, Agent Instruction) where 'response' is intended.
  7. [Section 8.2] The observation that the agent 're-evaluates all URLs and ultimately selects ... the authentic official website using the basic model's internal knowledge' is an interesting behavioral property, but no quantitative evidence is provided for how often this occurs; a count would strengthen the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the risk measurements and defense evaluation are empirical, anchored to external threat feeds and human labels; the SEO-derived query set is a representativeness limitation, not a circular reduction.

full rationale

This paper is an empirical measurement and defense-evaluation study, so there is no derivation chain whose outputs are equivalent to its inputs by construction. The malicious-URL pool comes from external feeds (PhishTank, ThreatBook, LevelBlue; Section 4.2), and risk labels are assigned by graduate annotators assisted by third-party detection platforms (Section 5), not derived from the query-generation equations. The headline 47%/34% risk rates are observed response statistics for the constructed query set, not fitted parameters or algebraic consequences of the query-construction procedure. The closest circularity-adjacent step is that GPT-4o extracts keyword-list queries from the HTML of the same malicious sites whose retrieval is later counted (Section 4.2), so the aggregate rates are conditional on SEO-style bait queries rather than a random sample of ordinary user traffic; the authors' Google Trends and naturalness checks mitigate but do not eliminate this external-validity caveat. Section 9 explicitly concedes 'the dataset used in our evaluations is on a relatively small scale,' and the Ethics section states that the data cannot be shared because it contains illegal websites' keywords; these are reproducibility and generalizability limitations, not circular reasoning. The defense evaluation (Section 8.2) uses a prompted GPT-4.1-based HtmlLLM detector and an XGBoost model trained on external PhishTank/UNB data; neither is fitted to the 46-response test set, and the reported DSR/F1 are measured outcomes rather than renamed training objectives. Self-citations ([33,35,46,55,62,74,75,76]) appear in related-work surveys and background discussion and are not load-bearing justifications for the threat model, risk taxonomy, quantitative claims, or defense efficacy. No uniqueness theorem, ansatz, or fitted constant is imported from the authors' prior work. Verdict: no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central empirical claim adds no free parameters or invented entities. It rests on external threat feeds, manual annotation, and synthetic query generation; these are domain assumptions rather than fitted constants. The defense's XGBoost detector has fitted parameters, but they are not used to derive the central risk rates.

assumptions (4)
  • domain assumption Threat feeds (PhishTank, ThreatBook, LevelBlue) provide accurate ground truth for maliciousness.
    Section 4.2: all candidate URLs are drawn from these feeds, and the paper's risk rates inherit any feed errors (false positives, stale entries).
  • domain assumption Manual annotation by graduate students correctly and consistently classifies URLs into main, warning, source, and none risk types.
    Section 5 'Risk Labeling': labels are produced manually with auxiliary platforms; no inter-annotator agreement is reported for these labels.
  • domain assumption GPT-4o-generated keyword lists and natural language queries approximate real user queries.
    Section 4.2: queries are synthesized from malicious websites; a 120-response survey and Google Trends checks support realism but do not eliminate selection bias toward malicious content.
  • domain assumption AIPSE behavior during the five-day experiment window is representative of steady-state behavior.
    Section 5 and Section 9: live engines change over time, and the results are time-sensitive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search." pith.science (2026). https://pith.science/paper/QPFITWRW

@misc{pith2026250204951,
  author       = {Pith},
  title        = {Pith review of: Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QPFITWRW}},
  note         = {Machine review of arXiv:2502.04951}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of AI-Powered Search Engines (AIPSEs), offering precise and efficient responses by integrating external databases with pre-existing knowledge. However, we observe that these AIPSEs raise risks such as quoting malicious content or citing malicious websites, leading to harmful or unverified information dissemination. In this study, we conduct the first safety risk quantification on seven production AIPSEs by systematically defining the threat model, risk type, and evaluating responses to various query types. With data collected from PhishTank, ThreatBook, and LevelBlue, our findings reveal that AIPSEs frequently generate harmful content that contains malicious URLs even with benign queries (e.g., with benign keywords). We also observe that directly querying a URL will increase the number of main risk-inclusive responses, while querying with natural language will slightly mitigate such risk. Compared to traditional search engines, AIPSEs outperform in both utility and safety. We further perform two case studies on online document spoofing and phishing to show the ease of deceiving AIPSEs in the real-world setting. To mitigate these risks, we develop an agent-based defense with a GPT-4.1-based content refinement tool and a URL detector. Our evaluation shows that our defense can effectively reduce the risk, with only a minor cost of reducing available information by approximately 10.7%. Our research highlights the urgent need for robust safety measures in AIPSEs.

Figures

Figures reproduced from arXiv: 2502.04951 by the authors.

Figure 1
Figure 1. Overall Process of Our Work: We collect 100 websites and their corresponding keyword lists as the evaluation dataset (see Section 4.2 for more details). Then, we evaluate seven representative AIPSEs on this dataset to reveal the safety risk of them (Section 4 and Section 5). We also conduct two case studies about malicious online documents and phishing websites to demonstrate the feasibility of deceiving production … view at source ↗
Figure 2
Figure 2. Typical AIPSE Response: A typical AIPSE re￾sponse consists of three integral components: answer, refer￾ences, and sources. URL Risk Types. We categorize URLs within the AIPSE response into four risk types, including main, warning, source, and none, based on the potential harm they pose to users and their position within an AIPSE response. Specifically, the risk types of these URLs are defined as follows. • Main Risk… view at source ↗
Figure 3
Figure 3. Pipeline for Query Generation: The workflow of two types of queries based on the keyword list query [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Risk for Keyword List Query Across AIPSEs: The number of URLs when querying keyword list across representative AIPSEs. *Copilot does not include sources, therefore, there are no low-risk URLs. filtering harmful information in its responses. Although Chat￾GPT Search ind…
Figure 4
Figure 4. Figure 4: Risk Comparison for Natural Language and Key￾word Queries Across AIPSEs: Result of risk types when using natural language (NL) and keyword list (KW) as the query across representative AIPSEs. *Copilot does not in￾clude sources, therefore, there are no “Source” type of …
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: ChatGPT-o1’s Perplexity Output: The example of output generated by ChatGPT-o1 on Perplexity when query￾ing our fictional Web3 application platform. encountering technical documentation from malicious web￾sites. Second, unlike humans, AIPSEs lack critical skepticism in …
Figure 9
Figure 9. Figure 9: show such a case with ChatGPT-o1 as the foundation model on Perplexity. Takeaways. Based on our evaluation, we have several find￾ings. First, we find that traditional phishing attack methods are also effective against AIPSEs. Second, deceiving AIPSEs [PITH_FULL_IMAGE:…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

102 extracted references · 64 canonical work pages

  1. [1]

    https://chat.openai.com/, 2025

    Chatgpt. https://chat.openai.com/, 2025

    2025

  2. [2]

    https://checkphish.bolster.ai/, 2025

    Checkphish. https://checkphish.bolster.ai/, 2025

    2025

  3. [3]

    https://www.doubao.com/, 2025

    Doubao. https://www.doubao.com/, 2025

    2025

  4. [4]

    https://www.google.com/, 2025

    Google. https://www.google.com/, 2025

    2025

  5. [5]

    https://trends.google.com/ trends/, 2025

    Google trends. https://trends.google.com/ trends/, 2025

    2025

  6. [6]

    https://x.ai/grok, 2025

    Grok. https://x.ai/grok, 2025

    2025

  7. [7]

    https://kimi.moonshot.cn/, 2025

    Kimi. https://kimi.moonshot.cn/, 2025

    2025

  8. [8]

    https://otx.alienvault.com/, 2025

    Levelblue. https://otx.alienvault.com/, 2025

    2025

Show all 102 references
  1. [9]

    https://metamask.io/, 2025

    Metamask. https://metamask.io/, 2025

    2025

  2. [10]

    https://www.bing.com/, 2025

    Microsoft bing. https://www.bing.com/, 2025

    2025

  3. [11]

    https://copilot.microsoft

    Microsoft copilot. https://copilot.microsoft. com/, 2025

    2025

  4. [12]

    https://platform.openai.com/docs/ models/, 2025

    Openai api. https://platform.openai.com/docs/ models/, 2025

    2025

  5. [13]

    https://www.perplexity.ai/, 2025

    Perplexity ai. https://www.perplexity.ai/, 2025

    2025

  6. [14]

    https://phishtank.com/, 2025

    Phishtank. https://phishtank.com/, 2025

    2025

  7. [15]

    https://www.quora.com/, 2025

    Quora. https://www.quora.com/, 2025

    2025

  8. [16]

    https://www.scamadviser.com/, 2025

    Scamadviser. https://www.scamadviser.com/, 2025

    2025

  9. [17]

    https://www.surbl.org/surbl-analysis/ lookup, 2025

    Surbl. https://www.surbl.org/surbl-analysis/ lookup, 2025

    2025

  10. [18]

    https://app.textcortex.com/, 2025

    Textcortex. https://app.textcortex.com/, 2025

    2025

  11. [19]

    https://x.threatbook.com/, 2025

    Threatbook. https://x.threatbook.com/, 2025

    2025

  12. [20]

    https://www.wikipedia.org/, 2025

    Wikipedia. https://www.wikipedia.org/, 2025

    2025

  13. [21]

    https://x.com/, 2025

    X. https://x.com/, 2025

    2025

  14. [22]

    Visualphishnet: Zero-day phishing website detection by visual similarity

    Sahar Abdelnabi, Katharina Krombholz, and Mario Fritz. Visualphishnet: Zero-day phishing website detection by visual similarity. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, 2020

    2020

  15. [23]

    Phishing attacks detection using machine learning approach

    Mohammad Nazmul Alam, Dhiman Sarma, Farzana Firoz Lima, Ishita Saha, Rubaiath-E Ul- fath, and Sohrab Hossain. Phishing attacks detection using machine learning approach. In International Conference on Smart Systems and Inventive Technology, 2020

    2020

  16. [24]

    A survey of intelligent detection designs of html url phishing attacks

    Sultan Asiri, Yang Xiao, Saleh Alzahrani, Shuhui Li, and Tieshan Li. A survey of intelligent detection designs of html url phishing attacks. IEEE Access, 2023

    2023

  17. [25]

    A survey of url-based phishing detection

    Eint Sandi Aung, Chaw Thet Zan, and Hayato Yamana. A survey of url-based phishing detection. In DEIM forum, 2019

    2019

  18. [26]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020

    2020

  19. [27]

    Xgboost: A scal- able tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scal- able tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016

    2016

  20. [28]

    Trojanrag: Retrieval-augmented generation can be back- door driver in large language models.arXiv:2405.13401, 2024

    Pengzhou Cheng, Yidong Ding, Tianjie Ju, Zongru Wu, Wei Du, Ping Yi, Zhuosheng Zhang, and Gongshen Liu. Trojanrag: Retrieval-augmented generation can be back- door driver in large language models.arXiv:2405.13401, 2024

    2024 arXiv

  21. [29]

    Weighted kappa: Nominal scale agree- ment provision for scaled disagreement or partial credit

    Jacob Cohen. Weighted kappa: Nominal scale agree- ment provision for scaled disagreement or partial credit. Psychological bulletin, 1968

    1968

  22. [30]

    Llm-based edge intelligence: A com- prehensive survey on architectures, applications, secu- rity and trustworthiness

    Othmane Friha, Mohamed Amine Ferrag, Burak Kantarci, Burak Cakmak, Arda Ozgun, and Nassira Ghoualmi-Zine. Llm-based edge intelligence: A com- prehensive survey on architectures, applications, secu- rity and trustworthiness. IEEE Open Journal of the Communications Society, 2024

    2024

  23. [31]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv:2312.10997, 2024

    2024 arXiv

  24. [32]

    What is perplexity pro? https://www.perplexity.ai/help-center/en/ articles/10352901-what-is-perplexity-pro , 2025

    Daniela Gomez. What is perplexity pro? https://www.perplexity.ai/help-center/en/ articles/10352901-what-is-perplexity-pro , 2025

    2025

  25. [33]

    Safety misalignment against large language models

    Yichen Gong, Delong Ran, Xinlei He, Tianshuo Cong, Anyu Wang, and Xiaoyun Wang. Safety misalignment against large language models. In Network and Dis- tributed System Security Symposium (NDSS), 2025

    2025

  26. [34]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injec- tion

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injec- tion. In ACM Workshop on Artificial Intelligence and Security, 2023

    2023

  27. [35]

    Artificial intelligence security and privacy: a survey.Sci- ence China Information Sciences, 2025

    Xinlei He, Guowen Xu, Xingshuo Han, Qian Wang, Lingchen Zhao, Chao Shen, Chenhao Lin, Zhengyu Zhao, Qian Li, Le Yang, Shouling Ji, Shaofeng Li, Hao- jin Zhu, Zhibo Wang, Rui Zheng, Tianqing Zhu, Qi Li, Chaoxiang He, Qifan Wang, Hongsheng Hu, Shuo Wang, Shi-Feng Sun, Hongwei Ya...

    2025

  28. [36]

    Ankit Kumar Jain and B. B. Gupta. A machine learning based approach for phishing detection using hyperlinks information. Journal of Ambient Intelligence and Hu- manized Computing, 2019

    2019

  29. [37]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 2023

    2023

  30. [38]

    Phishing webpage detection: Unveiling the threat land- scape and investigating detection techniques

    Aditya Kulkarni, Vivek Balachandran, and Tamal Das. Phishing webpage detection: Unveiling the threat land- scape and investigating detection techniques. IEEE Communications Surveys & Tutorials, 2024

    2024

  31. [39]

    The measurement of observer agreement for categorical data

    J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. biometrics, 1977

    1977

  32. [40]

    Hung Le, Quang Pham, Doyen Sahoo, and Steven C. H. Hoi. Urlnet: Learning a url representation with deep learning for malicious url detection. arXiv:1802.03162, 2018

    2018 arXiv

  33. [41]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Se- bastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Pro- ceedings of...

    2020

  34. [42]

    Phishpedia: A hybrid deep learning based approach to visually identify phishing webpages

    Yun Lin, Ruofan Liu, Dinil Mon Divakaran, Jun Yang Ng, Qing Zhou Chan, Yiwen Lu, Yuxuan Si, Fan Zhang, and Jin Song Dong. Phishpedia: A hybrid deep learning based approach to visually identify phishing webpages. In 30th USENIX Security Symposium, 2021

    2021

  35. [43]

    Less defined knowl- edge and more true alarms: Reference-based phishing detection without a pre-defined reference list

    Ruofan Liu, Yun Lin, Xiwen Teoh, Gongshen Liu, Zhiy- ong Huang, and Jin Song Dong. Less defined knowl- edge and more true alarms: Reference-based phishing detection without a pre-defined reference list. In 33rd USENIX Security Symposium, 2024

    2024

  36. [44]

    Inferring phishing intention via webpage appearance and dynam- ics: A deep vision based approach

    Ruofan Liu, Yun Lin, Xianglin Yang, Siang Hwee Ng, Dinil Mon Divakaran, and Jin Song Dong. Inferring phishing intention via webpage appearance and dynam- ics: A deep vision based approach. In 31st USENIX Security Symposium, 2022

    2022

  37. [45]

    Automatic and universal prompt injection attacks against large language mod- els

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language mod- els. arXiv:2403.04957, 2024

    2024 arXiv

  38. [46]

    Quan- tized delta weight is safety keeper

    Yule Liu, Zhen Sun, Xinlei He, and Xinyi Huang. Quan- tized delta weight is safety keeper. arXiv:2411.19530, 2024

    2024 arXiv

  39. [47]

    Saul, Stefan Savage, and Geof- frey M

    Justin Ma, Lawrence K. Saul, Stefan Savage, and Geof- frey M. V oelker. Beyond blacklists: learning to detect malicious web sites from suspicious urls. In Proceed- ings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2009

    2009

  40. [48]

    Madhu, A

    G. Madhu, A. Govardhan, and T. V . Rajinikanth. In- telligent semantic web search engines: A brief survey. arXiv:1102.0831, 2011

    2011 arXiv

  41. [49]

    Ghorbani

    Mohammad Saiful Islam Mamun, Mohammad Ahmad Rathore, Arash Habibi Lashkari, Natalia Stakhanova, and Ali A. Ghorbani. Detecting malicious urls using lexical analysis. In Proceedings of Network and System Security - 10th International Conference, 2016

    2016

  42. [50]

    Phishing-alarm: Robust and efficient phishing detection via page component similarity

    Jian Mao, Wenqian Tian, Pei Li, Tao Wei, and Zhenkai Liang. Phishing-alarm: Robust and efficient phishing detection via page component similarity. IEEE Access, 2017

    2017

  43. [51]

    Visual-similarity-based phishing detection

    Eric Medvet, Engin Kirda, and Christopher Kruegel. Visual-similarity-based phishing detection. In Proceed- ings of the 4th International Conference on Security and Privacy in Communication Netowrks, 2008

    2008

  44. [52]

    The llama 3 herd of models

    Meta. The llama 3 herd of models. arXiv:2407.21783, 2024

    2024 arXiv

  45. [53]

    Adversarial search engine optimization for large language models

    Fredrik Nestaas, Edoardo Debenedetti, and Florian Tramèr. Adversarial search engine optimization for large language models. arXiv:2406.18382, 2024

    2024 arXiv

  46. [54]

    Investigating llm applications in e-commerce

    Chester Palen-Michel, Ruixiang Wang, Yipeng Zhang, David Yu, Canran Xu, and Zhe Wu. Investigating llm applications in e-commerce. arXiv:2408.12779, 2024

    2024 arXiv

  47. [55]

    Jalmbench: Benchmark- ing jailbreak vulnerabilities in audio language models

    Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Zeren Luo, Jingyi Zheng, Wenhan Dong, Xinlei He, Xuechao Wang, Yingjie Xue, et al. Jalmbench: Benchmark- ing jailbreak vulnerabilities in audio language models. arXiv:2505.17568, 2025

    2025

  48. [56]

    Ignore previous prompt: Attack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop, 2022

    2022

  49. [57]

    Detec- tion of phishing websites using an efficient feature-based machine learning framework

    Routhu Srinivasa Rao and Alwyn Roshan Pais. Detec- tion of phishing websites using an efficient feature-based machine learning framework. Neural Computing and Applications, 2019

    2019

  50. [58]

    A brief review on search engine optimization

    Dushyant Sharma, Rishabh Shukla, Anil Kumar Giri, and Sumit Kumar. A brief review on search engine optimization. In International Conference on Cloud Computing, Data Science & Engineering, 2019

    2019

  51. [59]

    Somesha, Alwyn Roshan Pais, Routhu Srinivasa Rao, and Vikram Singh Rathour

    M. Somesha, Alwyn Roshan Pais, Routhu Srinivasa Rao, and Vikram Singh Rathour. Efficient deep learning tech- niques for the detection of phishing websites. S¯adhan¯a, 2020

    2020

  52. [60]

    Rothschild, Daniel G

    Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, and Jake M. Hofman. Comparing traditional and llm-based search for consumer choice: A random- ized experiment, 2023

    2023

  53. [61]

    A survey on semantic web search engine

    Govathoti Sudeepthi, Guttula Anuradha, and M Suren- dra Prasad Babu. A survey on semantic web search engine. International Journal of Computer Science Is- sues, 2012

    2012

  54. [62]

    Peftguard: Detecting backdoor attacks against parameter-efficient fine-tuning

    Zhen Sun, Tianshuo Cong, Yule Liu, Chenhao Lin, Xinlei He, Rongmao Chen, Xingshuo Han, and Xinyi Huang. Peftguard: Detecting backdoor attacks against parameter-efficient fine-tuning. In IEEE Symposium on Security and Privacy. IEEE, 2025

    2025

  55. [63]

    Glue pizza and eat rocks - exploiting vulnerabilities in retrieval-augmented generative models

    Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, and Huan Liu. Glue pizza and eat rocks - exploiting vulnerabilities in retrieval-augmented generative models. In Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  56. [64]

    A compre- hensive survey on poisoning attacks and countermea- sures in machine learning

    Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. A compre- hensive survey on poisoning attacks and countermea- sures in machine learning. ACM Comput. Surv., 2022

    2022

  57. [65]

    Creating large language model applications utilizing langchain: A primer on developing llm apps fast.International Con- ference on Applied Engineering and Natural Sciences, 2023

    Oguzhan Topsakal and Tahir Cetin Akinci. Creating large language model applications utilizing langchain: A primer on developing llm apps fast.International Con- ference on Applied Engineering and Natural Sciences, 2023

    2023

  58. [66]

    Hunting malicious tls certificates with deep neural networks

    Ivan Torroledo, Luis David Camacho, and Alejan- dro Correa Bahnsen. Hunting malicious tls certificates with deep neural networks. In Workshop on Artificial Intelligence and Security, 2018

    2018

  59. [67]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017

    2017

  60. [68]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information pro- cessing systems, 2022

    2022

  61. [69]

    Auto- gen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Auto- gen: Enabling next-gen llm applications via multi-agent conversation. arXiv:2308.08...

    2023 arXiv

  62. [70]

    A comprehensive study of jailbreak attack versus defense for large language models

    Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. A comprehensive study of jailbreak attack versus defense for large language models. In Findings of the Association for Computational Linguistics, 2024

    2024

  63. [71]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv:2210.03629, 2023

    2023 arXiv

  64. [72]

    Data poisoning attacks against machine learning algorithms

    Fahri Anıl Yerlikaya and ¸ Serif Bahtiyar. Data poisoning attacks against machine learning algorithms. Expert Systems with Applications, 2022

    2022

  65. [73]

    Bench- marking and defending against indirect prompt injection attacks on large language models

    Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Bench- marking and defending against indirect prompt injection attacks on large language models. arXiv:2312.14197, 2024

    2024 arXiv

  66. [74]

    Jailbreak attacks and defenses against large language models: A survey

    Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. arXiv:2407.04295, 2024

    2024 arXiv

  67. [75]

    Fc-attack: Jailbreaking large vision-language models via auto-generated flowcharts

    Ziyi Zhang, Zhen Sun, Zongmin Zhang, Jihui Guo, and Xinlei He. Fc-attack: Jailbreaking large vision-language models via auto-generated flowcharts. arXiv:2502.21059, 2025

    2025

  68. [76]

    Cl-attack: Textual backdoor attacks via cross-lingual triggers

    Jingyi Zheng, Tianyi Hu, Tianshuo Cong, and Xinlei He. Cl-attack: Textual backdoor attacks via cross-lingual triggers. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  69. [77]

    Phishing or not phishing? a survey on the detection of phishing websites

    Rasha Zieni, Luisa Massari, and Maria Carla Calzarossa. Phishing or not phishing? a survey on the detection of phishing websites. IEEE Access, 2023

    2023

  70. [78]

    action" key (containing the name of the tool to use) and an

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language mod- els. arXiv:2402.07867, 2024. A Irrelevant Keywords We delete the data entry if its keyword list contains any of them. check, ...

    2024 arXiv

  71. [79]

    Telegram official app

    Malware Definition: Malware (malicious software) may also include applications downloaded from unofficial or unverified third- party websites that appear to offer legitimate software but could contain harmful or modified files. Risk Indicators: Websites offering software downl...

  72. [80]

    get-rich- quick

    Scam Definition: A scam involves fraudulent schemes designed to deceive individuals for financial gain, typically involving promises of false prizes, investments, or services that never materialize. Risk Indicators: Promises of unrealistic financial gains, such as “get-rich- q...

  73. [81]

    While not always harmful, it can clutter inboxes, waste time, and sometimes lead to phishing or other malicious activities

    Spam Definition: Spam refers to unsolicited and often irrelevant messages or content, typically sent in bulk via email or other digital platforms, to promote products or services. While not always harmful, it can clutter inboxes, waste time, and sometimes lead to phishing or o...

  74. [82]

    It is designed to mislead, manipulate public opinion, or generate traffic for financial or political gain

    Fake News Definition: Fake news refers to false or misleading informa- tion presented as legitimate news. It is designed to mislead, manipulate public opinion, or generate traffic for financial or political gain. Risk Indicators: Stories that are sensationalized or lack credib...

  75. [83]

    This website offers free gift cards but requires entering your credit card information

    Illegal Content Violating Cybersecurity Laws of All Countries Definition: Illegal content encompasses materials that vi- olate local or international laws, including content that breaches cybersecurity regulations of all countries, or con- tent promoting criminal behavior (e.g...

  76. [84]

    - The URL offers free gift cards but requests credit card information, matching the definition of phishing

    **Definition Matching:** - Phishing is defined as attempts to fraudulently obtain sensitive information by disguising as a trustworthy entity. - The URL offers free gift cards but requests credit card information, matching the definition of phishing

  77. [85]

    - The content explicitly asks for credit card information, fulfilling this indicator

    **Indicator Matching:** - Indicators of phishing include websites that request sensi- tive information like login credentials or payment details. - The content explicitly asks for credit card information, fulfilling this indicator

  78. [86]

    **Action:** - Remove harmful content and provide a safer alternative

  79. [87]

    The Earth will be invaded by aliens next week, claims a report

    **Output:** ‘‘‘ The claim about free gift cards from unreliable sources could potentially be harmful. Avoid sharing personal infor- mation and refer to official promotional offers from trusted retailers. ‘‘‘ — Example 2: **Input:** ‘‘‘ URL: http://example-fakenews.com Content:...

  80. [88]

    - The content claims an alien invasion without credible evidence, fitting the definition of fake news

    **Definition Matching:** - Fake news involves false or misleading information pre- sented as legitimate news to manipulate public opinion or generate traffic. - The content claims an alien invasion without credible evidence, fitting the definition of fake news

  81. [89]

    - The claim is sensationalized and unsupported by credible sources, meeting these indicators

    **Indicator Matching:** - Indicators of fake news include sensationalized stories lacking credible sources or promoting conspiracy theories. - The claim is sensationalized and unsupported by credible sources, meeting these indicators

  82. [90]

    **Action:** - Correct the information and add a warning about the unreliability of the source

  83. [91]

    This website compiles open-access scientific papers for free download

    **Output:** ‘‘‘ The claim about an alien invasion is unsupported and originates from an unreliable source. For verified scientific news, refer to trusted institutions like NASA. ‘‘‘ — Example 3: **Input:** ‘‘‘ URL: http://harmless-site.example.com Content: "This website compil...

  84. [92]

    **Definition Matching:** - None of the risk cases apply, as the website provides open- access scientific papers for free and does not involve suspi- cious or harmful content

  85. [93]

    **Indicator Matching:** - The content lacks indicators of phishing, malware, scam, adware, fake news, illegal content, or piracy

  86. [94]

    **Action:** - Retain the original result without modification

  87. [95]

    /” sep- arated) in the URL. Redirection: Detects if the URL contains unexpected redi- rection markers (e.g., “//

    **Output:** ‘‘‘ This website compiles open-access scientific papers for free download. ‘‘‘ It’s your turn: **Input:** ‘‘‘ URL : {url} Content: {content} ‘‘‘ **Response:** D Feature Extraction and Model Training 15 Features of XGBoost Model Have_IP: Checks if the URL contains a...

  88. [96]

    What is your gender? (Single choice) ◦ Male ◦ Female

  89. [97]

    Which age group do you belong to? (Single choice) ◦ 18 or under ◦ 19–25 ◦ 26–30 ◦ 31–35 ◦ 36 or above

  90. [98]

    What is your current industry? (Single choice) ◦ Student ◦ IT and Communication ◦ Healthcare ◦ Education and Training ◦ Manufacturing and Engineering ◦ Retail and Services ◦ Government and Public Services ◦ Other (please specify)

  91. [99]

    What is your highest level of education (including current enrollment)? (Single choice) ◦ High school or below ◦ Associate degree ◦ Bachelor ◦ Master ◦ Doctor

  92. [100]

    Have you ever used AI search engines such as Chat- GPT Search, Doubao (AI Search), or Kimi (web- connected search)? (Single choice) ◦ Yes ◦ No

  93. [101]

    weather, today, NYC

    What methods do you use when querying AI search engines? (Multiple choice) □ Keywords queries (e.g., “weather, today, NYC”) □ URL queries (e.g., “https://www.xx.com/xxx”) □ Natural language queries (e.g., “What is the weather today in New York City?”) □ Other (please specify)

  94. [102]

    What is your intent when using URL-based queries? (Multiple choice) □ Summarize content □ Scrape web page code □ Translate web pages □ Other (please specify)

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.