Pith. sign in

REVIEW 4 major objections 4 minor 7 cited by

Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fully automated AI phishing emails matched human expert phishers in a 101-person trial, with both at 54% click-through.

desk verdict A genuinely useful empirical data point on fully automated LLM phishing, but the 'on par with human experts' claim is oversold because the human benchmark was deliberately weaker. read the letter →

arxiv 2412.00586 v1 pith:S3457VWZ submitted 2024-11-30 cs.CR

Fred Heiding , Simon Lermen , Andrew Kao , Bruce Schneier , Arun Vishwanath This is my paper · ORCID
classification cs.CR
keywords largelanguagemodelsspearphishingautomatedOSINTreconnaissancedetectionhumansubjectsexperimentcybersecurityeconomicssocialengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models can run a spear-phishing campaign end to end—reconnaissance, personalized email writing, sending, and tracking—without human help. In a 101-person experiment, fully automated AI emails achieved a 54% click-through rate, identical to emails written by human experts and far above the 12% control group. The authors argue this is a sharp jump from similar studies a year earlier, when AI needed human-in-the-loop help to match experts. They also report that AI-gathered background profiles were accurate and useful for 88% of targets and that a suspicion-primed language model detected phishing emails with over 97% accuracy and no false positives. If these numbers hold, the marginal cost of a convincing personalized phish has fallen to near zero, which changes who can mount such campaigns and at what scale.

What carries the argument

The carrying mechanism is a custom AI-agent tool that closes the whole phishing loop. It starts with OSINT reconnaissance: an agent built around GPT-4o searches a target's name, employer, publications, and social media through a search API and text-based browser, and compiles the findings into a vulnerability profile. A prompt-engineering database, currently written by human experts and exceeding 2,000 characters, encodes Cialdini's six influence principles and the V-Triad credibility-and-relevance framework, turning each profile into a personalized email whose sender, deadlines, and references are tailored per target. The tool then sends the emails, embeds a unique tracking URL per recipient, logs clicks, and feeds outcomes back for self-improvement. The key operating distinction is hyper-personalization per target rather than semi-personalization, and the remaining human-in-the-loop cost is small: editing averaged 2:41 minutes per message when needed, with 75% of edits being minor word changes.

What would settle it

Re-run the experiment with human experts given the same per-target OSINT profiles and enough time to write one personalized email per target; if their click-through rate exceeds the AI-automated rate by more than sampling error, the on-par claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that frontier language models have crossed a threshold: a fully automated pipeline can now perform the entire spear-phishing loop at human-expert effectiveness. The load-bearing numbers come from 101 consenting participants split into four groups: control emails clicked at 12%, human-expert emails at 54%, fully AI-automated emails at 54%, and AI emails with minor human editing at 56%. The AI system scraped each target's public digital footprint into a vulnerability profile, used that profile to compose a personalized email with Claude 3.5 Sonnet, and sent it without human intervention; the authors score 88% of the profiles as accurate and useful and only 4% as based on the wrong person. A companion detection experiment finds that Claude 3.5 Sonnet, when explicitly asked whether an email is suspicious, caught 97.25% of 363 phishing emails with zero false positives, outperforming humans on several seemingly benign emails. The paper concludes that AI-automated phishing is now on par with expert human phishing, that guardrail circumvention is trivial, and that the economics favor AI at scale.

Load-bearing premise

The claim that AI matches human experts assumes the human-expert arm was a fair test of expert skill, but the experts sent a single semi-personalized email to all 24 targets while the AI wrote a hyper-personalized email for each individual from an OSINT profile.

Editorial extensions

If this is right

  • Signature-based spam filters lose much of their value because every AI email is unique per target; detection must shift to intent and sender-behavior analysis.
  • Human-in-the-loop editing is becoming redundant: most edits were minor, and removing them barely changed the click-through rate (54% versus 56%).
  • The economics of phishing invert: with roughly four cents of API cost per email and a 54% click rate, AI-automated phishing is modeled as profitable for audiences above a few thousand, up to 50 times more profitable than manual phishing.
  • Suspicion priming is a cheap defensive lever: telling a model to look for suspicious intent rather than asking for the email's intention improved detection without raising false positives.
  • Because guardrails were bypassed by simple rephrasing, such as changing 'phishing email' to 'email', model providers cannot rely on refusal training alone to stop this use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 54% figure transfers beyond a university-recruited sample, the binding constraint on phishing volume is no longer attacker skill but target acquisition and email deliverability; defenses should concentrate on those choke points.
  • The parity result compares hyper-personalized AI emails against a single semi-personalized human email sent to the whole group; giving human experts the same per-target OSINT profiles would test whether the parity claim is about the AI or about the benchmark.
  • The OSINT profiles that enable attack could be turned into a personal defensive audit: users could generate their own vulnerability profile, see which public data points a phisher would exploit, and remove data that helps attackers more than it helps them.
  • A suspicion-primed model with zero false positives suggests an inbox-integrated 'second opinion' feature for suspicious messages, although its usefulness would depend on how often users consult it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a four-arm phishing field experiment with 101 human participants, comparing a control group of generic scam emails, emails written by human phishing experts, fully AI-automated spear phishing emails generated by a custom OSINT-and-email tool, and AI emails with human-in-the-loop intervention. The reported click-through rates are 12%, 54%, 54%, and 56%, respectively. The paper interprets the equal 54% rates for human experts and fully automated AI as evidence that LLMs now perform 'on par with human experts' in spear phishing. It also reports an 88% self-assessed accuracy rate for the AI OSINT profiles, evaluates five LLMs as phishing detectors (with Claude 3.5 Sonnet achieving 97.25% true-positive detection on a 381-email dataset at zero false positives), and presents a stylized economic model estimating that AI-based phishing is more profitable than manual phishing by up to roughly 50 times for large campaigns.

Significance. If the central claims hold, this is a timely and policy-relevant result: it would be one of the first demonstrations that a fully automated LLM pipeline, from OSINT reconnaissance to personalized email generation, can elicit real clicks from human targets at a rate comparable to human experts. The study's strengths include a genuine human-subjects experiment with IRB approval, transparent click-through measurement via per-target tracking links, a comparison with the authors' prior-year results, and an economic model with explicit sensitivity analysis over conversion rates and wage levels. The detection component is also a useful contribution. However, the headline parity claim rests on an asymmetric comparison between hyper-personalized AI emails and a single semi-personalized human-expert email, and the statistical evidence for 'on par' is weaker than the prose suggests. These issues are fixable but require additional analysis or an additional experimental condition.

major comments (4)
  1. [§3.5.2, §3.5.3, §5.1] The central claim that fully automated AI phishing is 'on par with human experts' is confounded by personalization. The human-expert condition used one semi-personalized email (a cross-disciplinary research invitation) sent identically to all 24 participants, whereas the AI condition generated a hyper-personalized email for each target from individual OSINT profiles. The paper itself labels these Category 2 and Category 3 personalization, respectively. Because the experiment never gives human experts the same per-target OSINT information and time budget, the equal 54% click-through rates cannot distinguish 'AI matches human skill' from 'hyper-personalization outperforms semi-personalization.' The abstract and conclusion should either be reframed to state the comparison as AI-hyper-personalized versus human-semi-personalized, or the authors should add a condition in which human experts receive the same per-target profiles and comparable time.
  2. [§5.1, Table 4] No statistical test supports the 'on par' wording. The four click-through rates are point estimates from groups of 24–26 participants; Table 4 reports standard errors around 12 percentage points for the human-expert and AI groups. The 54% versus 54% equality is exactly equal, but the confidence intervals are wide enough to include substantially different underlying rates. The power calculation in §3.2 is designed to detect a large effect, not to establish equivalence. The authors should report confidence intervals for each click-through rate and, if they wish to claim parity, conduct an equivalence or non-inferiority test (e.g., TOST with a pre-specified margin).
  3. [§5.1, Table 3; Abstract] The headline '88% accurate OSINT' figure is self-assessed by the authors using the subjective categories in Table 2 ('correct and sufficient information'), with no independent raters, no blinding, and no inter-rater reliability measure. Since the claim that the tool performs 'the entire spear phishing process' depends on OSINT quality, this number should be presented as an internal evaluation, not as a validated accuracy measurement. Independent coding of the collected profiles against the participants' actual public information, or at least a second annotator, would strengthen the claim.
  4. [§6.2, Table 4, Figure 9] The economic result that AI phishing is up to roughly 50 times more profitable than human phishing depends heavily on the calibrated 'Time spent' value of 1 minute for the fully automated AI group. Unlike the hybrid group's 4:24, which is recorded from actual intervention times (Section 5.1.1), the 1-minute figure appears to be an assumption rather than a measurement; the fully automated pipeline still requires some oversight, and Section 5.1.1 reports that checking each email took about one minute even in the hybrid condition. The profitability ratios and the break-even group sizes in Figure 9 should be accompanied by a sensitivity analysis over plausible per-email time (e.g., 0–5 minutes), since this parameter directly drives the 'up to 50 times' claim.
minor comments (4)
  1. [§3.5.1, Appendix A.6] The control-group email was iteratively edited 'to be less suspicious until it was accepted by all tested email clients.' This makes the control a weak baseline for 'ordinary phishing emails,' since it was modified specifically to avoid spam-filter rejection; the authors should acknowledge this as a limitation or describe how representative the final control email is.
  2. [§4.2, §5.2] The larger detection dataset includes AI-generated emails produced by the same tool and prompt template that generated the study emails. High detection accuracy on these emails may partly reflect recognition of the tool's own stylistic template rather than general phishing-detection ability. A temporal or source-based holdout would make the detection results more convincing.
  3. [§3.6] The full prompt template is withheld 'due to security considerations.' For reproducibility, the authors should provide a redacted or summarized version that preserves the core instruction structure without enabling direct misuse.
  4. [Throughout] There are several typographical and grammatical errors, including 'spar phishing' in Section 8, 'the this' in Section 6, 'Youíll' in Figure 3, and inconsistent hyphenation of 'human-in-the-loop' in places. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline click-through results are direct measurements, and the economic and projection claims are explicitly derived from external calibrations or labeled extrapolations.

full rationale

The paper's central results are empirical measurements rather than derived predictions: each group's click-through rate is an observed outcome from the 101-participant experiment (Section 5.1, Figure 5), and the human-expert comparison is a measured baseline, not a quantity defined in terms of the AI result. The economic analysis in Sections 6.1 and 6.2 plugs the measured success probabilities and externally calibrated parameters (payoff of $136, conversion rate q, wage rates, and per-email costs) into the stated profit formula, so the profitability conclusions follow by arithmetic from stated inputs rather than by construction. The 2025 performance projection in Appendix A.7 is explicitly described as a simple linear extrapolation, not a prediction derived from a model. The only mild concern is that the AI tool's OSINT accuracy is scored by the authors themselves, which is a self-assessment or measurement-bias issue rather than circular reasoning. The one self-citation to the authors' prior phishing study [50] is used only as a comparison baseline for the 'improvement over last year' claim and does not determine the current click-through measurements. No fitted parameter is renamed as a prediction, and no load-bearing claim reduces to its own input, so there is no significant circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The click-through experiment itself requires few free parameters, but the headline economic claim depends on several calibrated values (payoff, conversion, time, sunk costs). The main axioms are domain assumptions about applying marketing and industry statistics to phishing and about the fairness of the human-expert benchmark. The paper makes no claims of new physical entities; the custom tool is a software artifact, not a postulated entity.

free parameters (6)
  • Payoff per successful phish (m) = $136
    Calibrated from an industry estimate (Section 6.2, footnote 8). Used as the revenue per click in all profit/hour computations.
  • Conversion rate from click to revenue (q) = 0.6%, 2.35%, 7.9% (low, medium, high)
    Taken from marketing conversion statistics (Section 6.2). The profitability results, including the 'up to 50 times' claim, scale directly with q.
  • Time spent per email: fully automated AI = 1 minute
    Assumed in Table 4 and Section 6.2 for the pure AI arm. This excludes prompt engineering and tool development, which are treated as sunk costs.
  • Time spent per email: human expert = 30 minutes
    Measured from the authors' manual replication of OSINT and email writing (Section 5.1.1).
  • Sunk development cost for AI phishing tool = $16,120 (260 hours at $62/hour)
    Estimated from the authors' own development time (Section 6.2). Used to compute break-even group sizes in Figure 9.
  • Fixed costs per email (domain, AI API) = $0.05 per email
    Section 6.2: $0.01 for domain costs to avoid spam filters, $0.04 for AI API compute.
assumptions (6)
  • domain assumption Marketing conversion rates apply to phishing clicks.
    Section 6.2 uses q=2.35% (median), 0.6%, and 7.9% as conversion probabilities for a clicked phishing link, with no phishing-specific data. The 'up to 50 times' profitability claim depends on this.
  • domain assumption The $136 payoff per successful phish is unbiased.
    Section 6.2, footnote 8, states the industry estimate is assumed unbiased and orthogonal to success probability.
  • domain assumption The self-selected university sample represents phishing targets.
    Participants were recruited via flyers and university email lists (Section 3.3). The click-through rates are generalized without adjustment to broader populations.
  • domain assumption Human experts' semi-personalized email is a fair benchmark for human expertise.
    Section 3.5.2 restricts experts to one email for all 24 participants, while AI gets per-target OSINT profiles. This asymmetry is load-bearing for the 'on par with human experts' interpretation.
  • domain assumption LLM safety guardrails can be bypassed by simple rephrasing.
    Section 3.6 asserts that changing 'phishing email' to 'email' circumvents refusal. This is an empirical claim from the authors' own tests, not a proven property.
  • domain assumption The authors' retrospective scoring of AI-generated emails and OSINT is accurate.
    Table 3 scores were assigned by the authors after the fact; there is no second rater or independent verification of OSINT accuracy (88% level-3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects." pith.science (2026). https://pith.science/paper/S3457VWZ

@misc{pith2026241200586,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3457VWZ}},
  note         = {Machine review of arXiv:2412.00586}
}
read the original abstract

In this paper, we evaluate the capability of large language models to conduct personalized phishing attacks and compare their performance with human experts and AI models from last year. We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through). Thus, the AI-automated attacks performed on par with human experts and 350% better than the control group. The results are a significant improvement from similar studies conducted last year, highlighting the increased deceptive capabilities of AI models. Our AI-automated emails were sent using a custom-built tool that automates the entire spear phishing process, including information gathering and creating personalized vulnerability profiles for each target. The AI-gathered information was accurate and useful in 88% of cases and only produced inaccurate profiles for 4% of the participants. We also use language models to detect the intention of emails. Claude 3.5 Sonnet scored well above 90% with low false-positive rates and detected several seemingly benign emails that passed human detection. Lastly, we analyze the economics of phishing, highlighting how AI enables attackers to target more individuals at lower cost and increase profitability by up to 50 times for larger audiences.

Figures

Figures reproduced from arXiv: 2412.00586 by the authors.

Figure 1
Figure 1. Overview of AI-automated phishing campaigns. The process includes target identification, synthetic attacker profile creation, personalized email [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example of an abbreviated profile written about one of the authors [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Email message generated by Claude 3.5 Sonnet based on an [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: The control group emails received a click-through [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Top: Common reasons given in free text survey responses for why the email was trustworthy as a percentage of survey responses per group. Bottom: Common reasons given for why the email was suspicious. As noted in Section 3, half of the AI-generated emails used a human-i…
Figure 7
Figure 7. Figure 7: Success rate of the intent detection for each email category, [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Overview of suspicion scores evaluated by the Claude 3.5 Sonnet [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Estimated profitability of phishing groups of various sizes, using [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Overview of AI likelihood scores as evaluated by Claude 3.5 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: The values in this figure show the quality and relevance of emails [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Overview of AI performance growth and a linear projection of [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Control group email message used in the study. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Detection: Evaluating Defensive LLMs Against AI-Generated Social Engineering in Live Turn-by-Turn Interaction

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Defensive LLMs frequently intervene without identifying the right compromised trust component, and sometimes correctly diagnose a failure while still recommending no protective action.

  2. LLMs unlock new paths to monetizing exploits

    cs.CR 2025-05 conditional novelty 7.0 of 10

    LLMs can already mine personal data for blackmail, find real vulnerabilities in obscure software, and automate in-browser and server-side attacks, changing the economics of cybercrime.

  3. Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A large public red-teaming competition with 1.8 million attacks shows that nearly all 22 frontier LLM-based agents can be induced to violate their deployment policies within 10-100 queries, and that these attacks tran...

  4. Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A framework paper that adapts AI safety case methodology to the specific threat of manipulation attacks by internally deployed misaligned AI.

  5. The Impact of Emerging Phishing Threats: Assessing Quishing and LLM-generated Phishing Emails against Organizations

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Quishing emails are as effective as traditional link-based phishing in driving employees to fake login pages, and OSINT-fed LLM emails lured up to 31.5 percent of openers in one company.

  6. Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks

    cs.CR 2025-02 conditional novelty 6.0 of 10

    An autonomous LLM-driven agent can compromise accounts in a realistic Active Directory testbed, with reasoning models outperforming non-reasoning ones at competitive cost.

  7. LLM Cyber Evaluations Don't Capture Real-World Risk

    cs.CR 2025-01 conditional novelty 4.0 of 10

    The paper argues and demonstrates with a 100-prompt case study that LLM cyber risk evaluations need to include threat actor adoption and impact, not just model capability.

Reference graph

Works this paper leans on

61 extracted references · 41 canonical work pages · cited by 7 Pith papers

  1. [1]

    Why phishing works,

    R. Dhamija, J. D. Tygar, and M. Hearst, “Why phishing works,” Conference on Human Factors in Computing Systems - Proceedings , vol. 1, pp. 581–590, 2006. [Online]. Available: www.paypa1.com

    work page 2006

  2. [2]

    IBM finds that ChatGPT can generate phishing emails nearly as convincing as a human | VentureBeat

    “IBM finds that ChatGPT can generate phishing emails nearly as convincing as a human | VentureBeat.” [Online]. Available: https://venturebeat.com/ai/ibm-x-force-pits-chatgpt-against- humans-whos-better-at-phishing/

    work page

  3. [3]

    From Chatbots to Phishbots?: Phishing Scam Generation in Commercial Large Language Models,

    S. S. Roy, P. Thota, K. V . Naragam, and S. Nilizadeh, “From Chatbots to Phishbots?: Phishing Scam Generation in Commercial Large Language Models,” 2024 IEEE Symposium on Security and Privacy (SP), pp. 36–54, 5 2024

    work page 2024

  4. [4]

    Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT,

    N. Begou, J. Vinoy, A. Duda, and M. Korczynski, “Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT,” 2023 IEEE Conference on Communications and Network Security, CNS 2023 , 2023

    work page 2023

  5. [5]

    Digital deception: generative artificial intelligence in social engineering and phishing,

    M. Schmitt and I. Flechais, “Digital deception: generative artificial intelligence in social engineering and phishing,” Artificial Intelligence Review 2024 57:12 , vol. 57, no. 12, pp. 1–23, 10 2024. [Online]. Available: https://link.springer.com/article/10.1007/s10462-024-10973- 2

    work page doi:10.1007/s10462-024-10973- 2024

  6. [6]

    Vishwanath, The Weakest Link: How to Diagnose, Detect, and Defend Users from Phishing

    A. Vishwanath, The Weakest Link: How to Diagnose, Detect, and Defend Users from Phishing . MIT Press, 2022

    work page 2022

  7. [7]

    Hadnagy, Social Engineering: The Science of Human Hacking

    C. Hadnagy, Social Engineering: The Science of Human Hacking . John Wiley & Sons, 2018

    work page 2018

  8. [8]

    Raschka, Build a Large Language Model (From Scratch) , 2024

    S. Raschka, Build a Large Language Model (From Scratch) , 2024. [Online]. Available: https://www.google.com/books?hl=en&lr=&id=uSUmEQAAQBAJ& oi=fnd&pg=PA1&dq=Build+a+Large+Language+Model+(From+ Scratch)&ots=5B9d6TvtXm&sig=y9Vp3q_AIeV4Gizfx2nJh2s9eos

    work page 2024

Show all 61 references
  1. [9]

    Alammar and M

    J. Alammar and M. Grootendorst, Hands-On Large Language Models: Language Understanding and Generation , 2024. [Online]. Available: https://www.google.com/books?hl=en&lr=&id=hE8hEQAAQBAJ& oi=fnd&pg=PT24&dq=Hands-On+Large+Language+Models: +Language+Understanding+and+Generation+&...

    2024

  2. [10]

    The Persuasive Power of Large Language Models,

    S. M. Breum, D. V . Egdal, V . G. Mortensen, A. G. Møller, and L. M. Aiello, “The Persuasive Power of Large Language Models,” Proceedings of the International AAAI Conference on Web and Social Media , vol. 18, pp. 152–163, 5 2024. [Online]. Available: https://ojs.aaai.org/inde...

    2024

  3. [11]

    Working With AI to Persuade: Examining a Large Language Model’s Ability to Generate Pro-Vaccination Messages,

    E. Karinshak, S. X. Liu, J. S. Park, and J. T. Hancock, “Working With AI to Persuade: Examining a Large Language Model’s Ability to Generate Pro-Vaccination Messages,” Proceedings of the ACM on Human-Computer Interaction, vol. 7, no. CSCW1, 4 2023. [Online]. Available: https:/...

    2023 doi

  4. [12]

    Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language,

    A. B. Pauli, I. Augenstein, and I. Assent, “Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language,” 6 2024. [Online]. Available: https://arxiv.org/ abs/2406.17753v2

    2024 arXiv

  5. [13]

    Could what happened to sony happen to us?

    W. Houser, “Could what happened to sony happen to us?” IT Professional, vol. 17, no. 2, pp. 54–57, 2015

    2015

  6. [14]

    A taxonomy of cyber-harms: Defining the impacts of cyber-attacks and understanding how they propagate,

    I. Agrafiotis, J. R. C. Nurse, M. Goldsmith, S. Creese, and D. Upton, “A taxonomy of cyber-harms: Defining the impacts of cyber-attacks and understanding how they propagate,” Journal of Cybersecurity, vol. 4, no. 1, p. tyy006, 10 2018. [Online]. Available: https://doi.org/10.1...

    2018 doi

  7. [15]

    Casino giant MGM expects $100 million hit from hack that led to data breach | Reuters

    “Casino giant MGM expects $100 million hit from hack that led to data breach | Reuters.” [Online]. Avail- able: https://www.reuters.com/business/mgm-expects-cybersecurity- issue-negatively-impact-third-quarter-earnings-2023-10-05/

    2023

  8. [16]

    Cybersecurity threatscape: Q3 2022,

    P. Technologies, “Cybersecurity threatscape: Q3 2022,” 2022

    2022

  9. [17]

    Internet crime complaint center 2019 internet crime report,

    Federal Bureau of Investigation, “Internet crime complaint center 2019 internet crime report,” Federal Bureau of Investigation, Annual Report, 2020. [Online]. Available: https://www.ic3.gov/AnnualReport/ Reports/2019_IC3Report.pdf

    2019

  10. [18]

    Internet crime report 2023,

    Internet Crime Complaint Center (IC3), “Internet crime report 2023,” Federal Bureau of Investigation, Annual Report, 2024. [Online]. Available: https://www.ic3.gov/AnnualReport/Reports/2023_ IC3Report.pdf

    2023

  11. [19]

    “Memorandum on Advancing the United States’ Leadership in Artificial Intelligence; Harnessing Artificial Intelligence to Fulfill National Security Objectives; and Fostering the Safety, Security, and Trustworthiness of Artificial Intelligence | The White House.” [Online]. Avail...

    2024

  12. [20]

    How well does GPT phish people? An investigation involving cognitive biases and feedback,

    M. Sharma, K. Singh, P. Aggarwal, and V . Dutt, “How well does GPT phish people? An investigation involving cognitive biases and feedback,” Proceedings - 8th IEEE European Symposium on Security and Privacy Workshops, Euro S and PW 2023 , pp. 451–457, 2023

    2023

  13. [21]

    Devising and Detecting Phishing Emails Using Large Language Models,

    F. Heiding, B. Schneier, A. Vishwanath, J. Bernstein, and P. S. Park, “Devising and Detecting Phishing Emails Using Large Language Models,” IEEE Access, vol. 12, pp. 42 131–42 146, 2024

    2024

  14. [22]

    Targeted Phishing Campaigns using Large Scale Language Models,

    R. Karanjai, “Targeted Phishing Campaigns using Large Scale Language Models,” 12 2022. [Online]. Available: https://arxiv.org/ abs/2301.00665v1

    2022 arXiv

  15. [23]

    Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense,

    A. Kucharavy, Z. Schillaci, L. Loıc Maréchal, M. Würsch, L. Dolamic, R. Sabonnadiere, D. P. David, A. Mermoud, and V . Lenders, “Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense,” 3 2023. [Online]. Available: https://arxiv.org/abs/2303.12132v1

    2023 arXiv

  16. [24]

    Generating Phishing Attacks using ChatGPT,

    S. S. Roy, K. V . Naragam, and S. Nilizadeh, “Generating Phishing Attacks using ChatGPT,” 5 2023. [Online]. Available: https://arxiv.org/abs/2305.05133v1

    2023 arXiv

  17. [25]

    Generating Personalized Phishing Emails for Social Engineering Training Based on Neural Language Models,

    S. W. Guo, T. C. Chen, H. J. Wang, F. Y . Leu, and Y . C. Fan, “Generating Personalized Phishing Emails for Social Engineering Training Based on Neural Language Models,” Lecture Notes in Networks and Systems , vol. 570 LNNS, pp. 270–281, 2023. [Online]. Available: https://link...

    2023 doi

  18. [26]

    Measuring the persuasiveness of language models,

    E. Durmus, L. Lovitt, A. Tamkin, S. Ritchie, J. Clark, and D. Ganguli, “Measuring the persuasiveness of language models,”

  19. [27]

    Large Language Models Can Be Used To Effectively Scale Spear Phishing Campaigns,

    J. Hazell, “Large Language Models Can Be Used To Effectively Scale Spear Phishing Campaigns,” 5 2023. [Online]. Available: https://arxiv.org/abs/2305.06972v2

    2023 arXiv

  20. [28]

    Llm agents can autonomously exploit one-day vulnerabilities,

    R. Fang, R. Bindu, A. Gupta, and D. Kang, “Llm agents can autonomously exploit one-day vulnerabilities,” 2024. [Online]. Available: https://arxiv.org/abs/2404.08144

    2024 arXiv

  21. [29]

    Llm agents can autonomously hack websites,

    R. Fang, R. Bindu, A. Gupta, Q. Zhan, and D. Kang, “Llm agents can autonomously hack websites,” 2024. [Online]. Available: https://arxiv.org/abs/2402.06664

    2024 arXiv

  22. [30]

    Teams of llm agents can exploit zero-day vulnerabilities,

    R. Fang, R. Bindu, A. Gupta, Q. Zhan, and D. Kang, “Teams of llm agents can exploit zero-day vulnerabilities,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01637

    2024 arXiv

  23. [31]

    Pentestgpt: An llm-empowered automatic penetration testing tool,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “Pentestgpt: An llm-empowered automatic penetration testing tool,” 2024. [Online]. Available: https://arxiv.org/abs/2308.06782

    2024 arXiv

  24. [32]

    Purple llama cyberseceval: A secure coding benchmark for language models,

    M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y . Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V . V ontimitta, S. Whitman, and J. Saxe, “Purple llama cybe...

    2023 arXiv

  25. [33]

    Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models,

    A. K. Zhang, N. Perry, R. Dulepet, J. Ji, J. W. Lin, E. Jones, C. Menders, G. Hussein, S. Liu, D. Jasper, P. Peetathawatchai, A. Glenn, V . Sivashankar, D. Zamoshchin, L. Glikbarg, D. Askaryar, M. Yang, T. Zhang, R. Alluri, N. Tran, R. Sangpisit, P. Yiorkadjis, K. Osele, G. Ra...

    2024 arXiv

  26. [34]

    Detecting Phishing Sites Using ChatGPT,

    T. Koide, N. Fukushi, N. Security, J. Tokyo, J. H. Nakano, and D. Chiba, “Detecting Phishing Sites Using ChatGPT,” 6 2023. [Online]. Available: https://arxiv.org/abs/2306.05816v1

    2023 arXiv

  27. [35]

    LMs go Phishing: Adapting Pre-trained Language Models to Detect Phishing Emails,

    K. Misra and J. T. Rayz, “LMs go Phishing: Adapting Pre-trained Language Models to Detect Phishing Emails,” Proceedings - 2022 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2022 , pp. 135–142, 2022

    2022

  28. [36]

    A Large-Scale Pretrained Deep Model for Phishing URL Detection,

    Y . Wang, W. Zhu, H. Xu, Z. Qin, K. Ren, and W. Ma, “A Large-Scale Pretrained Deep Model for Phishing URL Detection,” pp. 1–5, 5 2023

    2023

  29. [37]

    URLTran: Improving Phishing URL Detection Using Transformers,

    P. Maneriker, J. W. Stokes, E. G. Lazo, D. Carutasu, F. Tajaddodianfar, and A. Gururajan, “URLTran: Improving Phishing URL Detection Using Transformers,” Proceedings - IEEE Military Communications Conference MILCOM, vol. 2021-November, pp. 197–204, 2021

    2021

  30. [38]

    SoK: Pragmatic As- sessment of Machine Learning for Network Intrusion Detection,

    G. Apruzzese, P. Laskov, and J. Schneider, “SoK: Pragmatic As- sessment of Machine Learning for Network Intrusion Detection,” Proceedings - 8th IEEE European Symposium on Security and Privacy, Euro S and P 2023 , pp. 592–614, 2023

    2023

  31. [39]

    R. Liu, Y . Lin, X. Teoh, G. Liu, Z. Huang, and J. S. Dong, Less Defined Knowledge and More True Alarms: Reference-based Phishing Detection without a Pre-defined Reference List . USENIX Association, 2024. [Online]. Available: https://www.usenix.org/ conference/usenixsecurity24...

    2024

  32. [40]

    P. G. Qi, A. Xin, Y . Li, Y . Liu, T. Zhang, and Y . Liu, Knowledge Expansion and Counterfactual Interaction for {Reference- Based} Phishing Detection , 2023. [Online]. Available: https: //sites.google.com/view/

    2023

  33. [41]

    Gpt-4o: Openai’s language model,

    OpenAI, “Gpt-4o: Openai’s language model,” 2024, https://openai. com/index/gpt-4o-fine-tuning

    2024

  34. [42]

    Claude 3.5 sonnet: Anthropic’s language model,

    Anthropic, “Claude 3.5 sonnet: Anthropic’s language model,” 2024, https://www.anthropic.com/index/claude-3.5-sonnet

    2024

  35. [43]

    The llama 3 herd of models,

    A. Dubey and A. J. et al., “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    2024 arXiv

  36. [44]

    On the planning abilities of openai’s o1 models: Feasibility, optimality, and generalizability,

    K. Wang, J. Li, N. P. Bhatt, Y . Xi, Q. Liu, U. Topcu, and Z. Wang, “On the planning abilities of openai’s o1 models: Feasibility, optimality, and generalizability,” 2024. [Online]. Available: https://arxiv.org/abs/2409.19924

    2024 arXiv

  37. [45]

    Refusal-trained llms are easily jailbroken as browser agents,

    P. Kumar, E. Lau, S. Vijayakumar, T. Trinh, S. R. Team, E. Chang, V . Robinson, S. Hendryx, S. Zhou, M. Fredrikson, S. Yue, and Z. Wang, “Refusal-trained llms are easily jailbroken as browser agents,” 2024. [Online]. Available: https://arxiv.org/abs/2410.13886

    2024 arXiv

  38. [46]

    Applying refusal-vector ablation to llama 3.1 70b agents,

    S. Lermen, M. Dziemian, and G. Pimpale, “Applying refusal-vector ablation to llama 3.1 70b agents,” 2024. [Online]. Available: https://arxiv.org/abs/2410.10871

    2024 arXiv

  39. [47]

    Agentharm: A benchmark for measuring harmfulness of llm agents,

    M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y . Gal, and X. Davies, “Agentharm: A benchmark for measuring harmfulness of llm agents,” 2024. [Online]. Available: https://arxiv.org/abs...

    2024 arXiv

  40. [48]

    R. B. Cialdini, Influence: The psychology of persuasion . Collins New York, 2007, vol. 55

    2007

  41. [49]

    A. v. d. Heijden and L. Allodi, Cognitive Triaging of Phishing Attacks, 2019. [Online]. Available: www.usenix.org/conference/ usenixsecurity19/presentation/van-der-heijden

    2019

  42. [50]

    Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models,

    F. Heiding, B. Schneier, A. Vishwanath, J. Bernstein, and P. S. Park, “Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models,” 8 2023. [Online]. Available: https://arxiv.org/abs/2308.12287v2

    2023 arXiv

  43. [51]

    Applicability of machine learning in spam and phishing email filtering: review and approaches,

    T. Gangavarapu, C. Jaidhar, and B. Chanduka, “Applicability of machine learning in spam and phishing email filtering: review and approaches,” Artificial Intelligence Review , vol. 53, pp. 5019–5081, 2020

    2020

  44. [52]

    A comprehensive survey of ai-enabled phishing attacks detection techniques,

    A. Basit, M. Zafar, X. Liu, A. R. Javed, Z. Jalil, and K. Kifayat, “A comprehensive survey of ai-enabled phishing attacks detection techniques,” Telecommunication Systems, vol. 76, pp. 139–154, 2021

    2021

  45. [53]

    Deep learning for phishing detection: Taxonomy, current challenges and future directions,

    N. Q. Do, A. Selamat, O. Krejcar, E. Herrera-Viedma, and H. Fujita, “Deep learning for phishing detection: Taxonomy, current challenges and future directions,” IEEE Access, vol. 10, pp. 36 429–36 463, 2022

    2022

  46. [54]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku,

    Anthropic, “Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku,” Anthropic News, 2024. [Online]. Available: https://www.anthropic.com/news/3-5-models-and-computer-use

    2024

  47. [55]

    Article 5: Prohibited AI Practices | EU Artificial Intelligence Act

    “Article 5: Prohibited AI Practices | EU Artificial Intelligence Act.” [Online]. Available: https://artificialintelligenceact.eu/article/5/

  48. [56]

    The right not to be subjected to ai profiling based on publicly available data—privacy and the exceptionalism of ai profiling,

    T. Ploug, “The right not to be subjected to ai profiling based on publicly available data—privacy and the exceptionalism of ai profiling,” Philosophy & Technology, vol. 36, no. 14, 2023. [Online]. Available: https://doi.org/10.1007/s13347-023-00616-9

    2023 doi

  49. [57]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” 2024. [Online]. Available: https://arxiv.org/abs/2406.11717

    2024 arXiv

  50. [58]

    Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b,

    S. Lermen, C. Rogers-Smith, and J. Ladish, “Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b,” 2024. [Online]. Available: https://arxiv.org/abs/2310.20624 A. Appendix A.1. Prohibited AI practices The EU AI Act outlines eight prohibited AI practices desig...

    2024 arXiv

  51. [60]

    https://trec.nist.gov/pubs/trec16/papers/SPAM.OVERVIEW16.pdf

  52. [61]

    list of available projects

    https://security.berkeley.edu/education-awareness/phishing/phishing- examples-archive legit n=18 AI combinedn=51 clauden=50 o1-previewn=50 gpt-4on=50gpt-3.5n=50 llaman=50phishingn=53 expertn=9 0% 20% 40% 60% 80% 100%Ai_likelihood Level Claude - Ai_likelihood Levels by Mail Cat...

  53. [2024]

    Available: https://www.anthropic.com/news/measuring- model-persuasiveness

    [Online]. Available: https://www.anthropic.com/news/measuring- model-persuasiveness

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.