REVIEW 3 major objections 5 minor 40 references
Cyri: A Conversational AI-based Assistant for Supporting the Human User in Detecting and Responding to Phishing Attacks
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that Cyri, a locally running conversational AI assistant, detects phishing emails by spotting 21 semantic persuasion features and explaining them to users, reaching 95.24% accuracy on a balanced 840-email dataset.
desk verdict Promising system, but the headline 95% accuracy is an in-sample fit: the prompt was tuned on the same 840 emails used for the final test, so the main claim is not supported as stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the combination of a 21-feature semantic taxonomy and a Chain-of-Thought prompt executed by the local LLM. The prompt fixes the model's role, supplies definitions and examples for each feature, assigns weights between 0.1 and 0.9, and requests a structured answer: likelihood category with percentage, plain-language explanation, list of features, per-feature evidence, and countermeasures. It also forces the model to form a base judgment before consulting the features and to interpret link-reputation results from external services. The taxonomy makes the model look for persuasion techniques; the weights and the three safe-email examples are what keep it from over-flagging legitimate mail.
What would settle it
Take a set of real phishing emails collected from reported attacks, have independent experts label which of the 21 features appear, run Cyri's final prompt without modification, and compare its verdicts and feature lists with those labels; if accuracy or per-feature recall falls substantially below the reported figures, the claim that Cyri generalizes to real phishing semantics fails.
Extended reading notes
Core claim
Cyri's central claim is that a carefully prompted 8-billion-parameter LLM, run locally, can classify an email as phishing or safe and, at sentence level, point to the social-engineering cues that justify the verdict. The model is given 21 named features, including urgency, authority, impersonation, exclusivity, and undesirable consequences, each with a description, examples, and a weight reflecting how strongly it indicates phishing. In the final configuration, Cyri correctly classifies 407 safe emails and 393 phishing emails out of 840, and the paper reports that the feature analysis agrees well with expert-curated ground truth on the features most critical to phishing detection. The paper also claims that the same system supports follow-up conversation, visual highlighting of suspicious text, and practical countermeasures, making it usable by people without security expertise.
Load-bearing premise
The evaluation assumes that phishing emails generated by WormGPT, each instructed to display exactly one of the 21 persuasion techniques, represent the phishing emails real users actually receive.
Editorial extensions
If this is right
- Cyri can be installed as an email-client plugin and run completely locally, so an inbox's content never has to leave the machine to receive a semantic phishing assessment.
- Non-expert users get sentence-level highlights of manipulation cues plus countermeasures, which the paper argues should counter the habituation caused by generic warning dialogs.
- The 21-feature taxonomy and the balanced synthetic dataset are offered as a public resource, giving other researchers a common annotation scheme for semantic phishing features.
- The conversational component lets users question a verdict and re-run detection on demand, so the system works as both a detector and a training aid.
Reading between the lines
- Inference: Because each generated phishing email was instructed to embody one target feature, the 95.24% accuracy figure should be read as a measure of the prompt's ability to recognize artificially isolated cues, not as a direct estimate of performance on real attacks that blend several cues.
- Inference: A more decisive evaluation would track user behavior, such as whether they click links or reply to suspicious mail, over weeks of real inbox use rather than relying on self-reported usefulness from a 60-minute session.
- Inference: The manually set feature weights could be treated as a prior and updated from user corrections or labeled real phishing mail, which would likely improve the low-accuracy features the paper itself identifies, such as social proof and false dilemma.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Cyri, a conversational AI assistant that uses a locally hosted LLaMA-3.1-8B model to detect phishing emails by extracting 21 semantic features, and that provides explanations, visual highlighting, and interactive dialogue for end users. The authors report a binary classification accuracy of 95.24% on a balanced 840-email dataset (420 synthetic phishing, 420 synthetic legitimate), feature-wise detection accuracy in Figure 7, and a 10-participant user study claiming that Cyri aided users in identifying phishing and improved their understanding of phishing tactics. The system contribution includes a Thunderbird plugin, an Electron-based interface, and a prompt-engineered local LLM pipeline.
Significance. The system design is well-motivated and addresses real gaps: local processing for privacy, explanation generation for non-experts, and a mixed visual/conversational interface to counter habituation are worthwhile ideas. The authors also release code and data, which is valuable for reproducibility. However, the central quantitative claims rest on an evaluation protocol that is both in-sample and circular, so the headline accuracy, feature-detection rates, and user-study conclusions are not supported by the evidence as presented.
major comments (3)
- [§6.2, Tables 3–6] The reported final accuracy of 95.24% (Table 6) is an in-sample result. The authors describe four iterative rounds of prompt refinement on the same 840-email dataset: Test 3 to Test 4 involved adjusting feature weights and adding safe-email examples specifically to reduce false positives observed on this dataset. No held-out partition, cross-validation, or independent test set is described anywhere in §6. Consequently, the headline claim in the abstract and §1 is not an unbiased estimate of performance on unseen emails; it reflects prompt optimization against the same examples used for measurement. This is load-bearing because the paper's central claim of high detection effectiveness rests on this single figure.
- [§6.1] The evaluation dataset is constructed from the detector's own semantic vocabulary, creating a circular test. Each phishing email is generated by WormGPT with the instruction to embody exactly one <FEATURE> from the same list of 21 features that Cyri is prompted to detect, and the safe emails are also synthetically generated with ChatGPT-4. In addition, the ground-truth feature labels are produced using ChatGPT-4o 'supplemented by manual review and additions' (end of §6.1), again using the same feature taxonomy. Real phishing emails typically mix, hide, or omit these cues, and a corpus in which each message is engineered to contain exactly one well-defined feature from the detector's own prompt cannot measure generalization to naturally occurring attacks. The accuracy in Table 6 and the feature-detection rates in Figure 7 therefore cannot be read as estimates of real-world detection performance.
- [§7, Figures 8–9] The abstract states that Cyri 'significantly aided users in identifying phishing emails and enhanced their understanding of phishing tactics,' but the user study provides no statistical support for this claim. The study has only 10 participants, no control condition, and no inferential test (such as a paired comparison of identification performance or confidence with and without Cyri). The reported results are descriptive averages from Likert-scale questions. In the absence of a controlled comparison or significance testing, the claim of a significant benefit is not supported by the reported data.
minor comments (5)
- [§4.2.1] The phrase 'Chain-of-Tought' appears twice and should be 'Chain-of-Thought.'
- [§3] In the description of the email client plugin, 'since It is an open-source email client' should be 'since it is an open-source email client'.
- [Tables 3–6] Percentage values use inconsistent decimal notation (e.g., '33,3%' in §6.2.2) while the rest of the text uses decimal points; please standardize.
- [§6.1] The reference for WormGPT V3.0 [34] is a FlowGPT chat page rather than a stable academic source; given the central role of this tool in dataset generation, the authors should provide a more citable reference or a detailed description of the model version and configuration.
- [§6.2.4] The statement that features with lower accuracy rates are 'less crucial for phishing detection' is not established by the data; lower detector accuracy could also reflect label ambiguity or annotation noise in the ground-truth procedure, and the paper should not assume a direct correspondence between detection difficulty and importance.
Circularity Check
The reported 95.24% accuracy is an in-sample fit: the prompt was iteratively tuned on the same 840 emails used for the headline metric, and the synthetic test emails are generated from the same feature taxonomy the detector is asked to recognize.
-
fitted input called prediction
[Section 6.2.2 (Validating LISA phishing detection), Tables 3-6 and Abstract]
"In the final evaluation, we implemented a comprehensive and optimized prompt corresponding to the one utilized by the Cyri system. To address the issues identified in the previous test, we adjusted the weights assigned to the features, aligning them more appropriately with their actual importance in phishing detection. We also provided multiple examples (3-shot learning) of safe emails to enhance the model's understanding of legitimate email patterns, having identified FP as the most problematic case."
Tests 1 through 4 in Section 6.2 are all run on the same 840-email dataset: Test 2 adds semantic features, Test 3 adds weighted features and one-shot examples, and Test 4 adjusts weights and adds three safe-email examples specifically to reduce false positives observed on that same set. Table 6 then reports the final metrics, including accuracy 95.24%, computed on the same 840 emails that guided the prompt modifications. The prompt is therefore a fitted parameter, and the reported accuracy is its in-sample training score, not a prediction on unseen emails.
-
self definitional
[Section 6.1 (Validation Datasets) and Section 6.2.3 (Validating LISA phishing semantic features detection)]
"The email must be characterized by the following persuasion technique: <FEATURE > With this methodology, a total of 420 phishing emails have been generated, with 20 emails dedicated to each of the 21 identified Cyri semantic features."
The same feature taxonomy that Cyri's prompt instructs the LLM to detect ('You have to find the following features: {features}') is the list used to tell WormGPT which single persuasion technique to plant in each generated test email. The ground-truth feature labels for the same emails are then produced using ChatGPT-4o plus manual review, again using the same feature set. Feature-detection accuracy in Section 6.2.3 therefore measures re-detection of deliberately implanted labels: the taxonomy is simultaneously the generation recipe, the ground-truth rubric, and the detection vocabulary. This makes the feature-level validation self-definitional rather than evidence that the model discovers these features in unconstrained, naturally occurring phishing emails.
full rationale
The paper's quantitative centerpiece is the 95.24% accuracy, 96.8% precision, and 93.56% recall reported in the abstract and Table 6. Section 6.2 describes four iterative rounds of prompt refinement, each responding to errors observed on the full 840-email validation set: feature weights are adjusted and safe-email examples are added to reduce false positives. The final Test 4 is then reported as Cyri's performance on the same dataset. Because the prompt was optimized against the very emails on which it is measured, the headline metric is an in-sample fit, not an unbiased estimate of detection on unseen emails; this is the strongest circular step and is load-bearing for the paper's central claim. A second, related circularity is that the test phishing emails are generated by instructing WormGPT to embody exactly one of Cyri's 21 semantic features, and the ground-truth feature labels are assigned with the same feature set and with ChatGPT-4o. The feature taxonomy is thus both the stimulus and the measurement instrument, so feature-detection accuracy is re-identification of planted labels rather than discovery in unconstrained data. There are no external benchmarks, held-out partitions, or cross-validation procedures that would break either loop. The user study is small and self-reported, but it is not itself circular; the circularity score is driven by the in-sample tuning and self-derived test corpus. This is not a case of self-citation load-bearing: the cited related work is external, and no uniqueness theorem is imported from the authors' prior work.
Assumptions & free parameters
free parameters (3)
- Semantic feature weights =
Authority 0.6; Impersonation 0.6; Instant Gratification 0.9; Exclusivity 0.8; Undesirable Consequences 0.9; Urgency…
- Likelihood thresholds =
20%, 60%, 90%
- Number of prompt examples =
3 safe, 1 phishing (3-shot)
assumptions (3)
- domain assumption The 21 semantic features are a sufficient and accurate taxonomy of phishing persuasion techniques.
- ad hoc to paper Emails generated by WormGPT with the instruction to embody exactly one listed feature are representative of real-world phishing emails.
- domain assumption Users' self-reported likelihood of continued use and understanding predicts actual secure behavior.
invented entities (1)
-
14 'Extracted' semantic features (e.g., Exclusivity, False Dilemma, Assurance of Legitimacy, Confidentiality Claims)
Cite this review
Pith. "Pith review of Cyri: A Conversational AI-based Assistant for Supporting the Human User in Detecting and Responding to Phishing Attacks." pith.science (2026). https://pith.science/paper/OSSQ4Y2L
@misc{pith2026250205951,
author = {Pith},
title = {Pith review of: Cyri: A Conversational AI-based Assistant for Supporting the Human User in Detecting and Responding to Phishing Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/OSSQ4Y2L}},
note = {Machine review of arXiv:2502.05951}
}
read the original abstract
This work introduces Cyri, an AI-powered conversational assistant designed to support a human user in detecting and analyzing phishing emails by leveraging Large Language Models. Cyri has been designed to scrutinize emails for semantic features used in phishing attacks, such as urgency, and undesirable consequences, using an approach that unifies features already established in the literature with others by Cyri features extraction methodology. Cyri can be directly plugged into a client mail or webmail, ensuring seamless integration with the user's email workflow while maintaining data privacy through local processing. By performing analyses on the user's machine, Cyri eliminates the need to transmit sensitive email data over the internet, reducing associated security risks. The Cyri user interface has been designed to reduce habituation effects and enhance user engagement. It employs dynamic visual cues and context-specific explanations to keep users alert and informed while using emails. Additionally, it allows users to explore identified malicious semantic features both through conversation with the agent and visual exploration, obtaining the advantages of both modalities for expert or non-expert users. It also allows users to keep track of the conversation, supports the user in solving additional questions on both computed features or new parts of the mail, and applies its detection on demand. To evaluate Cyri, we crafted a comprehensive dataset of 420 phishing emails and 420 legitimate emails. Results demonstrate high effectiveness in identifying critical phishing semantic features fundamental to phishing detection. A user study involving 10 participants, both experts and non-experts, evaluated Cyri's effectiveness and usability. Results indicated that Cyri significantly aided users in identifying phishing emails and enhanced their understanding of phishing tactics.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Social engineer- ing: A technique for managing human behavior
Bansla, N., Kunwar, S., and Jain, K. Social engineer- ing: A technique for managing human behavior. (2019). doi:10.5281/zenodo.2580822
-
[3]
Social engineering statistics (2023)
Sprinto.com. Social engineering statistics (2023). Available from: https://sprinto.com/blog/social-engineering-statistics
work page 2023
-
[4]
Ferreira, A. and Teles, S. Persuasion: How phishing emails can influence users and bypass security measures. International Journal of Human-Computer Studies, 125 (2019), 19. Available from: https://www.sciencedirect.com/science/article/pii/S1071581918306827, doi:https://doi.org/10. 1016/j.ijhcs.2018.12.004
work page 2019
-
[5]
The 5 biggest phishing scams of all time
Irwin, L. The 5 biggest phishing scams of all time. https://www. itgovernance.eu/blog/en/the-5-biggest-phishing-scams-of-all-time (2022)
work page 2022
-
[6]
Phishing activity trends reports
APWG. Phishing activity trends reports. https://apwg.org/ trend- sreports/ (2023)
work page 2023
-
[7]
Top phishing statistics for 2024: Latest figures and trends
Smith, G. Top phishing statistics for 2024: Latest figures and trends. https://www.stationx.net/phishing-statistics/ (2024)
work page 2024
-
[8]
Social Engineering: The Science of Human Hacking
Hadnagy, C. Social Engineering: The Science of Human Hacking. John Wiley & Sons (2018)
work page 2018
-
[9]
Sawyer, B. D. and Hancock, P. A. Hacking the human: The prevalence paradox in cybersecurity. Human Factors, 60 (2018),
work page 2018
Show all 40 references
-
[10]
Human factors in phishing attacks: A systematic literature review
Desolda, G., Ferro, L., Marrella, A., Costabile, M., and Catarci, T. Human factors in phishing attacks: A systematic literature review. ACM Computing Surveys, 54 (2022), 35. doi:10.1145/3469886
2022 doi
-
[11]
Social engineering and organisational dependencies in phishing attacks
Taib, R., Yu, K., Berkovsky, S., Wiggins, M., and Bayl-Smith, P. Social engineering and organisational dependencies in phishing attacks. In Human-Computer Interaction – INTERACT 2019: 17th IFIP TC 13 International Conference, Paphos, Cyprus, September 2–6, 2019, Proceedings, P...
2019 doi
-
[12]
F., Al-Sharai, A
Muneer, A., Ali, R. F., Al-Sharai, A. A., and Fati, S. M. A survey on phishing emails detection techniques. In 2021 International Conference on Innovative Computing (ICIC), pp. 1–6 (2021). doi:10.1109/ICIC53490.2021. 9692960
2021
-
[13]
Pujara, Er Purvi & Chaudhari, M. (2019). Phishing Website De- tection using Machine Learning : A Review
2019
-
[14]
Advancing phishing email detection: A comparative study of deep learning models
Altwaijry, N., Al-Turaiki, I., Alotaibi, R., and Alakeel, F. Advancing phishing email detection: A comparative study of deep learning models. Sensors, 24 (2024). Available from: https://www.mdpi.com/1424-8220/24/7/2077, doi:10.3390/s24072077
2024 doi
-
[15]
K., Buber, E., Demir, O., and Diri, B
Sahingoz, O. K., Buber, E., Demir, O., and Diri, B. Machine learning based phishing detection from urls. Expert Systems with Applications, 117 (2019), 345. Available from: https://www.sciencedirect.com/science/ article/pii/S0957417418306067, doi:https://doi.org/10.1016/j.eswa....
2019 doi
-
[16]
Phishing at- tacks detection a machine learning-based approach
Salahdine, F., El Mrabet, Z., and Kaabouch, N. Phishing at- tacks detection a machine learning-based approach. In 2021 IEEE 12th Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), pp. 0250–0255 (2021). doi:10.1109/UEMCON53757.2021.9666627
2021
-
[17]
Offensive deception in computing
Avery, J., Almeshekah, M., and Spafford, E. Offensive deception in computing. In 12th International Conference on Cyber Warfare and Security (ICCWS’17), pp. 23–31 (2017)
2017
-
[18]
and Isah, H
Beaman, C. and Isah, H. Anomaly detection in emails using machine learning and header information. CoRR, abs/2203.10408 (2022). Available from: https://arxiv.org/abs/2203.10408
2022 arXiv
-
[19]
and Kaabouch, N
Salahdine, F. and Kaabouch, N. Social engineering attacks: A survey. Future Internet, 11 (2019). doi:10.3390/fi11040089
2019 doi
-
[20]
van der Laan, J. J. The semantics of persuasion: A case study us- ing phishing emails. https://unbscholar.lib.unb.ca/items/ 716c75b0- cb66-4575-ada2-fedd2ea9ceeb (2021)
2021
-
[21]
The Weakest Link: How to Diagnose, Detect, and Defend Users from Phishing
Vishwanath, A. The Weakest Link: How to Diagnose, Detect, and Defend Users from Phishing. MIT Press (2022)
2022
-
[22]
Let warnings interrupt the interaction and explain: designing and evaluating phishing email warnings (2023)
Buono, P., Desolda, G., Greco, F., and Piccinno, A. Let warnings interrupt the interaction and explain: designing and evaluating phishing email warnings (2023). doi:10.1145/3544549.3585802
2023
-
[23]
Explanations in warning dialogs to help users defend against phishing attacks (2023)
Desolda, G., Aneke, J., Ardito, C., Lanzilotti, R., and Costabile, M. Explanations in warning dialogs to help users defend against phishing attacks (2023). doi:10.1016/j.ijhcs.2023.103056
2023
-
[24]
David versus goliath: Can machine learning detect llm-generated text? a case study in the detection of phishing emails (2024)
Greco, F., Desolda, G., Esposito, A., and Carelli, A. David versus goliath: Can machine learning detect llm-generated text? a case study in the detection of phishing emails (2024)
2024
-
[25]
Heiding, F., Schneier, B., Vishwanath, A., Bernstein, J., and Park, P. S. Devising and detecting phishing: Large language models vs. smaller human models (2023). Available from: https://arxiv.org/abs/2308.12287, arXiv:2308.12287
2023 arXiv
-
[26]
S., Thota, P., Naragam, K
Roy, S. S., Thota, P., Naragam, K. V ., and Nilizadeh, S. From chatbots to phishbots? – preventing phishing scams created us- ing chatgpt, google bard and claude (2024). Available from: https://arxiv.org/abs/2310.19181, arXiv:2310.19181
2024 arXiv
-
[27]
Chatspamdetec- tor: Leveraging large language models for effective phishing email detection (2024)
Koide, T., Fukushi, N., Nakano, H., and Chiba, D. Chatspamdetec- tor: Leveraging large language models for effective phishing email detection (2024). Available from: https://arxiv.org/abs/2402.18093, arXiv:2402.18093
2024 arXiv
-
[28]
L., Cao, T., Oo, N., Lim, H
Li, Y ., Huang, C., Deng, S., Lock, M. L., Cao, T., Oo, N., Lim, H. W., and Hooi, B. Knowphish: Large language models meet mul- timodal knowledge graphs for enhancing reference-based phishing detection (2024). Available from: https://arxiv.org/abs/2403.02253, arXiv:2403.02253
2024 arXiv
-
[29]
Llama 3.1 8b instruct (2024)
AI, M. Llama 3.1 8b instruct (2024). https://huggingface.co/meta- llama/Llama-3.1-8B-Instruct
2024
-
[30]
Google Safe Browsing
Google Developers. Google Safe Browsing. Available at: https://developers.google.com/safe-browsing
-
[31]
AbuseIPDB - IP Address Abuse Reports
AbuseIPDB. AbuseIPDB - IP Address Abuse Reports. Available at: https://www.abuseipdb.com/
-
[32]
WebExtension APIs for Thunderbird
Thunderbird Developers. WebExtension APIs for Thunderbird. Available at: https://webextension-api.thunderbird.net/en/stable/
-
[33]
Phishing email curated dataset (2023)
Unknown. Phishing email curated dataset (2023). Available from: https: //zenodo.org/records/8339691, doi:10.5281/zenodo.8339691
2023
-
[34]
Wormgpt v3.0 (2023)
Eternanet. Wormgpt v3.0 (2023). Available from: https://flowgpt.com/ chat/wormgpt-v30
2023
-
[35]
Spear phishing with large language models (2023)
Hazell, J. Spear phishing with large language models (2023). Avail- able from: https://arxiv.org/abs/2305.06972, arXiv:2305.06972
2023 arXiv
-
[36]
How polymorphic warnings reduce habituation in the brain
Anderson, B., Kirwan, B., Jenkins, J., Eargle, D., Howard, S., and Vance, A. How polymorphic warnings reduce habituation in the brain. pp. 2883–2892 (2015). doi:10.1145/2702123.2702322
2015
-
[37]
Cialdini, R. B. Influence: The Psychology of Persuasion. Collins Business Essentials, Harper Collins, revised edn. (2009)
2009
-
[38]
Large Language Models: A Comprehensive Survey of its Appli- cations, Challenges, Limitations, and Future Prospects
Muhammad Usman Hadi, Qasem Al Tashi, Rizwan Qureshi, et al. Large Language Models: A Comprehensive Survey of its Appli- cations, Challenges, Limitations, and Future Prospects. TechRxiv. September 05, 2024
2024
-
[39]
Apollo: A gpt-based tool to detect phishing emails and generate explanations that warn users (2024)
Desolda, G., Greco, F., and Vigan `o, L. Apollo: A gpt-based tool to detect phishing emails and generate explanations that warn users (2024). Available from: https://arxiv.org/abs/2410.07997, arXiv:2410.07997
2024 arXiv
-
[40]
Phishvision: A deep learning based visual brand impersonation detector for identifying phishing attacks
Graziano, G., Ucci, D., Bisio, F., and Oneto, L. Phishvision: A deep learning based visual brand impersonation detector for identifying phishing attacks. pp. 123–134. Springer Nature (2024). Appendix A. Cyri Materials and Components All materials and source code of Cyri, inclu...
2024
-
[597]
doi:10.1177/ 0018720818780472
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.