REVIEW 5 major objections 6 minor 14 references
AdaPhish: AI-Powered Adaptive Defense and Education Resource Against Deceptive Emails
T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read AdaPhish builds a phish bowl that anonymizes emails with an LLM, classifies by vector similarity, and reports 98.41% accuracy on a held-out test set.
desk verdict Solid engineering contribution that assembles known parts into a useful phish bowl tool, but the headline 'real-time adaptation' claim is not measured, and the reported numbers look tuned to the same fixed test set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the confidence-weighted ensemble of two analyzers. The phish bowl-based analyzer uses lazy learning: it memorizes training samples as vectors in a vector database and, at query time, labels a new email by a reciprocal-distance weighted sum over its $k=12$ closest neighbors. The GPT-based analyzer classifies from a prompted large language model. The mixing function $f(l'_{\mathrm{conf}})=0.8\sqrt{l'_{\mathrm{conf}}}$ uses the bowl's confidence, $l'_{\mathrm{conf}}=e^{-\lambda d_0^2}$, as a dynamic weight: near-duplicates of known phishes are decided by the bowl, while unfamiliar emails are deferred to the LLM. This design is what lets the system adapt to new tactics immediately, because adaptation means inserting a new vector rather than retraining a model.
What would settle it
Run the published pipeline unchanged on a fresh collection of phishing and benign emails from after the training corpus's time period and compare accuracy and precision with the reported 98.41% and 99.60%; a substantial drop would falsify the adaptation claim. Separately, measure end-to-end latency under sustained submission volume and check whether it keeps pace with the rate at which new phishes arrive, which would settle whether the system is actually real-time.
Extended reading notes
Core claim
The core claim is that a lazy-learning classifier over email embeddings can be combined with a prompted general-purpose LLM to achieve higher precision than either analyzer alone while staying adaptive. The phish bowl analyzer stores known emails as high-dimensional vectors and labels an incoming email by a distance-weighted sum over its nearest neighbors, with an exact match receiving full weight; a confidence term, $l'_{\mathrm{conf}}=e^{-\lambda d_0^2}$, suppresses predictions when no close neighbor exists. The GPT-based analyzer classifies from the text alone, and the ensemble weight $f(l'_{\mathrm{conf}})=0.8\sqrt{l'_{\mathrm{conf}}}$ shifts control from the LLM to the phish bowl as similarity confidence grows. In the final iteration the authors report 98.41% accuracy, 99.60% precision, and 97.22% recall on 4,096 held-out emails, and because the bowl only stores and retrieves embeddings, newly submitted phishes influence later classifications without retraining.
Load-bearing premise
The load-bearing premise is that the hand-tuned confidence mixing and decay parameters, and the fixed historical test set they were tuned on, represent how the system will behave on real, current email traffic; if real distributions differ, the reported accuracy and the real-time adaptation claims do not transfer.
Editorial extensions
If this is right
- Newly submitted phishing emails are embedded and stored, so they immediately influence the classification of subsequent similar emails without any retraining step.
- The confidence-decay mechanism lets the phish-bowl analyzer label emails as benign when no close vector neighbor exists, which the authors show prevents the bowl from collapsing when it contains only phishing samples.
- The ensemble outperforms either analyzer alone in the reported experiments, supporting the claim that a similarity-based memory and a general LLM can be complementary.
- The trend analyzer can issue alerts when groups of similar phishes accumulate above a threshold derived from daily volume, allowing the same infrastructure to track campaigns rather than just individual emails.
- Automatic anonymization via chain-of-thought prompting is intended to let organizations share phish-bowl data externally without manual redaction, making collaborative defense feasible.
Reading between the lines
- The paper leaves implicit that the same lazy-learning-plus-confidence ensemble could be applied to other fast-changing classification domains, such as spam, fraud, or content moderation, where a static model goes stale and user reports can be inserted into a similarity memory.
- A stricter test of the adaptation claim would use a temporal split, training the bowl only on older emails and testing on newer ones, because the reported numbers come from a randomly split historical corpus and may not reflect drift in attack tactics.
- The mixing factor 0.8 and the decay parameter $\lambda$ are hand-set without a described validation procedure; an automatic schedule that adjusts them to bowl size and class imbalance would be a natural extension.
- Because the reciprocal-distance weighting makes exact or near-exact matches dominate, adversarial rephrasing of an email could probe whether the vector similarity is robust to wording changes that preserve meaning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents AdaPhish, a phishing email detection and education platform that combines automatic anonymization by GPT-4o, a lazy-learning 'phish bowl' analyzer using embeddings and k-nearest-neighbor search, a GPT-4o-based zero-shot analyzer, a confidence-weighted ensemble, and a trend analyzer for alerts. The authors describe the architecture, prompts, and equations, and report an evaluation on 4,096 emails from a mixed phishing dataset, with the final ensemble achieving 98.41% accuracy, 99.60% precision, and 97.22% recall. The paper claims real-time detection, adaptation to new phishing tactics, and long-term tracking of phishing trends.
Significance. If substantiated, the contribution would be valuable: a privacy-preserving, collaborative phish bowl that removes the manual anonymization bottleneck and adapts without retraining is a practical and interesting direction. The manuscript is transparent about its design choices, provides concrete prompts (Templates I-III) and explicit formulas, and presents an iteration-by-iteration ablation that helps the reader see how each component affects accuracy. However, the current evidence does not support the central dynamic claims: the evaluation is a single static split, the hyperparameters are selected on the same test set, and no latency, temporal, or trend-analyzer experiments are reported. The strength of the paper lies in the system description and the reproducible evaluation setup, not yet in the evidence for adaptation.
major comments (5)
- [§V.A, §V.B, Eq. (2)] The final metrics (98.41% accuracy, 99.60% precision, 97.22% recall) are reported on a single fixed 4,096-email test set that was also used for all iteration decisions and for choosing the hyperparameters (k=12 in Section IV.D, λ in Section IV.D, f(l'_conf)=0.8√l'_conf in Section IV.F, and OCR thresholds in Section IV.A are all described as 'found' or 'shown to work best during testing' without a separate validation set). This protocol risks selection bias and gives no estimate of variance. Please report a train/validation/test split or nested cross-validation, and provide confidence intervals or multiple-split results.
- [§V.A] The train/test split is random, not temporal, on a dataset whose emails date back to 2002. Random assignment means train and test likely contain emails from the same campaigns and time periods, so the evaluation does not test the paper's central claim of rapid adaptation to new phishing tactics. Please evaluate with a temporal split (train on older emails, test on a later time window) or on a freshly collected corpus of recent phishing emails, and report the phish bowl analyzer's performance when the bowl is small.
- [Abstract, §II.C] The claim of 'real-time detection' is not measured. The lazy-learning design avoids retraining, but each classification still requires embedding the input and performing a k-nearest-neighbor search in Chroma, and no latency, throughput, or scaling experiments are reported. Please provide end-to-end classification latency on representative hardware and, ideally, latency as a function of phish bowl size.
- [§IV.G] The trend analyzer (grouping by distance threshold δ, alert threshold t_alert, and decay k_alert) is described but never evaluated, so the claims of 'real-time alerts' and 'long-term tracking of phishing trends' are unsupported. Please include an evaluation of the trend analyzer on a labeled temporal sequence, for example alert precision/recall, or explicitly scope this component as a design proposal rather than a demonstrated feature.
- [§V.B, Table I] No standard machine-learning baselines are evaluated on the same split; the only comparisons are the system's own iterative variants. Adding a classical baseline such as TF-IDF with logistic regression or a conventional kNN classifier on the same 4,096-email test set would substantially strengthen the claim that the LLM-based pipeline is competitive and would contextualize the reported 98.41% accuracy.
minor comments (6)
- [Figures] Figure 1 and Figure 3 both carry the caption 'Platform Architecture'; one appears to be a duplicate or the numbering is wrong.
- [§IV.C] The token-per-character estimate of 0.2815 is given without justification or source; please cite or derive it.
- [§IV.D, Eq. (2)] The description of the output label is confusing: the analyzer's output is already l_raw' × l_conf', and Equation (2) again multiplies by f(l_conf'), so the composition of the two steps should be stated more explicitly.
- [Table II] The asterisk on the 16,384 row (phish-only) should be explained in the caption, and the difference in λ (1.0 vs. 0.5) should be justified.
- [References] Reference [14] is incomplete (missing page numbers/DOI), and reference [6] is a blog post without a listed author; please complete the bibliography.
- [§II.D] There is a typo in the section title: 'Retireval' should be 'Retrieval'.
Circularity Check
No significant circularity: the paper reports an empirical system evaluation rather than a derived prediction, and its self-citation is not load-bearing.
full rationale
AdaPhish is an engineering paper with an empirical evaluation, not a derivation chain in which an output is shown to equal an input by construction. The nearest-neighbor classifier, GPT-4o prompt, and ensembling formula are all implemented mechanisms, and the reported 98.41% accuracy is an experimental measurement on a fixed 4,096-email test split. The choice of f(l_conf) = 0.8 sqrt(l_conf) in Section IV.F and the confidence decay lambda in Section IV.D are described as found to give the best balance, and the same subsets were used across iterations in Section V.A; this is a model-selection/overfitting concern about whether the headline metric would generalize, not a circularity in the paper's derivation. The strongest claimed capability, 'real-time detection and adaptation,' is supported only by the lazy-learning architecture, not by fresh-data or latency experiments; that is an evidence gap, not a self-referential reduction. The only self-citation is [9], a prior Federated Phish Bowl paper by a co-author, and it appears in Related Work for comparison without carrying any load-bearing argument. No quoted equation reduces to another by definition, no fitted parameter is renamed as a prediction, and no uniqueness theorem or self-citation chain forces the chosen design. Therefore the paper is self-contained as an empirical study and has no significant circularity.
Assumptions & free parameters
free parameters (7)
- k (number of nearest neighbors) =
12
- epsilon (distance stability constant) =
small value, unspecified
- lambda (confidence decay) =
0.5 or 1.0 in experiments
- ensemble mixing coefficient =
0.8 sqrt(l'_conf)
- OCR and header heuristics =
t_OCR=80, t_header=7, k_subject=1.25, t_logo=1.5
- trend analyzer thresholds =
delta, t_alert, k_alert depend on usage
- token-per-character estimate =
0.2815
assumptions (4)
- domain assumption Embedding distance in squared Euclidean space corresponds to semantic similarity relevant for phishing detection.
- domain assumption GPT-4o with prompt engineering is a reliable zero-shot phishing classifier.
- domain assumption The curated dataset [5] is representative of phishing emails that a deployed system would encounter.
- domain assumption Phish bowl labels are trustworthy.
Cite this review
Pith. "Pith review of AdaPhish: AI-Powered Adaptive Defense and Education Resource Against Deceptive Emails." pith.science (2026). https://pith.science/paper/Q2HLQL4L
@misc{pith2026250203622,
author = {Pith},
title = {Pith review of: AdaPhish: AI-Powered Adaptive Defense and Education Resource Against Deceptive Emails},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q2HLQL4L}},
note = {Machine review of arXiv:2502.03622}
}
read the original abstract
Phishing attacks remain a significant threat in the digital age, yet organizations lack effective methods to tackle phishing attacks without leaking sensitive information. Phish bowl initiatives are a vital part of cybersecurity efforts against these attacks. However, traditional phish bowls require manual anonymization and are often limited to internal use. To overcome these limitations, we introduce AdaPhish, an AI-powered phish bowl platform that automatically anonymizes and analyzes phishing emails using large language models (LLMs) and vector databases. AdaPhish achieves real-time detection and adaptation to new phishing tactics while enabling long-term tracking of phishing trends. Through automated reporting, adaptive analysis, and real-time alerts, AdaPhish presents a scalable, collaborative solution for phishing detection and cybersecurity education.
Figures
Reference graph
Works this paper leans on
-
[1]
2024 Data Breach Investigations Report,
“2024 Data Breach Investigations Report,” Verizon Enterprise Solutions. Accessed: Jul, 2024. [Online]. Available: https://www.verizon.com/business/resources/reports/2024-dbir-data- breach-investigations-report.pdf
work page 2024
-
[2]
An overview of the Tesseract OCR Engine,
R. Smith, "An overview of the Tesseract OCR Engine," Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil, 2007, pp. 629 -633, doi: 10.1109/ICDAR.2007.4376991
arXiv 2007
-
[3]
Chain-of-thought prompting elicits reasoning in Large Language Models,
J. Wei et al., “Chain-of-thought prompting elicits reasoning in Large Language Models,” Jan. 10, 2023, arXiv: arXiv:2201.11903. doi: 10.48550/arXiv.2201.11903
-
[4]
ChatSpamDetector: Leveraging Large Language Models for effective phishing email detection,
T. Koide, N. Fukushi, H. Nakano, and D. Chiba, “ChatSpamDetector: Leveraging Large Language Models for effective phishing email detection,” Aug. 23, 2024, arXiv: arXiv:2402.18093. doi: 10.48550/arXiv.2402.18093
-
[5]
Curated datasets and feature analysis for phishing email detection with machine learning,
A. I. Champa, M. F. Rabbi, and M. F. Zibran, “Curated datasets and feature analysis for phishing email detection with machine learning,” in 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI) , Apr. 2024, pp. 1 –7. doi: 10.1109/ICMI60790.2024.10585821
arXiv 2024
-
[6]
Rephrased labels improve zero-shot text classification by 30%
“Rephrased labels improve zero-shot text classification by 30%.” Accessed: Aug, 2024. [Online]. Available: https://jina.ai/news/rephrased-labels-improve-zero-shot-text- classification-30/
work page 2024
-
[7]
Detecting phishing websites using machine learning technique,
A. K. Dutta, “Detecting phishing websites using machine learning technique,” PLOS ONE, vol. 16, no. 10, p. e0258361, Oct. 2021, doi: 10.1371/journal.pone.0258361
-
[8]
A Deep Learning Model with Hierarchical LSTMs and Supervised Attention for Anti-Phishing
M. Nguyen, T. Nguyen, and T. H. Nguyen, “A Deep Learning Model with Hierarchical LSTMs and Supervised Attention for Anti - Phishing,” May 03, 2018, arXiv: arXiv:1805.01554. doi: 10.48550/arXiv.1805.01554
work page Pith review arXiv doi:10.48550/arxiv.1805.01554 2018
Show all 14 references
-
[9]
Federated Phish Bowl: LSTM - Based Decentralized Phishing Email Detection,
Y. Sun, N. Chong, and H. Ochiai, “Federated Phish Bowl: LSTM - Based Decentralized Phishing Email Detection,” in 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Oct. 2022, pp. 20–25. doi: 10.1109/SMC53654.2022.994558
2022
-
[10]
Exploring the Efficacy of Federated -Continual Learning Nodes with Attention - Based Classifier for Robust Web Phishing Detection: An Empirical Investigation,
J. M. Joshua, R. Adhithya, S. S. Dananjay, and M. Revathi, “Exploring the Efficacy of Federated -Continual Learning Nodes with Attention - Based Classifier for Robust Web Phishing Detection: An Empirical Investigation,” in 2024 International Conference on Advancements in Power...
2024
-
[11]
Secure Multi-Party Computation: Theory, practice and applications,
C. Zhao et al., “Secure Multi-Party Computation: Theory, practice and applications,” Information Sciences, vol. 476, pp. 357–372, Feb. 2019, doi: 10.1016/j.ins.2018.10.024
2019 doi
-
[12]
Secure multiparty computation,
Y. Lindell, “Secure multiparty computation,” Commun. ACM, vol. 64, no. 1, pp. 86–96, Dec. 2020, doi: 10.1145/3387108
2020 doi
-
[13]
Fine -Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero -Shot Generative AI Models in Text Classification,
M. J. J. Bucher and M. Martini, “Fine -Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero -Shot Generative AI Models in Text Classification,” arXiv.org. Accessed: Oct, 2024. [Online]. Available: https://arxiv.org/abs/2406.08660v1
2024 arXiv
-
[14]
Smart Expert System: Large Language Models as Text Classifiers,
Z. Wang, Y. Pang, and Y. Lin, “Smart Expert System: Large Language Models as Text Classifiers,” arXiv.org. Accessed: Oct, 2024. [Online]. Available: https://arxiv.org/abs/2405.10523v
2024 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.