REVIEW 3 major objections 5 minor 27 references
Catching the Phish: Detecting Phishing Attacks using Recurrent Neural Networks (RNNs)
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A recurrent neural network trained only on email text can detect phishing at 98.63% F-measure, beating text-based classifiers and matching the DMC state of the art.
desk verdict A plausible incremental result with an overclaimed headline; the evaluation does not support 'outperforms state-of-the-art' but the text-only LSTM idea is worth a rigorous look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a two-layer long short-term memory (LSTM) recurrent neural network. Emails are lowercased, cleaned to a 61-character alphabet, tokenized into single-word tokens with special tokens for URLs, email addresses, and punctuation runs, and reduced to a vocabulary of 5,000 stemmed words; very long emails are pruned with a cutout procedure that keeps the beginning, middle, and end. The final hidden state is mapped through a linear layer and sigmoid to a phishing probability, trained with binary cross-entropy, Adam, dropout, gradient clipping, and orthogonal initialization. This machinery carries the argument because it replaces expert feature selection: the network discovers the textual patterns that separate phish from ham.
What would settle it
Retrain the identical architecture on, say, twenty random seeds and evaluate on an independent corpus of phishing emails dated after the training period; if the F-measure falls below the DMCtext level or the variance across runs is large, the claim of outperformance lacks statistical grounding.
Extended reading notes
Core claim
On its own terms, the discovery is that representation learning over word sequences is sufficient for phishing detection at levels competitive with or better than the leading content-based method. The reported numbers on SA-JN are accuracy 98.91%, precision 98.74%, recall 98.53%, false positive rate 1.26%, false negative rate 1.47%, and F-measure 98.63%; on En-JN the F-measure is 96.71% with a 2.50% false positive rate. Because the model sees only text, it does not compete with link-analysis features; it provides an independent judgement, and the authors note that two emails with identical token sequences are inseparable, so perfect classification is impossible in this representation.
Load-bearing premise
The load-bearing premise is that one trained model evaluated on a chronological 10% slice of two public corpora estimates real-world phishing performance closely enough to support the outperformance claim.
Editorial extensions
If this is right
- Deployed alongside existing filters, the classifier would add an independent text-based signal that does not duplicate URL or metadata analysis.
- A false negative rate of 1.47% on SA-JN is the phishing-specific priority, since missed attacks are more dangerous than over-blocked mail in this domain.
- Because the features are learned, retraining on current email would let the system track new phishing language without redesigning feature sets.
- The classifier's efficiency and text-only input make it suitable for real-time screening before users see an email.
- Combining the RNN with complementary features should raise overall detection, since the paper shows text structure carries information gain beyond standard features.
Reading between the lines
- The authors leave implicit that the same word-sequence representation could be applied to phishing websites by treating visible page text or HTML as the sequence; their abstract says the approach can be extended but the paper only evaluates email.
- A direct test of the complementarity claim would be to strip URLs, sender headers, and metadata from test emails and confirm the RNN's accuracy barely drops; if it does drop, the text-only independence is weaker than claimed.
- The inseparability of identical token sequences suggests that adding structural or attachment-level features, rather than more text, is the next source of classification headroom.
- Because the evaluation reports a single trained instance, a practical deployment would need periodic re-evaluation to ensure the model tracks shifts in phishing tactics over time.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a phishing email classifier based on an LSTM recurrent neural network. The classifier takes the plain-text content of an email, tokenizes it into a vocabulary of 5000 stemmed word and special tokens, and labels the email as phishing or ham. The authors evaluate on two corpora, SA-JN (SpamAssassin plus Nazario) and En-JN (Enron plus Nazario), using chronological 81%-9%-10% splits. On SA-JN they report 98.91% accuracy, 98.74% precision, 98.53% recall, and 98.63% F-measure; on En-JN they report 96.74% accuracy, 97.45% precision, and 95.98% recall. They compare these results with the textAnalysis classifier of Verma et al. and the DMCtext classifier of Bergholz et al., and claim in the abstract and conclusion that the proposed system outperforms state-of-the-art tools.
Significance. If established, the contribution would be useful: a text-only LSTM that learns features automatically, captures word order, and is complementary to URL- and link-based phishing detectors could complement existing filters and support online retraining. The paper is also transparent about corpus construction and preprocessing, and it explicitly discloses that the results come from a single trained instance. However, the headline claim is not supported by the paper's own evidence. The F-measure advantage over DMCtext is small (0.68 points), DMCtext has higher accuracy and precision, and the baseline comparison is not shown to be controlled. The significance of the work is therefore better described as a promising and complementary content classifier rather than a demonstrated improvement over the state of the art.
major comments (3)
- [§5.2, Table 3] The abstract and conclusion claim that the proposed system outperforms state-of-the-art tools, but Table 3 does not support this for DMCtext. On SA-JN, DMCtext has higher accuracy (99.56% vs 98.91%) and higher precision (100.00% vs 98.74%); the RNN's F-measure advantage is 0.68 points (98.63 vs 97.95). Since Section 5.2 explicitly states that the results are from a single trained instance, this margin has no variance estimate or significance test and cannot be distinguished from training noise. The authors should either report multiple runs with standard deviations and a paired significance test, or reframe the claim as comparable or complementary rather than superior.
- [Table 3, textAnalysis row] The baseline comparison is not controlled. The textAnalysis row in Table 3 lists the corpus as '?-JN', and nowhere in Section 5 is it stated whether the published DMCtext and textAnalysis results were obtained on the same SA-JN or En-JN splits, with the same chronological ordering, or with the same metric definitions as the RNN. Without this information, or a reproduction of the baselines under the same protocol, the comparison does not provide valid evidence for the outperformance claim.
- [§3 and §5.1] The methodology states that 'we encode excess penalty for false positives in the optimisation criterion' to enforce the precision requirement, but the training description in Section 5.1 specifies only binary cross-entropy loss optimized with Adam, with no class weighting, cost-sensitive loss, or threshold adjustment. If such a penalty is implemented, it is not described; if it is not, the design principle asserted in Section 3 is unsupported. The exact loss formulation or weighting should be stated.
minor comments (5)
- [§4.3] In the sigmoid definition, the equation reads 1/(1 + exp(-x)) although the input to the sigmoid is z; this should be exp(-z).
- [§5] The text says 'with a 9-1 ration twice'; 'ration' should be 'ratio'.
- [§5] The split description says 'the most recent 10% of the emails forming the training set', which contradicts the earlier description of training, validation, and testing sets and the stated goal of training on present data to predict future data; presumably the most recent 10% is the test set.
- [§5.2] The sentence 'The RNN classifier clearly outperforms the textAnalysis classifier, and has comparable results to the state-of-the-art DMCtext feature' is weaker than the abstract's claim that 'the proposed system outperforms state-of-the-art tools'; these claims should be reconciled.
- [§6] There is a typo in the conclusion: 'oru system's performance' should be 'our system's performance'.
Circularity Check
No circularity: the paper is an empirical evaluation against external published baselines; any comparison-protocol concerns are experimental-validity issues, not circular reasoning.
full rationale
This paper is an empirical machine-learning evaluation, not a derivation chain in which an output is defined in terms of an input. The RNN classifier is trained on tokenized email text and evaluated on chronologically separated test splits; the paper states, 'the respective 81%-9%-10% splits respect the received datetime stamps' (Section 5), and the test set 'had been unseen during training' (Section 5.2). The central claim is a comparison with published results from textAnalysis (Verma et al.) and DMCtext (Bergholz et al.) in Table 3. This comparison may be imperfectly controlled because the paper does not establish that the published baselines used the same corpus splits and evaluation protocol, but an uncontrolled comparison is an experimental-rigor problem, not circularity. No fitted parameter is renamed as a prediction: hyperparameters such as hidden size 200, dropout rates, optimization choices, and the four-epoch training schedule are hand-selected design decisions, not quantities fit to the target metric. The precision-oriented loss weighting is a stated design choice, not a quantity derived from the evaluation result. The paper's self-citations, e.g., [10] and [16], support background claims about human susceptibility to phishing and are not load-bearing for the classifier's derivation. No uniqueness theorem, ansatz, or prior result by the same authors is invoked to force the model choice or its evaluation outcome. The paper is self-contained with respect to its own claims in the sense that the reported performance could in principle be falsified by external reimplementation; no step reduces to its own input by construction. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (6)
- LSTM hidden state size =
200
- Dictionary size =
5000
- Dropout rates =
0.2 and 0.5
- Pruning threshold =
1000 tokens with cutout of one third
- Batch size and training schedule =
batch 200, 4 epochs plus 1 validation epoch
- Adam learning rate =
not reported
assumptions (4)
- domain assumption Phishing signal is present in the tokenized word sequence of the email body after URLs and non-text structure are removed.
- domain assumption The public corpora and chronological split represent the deployment scenario.
- domain assumption Standard LSTM initialization, dropout, and gradient clipping yield a representative model.
- domain assumption Metrics computed on a single trained instance are treated as the system's performance.
Cite this review
Pith. "Pith review of Catching the Phish: Detecting Phishing Attacks using Recurrent Neural Networks (RNNs)." pith.science (2026). https://pith.science/paper/LT5EAL4F
@misc{pith2026190803640,
author = {Pith},
title = {Pith review of: Catching the Phish: Detecting Phishing Attacks using Recurrent Neural Networks (RNNs)},
year = {2026},
howpublished = {\url{https://pith.science/paper/LT5EAL4F}},
note = {Machine review of arXiv:1908.03640}
}
read the original abstract
The emergence of online services in our daily lives has been accompanied by a range of malicious attempts to trick individuals into performing undesired actions, often to the benefit of the adversary. The most popular medium of these attempts is phishing attacks, particularly through emails and websites. In order to defend against such attacks, there is an urgent need for automated mechanisms to identify this malevolent content before it reaches users. Machine learning techniques have gradually become the standard for such classification problems. However, identifying common measurable features of phishing content (e.g., in emails) is notoriously difficult. To address this problem, we engage in a novel study into a phishing content classifier based on a recurrent neural network (RNN), which identifies such features without human input. At this stage, we scope our research to emails, but our approach can be extended to apply to websites. Our results show that the proposed system outperforms state-of-the-art tools. Furthermore, our classifier is efficient and takes into account only the text and, in particular, the textual structure of the email. Since these features are rarely considered in email classification, we argue that our classifier can complement existing classifiers with high information gain.
Reference graph
Works this paper leans on
-
[1]
, González, F.A.: Classify- ing phishing urls using recurrent neural networks
Bahnsen, A.C., Bohorquez, E.C., Villegas, S., Vargas, J. , González, F.A.: Classify- ing phishing urls using recurrent neural networks. In: Proc eedings of the APWG Symposium on Electronic Crime Research. eCrime ’17, IEEE (A pril 2017)
work page 2017
-
[2]
IEEE Transactions on Neural Networks 5(2), 157–166 (March 1994)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (March 1994)
work page 1994
-
[3]
In: Proceedings of t he Fifth Conference on Email and Anti-Spam
Bergholz, A., Chang, J.H., Paaß, G., Reichartz, F., Strob el, S.: Improved phishing detection using model-based features. In: Proceedings of t he Fifth Conference on Email and Anti-Spam. CEAS ’08 (August 2008)
work page 2008
-
[4]
, Strobel, S.: New filtering approaches for phishing email
Bergholz, A., De Beer, J., Glahn, S., Moens, M.F., Paaß, G. , Strobel, S.: New filtering approaches for phishing email. Journal of Compute r Security – special issue on EU-funded ICT research on Trust and Security 18(1), 7–35 (January 2010)
work page 2010
-
[5]
In: Proceedings of the 9th A nnual NYS Cyber Security Conference
Chandrasekaran, M., Narayanan, K., Upadhyaya, S.: Phish ing email detection based on structural properties. In: Proceedings of the 9th A nnual NYS Cyber Security Conference. NYSCSC ’06 (June 2006) 12 L. Halgaš et al
work page 2006
-
[6]
In: Proceed- ings of the 16th international conference on World Wide Web
Fette, I., Sadeh, N., Tomasic, A.: Learning to detect phis hing emails. In: Proceed- ings of the 16th international conference on World Wide Web. pp. 649–656. WWW ’07, ACM (May 2007)
work page 2007
-
[7]
Neural Computation 12(10), 2451–2471 (October 2000)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to for get: Continual prediction with lstm. Neural Computation 12(10), 2451–2471 (October 2000)
work page 2000
-
[8]
MIT Press, Cambridge, MA, USA (2016)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning . MIT Press, Cambridge, MA, USA (2016)
work page 2016
Show all 27 references
-
[9]
Neural Computation 9(8), 1735–1780 (November 1997)
Hochreiter, S., Schmidhuber, J.: Long short-term memory . Neural Computation 9(8), 1735–1780 (November 1997)
1997
-
[10]
Human-centric Computing and In formation Sciences 6 (June 2016)
Iuga, C., Nurse, J.R.C., Erola, A.: Baiting the hook: Fac tors impacting suscepti- bility to phishing attacks. Human-centric Computing and In formation Sciences 6 (June 2016)
2016
-
[11]
In: Proceedings of the 32nd Interna tional Conference on Machine Learning
Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirica l exploration of recurrent network architectures. In: Proceedings of the 32nd Interna tional Conference on Machine Learning. pp. 2342–2350. ICML ’15 (July 2015)
2015
-
[12]
IEEE Communications Surveys & Tutorials 15(4), 2091–2121 (April 2013)
Khonji, M., Iraqi, Y., Jones, A.: Phishing detection: A l iterature survey. IEEE Communications Surveys & Tutorials 15(4), 2091–2121 (April 2013)
2013
-
[14]
Neural Computing and A pplications 25(2), 443– 458 (August 2014)
Mohammad, R.M., Thabtah, F., McCluskey, L.: Predicting phishing websites based on self-structuring neural network. Neural Computing and A pplications 25(2), 443– 458 (August 2014)
2014
-
[15]
Nazario, J.: https://monkey.org/~jose/phishing/
-
[16]
In: The Oxford Handbook of Cyberps ychology
Nurse, J.R.C.: Cybercrime and you: How criminals attack and the human factors that they seek to exploit. In: The Oxford Handbook of Cyberps ychology. Oxford University Press, Oxford, UK (May 2019)
2019
-
[17]
In: Proceedings of the 30th International Con ference on Machine Learning
Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of t raining recurrent neu- ral networks. In: Proceedings of the 30th International Con ference on Machine Learning. ICML ’13, vol. 28, pp. 1310–1318 (June 2013)
2013
-
[18]
PhishMe, Inc.: 2016 enterprise phishing susceptibilit y and resiliency report (2016)
2016
-
[19]
Porter, M.F.: Snowball: A language for stemming algorit hms, https://snowballstem.org/
-
[20]
Hand book of Information and Communication Security pp
Ramzan, Z.: Phishing attacks and countermeasures. Hand book of Information and Communication Security pp. 433–448 (2010)
2010
-
[21]
arXiv pre print arXiv:1312.6120 (2013)
Saxe, A.M., McClelland, J.L., Ganguli, S.: Exact soluti ons to the nonlinear dy- namics of learning in deep linear neural networks. arXiv pre print arXiv:1312.6120 (2013)
2013 arXiv
-
[22]
SpamAssassin: https://spamassassin.apache.org/old/publiccorpus/
-
[23]
Journal of Machine Learning Research 15(1), 1929–1958 (January 2014)
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskeve r, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfit ting. Journal of Machine Learning Research 15(1), 1929–1958 (January 2014)
1929
-
[24]
In: 2010 eCrime Researchers Summit
Toolan, F., Carthy, J.: Feature selection for spam and ph ishing detection. In: 2010 eCrime Researchers Summit. pp. 1–12. IEEE (October 2010)
2010
-
[25]
In: 17th European Symposium on Research in Com puter Security
Verma, R., Shashidhar, N., Hossain, N.: Detecting phish ing emails the natural language way. In: 17th European Symposium on Research in Com puter Security. pp. 824–841. ESORICS ’12, Springer, Berlin, Heidelberg (Se ptember 2012)
2012
-
[26]
Jour nal of Intelligent & Fuzzy Systems 34(3), 1333–1343 (March 2018)
Vinayakumar, R., Soman, K.P., Poornachandran, P.: Eval uating deep learning ap- proaches to characterize and classify malicious urls. Jour nal of Intelligent & Fuzzy Systems 34(3), 1333–1343 (March 2018)
2018
-
[27]
arXiv preprint arXiv:1409.2329 (2014) Catching the Phish 13
Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neur al network regularization. arXiv preprint arXiv:1409.2329 (2014) Catching the Phish 13
2014 arXiv
-
[28]
In: Proceedings of the 12th Inte rnational Conference on Innovative Mobile and Internet Services in Ubiquitous Co mputing
Zhao, J., Wang, N., Ma, Q., Cheng, Z.: Classifying malici ous urls using gated recurrent neural networks. In: Proceedings of the 12th Inte rnational Conference on Innovative Mobile and Internet Services in Ubiquitous Co mputing. pp. 385–394. IMIS ’18, Springer, Cham (July 2018)
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.