Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Catching the Phish: Detecting Phishing Attacks using Recurrent Neural Networks (RNNs)

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A recurrent neural network trained only on email text can detect phishing at 98.63% F-measure, beating text-based classifiers and matching the DMC state of the art.

desk verdict A plausible incremental result with an overclaimed headline; the evaluation does not support 'outperforms state-of-the-art' but the text-only LSTM idea is worth a rigorous look. read the letter →

arxiv 1908.03640 v1 pith:LT5EAL4F submitted 2019-08-09 cs.CR cs.CLcs.CYcs.LG

classification cs.CRcs.CLcs.CYcs.LG
keywords phishingdetectionrecurrentneuralnetworkLSTMemailsecuritytextclassificationnaturallanguageprocessingcontent-basedfilteringmachinelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that a recurrent neural network can detect phishing emails using only their text, with no hand-crafted features. On the public SA-JN corpus it reports an F-measure of 98.63%, a false negative rate of 1.47%, and precision of 98.74%, which the authors interpret as outperforming the textAnalysis classifier and matching or exceeding the DMCtext classifier. The broader claim is that the sequential structure of email language carries information that current filters largely ignore, so this classifier could sit alongside URL- and metadata-based defenses. If the result holds, phishing detection no longer has to depend on expert-chosen features that quickly become outdated.

What carries the argument

The central object is a two-layer long short-term memory (LSTM) recurrent neural network. Emails are lowercased, cleaned to a 61-character alphabet, tokenized into single-word tokens with special tokens for URLs, email addresses, and punctuation runs, and reduced to a vocabulary of 5,000 stemmed words; very long emails are pruned with a cutout procedure that keeps the beginning, middle, and end. The final hidden state is mapped through a linear layer and sigmoid to a phishing probability, trained with binary cross-entropy, Adam, dropout, gradient clipping, and orthogonal initialization. This machinery carries the argument because it replaces expert feature selection: the network discovers the textual patterns that separate phish from ham.

What would settle it

Retrain the identical architecture on, say, twenty random seeds and evaluate on an independent corpus of phishing emails dated after the training period; if the F-measure falls below the DMCtext level or the variance across runs is large, the claim of outperformance lacks statistical grounding.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that representation learning over word sequences is sufficient for phishing detection at levels competitive with or better than the leading content-based method. The reported numbers on SA-JN are accuracy 98.91%, precision 98.74%, recall 98.53%, false positive rate 1.26%, false negative rate 1.47%, and F-measure 98.63%; on En-JN the F-measure is 96.71% with a 2.50% false positive rate. Because the model sees only text, it does not compete with link-analysis features; it provides an independent judgement, and the authors note that two emails with identical token sequences are inseparable, so perfect classification is impossible in this representation.

Load-bearing premise

The load-bearing premise is that one trained model evaluated on a chronological 10% slice of two public corpora estimates real-world phishing performance closely enough to support the outperformance claim.

Editorial extensions

If this is right

  • Deployed alongside existing filters, the classifier would add an independent text-based signal that does not duplicate URL or metadata analysis.
  • A false negative rate of 1.47% on SA-JN is the phishing-specific priority, since missed attacks are more dangerous than over-blocked mail in this domain.
  • Because the features are learned, retraining on current email would let the system track new phishing language without redesigning feature sets.
  • The classifier's efficiency and text-only input make it suitable for real-time screening before users see an email.
  • Combining the RNN with complementary features should raise overall detection, since the paper shows text structure carries information gain beyond standard features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same word-sequence representation could be applied to phishing websites by treating visible page text or HTML as the sequence; their abstract says the approach can be extended but the paper only evaluates email.
  • A direct test of the complementarity claim would be to strip URLs, sender headers, and metadata from test emails and confirm the RNN's accuracy barely drops; if it does drop, the text-only independence is weaker than claimed.
  • The inseparability of identical token sequences suggests that adding structural or attachment-level features, rather than more text, is the next source of classification headroom.
  • Because the evaluation reports a single trained instance, a practical deployment would need periodic re-evaluation to ensure the model tracks shifts in phishing tactics over time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a phishing email classifier based on an LSTM recurrent neural network. The classifier takes the plain-text content of an email, tokenizes it into a vocabulary of 5000 stemmed word and special tokens, and labels the email as phishing or ham. The authors evaluate on two corpora, SA-JN (SpamAssassin plus Nazario) and En-JN (Enron plus Nazario), using chronological 81%-9%-10% splits. On SA-JN they report 98.91% accuracy, 98.74% precision, 98.53% recall, and 98.63% F-measure; on En-JN they report 96.74% accuracy, 97.45% precision, and 95.98% recall. They compare these results with the textAnalysis classifier of Verma et al. and the DMCtext classifier of Bergholz et al., and claim in the abstract and conclusion that the proposed system outperforms state-of-the-art tools.

Significance. If established, the contribution would be useful: a text-only LSTM that learns features automatically, captures word order, and is complementary to URL- and link-based phishing detectors could complement existing filters and support online retraining. The paper is also transparent about corpus construction and preprocessing, and it explicitly discloses that the results come from a single trained instance. However, the headline claim is not supported by the paper's own evidence. The F-measure advantage over DMCtext is small (0.68 points), DMCtext has higher accuracy and precision, and the baseline comparison is not shown to be controlled. The significance of the work is therefore better described as a promising and complementary content classifier rather than a demonstrated improvement over the state of the art.

major comments (3)
  1. [§5.2, Table 3] The abstract and conclusion claim that the proposed system outperforms state-of-the-art tools, but Table 3 does not support this for DMCtext. On SA-JN, DMCtext has higher accuracy (99.56% vs 98.91%) and higher precision (100.00% vs 98.74%); the RNN's F-measure advantage is 0.68 points (98.63 vs 97.95). Since Section 5.2 explicitly states that the results are from a single trained instance, this margin has no variance estimate or significance test and cannot be distinguished from training noise. The authors should either report multiple runs with standard deviations and a paired significance test, or reframe the claim as comparable or complementary rather than superior.
  2. [Table 3, textAnalysis row] The baseline comparison is not controlled. The textAnalysis row in Table 3 lists the corpus as '?-JN', and nowhere in Section 5 is it stated whether the published DMCtext and textAnalysis results were obtained on the same SA-JN or En-JN splits, with the same chronological ordering, or with the same metric definitions as the RNN. Without this information, or a reproduction of the baselines under the same protocol, the comparison does not provide valid evidence for the outperformance claim.
  3. [§3 and §5.1] The methodology states that 'we encode excess penalty for false positives in the optimisation criterion' to enforce the precision requirement, but the training description in Section 5.1 specifies only binary cross-entropy loss optimized with Adam, with no class weighting, cost-sensitive loss, or threshold adjustment. If such a penalty is implemented, it is not described; if it is not, the design principle asserted in Section 3 is unsupported. The exact loss formulation or weighting should be stated.
minor comments (5)
  1. [§4.3] In the sigmoid definition, the equation reads 1/(1 + exp(-x)) although the input to the sigmoid is z; this should be exp(-z).
  2. [§5] The text says 'with a 9-1 ration twice'; 'ration' should be 'ratio'.
  3. [§5] The split description says 'the most recent 10% of the emails forming the training set', which contradicts the earlier description of training, validation, and testing sets and the stated goal of training on present data to predict future data; presumably the most recent 10% is the test set.
  4. [§5.2] The sentence 'The RNN classifier clearly outperforms the textAnalysis classifier, and has comparable results to the state-of-the-art DMCtext feature' is weaker than the abstract's claim that 'the proposed system outperforms state-of-the-art tools'; these claims should be reconciled.
  5. [§6] There is a typo in the conclusion: 'oru system's performance' should be 'our system's performance'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation against external published baselines; any comparison-protocol concerns are experimental-validity issues, not circular reasoning.

full rationale

This paper is an empirical machine-learning evaluation, not a derivation chain in which an output is defined in terms of an input. The RNN classifier is trained on tokenized email text and evaluated on chronologically separated test splits; the paper states, 'the respective 81%-9%-10% splits respect the received datetime stamps' (Section 5), and the test set 'had been unseen during training' (Section 5.2). The central claim is a comparison with published results from textAnalysis (Verma et al.) and DMCtext (Bergholz et al.) in Table 3. This comparison may be imperfectly controlled because the paper does not establish that the published baselines used the same corpus splits and evaluation protocol, but an uncontrolled comparison is an experimental-rigor problem, not circularity. No fitted parameter is renamed as a prediction: hyperparameters such as hidden size 200, dropout rates, optimization choices, and the four-epoch training schedule are hand-selected design decisions, not quantities fit to the target metric. The precision-oriented loss weighting is a stated design choice, not a quantity derived from the evaluation result. The paper's self-citations, e.g., [10] and [16], support background claims about human susceptibility to phishing and are not load-bearing for the classifier's derivation. No uniqueness theorem, ansatz, or prior result by the same authors is invoked to force the model choice or its evaluation outcome. The paper is self-contained with respect to its own claims in the sense that the reported performance could in principle be falsified by external reimplementation; no step reduces to its own input by construction. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen hyperparameters (hidden size, dictionary size, dropout, pruning threshold, training schedule) and on domain assumptions about the informativeness of tokenized text and the representativeness of the public corpora. No new physical or conceptual entities are introduced; preprocessing tokens such as <url> and <cuts> are implementation details rather than independent entities.

free parameters (6)
  • LSTM hidden state size = 200
    Section 4.3 states 'we will choose the hidden state to be 200 variables large'; chosen by hand, not tuned or justified by data.
  • Dictionary size = 5000
    Section 4.4 reduces the vocabulary to the 4995 most common stemmed words plus 5 special tokens 'in order to acquire round values'; hand-selected.
  • Dropout rates = 0.2 and 0.5
    Section 5.1 applies dropout of 0.2 on the embedding layer and 0.5 between recurrent states and at the output; these are standard settings with no reported tuning.
  • Pruning threshold = 1000 tokens with cutout of one third
    Section 4.5 prunes long emails with a threshold of 1000 tokens and cuts out a third of the sequence; the choice is presented as a compromise, not an optimized value.
  • Batch size and training schedule = batch 200, 4 epochs plus 1 validation epoch
    Section 5.2 defines the training protocol; no sensitivity analysis or learning curves are reported.
  • Adam learning rate = not reported
    Section 5.1 names the Adam optimizer but does not give the learning rate or schedule, preventing exact replication.
assumptions (4)
  • domain assumption Phishing signal is present in the tokenized word sequence of the email body after URLs and non-text structure are removed.
    Sections 4.1 and 4.2 strip attachments, URLs, and non-text characters; the central claim depends on the remaining text retaining enough discriminative signal.
  • domain assumption The public corpora and chronological split represent the deployment scenario.
    Section 5 uses SpamAssassin/Nazario and Enron/Nazario, split by datetime to simulate training on past data to predict future data; this assumes those corpora reflect real phishing email streams.
  • domain assumption Standard LSTM initialization, dropout, and gradient clipping yield a representative model.
    Section 5.1 uses common techniques from the literature but does not compare architectures, random seeds, or alternative regularization settings.
  • domain assumption Metrics computed on a single trained instance are treated as the system's performance.
    Section 5.2 explicitly states 'the results provided are of the single trained instance', assuming that run-to-run variance is negligible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Catching the Phish: Detecting Phishing Attacks using Recurrent Neural Networks (RNNs)." pith.science (2026). https://pith.science/paper/LT5EAL4F

@misc{pith2026190803640,
  author       = {Pith},
  title        = {Pith review of: Catching the Phish: Detecting Phishing Attacks using Recurrent Neural Networks (RNNs)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LT5EAL4F}},
  note         = {Machine review of arXiv:1908.03640}
}
read the original abstract

The emergence of online services in our daily lives has been accompanied by a range of malicious attempts to trick individuals into performing undesired actions, often to the benefit of the adversary. The most popular medium of these attempts is phishing attacks, particularly through emails and websites. In order to defend against such attacks, there is an urgent need for automated mechanisms to identify this malevolent content before it reaches users. Machine learning techniques have gradually become the standard for such classification problems. However, identifying common measurable features of phishing content (e.g., in emails) is notoriously difficult. To address this problem, we engage in a novel study into a phishing content classifier based on a recurrent neural network (RNN), which identifies such features without human input. At this stage, we scope our research to emails, but our approach can be extended to apply to websites. Our results show that the proposed system outperforms state-of-the-art tools. Furthermore, our classifier is efficient and takes into account only the text and, in particular, the textual structure of the email. Since these features are rarely considered in email classification, we argue that our classifier can complement existing classifiers with high information gain.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 25 canonical work pages

  1. [1]

    , González, F.A.: Classify- ing phishing urls using recurrent neural networks

    Bahnsen, A.C., Bohorquez, E.C., Villegas, S., Vargas, J. , González, F.A.: Classify- ing phishing urls using recurrent neural networks. In: Proc eedings of the APWG Symposium on Electronic Crime Research. eCrime ’17, IEEE (A pril 2017)

  2. [2]

    IEEE Transactions on Neural Networks 5(2), 157–166 (March 1994)

    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (March 1994)

  3. [3]

    In: Proceedings of t he Fifth Conference on Email and Anti-Spam

    Bergholz, A., Chang, J.H., Paaß, G., Reichartz, F., Strob el, S.: Improved phishing detection using model-based features. In: Proceedings of t he Fifth Conference on Email and Anti-Spam. CEAS ’08 (August 2008)

  4. [4]

    , Strobel, S.: New filtering approaches for phishing email

    Bergholz, A., De Beer, J., Glahn, S., Moens, M.F., Paaß, G. , Strobel, S.: New filtering approaches for phishing email. Journal of Compute r Security – special issue on EU-funded ICT research on Trust and Security 18(1), 7–35 (January 2010)

  5. [5]

    In: Proceedings of the 9th A nnual NYS Cyber Security Conference

    Chandrasekaran, M., Narayanan, K., Upadhyaya, S.: Phish ing email detection based on structural properties. In: Proceedings of the 9th A nnual NYS Cyber Security Conference. NYSCSC ’06 (June 2006) 12 L. Halgaš et al

  6. [6]

    In: Proceed- ings of the 16th international conference on World Wide Web

    Fette, I., Sadeh, N., Tomasic, A.: Learning to detect phis hing emails. In: Proceed- ings of the 16th international conference on World Wide Web. pp. 649–656. WWW ’07, ACM (May 2007)

  7. [7]

    Neural Computation 12(10), 2451–2471 (October 2000)

    Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to for get: Continual prediction with lstm. Neural Computation 12(10), 2451–2471 (October 2000)

  8. [8]

    MIT Press, Cambridge, MA, USA (2016)

    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning . MIT Press, Cambridge, MA, USA (2016)

Show all 27 references
  1. [9]

    Neural Computation 9(8), 1735–1780 (November 1997)

    Hochreiter, S., Schmidhuber, J.: Long short-term memory . Neural Computation 9(8), 1735–1780 (November 1997)

  2. [10]

    Human-centric Computing and In formation Sciences 6 (June 2016)

    Iuga, C., Nurse, J.R.C., Erola, A.: Baiting the hook: Fac tors impacting suscepti- bility to phishing attacks. Human-centric Computing and In formation Sciences 6 (June 2016)

  3. [11]

    In: Proceedings of the 32nd Interna tional Conference on Machine Learning

    Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirica l exploration of recurrent network architectures. In: Proceedings of the 32nd Interna tional Conference on Machine Learning. pp. 2342–2350. ICML ’15 (July 2015)

  4. [12]

    IEEE Communications Surveys & Tutorials 15(4), 2091–2121 (April 2013)

    Khonji, M., Iraqi, Y., Jones, A.: Phishing detection: A l iterature survey. IEEE Communications Surveys & Tutorials 15(4), 2091–2121 (April 2013)

  5. [14]

    Neural Computing and A pplications 25(2), 443– 458 (August 2014)

    Mohammad, R.M., Thabtah, F., McCluskey, L.: Predicting phishing websites based on self-structuring neural network. Neural Computing and A pplications 25(2), 443– 458 (August 2014)

  6. [15]

    Nazario, J.: https://monkey.org/~jose/phishing/

  7. [16]

    In: The Oxford Handbook of Cyberps ychology

    Nurse, J.R.C.: Cybercrime and you: How criminals attack and the human factors that they seek to exploit. In: The Oxford Handbook of Cyberps ychology. Oxford University Press, Oxford, UK (May 2019)

  8. [17]

    In: Proceedings of the 30th International Con ference on Machine Learning

    Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of t raining recurrent neu- ral networks. In: Proceedings of the 30th International Con ference on Machine Learning. ICML ’13, vol. 28, pp. 1310–1318 (June 2013)

  9. [18]

    PhishMe, Inc.: 2016 enterprise phishing susceptibilit y and resiliency report (2016)

  10. [19]

    Porter, M.F.: Snowball: A language for stemming algorit hms, https://snowballstem.org/

  11. [20]

    Hand book of Information and Communication Security pp

    Ramzan, Z.: Phishing attacks and countermeasures. Hand book of Information and Communication Security pp. 433–448 (2010)

  12. [21]

    arXiv pre print arXiv:1312.6120 (2013)

    Saxe, A.M., McClelland, J.L., Ganguli, S.: Exact soluti ons to the nonlinear dy- namics of learning in deep linear neural networks. arXiv pre print arXiv:1312.6120 (2013)

  13. [22]

    SpamAssassin: https://spamassassin.apache.org/old/publiccorpus/

  14. [23]

    Journal of Machine Learning Research 15(1), 1929–1958 (January 2014)

    Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskeve r, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfit ting. Journal of Machine Learning Research 15(1), 1929–1958 (January 2014)

  15. [24]

    In: 2010 eCrime Researchers Summit

    Toolan, F., Carthy, J.: Feature selection for spam and ph ishing detection. In: 2010 eCrime Researchers Summit. pp. 1–12. IEEE (October 2010)

  16. [25]

    In: 17th European Symposium on Research in Com puter Security

    Verma, R., Shashidhar, N., Hossain, N.: Detecting phish ing emails the natural language way. In: 17th European Symposium on Research in Com puter Security. pp. 824–841. ESORICS ’12, Springer, Berlin, Heidelberg (Se ptember 2012)

  17. [26]

    Jour nal of Intelligent & Fuzzy Systems 34(3), 1333–1343 (March 2018)

    Vinayakumar, R., Soman, K.P., Poornachandran, P.: Eval uating deep learning ap- proaches to characterize and classify malicious urls. Jour nal of Intelligent & Fuzzy Systems 34(3), 1333–1343 (March 2018)

  18. [27]

    arXiv preprint arXiv:1409.2329 (2014) Catching the Phish 13

    Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neur al network regularization. arXiv preprint arXiv:1409.2329 (2014) Catching the Phish 13

  19. [28]

    In: Proceedings of the 12th Inte rnational Conference on Innovative Mobile and Internet Services in Ubiquitous Co mputing

    Zhao, J., Wang, N., Ma, Q., Cheng, Z.: Classifying malici ous urls using gated recurrent neural networks. In: Proceedings of the 12th Inte rnational Conference on Innovative Mobile and Internet Services in Ubiquitous Co mputing. pp. 385–394. IMIS ’18, Springer, Cham (July 2018)

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.