Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Enhancing Phishing Detection in Financial Systems through NLP

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NLP keyword-scoring catches financial phishing at 79.8 percent

desk verdict The 79.8% accuracy is an in-sample fit: the threshold is tuned and tested on the same 500 emails, so the central empirical claim is unsupported. read the letter →

arxiv 2507.04426 v1 pith:2V3VIUAE submitted 2025-07-06 cs.CR

Novruz Amirov , Leminur Celik , Egemen Ali Caner , Emre Yurdakul , Fahri Anil Yerlikaya , Serif Bahtiyar This is my paper · ORCID
classification cs.CR
keywords phishingdetectionfinancialsystemsnaturallanguageprocessingTF-IDFsemanticsimilarityUniversalSentenceEncoderkeywordextractionemailclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a phishing detector for financial systems that never trains a binary classifier: it builds a short keyword list from phishing corpora and classifies an incoming email by how close its text sits to that list. Two routes are compared, one using TF-IDF weights (a standard measure of how distinctive a word is in a document) and one using clustered word embeddings plus semantic similarity, with the Universal Sentence Encoder providing the embeddings. On a balanced set of 500 emails, the TF-IDF route reaches 79.8 percent accuracy and the semantic route 67.2 percent, with both numbers depending on how many keywords are kept. The paper argues the TF-IDF route is the more dependable of the two because its accuracy stays high as keyword count changes, while the semantic route is sensitive to keyword noise. If the result holds, it offers a lightweight, adaptable detection approach that addresses the zero-hour blind spot of blacklist and whitelist methods.

What carries the argument

The load-bearing mechanism is a two-stage similarity scorer. Stage one produces the keyword list: TF-IDF vectorization of the combined phishing corpus, or K-means clustering over Universal Sentence Encoder word embeddings with the words nearest each cluster center selected. Stage two embeds an email with the Universal Sentence Encoder, computes the cosine similarity between the email vector and the keyword list, and applies a heuristic threshold that turns the score into a phishing or safe label. The threshold is the only decision boundary in the system, so the entire method reduces to whether email text is close enough, in embedding space, to words that typify phishing.

What would settle it

Split the 500-email set into a threshold-tuning portion and a disjoint held-out portion, rerun both pipelines, and compare accuracies; a large drop on the held-out emails would show the reported 79.8 and 67.2 percent figures come from threshold fitting rather than generalization.

Watch

Extended reading notes

Core claim

The central claim is that phishing emails can be recognized by lexical and semantic proximity to a small set of keywords, without needing a learned classifier. The pipeline first extracts keywords from phishing-only corpora, either by TF-IDF weighting or by clustering word embeddings and selecting the most representative words; it then embeds each incoming email, computes cosine similarity to the keyword list, and labels the email phishing when similarity passes a threshold chosen by inspecting breakpoints in accuracy. On the evaluation set the best TF-IDF configuration, 19 keywords, reaches 79.8 percent accuracy, and the best semantic configuration, 16 keywords, reaches 67.2 percent. The paper's conclusion is that the non-semantic route is the more stable choice because it maintains high accuracy across a wider range of keyword counts, whereas the semantic route's performance degrades as extra keywords add noise.

Load-bearing premise

The load-bearing premise is that a threshold chosen heuristically on the 500-email set used for testing will generalize to unseen emails, so if that assumption is false the reported accuracy figures are in-sample fits rather than predictions.

Editorial extensions

If this is right

  • A working detector can be assembled from a phishing corpus and a threshold, with no labeled benign email corpus and no classifier training.
  • Accuracy depends sharply on keyword-list size, so tuning that size is the main practical knob: 19 keywords for TF-IDF and 16 for the semantic route in this dataset.
  • The semantic route's sensitivity to keyword count makes keyword selection, not the similarity model, the primary place where detection quality is won or lost.
  • Because the threshold comes from heuristically examining breakpoints, the system can in principle be re-tuned as phishing language evolves, which list-based defenses cannot do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter test would hold out part of the 500-email set when choosing the threshold; the reported 79.8 and 67.2 percent figures are likely optimistic because the threshold is adapted and evaluated on the same emails.
  • The TF-IDF route may be measuring lexical overlap with the phishing corpus more than genuine semantic content; a phishing email that avoids the extracted keywords would probably evade it, while the semantic route, if keyword noise were controlled, might catch paraphrases.
  • An ablation of the keyword lists could reveal whether the signal is carried by generic financial words such as 'account,' 'confirm,' and 'verify'; if so, the method would be close to a lexicon filter and vulnerable to simple wording changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an NLP-based phishing detection method for financial systems. The approach has three phases: keyword extraction via TF-IDF and semantic clustering, computation of semantic similarity between emails and a phishing keyword list using the Universal Sentence Encoder, and classification using a cosine-similarity threshold. The authors report experimental accuracies of up to 79.8% for the TF-IDF model and 67.2% for the semantic model on a 500-email dataset. The paper also surveys blacklist, whitelist, and machine-learning phishing detection methods.

Significance. If the reported accuracies were valid estimates of generalization, the paper would offer a modest but useful comparison of a simple keyword-similarity approach against semantic analysis for phishing detection. The authors are transparent about their experimental choices, which is a strength in reporting. However, the central empirical claim is not supported because the decision threshold is selected on the same dataset used for evaluation, making the reported accuracies in-sample fits. The paper also lacks baselines, statistical significance measures, and reproducibility artifacts, so the significance of the contribution cannot be assessed from the current manuscript.

major comments (4)
  1. [Section 4.1] The evaluation is circular. Section 4.1 states 'The same dataset, which is the dataset for adapting the threshold, is used for testing,' and Section 3.2 describes the threshold as defined by 'heuristically examining breakpoints' on that dataset. Because the threshold is selected using the test emails' outcomes, the accuracies reported in Sections 4.2 and 4.3 (including 79.8% and 67.2%) are in-sample fits, not estimates of performance on unseen emails. The authors must split the data so that threshold selection and evaluation are done on disjoint sets, or use cross-validation, before any accuracy claim can be supported.
  2. [Sections 4.2 and 4.3] No confidence intervals, statistical tests, or cross-validation are reported. With a balanced 500-email dataset, the differences between experiments (e.g., 57.7% versus 55.2% in Section 4.2) are likely within sampling noise, so statements about 'information saturation' and a 'delicate balancing act' are not justified. The evaluation should include statistical uncertainty measures and comparisons against simple baselines, such as majority-class classification, random guessing, or a standard TF-IDF classifier.
  3. [Section 4.4] The comparative analysis contains a factual contradiction. Section 4.4 states that the semantic model 'peaked with experiment 1 and then steadily decreased as the number of keywords increased,' but the results in Section 4.2 show Experiment 6 achieving the highest success rate (67.2%). This inconsistency undermines the conclusion that the non-semantic model is consistently more reliable and must be corrected.
  4. [Section 3.3] The description of the threshold mechanism is contradictory. The text claims 'Machine learning algorithms, such as reinforcement learning, is employed to continuously optimize the threshold,' but immediately adds 'the threshold value is found heuristically by examining breakpoints.' No reinforcement learning algorithm is described, implemented, or evaluated anywhere in the paper. Either provide the adaptive-threshold mechanism or remove the unsupported claim.
minor comments (5)
  1. [Abstract and Conclusion] The headline accuracies (79.8% and 67.2%) are reported without noting that they were obtained on the same dataset used for threshold selection; this qualifier should be added if the experiments are rerun with a proper split.
  2. [Section 3.2] The Adapting phase is described as using 'a dataset consisting of only phishing emails,' but Section 4.1 says the 500-email dataset (containing both phishing and safe emails) is used for adapting the threshold. This ambiguity should be resolved.
  3. [Section 2] Reference [18] contains a DOI ('10.1109/TCYB.2021.1234567') that appears to be a placeholder; please verify all references.
  4. [Conclusion] There is a typo: 'As as future work' should read 'As future work.'
  5. [General] The paper would benefit from a reproducibility statement and, if possible, a link to the code and datasets, since the experimental details are otherwise difficult to verify.

Circularity Check

1 steps flagged · score 8.0 of 10

Evaluation leakage: the decision threshold is adapted and tested on the same 500-email dataset, so the reported 79.8% and 67.2% accuracies are in-sample and do not support the central claim.

  1. fitted input called prediction [Section 3.2 (Phase 2: Adapting) and Section 4.1 (Dataset Overview)]
    "After analyzing the semantic similarities, a threshold value is defined for the dataset and a keyword list by heuristically examining breakpoints about relation between threshold increase and success rate gain to determine whether the corresponding similarity refers to ’phishing email’ or ’safe email’. ... The same dataset, which is the dataset for adapting the threshold, is used for testing."

    The decision threshold is selected by heuristically examining success-rate breakpoints on the same 500-email dataset on which the accuracy is later reported. Because the threshold is a fitted parameter tuned to maximize success on the test data, the reported accuracies (79.8% TF-IDF, 67.2% semantic) are measures of fit to the threshold choice rather than estimates of performance on unseen emails. The paper explicitly states the adapting dataset and testing dataset are the same, so the evaluation is in-sample by construction.

full rationale

The central empirical claim of the paper is that the proposed NLP detector reaches 79.8% accuracy with TF-IDF and 67.2% with semantic analysis. The load-bearing evaluation step in Section 3.2 chooses a threshold on a dataset by examining success-rate gains, and Section 4.1 states that the same dataset is used for testing. Thus the threshold is effectively fitted to the test labels, and the reported accuracies are computed on the very data used to select that decision boundary. This makes the reported figures in-sample fits rather than predictions on unseen emails. The paper is transparent about this design, but transparency does not correct the leakage: without a held-out set, cross-validation, or any separation between threshold adaptation and testing, the headline numbers do not estimate generalization. With a balanced 250/250 phishing/safe split, the chance baseline is 50%, and the reported margin could shrink or vanish on genuinely new data. The abstract and conclusion present these accuracies without this qualifier, so the central claim is not supported by the evaluation as designed.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a heuristic threshold fitted to the test set, an undisclosed K-means cluster count, and keyword-count sweeps in which the best result is reported as the headline. The domain assumptions concern dataset representativeness and the semantic link between keyword similarity and phishing intent.

free parameters (3)
  • Decision threshold = Not disclosed
    Chosen heuristically by examining breakpoints between threshold increase and success rate gain on the same dataset used for testing (Section 3.2, Section 4.1).
  • Number of K-means clusters = Not disclosed
    Controls keyword extraction in Phase 1; the paper states 'One variable controls the number of clusters' without specifying its value.
  • Number of keywords = 6, 8, 10, 11, 12, 16, 24, 34 (semantic); 6, 10, 12, 14, 16, 19, 20, 34 (TF-IDF)
    The experiments sweep this parameter and report the maximum accuracy as the headline result.
assumptions (3)
  • domain assumption The 190-email phishing dataset is representative enough to derive generalizable keywords.
    Keyword extraction in Phase 1 relies entirely on this dataset (Section 4.1); no analysis of diversity or coverage is provided.
  • domain assumption Cosine similarity between Universal Sentence Encoder embeddings and the keyword list captures phishing intent.
    The method assumes semantic proximity to phishing keywords indicates a phishing email (Section 3.2), without validating this mapping on independent data.
  • domain assumption The 500-email dataset is representative of real-world benign and phishing emails in financial systems.
    The test set is balanced 50/50 and its source is unspecified (Section 4.1), so the class distribution and content may not match deployment conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Phishing Detection in Financial Systems through NLP." pith.science (2026). https://pith.science/paper/2V3VIUAE

@misc{pith2026250704426,
  author       = {Pith},
  title        = {Pith review of: Enhancing Phishing Detection in Financial Systems through NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2V3VIUAE}},
  note         = {Machine review of arXiv:2507.04426}
}
read the original abstract

The threat of phishing attacks in financial systems is continuously growing. Therefore, protecting sensitive information from unauthorized access is paramount. This paper discusses the critical need for robust email phishing detection. Several existing methods, including blacklists and whitelists, play a crucial role in detecting phishing attempts. Nevertheless, these methods possess inherent limitations, emphasizing the need for the development of a more advanced solution. Our proposed solution presents a pioneering Natural Language Processing (NLP) approach for phishing email detection. Leveraging semantic similarity and TFIDF (Term Frequency-Inverse Document Frequency) analysis, our solution identifies keywords in phishing emails, subsequently evaluating the semantic similarities with a dedicated phishing dataset, ultimately contributing to the enhancement of cybersecurity and NLP domains through a robust solution for detecting phishing threats in financial systems. Experimental results show the accuracy of our phishing detection method can reach 79.8 percent according to TF-IDF analysis, while it can reach 67.2 percent according to semantic analysis.

Figures

Figures reproduced from arXiv: 2507.04426 by the authors.

Figure 1
Figure 1. Moreover, 6.3% of all attacks targeted e-commerce and retail sector. In this research, we primary consider [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Most targeted industry sectors by phishing attacks [1]. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A phishing attacks life cycle [2]. Since phishing attacks may have consequences, phishing detection is a significant research area. In general there are two approaches to detect phishing, namely list based and machine learning based. For instance, a blacklist contains malicious URLs, and whether the site to be entered is reliable or cannot be detected by comparing it with the URLs in the list. The whitelist includes… view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: Machine learning for phishing attack detection [10]. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 4
Figure 4. Figure 4: Semantic heat map. The main logic of phase one involves reading a CSV file, combining all text from a specified column into a single document, and getting embedding. After obtaining vectors of each unique lemmatized word, K-means clustering is 5 [PITH_FULL_IMAGE:figur…
Figure 5
Figure 5. Figure 5: Define keyword list using similarity algorithms. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Find threshold values using keyword lists. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Find accuracy rates using keyword lists and threshold values. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Dataset used for evaluation process of our model, 500 emails. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Output for semantic model. The significance of optimizing the trade-off between comprehensiveness and relevance in keyword extraction is highlighted by the semantic model’s sensitivity to the quantity of keywords. A greater number of keywords may be able to catch more …
Figure 10
Figure 10. Figure 10: Output for non-semantic model. The resilience of the non-semantic model is suggested by its ability to sustain high success rates across exper￾iments, even with different keyword numbers, which is especially beneficial in real-world applications. The non￾semantic meth…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 18 canonical work pages

  1. [1]

    Rep., 2014

    APWG, ”PHISHING ACTIVITY TRENDS REPORT 2Q 2023,” Anti-Phishing Working Group, Tech. Rep., 2014. [Online]. Accessed: Dec. 8, 2023. Available: https://docs.apwg.org/reports/apwg trends report q2 2023.pdf

    work page 2023

  2. [2]

    A. K. Jain and B. B. Gupta, ”A novel approach to protect against phishing attacks at client side using auto-updated white-list,” EURASIP J. on Info. Security, vol. 9, pp. 2016, 2016. doi: https://doi.org/10.1186/s13635-016-0034-3

    work page doi:10.1186/s13635-016-0034-3 2016

  3. [3]

    T. Li, A. Mehta and P. Yang, ”Security Analysis of Email Systems,” 2017 IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud), New York, NY , USA, 2017, pp. 91-96, doi: 10.1109/CSCloud.2017.20

    work page doi:10.1109/cscloud.2017.20 2017

  4. [4]

    Megaw and S

    G. Megaw and S. V . Flowerday, ”Phishing within e-commerce: A trust and confidence game,” 2010 Information Security for South Africa, Johannesburg, South Africa, 2010, pp. 1-8, doi: 10.1109/ISSA.2010.5588333

    work page arXiv 2010

  5. [5]

    M. C. Scheau, A.-L. Arsene, and G. Dinca, ”Phishing and E-Commerce: An Information Security Management Problem,” Journal of Defense Resources Management, vol. 7, no. 1, pp. 129-140, 2016

    work page 2016

  6. [6]

    Skula and M

    I. Skula and M. Kvet, ”Domain Blacklist E fficacy for Phishing Web-page Detection Over an Extended Time Period,” 2023 33rd Conference of Open Innovations Association (FRUCT), Zilina, Slovakia, 2023, pp. 257-263, doi: 10.23919/FRUCT58615.2023.10142999

    arXiv 2023

  7. [7]

    Sharifi and S

    M. Sharifi and S. H. Siadati, ”A phishing sites blacklist generator,” 2008 IEEE /ACS International Conference on Computer Systems and Applications, Doha, Qatar, 2008, pp. 840-843, doi: 10.1109/AICCSA.2008.4493625

    work page arXiv 2008

  8. [8]

    N. A. Azeez, S. Misra, I. A. Margaret, L. Fernandez-Sanz, and S. M. Abdulhamid, ”Adopting automated whitelist approach for detecting phishing attacks,” Computers & Security, vol. 108, pp. 102328, Sep. 2021. doi: 10.1016/j.cose.2021.102328

    work page arXiv 2021

Show all 23 references
  1. [9]

    Y . Cao, W. Han, and Y . Le, ”Anti-phishing based on automated individual white-list,” in Proceedings of the 4th ACM workshop on Digital identity management (DIM ’08), New York, NY , USA, 2008, pp. 51-60. doi: 10.1145/1456424.1456434

    2008

  2. [10]

    Basit, M

    A. Basit, M. Zafar, X. Liu, A. Javed, Z. Jalil, and K. Kifayat, ”A comprehensive survey of AI-enabled phishing attacks detection techniques” Telecommun Syst 76, 2021, pp. 139–154, doi: 10.1007/s11235-020-00733-2

    2021 doi

  3. [11]

    A. K. Dutta, ”Detecting phishing websites using machine learning technique,” PLoS ONE, vol. 16, no. 10, p. e0258361, 2021. doi: 10.1371/journal.pone.0258361

    2021 doi

  4. [12]

    Khonji, Y

    M. Khonji, Y . Iraqi and A. Jones, ”Phishing Detection: A Literature Survey,” in IEEE Communications Surveys & Tutorials, vol. 15, no. 4, pp. 2091-2121, Fourth Quarter 2013, doi: 10.1109/SURV .2013.032213.00009

    2013

  5. [13]

    Wei and Y

    Y . Wei and Y . Sekiya, ”Sufficiency of Ensemble Machine Learning Methods for Phishing Websites Detection,” in IEEE Access, vol. 10, pp. 124103-124113, 2022, doi: 10.1109/ACCESS.2022.3224781

    2022

  6. [14]

    Smeuninx, B

    N. Smeuninx, B. De Clerck, and W. Aerts, ”Measuring the Readability of Sustainability Reports: A Corpus-Based Analysis Through Standard Formulae and NLP,” International Journal of Business Communication, vol. 57, no. 1, pp. 52-85, 2020. doi: 10.1177/2329488416675456

    2020 doi

  7. [15]

    Patil, S

    R. Patil, S. Boit, V . Gudivada and J. Nandigam, ”A Survey of Text Representation and Embedding Techniques in NLP,” in IEEE Access, vol. 11, pp. 36120-36146, 2023, doi: 10.1109/ACCESS.2023.3266377

    2023

  8. [16]

    J. Kim, S. Hur, E. Lee, S. Lee and J. Kim, ”NLP-Fast: A Fast, Scalable, and Flexible System to Accelerate Large-Scale Heterogeneous NLP Models,” 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT), Atlanta, GA, USA, 2021, p.86, doi: 1...

    2021

  9. [17]

    Patil, M

    K. Patil, M. Badamikar and S. Sonawane, ”NLP based Text Summarization of Fintech RFPs,” 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS), Erode, India, 2023, p.869, doi: 10.1109/ICSCDS56580.2023.10104748

    2023

  10. [18]

    Salloum, T

    S. Salloum, T. Gaber, S. Vadera, and K. Shaalan, ”Phishing Email Detection Using Natural Language Processing Techniques: A Literature Survey,” IEEE Transactions on Cybersecurity, vol. 189, pp. 19-28, 2021. doi: 10.1109/TCYB.2021.1234567

    2021

  11. [19]

    Asiri, Y

    S. Asiri, Y . Xiao, S. Alzahrani, S. Li and T. Li, ”A Survey of Intelligent Detection Designs of HTML URL Phishing Attacks,” in IEEE Access, vol. 11, p.6422, 2023, doi: 10.1109/ACCESS.2023.3237798

    2023

  12. [20]

    Zieni, L

    R. Zieni, L. Massari and M. C. Calzarossa, ”Phishing or Not Phishing? A Survey on the Detection of Phishing Websites,” in IEEE Access, vol. 11, pp. 18503, 18515, 2023, doi: 10.1109/ACCESS.2023.3247135

    2023

  13. [21]

    Cer et al., ”Universal Sentence Encoder,” arXiv preprint, 2018, pp

    D. Cer et al., ”Universal Sentence Encoder,” arXiv preprint, 2018, pp. 1-7. doi: 10.48550 /arXiv.1803.11175

    2018 doi arXiv

  14. [22]

    Sharma and M

    T. Sharma and M. Bashir, ”An Analysis of Phishing Emails and How the Human Vulnerabilities are Exploited,” in Advances in Intelligent Systems and Computing, 2020, pp. 7-12. doi: 10.1007/978-3-030-52581-1 7

    2020 doi

  15. [23]

    26, 2023

    ”Phishing Datasets,” Kaggle, Nov. 26, 2023. [Online]. Accessed: Dec. 10, 2023. Available: https://www.kaggle.com/datasets/dineshsathya/phishing-datasets/data. 11

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.