Phishing URL Detection using Bi-LSTM

Sneha Baskota

REVIEW 4 major objections 6 minor 12 references

Phishing URL Detection using Bi-LSTM

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A character-level Bi-LSTM classifies URLs into four threat categories with about 97–98 percent accuracy, the paper argues.

desk verdict The confusion matrix is internally impossible, so the 98% accuracy claim is unsupported; the paper is a routine Bi-LSTM application with no reproducible evaluation. read the letter →

arxiv 2504.21049 v1 pith:GGTV743B submitted 2025-04-29 cs.CR cs.AI

Sneha Baskota This is my paper · ORCID

classification cs.CRcs.AI

keywords phishingdetectionBi-LSTMURLclassificationdeeplearningcharacter-leveltokenizationmulticlassmaliciousURLscybersecurity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a Bidirectional Long Short-Term Memory network reading URLs character by character—forward and backward—can sort URLs into benign, phishing, defacement, and malware categories with roughly 97 to 98 percent accuracy on a dataset of over 650,000 URLs (428,103 benign, 94,110 phishing, 96,456 defacement, 32,520 malware). The claim matters because phishing detection must keep false positives low while covering attack types beyond classic phishing, and a character-level sequence model removes the need for hand-crafted URL features. The author reports that phishing detection is the weakest class, with recall of 0.90, and presents the model as a lightweight, deployable alternative to traditional rule-based and heuristic systems.

What carries the argument

The load-bearing object is the Bi-LSTM layer applied to character-level URL sequences: each URL is tokenized into characters, padded to a fixed length, mapped to learned embeddings, and read both forward and backward, with the two hidden-state directions concatenated before a dropout layer and a dense softmax classifier. That bidirectional read is what lets the network use surrounding characters on both sides of a token when deciding whether a substring behaves like a phishing pattern, a defaced page, or a benign address. The trained model is served through a Flask API, but the classification claim rests on the Bi-LSTM sequence model itself.

What would settle it

Recompute the confusion matrix from the reported per-class counts: for any class, the true negatives must equal the number of test URLs not in that class minus the false positives, which here would be tens of thousands, not 25, 78, 5, or 20. If the rows do not sum to a consistent test set, the 98 percent accuracy cannot be reproduced; alternatively, rerun the training on a fixed random 80/20 split and compare per-class precision and recall on the same Kaggle data.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that bidirectional character-level sequence modeling is enough to separate the four URL classes on this dataset. The model treats each URL as a padded sequence of character indices, embeds them, passes the sequence through a Bi-LSTM in both directions, and applies dropout and a softmax output layer. The classification report in Section VI gives an overall accuracy of 98 percent (the abstract states 97 percent), with F1 scores of 0.99 for benign URLs, 0.93 for phishing, 0.99 for defacement, and 0.97 for malware. The paper reads the lower phishing recall (0.90) as the remaining weakness and attributes the overall result to the Bi-LSTM's ability to capture context from both past and future characters, such as suspicious domain structure and malformed URLs.

Load-bearing premise

The entire accuracy claim depends on the test set being a properly held-out random partition of the Kaggle URL dataset with trustworthy labels, but the paper never specifies the split, and the confusion-matrix true-negative counts (25, 78, 5, 20) are too small to be plausible totals for a multiclass test set of that size.

Editorial extensions

If this is right

If the reported accuracy holds on a proper held-out split, a character-level Bi-LSTM offers a URL-only detector that needs no hand-engineered features such as domain age or page rank.
The four-class output extends phishing detection beyond benign-versus-phishing binary decisions, so a single model can flag defacement and malware URLs in the same pass.
The low phishing recall (0.90) implies the model still misses one in ten phishing URLs, so a production system would need a secondary filter or additional features to close that gap.
The Flask deployment shows the trained model can answer classification requests with a confidence score in real time, which is the shape of a practical browser-side or server-side URL checker.
Because the input is just the URL string, the same pipeline can be retrained on fresh labeled URL streams without redesigning the feature set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The confusion-matrix entries in Section VI are internally inconsistent: the true-negative counts of 25, 78, 5, and 20 are far too small to represent the URLs outside each class in a test set of tens of thousands of instances, which suggests the matrix and the 98 percent figure may not describe the same evaluation.
If the central claim is right, a natural testable extension is to run the same Bi-LSTM on live, newly registered phishing URLs, since character-level models can overfit to surface patterns in a static Kaggle snapshot; periodic retraining would show how much of the accuracy survives distribution shift.
A hybrid that appends cheap external signals such as domain age, SSL validity, and URL length statistics to the learned character embedding could raise phishing recall above 0.90 without losing the model's lightweight deployment advantages.
The architecture is small enough that a quantized version could run in a browser extension or on a router, turning the Flask demo into an always-on filter that blocks malicious URLs before the page loads.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the headline accuracy isn't believable because the confusion matrix in Section VI is numerically impossible. That is the load-bearing problem. The paper applies a standard Bi-LSTM to four-class URL classification on a Kaggle dataset. The writing is clear and the related work is honestly cited, including several prior Bi-LSTM and attention-based detectors, so the authors don't overstate novelty. The Flask UI is a nice demo but it's packaging, not science.

The evaluation section is where it falls apart. There is no train/test split, no hyperparameters, no error bars, no code or data. More seriously, the confusion matrix entries cannot be correct. For example, True Negatives for benign is reported as 25, but the true positives for phishing, defacement, and malware (16,975 + 19,029 + 6,233 = 42,237) are all non-benign URLs correctly classified, so they are true negatives for benign. The same contradiction appears for every class, and the definitions of false positives and false negatives are swapped. That means the matrix doesn't correspond to any real classifier output, and the 98% accuracy claim has no supporting evidence. The 'significant improvements over traditional techniques' is also just asserted, with no baseline comparison on the same protocol.

There is one genuinely useful thing: a clear example of how to set up a Bi-LSTM for character-level URL classification with a Flask backend. If you are teaching someone the mechanics, this could be a starting point. But as a research paper it adds nothing to the crowded phishing detection literature. I would not cite it.

My recommendation: desk reject. The evaluation would need a complete redo, and given the internal inconsistency, I don't trust the current numbers enough to send it to a referee. If the author redoes the evaluation properly and reports a split, hyperparameters, and error bars, it might be worth a short workshop paper, but not in its present form.

Referee Report

4 major / 6 minor

Summary. The paper proposes a character-level bidirectional LSTM (Bi-LSTM) for classifying URLs into four categories: benign, phishing, defacement, and malware. It describes a Kaggle-sourced dataset of over 650,000 URLs, preprocessing by character tokenization and padding, the Bi-LSTM architecture with embedding and dropout layers, a Flask-based deployment, and an evaluation reporting 97–98% accuracy with precision, recall, and F1 scores. The central claim is that the proposed model achieves high accuracy and significant improvements over traditional phishing detection techniques.

Significance. If the accuracy claim were properly supported, a multiclass URL classifier with 97–98% accuracy would be a useful practical contribution, and the paper's intention to cover defacement and malware in addition to phishing is worthwhile. However, the evaluation as reported is not reproducible: no train/test split, hyperparameters, or code are given, and the sole evaluation artifact, the confusion matrix in Section VI, contains internally inconsistent numbers that cannot correspond to any real classifier output. The claimed accuracy is therefore unsupported. The paper's strengths are limited to a clear problem statement and a straightforward description of a deployable web interface; it provides no machine-checked proofs, reproducible code, or falsifiable benchmarking against baselines.

major comments (4)

[Section VI] The confusion matrix bullet list is numerically impossible. For the benign class, the True Negatives must include all correctly classified non-benign instances, i.e., at least the phishing TP of 16,975 plus the defacement TP of 19,029 plus the malware TP of 6,233, which is 42,237, not the reported 25. The same contradiction occurs for every class (phishing TN=78, defacement TN=5, malware TN=20). Additionally, the definitions of FP and FN are swapped: 'Incorrectly classified defacement URLs as benign' is a false positive for the benign class, not a false negative. Because this matrix is the only direct evidence for the reported accuracy, the central accuracy claim is unsupported.
[Sections II and VI] No train/test split is specified anywhere. The text refers to a 'test set' but never states the split ratio, whether stratification was used, or whether the same Kaggle dataset was used for both training and evaluation. Without a properly held-out partition, the reported accuracy cannot be interpreted as an estimate of generalization performance.
[Section IV] The model configuration is critically underspecified. The paper does not report the maximum URL sequence length, embedding dimension, number of Bi-LSTM hidden units, dropout rate, optimizer, learning rate, batch size, number of epochs, or any regularization or early-stopping criterion. These parameters are essential for replication and for assessing whether the reported result is plausible, so the 98% accuracy claim is not verifiable.
[Abstract, Table I, and Section VII] The reported accuracy is inconsistent: the abstract and Section VII state 97%, while Table I reports 98%. More importantly, the claim of 'significant improvements over traditional techniques' is never substantiated. No baseline algorithm is implemented, no comparative table is provided, and no statistical tests or error bars accompany the results. The improvement is asserted rather than demonstrated.

minor comments (6)

[Abstract] There is a typo in the abstract: 'Experimental results on dataset comprising' should be 'Experimental results on a dataset comprising'.
[Section VI and Figure 2] Figure 2 is referenced as the confusion matrix, but the figure content is not included in the manuscript text; the bullet list is the only source of the matrix numbers, and it is internally inconsistent as noted above.
[Section V] The Flask code snippet in Listing 1 is not runnable as shown because of the line break between 'prediction =' and 'model.predict', and the endpoint returns only a single prediction despite the text claiming it handles multiple URLs simultaneously.
[Section II] The paper does not provide a citation or link for the Kaggle dataset, which prevents verification of the reported class distribution (428,103 benign, 94,110 phishing, 96,456 defacement, 32,520 malware) and of label quality.
[Section VII] The claim that providing confidence scores is 'a new contribution' is overstated; softmax-based confidence scores are standard in deep learning classifiers.
[Section III] The literature review mentions several prior works but does not provide quantitative comparisons with them, making it difficult to place the claimed performance in context.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is a standard empirical ML evaluation; its internal inconsistencies are correctness issues, not circularity.

full rationale

The paper does not attempt a first-principles derivation; it trains a Bi-LSTM on a Kaggle URL dataset and reports classification metrics. There is no step in which a quantity is defined in terms of the very quantity it is supposed to predict, no fitted parameter is relabeled as a prediction, and no load-bearing claim rests on a self-citation. The cited prior work is external and does not supply the paper's core architecture or results. The statement 'achieving 97% accuracy and significant improvements over traditional techniques' is unsupported by a direct baseline comparison, and the confusion matrix in Section VI is arithmetically impossible, but these are failures of evidence and internal consistency rather than circular reasoning. Because the claimed accuracy is an empirical observation on a held-out test split (however poorly documented), it is not circular by construction. Under the hard rules, circularity requires exhibiting a specific reduction of the claimed result to its own inputs, and no such reduction appears in the manuscript. Therefore the appropriate circularity score is 0, with the caveat that the paper's central accuracy claim is not independently verifiable from the information provided.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim rests on an unverified dataset and a standard deep learning pipeline whose hyperparameters and split are not disclosed. No new entities are introduced.

free parameters (6)

Maximum URL sequence length
Padding to a fixed length is mentioned but the value is not given; it affects model capacity and input representation.
Embedding dimension
Not specified; chosen by hand without justification.
Bi-LSTM hidden units
Not specified; determines model capacity.
Dropout rate
Dropout is applied but the rate is not given.
Optimizer, learning rate, batch size, epochs
Training configuration is absent from the paper.
Train/test split ratio
No split specified, yet test metrics are reported.

assumptions (3)

domain assumption URL characters encode enough information to discriminate the four classes
The entire model relies on character-level sequences being predictive; no feature analysis is provided.
domain assumption Kaggle dataset labels are ground truth
No label verification or data provenance details are given; evaluation uses these labels as correct.
standard math Backpropagation and LSTM recurrence are standard and correctly implemented
The deep learning framework is assumed correct without verification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Phishing URL Detection using Bi-LSTM." pith.science (2026). https://pith.science/paper/GGTV743B

@misc{pith2026250421049,
  author       = {Pith},
  title        = {Pith review of: Phishing URL Detection using Bi-LSTM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGTV743B}},
  note         = {Machine review of arXiv:2504.21049}
}

read the original abstract

Phishing attacks threaten online users, often leading to data breaches, financial losses, and identity theft. Traditional phishing detection systems struggle with high false positive rates and are usually limited by the types of attacks they can identify. This paper proposes a deep learning-based approach using a Bidirectional Long Short-Term Memory (Bi-LSTM) network to classify URLs into four categories: benign, phishing, defacement, and malware. The model leverages sequential URL data and captures contextual information, improving the accuracy of phishing detection. Experimental results on a dataset comprising over 650,000 URLs demonstrate the model's effectiveness, achieving 97% accuracy and significant improvements over traditional techniques.

Figures

Figures reproduced from arXiv: 2504.21049 by the authors.

**Figure 1.** Phishing attack process II. DATASET AND PREPROCESSING The dataset used for this project is sourced from Kaggle, which contains a diverse set of URLs classified into four categories: benign, phishing, defacement, and malware. The benign URLs represent safe and legitimate web addresses, while phishing URLs are those designed to deceive users and steal sensitive data. Defacement URLs correspond to sites that have been … view at source ↗

**Figure 2.** Confusion Matrix of the Classification Model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

Phishingrtds: A real-time detection system for phishing attacks using a deep learning model,

S. Asiri, Y . Xiao, S. Alzahrani, and T. Li, “Phishingrtds: A real-time detection system for phishing attacks using a deep learning model,” Computers & Security , vol. 141, p. 103843, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167404824001445

work page 2024
[2]

Phishing url detection with neural networks: an empirical study,

H. Ghalechyan, E. Israyelyan, A. Arakelyan et al. , “Phishing url detection with neural networks: an empirical study,” Scientific Reports, vol. 14, p. 25134, 2024. [Online]. Available: https://doi.org/10.1038/ s41598-024-74725-6

work page 2024
[3]

Enhanced phishing url detection using a novel gru-cnn hybrid approach,

S. M, N. K, S. Ravva, R. Rudra, P. Balaji, and R. T, “Enhanced phishing url detection using a novel gru-cnn hybrid approach,” Journal of Machine and Computing , pp. 089–101, 01 2025

work page 2025
[4]

An integrated csppc and bilstm frame- work for malicious url detection,

J. Zhou, K. Zhang, A. Bilal et al., “An integrated csppc and bilstm frame- work for malicious url detection,” Scientific Reports , vol. 15, p. 6659,

work page
[5]

Phishing url detection using bi-lstm with attention mecha- nism,

K. Srinivasan, R. Prema, G. S. Chauhan, R. Jadon, R. Budda, and V . S. T. Gollapalli, “Phishing url detection using bi-lstm with attention mecha- nism,” in Handbook of Research on Cybersecurity Risk in Contemporary Business Systems, IGI Global, 2025, pp. 159–184. [Online]. Available: https://doi.org/10.4018/979-8-3693-7540-2.ch008

work page doi:10.4018/979-8-3693-7540-2.ch008 2025
[6]

A Sophisticated Framework for the Accurate Detection of Phishing Websites

A. Newaz, F. S. Haq, and N. Ahmed, “A sophisticated framework for the accurate detection of phishing websites,” 2024. [Online]. Available: https://arxiv.org/abs/2403.09735

work page Pith review arXiv 2024
[7]

Url based phishing attack detection using bilstm-gated highway attention block convolutional neural network,

M. Nanda and S. Goel, “Url based phishing attack detection using bilstm-gated highway attention block convolutional neural network,” Multimedia Tools and Applications , vol. 83, pp. 69 345–69 375, 2024. [Online]. Available: https://doi.org/10.1007/s11042-023-17993-0

work page doi:10.1007/s11042-023-17993-0 2024
[8]

Enhancing Phishing Detection through Feature Importance Analysis and Explainable AI: A Comparative Study of CatBoost, XGBoost, and EBM Models

A. Fajar, S. Yazid, and I. Budi, “Enhancing phishing detection through feature importance analysis and explainable ai: A comparative study of catboost, xgboost, and ebm models,” 2024. [Online]. Available: https: //arxiv.org/abs/2411.06860

work page Pith review arXiv 2024

Show all 12 references

[9]

Phishguard: A convolutional neural network based model for detecting phishing urls with explainability analysis,

M. R. Islam, M. M. Islam, M. S. Afrin, A. Antara, N. Tabassum, and A. Amin, “Phishguard: A convolutional neural network based model for detecting phishing urls with explainability analysis,” 2024. [Online]. Available: https://arxiv.org/abs/2404.17960

2024 arXiv
[10]

Phishing url detection using bilstm with attention mechanism,

J. K. S. and B. Arthi, “Phishing url detection using bilstm with attention mechanism,” in Machine Intelligence Applications in Cyber-Risk Man- agement, M. A. Almaiah and Y . Maleh, Eds., IGI Global, 2025, pp. 159–

2025
[184]

Available: https://doi.org/10.4018/979-8-3693-7540-2

[Online]. Available: https://doi.org/10.4018/979-8-3693-7540-2. ch008

doi
[2025]

Available: https://doi.org/10.1038/s41598-025-91148-z

[Online]. Available: https://doi.org/10.1038/s41598-025-91148-z

doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Phishingrtds: A real-time detection system for phishing attacks using a deep learning model,

S. Asiri, Y . Xiao, S. Alzahrani, and T. Li, “Phishingrtds: A real-time detection system for phishing attacks using a deep learning model,” Computers & Security , vol. 141, p. 103843, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167404824001445

work page 2024

[2] [2]

Phishing url detection with neural networks: an empirical study,

H. Ghalechyan, E. Israyelyan, A. Arakelyan et al. , “Phishing url detection with neural networks: an empirical study,” Scientific Reports, vol. 14, p. 25134, 2024. [Online]. Available: https://doi.org/10.1038/ s41598-024-74725-6

work page 2024

[3] [3]

Enhanced phishing url detection using a novel gru-cnn hybrid approach,

S. M, N. K, S. Ravva, R. Rudra, P. Balaji, and R. T, “Enhanced phishing url detection using a novel gru-cnn hybrid approach,” Journal of Machine and Computing , pp. 089–101, 01 2025

work page 2025

[4] [4]

An integrated csppc and bilstm frame- work for malicious url detection,

J. Zhou, K. Zhang, A. Bilal et al., “An integrated csppc and bilstm frame- work for malicious url detection,” Scientific Reports , vol. 15, p. 6659,

work page

[5] [5]

Phishing url detection using bi-lstm with attention mecha- nism,

K. Srinivasan, R. Prema, G. S. Chauhan, R. Jadon, R. Budda, and V . S. T. Gollapalli, “Phishing url detection using bi-lstm with attention mecha- nism,” in Handbook of Research on Cybersecurity Risk in Contemporary Business Systems, IGI Global, 2025, pp. 159–184. [Online]. Available: https://doi.org/10.4018/979-8-3693-7540-2.ch008

work page doi:10.4018/979-8-3693-7540-2.ch008 2025

[6] [6]

A Sophisticated Framework for the Accurate Detection of Phishing Websites

A. Newaz, F. S. Haq, and N. Ahmed, “A sophisticated framework for the accurate detection of phishing websites,” 2024. [Online]. Available: https://arxiv.org/abs/2403.09735

work page Pith review arXiv 2024

[7] [7]

Url based phishing attack detection using bilstm-gated highway attention block convolutional neural network,

M. Nanda and S. Goel, “Url based phishing attack detection using bilstm-gated highway attention block convolutional neural network,” Multimedia Tools and Applications , vol. 83, pp. 69 345–69 375, 2024. [Online]. Available: https://doi.org/10.1007/s11042-023-17993-0

work page doi:10.1007/s11042-023-17993-0 2024

[8] [8]

Enhancing Phishing Detection through Feature Importance Analysis and Explainable AI: A Comparative Study of CatBoost, XGBoost, and EBM Models

A. Fajar, S. Yazid, and I. Budi, “Enhancing phishing detection through feature importance analysis and explainable ai: A comparative study of catboost, xgboost, and ebm models,” 2024. [Online]. Available: https: //arxiv.org/abs/2411.06860

work page Pith review arXiv 2024