{"id":"cb348bed-16f5-474d-827e-22fda55b43a9","arxiv_id":"2504.21049","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A Bi-LSTM model trained on 650,000 URLs is claimed to classify URLs into four categories with roughly 97 percent accuracy, but the supporting evidence is incomplete and internally inconsistent.","lead":"This paper applies a Bidirectional LSTM neural network to classify URLs as benign, phishing, defacement, or malware, claiming about 97 percent accuracy on a Kaggle dataset of over 650,000 URLs. It is a demonstration that a standard deep learning architecture can be packaged into a lightweight web tool for URL screening.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported confusion matrix is numerically impossible, so the 98% accuracy claim is unsupported.","rationale":"The central claim of the paper is that a character-level Bi-LSTM achieves 97–98% accuracy on a four-class URL classification task. The only supporting evidence is the classification report in Table I and the confusion matrix in Section VI. The confusion matrix is not merely incomplete or missing a split; it is arithmetically impossible. Every class's reported true-negative count is far smaller than the minimum number of true negatives implied by the other classes' true positives. Additionally, the paper mislabels false positives and false negatives, so the matrix cannot be mapped to any real classifier output. This is a correctness failure in the core evaluation, not a mere stylistic issue. The reader's weakest_assumption correctly identified the confusion matrix inconsistency and the unspecified test split; I agree with that assessment. The paper also provides no hyperparameters, no code, no data, and no reproducibility artifacts, so even if the matrix were fixed there would be no way to verify the results. The verdict of REJECT is appropriate, and my stress-test does not change it.","tokens_in":4995,"tokens_out":2380,"duration_ms":25292,"concrete_test":"Analytically reconstruct the true confusion matrix from the paper's reported per-class TP/FP/FN/TN values and the class totals. For the benign row, compute the sum of correct non-benign predictions (TP_phishing + TP_defacement + TP_malware = 42,237); if this exceeds the reported benign TN of 25, the matrix is impossible. Then request the actual test-set confusion matrix and code from the author; if the original matrix is unavailable or still yields TN=25, the accuracy claim is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation section (Section VI) reports a confusion matrix whose numbers cannot be correct for any multiclass test set. For the benign class, the paper lists True Negatives (TN) = 25, yet the true positives for the other three classes (phishing 16,975, defacement 19,029, malware 6,233) are all correct predictions of non-benign URLs, and so are true negatives for the benign class. Therefore TN for benign must be at least 42,237, not 25. The same internal contradiction appears for every class (phishing TN=78, defacement TN=5, malware TN=20). Moreover, the definitions of FN and FP are swapped: 'Incorrectly classified defacement URLs as benign' is a false positive for benign, not a false negative; 'Incorrectly classified defacement URLs as phishing' is a false positive for phishing, not a false negative. Thus the reported confusion matrix does not correspond to any real classifier output. Because the paper's only evidence for its central accuracy claim is this internally inconsistent matrix, plus a classification report with no split, hyperparameters, code, or raw data, the 97–98% accuracy claim is not credible. The abstract's statement and Table I cannot be validated or trusted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a character-level bidirectional LSTM (Bi-LSTM) for classifying URLs into four categories: benign, phishing, defacement, and malware. It describes a Kaggle-sourced dataset of over 650,000 URLs, preprocessing by character tokenization and padding, the Bi-LSTM architecture with embedding and dropout layers, a Flask-based deployment, and an evaluation reporting 97–98% accuracy with precision, recall, and F1 scores. The central claim is that the proposed model achieves high accuracy and significant improvements over traditional phishing detection techniques.","tokens_in":5298,"tokens_out":3080,"duration_ms":32226,"significance":"If the accuracy claim were properly supported, a multiclass URL classifier with 97–98% accuracy would be a useful practical contribution, and the paper's intention to cover defacement and malware in addition to phishing is worthwhile. However, the evaluation as reported is not reproducible: no train/test split, hyperparameters, or code are given, and the sole evaluation artifact, the confusion matrix in Section VI, contains internally inconsistent numbers that cannot correspond to any real classifier output. The claimed accuracy is therefore unsupported. The paper's strengths are limited to a clear problem statement and a straightforward description of a deployable web interface; it provides no machine-checked proofs, reproducible code, or falsifiable benchmarking against baselines.","major_comments":[{"comment":"The confusion matrix bullet list is numerically impossible. For the benign class, the True Negatives must include all correctly classified non-benign instances, i.e., at least the phishing TP of 16,975 plus the defacement TP of 19,029 plus the malware TP of 6,233, which is 42,237, not the reported 25. The same contradiction occurs for every class (phishing TN=78, defacement TN=5, malware TN=20). Additionally, the definitions of FP and FN are swapped: 'Incorrectly classified defacement URLs as benign' is a false positive for the benign class, not a false negative. Because this matrix is the only direct evidence for the reported accuracy, the central accuracy claim is unsupported.","section":"Section VI"},{"comment":"No train/test split is specified anywhere. The text refers to a 'test set' but never states the split ratio, whether stratification was used, or whether the same Kaggle dataset was used for both training and evaluation. Without a properly held-out partition, the reported accuracy cannot be interpreted as an estimate of generalization performance.","section":"Sections II and VI"},{"comment":"The model configuration is critically underspecified. The paper does not report the maximum URL sequence length, embedding dimension, number of Bi-LSTM hidden units, dropout rate, optimizer, learning rate, batch size, number of epochs, or any regularization or early-stopping criterion. These parameters are essential for replication and for assessing whether the reported result is plausible, so the 98% accuracy claim is not verifiable.","section":"Section IV"},{"comment":"The reported accuracy is inconsistent: the abstract and Section VII state 97%, while Table I reports 98%. More importantly, the claim of 'significant improvements over traditional techniques' is never substantiated. No baseline algorithm is implemented, no comparative table is provided, and no statistical tests or error bars accompany the results. The improvement is asserted rather than demonstrated.","section":"Abstract, Table I, and Section VII"}],"minor_comments":[{"comment":"There is a typo in the abstract: 'Experimental results on dataset comprising' should be 'Experimental results on a dataset comprising'.","section":"Abstract"},{"comment":"Figure 2 is referenced as the confusion matrix, but the figure content is not included in the manuscript text; the bullet list is the only source of the matrix numbers, and it is internally inconsistent as noted above.","section":"Section VI and Figure 2"},{"comment":"The Flask code snippet in Listing 1 is not runnable as shown because of the line break between 'prediction =' and 'model.predict', and the endpoint returns only a single prediction despite the text claiming it handles multiple URLs simultaneously.","section":"Section V"},{"comment":"The paper does not provide a citation or link for the Kaggle dataset, which prevents verification of the reported class distribution (428,103 benign, 94,110 phishing, 96,456 defacement, 32,520 malware) and of label quality.","section":"Section II"},{"comment":"The claim that providing confidence scores is 'a new contribution' is overstated; softmax-based confidence scores are standard in deep learning classifiers.","section":"Section VII"},{"comment":"The literature review mentions several prior works but does not provide quantitative comparisons with them, making it difficult to place the claimed performance in context.","section":"Section III"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as a project report rather than a peer-reviewed research paper. The evaluation section contains an impossible confusion matrix and lacks all experimental details needed for reproducibility. These are load-bearing flaws: the paper's only evidence for its central claim is the unusable evaluation. Even a major revision would require completely re-running the experiments with a proper split and reporting, which is effectively a new submission. The paper is not suitable for publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the headline accuracy isn't believable because the confusion matrix in Section VI is numerically impossible. That is the load-bearing problem. The paper applies a standard Bi-LSTM to four-class URL classification on a Kaggle dataset. The writing is clear and the related work is honestly cited, including several prior Bi-LSTM and attention-based detectors, so the authors don't overstate novelty. The Flask UI is a nice demo but it's packaging, not science.\n\nThe evaluation section is where it falls apart. There is no train/test split, no hyperparameters, no error bars, no code or data. More seriously, the confusion matrix entries cannot be correct. For example, True Negatives for benign is reported as 25, but the true positives for phishing, defacement, and malware (16,975 + 19,029 + 6,233 = 42,237) are all non-benign URLs correctly classified, so they are true negatives for benign. The same contradiction appears for every class, and the definitions of false positives and false negatives are swapped. That means the matrix doesn't correspond to any real classifier output, and the 98% accuracy claim has no supporting evidence. The 'significant improvements over traditional techniques' is also just asserted, with no baseline comparison on the same protocol.\n\nThere is one genuinely useful thing: a clear example of how to set up a Bi-LSTM for character-level URL classification with a Flask backend. If you are teaching someone the mechanics, this could be a starting point. But as a research paper it adds nothing to the crowded phishing detection literature. I would not cite it.\n\nMy recommendation: desk reject. The evaluation would need a complete redo, and given the internal inconsistency, I don't trust the current numbers enough to send it to a referee. If the author redoes the evaluation properly and reports a split, hyperparameters, and error bars, it might be worth a short workshop paper, but not in its present form.","headline":"The confusion matrix is internally impossible, so the 98% accuracy claim is unsupported; the paper is a routine Bi-LSTM application with no reproducible evaluation.","tokens_in":5733,"tokens_out":2404,"would_cite":false,"duration_ms":22001,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A character-level Bi-LSTM classifies URLs into four threat categories with about 97–98 percent accuracy, the paper argues.","keywords":["phishing detection","Bi-LSTM","URL classification","deep learning","character-level tokenization","multiclass classification","malicious URLs","cybersecurity"],"falsifier":"Recompute the confusion matrix from the reported per-class counts: for any class, the true negatives must equal the number of test URLs not in that class minus the false positives, which here would be tens of thousands, not 25, 78, 5, or 20. If the rows do not sum to a consistent test set, the 98 percent accuracy cannot be reproduced; alternatively, rerun the training on a fixed random 80/20 split and compare per-class precision and recall on the same Kaggle data.","tokens_in":4841,"feed_emoji":"🛡️","tokens_out":5734,"duration_ms":57111,"temperature":0.7,"pith_summary":"The paper argues that a Bidirectional Long Short-Term Memory network reading URLs character by character—forward and backward—can sort URLs into benign, phishing, defacement, and malware categories with roughly 97 to 98 percent accuracy on a dataset of over 650,000 URLs (428,103 benign, 94,110 phishing, 96,456 defacement, 32,520 malware). The claim matters because phishing detection must keep false positives low while covering attack types beyond classic phishing, and a character-level sequence model removes the need for hand-crafted URL features. The author reports that phishing detection is the weakest class, with recall of 0.90, and presents the model as a lightweight, deployable alternative to traditional rule-based and heuristic systems.","feed_headline":"Deep learning model sorts malicious URLs with 98% accuracy","feed_subtitle":"A character-level neural network separates benign, phishing, defacement, and malware URLs in one pass.","key_machinery":"The load-bearing object is the Bi-LSTM layer applied to character-level URL sequences: each URL is tokenized into characters, padded to a fixed length, mapped to learned embeddings, and read both forward and backward, with the two hidden-state directions concatenated before a dropout layer and a dense softmax classifier. That bidirectional read is what lets the network use surrounding characters on both sides of a token when deciding whether a substring behaves like a phishing pattern, a defaced page, or a benign address. The trained model is served through a Flask API, but the classification claim rests on the Bi-LSTM sequence model itself.","core_discovery":"On the paper's own terms, the central discovery is that bidirectional character-level sequence modeling is enough to separate the four URL classes on this dataset. The model treats each URL as a padded sequence of character indices, embeds them, passes the sequence through a Bi-LSTM in both directions, and applies dropout and a softmax output layer. The classification report in Section VI gives an overall accuracy of 98 percent (the abstract states 97 percent), with F1 scores of 0.99 for benign URLs, 0.93 for phishing, 0.99 for defacement, and 0.97 for malware. The paper reads the lower phishing recall (0.90) as the remaining weakness and attributes the overall result to the Bi-LSTM's ability to capture context from both past and future characters, such as suspicious domain structure and malformed URLs.","pith_inferences":["The confusion-matrix entries in Section VI are internally inconsistent: the true-negative counts of 25, 78, 5, and 20 are far too small to represent the URLs outside each class in a test set of tens of thousands of instances, which suggests the matrix and the 98 percent figure may not describe the same evaluation.","If the central claim is right, a natural testable extension is to run the same Bi-LSTM on live, newly registered phishing URLs, since character-level models can overfit to surface patterns in a static Kaggle snapshot; periodic retraining would show how much of the accuracy survives distribution shift.","A hybrid that appends cheap external signals such as domain age, SSL validity, and URL length statistics to the learned character embedding could raise phishing recall above 0.90 without losing the model's lightweight deployment advantages.","The architecture is small enough that a quantized version could run in a browser extension or on a router, turning the Flask demo into an always-on filter that blocks malicious URLs before the page loads."],"forward_implications":["If the reported accuracy holds on a proper held-out split, a character-level Bi-LSTM offers a URL-only detector that needs no hand-engineered features such as domain age or page rank.","The four-class output extends phishing detection beyond benign-versus-phishing binary decisions, so a single model can flag defacement and malware URLs in the same pass.","The low phishing recall (0.90) implies the model still misses one in ten phishing URLs, so a production system would need a secondary filter or additional features to close that gap.","The Flask deployment shows the trained model can answer classification requests with a confidence score in real time, which is the shape of a practical browser-side or server-side URL checker.","Because the input is just the URL string, the same pipeline can be retrained on fresh labeled URL streams without redesigning the feature set."],"supporting_citations":[{"why":"Supplies the real-time deep-learning detection system that this paper positions itself against and extends to multiclass output.","marker":"[1]"},{"why":"Provides a BiLSTM-based malicious URL detection framework whose sequence-modeling result this work builds on.","marker":"[4]"},{"why":"Offers a Bi-LSTM with attention baseline for phishing URL detection, the direct architectural relative of this model.","marker":"[5]"},{"why":"Presents the stacking-ensemble binary phishing classifier that this paper extends by adding defacement and malware classes.","marker":"[6]"},{"why":"Introduces a BiLSTM-gated highway attention CNN for URL phishing detection, a strong deep-learning baseline the paper compares with.","marker":"[7]"},{"why":"Reports a 1D CNN with 21 features reaching 99 percent accuracy, the nearest high-accuracy benchmark the paper measures itself against.","marker":"[9]"}],"fun_headline_variants":["Bi-LSTM URL detector reaches 97% accuracy","Character-level Bi-LSTM sorts four URL threat types","Deep learning model identifies phishing URLs with 97% accuracy","Bi-LSTM classifies URL threats at 97% accuracy on 650K samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire accuracy claim depends on the test set being a properly held-out random partition of the Kaggle URL dataset with trustworthy labels, but the paper never specifies the split, and the confusion-matrix true-negative counts (25, 78, 5, 20) are too small to be plausible totals for a multiclass test set of that size.","fun_headline_variants_meta":{"raw":{"variants":["Bi-LSTM URL detector reaches 97% accuracy","Character-level Bi-LSTM sorts four URL threat types","Deep learning model identifies phishing URLs with 97% accuracy","Bi-LSTM classifies URL threats at 97% accuracy on 650K samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1215,"prompt_tokens":810,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":334}},"tokens_in":426,"tokens_out":405,"duration_ms":4675,"temperature":1.0,"reasoning_tokens":334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:30:21.376352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the confusion matrix from the reported per-class counts: for any class, the true negatives must equal the number of test URLs not in that class minus the false positives, which here would be tens of thousands, not 25, 78, 5, or 20. If the rows do not sum to a consistent test set, the 98 percent accuracy cannot be reproduced; alternatively, rerun the training on a fixed random 80/20 split and compare per-class precision and recall on the same Kaggle data.","supporting_citations":[{"cited_title":"Phishingrtds: A real-time detection system for phishing attacks using a deep learning model,","cited_arxiv_id":null,"evidence_quote":"Supplies the real-time deep-learning detection system that this paper positions itself against and extends to multiclass output."},{"cited_title":"An integrated csppc and bilstm frame- work for malicious url detection,","cited_arxiv_id":null,"evidence_quote":"Provides a BiLSTM-based malicious URL detection framework whose sequence-modeling result this work builds on."},{"cited_title":"Phishing url detection using bi-lstm with attention mecha- nism,","cited_arxiv_id":null,"evidence_quote":"Offers a Bi-LSTM with attention baseline for phishing URL detection, the direct architectural relative of this model."},{"cited_title":"A Sophisticated Framework for the Accurate Detection of Phishing Websites","cited_arxiv_id":"2403.09735","evidence_quote":"Presents the stacking-ensemble binary phishing classifier that this paper extends by adding defacement and malware classes."},{"cited_title":"Url based phishing attack detection using bilstm-gated highway attention block convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"Introduces a BiLSTM-gated highway attention CNN for URL phishing detection, a strong deep-learning baseline the paper compares with."},{"cited_title":"PhishGuard: A Convolutional Neural Network Based Model for Detecting Phishing URLs with Explainability Analysis","cited_arxiv_id":"2404.17960","evidence_quote":"Reports a 1D CNN with 21 features reaching 99 percent accuracy, the nearest high-accuracy benchmark the paper measures itself against."}],"review_version":1}