{"id":"380e3793-51ef-4a6e-9a80-e2f03ef461ea","arxiv_id":"2411.16751","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that ranks phishing URL classifiers by averaging published accuracy values, with naive Bayes lowest and random forest highest, plus an unvalidated proposal for a two-stage RF-CNN detector.","lead":"This survey compares machine learning and deep learning classifiers for phishing detection by averaging reported accuracies from prior studies. It finds random forest performs best and naive Bayes worst, then proposes an untested two-stage detector without validating it.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rank ordering in Section V is not supported because Table III averages accuracy over heterogeneous datasets, feature types, and study counts with no paired comparison or variance control; the CNN-vs-XGBoost ordering flips if a single outlier is removed.","rationale":"The reader's weakest assumption correctly identifies the load-bearing weakness: the paper pools accuracy numbers from heterogeneous studies and averages them without statistical controls, then uses those means to produce the headline classifier ranking. My stress-test confirms this is not a stylistic objection but a correctness risk for the central claim. The paper is transparent about some of these issues — Section IV explicitly notes that accuracy is affected by dataset quality, imbalance, and preprocessing — but transparency does not make the pooling valid. The strongest claim depends on the means in Table III being comparable across studies, and the table itself contains direct evidence that they are not: variable numbers of entries per classifier, mixed feature sources (URL vs. content vs. images), mixed dataset provenance, and at least one extreme outlier that changes the ordering when removed. I credit the paper for acknowledging the limitations of accuracy as a metric and for not pretending to have formal verification or a parameter-free derivation, but the empirical ranking is the core contribution and it is not robust under a simple sensitivity check. A paired within-study reanalysis would settle the question: if the ranking survives pairwise comparisons on shared datasets, the qualitative conclusion is plausible; if not, the reported order is an artifact of which studies were included. Because this concern reinforces the reader's rejection rather than changing it, the verdict should remain unchanged.","tokens_in":17681,"tokens_out":4073,"duration_ms":38841,"concrete_test":"Recompute the Table III ranking using only paired within-study observations: for every primary source that reports accuracy for at least two of {Naive Bayes, SVM, Random Forest, Decision Tree, CNN, XGBoost, RNN} on the same dataset and protocol, compute per-study differences between each pair (e.g., RF_i - NB_i). Then test whether the mean paired differences are positive and whether a sign test rejects equality at alpha = 0.05. Also recompute the means restricted to studies using the UCI phishing dataset only. If Random Forest no longer beats Naive Bayes in the paired analysis, or if XGBoost and CNN flip when the outlier [47] is excluded, the Section V ranking is not robust and the rejection stands; if the paired ordering matches Table III, the averaging concern is substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V's central claim — Random Forest, Decision Tree, CNN, and XGBoost are the top 4, while Naive Bayes, SVM, and RNN are the worst 3 for URL-property phishing detection — is derived from the arithmetic means in Table III. That derivation is only valid if the accuracy numbers being averaged are commensurable. They are not. (1) The primary studies use different datasets, ranging from public UCI data (e.g., [61]) to internally generated or non-public data (e.g., [53], [60] in Table I); a 95% accuracy on a small clean dataset is not directly comparable to 90% on a larger noisier one. (2) Class balance and preprocessing differ; Section IV itself lists imbalance, bias, and null-value handling as accuracy distorters, yet no correction is applied before pooling. (3) The claim is specifically about URL properties, but several CNN/RNN entries are trained on HTML/content/image inputs (e.g., [19], [77], [86]), so the comparison mixes feature types. (4) Each mean has different support: Naive Bayes has 15 entries, SVM 12, Random Forest 13, CNN 9, XGBoost 11; a single outlier can move the ranking. For instance, the XGBoost mean (94.15) is dragged down by the 70.34 entry from [47]; dropping that one study raises XGBoost to about 96.5, above CNN (94.2). No confidence intervals, significance tests, or dataset-stratified analyses are reported. Therefore the precise ordering in Section V is an artifact of an unvalidated pooling rule, not an established empirical finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper surveys machine learning and deep learning classifiers for phishing detection, grouping methods into Naive Bayes variants, non-Bayesian classifiers, and deep learning models. It aggregates published accuracy results in Tables II and III and computes mean accuracies per classifier. The central empirical claim, stated in Section V, is that Random Forest, Decision Tree, CNN, and XGBoost have the top four mean accuracies (97.1%, 95.2%, 94.2%, 94.1%) for URL-property-based phishing detection, while Naive Bayes, SVM, and RNN have the worst three (80.4%, 89.4%, 91.6%). The paper also proposes a two-stage prediction model (Random Forest followed by CNN) and a regularized variant of the Bayes rule to improve Naive Bayes performance.","tokens_in":18046,"tokens_out":1983,"duration_ms":20317,"significance":"If the ranking were valid, it would provide useful guidance for practitioners selecting classifiers for URL-based phishing detection, and the two-stage proposal would be a plausible direction for future work. The paper also makes a fair point that accuracy alone is not a sufficient evaluation metric and that dataset imbalance, preprocessing differences, and heterogeneous data sources complicate any comparison across studies. However, the central contribution is the ranking in Table III and Section V, and that ranking rests on a pooling methodology that the authors themselves acknowledge, in Section IV, is threatened by the very factors they list. The paper does not provide machine-checked proofs or reproducible code; its empirical content is a hand-aggregated table of published numbers without statistical controls. The survey also usefully catalogs limitations of existing URL-based detectors, especially the vulnerability to so-called friendly URLs and the false-positive problem for newly registered legitimate domains.","major_comments":[{"comment":"The central ranking is derived by averaging accuracy values from heterogeneous studies without establishing commensurability. The primary studies differ in dataset source and size (e.g., public UCI data in [61] versus internally generated or non-public data in [53] and [60]), in class balance, in preprocessing (removal vs. replacement of null values), and in the exact task formulation. Section IV explicitly lists imbalance, bias, preprocessing, and author error as accuracy distorters, yet no correction, stratification, or sensitivity analysis is applied before pooling. The resulting means are therefore not a defensible basis for the claim that Random Forest, Decision Tree, CNN, and XGBoost are the top four and Naive Bayes, SVM, and RNN the worst three. A single outlier can change the ordering: for XGBoost, dropping the 70.34 value from [47] raises its mean from about 94.2 to about 96.5, above CNN. No confidence intervals or significance tests are reported, so the precise ordering in Section V is unsupported.","section":"Table III and Section V"},{"comment":"The claim in Section V is specifically about URL properties-based phishing detection, but several deep learning entries included in the averaged rows are not trained on URL-only features. For example, [19] uses HTML and text obtained from web pages, [86] and [77] involve image or content-based inputs, and [8] uses a broader deep-learning pipeline. Mixing feature types invalidates the comparison across classifier categories, because the performance differences may reflect input modalities rather than the classifier family. The table would need to separate URL-only studies from content/image-based studies before any classifier ranking can be drawn.","section":"Table III, CNN/RNN rows"},{"comment":"The proposed two-stage model (Random Forest on URL properties, then CNN on web-scraped content) is motivated by the aggregated ranking, so it inherits the pooling problem. The choice of Random Forest for the first stage is justified by a mean accuracy of 97% from Table III, but if that mean is not commensurable, the design rationale is not established. The proposal is also not evaluated in any experiment; it is a suggestion rather than a validated contribution. The paper would need at least a small pilot evaluation or a clearly stated feasibility argument to make this a load-bearing part of the survey.","section":"Section IV and Section V, two-stage proposal"},{"comment":"The text says mean accuracy was adopted 'to counter the effect of' uncertainty in dataset quality, imbalance, preprocessing, and author error. Averaging does not counter these effects; it merely propagates them into a single number and then treats dissimilar numbers as comparable. The manuscript contains its own admission that accuracy alone is not a perfect metric and that imbalance can 'tilt the accuracy in favor or against a classifier.' This is an internal inconsistency: the acknowledged limitations are not addressed by the chosen methodology, and no alternative analysis (e.g., stratified means by dataset, paired comparisons within studies that report multiple classifiers, or rank-based aggregation) is provided.","section":"Section IV, mean accuracy rationale"},{"comment":"The survey aggregates values from many independent studies, which is appropriate, but two of the author's own prior papers ([35] on multinomial Naive Bayes and [36] on Bayesian variants for network intrusion detection) are cited in Section III as evidence for the limitations of Naive Bayes. Those papers are not phishing detection studies, so their inclusion in the narrative about phishing performance is not relevant, and their use in the motivation for the proposed regularized Bayes rule gives the proposal a weaker evidential base than the text suggests. This is a minor self-citation concern rather than a circularity problem, but it should be cleaned up.","section":"Table III and references [35], [36]"}],"minor_comments":[{"comment":"The abstract contains an ungrammatical sentence: 'we also made a series of proposals on how the performance of the under-performing algorithm can improved' should read 'can be improved.' The Introduction also repeats several statistics with obvious inconsistencies (e.g., 4.8 million vs. 1.6 million vs. 4.7 billion dollars in adjacent sentences); these should be reconciled against the cited sources.","section":"Abstract and Introduction"},{"comment":"There are unresolved citation placeholders such as ' [ ?]' in the Introduction and ' [ ?]' in the CNN subsection. Figure 1 is said to show phishing statistics from 2013 Q3 to 2022 Q3, but the reader is never told what the plotted quantity is or where the data come from.","section":"Figure 1 and reference placeholders"},{"comment":"The classification of URL properties is confusing. The text first lists length of URL, hostname length, average word, and character repetition as controllable by attackers, then later proposes using 'Uncontrollable properties like the length of the URL, length of the hostname, average URL, longest word, character repetition...' The two lists are nearly identical, so the distinction between controllable and uncontrollable properties is not made clear. Please revise the terminology and the example properties so the intended contrast is precise.","section":"Section IV, controllable/uncontrollable properties"},{"comment":"The description of SVM states that two theories must hold before the suitability of SVM can be determined, namely high-dimensional input space and linearly separable categories. This is an oversimplification; SVMs with kernels are specifically designed to handle non-linearly separable data, and the claim as written is misleading. A more standard description would improve the survey's accuracy.","section":"Section III, SVM motivation"},{"comment":"There are numerous typos and formatting errors throughout, including 'Munitinomial Naive Bayes' in the index terms, 'Nave Bayes' in the text, 'UCL' for the UCI dataset in Table I, and inconsistent capitalization (e.g., 'Bayesian' vs. 'Bayes'). The table captions are also inconsistent: Table I is described as 'LIMITATIONS' but contains a mix of summaries and limitations, and Tables II and III use different column layouts for essentially the same information. A careful proofread is needed.","section":"Minor typographical and formatting issues"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is a classifier ranking built from averaged accuracies across heterogeneous studies, and the authors themselves enumerate the factors that make such averaging invalid in Section IV. The ranking is not robust to the removal of a single outlier, as shown by the XGBoost example, and the feature-type mix in the deep learning rows further undermines the comparison. The two-stage model proposal is a reasonable research idea but is not evaluated. Because the load-bearing empirical contribution is unsupported and the paper is framed as a survey rather than as a position paper, I do not see a feasible revision path within the manuscript's current scope that would salvage the ranking. The paper could be restructured as a qualitative survey of limitations and research directions, but that would be a substantially different paper. This is why I recommend reject rather than major revision, despite the existence of some useful observations about URL-based detection vulnerabilities."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: a survey with a plausible high-level message and an unsupported central ranking. The paper catalogs recent phishing detection work, sorts classifiers into Bayesian, non-Bayesian, and deep learning buckets, and offers a sensible distinction between controllable and uncontrollable URL properties. The two-stage RF+CNN detector and the regularized naive Bayes idea are reasonable research directions, but both are only proposals; nothing is implemented or tested.\n\nCredit where due: the literature coverage is broad and recent, and the authors openly acknowledge the field's over-reliance on accuracy as the sole metric, plus dataset imbalance and preprocessing variation as threats to validity. The qualitative conclusion that tree ensembles and deep nets generally beat naive Bayes on URL-based detection is consistent with prior comparative work and probably true.\n\nThe soft spot is load-bearing. Section V's ranking is built by averaging accuracies from studies that used different datasets (UCI, internal, non-public), different preprocessing and class balances, and different feature types. Several CNN/RNN entries are trained on HTML/content or image inputs, not URL properties, so the table does not isolate the URL-property task. Support per mean varies from 9 to 15 studies, and one outlier can flip the ordering—dropping the 70.34 XGBoost entry moves XGBoost above CNN. No confidence intervals, significance tests, or dataset-stratified sensitivity checks are reported. The paper itself lists these problems in Section IV and then proceeds anyway. That makes the precise ranking an artifact of an unvalidated pooling rule, not an empirical finding.\n\nThere are also craft issues: rough prose, missing citations (the [?] after the CNN definition), a duplicate [17] entry in the XGBoost row, and some self-citations that are fine as background but add noise. The self-citations do not drive the ranking.\n\nWho this is for: a practitioner wanting a quick list of recently published phishing detection accuracies, provided they treat the numbers as approximate. The exact ordering should not be trusted.\n\nRecommendation: not a serious research contribution as it stands. If the authors dropped the false-precision ranking and presented the comparison as a qualitative trend, this could be a useful resource. As is, I would desk reject rather than send it to referees.","headline":"A survey with a plausible big-picture conclusion but a central ranking that is not supported by its own meta-analytic method.","tokens_in":18518,"tokens_out":3265,"would_cite":false,"duration_ms":31079,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey of recent phishing-detection studies finds Naive Bayes is the weakest URL-based classifier, with Random Forest and Decision Tree on top.","keywords":["phishing detection","Naive Bayes","Random Forest","deep learning","URL features","accuracy comparison","two-stage detection","survey"],"falsifier":"Compute mean accuracy for each classifier using only studies that share one public dataset and identical preprocessing, and check whether Naive Bayes remains the lowest and Random Forest the highest; if the order changes or the margin collapses, the paper's ranking is an artifact of averaging incomparable studies.","tokens_in":17542,"feed_emoji":"🎣","tokens_out":5171,"duration_ms":46499,"temperature":0.7,"pith_summary":"The paper sets out to settle a practical question: among current machine-learning and deep-learning classifiers trained on URL properties, where does Naive Bayes actually stand? By pooling the accuracy reported in roughly five years of phishing-detection studies and averaging per classifier, it claims that Naive Bayes is the weakest of the ten methods compared, with mean accuracy around 80.4%, while Random Forest leads at about 97.1%. A sympathetic reader would care because Naive Bayes is a common lightweight baseline in security products, and a 16-point average gap is large enough to change deployment choices. The paper also argues that URL-only models are fundamentally limited because attackers can craft 'friendly' URLs that evade controllable URL features, and it proposes a two-stage detector plus a regularized Bayes variant to close the gap.","feed_headline":"Naive Bayes trails tree models by 17 points on phishing URLs","feed_subtitle":"Survey of recent studies ranks Random Forest, Decision Tree, CNN, and XGBoost ahead on URL-based detection.","key_machinery":"The argument is carried by mean-accuracy aggregation: the paper converts each cited study's reported accuracy into a per-classifier average, treating those numbers as commensurable evidence of real-world performance. Around that device it builds a second distinction, controllable versus uncontrollable URL properties, which explains why URL-trained models fail against attacker-crafted URLs and against young legitimate domains. The proposed two-stage pipeline (Random Forest for URL screening, CNN for page content) and the regularized Bayes rule are the paper's forward-looking mechanisms.","core_discovery":"The central claim is a performance ranking for URL-properties-based phishing detection, computed as mean accuracy across recent studies: Random Forest (97.1%), Decision Tree (95.2%), CNN (94.2%), and XGBoost (94.1%) are the top four, while Naive Bayes (80.4%), SVM (89.4%), and RNN (91.6%) are the bottom three. The paper further claims the poor showing of Naive Bayes follows from its independence assumption, which rarely holds for URL features, and that both Bayesian and non-Bayesian URL-based models share a vulnerability to 'friendly' URLs that defeat controllable URL properties. It concludes with two remedies: a two-stage model in which Random Forest screens the URL and, if the site looks legitimate, a CNN classifies scraped page content; and a regularized Bayes rule that accounts for feature correlation and distribution shape.","pith_inferences":["If the pooled accuracies are distorted by uneven dataset difficulty, the absolute gap may not transfer to a new deployment even if the ordinal ranking does.","A natural testable extension is to run all ten classifiers on a single shared phishing URL dataset with the same feature set and class balance; that would separate classifier ability from study-level confounds.","The paper's controllable-versus-uncontrollable URL distinction suggests a concrete adversary model: an attacker who edits URL length, hostname, and path tokens can evade models trained on those features, which may explain why high laboratory accuracy has not stopped phishing.","The regularized Bayes proposal points toward a family of correlation-aware Bayes variants; their success would depend on whether the added correlation terms remain cheap enough for real-time URL screening."],"forward_implications":["If the ranking holds, teams building URL-only phishing detectors should prefer Random Forest or XGBoost over Naive Bayes as the first-line classifier.","A two-stage design that checks the URL first and scrapes page content only when the URL looks benign could cut false positives on newly registered legitimate sites.","Regularizing the Bayes rule to account for correlated features would give Naive Bayes variants a path to competitiveness without abandoning their simplicity.","URL-only detection is unlikely to be sufficient on its own: attacker-controlled URL properties can be manipulated, so content and image signals are needed.","Because accuracy alone can be inflated by imbalanced data, the paper's choice of mean accuracy across studies is itself a pragmatic compromise."],"supporting_citations":[{"why":"Supplies accuracy values for Naive Bayes, SVM, Random Forest, XGBoost, KNN, and Logistic Regression used in the pooled comparison.","marker":"[17]"},{"why":"Contributes the Naive Bayes, ANN, SVM, KNN, and tree-classifier accuracies pooled in Table III.","marker":"[67]"},{"why":"Provides the Random Forest and other classifier results, including the 97.1% figure cited for Random Forest.","marker":"[61]"},{"why":"Supplies the hybrid decision-tree, logistic-regression, and SVM results that populate the non-Bayesian column.","marker":"[47]"},{"why":"Contributes machine-learning and deep-learning accuracy values used in the categorical comparison.","marker":"[12]"},{"why":"Adds Naive Bayes, Random Forest, XGBoost, and KNN accuracy values to the pooled means.","marker":"[6]"},{"why":"Provides deep-learning and URL-based detection accuracies that feed the CNN and RNN averages.","marker":"[8]"},{"why":"Contributes CNN and attention-based RNN accuracy numbers used in the deep-learning comparison.","marker":"[34]"}],"fun_headline_variants":["Naive Bayes worst in phishing URL detection survey","Random Forest tops phishing URL detector survey","Survey: Naive Bayes lags 17 points behind tree models","Phishing URL detection: trees beat Bayes, deep nets mid-table","Two-stage model proposed to boost naive Bayes phishing defense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes that accuracy numbers reported in different studies, on different datasets with different preprocessing and class balances, can be averaged as if they measured the same thing.","fun_headline_variants_meta":{"raw":{"variants":["Naive Bayes worst in phishing URL detection survey","Random Forest tops phishing URL detector survey","Survey: Naive Bayes lags 17 points behind tree models","Phishing URL detection: trees beat Bayes, deep nets mid-table","Two-stage model proposed to boost naive Bayes phishing defense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1320,"prompt_tokens":995,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":611,"tokens_out":325,"duration_ms":3817,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:57:51.778252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute mean accuracy for each classifier using only studies that share one public dataset and identical preprocessing, and check whether Naive Bayes remains the lowest and Random Forest the highest; if the order changes or the margin collapses, the paper's ranking is an artifact of averaging incomparable studies.","supporting_citations":[{"cited_title":"A high-accuracy phishing website detection method based on machine learning","cited_arxiv_id":null,"evidence_quote":"Supplies accuracy values for Naive Bayes, SVM, Random Forest, XGBoost, KNN, and Logistic Regression used in the pooled comparison."},{"cited_title":"Classification of phishing websites using machine learning models","cited_arxiv_id":null,"evidence_quote":"Contributes the Naive Bayes, ANN, SVM, KNN, and tree-classifier accuracies pooled in Table III."},{"cited_title":"Comparative study of machine learning algorithms for phishing website detection","cited_arxiv_id":null,"evidence_quote":"Provides the Random Forest and other classifier results, including the 97.1% figure cited for Random Forest."},{"cited_title":"Phishing detection system through hybrid machine learning based on url","cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid decision-tree, logistic-regression, and SVM results that populate the non-Bayesian column."},{"cited_title":"Detecting phishing domains using machine learning","cited_arxiv_id":null,"evidence_quote":"Contributes machine-learning and deep-learning accuracy values used in the categorical comparison."},{"cited_title":"Phishing url detection via cnn and attention-based hierarchical rnn","cited_arxiv_id":null,"evidence_quote":"Contributes CNN and attention-based RNN accuracy numbers used in the deep-learning comparison."}],"review_version":1}