{"id":"d2a01f0b-ab5d-464e-bd01-c7ab3104cd2f","arxiv_id":"2509.08424","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey categorizing phishing webpage detection into URL, content, and visual approaches, with an analysis of research gaps and suggested directions.","lead":"This paper surveys how researchers detect phishing webpages, grouping methods into those that analyze URLs, webpage content, or screenshots. It lists open problems in the field and suggests some fixes, which is useful for people building or evaluating anti-phishing tools.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparative performance claims in §VIII ('RF predominantly favored', 'hybrid outperforms') are unsupported because Table V pools metrics from different datasets, class balances, and protocols without a common baseline.","rationale":"I read the paper as a survey whose contribution is a categorization and an enumeration of research gaps. The abstract promises a 'systematic categorization' and says the study 'underscores current research gaps' and 'contributes valuable insights'. The conclusion (Section VIII) makes two explicit comparative statements: RF outperforms other ML classifiers, and hybrid-based approaches outperform other approaches. These are the most concrete, falsifiable claims in the paper. For these claims to hold, the performance numbers in Table V must be comparable across the cited papers. That condition is not established and is likely false: papers use different datasets (Alexa, PhishTank, Common Crawl, DMOZ, UCI, OpenPhish), different class ratios (some heavily imbalanced), and different splits, and there is no shared evaluation protocol. Moreover, the paper does not report variance or statistical significance. The reader's weakest_assumption identifies exactly this. My independent check of the table confirms the problem: in several entries the paper cited as 'best' for RF is not best in its own comparison (e.g., [111] and [11]). The hybrid claim is even thinner, resting on two papers. Therefore the strongest claim of the paper is weakened, but not the entire survey; a revised version that removes or qualifies these comparative claims and adds a reproducible selection protocol would be of value. Hence I agree with the reader and keep the CONDITIONAL verdict (UNCHANGED).","tokens_in":39303,"tokens_out":4537,"duration_ms":46862,"concrete_test":"Reconstruct Table V from the original cited papers; for each row record the dataset, class distribution, train/test split, and evaluation metric. Then (1) count in how many rows RF has the highest accuracy among the classifiers in that row; if it is not the best in a majority of rows, 'predominantly favoured' is false. (2) Search for any pair of hybrid (e.g., [57] or [58]) and non-hybrid paper that use the same dataset and identical evaluation protocol; if no such pair exists, the claim that hybrid outperforms other approaches cannot be derived from these data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's central contribution is a systematic map of detection techniques and open problems (Abstract; §VII). But its most distinctive claims, in §VIII and §IV.D, are that 'the RF classifier is predominantly favored' and 'the hybrid-based approach outperforms other approaches'. These rest on Table V, which tabulates accuracy/precision/recall/F1 from papers that use different repositories (Alexa, PhishTank, Common Crawl, DMOZ, UCI, OpenPhish), different class proportions (e.g., [73] uses 14,000 phishing vs 1,000 legitimate; [57] uses 2,119 vs 1,407), and different train/test splits. No statistical test or common baseline is provided. Direct inspection undermines the RF claim: in [111] LSTM achieves 98.76% vs RF 93.47%; in [11] LR achieves 98.42% vs RF 97.37%; in [57] PCA-RF achieves 99.55% vs RF 99.31%. Merely counting papers where RF appears does not establish superiority. The hybrid claim is based only on [57] and [58]; neither is compared against non-hybrid approaches on the same data. Thus the load-bearing assumption of cross-study metric comparability is false, and the paper's headline conclusions are not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a survey of phishing webpage detection. It categorizes detection approaches into URL-based, webpage content-based, and visual/hybrid methods; reviews inputs, dataset repositories, feature sets, feature selection algorithms, ML/DL classifiers, and performance metrics; discusses evasion tactics such as compromised domains, URL shortening, adversarial attacks, and LLM-based phishing; and lists open issues with proposed solutions. The contribution is a structured literature synthesis and a research-gap list rather than a new detector.","tokens_in":39636,"tokens_out":6342,"duration_ms":75516,"significance":"If read as a descriptive overview, the paper is useful: it organizes a large body of recent work, covers evolving threats (LLM-generated phishing, adversarial attacks), and provides tables of datasets and features that practitioners will find convenient. The survey does not ship reproducible code, machine-checked proofs, or parameter-free derivations; its value is as a synthesis. However, the paper's headline comparative conclusions are not supported by the evidence it presents. The claims that 'RF classifier is predominantly favored' and that 'hybrid-based approach outperforms other approaches' (Sections IV.D and VIII, Table V) rest on pooled accuracy/precision/recall/F1 numbers obtained on different datasets, class ratios, and evaluation protocols. These conclusions need to be reframed or removed before the survey can be accepted.","major_comments":[{"comment":"The statement that 'the RF classifier is predominantly favored' and 'outperforms other ML classifiers' is a load-bearing conclusion, but Table V pools metrics from papers using different repositories (Alexa, PhishTank, Common Crawl, DMOZ, UCI, OpenPhish), different class proportions (e.g., [73] uses 14,000 phishing vs 1,000 legitimate; [57] uses 2,119 vs 1,407), and different train/test splits. No common baseline or statistical test is provided. Direct inspection of Table V contradicts the superiority claim: in [111], LSTM achieves 98.76% accuracy vs RF 93.47%; in [11], LR achieves 98.42% vs RF 97.37%; in [57], PCA-RF achieves 99.55% vs RF 99.31%; in [77], multiple classifiers outperform RF on some metrics. At most, RF's frequency of use could be reported as a descriptive count, but that requires an explicit count and should be separated from any performance ranking.","section":"§IV.D, Table V, §VIII"},{"comment":"The claim that 'the hybrid-based approach outperforms other approaches' is based only on [57] and [58]. Neither paper compares its hybrid feature combination against URL-only or content-only variants on the same data, so the conclusion 'the research works have proven' (Section III.C) is unsupported. The related survey [145] reaches a similar conclusion, but citing another survey's opinion does not provide the controlled comparison needed. This conclusion should be removed or rephrased as an observation about the two cited papers, not a general performance ordering.","section":"§III.C, §VIII"},{"comment":"The paper positions itself as systematic and claims to make a substantial contribution by contrasting prior surveys in Table VI. However, the symbols used in the table legend (discussed, highlighted, not mentioned) are not rendered in the text, making the comparison impossible to verify. In addition, the selection of surveys is justified only by 'publication records, citation counts' with no explicit inclusion/exclusion criteria or search protocol. Adding a short methodology paragraph and fixing the table symbols is necessary to support the 'systematic' claim.","section":"§VII, Table VI"}],"minor_comments":[{"comment":"Typo: 'Tanco' should be 'Tranco'.","section":"§II.B.1"},{"comment":"Typo in the listed suspicious word: 'siginin' should be 'signin'.","section":"Table II"},{"comment":"'FR rate' appears to be a typo for 'FP rate'.","section":"§V.D"},{"comment":"The many '-' entries make it unclear which metric a value refers to. The table would be much clearer if each metric column were explicitly labeled and every cell had a value or a footnote.","section":"Table V"},{"comment":"DBN is discussed as a DL algorithm, but the same algorithm family is partly listed under neural networks in §II.E. This duplication could be consolidated to avoid confusion.","section":"§II.F"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about cross-study comparability is valid and lands directly on the paper's central comparative claims. The descriptive survey content is largely sound and the paper is salvageable by removing or downgrading the performance ordering claims and reporting usage frequency instead. I do not see grounds for rejection, but the current form should not appear with unsupported headlined conclusions in a serious venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2509.08424. It's a survey of phishing webpage detection, and it's a reasonable map for newcomers, but the headline conclusions about classifier performance don't hold up.\n\nWhat's actually new: not much—the taxonomy (URL, content, visual) and most challenges appear in earlier surveys. What it does well: it covers recent/emerging topics like LLM-phishing, tiny URLs, adversarial attacks, and compromised domains, and it includes a useful table (Table VI) contrasting prior surveys on several attributes. The descriptive summaries of individual papers are broadly faithful, and the authors are honest about dataset imbalance issues. As a reference for someone entering the field, it has value.\n\nThe soft spots are in the comparative claims. The stress-test note is right: Table V pools accuracy/precision/recall/F1 from papers using different repositories (Alexa, PhishTank, Common Crawl, UCI), different class ratios (e.g., 14,000 vs 1,000 in one, 2,119 vs 1,407 in another), and different splits. No common baseline or statistical test is provided. The conclusion that 'RF classifier is predominantly favored' and 'hybrid-based approach outperforms' is not supported. Direct inspection of Table V undermines the RF claim: in [111] LSTM beats RF by 5 points; in [11] LR beats RF by about a point; in [57] PCA-RF beats RF. Counting papers where RF appears is not evidence of superiority. The hybrid claim rests on just [57] and [58], and neither compares against non-hybrid methods on the same data. The paper's own Section III-C asserts 'research works have proven' hybrid is better, with no cross-study comparability argument. So the paper's main conclusions in Section VIII are unsupported.\n\nThat said, this doesn't destroy the survey's descriptive value. It's a compilation, not a derived result. If the authors weakened these claims to 'RF is commonly used' and 'hybrid approaches are promising,' the paper would be fine. One more minor quibble: the 'systematic' label is a bit generous—no search strategy or inclusion criteria are given—but that's common in this area.\n\nWho is this for? Newcomers wanting a structured overview and a checklist of open problems. It deserves a serious referee, because a survey with useful organization and a genuine gap list can be made publishable with revision. My recommendation: send it to peer review, but reviewers should insist on either removing the cross-study performance rankings or reframing them as descriptive frequency claims.","headline":"A decent descriptive survey of phishing detection, but its headline claims that RF is best and hybrids outperform are unsupported because they pool numbers from non-comparable studies.","tokens_in":40058,"tokens_out":2419,"would_cite":false,"duration_ms":26814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey maps phishing webpage detection into URL, content, and visual approaches, and catalogs the open problems that keep them from stopping zero-day attacks.","keywords":["phishing detection","phishing webpage","URL-based detection","visual similarity","machine learning","deep learning","zero-day phishing","survey"],"falsifier":"Run the surveyed classifiers (RF, hybrid, etc.) on one common benchmark corpus with a fixed train/test split, balanced classes, and a single evaluation protocol; if RF's edge and the hybrid advantage shrink or reverse, those comparative conclusions fail.","tokens_in":39263,"feed_emoji":"🎣","tokens_out":3398,"duration_ms":39725,"temperature":0.7,"pith_summary":"This survey tries to bring order to the scattered field of phishing webpage detection. It categorizes existing approaches by what input they inspect—the URL string, the webpage's content and code, or its visual appearance—and then draws from the literature a list of recurring weaknesses: unbalanced and undiverse datasets, correlated features, tiny URLs, compromised domains, adversarial attacks, and AI-generated phishing pages. The authors argue that this structure makes the research gaps visible and propose practical fixes, including a synthetic phishing-page generator for dataset creation, feature-selection algorithms, stacked classifiers, and browser-based user education. A careful reader would care because the survey offers a single map of what has been tried and where defenses consistently fall short.","feed_headline":"Survey maps phishing defenses and the gaps that let zero-days through","feed_subtitle":"URL, content, and visual techniques are catalogued; datasets and metrics stay too varied to compare fairly.","key_machinery":"The survey's organizing device is a three-branch taxonomy of detection methodologies: URL-based approaches (list-based, certificate-based, heuristic, ML, DL), webpage-based approaches (content similarity, ML on content, screenshot similarity, DL on screenshots), and hybrid approaches that concatenate URL and content feature vectors before classification. The taxonomy carries the argument by turning a scattered literature into a map, and that map is what the authors use to identify which gaps appear in every branch.","core_discovery":"The paper claims that phishing webpage detection research can be organized by the input it inspects—the URL string, the page's content and code, or its visual appearance (screenshots, logos, favicons)—and that this organization exposes where current defenses fall short. On the evidence it surveys, it further claims that the Random Forest classifier is the most frequently favored machine-learning choice and that hybrid approaches combining URL and content features give the best zero-day detection performance. It then lists open problems and offers remedies: balanced dataset generation, feature-selection algorithms, stacked classifiers, brand prediction for proactive defense, and user-educatio","pith_inferences":["The survey's comparative claims rest on cross-paper metric comparisons that its own tables show involve different datasets, class balances, and evaluation protocols; a shared benchmark would be the natural next step to verify whether RF and hybrid superiority actually holds.","The proposed phishing-page generator could double as an adversarial robustness testbed: generated pages with varied phishing attributes can probe how classifiers generalize beyond a fixed repository.","Brand prediction via NLP on business rankings is a proactive angle that could let defenses anticipate which domains attackers will impersonate next, rather than reacting after phishing pages appear.","Stacking classifiers, suggested as a performance booster, is not itself evaluated in the survey; testing it on a balanced dataset would be a direct, low-cost follow-up experiment."],"forward_implications":["If the taxonomy is accurate, newcomers can locate any detection method by its input and immediately see which techniques already exist for that branch.","If Random Forest is indeed the consistently favored classifier, new work can reasonably start with RF as a baseline before trying more complex models.","If hybrid detection outperforms single-input approaches, combining URL and content features is a promising design direction for zero-day phishing detection.","If the listed open problems are real, detection research should prioritize diverse balanced datasets, tiny-URL handling, compromised-domain detection, adversarial robustness, and LLM-generated-page detection.","If the proposed phishing-page generator (synthesizing phishing pages from legitimate URLs plus random phishing attributes) is adopted, it could supply the balanced, diverse data the field currently lacks."],"supporting_citations":[{"why":"Supplies the lexical and host-feature method that anchors the URL-based ML/heuristic branch.","marker":"[8]"},{"why":"CANTINA's content-based TF-IDF approach is the basis of the webpage content similarity branch.","marker":"[67]"},{"why":"VisualPhishNet's Siamese-network visual similarity is the basis of the DL screenshot branch.","marker":"[12]"},{"why":"PhishAlarm's CSS component-similarity method grounds the visual component comparison approach.","marker":"[9]"},{"why":"Phishpedia's hybrid deep visual identification anchors the screenshot/logo comparison category.","marker":"[63]"},{"why":"Provides the hybrid feature-based machine-learning framework that supports the claim that hybrid approaches outperform single-input ones.","marker":"[57]"},{"why":"Demonstrates deep-learning models on hybrid features, reinforcing the hybrid advantage claim.","marker":"[58]"},{"why":"Shows high accuracy with RF and DT on a hand-crafted feature set, used to support ML-based URL detection and the RF emphasis.","marker":"[10]"},{"why":"URLNet's CNN over URL characters and words anchors the DL-based URL analysis subcategory.","marker":"[70]"},{"why":"The SoK-style comprehensive reexamination supplies many of the open issues and evaluation pitfalls the survey adopts.","marker":"[124]"}],"fun_headline_variants":["Survey: Hybrid URL-content features best for zero-day phishing","Phishing detection survey maps URL, content, visual methods","Study: Random Forest popular, but hybrid beats zero-days","Phishing survey exposes gaps in detection benchmarks","Best phishing defense? Hybrid methods, survey says"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The survey's comparative conclusions—that Random Forest is predominantly favored and hybrid approaches outperform others—assume that accuracy, precision, recall, and F1 values reported by different papers on different datasets with different evaluation protocols can be compared directly.","fun_headline_variants_meta":{"raw":{"variants":["Survey: Hybrid URL-content features best for zero-day phishing","Phishing detection survey maps URL, content, visual methods","Study: Random Forest popular, but hybrid beats zero-days","Phishing survey exposes gaps in detection benchmarks","Best phishing defense? Hybrid methods, survey says"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":2909,"prompt_tokens":649,"completion_tokens":2260,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":393,"completion_tokens_details":{"reasoning_tokens":2183}},"tokens_in":393,"tokens_out":2260,"duration_ms":17736,"temperature":1.0,"reasoning_tokens":2183,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:37:59.037542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the surveyed classifiers (RF, hybrid, etc.) on one common benchmark corpus with a fixed train/test split, balanced classes, and a single evaluation protocol; if RF's edge and the hybrid advantage shrink or reverse, those comparative conclusions fail.","supporting_citations":[],"review_version":1}