{"id":"bff3f9ff-870d-4c8d-8ea7-8707b4e5182a","arxiv_id":"2608.07451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Analysis of 1,300 phishkits (2020-2023) finds heavy code reuse, simple evasion tricks, and widespread Telegram bot exfiltration, making phishkit-based phishing highly predictable.","lead":"This study analyzed 1,300 phishing kits collected between 2020 and 2023, finding that most rely on simple, highly reused code for evasion and credential theft. It quantifies the growing use of Telegram bots and email to exfiltrate stolen data, suggesting that large-scale phishing operations may be easier to detect than feared.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The predictability result may be an artifact of collection bias: 732 of 1,300 kits come from one 3-month commercial feed, and the 21% 'structurally identical' figure is computed on the pooled set without source stratification or released clustering parameters.","rationale":"The reader's weakest assumption—dataset representativeness—is indeed the most load-bearing concern. The paper itself acknowledges in Section VIII that the dataset is a best-effort lower bound, but the central claim in the abstract and conclusion is stated without that caveat. My reading sharpens the concern by pointing to a concrete mechanism: a single commercial feed covering three months supplies 56% of the dataset, and such feeds typically detect phishing pages by matching known source-code signatures. This creates a circularity risk for the code-reuse conclusion: if the collection method preferentially selects kits that resemble previously seen phishkits, then observing heavy reuse is partly an artifact of the sampling process. I do not claim this is definitely the case; the commercial feed could be broad and unbiased, and the Phishunt repository could independently confirm the pattern. That is why the source-stratified recomputation is the right test. The clustering methodology is also underspecified, but that is secondary: even with perfect methodology, the sample-bias issue would remain. I agree with the reader that the paper is otherwise transparent and internally consistent, with plausible findings that align with prior work. The CONDITIONAL verdict is appropriate; my concern reinforces the condition that the authors provide source-stratified analysis and methodology details. I therefore set verdict_should_be to UNCHANGED rather than moving it, since the reader already assigned CONDITIONAL and the required action is the same: supply the missing stratification and clustering details, ideally with released analysis scripts.","tokens_in":15258,"tokens_out":3630,"duration_ms":40349,"concrete_test":"Ask the authors to release the clustering pipeline and per-kit source labels, then recompute the 100%-structural-similarity rate separately for the 589 Phishunt kits and the 732 commercial-feed kits using identical hyperparameters and preprocessing. Also recompute it after removing kits from the Dec 2022–Mar 2023 feed. If the Phishunt-only rate is materially lower (e.g., below 10%) or the 276-kit figure is concentrated in the three-month feed, the conclusion should be re-scoped from 'predictable in the wild' to 'predictable within one feed's observation window.' As an auxiliary check, compare the embedding-based result against exact file-hash or normalized-source deduplication to verify that the reported structural identity reflects actual code reuse rather than a loose embedding threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that heavy code reuse makes phishkits predictable and easier to detect at scale—depends on the code-reuse measurements in Section VI. There, the authors report 171 similar clusters and 276 kits (21%) with '100% structural similarity,' computed by Node2Vec embeddings of PHP control-flow graphs followed by agglomerative clustering. However, the pooled dataset is dominated by 732 kits from a single commercial daily feed collected over only three months (Dec 21, 2022–Mar 20, 2023), and 589 kits from one public repository. A feed that identifies phishing pages by source code is likely to surface kits that match known phishkit families, biasing the sample toward exactly the reuse the paper claims to discover. The temporal concentration reinforces this: the largest exemplar cluster spans only November 2022 to February 2023. Section VIII's lower-bound caveat is a useful honesty marker, but it does not remove the concern because the abstract generalizes to 'a large number of cases' and 'detection ... at scale.' Furthermore, the clustering hyperparameters, similarity threshold, and definition of 'structurally identical' are not specified and the analysis code is not released, so the 21% figure cannot be independently audited. If the structural-similarity rate drops sharply when computed on the Phishunt-only subset or after controlling for the feed's three-month window, the paper's quantitative support for the central claim would no longer hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an observational study of 1,300 phishkits collected from 2020 to 2023, drawing on a commercial daily feed (732 kits) and a public repository (589 kits). It analyzes backdoor mechanisms (email and Telegram), traffic attribution and cloaking, structural code reuse over time, and characterizes the ecosystem as highly predictable and therefore easier to defend against at scale. The authors also report responsible handling of leaked victim data, including IRB approval and disclosure to a credit-card issuer.","tokens_in":15526,"tokens_out":3930,"duration_ms":41199,"significance":"If the quantitative claims hold, the paper provides useful longitudinal evidence on the commoditization of phishing infrastructure. The study's strengths include a real-world dataset spanning two collection sources, live interaction with Telegram bots, a clear limitations section, and internally consistent arithmetic. The claim that heavy code reuse makes phishkit-based phishing predictable is relevant for detection research. However, the headline code-reuse measurements depend on unspecified clustering hyperparameters and an unstratified dataset, so the quantitative support for the main claim needs substantial clarification before the paper is publication-ready.","major_comments":[{"comment":"The central quantitative claim that 276 kits (21%) have 'structurally identical layouts and 100% structural similarity' is not auditable as reported. The pipeline uses Node2Vec embeddings and agglomerative clustering, but the embedding dimension, random-walk parameters, distance threshold, and the definition of the '100%' cutoff are not specified. Since cluster membership is used to assert '100% structural similarity,' the metric is partly circular. Please report all hyperparameters and the exact similarity threshold, or replace this with a direct pairwise code-similarity measure (e.g., normalized AST/CFG edit distance) computed with a fixed threshold. Also state whether the 21% figure is stable under reasonable parameter variation.","section":"Section VI (Analysis of Phishkits Over the Years)"},{"comment":"The dataset is dominated by 732 kits from a single commercial feed collected over a three-month window (Dec 21, 2022–Mar 20, 2023) and 589 kits from one public repository. The abstract and conclusion generalize to 'a large number of cases' and 'detection ... at scale' without the Section VIII lower-bound caveat. If the commercial feed preferentially surfaces kits that match known phishkit families, the code-reuse estimates are inflated. Please report the similarity statistics separately for the two sources and by collection period, and show that the largest clusters are not an artifact of the short feed window. At a minimum, the abstract should carry the same 'lower-bound estimate' qualification as Section VIII.","section":"Section III (Methodology) and Abstract"},{"comment":"The first paragraph of Section VIII states that 'our approach for creating clusters was based on the network address and the company names,' with judgments made from reverse DNS lookups. This conflicts with the clustering methodology in Section VI, which builds clusters from Node2Vec embeddings of PHP control-flow graphs. The sentence appears to describe the sector classification in Table III, not the code-similarity clusters, but as written it undermines the methodological narrative. Please rewrite this passage to disambiguate the cluster-construction process for code similarity from the classification of target sectors.","section":"Section VIII (Limitations) vs. Section VI"},{"comment":"The paper states in the Ethics subsection that 'we did not perform any analysis of the leaked data,' yet Section IV reports 370 leaked credit card instances, 425 SMS verification codes, and 509 username/password pairs, and the abstract summarizes these counts. These statements are inconsistent unless the authors distinguish 'categorization for responsible disclosure' from 'analysis for research.' If the counts were needed for disclosure, please state that explicitly and describe how the counting was performed without using the PII for research. If the counts came from a different workflow, clarify that workflow.","section":"Section IV (Backdoor Mechanisms) and Ethics"}],"minor_comments":[{"comment":"References [21] and [22] are duplicates of the same paper (Bijmans et al., USENIX Security 2021) and should be merged.","section":"References"},{"comment":"The abstract reports 284 phishkits (21.8%) with no evasion mechanism, while Section V reports 600 kits with IP blocking and 439 with user-agent attribution. Please clarify the overlap between these categories so the reader can follow how the 284 figure is derived with respect to the other 600 and 439 counts.","section":"Abstract and Section V"},{"comment":"The largest cluster is described as having similarity scores 'ranging from 82%–100%' after the paper says all 276 kits in similar clusters had '100% structural similarity.' Please clarify whether the 82–100% range refers to pairwise similarities within the cluster and whether the 100% figure applies only to a subset or to a particular centroid-based definition.","section":"Section VI"},{"comment":"The selection of Node2Vec over spectral embedding and GNNs is justified qualitatively, but no evaluation of the chosen set of features (call graph and CFG) is reported. A short comparison of clustering quality across the three approaches would make the choice reproducible.","section":"Section VI"},{"comment":"The phrase 'After removing duplicate and irrelevant samples, and accounting for 21 kits shared between the two sources' should specify how 'irrelevant' was determined and whether 'duplicate' was based on hashes, filenames, or a stricter comparison.","section":"Section III"},{"comment":"The introduction's contribution list is brief and does not mention the longitudinal analysis or the code-reuse clustering, which are the paper's most distinctive and risky parts. Consider expanding the contribution list to state these explicitly.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important and understudied topic. The main risk is that the headline predictability claim is supported by measurements whose parameters are not disclosed and whose dataset is dominated by a single three-month feed. These issues are fixable with additional analysis and transparency, so I do not recommend rejection. The authors should be asked to provide the missing hyperparameters, release or otherwise make auditable the clustering code, and perform source- and time-stratified analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper's genuinely new material is its census of phishkit exfiltration channels and evasion artifacts from 1,300 kits: 353 unique Telegram bot tokens, 8,635 live messages polled from 292 bots, 456 email accounts, 452,717 unique blocked IP ranges, and a credible body of code-level evidence that most kits use simple, repeated tricks. The soft spot is the structural-similarity analysis: the 276 kits labeled '100% structurally identical' come from a clustering pipeline with unreported thresholds and no released code, and the dataset is dominated by one commercial feed covering about three months. The qualitative findings stand; the quantitative reuse rate is not independently auditable as written.\n\nWhat the paper does well: the ethics and disclosure process is unusually careful - IRB review, no analysis of PII, responsible disclosure to card issuers and Telegram. The arithmetic is internally consistent, and Section VIII contains a frank lower-bound caveat about generalizability. The descriptive results about Telegram-based credential exfiltration and IP blocking are new to this scale and are methodologically simple enough to trust.\n\nThe soft spots, in proportion. The main concern is sampling: 732 of the 1,300 kits come from a single daily feed collected over three months, and the other 589 from one public repository. If the feed identifies phishing pages by matching known kit structures, it will over-represent exactly the code reuse the paper claims to find. The authors acknowledge this in Section VIII, but the abstract states the predictability claim without the caveat, which is overreach. That said, the finding that a substantial share of kits are trivial is consistent with prior work and does not collapse under the bias concern; it just should be presented as a lower bound more consistently. The clustering hyperparameters, distance threshold, and definition of 'structurally identical' are missing, so the 276 figure cannot be checked. Citation errors: [23] is misdescribed (Peng et al. is about credential sharing, not fingerprinting), and [21]/[22] are the same paper listed twice. These are minor but should be cleaned.\n\nWho it's for: security researchers studying phishing infrastructure and abuse of messaging platforms; platform and browser defenders. It deserves a serious referee - the dataset and measurements are worth scrutiny, and the paper could be a useful reference after revision. I'd send it to review, with a request for clustering methodology details and analysis scripts, and a firm nudge to soften the abstract in line with the limitations section.","headline":"A valuable descriptive census of phishkit exfiltration channels and evasion artifacts, whose reuse-rate figure needs methodological transparency before the broad predictability claim can be taken at face value.","tokens_in":16128,"tokens_out":2577,"would_cite":true,"duration_ms":26081,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper analyzes 1,300 phishing kits and argues that heavy code reuse and known evasion tricks make most phishkit-built sites predictable enough to detect at scale.","keywords":["phishkits","phishing attacks","evasion and cloaking","code reuse","Telegram bot exfiltration","traffic attribution","structural similarity","phishing detection"],"falsifier":"Run the same structural clustering on an independent, larger collection of phishkits gathered from multiple commercial feeds and underground forums over a longer window; if the proportion of kits with 100% structurally identical PHP layouts falls well below the 21% observed here, or if most new kits carry bespoke evasion code, the predictability claim is contradicted.","tokens_in":15020,"feed_emoji":"🎣","tokens_out":8390,"duration_ms":77652,"temperature":0.7,"pith_summary":"This paper analyzes 1,300 phishing kits observed between 2020 and 2023 to understand what makes phishing pages cheap to build and hard to catch. It reports that the kits' core modules—page templates, traffic filtering, and stolen-data exfiltration—are heavily reused, with 276 kits sharing 100% structurally identical PHP layouts, 126 kits using identical IP-blocking code, and 284 kits using no evasion at all. It also finds that messaging-bot APIs have become the main exfiltration channel: 292 bot tokens embedded in the live-feed kits produced 8,635 leaked messages with credentials, card numbers, and SMS codes when polled over 101 days. The paper argues that because most phishkits are this predictable, structural-similarity and known-evasion-pattern detection can catch a large share of phishkit-built websites at scale.","feed_headline":"Most phishing kits reuse the same code, making them easy to catch","feed_subtitle":"Analysis of 1,300 kits finds identical evasion code and Telegram exfiltration across many campaigns.","key_machinery":"The load-bearing object is the three-module structure common to modern phishkits: an app template loader that clones a legitimate login page, a backdoor that sends stolen data to the operator over email or messaging-bot APIs, and a traffic-attribution layer that inspects IP address, ISP, user agent, hostname, or geolocation before serving the fake page. The argument is carried by a structural-similarity pipeline that parses each kit's PHP source into control-flow graphs, embeds those graphs as vectors using the Node2Vec method, and clusters them with hierarchical agglomerative clustering, so near-identical layouts become visible as clusters rather than as superficial template differences. That pipeline, together with shared blocklists and repeated bot tokens, is what transforms \"kits look different\" into \"the ecosystem is homogeneous and predictable.\"","core_discovery":"The paper's central claim is that the phishkit ecosystem is far less sophisticated than many accounts suggest: the components that keep a phishing page alive are nearly identical across kits and have not changed critically over the 2020–2023 observation window. The evidence is a set of large overlapping measurements: 600 of 1,300 kits (46.1%) filter incoming traffic by IP address, 312 (24%) block by ISP, 498 call a geolocation service, and 126 kits ship the same evasion file set. Structural analysis of the PHP control-flow graphs puts 276 kits (21%) in clusters with 100% identical structure, with the largest cluster of 17 banking kits sharing 82–100% similarity. On the operational side, 292 Telegram bot tokens appeared in the 732 live-feed kits, and controlled polling of those bots over 101 days captured 8,635 messages, including 370 credit-card numbers and 509 username/password pairs. From these observations the paper concludes that most phishkits manifest similar, predictable behavior and that unsupervised or semi-supervised structure-aware detection can exploit that predictability at scale.","pith_inferences":["One consequence the paper leaves implicit is that defenders can treat a single discovered kit as a starting point for a whole family: because kit code is reused heavily, one cluster's shared artifacts can be used to hunt for its other deployments, effectively turning each new sample into multiple detections.","A testable extension would be to measure how quickly the predictability advantage decays: after this analysis is public, adversarial kit authors may diversify their evasion code, and the share of structurally identical kits in a fresh post-2023 sample would show whether the ecosystem responds to published measurement.","Another unstated implication is that the lower-bound caveat cuts both ways: the same homogeneity that makes detection easy also means a single published countermeasure, such as blocking the observed messaging-bot infrastructure, could force kit authors to rework a large fraction of their operational chain at once."],"forward_implications":["A detector that compares the control-flow structure of a live phishing page against known kit clusters could flag a substantial share of sites without retraining on each new campaign.","Because 284 kits (21.8%) apply no evasion at all, ordinary crawler-based blocklist scanning should already catch them, and a structural detector would catch the other kits that only hide behind IP and user-agent filtering.","The widespread, near-identical use of messaging-bot exfiltration means that tracking and revoking the embedded bot tokens and email addresses can disrupt data theft across many campaigns at once.","The stability of core components over the study window means behavioral signatures built from current kits are likely to retain value for some time, reducing the need for constant retraining."],"supporting_citations":[{"why":"Public phishkit repository that supplied 589 of the 1,300 samples; fixes the dataset's composition and limits.","marker":"[17]"},{"why":"Prior large-scale study of client-side cloaking that documents the evasion techniques the kits are found to reuse.","marker":"[12]"},{"why":"Earlier end-to-end study of phishkit lifecycles; supplies the comparison point for kit prevalence and short-lived deployments.","marker":"[21]"},{"why":"Introduced fingerprinting of phishkits into families; the dataset and code-reuse idea this paper extends with structural-similarity clustering.","marker":"[23]"},{"why":"The safe-browsing service whose crawl behavior is the assumed target of the kits' IP, ISP, and user-agent filtering.","marker":"[30]"},{"why":"Node2Vec embedding used to convert each kit's control-flow graph into a feature vector for similarity clustering.","marker":"[32]"},{"why":"Hierarchical agglomerative clustering algorithm that produces the 171 similarity clusters.","marker":"[34]"}],"fun_headline_variants":["Phishkits are shockingly predictable, and that's a defense win","1,300 phishkits reveal a boring, catchable ecosystem","Telegram bots leak 370 credit cards from phishing kits","Phishkit analysis: 21% are clones, 46% IP-filter","Most phishing kits are lazy—same code, same tricks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion depends on the 1,300 collected kits—732 from a single commercial feed covering about three months and 589 from one public repository—being representative of the broader phishkit ecosystem; if more diverse or sophisticated kits escape collection, the true ecosystem is less predictable than the paper claims.","fun_headline_variants_meta":{"raw":{"variants":["Phishkits are shockingly predictable, and that's a defense win","1,300 phishkits reveal a boring, catchable ecosystem","Telegram bots leak 370 credit cards from phishing kits","Phishkit analysis: 21% are clones, 46% IP-filter","Most phishing kits are lazy—same code, same tricks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1514,"prompt_tokens":1039,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":655,"tokens_out":475,"duration_ms":4587,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:25:17.594898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same structural clustering on an independent, larger collection of phishkits gathered from multiple commercial feeds and underground forums over a longer window; if the proportion of kits with 100% structurally identical PHP layouts falls well below the 21% observed here, or if most new kits carry bespoke evasion code, the predictability claim is contradicted.","supporting_citations":[{"cited_title":"Phishing kits collected from phishunt,","cited_arxiv_id":null,"evidence_quote":"Public phishkit repository that supplied 589 of the 1,300 samples; fixes the dataset's composition and limits."},{"cited_title":"Crawlphish: Large-scale analysis of client-side cloaking techniques in phishing,","cited_arxiv_id":null,"evidence_quote":"Prior large-scale study of client-side cloaking that documents the evasion techniques the kits are found to reuse."},{"cited_title":"Catching phishers by their bait: Investigating the dutch phishing landscape through phishing kit detection,","cited_arxiv_id":null,"evidence_quote":"Earlier end-to-end study of phishkit lifecycles; supplies the comparison point for kit prevalence and short-lived deployments."},{"cited_title":"Google Safe Browsing,","cited_arxiv_id":null,"evidence_quote":"The safe-browsing service whose crawl behavior is the assumed target of the kits' IP, ISP, and user-agent filtering."}],"review_version":1}