pith. sign in

arxiv: 2606.21690 · v2 · pith:JNLRBYXBnew · submitted 2026-06-19 · 💻 cs.CR · cs.CL· cs.LG

A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage

Saifelden M. Ismail , Aser O. Ibrahim , Omar A. Mahmoud This is my paper

Pith reviewed 2026-06-26 13:36 UTC · model grok-4.3

classification 💻 cs.CR cs.CLcs.LG
keywords phishing detectionURL classificationNLP classifierdecision-level fusionthreat intelligenceDistilBERTmulti-channel pipelinefalse positive reduction
0
0 comments X

The pith

A hybrid pipeline fuses independent URL, NLP and threat engines at decision level to reach F1 0.914 on a 10,677-email benchmark while cutting real-spam false positives to 3.6%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper constructs three separately validated engines for phishing signals: a four-stage URL stack, a DistilBERT text classifier hardened for generalization, and an instrumented threat-intelligence synchronizer. These outputs are combined through a calibrated probabilistic-OR fusion layer that operates on URL, header and phishing-probability channels. The integrated system is tested end-to-end on a large email benchmark, yielding the reported performance numbers, yet the authors flag the result as preliminary because the test uses proxy channels and the operating point still requires recalibration. The central message is that generalization to unseen threats, not accuracy on training-like data, determines whether such a detector can be deployed.

Core claim

The paper demonstrates that a decision-level fusion stage, using a calibrated probabilistic-OR over URL, header and phishing-probability channels, attains an F1 score of 0.914 on a 10,677-email whole-system benchmark and reduces held-out real-spam false positives to 3.6 percent. The three engines are built and benchmarked independently: a URL stack with domain guard, lexical model and threat intelligence; a generalization-hardened DistilBERT model whose real-phishing recall improves from 0.8 percent to 87.3 percent; and a threat-intelligence synchronizer with OpenTelemetry instrumentation that preserves 1:1 message conservation. The authors explicitly present the fused result as preliminary

What carries the argument

Decision-level fusion stage that applies a calibrated probabilistic-OR across URL, header and phishing-probability channels.

If this is right

  • Each modality can be developed, validated and updated independently without retraining the entire system.
  • The limiting factor for production use shifts from training-set accuracy to how well each engine generalizes to novel phishing variants.
  • Instrumentation confirming 1:1 message conservation removes one source of silent data loss in the pipeline.
  • Recalibration of the fusion thresholds becomes a required step before any real-world deployment.
  • The architecture separates concerns so that improvements in one engine translate directly into the fused score.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same fusion logic could be applied to other multi-modal security tasks such as malware or fraud detection where independent signals exist.
  • If generalization remains the bottleneck, future benchmarks should prioritize temporally separated test sets drawn from later time periods rather than random splits.
  • The emphasis on independent engine validation suggests that open-sourcing the individual components would allow community testing of each channel in isolation.
  • Deployment would likely require continuous monitoring of channel drift so that the calibrated fusion weights can be adjusted over time.

Load-bearing premise

The reported benchmark performance rests on proxy URL and header channels rather than live integrated data, and the chosen operating point still needs recalibration before any deployment claim can be made.

What would settle it

Run the full pipeline on a new collection of live emails that supplies genuine real-time URL and header data, then measure whether F1 stays at or above 0.9 and real-spam false positives remain at or below 4 percent after the operating point is recalibrated.

Figures

Figures reproduced from arXiv: 2606.21690 by Aser O. Ibrahim, Omar A. Mahmoud, Saifelden M. Ismail.

Figure 1
Figure 1. Figure 1: Processing topology of the system. Thick solid boxes (SVC-03, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
read the original abstract

Phishing is a multi-modal threat. We present a hybrid pipeline that scores each modality with its own engine and fuses the results. Three engines are built, deployed, and independently benchmarked: a four-stage URL stack (Domain Guard, lexical model, threat intelligence, and an asymmetric L2 fusion sidecar); a generalization-hardened DistilBERT NLP classifier whose held-out real-phishing recall rises from 0.8% to 87.3%; and a threat-intelligence synchronizer with end-to-end OpenTelemetry instrumentation confirming 1:1 message conservation. A decision-level fusion stage, characterized on a 10,677-email whole-system benchmark, reaches F1=0.914 with a calibrated probabilistic-OR over URL, header, and phishing-probability channels while reducing held-out real-spam false positives to 3.6%. Because that benchmark uses proxy URL and header channels and an operating point still needing recalibration, we present it as a preliminary integrated result. For deployable detection, the limiting factor is how well a model generalizes, not how accurately it scores data drawn from its own training distribution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript describes a hybrid pipeline for phishing and threat classification that combines three independently developed and benchmarked engines—a four-stage URL stack (Domain Guard, lexical model, threat intelligence, asymmetric L2 fusion sidecar), a generalization-hardened DistilBERT NLP classifier (held-out real-phishing recall improved from 0.8% to 87.3%), and a threat-intelligence synchronizer with OpenTelemetry instrumentation for 1:1 message conservation—with a decision-level fusion stage. The fusion uses a calibrated probabilistic-OR over URL, header, and phishing-probability channels and is characterized on a 10,677-email whole-system benchmark, yielding F1=0.914 and 3.6% held-out real-spam false positives; the result is explicitly presented as preliminary because the benchmark substitutes proxy channels and requires further recalibration. The paper emphasizes that generalization to real data, rather than in-distribution accuracy, is the limiting factor for deployable detection.

Significance. If the fusion performance holds after recalibration with the actual engines, the multi-channel approach could provide a practical way to combine modality-specific signals for improved phishing detection. Credit is due for the independent validation of each engine, the use of held-out real data rather than synthetic benchmarks, and the OpenTelemetry instrumentation that verifies end-to-end conservation. These elements strengthen the methodological contribution even if the headline numbers remain preliminary.

major comments (1)
  1. [Fusion stage / whole-system benchmark] Benchmark description (near the 10,677-email result): the F1=0.914 and 3.6% false-positive figures are obtained with proxy URL and header channels rather than the deployed four-stage URL engine and DistilBERT NLP classifier; because this substitution is load-bearing for any claim about the calibrated probabilistic-OR fusion stage, the preliminary qualification must be retained and the path to a non-proxy evaluation made explicit.
minor comments (1)
  1. [Abstract] The phrase 'asymmetric L2 fusion sidecar' appears in the abstract without an accompanying definition or forward reference to the section that describes its operation.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the thorough review and for recognizing the independent validation of each engine, the use of held-out real data, and the OpenTelemetry instrumentation. We address the single major comment below.

read point-by-point responses

  1. Referee: [Fusion stage / whole-system benchmark] Benchmark description (near the 10,677-email result): the F1=0.914 and 3.6% false-positive figures are obtained with proxy URL and header channels rather than the deployed four-stage URL engine and DistilBERT NLP classifier; because this substitution is load-bearing for any claim about the calibrated probabilistic-OR fusion stage, the preliminary qualification must be retained and the path to a non-proxy evaluation made explicit.

    Authors: We agree that the substitution of proxy channels renders the 10,677-email benchmark preliminary and that this qualification is load-bearing for claims about the fusion stage. The manuscript already qualifies the result as preliminary on exactly these grounds. In revision we will expand the benchmark description and future-work section to make the path to a non-proxy evaluation explicit, including the planned integration of the four-stage URL engine and DistilBERT classifier followed by recalibration on the same held-out corpus. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper describes an engineering pipeline with three independently benchmarked engines (URL stack, DistilBERT NLP, threat-intelligence synchronizer) whose outputs are fused via a calibrated probabilistic-OR. The central performance numbers (F1=0.914, 3.6% false positives) are obtained from an explicit 10,677-email held-out benchmark that the authors themselves qualify as preliminary because it substitutes proxy channels and requires further recalibration. No equations, parameter fits, or self-citations are presented that would make these metrics reduce by construction to quantities defined by the same training data or inputs. The derivation chain is therefore empirical benchmarking rather than a closed mathematical loop.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review reveals no mathematical model, free parameters, or new postulated entities; the contribution is an engineering pipeline whose performance claims rest on benchmark measurements.

pith-pipeline@v0.9.1-grok · 5756 in / 1175 out tokens · 21443 ms · 2026-06-26T13:36:04.377999+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 8 canonical work pages · 3 internal anchors

  1. [1]

    A new hybrid ensemble feature selection framework for machine learning- based phishing detection system,

    K. L. Chiew, C. L. Tan, K. Wong, K. S. C. Yong, and W. K. Tiong, “A new hybrid ensemble feature selection framework for machine learning- based phishing detection system,”Information Sciences, vol. 484, pp. 153–166, 2019

    2019

  2. [2]

    Dataset of suspicious phishing URL detection,

    M. A. Tamal, M. K. Islam, T. Bhuiyan, and A. Sattar, “Dataset of suspicious phishing URL detection,”Frontiers in Computer Science, vol. 6, p. 1308634, 2024

    2024

  3. [3]

    An assessment of features related to phishing websites using an automated technique,

    R. M. Mohammad, F. Thabtah, and L. McCluskey, “An assessment of features related to phishing websites using an automated technique,” in Int. Conf. for Internet Technology and Secured Transactions (ICITST- 2012). IEEE, 2012

    2012

  4. [4]

    LegitPhish: A large-scale annotated dataset for URL-based phishing detection,

    R. S. Potpelwar, U. V . Kulkarni, and J. M. Waghmare, “LegitPhish: A large-scale annotated dataset for URL-based phishing detection,”Data in Brief, vol. 63, p. 111972, 2025

    2025

  5. [5]

    PhiUSIIL: A diverse security profile em- powered phishing URL detection framework based on similarity index and incremental learning,

    A. Prasad and S. Chandra, “PhiUSIIL: A diverse security profile em- powered phishing URL detection framework based on similarity index and incremental learning,”Computers & Security, vol. 136, p. 103545, 2024

    2024

  6. [6]

    Comparison of the predicted and observed secondary structure of T4 phage lysozyme,

    B. W. Matthews, “Comparison of the predicted and observed secondary structure of T4 phage lysozyme,”Biochimica et Biophysica Acta (BBA) - Protein Structure, vol. 405, no. 2, pp. 442–451, 1975

    1975

  7. [7]

    Cisco umbrella popularity list,

    Cisco Umbrella, “Cisco umbrella popularity list,” https://s3-us-west-1. amazonaws.com/umbrella-static/index.html, daily ranking of most- queried domains on Cisco recursive resolvers

  8. [8]

    Binary codes capable of correcting deletions, inser- tions, and reversals,

    V . I. Levenshtein, “Binary codes capable of correcting deletions, inser- tions, and reversals,”Soviet Physics Doklady, vol. 10, no. 8, pp. 707–710, 1966

    1966

  9. [9]

    Brand impersonation in domain names,

    B. Edelman, “Brand impersonation in domain names,” OECD/APWG industry reports on subdomain-based brand abuse, Tech. Rep., 2003

    2003

  10. [10]

    Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,

    J. C. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” inAdvances in Large Margin Classifiers. MIT Press, 1999

    1999

  11. [11]

    Greedy function approximation: A gradient boosting machine,

    J. H. Friedman, “Greedy function approximation: A gradient boosting machine,”The Annals of Statistics, vol. 29, no. 5, pp. 1189–1232, 2001

    2001

  12. [12]

    LightGBM: A highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “LightGBM: A highly efficient gradient boosting decision tree,” in Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  13. [13]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,”arXiv preprint arXiv:1910.01108, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  14. [14]

    Dual-path phishing detection: Integrating transformer-based NLP with structural URL analysis,

    I. Altan, A. Bachir, Y . Parbhulkar, A. M. Rizvi, and M. Farazi, “Dual-path phishing detection: Integrating transformer-based NLP with structural URL analysis,”arXiv preprint arXiv:2509.20972, 2025

    work page arXiv 2025

  15. [15]

    PhishingGNN: Phishing email detection us- ing graph attention networks and transformer-based feature extraction,

    M. Safran and A. Musleh, “PhishingGNN: Phishing email detection us- ing graph attention networks and transformer-based feature extraction,” IEEE Access, vol. 13, pp. 131 390–131 399, 2025

    2025

  16. [16]

    Lightweight transformer models for scalable phishing email detection: A comparative study of ALBERT and TinyBERT on a balanced email corpus,

    O. Atanda, H. Aworinde, and B. van Niekerk, “Lightweight transformer models for scalable phishing email detection: A comparative study of ALBERT and TinyBERT on a balanced email corpus,”Int. J. of Innovative Research and Scientific Studies (IJIRSS), vol. 9, no. 2, pp. 10–20, 2026

    2026

  17. [17]

    Explainable few-shot learning with modern BERT for detecting emerging phishing attacks using XF-PhishBERT,

    M. Tawfik, A. A. Abu-Ein, A. H. Abdelhaliem, Y . M. Al-Sharo, and I. S. Fathi, “Explainable few-shot learning with modern BERT for detecting emerging phishing attacks using XF-PhishBERT,”Scientific Reports, vol. 15, 2025

    2025

  18. [18]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” inProc. Int. Conf. on Learning Representations (ICLR), 2015

    2015

  19. [19]

    A Robust and Explainable Transformer-Based Framework for Phishing Email Detection

    U. P. Sajad, “Explainable transformer-based email phishing classification with adversarial robustness,”arXiv preprint arXiv:2511.12085, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    The use of multi-task learning in cybersecurity applications: a systematic literature review,

    S. Ibrahim, C. Catal, and T. Kacem, “The use of multi-task learning in cybersecurity applications: a systematic literature review,”Neural Computing and Applications, 2024

    2024

  21. [21]

    Multi-task learning using uncer- tainty to weigh losses for scene geometry and semantics,

    A. Kendall, Y . Gal, and R. Cipolla, “Multi-task learning using uncer- tainty to weigh losses for scene geometry and semantics,” inProc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  22. [22]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” inProc. IEEE Int. Conf. on Computer Vision (ICCV), 2017

    2017

  23. [23]

    MeAJOR corpus: A multi-source dataset for phishing email detection,

    P. Mendes, E. Maia, and I. Prac ¸a, “MeAJOR corpus: A multi-source dataset for phishing email detection,”arXiv preprint arXiv:2507.17978, 2025

    work page arXiv 2025

  24. [24]

    Phishing codebook: A structured framework for the characterization of phishing emails,

    T. Saka, R. Jain, K. Vaniea, and N. K ¨okciyan, “Phishing codebook: A structured framework for the characterization of phishing emails,”arXiv preprint arXiv:2408.08967, 2024

    work page arXiv 2024

  25. [25]

    The impact of quantization on the robustness of transformer-based text classifiers,

    S. P. Neshaei, Y . Boreshban, G. Ghassem-Sani, and S. A. Mirroshandel, “The impact of quantization on the robustness of transformer-based text classifiers,”arXiv preprint arXiv:2403.05365, 2024

    work page arXiv 2024

  26. [26]

    The phish, the spam, and the valid: Generating feature-rich emails for benchmarking LLMs,

    R. Toth, T. Bisztray, and N. Gruschka, “The phish, the spam, and the valid: Generating feature-rich emails for benchmarking LLMs,”arXiv preprint arXiv:2511.21448, 2025

    work page arXiv 2025

  27. [27]

    The biggest spam ham phish email dataset (300000+),

    A. Sharma, “The biggest spam ham phish email dataset (300000+),” https://www.kaggle.com/datasets/akshatsharma2/ the-biggest-spam-ham-phish-email-dataset-300000

  28. [28]

    Education-targeted phishing email dataset,

    T. Ahmed, “Education-targeted phishing email dataset,” https://www.kaggle.com/datasets/tanvirahmed0981/ education-targeted-phishing-email-dataset

  29. [29]

    David versus Goliath: Can machine learning detect LLM-generated text? a case study in the detection of phishing emails,

    F. Greco, G. Desolda, A. Esposito, and A. Carelli, “David versus Goliath: Can machine learning detect LLM-generated text? a case study in the detection of phishing emails,” inProc. Italian Conf. on CyberSecurity (ITASEC 2024), ser. CEUR Workshop Proceedings, vol. 3731, 2024. [Online]. Available: https://ceur-ws.org/V ol-3731/paper41.pdf

    2024

  30. [30]

    Nazario phishing corpus,

    J. Nazario, “Nazario phishing corpus,” https://monkey.org/ ∼jose/ phishing/, 2005, cC-BY-4.0

    2005

  31. [31]

    CLAIR collection of fraud email,

    D. Radev, “CLAIR collection of fraud email,” ACL Data and Code Repository, ADCR2008T001, 2008. [Online]. Available: https://www. aclweb.org/aclwiki/CLAIR collection of fraud email (Repository)

    2008

  32. [32]

    Seven phishing email datasets,

    Puyang, “Seven phishing email datasets,” https://huggingface.co/ datasets/puyang2025/seven-phishing-email-datasets, 2025

    2025

  33. [33]

    On the resemblance and containment of documents,

    A. Z. Broder, “On the resemblance and containment of documents,” inProc. Compression and Complexity of Sequences (SEQUENCES’97), 1997, pp. 21–29

    1997

  34. [34]

    XGBoost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” inProc. ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, 2016

    2016

  35. [35]

    Public suffix list,

    Mozilla Foundation, “Public suffix list,” https://publicsuffix.org/, community-maintained list of public suffixes for registrable-domain extraction

  36. [36]

    Openphish phishing intelligence,

    OpenPhish, “Openphish phishing intelligence,” https://openphish.com/, commercial phishing URL feed

  37. [37]

    Phishtank,

    Cisco Talos Intelligence Group, “Phishtank,” https://data.phishtank.com/, community-verified phishing URL feed

  38. [38]

    Urlhaus malware url exchange,

    abuse.ch, “Urlhaus malware url exchange,” https://urlhaus.abuse.ch/, community malware URL feed

  39. [39]

    Threatfox IOC sharing platform,

    ——, “Threatfox IOC sharing platform,” https://threatfox.abuse.ch/, community indicator-of-compromise sharing

  40. [40]

    Malwarebazaar,

    ——, “Malwarebazaar,” https://bazaar.abuse.ch/, malware sample and hash exchange

  41. [41]

    Decoupled Weight Decay Regularization

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2019, published at ICLR 2019

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  42. [42]

    Detecting near-duplicates for web crawling,

    G. S. Manku, A. Jain, and A. D. Sarma, “Detecting near-duplicates for web crawling,” inProc. Int. Conf. on World Wide Web (WWW), 2007

    2007

  43. [43]

    Pearl,Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference

    J. Pearl,Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo, CA: Morgan Kaufmann, 1988

    1988

  44. [44]

    The Enron corpus: A new dataset for email classification research,

    B. Klimt and Y . Yang, “The Enron corpus: A new dataset for email classification research,” inEuropean Conference on Machine Learning, 2004, pp. 217–226

    2004

  45. [45]

    Spamassassin public mail corpus,

    J. Mason, “Spamassassin public mail corpus,” https://spamassassin. apache.org/old/publiccorpus/, 2002

    2002