pith. sign in

arxiv: 2604.09849 · v1 · submitted 2026-04-09 · 💻 cs.CR · cs.LG

Improving DNS Exfiltration Detection via Transformer Pretraining

Milo\v{s} Tomi\'c , Aleksa Cvetanovi\'c , Predrag Tadi\'c This is my paper

Pith reviewed 2026-05-10 17:42 UTC · model grok-4.3

classification 💻 cs.CR cs.LG
keywords DNS exfiltrationBERT pretrainingTransformer modelsROC curveFalse positive ratesNetwork securityIntrusion detection
0
0 comments X

The pith

Pretrained BERT improves subdomain DNS exfiltration detection at low false positive rates.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper investigates whether pretraining a BERT model on DNS traffic data before fine-tuning improves detection of exfiltration attempts at the subdomain level. It develops a controlled pipeline that selects classification thresholds on validation data and applies them unchanged to test data, allowing clean comparisons across different pretraining lengths and amounts of labeled fine-tuning data. Results indicate meaningful gains in the low false-positive region of the ROC curve relative to randomly initialized models, with larger gains from additional pretraining steps when more labeled data is available. These findings matter because operational detection systems must catch stealthy DNS-based data leaks while keeping false alarms low enough to avoid overwhelming analysts.

Core claim

By pretraining BERT on in-domain DNS data and then fine-tuning on labeled exfiltration examples, the model achieves better true positive rates at low false positive rates compared to random initialization, and within pretrained variants, increasing the number of pretraining steps helps the most when more labeled data are available for fine-tuning.

What carries the argument

The controlled ablation pipeline that freezes operating points selected on the validation set and transfers them unchanged to the test set, enabling direct comparison of pretraining and label budgets without threshold selection bias.

Load-bearing premise

That freezing operating points on the validation set and transferring them unchanged to the test set produces unbiased, clean ablations across pretraining and label budgets without introducing selection bias or distribution shift.

What would settle it

Repeating the experiments but selecting operating points to optimize performance directly on the test set and checking whether the reported gains in the low false-positive regime disappear.

Figures

Figures reproduced from arXiv: 2604.09849 by Aleksa Cvetanovi\'c, Milo\v{s} Tomi\'c, Predrag Tadi\'c.

Figure 1
Figure 1. Figure 1: Confusion matrix heatmaps at 0.1% FPR for Randomly initialized, PT-37.5k and HF-PT-37.5k (full-data [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
read the original abstract

We study whether in-domain pretraining of Bidirectional Encoder Representations from Transformer (BERT) model improves subdomain-level detection of exfiltration at low false positive rates. While previous work mostly examines fine-tuned generic Transformers, it does not aim to isolate the effect of pretraining on the downstream task of classification. To address this gap, we develop a controlled pipeline where we freeze operating points on validation and transfer them to the test set, thus enabling clean ablations across different label and pretraining budgets. Our results show significant improvements in the left tail of the Receiver Operating Characteristic (ROC) curve, especially against randomly initialized baseline. Additionally, within pretrained model variants, increasing the number of pretraining steps helps the most when more labeled data are available for fine-tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript examines whether in-domain pretraining of BERT models enhances subdomain-level detection of DNS exfiltration at low false positive rates. It introduces a controlled pipeline that selects operating points on a validation set and transfers them unchanged to the test set to support ablations across pretraining step counts and labeled data budgets for fine-tuning. The reported results indicate significant improvements in the left tail of the ROC curve relative to randomly initialized baselines, with the benefit of additional pretraining steps being larger when more labeled data is available.

Significance. If the empirical claims hold after verification of operating-point equivalence, the work would provide concrete evidence that domain-specific pretraining improves transformer performance on an imbalanced security classification task in the low-FPR regime. The emphasis on controlled ablations over label and pretraining budgets is a methodological strength that could guide future studies on pretraining for DNS and related security detection problems.

major comments (2)
  1. [Controlled pipeline (abstract and methods)] The central claim of clean ablations and interpretable left-tail ROC gains rests on the assumption that validation-derived thresholds produce equivalent realized FPRs on the test set across model variants. DNS traffic exhibits temporal, domain, and query-volume shifts; without reported verification (e.g., measured test FPRs at the transferred thresholds or explicit shift statistics), the comparison between pretrained and randomly initialized models risks confounding by non-equivalent operating points.
  2. [Abstract] The abstract asserts 'significant improvements' and that 'increasing the number of pretraining steps helps the most' but supplies no dataset sizes, exact TPR/FPR values, error bars, or statistical tests. This absence prevents assessment of the practical magnitude and reliability of the reported gains, which are load-bearing for the paper's empirical contribution.
minor comments (1)
  1. [Abstract] The abstract would be strengthened by including one or two key quantitative results (e.g., TPR at FPR=10^{-3}) to allow immediate evaluation of the claimed improvements.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. The comments highlight important aspects of ensuring robust empirical comparisons and clear communication of results. We address each major comment point by point below, with planned revisions to enhance the manuscript.

read point-by-point responses

  1. Referee: [Controlled pipeline (abstract and methods)] The central claim of clean ablations and interpretable left-tail ROC gains rests on the assumption that validation-derived thresholds produce equivalent realized FPRs on the test set across model variants. DNS traffic exhibits temporal, domain, and query-volume shifts; without reported verification (e.g., measured test FPRs at the transferred thresholds or explicit shift statistics), the comparison between pretrained and randomly initialized models risks confounding by non-equivalent operating points.

    Authors: We appreciate the referee's focus on verifying operating-point equivalence to support the ablation claims. Our controlled pipeline explicitly selects thresholds on the validation set to achieve target low FPRs and applies those identical thresholds to the test set for every model variant (pretrained and random-initialized). This design choice ensures that all comparisons occur under the same selection procedure, enabling interpretable differences attributable to pretraining rather than threshold choice. To further strengthen this, we will add explicit reporting of the realized test-set FPRs achieved by each model variant at the transferred thresholds, along with basic statistics on query-volume and domain shifts observed between validation and test splits if they are material. revision: yes

  2. Referee: [Abstract] The abstract asserts 'significant improvements' and that 'increasing the number of pretraining steps helps the most' but supplies no dataset sizes, exact TPR/FPR values, error bars, or statistical tests. This absence prevents assessment of the practical magnitude and reliability of the reported gains, which are load-bearing for the paper's empirical contribution.

    Authors: We agree that including concrete quantitative anchors in the abstract would help readers immediately gauge the scale and reliability of the gains. The current abstract prioritizes a high-level summary of the controlled pipeline and key trends; full dataset sizes, exact TPR values at low-FPR operating points, and variability across runs are reported in the methods and results sections. We will revise the abstract to incorporate representative numbers (e.g., dataset scale for fine-tuning, TPR at FPR = 0.001, and mention of multiple-run variability) while remaining within length limits. This will make the practical magnitude clearer without altering the abstract's focus. revision: yes

Circularity Check

0 steps flagged

Empirical ablation study with no circular derivations

full rationale

The paper is an empirical ML study that reports ROC improvements from in-domain BERT pretraining on DNS data, using a controlled pipeline that freezes validation-derived operating points for test-set evaluation across pretraining steps and label budgets. No equations, derivations, or load-bearing self-citations appear in the provided text that reduce any claimed result to a quantity defined by the inputs or fitted parameters on the same data. The ablations compare pretrained variants against random-initialization baselines via standard experimental controls, leaving the reported gains independent of the evaluation choices.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only abstract available; no explicit free parameters, axioms, or invented entities are stated or can be inferred.

pith-pipeline@v0.9.0 · 5431 in / 979 out tokens · 25744 ms · 2026-05-10T17:42:52.160182+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    Dns exfiltration detection in the presence of adversarial attacks and modified exfiltrator behaviour,

    K. Žiža, P. Tadić, and P. Vuletić, “Dns exfiltration detection in the presence of adversarial attacks and modified exfiltrator behaviour,”Int. J. Inf. Secur., vol. 22, no. 6, p. 1865–1880, 2023. [Online]. Available: https://doi.org/10.1007/s10207-023-00723-w

    work page doi:10.1007/s10207-023-00723-w 2023

  2. [2]

    Information-based heavy hitters for real-time dns data exfiltration detection and prevention,

    Y. Ozery, A. Nadler, and A. Shabtai, “Information-based heavy hitters for real-time dns data exfiltration detection and prevention,” 2023. [Online]. Available: https://arxiv.org/abs/ 2307.02614

    work page arXiv 2023

  3. [3]

    Dns exfiltration guided by generative adversarial networks,

    A. Fahim, S. Zhu, Z. Qian, C. Song, E. Papalexakis, S. Chakraborty, K. Chan, P. Yu, T. Jaeger, and S. V. Krish- namurthy, “Dns exfiltration guided by generative adversarial networks,” in2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P), 2024, pp. 580–599

    work page 2024

  4. [4]

    Netgpt: Generative pretrained transformer for network traffic,

    X. Meng, C. Lin, Y. Wang, and Y. Zhang, “Netgpt: Generative pretrained transformer for network traffic,”arXiv preprint arXiv:2304.09513, 2023. [Online]. Available: https: //arxiv.org/abs/2304.09513

    work page arXiv 2023

  5. [5]

    Flowtransformer: A transformer framework for flow-based network intrusion detection systems,

    L. D. Manocchio, S. Layeghy, W. W. Lo, G. K. Kulatilleke, M. Sarhan, and M. Portmann, “Flowtransformer: A transformer framework for flow-based network intrusion detection systems,”

    work page

  6. [6]

    Available: https://arxiv.org/abs/2304.14746

    [Online]. Available: https://arxiv.org/abs/2304.14746

    work page arXiv

  7. [7]

    Domurls_bert: Pre-trained bert-based model for malicious domains and urls detection and classification,

    A. E. Mahdaouy, S. Lamsiyah, M. J. Idrissi, H. Alami, Z. Yartaoui, and I. Berrada, “Domurls_bert: Pre-trained bert-based model for malicious domains and urls detection and classification,” 2024. [Online]. Available: https://arxiv.org/abs/ 2409.09143

    work page arXiv 2024

  8. [8]

    Malicious dns detection by combining improved transformer and cnn,

    H. Li, Z. Li, S. Zhang, and X. Pu, “Malicious dns detection by combining improved transformer and cnn,”Scientific Reports, vol. 14, 12 2024

    work page 2024

  9. [9]

    Dom-bert: Detecting malicious domains with pre-training model,

    Y. Tian and Z. Li, “Dom-bert: Detecting malicious domains with pre-training model,” inPassive and Active Measurement: 25th International Conference, PAM 2024, Virtual Event, March 11–13, 2024, Proceedings, Part I. Berlin, Heidelberg: Springer-Verlag, 2024, p. 133–158. [Online]. Available: https: //doi.org/10.1007/978-3-031-56249-5_6

    work page doi:10.1007/978-3-031-56249-5_6 2024

  10. [10]

    Dns exfiltration dataset,

    K. Ziza, P. Vuletić, and P. Tadić, “Dns exfiltration dataset,”

    work page

  11. [11]

    Available: https://doi.org/10.17632/c4n7fckkz3.3

    [Online]. Available: https://doi.org/10.17632/c4n7fckkz3.3

    work page doi:10.17632/c4n7fckkz3.3

  12. [12]

    Subdomain Statistics from scanner.ducks.party,

    nyuuzyou, “Subdomain Statistics from scanner.ducks.party,” https://huggingface.co/datasets/nyuuzyou/subdomains, 2025

    work page 2025