Pith. sign in

REVIEW 4 major objections 7 minor 48 references

EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability

T0 review · 4 major / 7 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read When accuracy, time drift, attacks, and speed are scored together, hand-crafted features still beat most deep networks for real-world Windows malware detection.

desk verdict Solid multi-axis malware-detector benchmark; GBDT wins the joint score, but the #1 vs #2 gap is thin and mostly rides on a timestamp-proxied temporal metric. read the letter →

arxiv 2607.24177 v1 pith:6UN44APS submitted 2026-07-27 cs.CR cs.LG

Andrea Ponte , Daniel Gibert , Matous Kozak , Dmitrijs Trizna , Maura Pintor , Battista Biggio , Fabio Roli , Luca Demetrio This is my paper · ORCID
classification cs.CRcs.LG
keywords malwaredetectionbenchmarkWindowsPEmachinelearningdeepadversarialrobustnessconceptdriftinferencecost
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing comparisons of AI Windows malware detectors cannot decide what to deploy: they train on different data, ignore concept drift, skip systematic adversarial testing, and overlook inference cost on ordinary endpoints. EXE-Bench fixes that by training thirty models on identical data and ranking them with one average of four metrics—immediate F1, sample-weighted F1 on later time bins, robustness under growing content-injection budgets, and inverse inference time. Under this ranking the classic gradient-boosted tree on EMBER features comes first, while most end-to-end deep networks look strong only right after training and then fall behind. The result matters because it shows practitioners which detectors remain usable once time and attacks enter the picture, rather than which ones merely top a static test set.

What carries the argument

EXE-Bench score S(f)=(P+T+R+I)/4: P is F1 on the held-out test set, T is the sample-weighted average F1 across future four-month bins, R is the normalized area under the detection-rate-versus-perturbation-budget curve, and I=exp(−average inference time). This single scalar produces the leaderboard.

What would settle it

Replace PE timestamps with reliable first-seen dates for most Speakeasy samples, recompute the temporal metric on the same thirty models, and check whether the EMBER gradient-boosted tree still tops the four-metric leaderboard.

Watch

Extended reading notes

Core claim

When thirty AI-based Windows malware detectors are trained and tested under identical conditions and scored jointly on immediate performance, stability under temporal drift, resistance to adversarial content injection, and inference speed, the gradient-boosting model that uses hand-engineered EMBER features achieves the highest aggregate score. Most end-to-end deep networks excel mainly on the deployment-time test set and degrade under later data or under modest byte manipulations, while image-based and randomized-smoothing variants rank near the bottom.

Load-bearing premise

The ranking over time treats PE compilation timestamps (after discarding obvious outliers) as a good enough proxy for when each sample first appeared, even though those timestamps can be forged and external first-seen coverage is sparse.

Editorial extensions

If this is right

  • Practitioners should prefer feature-engineered models over raw-byte deep networks when deployment must survive drift and content-injection attacks.
  • Image-based malware classifiers should be avoided in production under these four criteria.
  • De-randomized smoothing can improve speed and sometimes rank, but does not automatically confer robustness to section-injection attacks.
  • Accuracy-only leaderboards systematically over-rank models that overfit the training-time distribution.
  • New detectors can be dropped into the same four-metric average without redesigning the comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same multi-axis scoring template could be applied to Android or Linux detectors to test whether feature engineering retains the same edge outside PE files.
  • If PE-timestamp noise is the main bias, the relative ordering of feature-based versus end-to-end models may stay stable once cleaner time labels arrive, because both families see identical bins.
  • Certification schemes that slice files into chunks look more practical for endpoint latency than randomized smoothing, so research effort may pay off more by raising their clean accuracy than by adding further noise-based certificates.
  • The gap between immediate F1 and temporal/robustness scores implies continuous re-training will remain necessary for deep models even when feature models can be updated more cheaply.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces EXE-Bench, a benchmark that re-trains 30 AI-based static Windows malware detectors (feature-based GBDT, end-to-end byte CNNs, an image classifier, and randomized/de-randomized smoothing certifiable variants) on one retrieved EMBER-2017 split and ranks them by an equally-weighted average S=(P+T+R+I)/4 of: F1 at deployment (P, at 1% FPR where thresholds are tunable), a novel sample-cardinality-weighted F1 over 10 four-month temporal bins from the Speakeasy dataset (T), a normalized AUC of detection rate vs. Levenshtein perturbation budget over three black-box genetic attacks (R), and an exponential of CPU inference time (I). The headline finding is that EmberGBDT tops the leaderboard (S=0.86 vs BBDnn 0.85), supporting the claim that feature engineering still beats end-to-end deep learning for deployable detectors, while certifiable and image-based models fare poorly. Code and an interactive reweighting dashboard are released. Statistical support includes Kendall W=0.48 (p=0.0021) across metric rankings and a leave-one-out rank-correlation sensitivity analysis.

Significance. If the central ordering holds, this is a useful community artifact: a controlled, apples-to-apples comparison of 30 Windows malware detectors with released code and an interactive reweighting dashboard, a principled sample-weighted temporal metric, explicit attack budgets with security-evaluation curves, and a realistic CPU inference protocol. The finding that certifiable defenses collapse under multi-axis evaluation, and that image-based detectors are both brittle and drift-prone, is genuinely informative for deployment decisions. The paper is also unusually candid about its limitations (timestomping, missing attacks, truncated inputs for RS models). However, the marquee claim — GBDT as the single winner — currently rests on a 0.01 margin whose deciding component (T) is the least-validated axis, so the practical significance is closer to "GBDT and BBDnn dominate, with the ordering dependent on deployment weights" than the abstract suggests.

major comments (4)
  1. [§V-A, Table I, Eq. (5)] The headline result ("feature engineering remains superior", GBDT ranked #1) is load-bearing on the temporal metric and on equal weighting, and the margin is 0.01. From Table I: GBDT vs BBDnn is nearly tied on P (0.99/0.97) and R (0.80/0.79), BBDnn wins I (0.98/0.87), and GBDT's edge is entirely T (0.78/0.67). Simple arithmetic shows the #1 slot flips under plausible changes: averaging only P, R, I gives BBDnn 0.913 vs GBDT 0.887. Meanwhile T is the least-secured axis (forgeable PE timestamps, sparse VT validation per Tab. IV). The paper's own LOO analysis (Tab. V) does not test the top of the leaderboard, only global rank correlation. The authors should (i) report a sensitivity analysis of the top-3 ordering to re-binning (e.g., quarterly vs 4-month windows, different outlier thresholds), timestamp perturbation, and metric weight perturbations around the equal-weight point; and (ii) tem
  2. [§IV-A, Fig. 3, Table IV] The temporal protocol removes an apparently very large fraction of Speakeasy as outliers (Fig. 3's "Past Out." bar dwarfs the retained bins, on the order of half the 125,921 samples), and the retained bins are extremely uneven (roughly 192 to 18,864 samples per bin per class). Since T weights bins by |Si|/|D| (Eq. 1), a few dominant bins — whose timestamps are exactly the ones most likely to be build-farm artifacts or timestomped values that survived the outlier filter — effectively set T for every model. Table IV shows the VT cross-check cannot adjudicate this: 33% presence, ~7% non-null first-seen, and 45% of first submissions after collection. Please report exact discarded counts and per-bin class composition, and quantify T's stability (e.g., bootstrap over bins, or T recomputed excluding the two largest bins). Without this, the reader cannot tell whether the GBDT–BBDnn T gap (0.11)
  3. [§III (Eqs. 2-3), §IV-C, §V-D] R values in Table I mix two different evaluation regimes: non-certifiable models are attacked directly, while certifiable models are evaluated only via transfer attacks (§IV-C), because direct attacks were computationally infeasible. Presenting both in one column invites misreading (e.g., BBDnnRS R=0.62 ranking above directly-attacked models may reflect attack non-transfer rather than robustness; cf. BBDnnRS's odd P=0.67). Additionally: (a) the normalization of R by the global maximum ε across all models couples each model's score to the hardest attack run against any model — adding a model or attack to the benchmark changes existing R values, which should be stated; (b) GAMMA with λ=0 admits empty .rdata section injections counted toward the perturbation budget; (c) robustness is measured on only 500 unpacked EMBER test samples with a single 500-query GA budget and no white-box evaluati
  4. [§IV-B, §V-A] All metrics are computed from a single random 80/10/10 split and a single training run per model, with no variance estimates anywhere (Figs. 5-7, Tables I-II). Neural network rankings near the top (e.g., MalConv vs NGramConv on P; the T gap itself) could plausibly be within seed variance. Given that the benchmark's purpose is a durable ordering, at least the key claims (GBDT vs BBDnn on T and R) should be supported by repeated runs or split-level confidence intervals. Relatedly, Kendall W=0.48 among four metric rankings is invoked to "justify" aggregation into S, but concordance among metrics does not justify equal weights — these are distinct statements; please rephrase or support the weighting choice directly.
minor comments (7)
  1. [Table I] The rows are not sorted by S: e.g., BBDnnRsDel (0.56) appears above MalConv (0.67), BBDnnRDRS/SDRS (0.68/0.69) above several 0.65-0.67 entries. Bold/tie marking is also inconsistent (e.g., I=0.99 ties across several models are variously bolded). Please re-sort and re-check.
  2. [Notation/typos] Model naming oscillates between BBDNN (§IV-B) and BBDnn/BBDnn (tables/figures); also "subject toconcept drift" (missing space) in §I, and Table I has run-together values ("0.520.99"). Table IV's column alignment is hard to parse.
  3. [Fig. 4] Radar plots are nearly unreadable at this density; consider a per-metric heatmap or parallel-coordinates plot, and mark the top-3 models.
  4. [§III, Inference Metric] I = exp(-t) squashes differences above ~1s (BBDnnRsDel I=0.05 corresponds to ~3s; the distinction between 3s and 30s is deployment-relevant but nearly invisible in I). The footnote mentions a linear variant; please report in an appendix whether the leaderboard changes under it.
  5. [§IV-D] The inference-timing sample is restricted to file sizes within µ±0.5σ of Speakeasy (§IV-D), which excludes large files where feature extraction and input-window differences matter most. A sentence on how results change with k (or on the full size distribution) would strengthen the I metric.
  6. [§V-D] BBDnn's robustness is partly attributed to its 100KB input window letting injected content fall outside the model's view (§V-D) — this is an important caveat that deserves prominence in the main results, since it means R partly rewards input truncation rather than robustness.
  7. [§VI] EMBER2024 [42] is cited as future work but, given the timestamp concerns with Speakeasy, a brief discussion of whether EMBER2024's temporal splits would give a better-validated T axis would be valuable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EXE-Bench is an empirical multi-metric ranking, not a derivation that forces its headline by construction.

full rationale

The paper’s central claim—that EMBER GBDT tops an aggregate of performance, temporal stability, adversarial robustness, and inference cost under identical data/hardware—is obtained by measuring four independently defined quantities (P = F1 on a fixed test set; T = cardinality-weighted F1 over temporal bins, Eq. 1; R = normalized AUC of detection rate vs. perturbation budget, Eqs. 2–3; I = exp(−t_i), Eq. 4) and averaging them (S, Eq. 5). None of these equations defines a metric in terms of the ranking outcome, fits a free parameter to the target ordering, or imports a uniqueness theorem that forbids alternatives. Self-citations (e.g., Demetrio et al. for GAMMA/FullDOS/Content-shift; Gibert et al. for certifiable variants) supply attack and model implementations used as experimental methods; they do not underwrite the leaderboard by construction. Timestamp-based binning and the small S gap (0.86 vs 0.85) are external-validity / sensitivity concerns, not circular reductions. The work is self-contained against its stated evaluation protocol; honest finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 7 assumptions · 2 invented entities

The central ranking rests on standard ML evaluation practice plus several domain and design choices: identical EMBER training, Speakeasy as future distribution, PE timestamps as time labels, a fixed menu of black-box problem-space attacks, CPU inference on size-filtered samples, and uniform averaging of four normalized metrics. No new physical entities are postulated; free choices are metric design and experimental knobs rather than fitted physical constants.

free parameters (6)
  • Equal metric weights in S=(P+T+R+I)/4 = 1/4 each
    Aggregation treats performance, temporal, robustness, and inference as equally important; LOO shows inference moves ranks most, so the default average is a design choice that shapes the winner.
  • Detection threshold at 1% FPR (non-certifiable models) = 1% FPR
    Fixes operating point for F1 in P and T; certifiable models keep majority vote, slightly advantaged then rebalanced via F1.
  • Attack query budget and population size = 500 queries, pop=10
    Genetic black-box attacks capped at 500 queries and 10 variants per iteration; strength of R depends on this budget.
  • Temporal bin width and PE-timestamp outlier rules = 4-month bins; discard past/future outliers
    Four-month bins from 2019-Q1 to 2022-Q1 after dropping past/future outliers define T; binning is a free experimental choice.
  • Inference sample size filter k=0.5σ = k=0.5
    Times measured on 1000 files within μ±0.5σ of Speakeasy size distribution; changes which models look fast.
  • GAMMA section counts and λ=0 = λ=0; up to 50 sections
    Injected section counts (5–50) and zero size penalty maximize perturbation; defines the right tail of the security curve.
assumptions (7)
  • domain assumption i.i.d. training on EMBER 2017 retrieved binaries is a fair shared starting point for comparing static detectors
    §IV-A: all models share the same 80/10/10 split of ~750k retrieved PE files.
  • domain assumption Speakeasy (2019–2022) represents relevant post-deployment concept drift for models trained on ≤2017 data
    §IV-A Temporal Analysis; vendor/era coverage may not match all production streams (§VI).
  • domain assumption PE header timestamps (after outlier removal) are an acceptable time axis when VT first-seen is mostly missing or post-collection
    §IV-A and Tab. IV; authors argue alignment with prior best practices [18] despite timestomping risk.
  • ad hoc to paper Robustness can be summarized as normalized AUC of detection rate vs Levenshtein byte budget over FullDOS, Content-shift, and GAMMA
    §III Robustness Metric Eqs. (2)–(3); attack menu is chosen, not exhaustive (§VI Missing attacks).
  • ad hoc to paper Inference utility is well captured by I=exp(-t_i) on CPU without GPU
    §III Inference Metric Eq. (4); exponential form is a modeling choice (linear alternative said not to change results).
  • domain assumption Functionality-preserving problem-space manipulations are the right threat model for static PE detectors
    §II Adversarial EXEmples; standard in this literature [19]–[22].
  • standard math F1 (and weighted F1 over bins) is an adequate scalar for detection quality at the chosen threshold
    §III Performance and Temporal metrics; standard classification practice.
invented entities (2)
  • EXE-Bench composite score S and four-metric leaderboard protocol independent evidence
    purpose: Single comparable ranking of Windows static ML detectors across deployability axes
    Defined in §III Eqs. (1)–(5) and Fig. 2; the paper’s main methodological object.
  • Sample-cardinality-weighted temporal metric T independent evidence
    purpose: Avoid AUC fallacy when temporal bins have unequal sample counts
    §III Eq. (1) and Appendix A contrast with unweighted AUC used in prior drift work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability." pith.science (2026). https://pith.science/paper/6UN44APS

@misc{pith2026260724177,
  author       = {Pith},
  title        = {Pith review of: EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UN44APS}},
  note         = {Machine review of arXiv:2607.24177}
}
read the original abstract

Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.

Figures

Figures reproduced from arXiv: 2607.24177 by the authors.

Figure 1
Figure 1. PE File Format. in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Design of EXE-Bench. Models are trained on the same data (1), evaluated through the lenses of test-time performance, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Temporal bins of Speakeasy Dataset in quarters, each [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Radar plots summarizing the results of EXE-Bench, ordering models by their score (S). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: ROC curves of the top-six models according to the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Security evaluations of the top-six models in terms of [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Boxplot with average (dot), median (line), and standard [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 1 linked inside Pith

  1. [1]

    EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models,

    H. S. Anderson and P. Roth, “EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models,”ArXiv e-prints, Apr. 2018

    2018

  2. [2]

    Deep neural network based malware detection using two dimensional binary program features,

    J. Saxe and K. Berlin, “Deep neural network based malware detection using two dimensional binary program features,” in10th Int. Conf. on Malicious and Unwanted Software (MALWARE), 2015, pp. 11–20

    2015

  3. [3]

    Novel feature extraction, selection and fusion for effective malware family classification,

    M. Ahmadi, D. Ulyanov, S. Semenov, M. Trofimov, and G. Giacinto, “Novel feature extraction, selection and fusion for effective malware family classification,” inProc. of 6th ACM Conference on Data and Application Security and Privacy, 2016, p. 183–194

    2016

  4. [4]

    Malware detection by eating a whole EXE,

    E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. K. Nicholas, “Malware detection by eating a whole EXE,” inWorkshop of the The 32nd AAAI Conference on Art. Int., 2018

    2018

  5. [5]

    Deep convolutional malware classifiers can learn from raw executables and labels only,

    M. Kr ˇc´al, O. ˇSvec, M. B ´alek, and O. Ja ˇsek, “Deep convolutional malware classifiers can learn from raw executables and labels only,” inICLR 2018 Workshop, 2018

    2018

  6. [6]

    Activation analysis of a byte-based deep neural network for malware classification,

    S. E. Coull and C. Gardner, “Activation analysis of a byte-based deep neural network for malware classification,” inIEEE Security and Privacy Workshops (SPW), 2019

    2019

  7. [7]

    Auditing static machine learning anti-malware tools against metamorphic attacks,

    D. Gibert, C. Mateu, J. Planes, and J. Marques-Silva, “Auditing static machine learning anti-malware tools against metamorphic attacks,” Computers & Security, vol. 102, p. 102159, 2021

    2021

  8. [8]

    Malware images: visualization and automatic classification,

    L. Nataraj, S. Karthikeyan, G. Jacob, and B. S. Manjunath, “Malware images: visualization and automatic classification,” in8th Int. Sympo- sium on Visualization for Cyber Security (VizSec). ACM, 2011, p. 4

    2011

Show all 48 references
  1. [9]

    Using convolutional neural networks for classification of malware represented as images,

    D. Gibert, C. Mateu, J. Planes, and R. Vicens, “Using convolutional neural networks for classification of malware represented as images,”J. Comput. Virol. Hacking Tech., vol. 15, no. 1, pp. 15–28, 2019

    2019

  2. [10]

    RS-Del: Edit distance robustness certificates for sequence classifiers via randomized deletion,

    Z. Huang, N. Marchant, K. Lucas, L. Bauer, O. Ohrimenko, and B. I. P. Rubinstein, “RS-Del: Edit distance robustness certificates for sequence classifiers via randomized deletion,” inAdv. in Neural Information Processing Systems, 2023

    2023

  3. [11]

    Towards a practical defense against adversarial attacks on deep learning-based malware detectors via ran- domized smoothing,

    D. Gibert, G. Zizzo, and Q. Le, “Towards a practical defense against adversarial attacks on deep learning-based malware detectors via ran- domized smoothing,” in28th Eu. Sym. on Research in Computer Security Workshops. Springer, 2023

    2023

  4. [12]

    Certified robustness of static deep learning-based malware detec- tors against patch and append attacks,

    ——, “Certified robustness of static deep learning-based malware detec- tors against patch and append attacks,” inProc. of 16th ACM Workshop on Art. Int. and Security, 2023, p. 173–184

    2023

  5. [13]

    Drsm: De- randomized smoothing on malware classifier providing certified robust- ness,

    S. Saha, W. Wang, Y . Kaya, S. Feizi, and T. Dumitras, “Drsm: De- randomized smoothing on malware classifier providing certified robust- ness,”ICLR 2024, 2024

    2024

  6. [14]

    Adversarial robustness of deep learning-based malware detectors via (de)randomized smoothing,

    D. Gibert, G. Zizzo, Q. Le, and J. Planes, “Adversarial robustness of deep learning-based malware detectors via (de)randomized smoothing,” IEEE Access, vol. 12, pp. 61 152–61 162, 2024

    2024

  7. [15]

    Certified adversarial robustness of machine learning-based malware detectors via (de)randomized smoothing,

    D. Gibert, L. Demetrio, G. Zizzo, Q. Le, J. Planes, and B. Biggio, “Certified adversarial robustness of machine learning-based malware detectors via (de)randomized smoothing,” 2024. [Online]. Available: https://arxiv.org/abs/2405.00392

    2024 arXiv

  8. [16]

    Certified adversarial robustness via randomized smoothing,

    J. Cohen, E. Rosenfeld, and J. Z. Kolter, “Certified adversarial robustness via randomized smoothing,” inProc. of 36th Int. Conf. on Machine Learning, vol. 97, 2019, pp. 1310–1320

    2019

  9. [17]

    (de)randomized smoothing for certifiable defense against patch attacks,

    A. Levine and S. Feizi, “(de)randomized smoothing for certifiable defense against patch attacks,” inAdv. in Neural Information Processing Systems, vol. 33, 2020, pp. 6465–6475

    2020

  10. [18]

    {TESSERACT}: Eliminating experimental bias in malware classifi- cation across space and time,

    F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro, “{TESSERACT}: Eliminating experimental bias in malware classifi- cation across space and time,” in28th USENIX sec. sym., 2019, pp. 729–746

    2019

  11. [19]

    Adversarial malware binaries: Evading deep learning for malware detection in executables,

    B. Kolosnjaji, A. Demontis, B. Biggio, D. Maiorca, G. Giacinto, C. Eckert, and F. Roli, “Adversarial malware binaries: Evading deep learning for malware detection in executables,” in26th European signal processing conference (EUSIPCO). IEEE, 2018, pp. 533–537

    2018

  12. [20]

    Malware makeover: Breaking ml-based static analysis by modifying executable bytes,

    K. Lucas, M. Sharif, L. Bauer, M. K. Reiter, and S. Shintre, “Malware makeover: Breaking ml-based static analysis by modifying executable bytes,” inProc. of ACM Asia Conference on Computer and Communi- cations Security, 2021, pp. 744–758

    2021

  13. [21]

    Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection,

    L. Demetrio, S. E. Coull, B. Biggio, G. Lagorio, A. Armando, and F. Roli, “Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection,” ACM Trans. Priv. Secur., vol. 24, no. 4, Sep. 2021

    2021

  14. [22]

    Functionality-preserving black-box optimization of adversarial win- dows malware,

    L. Demetrio, B. Biggio, G. Lagorio, F. Roli, and A. Armando, “Functionality-preserving black-box optimization of adversarial win- dows malware,”IEEE Tran. on Information Forensics and Security, vol. 16, pp. 3469–3478, 2021

    2021

  15. [23]

    Adversarial training for{Raw-Binary}malware classifiers,

    K. Lucas, S. Pai, W. Lin, L. Bauer, M. K. Reiter, and M. Sharif, “Adversarial training for{Raw-Binary}malware classifiers,” in32nd USENIX sec. sym., 2023, pp. 1163–1180

    2023

  16. [24]

    Wild patterns: Ten years after the rise of adversarial machine learning,

    B. Biggio and F. Roli, “Wild patterns: Ten years after the rise of adversarial machine learning,”Elsevier Pattern Recognition, vol. 84, pp. 317–331, 2018

    2018

  17. [25]

    Attackbench: Evaluating gradient-based attacks for adversarial examples,

    A. E. Cin `a, J. Rony, M. Pintor, L. Demetrio, A. Demontis, B. Biggio, I. B. Ayed, and F. Roli, “Attackbench: Evaluating gradient-based attacks for adversarial examples,” inProc. of AAAI Conference on Art. Int., vol. 39, no. 3, 2025, pp. 2600–2608

    2025

  18. [26]

    Robustbench: a standardized adversarial robustness benchmark,

    F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flammar- ion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,” in35th Conf. on Neural Information Processing Systems Datasets and Benchmarks Track, 2021

    2021

  19. [27]

    Intriguing properties of adversarial ml attacks in the problem space,

    F. Pierazzi, F. Pendlebury, J. Cortellazzi, and L. Cavallaro, “Intriguing properties of adversarial ml attacks in the problem space,” inIEEE symposium on security and privacy (SP). IEEE, 2020, pp. 1332–1349

    2020

  20. [28]

    Robust intelligent malware detection using deep learning,

    R. Vinayakumar, M. Alazab, K. P. Soman, P. Poornachandran, and S. Venkatraman, “Robust intelligent malware detection using deep learning,”IEEE Access, vol. 7, pp. 46 717–46 738, 2019

    2019

  21. [29]

    Optimized approaches to malware detection: A study of machine learning and deep learning techniques,

    A. Fahim, S. Dey, M. N. Absur, M. Kamrul Siam, M. T. Huque, and J. Jafor Godhuli, “Optimized approaches to malware detection: A study of machine learning and deep learning techniques,” inIEEE 14th Int. Conf. on Communication Systems and Network Technologies, 2025, pp. 269–275. 13

    2025

  22. [30]

    Evaluating realistic adversarial attacks against machine learning models for windows pe malware detection,

    M. Imran, A. Appice, and D. Malerba, “Evaluating realistic adversarial attacks against machine learning models for windows pe malware detection,”Future Internet, vol. 16, no. 5, 2024

    2024

  23. [31]

    A comparison of adversarial malware generators,

    P. Louth ´anov´a, M. Koz ´ak, M. Jure ˇcek, M. Stamp, and F. Di Troia, “A comparison of adversarial malware generators,”Journal of Computer Virology and Hacking Techniques, vol. 20, no. 4, pp. 623–639, 2024

    2024

  24. [32]

    The robust malware detection challenge and greedy random accelerated multi-bit search,

    S. Verwer, A. Nadeem, C. Hammerschmidt, L. Bliek, A. Al-Dujaili, and U.-M. O’Reilly, “The robust malware detection challenge and greedy random accelerated multi-bit search,” inProc. of 13th ACM Workshop on Art. Int. and Security, 2020, p. 61–70

    2020

  25. [33]

    Fast minimum-norm adversarial attacks through adaptive norm constraints,

    M. Pintor, F. Roli, W. Brendel, and B. Biggio, “Fast minimum-norm adversarial attacks through adaptive norm constraints,”Adv. in Neural Information Processing Systems, vol. 34, pp. 20 052–20 062, 2021

    2021

  26. [34]

    Quo vadis: hybrid machine learning meta-model based on contextual and behavioral malware representations,

    D. Trizna, “Quo vadis: hybrid machine learning meta-model based on contextual and behavioral malware representations,” inProc. of 15th ACM Workshop on Art. Int. and Security, 2022, pp. 127–136

    2022

  27. [35]

    Greedy function approximation: a gradient boosting machine,

    J. H. Friedman, “Greedy function approximation: a gradient boosting machine,”Annals of statistics, pp. 1189–1232, 2001

    2001

  28. [36]

    Lightgbm: A highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “Lightgbm: A highly efficient gradient boosting decision tree,”Adv. in Neural Information Processing Systems, vol. 30, 2017

    2017

  29. [37]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inIEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778

    2016

  30. [38]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Adv. in Neural Information Processing Systems, vol. 32, 2019

    2019

  31. [39]

    Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks,

    A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, and F. Roli, “Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks,” in28th USENIX sec. sym., 2019, pp. 321–338

    2019

  32. [40]

    The rise of machine learning for detection and classification of malware: Research developments, trends and challenges,

    D. Gibert, C. Mateu, and J. Planes, “The rise of machine learning for detection and classification of malware: Research developments, trends and challenges,”Elsevier Journal of Network and Computer Applications, vol. 153, p. 102526, 2020

    2020

  33. [41]

    Sorel-20m: A large scale benchmark dataset for malicious pe detection,

    R. Harang and E. M. Rudd, “Sorel-20m: A large scale benchmark dataset for malicious pe detection,”arXiv preprint, 2020

    2020

  34. [42]

    Ember2024-a benchmark dataset for holistic evaluation of malware classifiers,

    R. J. Joyce, G. Miller, P. Roth, R. Zak, E. Zaresky-Williams, H. Ander- son, E. Raff, and J. Holt, “Ember2024-a benchmark dataset for holistic evaluation of malware classifiers,” inProc. of 31st ACM SIGKDD Conf. on Knowledge Discovery and Data Mining V . 2, 2025, pp. 5516–5526

    2025

  35. [43]

    Beyond raw bytes: Towards large language models,

    L. Kurlandski, H. Berger, Y . Pan, and M. Wright, “Beyond raw bytes: Towards large language models,” inProc. of 33rd Network and Dis- tributed Systems (NDSS) Symposium, 2026

    2026

  36. [44]

    Guided malware sample analysis based on graph neural networks,

    Y .-H. Chen, S.-C. Lin, S.-C. Huang, C.-L. Lei, and C.-Y . Huang, “Guided malware sample analysis based on graph neural networks,” IEEE Transactions on Information Forensics and Security, vol. 18, pp. 4128–4143, 2023

    2023

  37. [45]

    Training robust ml-based raw-binary malware detectors in hours, not months,

    K. Lucas, W. Lin, L. Bauer, M. K. Reiter, and M. Sharif, “Training robust ml-based raw-binary malware detectors in hours, not months,” inProc. of 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, p. 124–138

    2024

  38. [46]

    Updating windows mal- ware detectors: Balancing robustness and regression against adversarial exemples,

    M. Kozak, L. Demetrio, D. Trizna, and F. Roli, “Updating windows mal- ware detectors: Balancing robustness and regression against adversarial exemples,”Elsevier Computers & Security, vol. 155, p. 104466, 2025

    2025

  39. [47]

    Mab- malware: A reinforcement learning framework for blackbox generation of adversarial malware,

    W. Song, X. Li, S. Afroz, D. Garg, D. Kuznetsov, and H. Yin, “Mab- malware: A reinforcement learning framework for blackbox generation of adversarial malware,” inProc. of ACM on Asia conference on computer and communications security, 2022, pp. 990–1003

    2022

  40. [48]

    Creating valid ad- versarial examples of malware,

    M. Koz ´ak, M. Jure ˇcek, M. Stamp, and F. D. Troia, “Creating valid ad- versarial examples of malware,”Springer Journal of Computer Virology and Hacking Techniques, vol. 20, no. 4, pp. 607–621, 2024. APPENDIX A: Area Under the Curve fallacyThe Area Under the Curve (AUC) quant...

    2024

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.