REVIEW 3 major objections 7 minor 2 cited by
EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read EMBER2024 gives malware researchers one benchmark with 3.2M files, six formats, seven label types, and a challenge set of files that initially evaded every antivirus engine.
desk verdict Genuinely useful benchmark artifact that fills a real gap; core claims hold, but a few fixable errors and one substantive concern about cross-week near-duplicate leakage in the challenge set should be addressed before publication. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-stage VirusTotal labeling protocol. Each file is queried within 24 hours of first submission and again 90 or more days later; files suspected to be benign are re-scanned at least 30 days after submission. Zero detected products at the later scan labels a file benign, five or more independent detections labels it malicious, and files that were undetected at first submission but malicious by the later scan form the challenge set. This temporal delay turns antivirus signature updates into a signal for evasive malware. The other central object is EMBER feature version 3, a 2,568-dimensional vector that extends the prior PE-focused feature format and still extracts general, string, byte-histogram, and byte-entropy features from non-PE or unparsable files, so one representation can cover Win32, Win64, .NET, APK, ELF, and PDF files.
What would settle it
Manually analyze a random sample of files EMBER2024 labels benign by executing them in an isolated sandbox and inspecting network, filesystem, and process behavior; if a nontrivial fraction turn out to be malicious, the benign labels and the challenge-set definition are biased. A cheaper check is to rerun the labeling protocol with an independent second scanner or a stricter detection threshold and see whether reported benchmark rankings on the challenge set change.
Extended reading notes
Core claim
The authors introduce EMBER2024 as the first public malware benchmark that is simultaneously multi-platform, multi-label, and evasive-aware. Using antivirus verdicts taken within 24 hours of first submission and again 90 or more days later, they label files as benign if no product detects them, malicious if five or more independent products agree, and as a challenge set if the file was undetected at first submission but malicious by re-scan. Baseline gradient-boosted tree classifiers reach roughly 0.995 ROC AUC on the ordinary temporal test set, but their precision-recall scores collapse on the challenge set—for all files combined the PR AUC is 0.5722, and for APK files it falls to 0.2374—which the paper reads as evidence that evasive malware is a measurable, still-open problem. The companion feature version 3 vector, together with released construction code, is what lets this single benchmark cover multiple formats and tasks.
Load-bearing premise
The entire label set—benign, malicious, and challenge—rests on the assumption that antivirus verdicts collected after 30 or more days are reliable ground truth, so any malicious file that no antivirus product ever flags is silently counted as benign.
Editorial extensions
If this is right
- A researcher can benchmark malware detection, family identification, behavior prediction, packer identification, file-property prediction, exploit tagging, and threat-group attribution on the same train/test split and the same feature format.
- The challenge set gives a compact standard evaluation for evasive malware; the paper's all-files model achieves PR AUC 0.9971 on the ordinary test set but only 0.5722 on the challenge set.
- The fixed weekly time split makes concept drift directly testable, including the 75 families that appear in the test set but never in training.
- Because the feature extractor and dataset-construction pipeline are released, researchers can rerun the methodology on new time windows rather than depending on a static snapshot.
- Classifiers trained on reduced features for ELF, PDF, and APK still reach high AUC on ordinary test sets, so the multi-format feature approach is viable beyond Windows PE files.
Reading between the lines
- A natural extension the paper does not pursue is per-format challenge sets; the APK challenge PR AUC of 0.2374 suggests evasion mechanisms differ by platform and deserve separate study.
- The delayed-detection criterion could be graded—files first detected after 30, 90, or 180 days may form an evasion difficulty ladder, letting researchers measure how much longer a classifier resists newer malware.
- Because only hashes, metadata, and feature vectors are released, not raw file bytes, byte-level deep-learning detectors and adversarial perturbation experiments cannot be run directly on EMBER2024; its holistic scope is mainly for static-feature classifiers.
- Strong AUC on a temporal split can coexist with weak performance on evasive files, so future benchmark reports that cite EMBER2024 should report challenge-set precision-recall alongside standard AUC.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents EMBER2024, a new large-scale benchmark dataset for malware classification, containing 3,238,315 files collected over 64 weeks from six file formats (Win32, Win64, .NET, APK, ELF, PDF). The dataset provides seven types of labels and tags, supports tasks including malware detection, family classification, and behavior prediction, and includes a 'challenge set' of 6,315 malicious files that were initially undetected by all VirusTotal AV engines but later detected. The paper also introduces EMBER feature version 3, releases code for dataset construction and model training, and reports baseline LightGBM results. The central claim is that this is the first holistic benchmark combining multi-format support, multi-task labels, and an evasive-malware evaluation subset, with reproducible construction methodology.
Significance. If the dataset construction is sound, EMBER2024 would be a significant contribution: it addresses the lack of recent, multi-format, multi-task malware benchmarks and is the first to offer a dedicated evasive-malware challenge set. The release of construction code is a strong reproducibility feature, and the baseline results provide a useful reference. However, the paper contains concrete numerical errors and an under-specified aspect of the challenge set's construction that could bias evaluation. These issues are fixable, but they are load-bearing for the paper's reliability.
major comments (3)
- [Table 3 and abstract] Table 3 contains an arithmetic error: the PDF row reports a total of 12,805, but its components (52,000 + 12,000 + 805) sum to 64,805. The challenge-set column in Table 3 sums to 6,291, not the stated 6,315, and the file-type totals sum to 3,238,291, not the 3,238,315 given in the abstract and Section 1.1. The paper must correct these numbers and explain the 24-file discrepancy.
- [Section 4.1 vs Table 5] Section 4.1 reports a ROC AUC of 0.9949 for the LightGBM classifier trained on the full training set, while Table 5 reports 0.9969 for the 'All files / All files' entry. Since Section 4.2 says all classifiers use the same hyperparameters described in Section 4.1, these numbers should match unless there were separate training runs. Please clarify the source of the discrepancy and, if multiple runs were performed, report the variability.
- [Section 2.3] The challenge-set near-duplicate exclusion only covers files from the same week of data collection. Because the training and test sets span 64 weeks, a challenge file may have a TLSH near-duplicate (distance of 30 or less) in a different week's portion of the training set, which would allow a classifier to exploit that near-duplicate rather than generalize to evasive malware. The paper does not report any cross-week TLSH distance analysis between the challenge set and the training/test sets. Please provide this analysis (for example, the distribution of minimum TLSH distances from each challenge file to any train/test file) or justify why cross-week near-duplicates are unlikely. This is essential to support the claim that the challenge set measures robustness against evasive malware.
minor comments (7)
- [Section 2.4] The benign label is defined as no AV detection after re-scanning at least 30 days later. This is a noisy ground-truth assumption that also affects the challenge-set definition. The paper should explicitly acknowledge the risk that never-detected malware is labeled benign and that the challenge set depends on the same AV threshold. A short limitations discussion would improve the paper.
- [Figure 3] The JSON example is missing a comma after the 'md5' value.
- [Section 3.3] The text says the implementation switched from 'LEIF' to pefile; this should be 'LIEF'.
- [Table 7] The column header 'weighed averaging' should be 'weighted averaging'.
- [References] Reference [20] is incomplete; it lacks the publication venue.
- [Section 4.3] The statement that ROC AUC scores in Table 6 do not accurately reflect model performance due to class imbalance is imprecise; ROC AUC is generally invariant to class imbalance, whereas PR AUC is not. Please clarify the intended meaning.
- [Throughout] The text uses 'A V' with a space in several places; please use 'AV' consistently.
Circularity Check
No circular derivation: EMBER2024's claims are dataset properties and benchmark measurements, not predictions reduced to their inputs.
full rationale
The paper's central claims are descriptive and empirical: it constructs a dataset, describes its labels, and reports benchmark classifier scores. There is no derivation chain in which a fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The challenge set is defined by VirusTotal detection status (initially undetected, later detected), and the benign/malicious labels use the same AV-detection criterion; this makes the challenge set a selected subpopulation, not a circular prediction. The statement that classifiers struggle on the challenge set is a measurement against that selection, not an equivalence forced by construction. The family and behavior labels come from the authors' own ClarA Vy tool, which is a disclosed self-citation; however, the paper does not argue for the labels' correctness by citing itself, and the benchmark tasks measure classifier agreement with those labels rather than deriving the labels from the classifiers. Similarly, the near-duplicate exclusion being limited to same-week files is a possible leakage limitation but not a circularity, because it does not make the evaluation score equal to the construction criterion by definition. Overall, the novelty claims and benchmark results are self-contained and externally checkable; the only adjacent concerns are validity risks, not circular reasoning.
Assumptions & free parameters
free parameters (5)
- Malicious AV detection threshold =
5 or more independent AV detections
- Benign AV detection threshold =
0 detections after re-scan at 30+ days
- TLSH near-duplicate distance threshold =
30
- Re-scan delay for label stabilization =
30 days minimum for benign files, 90 days for all files
- Weekly file quotas per format =
Win32 30k, Win64 10k, .NET 5k, APK 4k, ELF 500, PDF 1k per week
assumptions (4)
- domain assumption VirusTotal AV detection counts are a valid proxy for file maliciousness
- domain assumption ClarA Vy family and tag labels are sufficiently accurate for benchmarking
- domain assumption Files first submitted to VirusTotal are representative of malware and benign files encountered in practice
- standard math TLSH distance of 30 identifies near-duplicates with negligible false positive rate
invented entities (2)
-
EMBER2024 challenge set
independent evidence
-
EMBER feature version 3
independent evidence
Cite this review
Pith. "Pith review of EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers." pith.science (2026). https://pith.science/paper/KQFLU3F3
@misc{pith2026250605074,
author = {Pith},
title = {Pith review of: EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers},
year = {2026},
howpublished = {\url{https://pith.science/paper/KQFLU3F3}},
note = {Machine review of arXiv:2506.05074}
}
read the original abstract
A lack of accessible data has historically restricted malware analysis research, and practitioners have relied heavily on datasets provided by industry sources to advance. Existing public datasets are limited by narrow scope - most include files targeting a single platform, have labels supporting just one type of malware classification task, and make no effort to capture the evasive files that make malware detection difficult in practice. We present EMBER2024, a new dataset that enables holistic evaluation of malware classifiers. Created in collaboration with the authors of EMBER2017 and EMBER2018, the EMBER2024 dataset includes hashes, metadata, feature vectors, and labels for more than 3.2 million files from six file formats. Our dataset supports the training and evaluation of machine learning models on seven malware classification tasks, including malware detection, malware family classification, and malware behavior identification. EMBER2024 is the first to include a collection of malicious files that initially went undetected by a set of antivirus products, creating a "challenge" set to assess classifier performance against evasive malware. This work also introduces EMBER feature version 3, with added support for several new feature types. We are releasing the EMBER2024 dataset to promote reproducibility and empower researchers in the pursuit of new malware research topics.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 2 Pith papers
-
Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach
PL-HCL detects cross-layer misalignment in Agent Skills by learning consistency among metadata, instructions, and resources, lifting Macro-F1 to 0.87–0.89 on a human-verified challenge set.
-
Latent Stability Analysis of Malware Representations Under Feature-Space Perturbations
Latent-stability metrics (LED and PINNFlow shifts) diagnose malware representation movement under feature-space probes, without beating full EMBER or PCA-64 on clean classification.
Reference graph
Works this paper leans on
-
[1]
[n. d.]. VirusShare.com - Because Sharing is Caring. https://virusshare.com/, Last accessed on 2025-02-17
work page 2025
-
[2]
Hyrum S Anderson and Phil Roth. 2018. Ember: an open dataset for training static pe malware machine learning models. arXiv preprint arXiv:1804.04637 (2018)
arXiv 2018
-
[3]
G. Apruzzese, H. S. Anderson, S. Dambra, D. Freeman, F. Pierazzi, and K. Roundy
-
[4]
Marcus Botacin, Fabricio Ceschin, Paulo De Geus, and André Grégio. 2020. We need to talk about antiviruses: challenges & pitfalls of av evaluations. Computers & Security 95 (2020), 101859
work page 2020
-
[5]
Ero Carrera. 2004. pefile: Python module to read and work with PE files. https: //github.com/erocarrera/pefile
work page 2004
-
[6]
William W Cohen. 1995. Fast effective rule induction. In Proceedings of the twelfth international conference on machine learning. 115–123
work page 1995
-
[7]
Colin Galen and Robert Steele. 2020. Evaluating performance maintenance and deterioration over time of machine learning-based malware detection models on the ember pe dataset. In 2020 Seventh International Conference on Social Networks Analysis, Management and Security (SNAMS). IEEE, 1–7
work page 2020
-
[8]
Richard Harang and Ethan M Rudd. 2020. SOREL-20M: A large scale benchmark dataset for malicious PE detection. arXiv preprint arXiv:2012.07634 (2020)
arXiv 2020
Show all 32 references
-
[9]
Wenyi Huang and Jack W Stokes. 2016. MtNet: a multi-task neural network for dynamic malware classification. In Detection of Intrusions and Malware, and Vulnerability Assessment: 13th International Conference, DIMVA 2016, San Sebastián, Spain, July 7-8, 2016, Proceedings 13. Sp...
2016
-
[10]
Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro
Roberto Jordaney, Kumar Sharad, Santanu K. Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2017. Transcend: Detecting Concept Drift in Malware Classification Models. In 26th USENIX Security Symposium (USENIX Security 17). USENIX Association, Vancouver, ...
2017
-
[11]
Robert J Joyce, Dev Amlani, Charles Nicholas, and Edward Raff. 2023. Motif: A malware reference dataset with ground truth family labels. Computers & Security 124 (2023), 102921
2023
-
[12]
Joyce, Derek Everett, Maya Fuchs, Edward Raff, and James Holt
Robert J. Joyce, Derek Everett, Maya Fuchs, Edward Raff, and James Holt. 2025. ClarA Vy: A Tool for Scalable and Accurate Malware Family Labeling. InCompan- ion of the 16th ACM/SPEC International Conference on Performance Engineering (WWW Companion ’25)
2025
-
[13]
Robert J Joyce, Edward Raff, Charles Nicholas, and James Holt. 2023. MalDICT: Benchmark Datasets on Malware Behaviors, Platforms, Exploitation, and Packers. arXiv preprint arXiv:2310.11706 (2023)
2023 arXiv
-
[14]
Joseph, and J
Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D. Joseph, and J. D. Tygar. 2015. Better Malware Ground Truth: Techniques for Weighting Anti-Virus Vendor Labels. In Proceedings of the 8th ACM Workshop on Artificial I...
2015
-
[15]
Jeremy Z Kolter and Marcus A Maloof. 2004. Learning to detect malicious executables in the wild. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 470–478
2004
-
[16]
Nicola Loi, Claudio Borile, and Daniele Ucci. 2021. Towards an automated pipeline for detecting and classifying malware through machine learning. arXiv preprint arXiv:2106.05625 (2021)
2021 arXiv
-
[17]
Microsoft Security Team. 2021. Attack AI Systems in Machine Learning Evasion Competition. https://www.microsoft.com/en-us/security/blog/2021/07/29/attack- ai-systems-in-machine-learning-evasion-competition/
2021
-
[18]
Aziz Mohaisen, Omar Alrawi, and Manar Mohaisen. 2015. AMAL: High-fidelity, behavior-based automated malware analysis and classification. Computers & Security 52 (2015), 251 – 266. doi:10.1016/j.cose.2015.04.001
2015 doi
-
[19]
Jonathan Oliver, Chun Cheng, and Yanggui Chen. 2013. TLSH–a locality sensitive hash. In 2013 Fourth Cybercrime and Trustworthy Computing Workshop. IEEE, 7–13
2013
-
[20]
Tirth Patel, Fred Lu, Edward Raff, Charles Nicholas, Cynthia Matuszek, and James Holt. 2023. Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits! (2023). EMBER2024 - A Benchmark Dataset for Holistic Evaluation of Malware Classifiers KDD’25, August 3–7, 2025,...
2023
-
[21]
Edward Raff and Charles Nicholas. 2020. A survey of machine learning methods and challenges for windows malware classification. arXiv preprint arXiv:2006.09271 (2020)
2020 arXiv
-
[22]
Phil Roth. 2019. EMBER Improvements. (2019). https://docs.google.com/ presentation/d/1A13tsUkgWeujTy9SD-vDFfQp9fnIqbSE_tCihNPlArQ Confer- ence on Applied Machine Learning in Information Security
2019
-
[23]
Joshua Saxe and Konstantin Berlin. 2015. Deep neural network based malware detection using two dimensional binary program features. In Malicious and Un- wanted Software (MALWARE), 2015 10th International Conference on . IEEE, 11–20
2015
-
[24]
Matthew G Schultz, Eleazar Eskin, F Zadok, and Salvatore J Stolfo. 2001. Data mining methods for detection of new malicious executables. In Security and Privacy, 2001. S&P 2001. Proceedings. 2001 IEEE Symposium on. IEEE, 38–49
2001
-
[25]
Marcos Sebastián, Richard Rivera, Platon Kotzias, and Juan Caballero. 2016. Av- class: A tool for massive malware labeling. InResearch in Attacks, Intrusions, and Defenses: 19th International Symposium, RAID 2016, Paris, France, September 19-21, 2016, Proceedings 19. Springer, 230–253
2016
-
[26]
John Seymour. [n. d.]. label-virusshare. https://github.com/seymour1/label- virusshare, Last accessed on 2025-02-14
2025
-
[27]
Wei Song, Xuezixiang Li, Sadia Afroz, Deepali Garg, Dmitry Kuznetsov, and Heng Yin. 2021. MAB-Malware: A Reinforcement Learning Framework for Attacking Static Malware Classifiers. arXiv:2003.03100 [cs.CR] https://arxiv.org/ abs/2003.03100
2021 arXiv
-
[28]
Romain Thomas. 2017. LIEF - Library to Instrument Executable Formats. https://lief.quarkslab.com/
2017
-
[29]
Xabier Ugarte-Pedrero, Mariano Graziano, and Davide Balzarotti. 2019. A close look at a daily dataset of malware samples. ACM Transactions on Privacy and Security (TOPS) 22, 1 (2019), 1–30
2019
-
[30]
VirusTotal. [n. d.]. Analyse suspicious files, domains, IPs and URLs to detect mal- ware and other breaches, automatically share them with the security community. https://www.virustotal.com/en/, Last accessed on 2025-02-18
2025
-
[31]
histogram
Shao-Huai Zhang, Cheng-Chung Kuo, and Chu-Sing Yang. 2019. Static PE Malware Type Classification Using Machine Learning Techniques. In 2019 In- ternational Conference on Intelligent Computing and its Emerging Applications (ICEA). 81–86. doi:10.1109/ICEA.2019.8858297 KDD’25, Au...
2019
-
[2023]
Real attackers don’t compute gradients
“Real attackers don’t compute gradients”: Bridging the gap between ad- versarial ML research and practice. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 339–364
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.