Pith. sign in

REVIEW 5 major objections 8 minor 31 references

RawMal-TF: Raw Malware Dataset Labeled by Type and Family

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper introduces RawMal-TF, a public dataset pairing raw malware binaries with type and family labels, and reports 98.98% family-detection accuracy.

desk verdict A real, useful raw-PE dataset whose label validity — AV-name family parsing and ClarAVy type tags — is never checked, making the benchmark numbers a measure of labeling consistency rather than ground truth. read the letter →

arxiv 2506.23909 v1 pith:JFK22SOL submitted 2025-06-30 cs.CR cs.LG

classification cs.CRcs.LG
keywords malwareclassificationrawdatasettypeandfamilylabelsstaticPEanalysismachinelearningmulticlassbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that malware classification suffers from a shortage of datasets that provide both raw binaries and two levels of labels, type and family. It introduces RawMal-TF, a collection of 14 malware types and 17 malware families assembled from public repositories, with binaries shipped alongside extracted static features. Using a standard Windows PE feature pipeline, the authors show that Random Forest and XGBoost distinguish these malware from benign files at 98.5% (type) and 98.98% (family) accuracy, while multiclass type assignment reaches 81.1% with SVM. If the dataset and labels are reliable, researchers gain a reusable raw-binary resource for custom feature extraction, adversarial testing, and fine-grained malware analysis.

What carries the argument

The load-bearing object is the dual-label construction pipeline: an automated behavioral-labeling tool assigns type tags from aggregated antivirus scan data, and filename substring parsing assigns family names from archived malware filenames. On top of this sits a standardized static feature extraction pipeline for Windows Portable Executable files, drawn from an established benchmark, which turns every binary into a fixed-length vector of header, histogram, entropy, string, section, import, and export features. The pipeline is what lets the same labeled binaries support binary, interclass, and multiclass classification tasks, and what makes the raw binaries re-usable for future feature engineering.

What would settle it

Randomly sample, say, 200 family-labeled binaries from RawMal-TF and 200 type-labeled binaries, then have two independent analysts or an alternative automated labeler assign family and type from the binaries themselves; if agreement between the dataset's labels and the independent labels falls well below the reported accuracies, the headline numbers largely measure agreement with a noisy labeling source.

Watch

Extended reading notes

Core claim

The central claim is that a raw malware dataset labeled at both behavioral type and family granularities can be built entirely from existing public sources without paid APIs, and that static PE features extracted from it support strong classification. Type labels come from precomputed behavioral tags generated by an automated antivirus-label parser applied to large unlabeled malware archives; family labels are parsed from filenames in redistributed malware archives. After filtering to categories with at least 1,000 samples, the dataset contains 14 types and 17 families. Experiments across binary, interclass, and multiclass tasks show that malware-versus-benign detection is nearly saturated, while finer-grained distinctions remain harder: family multiclass accuracy peaks near 73.4% and type multiclass at 81.1%.

Load-bearing premise

The dataset's value rests on the labels being true: family names parsed from filenames and behavioral types from automated antivirus-label parsing must actually describe what each binary does.

Editorial extensions

If this is right

  • Any researcher can download the raw binaries and re-extract features with newer techniques rather than being locked into precomputed vectors.
  • Type-level and family-level labels allow studying how label granularity affects detection, interclass separability, and multiclass accuracy.
  • Reported near-perfect binary detection suggests static PE features alone are sufficient to separate these families from benign files.
  • The multiclass gaps (81.1% types, 73.4% families) point to specific confusable pairs, such as worm/auto-run and Taskun/Agensla, as targets for richer features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the labels are sound, RawMal-TF could serve as a testbed for adversarial perturbations of raw binaries, since the raw bytes are provided rather than only vectors.
  • A natural extension the paper does not perform is label-quality auditing: comparing filename-derived family labels against an independent labeling source on a sample subset.
  • The observed confusion patterns suggest type boundaries such as worm vs. auto-run may be intrinsically fuzzy, so future taxonomies might treat some categories as multi-label rather than exclusive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces RawMal-TF, a dataset of Windows PE malware binaries labeled at two granularities: 14 behavioral types derived from ClarAVy BEH tags on VirusShare samples, and 17 families parsed from filenames in VX Underground's MalwareBazaar archives. The authors extract EMBER static features from the raw binaries, release both binaries and feature vectors, and evaluate five classifiers (KNN, MLP, SVM, XGBoost, Random Forest) on malware-vs-benign detection, pairwise interclass discrimination, and multiclass type/family classification. Reported results include 98.5% and 98.98% average accuracy for full-scope type- and family-based binary detection, 97.5% and 93.7% average pairwise accuracy for inter-type and inter-family tasks, and 81.1% (SVM) and 73.4% (RF/XGBoost) multiclass accuracy for types and families, respectively.

Significance. If the labels are trustworthy, RawMal-TF addresses a genuine gap: most public malware benchmarks provide only feature vectors, whereas this work releases raw binaries alongside dual-level type/family labels and EMBER-compatible features. The public GitHub release, the transparent description of the feature-processing pipeline, and the detailed per-class performance tables are useful contributions. However, the central value of the dataset depends on label correctness, and the paper does not independently validate either the type labels or the family labels. The label-generation pipeline contains an unspecified multi-tag assignment rule, inconsistent category-count arithmetic, and a temporal mismatch between 2018 benign samples and 2023-2025 malware. These issues mean the reported accuracies should currently be interpreted as agreement with the labeling procedure rather than as validation against ground truth. If the labeling concerns are resolved, the resource would be a solid community asset; in the present form, the core dataset claim needs additional support.

major comments (5)
  1. [Section 3.3.1, Listing 1] The paper does not specify how samples carrying multiple BEH tags are assigned to the single-label type categories used in the experiments. The example record contains both BEH:adware and BEH:downloader, each supported by six engines; if such samples are placed in multiple category directories, the class directories overlap and the multiclass experiments in Section 4.3 are invalid because one sample can appear in more than one class. Please state the assignment rule (e.g., first tag, majority vote, or exclusion) and report how many of the retained samples had multiple BEH tags.
  2. [Section 3.3.2] Family labels are obtained by parsing antivirus detection names from filenames such as 'UDS-Backdoor.Win32.Bladabindi.dej', with no independent verification. Antivirus naming conventions vary by vendor, and one family can appear under multiple names (njRAT and Bladabindi refer to the same family), while some names are generic or vendor-specific. The exclusion of generic names is described, but no check is provided that the remaining parsed names correspond to the intended families. Please add a validation step on a random subset using an independent source (e.g., MalwareBazaar tags, YARA rules, or a second labeling tool) and report agreement rates; without this, the family-level accuracies measure consistency with the parsing rule rather than correctness of the family labels.
  3. [Section 3.3.1, Table 1] The category-count arithmetic is internally inconsistent. The text reports 76 unique behavioral categories, 56 categories excluded for having fewer than 1,000 samples, and 17 categories with no samples in the selected archives, yet 14 categories are retained; 76 - 56 - 17 = 3, and 76 - 56 = 20, leaving six categories unaccounted for. In addition, Table 1 includes BEH adware with 967 samples, below the stated 1,000-sample threshold. Please reconcile these numbers and specify which categories were retained at each filtering step so that Table 1 can be reproduced exactly.
  4. [Section 4.1.1 Step 3 and Section 4.2] The benign samples come from EMBER 2018, whereas the malware samples were collected from 2023-2025 (Section 3.3). This temporal mismatch is a possible confound in the malware-vs-benign experiments: models may separate malware from benign files based on compiler toolchains, packers, or PE-layout changes over time rather than on malicious behavior. Please report a time-matched control (e.g., benign PE files from the same period) or analyze the extent of the distribution shift, and temper the corresponding accuracy claims accordingly.
  5. [Section 4.3, Tables 15 and 18] The high-level multiclass accuracy claims should be qualified by the per-class results. Type-based multiclass accuracy of 81.1% is accompanied by per-class recall of 49% for worm, 51% for virus, and 57% for downloader; family-based multiclass accuracy of 73.4% is accompanied by recall of 35% for Noon and 41% for Agensla. These low-recall classes are central to malware taxonomy, so the abstract's characterization of the results as enabling 'more fine-grained and insightful malware classification' goes beyond what the reported per-class metrics support. Please report macro-averaged results prominently and discuss the class-dependent difficulty explicitly.
minor comments (8)
  1. [Section 1, last paragraph] The text says 'Section 1 introduces the necessary background,' but the background appears in Section 2; the cross-reference should be corrected.
  2. [Throughout] The name ClarAVy is typeset inconsistently as 'ClarA Vy' in the Abstract and Section 3.3.1; the official spelling should be used uniformly.
  3. [Listing 1] The meaning of the integer after the hash (e.g., '53', '54') is not explained in the text; clarify whether it is the total number of antivirus detections for that sample.
  4. [Table 18] The columns 'Avg FP' and 'Avg FN' are labeled ambiguously; they appear to be counts rather than rates, but the text does not state this, and Table 15 uses FPR/FNR percentages. The units should be unified or explicitly defined.
  5. [Section 3.3.2] The exact family-name extraction rule (substring matching vs. regular expression, and which token of the filename is taken) is not specified; providing the parser pseudocode or a regex pattern would make the dataset construction reproducible.
  6. [Section 4.1.1 Step 3] The benign samples are EMBER 2018 feature vectors, not raw benign binaries; the dataset description should state this limitation explicitly, since the abstract claims that raw binaries are included alongside features and labels.
  7. [Section 4.2.2, Table 8] The arrow notation in misclassification rates (e.g., 'autorun↔ worm 20% (autorun), 44.5% (worm)') is ambiguous about which class is the true class; clarify the direction for each reported pair.
  8. [Section 4.3.1] The text states the type-based multiclass dataset contains approximately 14,000 samples, but Table 1 lists BEH adware with only 967 samples; the exact total should be stated.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the dataset's labels are external AV-derived metadata, and the feature extraction and classification pipeline is independent of the labeling procedure, so the accuracy results are not forced by construction.

full rationale

The paper's central claims are that RawMal-TF is a new dataset with raw binaries, family labels parsed from filenames, type labels from ClarAVy, and that classifiers reach the reported accuracies. Checking the derivation chain: (1) family labels come from parsing externally supplied antivirus filenames such as 'UDS-Backdoor.Win32.Bladabindi.dej' (Section 3.3.2), and type labels are taken from precomputed ClarAVy BEH: tags (Section 3.3.1, Listing 1); neither label is defined in terms of the extracted EMBER features or the models' outputs, so the classification accuracy is not circular. (2) The features are extracted independently from the binaries via EMBER's PEFeatureExtractor() after the labels are fixed, and the models are trained on those features, so no fitted parameter is renamed as a prediction. (3) The paper has no self-citations and invokes no uniqueness theorem; the only reuse of another dataset is benign samples drawn from EMBER 2018 with verified labels, which is external and legitimate support rather than circularity. (4) The models are evaluated on held-out test splits generated after random shuffling, not on the labeling source. The real weaknesses are external validity issues, not circularity: family labels inherit AV-vendor naming conventions, ClarAVy behavioral tags may be noisy, and the paper does not specify how multi-tag records such as 'BEH:adware|6,BEH:downloader|6' are collapsed into the single type labels used in the multiclass experiments (Section 3.3.1). Those concerns affect whether the labels mean what the paper claims, but they do not make the derivations assume their own conclusions. Accordingly, no circular step meeting the evidentiary bar is present, and the score reflects only the minor label-source validation caveat.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on label reliability and representativeness assumptions rather than on fitted equations; no new entities are introduced.

free parameters (3)
  • minimum category size threshold = 1,000 samples
    Categories with fewer than 1,000 samples were excluded for both types and families, shaping the final 14-type/17-family dataset and influencing class balance and measured accuracy (Section 3.3.1, 3.3.2).
  • truncated dataset size = 1,000 malware + 1,000 benign
    Used for controlled comparisons; this hand-chosen size affects reported limited-data accuracy (Section 4.2.1).
  • feature selection k (binary tasks) = chosen among {5,10,20,30,40,50,100}
    Best k selected by validation accuracy, so reported accuracies depend on this data-driven choice (Section 4.2).
assumptions (4)
  • domain assumption PE filenames in VX Underground/MalwareBazaar exports encode correct malware family names.
    Family labels are assigned by parsing filenames, with no independent verification (Section 3.3.2).
  • domain assumption ClarAVy behavioral tags from AV scan reports are accurate enough to serve as type ground truth.
    Type labels are taken from precomputed ClarAVy tags without manual validation (Section 3.3.1).
  • domain assumption EMBER 2018 benign samples are representative of benign Windows binaries in 2023-2025.
    Benign samples come from EMBER 2018, while malware is from 2023-2025; any temporal or distribution shift inflates detection accuracy (Section 4.1 Step 3).
  • domain assumption The EMBER static feature extractor produces features that are sufficient and consistent for these binaries.
    All classification is based on EMBER PE features; packed or unusual binaries may not be well represented (Section 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RawMal-TF: Raw Malware Dataset Labeled by Type and Family." pith.science (2026). https://pith.science/paper/JFK22SOL

@misc{pith2026250623909,
  author       = {Pith},
  title        = {Pith review of: RawMal-TF: Raw Malware Dataset Labeled by Type and Family},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFK22SOL}},
  note         = {Machine review of arXiv:2506.23909}
}
read the original abstract

This work addresses the challenge of malware classification using machine learning by developing a novel dataset labeled at both the malware type and family levels. Raw binaries were collected from sources such as VirusShare, VX Underground, and MalwareBazaar, and subsequently labeled with family information parsed from binary names and type-level labels integrated from ClarAVy. The dataset includes 14 malware types and 17 malware families, and was processed using a unified feature extraction pipeline based on static analysis, particularly extracting features from Portable Executable headers, to support advanced classification tasks. The evaluation was focused on three key classification tasks. In the binary classification of malware versus benign samples, Random Forest and XGBoost achieved high accuracy on the full datasets, reaching 98.5% for type-based detection and 98.98% for family-based detection. When using truncated datasets of 1,000 samples to assess performance under limited data conditions, both models still performed strongly, achieving 97.6% for type-based detection and 98.66% for family-based detection. For interclass classification, which distinguishes between malware types or families, the models reached up to 97.5% accuracy on type-level tasks and up to 93.7% on family-level tasks. In the multiclass classification setting, which assigns samples to the correct type or family, SVM achieved 81.1% accuracy on type labels, while Random Forest and XGBoost reached approximately 73.4% on family labels. The results highlight practical trade-offs between accuracy and computational cost, and demonstrate that labeling at both the type and family levels enables more fine-grained and insightful malware classification. The work establishes a robust foundation for future research on advanced malware detection and classification.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

  1. [1]

    Computers & Security 81, 123– 147 (2019) https://doi.org/10.1016/j.cose

    Ucci, D., Aniello, L., Baldoni, R.: Survey of machine learning techniques for malware analysis. Computers & Security 81, 123– 147 (2019) https://doi.org/10.1016/j.cose. 2018.11.001

  2. [2]

    Machine Learning, Deep Learning and AI for Cyber- security, 113–142 (2025)

    Dˇ ed, L., Jureˇ cek, M.: Selecting representa- tive samples from malware datasets. Machine Learning, Deep Learning and AI for Cyber- security, 113–142 (2025)

  3. [3]

    ArXiv e-prints (2018) arXiv:1804.04637 [cs.CR]

    Anderson, H.S., Roth, P.: EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. ArXiv e-prints (2018) arXiv:1804.04637 [cs.CR]

  4. [4]

    In: Companion of the 16th ACM/SPEC Interna- tional Conference on Performance Engineer- ing (WWW Companion ’25) (2025)

    Joyce, R.J., Everett, D., Fuchs, M., Raff, E., Holt, J.: Claravy: A tool for scalable and accurate malware family labeling. In: Companion of the 16th ACM/SPEC Interna- tional Conference on Performance Engineer- ing (WWW Companion ’25) (2025)

  5. [5]

    Online; cit

    VirusShare: VirusShare.com. Online; cit. 2025-02-28. https://virusshare.com/

  6. [6]

    Online; cit

    VX Underground: VX Underground. Online; cit. 2025-03-18. https://vx-underground. org/

  7. [7]

    Online; cit

    Abuse.ch: MalwareBazaar. Online; cit. 2025- 03-02. https://bazaar.abuse.ch/browse/ 30

  8. [8]

    Online; cit

    VirusTotal: VirusTotal. Online; cit. 2025-03-

Show all 31 references
  1. [9]

    Nist interagency/internal report 7621 rev

    Paulsen, C., Toth, P.: Small business infor- mation security: The fundamentals. Nist interagency/internal report 7621 rev. 1, National Institute of Standards and Tech- nology, Gaithersburg, MD (November 2016). https://doi.org/10.6028/NIST.IR.7621r1

  2. [10]

    2025-03-15

    Belcic, I.: What Is Malware and How to Protect Against Malware Attacks? Online; cit. 2025-03-15. https://www.avast.com/c-malware

  3. [11]

    arXiv preprint arXiv:2407.12793v1; Online; cit

    Huang, Q., Zhao, T.: Data Collection and Labeling Techniques for Machine Learning. arXiv preprint arXiv:2407.12793v1; Online; cit. 2025-04-01 (2024). https://doi.org/10. 48550/arXiv.2407.12793 . https://arxiv.org/ abs/2407.12793

  4. [12]

    Online; cit

    Lanier, K.: Worm Malware Foren- sic Analysis. Online; cit. 2025-02-25. https://westoahu.hawaii.edu/cyber/ forensics-weekly-executive-summmaries/ worm-malware-forensic-analysis

  5. [13]

    Online; cit

    Malwarebytes: Downloader. Online; cit. 2025-03-10. https://www.malwarebytes. com/glossary/downloader

  6. [14]

    Technical Report NIST Special Publication 800-83 Revision 1, National Institute of Standards and Technol- ogy, Gaithersburg, MD (July 2013)

    Souppaya, M., Scarfone, K.: Guide to mal- ware incident prevention and handling for desktops and laptops. Technical Report NIST Special Publication 800-83 Revision 1, National Institute of Standards and Technol- ogy, Gaithersburg, MD (July 2013). https: //doi.org/10.6028/NIST....

  7. [15]

    Technical Report Draft NISTIR 8374, National Institute of Standards and Technol- ogy, Gaithersburg, MD (September 2021)

    Barker, W.C., Scarfone, K., Fisher, W., Souppaya, M.: Cybersecurity framework profile for ransomware risk management. Technical Report Draft NISTIR 8374, National Institute of Standards and Technol- ogy, Gaithersburg, MD (September 2021). https://doi.org/10.6028/NIST.IR.8374-d...

  8. [16]

    Online; cit

    National Institute of Standards and Tech- nology: Spyware. Online; cit. 2025- 03-10. https://csrc.nist.gov/glossary/term/ spyware

  9. [17]

    Online; cit

    Kaspersky: What Is Adware? – Def- inition and Explanation. Online; cit. 2025-04-05. https://www.kaspersky.com/ resource-center/threats/adware

  10. [18]

    https://doi.org/10.48550/ arXiv.2111.15031

    Joyce, R.J., Amlani, D., Nicholas, C., Raff, E.: MOTIF: A Large Malware Ref- erence Dataset with Ground Truth Fam- ily Labels (2021). https://doi.org/10.48550/ arXiv.2111.15031

  11. [19]

    Online; cit

    Trend Micro: Exploring Emotet’s Activ- ities. Online; cit. 2025-04-12 (n.d.). https://documents.trendmicro.com/assets/ white papers/ExploringEmotetsActivities Final.pdf

  12. [20]

    2025-04-15

    Baker, K.: What Is TrickBot Malware? Online; cit. 2025-04-15. https://www. crowdstrike.com/en-us/cybersecurity-101/ malware/trickbots/

  13. [21]

    Origi- nally published 2007; updated 2011; Online; cit

    Microsoft: Win32/Allaple. Origi- nally published 2007; updated 2011; Online; cit. 2025-04-18. https://www. microsoft.com/en-us/wdsi/threats/ malware-encyclopedia-description?Name= Win32/Allaple&threatId=

  14. [22]

    Online; cit

    Hunt.io: Emotet. Online; cit. 2025-04-20. https://hunt.io/malware-families/emotet

  15. [23]

    Online; cit

    KnowBe4: Virlock Ransomware. Online; cit. 2025-04-22. https://www.knowbe4.com/ ransomware-knowledgebase/virlock

  16. [24]

    Origi- nally published 2007; updated 2017; Online; cit

    Microsoft: Win32/Virut. Origi- nally published 2007; updated 2017; Online; cit. 2025-04-18. https://www. microsoft.com/en-us/wdsi/threats/ malware-encyclopedia-description?Name= Win32/Virut

  17. [25]

    https://www.virustotal.com/

  18. [26]

    Online; cit

    Hunt.io: njRAT. Online; cit. 2025-04-27. https://hunt.io/malware-families/njrat

  19. [27]

    Online; 31 cit

    F-Secure: Trojan-Spy:W32/Zbot. Online; 31 cit. 2025-04-29. https://www.f-secure.com/ v-descs/trojan-spy-w32-zbot.shtml

  20. [28]

    In: Research in Attacks, Intru- sions, and Defenses: 19th International Sym- posium, RAID 2016, Paris, France, Septem- ber 19-21, 2016, Proceedings 19, pp

    Sebasti´ an, M., Rivera, R., Kotzias, P., Caballero, J.: Avclass: A tool for massive mal- ware labeling. In: Research in Attacks, Intru- sions, and Defenses: 19th International Sym- posium, RAID 2016, Paris, France, Septem- ber 19-21, 2016, Proceedings 19, pp. 230–253 (2016). Springer

  21. [29]

    Avail- able: https://github.com/lief-project/LIEF

    LIEF: [Online; accessed May 22, 2025]. Avail- able: https://github.com/lief-project/LIEF

  22. [30]

    Online; cit

    scikit-learn developers: SelectKBest. Online; cit. 2025-04-01. https://scikit-learn. org/stable/modules/generated/sklearn. feature selection.SelectKBest.html

  23. [31]

    Online; cit

    scikit-learn developers: Support Vector Machines (SVM). Online; cit. 2025-04-14. https://scikit-learn.org/stable/modules/ svm.html 32

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.