REVIEW 4 major objections 5 minor 24 references
Using Malware Detection Techniques for HPC Application Classification
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A Random Forest trained on SSDeep fuzzy-hash similarities can label HPC application executables, including unseen classes, with about 90% F1.
desk verdict The fuzzy-hashing idea is genuinely new for HPC executable classification, but the evaluation as described almost certainly leaks test information, so the 90% F1 is not believable without a version-based split and a restricted feature matrix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the SSDeep fuzzy hash, a context-triggered piecewise hash that produces a similarity score from 0 to 100 via the Damerau-Levenshtein edit distance between hash strings. It does the work of measuring how close an unseen executable is to known application samples despite code or version changes. The Random Forest Classifier consumes a matrix of these similarity scores across three features—raw binary, printable strings from the strings command, and global symbol names from the nm command—and a confidence threshold converts low-confidence predictions into the unknown label (-1). Balanced class weights counteract the dataset's heavy imbalance.
What would settle it
Collect a held-out set of user-compiled or stripped binaries from a production HPC system, run the trained Fuzzy Hash Classifier, and measure F1: if classification accuracy falls well below the reported 90% on these realistic workloads, the central claim does not transfer. Alternatively, feed known malicious binaries such as compiled cryptocurrency miners and check whether the unknown label reliably catches them.
Extended reading notes
Core claim
The central claim is that SSDeep fuzzy hashes of application executables preserve enough structural similarity across versions of the same application that a Random Forest classifier can label both known and unknown samples. Instead of cryptographic hashes, which only match identical files, SSDeep uses context-triggered piecewise hashing and compares hashes by Damerau-Levenshtein distance, so slightly modified versions of a program remain recognizable. The authors extract three fuzzy-hash features—raw binary content, printable strings, and global symbol names—and compute pairwise similarities against training samples. After an 80-20 split that holds out whole application classes as unknown and a stratified 60-40 split of the remaining classes, the Fuzzy Hash Classifier reports macro F1 of 90%, micro F1 of 89%, and weighted F1 of 90% over 92 classes and 5,333 samples. The symbol-table feature carries most of the predictive weight, consistent with function names staying stable across versions.
Load-bearing premise
The load-bearing premise is that the preinstalled, non-malicious executables collected for the dataset—all of which still contain their symbol table—are representative of what an HPC system actually needs to classify; user-compiled binaries, stripped binaries, and wrapper scripts are explicitly excluded.
Editorial extensions
If this is right
- If correct, HPC centers can label a running job's executable as a known application class or as unknown without collecting per-job resource-usage data or trusting user-supplied job names.
- The unknown label gives a concrete signal for deviation from allocation purpose: a user project suddenly executing executables outside its known set can be flagged for inspection.
- Because symbol-table function names dominate the feature importance, the method works best on non-stripped binaries; stripped executables and wrapper scripts such as bash or Python are outside its current reach.
- The approach is static, so it can run before or alongside dynamic resource-based classifiers, potentially reducing monitoring overhead.
- The reported F1 scores around 90% on 92 imbalanced classes suggest the method scales to a realistic multi-class setting, though classes with very few samples have noisy performance.
Reading between the lines
- A natural next test is to point the same classifier at known malicious binaries, such as compiled cryptocurrency miners; the paper's dataset intentionally excludes malicious software, so this would probe whether 'unknown' catches actual abuse rather than just unfamiliar scientific codes.
- The dominance of symbol-table features suggests stripped binaries could be handled by reconstructing function-level structure from dynamic call trees, an extension the authors name but do not implement; one could test whether the classification gap closes when such features replace nm output.
- The two-phase split mimics a deployment where only some application classes are known; in production, the confidence threshold would likely need site-specific tuning, since the paper reports that raising it trades unknown recall for known-class performance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a static, hash-based approach to classifying application executables on HPC systems. The authors extract SSDeep fuzzy hashes of three executable features (raw binary content, printable strings, and symbol-table function names), build a similarity feature matrix, and train a Random Forest classifier, which they call the Fuzzy Hash Classifier. Using 5,333 preinstalled application samples from the sciCORE cluster across 92 classes, the model reports micro F1 of 89%, macro F1 of 90%, and weighted F1 of 90%, including a manually defined '-1' unknown class formed from classes held out in an 80-20 class-level split. The paper argues that fuzzy hashing can recognize known applications and flag previously unseen ones, complementing dynamic resource-usage-based classification.
Significance. If the evaluation is sound, the paper offers a useful and low-overhead complement to dynamic HPC application classification: static executable fingerprints can be collected without runtime monitoring, avoid many privacy and integrity concerns of inspecting raw user files, and are cheap to compare. The dataset of 5,333 real preinstalled HPC executables with version structure is itself a valuable resource, and the explicit handling of completely unseen application classes is a relevant step toward production use. The approach is not conceptually novel — fuzzy hashing is standard in malware detection — but applying it to HPC workload classification and evaluating it on a production cluster dataset is a reasonable contribution. The reported results, however, rest on several evaluation-protocol details that must be clarified and corrected before the 90% F1 claim can be accepted.
major comments (4)
- [Section 3 (Feature Extraction and Fuzzy Hash Classifier)] The manuscript states that 'We compute a feature matrix for our dataset based on the SSDeep fuzzy hash similarity between sample features' and only afterward describes the two-phase train-test split. If this feature matrix is all-pairs over all 5,333 samples, then every training row contains similarity values to test samples, and the Random Forest's feature dimensions are indexed by the full dataset. A genuinely new executable could only be compared against the training library, so its feature vector would have fewer dimensions and different semantics; the reported 90% macro F1 would then reflect a transductive setup rather than classification of unseen applications. The authors must clarify whether the similarity matrix was computed only within the training set, and if not, rerun the evaluation with the feature matrix restricted to training-sample references.
- [Section 3 (Fuzzy Hash Classifier, second-phase split)] The second split is described as 'a stratified 60-40 train-test split on the samples,' not on versions. As Table 1 shows for Velvet, a single version can contain multiple executables (velveth and velvetg); a sample-level split can place executables from the same version in both training and test sets, inflating similarity because those executables were built from the same source and compiler configuration. The claim that the test set contains unseen applications is stronger than what this split guarantees. The authors should split by version (or by unique build) so that all samples from one version are assigned to the same fold, and report results under that split.
- [Section 3 (Unknown Samples and Confidence Threshold) and Section 4 (Confidence Threshold)] The training set of 2,688 samples is formed from the known classes only; the 20 unknown classes (852 test samples, Table 3) appear exclusively in the test set. The confidence threshold is said to be tuned 'as part of the hyperparameter grid search within the training set,' but it is unclear how the threshold can be optimized for an unknown class when no unknown-class samples exist in the training set. If the threshold was selected using the test set's unknown samples, that selection is a form of test-set leakage and the reported unknown-class performance is optimistically biased. The authors must specify exactly how the threshold was tuned and, ideally, use a validation split that contains held-out unknown classes.
- [Section 4 (Results) and Section 5 (Overall Performance)] The evaluation relies on a single, fixed train-test split with no cross-validation, repeated splits, or error bars. Given the high class imbalance (support values range from 1 to 352 in Table 4) and the large per-class variability (e.g., BWA recall 0.40, HTSlib F1 0.36, CAD-score precision 0.25), a single split is insufficient to establish that the 90% macro F1 is stable rather than an artifact of one particular split. The authors should report results over multiple re-splits (e.g., repeated stratified splits or cross-validation with class-level grouping) including variance, especially for the low-support classes.
minor comments (5)
- [Section 2 (Related Work)] Several references are malformed or incomplete: reference [15] reads 'Sean. Fingerprinting communication Peisert and computation on HPC machines' and [16] has the author name misplaced; reference [7] appears to omit co-authors. These should be corrected.
- [Section 6 (Conclusion)] There is a typo in '5‘333' (curly quote instead of apostrophe) and several headings contain stray spaces ('F uzzy', 'F eature', 'F uture'). Please proofread.
- [Section 5 (Feature Importance)] The text says 'Figure 5 shows the feature importance' but the feature importance is presented in Table 5; the figure/table citation should be corrected.
- [Section 5 (Limitations)] The limitations paragraph appropriately acknowledges that stripped binaries and wrapper scripts are excluded, but the introduction and conclusion frame the method as broadly applicable to HPC application classification. The paper should state more prominently that the evaluation covers preinstalled, symbol-table-intact executables only, and that transfer to user-compiled or stripped executables is untested.
- [General] No code or data artifacts are released. Given that the evaluation protocol is central to the paper's claims, releasing the dataset (or a representative subset) and the feature-construction pipeline would substantially increase confidence and reproducibility.
Circularity Check
No significant circularity: the fuzzy-hash classifier evaluation is an empirical supervised-learning study whose predictions do not reduce to the training inputs by construction.
full rationale
The paper makes an empirical claim—that a Random Forest trained on SSDeep similarity features can label HPC application executables, including held-out application classes—and evaluates it with a described train/test protocol. Labels are derived from directory paths, features are fuzzy hashes of file content, strings, and symbols, and the two-phase split is intended to include completely unseen classes. The confidence threshold is tuned within the training set, which is standard practice rather than a fitted parameter renamed as a prediction. The only self-citation is to the authors' earlier fingerprint-dictionary work [7], and it appears as related work rather than as load-bearing justification. No uniqueness theorem, ansatz, or renamed known result is invoked, and the acknowledged limitations (stripped binaries and wrapper scripts) affect external validity, not circularity. One passage, 'We compute a feature matrix for our dataset based on the SSDeep fuzzy hash similarity between sample features,' appears before the train-test split and may indicate that all-pairs similarities including test samples were used as features; if so, the reported 90% F1 would be a transductive-leakage validity threat. That is a serious evaluation concern, but the manuscript does not explicitly state that test labels or test-derived parameters entered training, so it is not a demonstrated circular reduction of the claimed result to its inputs.
Assumptions & free parameters
free parameters (3)
- confidence_threshold =
not reported (max combined F1 during grid search)
- random_forest_hyperparameters =
not reported (n_estimators, criterion, max_depth, min_samples_split, min_samples_leaf, max_features)
- class_weights =
inversely proportional to class frequencies
assumptions (5)
- domain assumption The directory root folder of each preinstalled application version is a correct application class label.
- domain assumption Executables with intact symbol tables are representative of HPC applications worth classifying.
- domain assumption SSDeep fuzzy hash similarity preserves application identity across versions and compilers.
- standard math A sample-level 60-40 split of known classes gives an unbiased estimate of performance on unseen versions.
- ad hoc to paper Low classifier confidence is an adequate operationalization of 'unknown' applications.
Cite this review
Pith. "Pith review of Using Malware Detection Techniques for HPC Application Classification." pith.science (2026). https://pith.science/paper/66NAEWD6
@misc{pith2026241118327,
author = {Pith},
title = {Pith review of: Using Malware Detection Techniques for HPC Application Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/66NAEWD6}},
note = {Machine review of arXiv:2411.18327}
}
read the original abstract
HPC systems face security and compliance challenges, particularly in preventing waste and misuse of computational resources by unauthorized or malicious software that deviates from allocation purpose. Existing methods to classify applications based on job names or resource usage are often unreliable or fail to capture applications that have different behavior due to different inputs or system noise. This research proposes an approach that uses similarity-preserving fuzzy hashes to classify HPC application executables. By comparing the similarity of SSDeep fuzzy hashes, a Random Forest Classifier can accurately label applications executing on HPC systems including unknown samples. We evaluate the Fuzzy Hash Classifier on a dataset of 92 application classes and 5333 distinct application samples. The proposed method achieved a macro f1-score of 90% (micro f1-score: 89%, weighted f1-score: 90%). Our approach addresses the critical need for more effective application classification in HPC environments, minimizing resource waste, and enhancing security and compliance.
Figures
Reference graph
Works this paper leans on
-
[1]
User environment tracking and problem detection with xalt
Kapil Agrawal, Mark R Fahey, Robert McLay, and Doug James. User environment tracking and problem detection with xalt. In 2014 First International Workshop on HPC User Support Tools , pages 32–40. IEEE, 2014
work page 2014
-
[2]
Taxonomist: Application detection through rich monitoring data
Emre Ates, Ozan Tuncer, Ata Turk, Vitus J Leung, Jim Brandt, Manuel Egele, and Ayse K Coskun. Taxonomist: Application detection through rich monitoring data. In Euro-Par 2018: Parallel Processing: 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27-31, 2018, Proceedings 24 , pages 92–105. Springer, 2018
work page 2018
-
[3]
Automatic application tuning for hpc architectures (dagstuhl seminar 13401)
Siegfried Benkner, Franz Franchetti, Hans Michael Gerndt, and Jeffrey K Hollingsworth. Automatic application tuning for hpc architectures (dagstuhl seminar 13401). In Dagstuhl Reports, volume 3. Schloss Dagstuhl-Leibniz- Zentrum fuer Informatik, 2014
work page 2014
-
[4]
Systematically inferring i/o performance variability by examining repetitive job behavior
Emily Costa, Tirthak Patel, Benjamin Schwaller, Jim M Brandt, and Devesh Tiwari. Systematically inferring i/o performance variability by examining repetitive job behavior. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1–15, 2021
work page 2021
-
[5]
A technique for computer detection and correction of spelling errors
Fred J Damerau. A technique for computer detection and correction of spelling errors. Communications of the ACM , 7(3):171–176, 1964
work page 1964
-
[6]
Code characterization with graph convolutions and capsule networks
Poornima Haridas, Gopinath Chennupati, Nandakishore Santhi, Phillip Romero, and Stephan Eidenbenz. Code characterization with graph convolutions and capsule networks. IEEE Access, 8:136307–136315, 2020
work page 2020
-
[7]
An execution fingerprint dictionary for hpc application recognition
Thomas Jakobsche, Nicolas Lachiche, Aur´ elien Cavelan, and Florina M Ciorba. An execution fingerprint dictionary for hpc application recognition. In 2021 IEEE International Conference on Cluster Computing (CLUSTER) , pages 604–608. IEEE, 2021
work page 2021
-
[8]
Identifying almost identical files using context triggered piecewise hashing
Jesse Kornblum. Identifying almost identical files using context triggered piecewise hashing. Digital investigation , 3:91–97, 2006. 10
work page 2006
Show all 24 references
-
[9]
Binary codes capable of correcting deletions, insertions, and reversals
Vladimir I Levenshtein et al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union, 1966
1966
-
[10]
Tracking Module Usage – Lmod 8.7.19 documentation
Robert. Tracking Module Usage – Lmod 8.7.19 documentation. 17 January 2023 McLay
2023
-
[11]
”US national agency computers misused to mine bitcoins”
BBC News Technology. ”US national agency computers misused to mine bitcoins”. 10 June 2014. https://www.bbc. com/news/technology-27779030
2014
-
[12]
An Action Plan for High Performance Computing Security
NIST Working Draft. An Action Plan for High Performance Computing Security. November 2016. https://www. nist.gov/system/files/documents/2018/03/15/working_draft_actionplanhpc.pdf
2016
-
[13]
”Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer”
Tiffany Trader. ”Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer”. 12 February 2018.https: //www.hpcwire.com/2018/02/12/russian-nuclear-engineers-caught-cryptomining-lab-supercomputer/
2018
-
[14]
Scikit-learn: Machine learning in python fabian
Fabian Pedregosa. Scikit-learn: Machine learning in python fabian. Journal of machine learning research , 12:2825, 2011
2011
-
[15]
Fingerprinting communication Peisert and computation on HPC machines
Sean. Fingerprinting communication Peisert and computation on HPC machines. 2010
2010
-
[16]
Security in high-performance computing environments
Sean. Security in high-performance computing environments. Communications of the ACM 60.9 (2017): 72-80. Peisert
2017
-
[17]
An accurate tool for modeling, fingerprinting, comparison, and clustering of parallel applications based on performance counters
Vitor Ramos, Carlos Valderrama, Samuel Xavier de Souza, and Pierre Manneback. An accurate tool for modeling, fingerprinting, comparison, and clustering of parallel applications based on performance counters. In 2019 IEEE International Parallel and Distributed Processing Sympos...
2019
-
[18]
sciCORE ( http://scicore.unibas.ch/) the scientific computing center at the University of Basel
Switzerland. sciCORE ( http://scicore.unibas.ch/) the scientific computing center at the University of Basel
-
[19]
”Europe’s supercomputers hijacked by attackers for crypto mining”
BBC News Technology. ”Europe’s supercomputers hijacked by attackers for crypto mining”. 18 May 2020. https: //www.bbc.com/news/technology-52709660
2020
-
[20]
https://github.com/spiffe/spire
SPIRE (the SPIFFE Runtime Environment). https://github.com/spiffe/spire
-
[21]
https://www.thecrimson.com/article/2014/2/20/harvard-odyssey-dogecoin/ Theodore R
2014. https://www.thecrimson.com/article/2014/2/20/harvard-odyssey-dogecoin/ Theodore R. Delwiche. Harvard Research Computing Resources Misused for ‘Dogecoin’ Mining Operation. February 20
2014
-
[22]
”Combing through the fuzz: Using fuzzy hashing and 2021
Edir Garcia Lazo. ”Combing through the fuzz: Using fuzzy hashing and 2021. https://www.microsoft.com/ security/blog/2021/07/27/combing-through-the-fuzz-using-fuzzy-hashing-and-deep-learning-to-counter-malware-detection-evasion-techniques/ deep learning to counter malware detec...
2021
-
[23]
Information retrieval
Cornelius Joost Van Rijsbergen. Information retrieval. 2nd. newton, ma, 1979
1979
-
[24]
Classifying jobs and predicting applications in hpc systems
Keiji Yamamoto, Yuichi Tsujita, and Atsuya Uno. Classifying jobs and predicting applications in hpc systems. In High Performance Computing: 33rd International Conference, ISC High Performance 2018, Frankfurt, Germany, June 24-28, 2018, Proceedings 33 , pages 81–99. Springer, 2018. 11
2018
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.