{"id":"cd69dcde-1c82-47dd-abcd-3d0e17e97ee0","arxiv_id":"2411.18327","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SSDeep fuzzy hashes of executable files, combined with a random forest, classify preinstalled HPC applications into 92 classes and an unknown class with roughly 90% macro F1.","lead":"An HPC security team tests whether SSDeep fuzzy hashes, a malware-detection tool, can identify which scientific application an executable belongs to. They report around 90% macro F1 on 92 application classes, which could help detect resource misuse and stolen credentials.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol likely leaks test information: the all-pairs feature matrix appears to be computed before the train/test split, and the known-class split is at sample level, so the reported 90% F1 may not reflect classifying genuinely unseen applications.","rationale":"The reader chose dataset representativeness as the weakest assumption, but the more urgent issue is internal validity: the evaluation protocol may not simulate the intended deployment setting. If the feature matrix is all-pairs over the full dataset before splitting, training rows contain similarities to test samples, so the model's feature space differs from what a deployed classifier would see. The paper's own wording in Section 3 makes this plausible, and the sample-level (rather than version-level) split creates an additional same-version leakage path. The reader's rationale did mention the sample-level split and under-specified feature matrix, so there is partial agreement, but the reader did not identify this as the central load-bearing concern. The dataset representativeness limitation is explicitly acknowledged in Section 5, whereas the potential leakage is not acknowledged anywhere. Since the paper provides no code or data, the concern cannot be resolved from the manuscript alone. The appropriate response is to keep the conditional verdict: the central claim is plausible but unverified, and it must be re-tested with a version-based, training-reference-only protocol before the 90% F1 can be trusted.","tokens_in":11075,"tokens_out":8184,"duration_ms":78256,"concrete_test":"Release code or a precise specification and re-run the pipeline with two changes: (1) split known classes by version, not by sample, so all executables of a given version stay in one fold; (2) compute features for each training sample only against the training reference set and for each test sample only against that same training reference set, never using all-pairs similarities that include test samples. Compare macro F1 to the reported 90%. A drop of more than a few points, or an inability to define fixed-dimension features for a truly new sample, would confirm that the reported performance depends on leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is internal to the evaluation, not the dataset. Section 3 states that 'We compute a feature matrix for our dataset based on the SSDeep fuzzy hash similarity between sample features' before describing the two-phase train/test split. If this matrix is all-pairs over all 5333 samples, then every training row contains columns whose values are similarities to test samples. A Random Forest trained on these rows has a feature space indexed by the full dataset; a genuinely new executable, however, can only be compared against the training library, so its feature vector has fewer dimensions and different semantics. The reported 90% macro F1 would then be an artifact of a transductive setup rather than evidence that unseen applications can be classified. The description also says the second split is 'a stratified 60-40 train-test split on the samples,' not on versions; for classes with multiple executables per version (Table 1, Velvet), the same version can appear in both training and test, further inflating similarity. Section 5 discusses class imbalance and mislabeled classes but never mentions either form of potential leakage. Because no code or data are released, the reader cannot tell whether the feature matrix was restricted to training similarities; the manuscript's own description is consistent with the leaky construction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a static, hash-based approach to classifying application executables on HPC systems. The authors extract SSDeep fuzzy hashes of three executable features (raw binary content, printable strings, and symbol-table function names), build a similarity feature matrix, and train a Random Forest classifier, which they call the Fuzzy Hash Classifier. Using 5,333 preinstalled application samples from the sciCORE cluster across 92 classes, the model reports micro F1 of 89%, macro F1 of 90%, and weighted F1 of 90%, including a manually defined '-1' unknown class formed from classes held out in an 80-20 class-level split. The paper argues that fuzzy hashing can recognize known applications and flag previously unseen ones, complementing dynamic resource-usage-based classification.","tokens_in":11278,"tokens_out":3307,"duration_ms":32643,"significance":"If the evaluation is sound, the paper offers a useful and low-overhead complement to dynamic HPC application classification: static executable fingerprints can be collected without runtime monitoring, avoid many privacy and integrity concerns of inspecting raw user files, and are cheap to compare. The dataset of 5,333 real preinstalled HPC executables with version structure is itself a valuable resource, and the explicit handling of completely unseen application classes is a relevant step toward production use. The approach is not conceptually novel — fuzzy hashing is standard in malware detection — but applying it to HPC workload classification and evaluating it on a production cluster dataset is a reasonable contribution. The reported results, however, rest on several evaluation-protocol details that must be clarified and corrected before the 90% F1 claim can be accepted.","major_comments":[{"comment":"The manuscript states that 'We compute a feature matrix for our dataset based on the SSDeep fuzzy hash similarity between sample features' and only afterward describes the two-phase train-test split. If this feature matrix is all-pairs over all 5,333 samples, then every training row contains similarity values to test samples, and the Random Forest's feature dimensions are indexed by the full dataset. A genuinely new executable could only be compared against the training library, so its feature vector would have fewer dimensions and different semantics; the reported 90% macro F1 would then reflect a transductive setup rather than classification of unseen applications. The authors must clarify whether the similarity matrix was computed only within the training set, and if not, rerun the evaluation with the feature matrix restricted to training-sample references.","section":"Section 3 (Feature Extraction and Fuzzy Hash Classifier)"},{"comment":"The second split is described as 'a stratified 60-40 train-test split on the samples,' not on versions. As Table 1 shows for Velvet, a single version can contain multiple executables (velveth and velvetg); a sample-level split can place executables from the same version in both training and test sets, inflating similarity because those executables were built from the same source and compiler configuration. The claim that the test set contains unseen applications is stronger than what this split guarantees. The authors should split by version (or by unique build) so that all samples from one version are assigned to the same fold, and report results under that split.","section":"Section 3 (Fuzzy Hash Classifier, second-phase split)"},{"comment":"The training set of 2,688 samples is formed from the known classes only; the 20 unknown classes (852 test samples, Table 3) appear exclusively in the test set. The confidence threshold is said to be tuned 'as part of the hyperparameter grid search within the training set,' but it is unclear how the threshold can be optimized for an unknown class when no unknown-class samples exist in the training set. If the threshold was selected using the test set's unknown samples, that selection is a form of test-set leakage and the reported unknown-class performance is optimistically biased. The authors must specify exactly how the threshold was tuned and, ideally, use a validation split that contains held-out unknown classes.","section":"Section 3 (Unknown Samples and Confidence Threshold) and Section 4 (Confidence Threshold)"},{"comment":"The evaluation relies on a single, fixed train-test split with no cross-validation, repeated splits, or error bars. Given the high class imbalance (support values range from 1 to 352 in Table 4) and the large per-class variability (e.g., BWA recall 0.40, HTSlib F1 0.36, CAD-score precision 0.25), a single split is insufficient to establish that the 90% macro F1 is stable rather than an artifact of one particular split. The authors should report results over multiple re-splits (e.g., repeated stratified splits or cross-validation with class-level grouping) including variance, especially for the low-support classes.","section":"Section 4 (Results) and Section 5 (Overall Performance)"}],"minor_comments":[{"comment":"Several references are malformed or incomplete: reference [15] reads 'Sean. Fingerprinting communication Peisert and computation on HPC machines' and [16] has the author name misplaced; reference [7] appears to omit co-authors. These should be corrected.","section":"Section 2 (Related Work)"},{"comment":"There is a typo in '5‘333' (curly quote instead of apostrophe) and several headings contain stray spaces ('F uzzy', 'F eature', 'F uture'). Please proofread.","section":"Section 6 (Conclusion)"},{"comment":"The text says 'Figure 5 shows the feature importance' but the feature importance is presented in Table 5; the figure/table citation should be corrected.","section":"Section 5 (Feature Importance)"},{"comment":"The limitations paragraph appropriately acknowledges that stripped binaries and wrapper scripts are excluded, but the introduction and conclusion frame the method as broadly applicable to HPC application classification. The paper should state more prominently that the evaluation covers preinstalled, symbol-table-intact executables only, and that transfer to user-compiled or stripped executables is untested.","section":"Section 5 (Limitations)"},{"comment":"No code or data artifacts are released. Given that the evaluation protocol is central to the paper's claims, releasing the dataset (or a representative subset) and the feature-construction pipeline would substantially increase confidence and reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the dataset is a useful contribution, but the evaluation protocol as described is not yet trustworthy. The all-pairs feature-matrix concern and the sample-level split are load-bearing: both can inflate the reported F1 scores, and the manuscript's own wording is consistent with both problems. These issues are fixable within the scope of the paper by recomputing the feature matrix on training data only, splitting by version, and specifying the unknown-class threshold tuning procedure. I would not reject the paper, but I would not accept it without these corrections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper takes a sensible idea—SSDeep fuzzy hashing, borrowed from malware detection—and applies it to classify HPC executables. That combination is new relative to the prior work (Yamamoto et al. used cryptographic hashes and function-name paragraph vectors). The framing is clear, and the paper honestly discusses limitations like wrapper scripts and stripped binaries. The feature-importance result, where symbol-table hashes dominate over raw bytes and strings, is a nice insight. The evaluation, however, has a load-bearing flaw. Section 3 says a feature matrix is computed from SSDeep similarity between sample features, and this is described before the train/test split. If that matrix is all-pairs over all 5333 samples, then every training row contains similarity values to test samples. A Random Forest trained on those rows has a feature space indexed by the full dataset. A genuinely new executable can only be compared against the training library, so it would not have those columns. The reported 90% macro F1 would then be an artifact of a transductive setup, not evidence of classifying unseen applications. The manuscript never states that the matrix is restricted to training similarities, and the wording suggests otherwise. Second, the known-class split is at the sample level, not the version level. For classes like Velvet, where velveth and velvetg are two executables from the same version, a 60-40 sample split can put one in train and the other in test. That makes recognition easier and further inflates the reported scores. The paper also gives no baseline comparison (e.g., a trivial byte-exact match or a simple hash classifier), no cross-validation or error bars, and no code or data, so the reader cannot check any of this. None of this kills the underlying premise. Fuzzy hashing may well work for this task if the feature matrix is built correctly and the split is by version or by class. But as written, the central claim is uncertain. The paper is worth engaging with because the idea is good and the discussion is thoughtful, but it needs major revision before the numbers can be trusted. Who is this for? HPC operators and researchers working on workload classification or security monitoring. I would not cite it in its current form, but I would send it to a serious referee. My recommendation: accept for peer review with a request for a version-based split, a clear description of the feature matrix construction, and a baseline comparison.","headline":"The fuzzy-hashing idea is genuinely new for HPC executable classification, but the evaluation as described almost certainly leaks test information, so the 90% F1 is not believable without a version-based split and a restricted feature matrix.","tokens_in":723,"tokens_out":837,"would_cite":false,"duration_ms":26398,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Random Forest trained on SSDeep fuzzy-hash similarities can label HPC application executables, including unseen classes, with about 90% F1.","keywords":["HPC application classification","SSDeep fuzzy hashing","context-triggered piecewise hashing","Random Forest classifier","static executable analysis","unknown sample detection","resource misuse detection","Damerau-Levenshtein similarity"],"falsifier":"Collect a held-out set of user-compiled or stripped binaries from a production HPC system, run the trained Fuzzy Hash Classifier, and measure F1: if classification accuracy falls well below the reported 90% on these realistic workloads, the central claim does not transfer. Alternatively, feed known malicious binaries such as compiled cryptocurrency miners and check whether the unknown label reliably catches them.","tokens_in":10816,"feed_emoji":"🛡️","tokens_out":5553,"duration_ms":46346,"temperature":0.7,"pith_summary":"This paper tries to establish that a malware-detection technique—similarity-preserving fuzzy hashing—can identify which applications are running on a high-performance computing system. The idea is to hash static features of executable files (raw binary content, printable strings, and symbol-table function names) and train a Random Forest to label new executables by their similarity to known samples. On 92 application classes and 5,333 preinstalled executables from a production cluster, the classifier reaches macro, micro, and weighted F1 scores of 90%, 89%, and 90%, including a test set with completely unseen classes that should be flagged as unknown. If this holds, HPC operators could detect wasted or misused compute resources, such as cryptocurrency miners, by recognizing that a job's executable does not match any known, allocation-approved application.","feed_headline":"Fuzzy hashes identify 92 HPC applications at 90% F1","feed_subtitle":"Malware-style fuzzy fingerprints let operators flag executables that do not match any known, approved application.","key_machinery":"The central object is the SSDeep fuzzy hash, a context-triggered piecewise hash that produces a similarity score from 0 to 100 via the Damerau-Levenshtein edit distance between hash strings. It does the work of measuring how close an unseen executable is to known application samples despite code or version changes. The Random Forest Classifier consumes a matrix of these similarity scores across three features—raw binary, printable strings from the strings command, and global symbol names from the nm command—and a confidence threshold converts low-confidence predictions into the unknown label (-1). Balanced class weights counteract the dataset's heavy imbalance.","core_discovery":"The central claim is that SSDeep fuzzy hashes of application executables preserve enough structural similarity across versions of the same application that a Random Forest classifier can label both known and unknown samples. Instead of cryptographic hashes, which only match identical files, SSDeep uses context-triggered piecewise hashing and compares hashes by Damerau-Levenshtein distance, so slightly modified versions of a program remain recognizable. The authors extract three fuzzy-hash features—raw binary content, printable strings, and global symbol names—and compute pairwise similarities against training samples. After an 80-20 split that holds out whole application classes as unknown and a stratified 60-40 split of the remaining classes, the Fuzzy Hash Classifier reports macro F1 of 90%, micro F1 of 89%, and weighted F1 of 90% over 92 classes and 5,333 samples. The symbol-table feature carries most of the predictive weight, consistent with function names staying stable across versions.","pith_inferences":["A natural next test is to point the same classifier at known malicious binaries, such as compiled cryptocurrency miners; the paper's dataset intentionally excludes malicious software, so this would probe whether 'unknown' catches actual abuse rather than just unfamiliar scientific codes.","The dominance of symbol-table features suggests stripped binaries could be handled by reconstructing function-level structure from dynamic call trees, an extension the authors name but do not implement; one could test whether the classification gap closes when such features replace nm output.","The two-phase split mimics a deployment where only some application classes are known; in production, the confidence threshold would likely need site-specific tuning, since the paper reports that raising it trades unknown recall for known-class performance."],"forward_implications":["If correct, HPC centers can label a running job's executable as a known application class or as unknown without collecting per-job resource-usage data or trusting user-supplied job names.","The unknown label gives a concrete signal for deviation from allocation purpose: a user project suddenly executing executables outside its known set can be flagged for inspection.","Because symbol-table function names dominate the feature importance, the method works best on non-stripped binaries; stripped executables and wrapper scripts such as bash or Python are outside its current reach.","The approach is static, so it can run before or alongside dynamic resource-based classifiers, potentially reducing monitoring overhead.","The reported F1 scores around 90% on 92 imbalanced classes suggest the method scales to a realistic multi-class setting, though classes with very few samples have noisy performance."],"supporting_citations":[{"why":"Defines SSDeep context-triggered piecewise hashing, the similarity-preserving hash at the center of the method.","marker":"[8]"},{"why":"Prior HPC work classifying jobs from executables with cryptographic hashes and function names; the approach extends it to fuzzy hashes.","marker":"[24]"},{"why":"Malware research showing fuzzy hashing catches near-duplicate variants, the core analogy the paper borrows.","marker":"[22]"},{"why":"Supplies the Random Forest Classifier and evaluation metrics used in the experiments.","marker":"[14]"},{"why":"Defines the Damerau-Levenshtein distance that SSDeep uses to score similarity between hashes.","marker":"[5]"},{"why":"Levenshtein distance is the base edit-distance concept behind the similarity scoring.","marker":"[9]"},{"why":"Source of the preinstalled application executable dataset used for training and testing.","marker":"[18]"}],"fun_headline_variants":["Fuzzy hashes classify HPC apps at 90% F1","Malware-style hashing IDs HPC applications with 90% F1","HPC app classification via fuzzy hashes hits 90% F1","Fuzzy hashing for HPC: 90% F1 across 92 classes","Recognizing HPC executables with fuzzy hashes and ML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the preinstalled, non-malicious executables collected for the dataset—all of which still contain their symbol table—are representative of what an HPC system actually needs to classify; user-compiled binaries, stripped binaries, and wrapper scripts are explicitly excluded.","fun_headline_variants_meta":{"raw":{"variants":["Fuzzy hashes classify HPC apps at 90% F1","Malware-style hashing IDs HPC applications with 90% F1","HPC app classification via fuzzy hashes hits 90% F1","Fuzzy hashing for HPC: 90% F1 across 92 classes","Recognizing HPC executables with fuzzy hashes and ML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2011,"prompt_tokens":902,"completion_tokens":1109,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1011}},"tokens_in":518,"tokens_out":1109,"duration_ms":8739,"temperature":1.0,"reasoning_tokens":1011,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:17:52.678425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of user-compiled or stripped binaries from a production HPC system, run the trained Fuzzy Hash Classifier, and measure F1: if classification accuracy falls well below the reported 90% on these realistic workloads, the central claim does not transfer. Alternatively, feed known malicious binaries such as compiled cryptocurrency miners and check whether the unknown label reliably catches them.","supporting_citations":[{"cited_title":"Identifying almost identical files using context triggered piecewise hashing","cited_arxiv_id":null,"evidence_quote":"Defines SSDeep context-triggered piecewise hashing, the similarity-preserving hash at the center of the method."},{"cited_title":"Classifying jobs and predicting applications in hpc systems","cited_arxiv_id":null,"evidence_quote":"Prior HPC work classifying jobs from executables with cryptographic hashes and function names; the approach extends it to fuzzy hashes."},{"cited_title":"”Combing through the fuzz: Using fuzzy hashing and 2021","cited_arxiv_id":null,"evidence_quote":"Malware research showing fuzzy hashing catches near-duplicate variants, the core analogy the paper borrows."},{"cited_title":"Scikit-learn: Machine learning in python fabian","cited_arxiv_id":null,"evidence_quote":"Supplies the Random Forest Classifier and evaluation metrics used in the experiments."},{"cited_title":"A technique for computer detection and correction of spelling errors","cited_arxiv_id":null,"evidence_quote":"Defines the Damerau-Levenshtein distance that SSDeep uses to score similarity between hashes."},{"cited_title":"sciCORE ( http://scicore.unibas.ch/) the scientific computing center at the University of Basel","cited_arxiv_id":null,"evidence_quote":"Source of the preinstalled application executable dataset used for training and testing."}],"review_version":1}