REVIEW 2 major objections 6 minor 46 references
Addressing malware family concept drift with triplet autoencoder
T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a triplet-loss autoencoder, clustered with DBSCAN in its latent space, can flag malware samples that belong to no known family, reaching overall F1 0.81 on the Drebin Android dataset and beating two drift-detection…
desk verdict A coherent open-set malware family detector whose evaluation never tests temporal drift: the held-out family is withheld by label, not by timestamp, so the central claim does not follow. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a triplet autoencoder: an autoencoder whose 32-dimensional bottleneck is trained with both reconstruction loss and triplet loss $L = \max\{0, D(x_a, x_p) - D(x_a, x_n) + \alpha\}$, so that the latent space is both low-dimensional and metric-structured. On top of this, DBSCAN is the load-bearing mechanism: it discovers multiple clusters within a single malware family (for example, two clusters in FakeInstaller, reducing the mean sample-to-centroid distance by 26 percent), marks outliers, and produces centroids and per-family thresholds defined as the maximum distance from a centroid to any core point. The threshold is what converts the embedding into an open-set decision rule: within threshold means known family, beyond means likely new family.
What would settle it
Run the same Drebin protocol but restrict each withheld family's test set to samples created after the training cutoff; if the DBSCAN method's overall F1 no longer beats the MAD and CADE baselines, the claim that it detects temporally emerging families would be contradicted.
Extended reading notes
Core claim
The central claim is that the appearance of new malware families, one of the two named forms of concept drift, can be detected by learning a metric space instead of a fixed classifier. A triplet autoencoder maps malware into a 32-dimensional embedding in which known families form compact, separated clusters; DBSCAN then exposes sub-clusters inside any single family, discards outliers, and defines each cluster's centroid and membership threshold as the maximum distance from the centroid to a core point. A test sample is assigned to the nearest known family only if it falls within that learned boundary; otherwise it is labeled unknown and sent for expert review. The paper reports that this DBSCAN-based threshold consistently beats the MAD threshold on the same embedding, with overall F1 of 0.81 versus 0.78 on Drebin and 0.62 for the CADE baseline, and that the approach transfers to the Windows PE dataset BODMAS with per-family F1 between 0.51 and 0.99. The authors present the automatic threshold as the practical advantage: no empirical coefficient is needed.
Load-bearing premise
The evaluation rests on treating a held-out malware family as genuinely unseen at test time even though its samples' creation timestamps overlap the training period, because no temporal split is applied within the withheld family.
Editorial extensions
If this is right
- A deployed detector can flag malware as belonging to no known family instead of forcing a wrong family label, allowing new samples to be routed to expert analysis.
- Because DBSCAN sets each family's threshold automatically, the method removes the empirical coefficient that a statistical dispersion threshold such as MAD requires.
- The same triplet-autoencoder-plus-DBSCAN pipeline reports strong F1 scores on both Android and Windows PE malware, indicating the approach transfers across feature representations and operating systems.
- Sub-clustering inside a family fixes a known failure of single-centroid methods: for FakeInstaller, DBSCAN found two clusters and reduced the mean sample-to-centroid distance by 26 percent, so distances and thresholds become more trustworthy.
- The method targets the emergence of new families rather than gradual evolution within existing families, so it complements rather than replaces drift detection aimed at shifting distributions.
Reading between the lines
- The paper does not apply a temporal split inside the withheld family; a stricter evaluation would train on each family's early samples and test only on that family's later samples, and the reported F1 advantage may shrink under that protocol.
- The threshold is fixed at the maximum core-point distance, so the boundary is sensitive to the most extreme inlier in each cluster; testing a quantile-based boundary could improve the worst BODMAS families (mira at 0.58 and sfone at 0.51).
- Only one family is withheld at a time; when several new families appear together, DBSCAN's noise points could be clustered to recognize multiple simultaneous unknowns, an extension the paper leaves for future work.
- Flagged unknowns naturally feed an active-learning loop: an analyst labels rejected samples and the centroids are recomputed, so the system can absorb a new family without a full retrain.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a triplet-autoencoder plus DBSCAN method for detecting new malware families, framed as addressing concept drift caused by the emergence of previously unseen families. The method trains an autoencoder with a combination of reconstruction loss and triplet loss, clusters the latent space with DBSCAN, computes per-cluster centroids and distance thresholds, and labels test samples as known or unknown based on proximity to the nearest centroid. The authors evaluate on the Drebin Android dataset and the BODMAS Windows PE dataset, comparing against CADE and a MAD-threshold variant, and report higher F1 scores for their DBSCAN-based approach.
Significance. If the evaluation were sound, the paper would offer a useful and relatively simple contribution to malware family detection under drift, with the strength of explicit sub-cluster modeling via DBSCAN and evaluation on two datasets. The method is clearly described, and the comparison with CADE and MAD is reasonable. However, the central claimed contribution is temporal concept drift, and the reported experimental protocol does not actually isolate temporal emergence of new families. The significance of the empirical results is therefore not established as stated.
major comments (2)
- [Section IV-C, Tables III and IV] The drift simulation uses the entire held-out malware family as the unknown set regardless of timestamps, while the temporal 80:20 split is applied only to the remaining families. For example, in Table III the FakeInstaller row uses all 925 FakeInstaller samples as unknown, even though a large fraction of those samples fall within the training time window; the known set consists of the time-based test slice of the other families. This protocol measures arbitrary family withholding (leave-one-family-out open-set recognition), not the temporal emergence of a new family after the training cutoff, which is the paper's stated central claim. The experiments should be re-run with the unknown set restricted to held-out-family samples occurring after the training cutoff, or the paper should be explicitly reframed as addressing open-set family detection rather than concept drift.
- [Table IV] The reported 'No. of known samples' values are not consistent with the stated per-family 80:20 temporal split and the family sizes in Table II. For the small row, the remaining families contain 12,852 samples, so a 20% test split would give approximately 2,570 known samples, not the reported 1,473; for the berbew row, the expected value is approximately 2,943, not 2,817. If the authors instead use a global date-based split, the exact cutoff dates and per-family test counts must be reported. As written, Table IV cannot be reproduced from the described experimental protocol.
minor comments (6)
- [Tables I and III] The family name is written as 'GingerMaster' in Table I but 'GinMaster' in Table III; please use a single consistent name.
- [Equations (1) and (2)] Equation (2) defines D as squared Euclidean distance, but the text and the threshold discussion refer to plain distances between samples and centroids; please clarify which distance is actually used in the centroid and threshold computations.
- [Section V-A] The MAD baseline is said to require an empirically determined coefficient, but the coefficient value is never reported; please include it for reproducibility.
- [Section III, 'Impact of DBSCAN'] The claim that DBSCAN 'calculates the threshold automatically without requiring a manual coefficient' should be qualified, since the DBSCAN parameters epsilon and minPts are still chosen by heuristics (the k-distance plot and the 2 x dim rule).
- [Section IV-B] The paper states that the BODMAS training and test sets are split by first-seen timestamps from August 2019 to September 2020, but Table II does not report how many samples of each family fall into the test window; adding per-family test counts would make the setup clearer.
- [References] Reference [2] is spelled 'A V-Test' but should be 'AV-TEST'; please correct the citation.
Circularity Check
No circularity: the F1 evaluation is on held-out families with thresholds derived from training data alone; the temporal-split weakness is an experimental-validity concern, not a circular derivation.
full rationale
The derivation chain is self-contained. The model is trained on known families with one family excluded, and DBSCAN centroids and per-family thresholds are computed exclusively from the training latent space: "Each family's threshold is determined by the distance from the centroid to the furthest point within the cluster." Test samples are then classified by comparing their distance to the nearest centroid against that threshold. The reported F1 scores in Tables III and IV are computed on samples whose family labels were never used to set the thresholds, so the "new family detection" result is not equivalent to a fitted parameter or to the training labels. The only self-citation, reference [25], is a metric-learning BCI paper co-authored by one of the present authors and appears only in related work as an example application; it is not load-bearing. Section VI openly acknowledges limitations such as considering only one unknown family and the possibility of fine-tuning DBSCAN's epsilon and minPts; these are honest scope statements rather than circular moves. The skeptic's concern that the held-out family's early samples are contemporaneous with the training window, based on the statement in Section IV that "we simulate the presence of drifting samples by systematically excluding one malware family from the training data," is a threat to the temporal-drift interpretation of the experiment, but it does not make the predicted F1 scores an input of the method. The apparent BODMAS row inconsistency in Table IV (for 'small', 1473 known + 3606 unknown = 5079, while Table II reports 3606 samples for that family) is a reporting or preprocessing consistency issue, not a circular derivation. No step in the paper reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (6)
- DBSCAN epsilon =
not reported
- DBSCAN minPts =
2 x bottleneck_dim = 64 (for 32-d latent)
- Triplet loss margin alpha =
not reported
- Reconstruction vs triplet loss weighting =
not reported
- Feature variance threshold =
not reported
- MAD coefficient for baseline =
not reported
assumptions (4)
- domain assumption All inputs are malware samples; only family labels matter.
- domain assumption Emergence of a new family manifests as out-of-distribution samples in a 32-d metric space learned from static features.
- domain assumption DBSCAN clusters in the latent space correspond to meaningful subfamilies or variants.
- domain assumption Static features (Drebin manifest/code features, BODMAS feature vector) are sufficient to detect family drift.
Cite this review
Pith. "Pith review of Addressing malware family concept drift with triplet autoencoder." pith.science (2026). https://pith.science/paper/PIPQF3JJ
@misc{pith2026250700348,
author = {Pith},
title = {Pith review of: Addressing malware family concept drift with triplet autoencoder},
year = {2026},
howpublished = {\url{https://pith.science/paper/PIPQF3JJ}},
note = {Machine review of arXiv:2507.00348}
}
read the original abstract
Machine learning is increasingly vital in cybersecurity, especially in malware detection. However, concept drift, where the characteristics of malware change over time, poses a challenge for maintaining the efficacy of these detection systems. Concept drift can occur in two forms: the emergence of entirely new malware families and the evolution of existing ones. This paper proposes an innovative method to address the former, focusing on effectively identifying new malware families. Our approach leverages a supervised autoencoder combined with triplet loss to differentiate between known and new malware families. We create clear and robust clusters that enhance the accuracy and resilience of malware family classification by utilizing this metric learning technique and the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. The effectiveness of our method is validated using an Android malware dataset and a Windows portable executable (PE) malware dataset, showcasing its capability to sustain model performance within the dynamic landscape of emerging malware threats. Our results demonstrate a significant improvement in detecting new malware families, offering a reliable solution for ongoing cybersecurity challenges.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Learning under concept drift: A review,
J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE transactions on knowledge and data engineering, vol. 31, no. 12, pp. 2346– 2363, 2018
work page 2018
-
[2]
Malware statistics & trends report,
A V-Test, “Malware statistics & trends report,” Accessed: 14 July 2024, 2023, [Online]. Available: https://www.av-test.org/ en/statistics/malware/
work page 2024
-
[3]
Tesseract: Eliminating experimental bias in malware classification across space and time,
F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cav- allaro, “Tesseract: Eliminating experimental bias in malware classification across space and time,” in 28th USENIX security symposium (USENIX Security 19) , 2019, pp. 729–746
work page 2019
-
[4]
Transcend: Detecting concept drift in malware classification models,
R. Jordaney et al. , “Transcend: Detecting concept drift in malware classification models,” in 26th USENIX security sym- posium (USENIX security 17) , 2017, pp. 625–642
work page 2017
-
[5]
Towards open set deep networks,
A. Bendale and T. E. Boult, “Towards open set deep networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1563–1572
work page 2016
-
[6]
A simple unified frame- work for detecting out-of-distribution samples and adversarial attacks,
K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified frame- work for detecting out-of-distribution samples and adversarial attacks,” Advances in neural information processing systems , vol. 31, 2018
work page 2018
-
[7]
A. Guerra-Manzanares, “Android malware detection: Mission accomplished? a review of open challenges and future per- spectives,” Computers & Security , p. 103 654, 2023
work page 2023
-
[8]
Novel feature extraction, selection and fusion for effective malware family classification,
M. Ahmadi, D. Ulyanov, S. Semenov, M. Trofimov, and G. Giacinto, “Novel feature extraction, selection and fusion for effective malware family classification,” in Proceedings of the sixth ACM conference on data and application security and privacy, 2016, pp. 183–194
work page 2016
Show all 46 references
-
[9]
Malware detection based on mining api calls,
A. Sami, B. Yadegari, H. Rahimi, N. Peiravian, S. Hashemi, and A. Hamze, “Malware detection based on mining api calls,” in Proceedings of the 2010 ACM symposium on applied computing, 2010, pp. 1020–1025
2010
-
[10]
Byte level n–gram analysis for malware detection,
S. Jain and Y . K. Meena, “Byte level n–gram analysis for malware detection,” in Computer Networks and Intelligent Computing: 5th International Conference on Information Pro- cessing, ICIP 2011, Bangalore, India, August 5-7, 2011. Proceedings, Springer, 2011, pp. 51–59
2011
-
[11]
Malware detection and classifica- tion based on n-grams attribute similarity,
Z. Fuyong and Z. Tiezhu, “Malware detection and classifica- tion based on n-grams attribute similarity,” in2017 IEEE inter- national conference on computational science and engineering (CSE) and IEEE international conference on embedded and ubiquitous computing (EUC), IEEE, vo...
2017
-
[12]
Deep android malware detection,
N. McLaughlin et al. , “Deep android malware detection,” in Proceedings of the seventh ACM on conference on data and application security and privacy , 2017, pp. 301–308
2017
-
[13]
Sequential op- code embedding-based malware detection method,
A. G. Kakisim, S. Gulmez, and I. Sogukpinar, “Sequential op- code embedding-based malware detection method,”Computers & Electrical Engineering , vol. 98, p. 107 703, 2022
2022
-
[14]
Malware detection based on deep learn- ing algorithm,
D. Yuxin and Z. Siyi, “Malware detection based on deep learn- ing algorithm,” Neural Computing and Applications , vol. 31, pp. 461–472, 2019
2019
-
[15]
Drebin: Effective and explainable detection of android malware in your pocket.,
D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, K. Rieck, and C. Siemens, “Drebin: Effective and explainable detection of android malware in your pocket.,” in Ndss, vol. 14, 2014, pp. 23–26
2014
-
[16]
Ember: An open dataset for training static pe malware machine learning models,
H. S. Anderson and P. Roth, “Ember: An open dataset for training static pe malware machine learning models,” arXiv preprint arXiv:1804.04637, 2018
2018 arXiv
-
[17]
Bodmas: An open dataset for learning based temporal analysis of pe malware,
L. Yang, A. Ciptadi, I. Laziuk, A. Ahmadzadeh, and G. Wang, “Bodmas: An open dataset for learning based temporal analysis of pe malware,” in 2021 IEEE Security and Privacy Workshops (SPW), IEEE, 2021, pp. 78–84
2021
-
[18]
Maar: Robust features to detect malicious activity based on api calls, their arguments and return values,
Z. Salehi, A. Sami, and M. Ghiasi, “Maar: Robust features to detect malicious activity based on api calls, their arguments and return values,” Engineering Applications of Artificial In- telligence, vol. 59, pp. 93–102, 2017
2017
-
[19]
Malware detection and classification based on extraction of api sequences,
D. Uppal, R. Sinha, V . Mehra, and V . Jain, “Malware detection and classification based on extraction of api sequences,” in 2014 International conference on advances in computing, com- munications and informatics (ICACCI), IEEE, 2014, pp. 2337– 2342
2014
-
[20]
Detecting obfus- cated malware using reduced opcode set and optimised runtime trace,
P. O’kane, S. Sezer, and K. McLaughlin, “Detecting obfus- cated malware using reduced opcode set and optimised runtime trace,” Security Informatics, vol. 5, pp. 1–12, 2016
2016
-
[21]
Network malware classification comparison using dpi and flow packet headers,
A. Boukhtouta, S. A. Mokhov, N.-E. Lakhdari, M. Debbabi, and J. Paquet, “Network malware classification comparison using dpi and flow packet headers,” Journal of Computer Virology and Hacking Techniques, vol. 12, pp. 69–100, 2016
2016
-
[22]
Malicious software classification using transfer learning of resnet-50 deep neural network,
E. Rezende, G. Ruppert, T. Carvalho, F. Ramos, and P. De Geus, “Malicious software classification using transfer learning of resnet-50 deep neural network,” in 2017 16th IEEE international conference on machine learning and applications (ICMLA), IEEE, 2017, pp. 1011–1014
2017
-
[23]
In defense of the triplet loss for person re-identification,
A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv preprint arXiv:1703.07737, 2017
2017 arXiv
-
[24]
Triplet loss in siamese network for object tracking,
X. Dong and J. Shen, “Triplet loss in siamese network for object tracking,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 459–474
2018
-
[25]
A zero-shot deep metric learning approach to brain–computer interfaces for image retrieval,
B. McCartney, B. Devereux, and J. Martinez-del-Rincon, “A zero-shot deep metric learning approach to brain–computer interfaces for image retrieval,” Knowledge-Based Systems , vol. 246, p. 108 556, 2022
2022
-
[26]
Representation learning with contrastive predictive coding,
A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018
2018 arXiv
-
[27]
Metric learning- based multimodal audio-visual emotion recognition,
E. Ghaleb, M. Popa, and S. Asteriadis, “Metric learning- based multimodal audio-visual emotion recognition,” IEEE Multimedia, vol. 27, no. 1, pp. 37–48, 2019
2019
-
[28]
In defence of metric learning for speaker recognition,
J. S. Chung et al., “In defence of metric learning for speaker recognition,” arXiv preprint arXiv:2003.11982 , 2020
2003 arXiv
-
[29]
Multi-instance multi- label distance metric learning for genome-wide protein func- tion prediction,
Y . Xu, H. Min, H. Song, and Q. Wu, “Multi-instance multi- label distance metric learning for genome-wide protein func- tion prediction,” Computational biology and chemistry, vol. 63, pp. 30–40, 2016
2016
-
[30]
A novel drug repositioning approach based on collaborative metric learning,
H. Luo, J. Wang, C. Yan, M. Li, F.-X. Wu, and Y . Pan, “A novel drug repositioning approach based on collaborative metric learning,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 18, no. 2, pp. 463–471, 2019
2019
-
[31]
Contrastive learning for robust android malware familial classification,
Y . Wu, S. Dou, D. Zou, W. Yang, W. Qiang, and H. Jin, “Contrastive learning for robust android malware familial classification,” IEEE Transactions on Dependable and Secure Computing, 2022
2022
-
[32]
Application of distance metric learning to automated malware detection,
M. Jure ˇcek and R. Lórencz, “Application of distance metric learning to automated malware detection,”IEEE Access, vol. 9, pp. 96 151–96 165, 2021
2021
-
[33]
Fewm- hgcl: Few-shot malware variants detection via heterogeneous graph contrastive learning,
C. Liu, B. Li, J. Zhao, Z. Zhen, X. Liu, and Q. Zhang, “Fewm- hgcl: Few-shot malware variants detection via heterogeneous graph contrastive learning,” IEEE Transactions on Dependable and Secure Computing , 2022
2022
-
[34]
Autoencoder- based deep metric learning for network intrusion detection,
G. Andresini, A. Appice, and D. Malerba, “Autoencoder- based deep metric learning for network intrusion detection,” Information Sciences, vol. 569, pp. 706–727, 2021
2021
-
[35]
Tracking concept drift in malware families,
A. Singh, A. Walenstein, and A. Lakhotia, “Tracking concept drift in malware families,” in Proceedings of the 5th ACM workshop on Security and artificial intelligence, 2012, pp. 81– 92
2012
-
[36]
Transcending transcend: Revisiting malware classification in the presence of concept drift,
F. Barbero, F. Pendlebury, F. Pierazzi, and L. Cavallaro, “Transcending transcend: Revisiting malware classification in the presence of concept drift,” in 2022 IEEE Symposium on Security and Privacy (SP) , IEEE, 2022, pp. 805–823
2022
-
[37]
Cade: Detecting and explaining concept drift samples for security applications,
L. Yang et al., “Cade: Detecting and explaining concept drift samples for security applications,” in 30th USENIX Security Symposium (USENIX Security 21) , 2021, pp. 2327–2344
2021
-
[38]
Insomnia: Towards concept-drift robustness in network intrusion detection,
G. Andresini, F. Pendlebury, F. Pierazzi, C. Loglisci, A. Appice, and L. Cavallaro, “Insomnia: Towards concept-drift robustness in network intrusion detection,” in Proceedings of the 14th ACM workshop on artificial intelligence and security , 2021, pp. 111–122
2021
-
[39]
Temporal analysis of dis- tribution shifts in malware classification for digital forensics,
F. Zola, J. L. Bruse, and M. Galar, “Temporal analysis of dis- tribution shifts in malware classification for digital forensics,” in 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), IEEE, 2023, pp. 439–450
2023
-
[40]
Deep metric learning: A survey,
M. Kaya and H. ¸ S. Bilge, “Deep metric learning: A survey,” Symmetry, vol. 11, no. 9, p. 1066, 2019
2019
-
[41]
The curse (s) of dimension- ality,
N. Altman and M. Krzywinski, “The curse (s) of dimension- ality,” Nat Methods, vol. 15, no. 6, pp. 399–400, 2018
2018
-
[42]
Dbscan revisited, revisited: Why and how you should (still) use dbscan,
E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu, “Dbscan revisited, revisited: Why and how you should (still) use dbscan,” ACM Transactions on Database Systems (TODS), vol. 42, no. 3, pp. 1–21, 2017
2017
-
[43]
The k-means algo- rithm: A comprehensive survey and performance evaluation,
M. Ahmed, R. Seraj, and S. M. S. Islam, “The k-means algo- rithm: A comprehensive survey and performance evaluation,” Electronics, vol. 9, no. 8, p. 1295, 2020
2020
-
[44]
A density-based algorithm for discovering clusters in large spatial databases with noise,
M. Ester et al. , “A density-based algorithm for discovering clusters in large spatial databases with noise,” in KDD, vol. 96, 1996, pp. 226–231
1996
-
[45]
Fesa: Feature selection architecture for ransomware detection under concept drift,
D. W. Fernando and N. Komninos, “Fesa: Feature selection architecture for ransomware detection under concept drift,” Computers & Security , vol. 116, p. 102 659, 2022
2022
-
[46]
Visualizing data using t- sne.,
L. Van der Maaten and G. Hinton, “Visualizing data using t- sne.,” Journal of machine learning research , vol. 9, no. 11, 2008
2008
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.