REVIEW 4 major objections 5 minor 27 references
Comparison of Multiple Classifiers for Android Malware Detection with Emphasis on Feature Insights Using CICMalDroid 2020 Dataset
T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read XGBoost trained on the 564-dimensional hybrid static-dynamic feature vector reaches 0.9747 accuracy on CICMalDroid2020, beating six other classifiers and both PCA and LDA preprocessing.
desk verdict Useful but unremarkable benchmark on CICMalDroid2020; the accuracy claim is compromised by an unresolved ambiguity about how the 4,264 failed-execution apps enter the evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the 564-dimensional hybrid feature vector, built by applying a variance threshold of 0.1 to reduce a 50,621-length static vector to 301 features and a 470-length dynamic vector to 263 features, then L2-normalizing the concatenation. The second mechanism is gradient boosting as implemented by XGBoost, which combines many shallow trees and handles sparse mixed categorical-numerical inputs. The third is a depth-two surrogate decision tree, which approximates the XGBoost model's splits to make the decision drivers readable. Together they support the argument that rich raw features plus boosting beat dimensionality-reduced inputs for this detection task.
What would settle it
Train the same XGBoost pipeline on CICMalDroid2020, then evaluate it on a held-out set of Android apps collected after 2020, or on the same apps with package names and manifest fields randomly mutated. If accuracy or F1 drops substantially, the paper's central claim of a practical, generalizable detector is refuted.
Extended reading notes
Core claim
The paper's central claim is that, on CICMalDroid2020 (17,341 apps labeled Benign, Adware, Banking, SMS malware, and Riskware), gradient boosting on the original 564-dimensional hybrid static-dynamic feature vector is the best of seven classifiers tested. XGBoost achieves 0.9747 accuracy, 0.9703 precision, 0.9731 recall, and 0.9716 F1; HistGradientBoosting is nearly tied, while CatBoost and Random Forest trail slightly, and KNN and SVM lag substantially. PCA that retains 95% variance degrades every model, dropping XGBoost to 0.9164 accuracy and 0.8988 F1, whereas LDA maintains mid-90s accuracy and clarifies class structure. A depth-two surrogate tree shows that the most influential features
Load-bearing premise
The central claim assumes that the features the model leans on most—package name, main activity, and target SDK—capture genuine malicious behavior rather than dataset-specific naming conventions; if those fields are spoofable or drift across time, the high accuracy will not transfer to real-world deployment.
Editorial extensions
If this is right
- Future work on CICMalDroid2020 can treat XGBoost on the original hybrid features as a supervised baseline instead of defaulting to PCA or LDA-reduced inputs.
- Dimensionality reduction is not a free lunch: retaining 95% variance still removes signal that tree ensembles rely on, so preprocessing choices should be justified per task.
- Because the top influential features are manifest metadata, the model's decisions are partly driven by app identity fields that are cheap for attackers to alter.
- Per-class metrics indicate SMS malware is the easiest class and adware and benign apps are the hardest, so a deployed detector should use class-specific thresholds or costs.
- The results support the paper's conclusion that hybrid static-dynamic features with gradient boosting offer a practical, interpretable foundation, provided the deployment feature distribution matches the 2017-2018 collection.
Reading between the lines
- A direct test of generalization would be to retrain the same pipeline after replacing package names and manifest strings with random or anonymized values; a large accuracy drop would show the reported F1 is partly an artifact of dataset naming conventions.
- The same 564-feature pipeline could be compared against sequence models that consume raw API call traces in order, testing whether gradient boosting's edge persists when dynamic behavior is used in its natural temporal order.
- Because the dataset is from 2017-2018 and SMS-heavy, collecting a fresh sample of apps and re-running the comparison would reveal whether the classifier ranking and the top features have shifted.
- An adversarial probe follows directly from the surrogate tree: renaming a malicious app to a benign-looking package string should flip the prediction if the model leans on naming rather than behavior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript reports an empirical comparison of seven classifiers (Random Forest, Extra Trees, HistGradientBoosting, KNN, SVM, XGBoost, CatBoost) under three preprocessing schemes (original features, PCA, LDA) on the CICMalDroid2020 dataset. The authors construct a 564-dimensional hybrid static/dynamic feature vector, evaluate a single 70/30 stratified split, and report that XGBoost on the original features achieves the highest accuracy (0.9747), precision (0.9703), recall (0.9731), and F1 (0.9716). They also present LDA/t-SNE visualizations, per-class F1 analysis, and a depth-two surrogate decision tree highlighting package name, main activity, and target SDK as key drivers. The paper concludes that gradient boosting with rich hybrid features is a practical baseline for Android malware detection.
Significance. If the central empirical claim holds, the paper provides a useful benchmark confirming that gradient boosting on rich static+dynamic features is strong for the CICMalDroid2020 dataset. The use of a public dataset and per-class analyses are commendable. However, the novelty is limited: the result is a single-split classifier comparison, not a new method, and many prior works report comparable or higher accuracy on similar data. The interpretability section honestly flags that spoofable manifest metadata are key drivers, which tempers the deployment-oriented conclusion. The overall significance is modest and rests on the reliability of the reported evaluation, which is currently not fully established.
major comments (4)
- [Section II vs Section III] The sample count is ambiguous. Section II states that 'Of the 17341 apps, 13077 executed successfully' and that feature extraction was performed 'From each JSON log.' Section III then says 'CICMalDroid2020 contained 17341 Android apps... After preprocessing, each app had a 564 dimensional hybrid vector.' If dynamic features come only from successful executions, the 4,264 failed apps cannot contribute dynamic traces. Please state clearly whether the failed apps were included with zero-filled dynamic features or excluded from training and testing. If included, accuracy can be inflated by a trivial 'execution success' signal correlated with class; if excluded, the headline claim that the result is for 'CICMalDroid2020' with 17,341 apps is misleading. Also report the actual number of instances in each train/test split and the per-class breakdown of the subset used. This issue is load-bearing
- [Section III, Table II] The evaluation uses a single stratified 70/30 split with no repeated runs, cross-validation, or confidence intervals. The reported difference between XGBoost (accuracy 0.9747) and HistGradientBoosting (accuracy 0.9741) is only 0.0006, well within the range of split-to-split variability. Without multiple seeds or k-fold cross-validation, the central claim that 'XGBoost performs best' is not statistically supported. Please report the hyperparameter settings for each model (including whether defaults were used), the random seed(s), and either repeated random splits with mean and standard deviation or stratified k-fold cross-validation with per-fold metrics.
- [Section II, feature preprocessing] The feature reduction pipeline appears to be applied before the train/test split. Section II describes reducing the static vector from 50,621 to 301 and the dynamic vector from 470 to 263 using a variance threshold of 0.1, and then normalizing, all before Section III's split. If this variance-based feature selection is computed on the entire dataset, it is a form of information leakage that can optimistically bias the evaluation. Please either perform all feature selection steps within the training fold only, or provide evidence that the leakage is negligible. This is important for the paper's stated aim of a 'rigorous, transparent evaluation.'
- [Reproducibility / artifact availability] The exact construction of the 564-dimensional hybrid vector is not reproducible from the manuscript. The text states that static features were obtained by enumerating all unique categorical values, but the specific feature names, the mapping from JSON logs to vector entries, the handling of the 4,264 failed executions, and the code used for preprocessing and evaluation are not provided. No repository or supplemental artifact is mentioned. For an empirical benchmarking paper, this is a significant gap. Please release the code and the final feature matrix (or a detailed feature-extraction specification) so the reported numbers can be independently verified.
minor comments (5)
- [Abstract/Index Terms] There are typos: 'Androind' should be 'Android'; 'summaries' should be 'summarizes' in Section I; Section IV contains the phrase 'slight typo deliberate' which appears to be an editorial artifact and should be removed.
- [Fig. 1 caption] The caption says 'balanced distribution,' but the text in Section II and Fig. 1 show a clear imbalance (SMS malware dominates, Adware is smallest). Please correct the caption to avoid contradiction.
- [Section IV-D / Fig. 9] The surrogate decision tree is not described with enough detail: on which data was it trained (original features? training set?), and does it approximate XGBoost's predictions or the true labels? Please specify.
- [Section V] The conclusion states that 'confusion matrix and ROC analyses showed rare benign labels for malicious apps,' but no ROC analysis or ROC curves are presented in the paper. Either include the ROC results or remove this claim.
- [Table I / References] Table I has several formatting issues: some columns are missing entries, and entries are not consistently aligned. Also, some reference URLs are malformed (e.g., [17] contains concatenated links). Please clean these up.
Circularity Check
No significant circularity: core results are empirical train/test comparisons on an external dataset, not derivations from their own outputs.
full rationale
The paper's central claims are benchmark accuracy/F1 numbers obtained from a fixed 70/30 stratified split on the external CICMalDroid2020 dataset. There is no derivation chain in which a fitted parameter is renamed as a prediction, no self-definitional feature/label identity, and no uniqueness/ansatz claim imported from the authors' prior work. The only self-citation ([1], plus incidental [6], [7]) is used for a generic motivational statement about dynamic analysis cost, not to support the reported accuracies. The interpretability section uses a depth-two surrogate tree fit to XGBoost's own predictions; explaining a model with a surrogate trained on its predictions is the standard definition of surrogate modeling, not an independent causal claim, so it does not constitute circularity. Reported ambiguities—such as how the 4,264 non-executing apps are represented in the final 17,341-row feature matrix, and the spoofability of package-name features—are data-validity/external-validity concerns, not circular reductions of the results to their inputs. No quoted reduction to inputs was found.
Assumptions & free parameters
free parameters (4)
- Variance threshold =
0.1
- PCA variance retention =
95%
- Train/test split ratio =
70/30
- Model hyperparameters =
unreported
assumptions (4)
- domain assumption CICMalDroid2020 labels are correct and representative of real Android malware.
- domain assumption Manifest metadata (package name, main activity, target SDK) are legitimate predictive features for malware detection.
- domain assumption The 13,077 successfully executed apps are representative of the full 17,341 apps.
- domain assumption Standard supervised learning assumptions: i.i.d. train/test split and no distribution shift between the 2017-2018 collection period and deployment.
Cite this review
Pith. "Pith review of Comparison of Multiple Classifiers for Android Malware Detection with Emphasis on Feature Insights Using CICMalDroid 2020 Dataset." pith.science (2026). https://pith.science/paper/2LJAYW74
@misc{pith2026260200058,
author = {Pith},
title = {Pith review of: Comparison of Multiple Classifiers for Android Malware Detection with Emphasis on Feature Insights Using CICMalDroid 2020 Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/2LJAYW74}},
note = {Machine review of arXiv:2602.00058}
}
read the original abstract
Accurate Android malware detection was critical for protecting users at scale. Signature scanners lagged behind fast release cycles on public app stores. We aimed to build a trustworthy detector by pairing a comprehensive dataset with a rigorous, transparent evaluation, and to identify interpretable drivers of decisions. We used CICMalDroid2020, which contained 17,341 apps across Benign, Adware, Banking, SMS malware, and Riskware. We extracted 301 static and 263 dynamic features into a 564 dimensional hybrid vector, then evaluated seven classifiers under three schemes, original features, principal component analysis, PCA, and linear discriminant analysis, LDA, with a 70 percent training and 30 percent test split. Results showed that gradient boosting on the original features performed best. XGBoost achieved 0.9747 accuracy, 0.9703 precision, 0.9731 recall, and 0.9716 F1, and the confusion matrix indicated rare benign labels for malicious apps. HistGradientBoosting reached 0.9741 accuracy and 0.9708 F1, while CatBoost and Random Forest were slightly lower at 0.9678 and 0.9687 accuracy with 0.9636 and 0.9637 F1. KNN and SVM lagged. PCA reduced performance for all models, with XGBoost dropping to 0.9164 accuracy and 0.8988 F1. LDA maintained mid 90s accuracy and clarified separable clusters in projections. A depth two surrogate tree highlighted package name, main activity, and target SDK as key drivers. These findings established high fidelity supervised baselines for Android malware detection and indicated that rich hybrid features with gradient boosting offered a practical and interpretable foundation for deployment.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
M. M.-H.-Z. Abedin and T. Mehrub, “Evaluating ensemble and deep learning models for static malware detection with dimensionality reduction using the ember dataset,” 7 2025. [Online]. Available: https://arxiv.org/pdf/2507.16952
arXiv 2025
-
[2]
Divergence unveils further distinct phenotypic traits of human brain connectomics fingerprint,
M. K. Uddin, N. Nguyen, H. Huang, D. Duong-Tran, and J. Zheng, “Divergence unveils further distinct phenotypic traits of human brain connectomics fingerprint,”iScience, vol. 29, no. 1, 2026
2026
-
[3]
Drebin: Effective and explainable detection of android malware in your pocket,
D. Arp, M. Spreitzenbarth, M. H ¨ubner, H. Gascon, and K. Rieck, “Drebin: Effective and explainable detection of android malware in your pocket,” 2014. [Online]. Available: http://dx.doi.org/
2014
-
[4]
YuanZhenlong, LuYongqiang, WangZhaoguo, and XueYibo, “Droid- sec,”ACM SIGCOMM Computer Communication Review, vol. 44, pp. 371–372, 8 2014. [Online]. Available: /doi/pdf/10.1145/2740070. 2631434?download=true
-
[5]
Droiddelver: An android malware detection system using deep belief network based on api call blocks,
S. Hou, A. Saas, Y . Ye, and L. Chen, “Droiddelver: An android malware detection system using deep belief network based on api call blocks,”Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9998 LNCS, pp. 54–66, 2016. [Online]. Available: https://link.springer.com/cha...
-
[6]
Stegopix2pix: Image steganography method via pix2pix networks,
M. M. ha-zul Abedin and M. A. Yousuf, “Stegopix2pix: Image steganography method via pix2pix networks,”Lecture Notes in Networks and Systems, vol. 618 LNNS, pp. 343–356, 2023. [Online]. Available: https://link.springer.com/chapter/10.1007/978-981-19-9483-8 29
-
[7]
Bangla printed character generation from handwritten character using gan,
M. M. ha zul Abedin, T. Ghosh, T. Mehrub, and M. A. Yousuf, “Bangla printed character generation from handwritten character using gan,”Studies in Fuzziness and Soft Computing, vol. 413, pp. 153–165, 2022. [Online]. Available: https://link.springer.com/chapter/ 10.1007/978-3-030-92026-5 9
-
[8]
A. Albladi, M. K. Uddin, M. Islam, and C. Seals, “Twssenti: A novel hybrid framework for topic-wise sentiment analysis on social media using transformer models,”arXiv preprint arXiv:2504.09896, 2025
arXiv 2025
Show all 27 references
-
[9]
Classification of android apps and malware using deep neural networks,
R. Nix and J. Zhang, “Classification of android apps and malware using deep neural networks,”Proceedings of the International Joint Conference on Neural Networks, vol. 2017-May, pp. 1871–1878, 6 2017
2017
-
[10]
R2-d2: Color-inspired convolutional neural network (cnn)-based android malware detections,
T. H. D. Huang and H. Y . Kao, “R2-d2: Color-inspired convolutional neural network (cnn)-based android malware detections,”Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, pp. 2633–2642, 7 2018
2018
-
[11]
Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network,
W. Wang, M. Zhao, and J. Wang, “Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network,”Journal of Ambient Intelligence and Humanized Computing, vol. 10, pp. 3035–3043, 8 2019. [Online]. Available: https://link.spring...
2019 doi
-
[12]
Dysign: Dynamic fingerprinting for the automatic detection of android malware,
E. M. B. Karbab, M. Debbabi, S. Alrabaee, and D. Mouheb, “Dysign: Dynamic fingerprinting for the automatic detection of android malware,” 2016 11th International Conference on Malicious and Unwanted Soft- ware, MALWARE 2016, pp. 139–146, 3 2017
2016
-
[13]
Droidcat: Effective android malware detection and categorization via app-level profiling,
H. Cai, N. Meng, B. Ryder, and D. Yao, “Droidcat: Effective android malware detection and categorization via app-level profiling,”IEEE Transactions on Information Forensics and Security, vol. 14, pp. 1455– 1470, 6 2019
2019
-
[14]
Android malware detection based on system call sequences and lstm,
X. Xiao, S. Zhang, F. Mercaldo, G. Hu, and A. K. Sangaiah, “Android malware detection based on system call sequences and lstm,”Multimedia Tools and Applications, vol. 78, pp. 3979–3999, 2 2019. [Online]. Available: https://link.springer.com/article/10.1007/s11042-017-5104-0
2019 doi
-
[15]
A multimodal deep learning method for android malware detection using various features,
T. Kim, B. Kang, M. Rho, S. Sezer, and E. G. Im, “A multimodal deep learning method for android malware detection using various features,” IEEE Transactions on Information Forensics and Security, vol. 14, pp. 773–788, 3 2019
2019
-
[16]
Dynamic android malware category classification using semi-supervised deep learning,
S. Mahdavifar, A. F. A. Kadir, R. Fatemi, D. Alhadidi, and A. A. Ghorbani, “Dynamic android malware category classification using semi-supervised deep learning,”2020 IEEE Intl Conf on Depend- able, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computi...
2020
-
[17]
Android malware detection based on a hybrid deep learning model,
T. Lu, Y . Du, L. Ouyang, Q. Chen, and X. Wang, “Android malware detection based on a hybrid deep learning model,” Security and Communication Networks, vol. 2020, p. 8863617, 1
2020
-
[18]
Active semi-supervised approach for checking app behavior against its description,
S. Ma, S. Wang, D. Lo, R. H. Deng, and C. Sun, “Active semi-supervised approach for checking app behavior against its description,”Proceedings - International Computer Software and Applications Conference, vol. 2, pp. 179–184, 9 2015
2015
-
[19]
Detecting and classifying android malware using static analysis along with creator information,
H. Kang, J. wook Jang, A. Mohaisen, and H. K. Kim, “Detecting and classifying android malware using static analysis along with creator information,”International Journal of Distributed Sensor Networks, vol. 2015, 3 2019. [Online]. Available: http://arxiv.org/abs/1903.01618http...
2015 arXiv
-
[20]
Maldozer: Automatic framework for android malware detection using deep learning,
E. M. B. Karbab, M. Debbabi, A. Derhab, and D. Mouheb, “Maldozer: Automatic framework for android malware detection using deep learning,”Digital Investigation, vol. 24, pp. S48–S59, 3 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S1742287618300392
2018
-
[21]
Fossil: A resilient and efficient system for identifying foss functions in malware binaries,
S. Alrabaee, P. Shirani, L. Wang, and M. Debbabi, “Fossil: A resilient and efficient system for identifying foss functions in malware binaries,” ACM Transactions on Privacy and Security, vol. 21, 1 2018. [Online]. Available: /doi/pdf/10.1145/3175492?download=true
2018 doi
-
[22]
Effective and efficient hybrid android malware classification using pseudo- label stacked auto-encoder,
S. Mahdavifar, D. Alhadidi, and A. A. Ghorbani, “Effective and efficient hybrid android malware classification using pseudo- label stacked auto-encoder,”Journal of Network and Systems Management, vol. 30, pp. 1–34, 1 2022. [Online]. Available: https://link.springer.com/article...
2022 doi
-
[23]
Amddlmodel: Android smartphones malware detection using deep learning model,
M. Aamir, M. W. Iqbal, M. Nosheen, M. U. Ashraf, A. Shaf, K. A. Almarhabi, A. M. Alghamdi, and A. A. Bahaddad, “Amddlmodel: Android smartphones malware detection using deep learning model,” PLOS ONE, vol. 19, p. e0296722, 1 2024. [Online]. Available: https: //journals.plos.org...
2024 doi
-
[24]
Benchmarking android malware detection: Traditional vs. deep learning models,
G. Liu, D. Caragea, X. Ou, and S. Roy, “Benchmarking android malware detection: Traditional vs. deep learning models,” 2 2025. [Online]. Available: https://arxiv.org/pdf/2502.15041
2025 arXiv
-
[25]
An android mutation malware detection based on deep learning using visualization of importance from codes,
Y . S. Yen and H. M. Sun, “An android mutation malware detection based on deep learning using visualization of importance from codes,”Microelectronics Reliability, vol. 93, pp. 109–114, 2 2019. [Online]. Available: https://www.sciencedirect.com/science/article/abs/ pii/S002627...
2019
-
[26]
Semi-supervised classification for dynamic android malware detection,
L. Chen, M. Zhang, C. Y . Yang, and R. Sahita, “Semi-supervised classification for dynamic android malware detection,”Proceedings of the ACM Conference on Computer and Communications Security, pp. 2479–2481, 4 2017. [Online]. Available: https://arxiv.org/pdf/1704. 05948
2017
-
[2020]
Available: /doi/pdf/10.1155/2020/8863617https: //onlinelibrary.wiley.com/doi/abs/10.1155/2020/8863617https: //onlinelibrary.wiley.com/doi/10.1155/2020/8863617
[Online]. Available: /doi/pdf/10.1155/2020/8863617https: //onlinelibrary.wiley.com/doi/abs/10.1155/2020/8863617https: //onlinelibrary.wiley.com/doi/10.1155/2020/8863617
2020 doi
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.