{"id":"60757dd0-7f64-4d56-8985-0c6037e47305","arxiv_id":"2504.17930","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A routine malware-detection benchmark whose central 'DNN is best' claim is contradicted by its own tables and undermined by data leakage and an impossible preprocessing step.","lead":"This paper compares six machine learning and three deep learning models on a public Kaggle malware dataset and reports near-perfect accuracies for all of them. The paper's own results tables contradict its headline claim that the DNN is best, and its preprocessing steps include an impossible 'unhashing' step and feature selection that leaks test information.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data leakage from running RFE before the train/test split invalidates the accuracy ranking, and Table VI contradicts the claimed DNN superiority.","rationale":"The reader's weakest assumption is the preprocessing pipeline, and this is exactly where the central claim collapses. In good faith, the paper's stated contribution is a fair comparison of ML/DL models for malware detection; such a comparison requires that test data never influence model or feature choice. The text and workflow place RFE before the split, so this condition is violated. Even setting leakage aside, the conclusion misquotes the paper's own Table VI: CNN is reported as more accurate than DNN on the test set, and the reported DNN AUC of 0.9993 is not the perfect 100% claimed in the conclusion. The impossible 'unhashing' step reinforces that the preprocessing narrative is unreliable. I therefore see no way to accept the headline claim; because the reader already reached REJECT, the stress-test leaves that verdict unchanged.","tokens_in":7493,"tokens_out":3691,"duration_ms":36593,"concrete_test":"Obtain the Kaggle 'Malware Detection' dataset cited as [16], rerun the exact pipeline but move RFE inside the 80% training fold so the 20% test fold is untouched by feature selection, then compare the resulting DNN test accuracy and AUC with the reported 99.90%/0.9993 and with CNN's Table VI values; if the scores drop or the ranking changes, the reported superiority is an artifact of leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires both a valid preprocessing pipeline and consistent support in the reported tables; neither holds. Section III.B describes Recursive Feature Elimination applied to select the top 25 features before the 80/20 split, and the workflow diagram in Fig. 2 confirms that RFE precedes the split. Because RFE is fit on all 100,000 labeled records, the chosen features encode test-set label information, so every test accuracy and AUC in Tables III-VI is optimistically biased and the model ranking is not a fair comparison. The paper's own numbers also contradict the headline: Table VI reports CNN test accuracy 99.99% and DNN 99.90%, while the Results section gives DNN AUC 0.9993; the conclusion instead credits DNN with 99.99% accuracy and 100% AUC. Finally, Section III.B says a hashed column was 'unhashed using the hashlib library to retrieve the original data'; for a cryptographic hash this is not generally possible, which further undermines confidence that the data handling is correctly described. The DNN-superiority claim is therefore not supported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares several machine learning and deep learning classifiers (RF, SVC, LR, KNN, MLP, CNN, LSTM, DNN) for malware detection on a 100,000-record Kaggle dataset, using Recursive Feature Elimination to select 25 features and reporting accuracy, AUC, MCC, and Kappa. The central claim is that the DNN outperforms all other models, with the abstract reporting 99.92% training accuracy and near-perfect AUC, and the conclusion claiming 99.99% accuracy and 100% AUC. The manuscript also includes a comparison with prior studies, positioning the DNN as achieving the best reported accuracy.","tokens_in":7737,"tokens_out":3690,"duration_ms":35347,"significance":"The problem addressed is relevant, and the paper makes an effort to compare several standard classifiers under a common preprocessing pipeline. If the results were valid, the near-perfect detection rates would be a useful data point for the malware-detection literature. However, the reported evidence does not support the central claim: the experimental protocol leaks test-set information into feature selection, the tables and prose contradict each other, and the DNN is not the highest-accuracy model in the paper's own Table VI. The manuscript also provides no code or detailed reproducibility artifacts, so the numerical claims cannot be independently checked. These issues are load-bearing rather than presentational.","major_comments":[{"comment":"Recursive Feature Elimination is applied to the full 100,000-record dataset before the 80/20 train-test split. Because RFE is fit on all labeled records, the chosen 25 features encode test-set label information, making every test accuracy, AUC, MCC, and Kappa value in Tables III-VII optimistically biased. The model ranking is therefore not a valid comparison, and the central claim of DNN superiority is not supported by the experimental protocol.","section":"Section III.B, Fig. 2"},{"comment":"Table VI reports CNN test accuracy of 99.99% and DNN test accuracy of 99.90%, yet the Conclusion credits the DNN with 99.99% accuracy and a perfect AUC of 100%. Section V also states the DNN achieves 0.9993 AUC while Table VI lists DNN AUC as 100%, and Table VII gives CNN MCC/Kappa 99.46% versus DNN 99.99%/100%. These internal contradictions mean the paper's own reported numbers do not establish that the DNN is the best model.","section":"Section IV.B, Table VI, and Conclusion"},{"comment":"The preprocessing description states that 'A hashed column was unhashed using the hashlib library to retrieve the original data.' For a cryptographic hash, recovering the original data is not generally possible. This claim suggests the data handling is not correctly understood or described, and it undermines confidence in the entire preprocessing pipeline.","section":"Section III.B"},{"comment":"The 'proposed' DNN is selected after observing its performance on the same test set used to evaluate all models, with no separate validation set or model-selection procedure. Combined with the RFE leakage described above, this makes the DNN-superiority claim a post-hoc selection artifact rather than a principled finding.","section":"Section V"},{"comment":"The prose states that Multi-Layer Perceptron and Random Forest achieve test accuracy of 99.99% and perfect 100% AUC, but Table III lists RF test accuracy as 99.98% with AUC 99.8% and MLP accuracy as 99.99% with AUC 99.6%. The table and text disagree on these key metrics, so the reported performance numbers cannot be relied on as consistent evidence.","section":"Section IV.A, Table III"}],"minor_comments":[{"comment":"The caption text 'Figure ?? shown that all the deep learning model have a perfect roc curve' contains an unresolved cross-reference and should be fixed.","section":"Section IV.B"},{"comment":"AUC values are inconsistently expressed as percentages and decimals (e.g., DNN AUC appears as 0.9993 in Section V and 100% in Table VI); the notation should be unified throughout.","section":"Tables V, VI, and VII"},{"comment":"The sentence 'Although its performance was not as high as the CNN with an accuracy of 96.41%' has an unclear pronoun antecedent; the intended model should be identified explicitly.","section":"Section II"},{"comment":"Hyperparameters are given only for the DNN; the CNN and LSTM architectures and training configurations are not specified, which limits the reproducibility of the deep-learning comparison.","section":"Tables V-VIII"}],"recommendation":"reject","confidential_remarks":"The data leakage from applying RFE before the train-test split is a fundamental flaw that invalidates all reported test metrics and the model ranking. Even setting that aside, the manuscript's own tables contradict the headline claim that the DNN is the most accurate model. A corrected version would require rerunning the entire experimental pipeline, and the inconsistencies would need to be resolved before the results could be taken seriously. I do not see how these issues can be fixed within the scope of a normal revision, so I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the paper's central claim does not survive contact with its own tables or its own methodology. The DNN-is-best conclusion is contradicted by Table VI, where CNN test accuracy (99.99%) exceeds DNN (99.90%), and the preprocessing pipeline leaks test information into the training set via RFE before the split.\n\nWhat the paper does well: it runs a standard set of ML and DL models on a public Kaggle malware dataset and reports standard metrics. The DNN hyperparameters are explicitly listed, which is useful for reproduction. The related-work section is adequate and shows prior benchmarks in the same accuracy range.\n\nThe problems are load-bearing, not cosmetic. First, data leakage. Section III.B says RFE selects the top 25 features using the full 100,000-record dataset before the 80/20 split, and Figure 2 confirms this order. That means the selected features encode test-set labels, so every test accuracy and AUC is optimistically biased and the model ranking is not a fair comparison. Second, the central claim is internally inconsistent. Table VI reports CNN test accuracy 99.99% and DNN 99.90%, with DNN AUC 0.9993. The conclusion credits DNN with 99.99% accuracy and a perfect AUC of 100%. These cannot all be right. Third, the text asserts a hashed column was 'unhashed using the hashlib library to retrieve the original data.' Cryptographic hashes are one-way; this is not generally possible, and the statement suggests the data handling is not well understood.\n\nThe study is a routine benchmark: stock models, public dataset, no new method or reusable artifact. The reported accuracies match prior work. As it stands, the paper's conclusion is unsupported by its own evidence.\n\nWho is this for? Possibly as a negative teaching example in a methods class, but not as a research result. I would not cite it, and I would not send it to peer review in its current form. The authors could redo the experiments with a proper train/test split before feature selection and fix the internal inconsistencies, but that would be a substantially different paper.","headline":"Data leakage and internal contradictions sink the DNN-superiority claim; this is a routine benchmark that should be desk-rejected.","tokens_in":1,"tokens_out":2419,"would_cite":false,"duration_ms":58986,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a deep neural network detects malware with up to 99.99 percent accuracy and a perfect AUC, outperforming random forest, MLP, CNN, and LSTM on a 100,000-record dataset.","keywords":["malware detection","deep neural network","machine learning","random forest","feature selection","recursive feature elimination","AUC","binary classification"],"falsifier":"Rerun the same pipeline with the same dataset but move recursive feature elimination inside the 80/20 split, fitting it only on the 80% training portion, and compare the DNN's test accuracy and AUC to the paper's reported 99.90-99.99% and 100%. If the accuracy drops substantially, the original result depended on test-set information rather than on the DNN itself. A second test is to check whether 'unhashing' with hashlib is possible for the dataset's hash column; if the column is a cryptographic hash, no Python library can recover the original values, and the preprocessing description is not reproducible.","tokens_in":7279,"feed_emoji":"🛡️","tokens_out":9329,"duration_ms":74036,"temperature":0.7,"pith_summary":"The paper tries to show that a deep neural network can detect malware more accurately than classical machine-learning classifiers if the input features are first carefully selected and cleaned. On a 100,000-sample public malware dataset, the authors report their DNN reaching a training accuracy of 99.92%, test accuracy up to 99.99%, and a perfect AUC, beating random forest, support vector classifier, logistic regression, K-nearest neighbors, CNN, and LSTM. The authors credit the combination of feature selection and preprocessing for the DNN's near-perfect separation of benign and malicious samples. If the result is real, it would make DNNs the strongest candidate for building intelligent malware detection systems against newly emerging threats. It would also show that feature engineering remains a decisive factor even when deep models are used.","feed_headline":"DNN tops malware detection at 99.99 percent accuracy","feed_subtitle":"The deep net edges out random forest and MLP with near-perfect scores on a 100,000-record dataset.","key_machinery":"The model that carries the argument is a fully connected deep neural network with two hidden layers of 128 and 64 units, ReLU activation, dropout at 0.5, and a sigmoid output, trained with the Adam optimizer on binary cross-entropy. What the paper treats as the enabling mechanism is the preprocessing pipeline that feeds it: label-encoding two object columns, removing z-score outliers, and applying recursive feature elimination to select the top 25 of 35 features. The same 25-feature set is then used for every classifier, so the DNN's reported edge is attributed to the model architecture plus this prepared input.","core_discovery":"The paper's central claim is that a deep neural network outperforms all traditional models in malware detection, with the highest training accuracy of 99.92% and near-perfect AUC, and the conclusion states an accuracy of 99.99% with a perfect AUC of 100%. The DNN uses two hidden layers (128 and 64 units), ReLU activations, dropout, and Adam optimization with binary cross-entropy. The authors report zero false positives for the DNN, a Matthews correlation coefficient of 99.99%, and a Kappa score of 100%, which they interpret as the model capturing the underlying benign-versus-malware pattern without overfitting. They further claim the DNN outperforms CNN and LSTM, with the DNN's confusion matrix showing no omissions, making it the most suitable approach among those studied.","pith_inferences":["Because recursive feature elimination is applied before the train-test split rather than inside it, the 25 features are selected using information from the test records; a fair re-run would likely show a smaller gap between the DNN and the other classifiers.","The claim that a hashed column was 'unhashed' with hashlib is not physically possible for cryptographic hashes, so the paper's description of the data handling cannot be taken at face value.","A direct test of the paper's general claim would be to train the same DNN on a different malware dataset, such as EMBER or Microsoft Big, and see whether the near-perfect accuracy transfers.","The dataset features named in the selected list (for example, 'mm users' and 'prio reserved vm') resemble Linux process-scheduling counters rather than network traffic attributes, so the type of data the model is actually seeing deserves scrutiny."],"forward_implications":["If the ranking holds, DNN becomes the model of choice for malware detection on this type of data, ahead of random forest, SVC, and MLP.","The reported MCC of 99.99% and Kappa of 100% are interpreted by the authors as proof the model is not just accurate but reliable in classifying all samples.","The paper's comparison against previous studies positions its DNN above reported results across several datasets, including EMBER and Windows PE.","A malware detector with the DNN's reported zero false positives would flood an analyst's queue with almost no benign alerts, easing manual triage."],"supporting_citations":[{"why":"Supplies the 100,000-sample dataset and 35 features that all experiments use.","marker":"[16]"},{"why":"Provides the 99% decision-tree baseline that the proposed DNN is compared against.","marker":"[7]"},{"why":"Supplies the 99.44% random forest result on Windows malware, a key ML comparator.","marker":"[8]"},{"why":"Gives the 87.76% deep-learning baseline on EMBER that the paper claims to improve.","marker":"[9]"},{"why":"Contributes a 99.24% ensemble-learning baseline from the PE malware dataset.","marker":"[10]"},{"why":"Provides the 99.743% LSVC hybrid-feature result that is the strongest prior accuracy in the comparison table.","marker":"[12]"},{"why":"Supports the paper's premise that preprocessing meaningfully changes model performance.","marker":"[15]"}],"fun_headline_variants":["DNN hits 99.99% accuracy in malware detection","Deep neural net beats classic models on malware","99.99% accuracy: DNN dominates malware detection","Malware detection: DNN outperforms with near-perfect score","DNN achieves 99.99% accuracy, zero false positives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the preprocessing pipeline is sound, meaning recursive feature elimination selects the 25 features without seeing the test set and a hashed column can indeed be 'unhashed' with hashlib; if either is false, the reported accuracies no longer rank the models as claimed.","fun_headline_variants_meta":{"raw":{"variants":["DNN hits 99.99% accuracy in malware detection","Deep neural net beats classic models on malware","99.99% accuracy: DNN dominates malware detection","Malware detection: DNN outperforms with near-perfect score","DNN achieves 99.99% accuracy, zero false positives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000508,"raw_usage":{"total_tokens":2472,"prompt_tokens":940,"completion_tokens":1532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":1449}},"tokens_in":556,"tokens_out":1532,"duration_ms":11927,"temperature":1.0,"reasoning_tokens":1449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:28:35.642479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same pipeline with the same dataset but move recursive feature elimination inside the 80/20 split, fitting it only on the 80% training portion, and compare the DNN's test accuracy and AUC to the paper's reported 99.90-99.99% and 100%. If the accuracy drops substantially, the original result depended on test-set information rather than on the DNN itself. A second test is to check whether 'unhashing' with hashlib is possible for the dataset's hash column; if the column is a cryptographic hash, no Python library can recover the original values, and the preprocessing description is not reproducible.","supporting_citations":[{"cited_title":"Malware Detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the 100,000-sample dataset and 35 features that all experiments use."},{"cited_title":"”Malware Analysis and Detection Using Machine Learning Algorithms.” Symmetry 14, no","cited_arxiv_id":null,"evidence_quote":"Provides the 99% decision-tree baseline that the proposed DNN is compared against."},{"cited_title":"Arslan Raza","cited_arxiv_id":null,"evidence_quote":"Supplies the 99.44% random forest result on Windows malware, a key ML comparator."},{"cited_title":"”A Novel Approach towards Windows Malware Detection System Using Deep Neural Networks.” Procedia Computer Science 215 (2022): 148-157","cited_arxiv_id":null,"evidence_quote":"Gives the 87.76% deep-learning baseline on EMBER that the paper claims to improve."},{"cited_title":"”Windows PE Malware Detection Using Ensemble Learning.” Informatics 8, no","cited_arxiv_id":null,"evidence_quote":"Contributes a 99.24% ensemble-learning baseline from the PE malware dataset."},{"cited_title":"Windows malware detection system based on LSVC recommended hybrid features,","cited_arxiv_id":null,"evidence_quote":"Provides the 99.743% LSVC hybrid-feature result that is the strongest prior accuracy in the comparison table."},{"cited_title":"”Data preprocessing impact on machine learning algorithm performance.” Open Computer Science 13, no","cited_arxiv_id":null,"evidence_quote":"Supports the paper's premise that preprocessing meaningfully changes model performance."}],"review_version":1}