{"id":"f044ae0d-a80c-47dc-b42f-da92ce8855f1","arxiv_id":"2608.03250","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A random forest plus multilayer perceptron stacking model reached 97.5% accuracy on CICMalDroid 2020 Android malware data, exceeding its single-model parts.","lead":"ShielDroid is an Android malware detector that combines two machine learning models: a random forest first, then a neural network that also sees the forest's predictions. On the CICMalDroid 2020 dataset it reports 97.5% accuracy, though the evaluation has gaps a reader should check.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 97.5% accuracy is compromised by data leakage: feature selection uses the full dataset before the 80/20 split, and Random Forest predictions fed to the MLP are not shown to be out-of-fold.","rationale":"The reader's weakest assumption correctly identifies the two most serious threats to the central claim: feature selection performed before the train/test split, and ambiguity about whether the Random Forest predictions used as MLP input are out-of-fold. Both are confirmed by the manuscript text. The paper's ordering in Section V explicitly places feature selection before the split, which is a textbook data-leakage error that can inflate accuracy by allowing the test set to influence which features are chosen. Section III-C describes training the MLP with an additional RF prediction column but does not specify the protocol; the most straightforward reading is that the RF model used to generate predictions for the training set was also trained on that same set, producing in-sample predictions that are artificially correlated with the target. This would let the MLP over-rely on that column during training, and the reported test accuracy would reflect a meta-model that was trained with leaked information. The paper also lacks code and data, so there is no way to verify the magnitude of the bias. These concerns are load-bearing because the paper's sole quantitative contribution is the 97.5% accuracy; without a clean evaluation, the central claim is unsupported. The reader's verdict of REJECT is therefore appropriate, and our stress-test does not change that verdict.","tokens_in":6999,"tokens_out":3155,"duration_ms":29508,"concrete_test":"Re-run the full ShielDroid pipeline with two corrections: (1) perform SelectKBest feature selection inside the training fold only, and (2) generate the Random Forest prediction column for the training set via 5-fold cross-validation (out-of-fold) and for the test set with the model trained on the full training fold. Then train the MLP on this leakage-free dataset and evaluate on the original test split. If the accuracy falls materially below 97.5% (or below standalone RF/MLP), the original evaluation protocol is the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V states: \"The top 120 features were extracted out of the 470 features from the dataset during feature selection using weka and sklearn. The dataset was then divided into train and test data in an 80:20 ratio.\" This ordering means the test split contributes to feature selection, so the reported accuracy is not an unbiased estimate of performance on new data. Additionally, Section III-C describes using \"the prediction result\" from Random Forest as an additional input to the MLP but does not specify whether these predictions were generated out-of-fold. If the MLP is trained on in-sample RF predictions, those predictions encode the true training labels, leaking target information into the second-stage features and inflating accuracy. Both issues are load-bearing because the central claim is that the hybrid model attains 97.5% accuracy; a corrected evaluation could substantially lower this figure. The paper provides no code or data to assess the magnitude of the bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents ShielDroid, a hybrid Android malware detector. On the CICMalDroid 2020 dataset (five categories), the authors propose selecting the top 120 features via ANOVA, training a Random Forest (RF), and using the RF prediction as an additional input to a multilayer perceptron (MLP). They report that this RF+MLP hybrid achieves 97.5% accuracy with a training time of 22.945 seconds, outperforming RF, MLP, SVM, KNN, and Naive Bayes. The central claim is that the hybrid stacking approach improves detection accuracy over either model alone.","tokens_in":7251,"tokens_out":5429,"duration_ms":61153,"significance":"If the reported performance were obtained under a sound evaluation protocol, this would be a useful empirical benchmark: the paper uses a public dataset, compares several standard baselines, and includes per-class metrics and a confusion matrix. The explicit reporting of training time is also a positive. However, the manuscript provides no code or data, and the evaluation protocol as described contains two forms of target-information leakage (feature selection before the split and ambiguous generation of the RF prediction column). The headline 97.5% accuracy therefore cannot currently be interpreted as a generalization estimate, and the contribution is conditional on a corrected experimental evaluation.","major_comments":[{"comment":"Feature selection is performed on the full dataset before the train/test split. The paper states: 'The top 120 features were extracted out of the 470 features from the dataset during feature selection using weka and sklearn. The dataset was then divided into train and test data in an 80:20 ratio.' Because the test portion contributes to the chosen feature set, the reported 97.5% accuracy is optimistically biased and is not an unbiased estimate of performance on unseen data. Feature selection must be nested inside the training folds (e.g., in a pipeline or cross-validation loop), and all reported metrics must be recomputed under that protocol.","section":"Section V"},{"comment":"The stacking procedure uses 'the prediction result' of Random Forest as an additional input to the MLP, but the paper does not state whether those predictions were generated out-of-fold. If the RF predictions are in-sample predictions on the same training data used to train the MLP, they encode the true training labels and leak target information into the second-stage classifier, which can substantially inflate the reported accuracy. The authors must specify whether out-of-fold predictions were used; if not, the experiment must be rerun with out-of-fold RF predictions.","section":"Section III-C"},{"comment":"The main quantitative claim is internally inconsistent. The abstract and one sentence in Section V report the hybrid accuracy as 97.5%; Section V also says the proposed model 'gave the highest accuracy of 0.98' and Table I lists Accuracy as .98. Section V reports RF alone as 93.6% in one sentence and 94% in the comparison list. The discrepancy between 97.5% and 0.98, and between 93.6% and 0.94, must be resolved; otherwise the headline result is ambiguous.","section":"Sections V and VI, Table I"},{"comment":"The dataset size is inconsistent. Section IV states that 13,077 out of 17,341 samples were successfully analyzed by CopperDroid, while Section V says the dataset used has 11,598 rows and 470 columns. The relationship between these numbers is unexplained. Since the entire contribution is empirical, the exact preprocessing chain that yields the 11,598-row dataset must be documented, including any filtering or deduplication steps.","section":"Sections IV and V"}],"minor_comments":[{"comment":"A stratified train-test split is mentioned only at the end of Section VI, after the experimental results are described, while Section V refers only to an 80:20 split. Please state at the point of the split whether stratification was applied and report the class proportions in train and test.","section":"Section VI"},{"comment":"There is a recurring typo: 'MPL' should be 'MLP' (e.g., in Section VI). Please correct throughout.","section":"Section III-C"},{"comment":"The confusion matrix is not readable in the compressed figure and the numerical values are not described in the text beyond macro averages. Please provide a larger figure or a table with the full confusion matrix.","section":"Figure 3"},{"comment":"The sentence 'The above figure demonstrates...' appears before the figure is displayed. Please fix the cross-reference so the text points to the figure correctly.","section":"Figure 4"},{"comment":"No information is given about the computing environment used for the timing measurements (CPU/GPU, memory, operating system). Without this, the reported execution time of 22.945 seconds is not reproducible or comparable.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-protocol problems are serious and the current numbers should not be cited. I recommend major revision rather than immediate rejection because the misuse can in principle be corrected by re-running the experiments with proper feature selection and out-of-fold stacking, and the idea is simple and potentially useful. However, if the authors cannot supply corrected results and code/data for verification, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this before spending time on it: the central accuracy claim is undermined by a textbook data-leakage error, and the real-time claim is based on training time, not inference. The rest is a routine stacking exercise on a public benchmark.\n\nWhat the paper does well: it builds a hybrid where a Random Forest's predictions become an extra input feature to an MLP on CICMalDroid 2020, and it compares against several baselines with confusion matrices and a classification report. The writing is clear enough to follow. That is honest empirical work buried under the errors.\n\nThe soft spots are serious. Section V states: \"The top 120 features were extracted out of the 470 features from the dataset during feature selection using weka and sklearn. The dataset was then divided into train and test data in an 80:20 ratio.\" That is leakage: the test set influences feature choice, so the reported 97.5% is an optimistic estimate. The RF+MLP description does not say whether the RF predictions fed to the MLP were out-of-fold; in-sample predictions would leak target information into the second stage. The paper also conflates 22.945 seconds of training time with \"real-time\" detection, which is training cost, not per-sample latency. All three touch the central claim.\n\nThere are smaller problems: abstract says 97.5%, classification report says 98%; the prototype application mentioned in the introduction never appears later; some references are malformed. No code or data is provided, so the leakage bias cannot be quantified.\n\nWho this is for: it could serve as a cautionary example of why feature selection must be nested inside cross-validation. It is not a useful reference for state-of-the-art Android malware detection, since published detectors already exceed 99% on similar datasets.\n\nMy recommendation: desk reject. If the authors redo the evaluation with a proper nested protocol and honestly report inference time, the topic could support a short workshop paper, but this submission is not there.","headline":"Routine stacking paper with a load-bearing data leakage bug; reported accuracy and real-time claims are not credible.","tokens_in":7726,"tokens_out":3611,"would_cite":false,"duration_ms":36955,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Random Forest plus neural net catches Android malware at 97.5% accuracy.","keywords":["Android malware detection","dynamic analysis","Random Forest","multilayer perceptron","hybrid model","machine learning","CICMalDroid 2020","cyber-security"],"falsifier":"Re-run the pipeline with the independence conditions enforced: pick the top 120 features inside each cross-validation fold and produce the Random Forest prediction column out-of-fold, so no test-row information reaches the MLP. If accuracy falls to near the 93–94% level of the individual models, the reported gain is leakage; if it stays at 97.5%, the hybrid effect is real and the paper's claim survives.","tokens_in":6945,"feed_emoji":"🛡️","tokens_out":8849,"duration_ms":87501,"temperature":0.7,"pith_summary":"The paper tries to establish that a hybrid classifier — a Random Forest whose predictions are fed as an extra column into a multilayer perceptron — detects Android malware more accurately than any single classifier it tested, and fast enough to be used in near-real time. On the CICMalDroid 2020 dataset of 11,598 dynamically analyzed apps, the hybrid reaches 97.5% accuracy with a combined training time of 22.9 seconds, beating Random Forest (94%), the MLP alone (93%), KNN (84%), Naive Bayes (53%), and SVM (45%). The authors argue this matters because static screening misses malware that only activates at runtime, so a fast, accurate dynamic classifier could catch malicious apps before they damage devices. The paper also shows the hybrid repairs specific blind spots: it correctly identifies 89 samples Random Forest got wrong and 308 samples the MLP got wrong, most of them Banking malware.","feed_headline":"Hybrid model catches Android malware with 97.5% accuracy","feed_subtitle":"Random Forest predictions fed into a neural net beat either model alone, training in about 23 seconds.","key_machinery":"The load-bearing object is the two-stage stack: the Random Forest's 'predicted column' — one class label per sample, produced by 100 decision trees — is treated as an ordinary input feature (the 121st) for the multilayer perceptron. The paper's stated rationale is that this column gives the neural network a distilled, higher-correlation signal to learn from, so the net inherits the forest's strengths and corrects residual errors; the 89 plus 308 recovered samples are offered as evidence that the extra column, not either model alone, carries the gain.","core_discovery":"The paper claims that stacking works: a Random Forest (100 trees, Gini-impurity splits) trained on 11,598 Android apps described by 120 selected behavioral features emits a predicted class per sample; that prediction is appended as an extra column, and a multilayer perceptron (four hidden layers, 256/128/64/32 neurons, ReLU, 5-way softmax) is trained on the resulting 121 inputs. It reports 97.5% accuracy versus 93.6–94% for Random Forest alone and 93% for MLP alone, with total training time of 22.945 seconds. Per class: 99% SMS malware, 97% Adware and Benign, 96% Banking and Riskware; macro precision 0.98, recall 0.97, F1 0.98. The combination is claimed to repair real blind spots: 89 sample","pith_inferences":["My reading: the 'predicted column' trick is a two-stage stacking ensemble, and its gain should be sensitive to how the forest's prediction is produced; testing out-of-fold RF predictions would separate a genuine ensemble effect from target leakage, a check the paper does not report.","My reading: the paper measures 'execution time' as training time, so the real-time claim is really about fast retraining; a field deployment would need per-app inference latency and memory footprint measured before calling it real-time protection.","My reading: the dataset is class-imbalanced (SMS malware is roughly a third of samples), so the reported macro precision 0.98 / recall 0.97 are the informative numbers; a rerun on a balanced subset would show whether 97.5% survives when classes are equalized."],"forward_implications":["If the claim holds, stacking RF predictions into an MLP is a cheap accuracy boost: no new features or data, just re-use of the forest's own outputs, lifting accuracy roughly 3–4 points over either component.","The combined training cost of about 23 seconds means the model can be rebuilt quickly on fresh malware samples, which is the paper's premise for keeping detection current in real-world deployment.","The class-level recoveries — especially 124 Banking malware samples the MLP alone missed — imply the hybrid helps most on categories where the single models are weakest.","The 470-to-120 feature reduction via ANOVA gives the trained model a small input footprint, consistent with the paper's claim that the framework transfers to new high-dimensional datasets."],"supporting_citations":[{"why":"Supplies the CICMalDroid 2020 dataset of dynamically analyzed Android samples; every reported accuracy is measured on this data.","marker":"[13]"},{"why":"The ANOVA reference behind the SelectKBest feature-selection step that reduces 470 features to the top 120.","marker":"[11]"},{"why":"Cited for the MLPClassifier implementation used to build the neural-network stage that receives the Random Forest prediction column.","marker":"[2]"},{"why":"SeqMobile, the sequence-based detector with 97.85% accuracy whose run-time-performance goal the hybrid is presented as meeting.","marker":"[1]"},{"why":"A prior hybrid (deep autoencoder plus CNN) reaching high accuracy slowly; the comparison point for why a faster hybrid is worth building.","marker":"[10]"}],"fun_headline_variants":["Stacked RF-MLP beats single models at 97.5% accuracy","Hybrid detection: RF+MLP stack scores 97.5% on Android","Malware classifier stacks RF and neural net to 97.5%","Random Forest plus MLP raises Android malware detection to 97.5%","97.5% Android malware accuracy achieved by stacked RF-MLP"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The 97.5% figure assumes the 20% test set stayed fully independent of model construction, but the top 120 features were selected from all 470 before the 80:20 split, so test rows influenced which features were used, and the paper does not state that the Random Forest prediction column given to the MLP was generated out-of-fold rather than from the rows the MLP trained on.","fun_headline_variants_meta":{"raw":{"variants":["Stacked RF-MLP beats single models at 97.5% accuracy","Hybrid detection: RF+MLP stack scores 97.5% on Android","Malware classifier stacks RF and neural net to 97.5%","Random Forest plus MLP raises Android malware detection to 97.5%","97.5% Android malware accuracy achieved by stacked RF-MLP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2265,"prompt_tokens":806,"completion_tokens":1459,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1373}},"tokens_in":550,"tokens_out":1459,"duration_ms":12775,"temperature":1.0,"reasoning_tokens":1373,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:29:04.430452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline with the independence conditions enforced: pick the top 120 features inside each cross-validation fold and produce the Random Forest prediction column out-of-fold, so no test-row information reaches the MLP. If accuracy falls to near the 93–94% level of the individual models, the reported gain is leakage; if it stays at 97.5%, the hybrid effect is real and the paper's claim survives.","supporting_citations":[{"cited_title":"Ambielli, Gini Impurity (With Examples), Oct","cited_arxiv_id":null,"evidence_quote":"Supplies the CICMalDroid 2020 dataset of dynamically analyzed Android samples; every reported accuracy is measured on this data."},{"cited_title":"Analysis of variance (anova),","cited_arxiv_id":null,"evidence_quote":"The ANOVA reference behind the SelectKBest feature-selection step that reduces 470 features to the top 120."},{"cited_title":"Hendricks, Study.com — Take Online Courses","cited_arxiv_id":null,"evidence_quote":"Cited for the MLPClassifier implementation used to build the neural-network stage that receives the Random Forest prediction column."},{"cited_title":"SeqMobile: A Sequence Based Efficient Android Malware Detection System Using RNN on Mobile Devices","cited_arxiv_id":"2011.05218","evidence_quote":"SeqMobile, the sequence-based detector with 97.85% accuracy whose run-time-performance goal the hybrid is presented as meeting."},{"cited_title":"Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"A prior hybrid (deep autoencoder plus CNN) reaching high accuracy slowly; the comparison point for why a faster hybrid is worth building."}],"review_version":1}