{"id":"6c4f9811-edb9-4501-ac2c-daa3692a478d","arxiv_id":"1909.03837","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A genetic algorithm-selected ensemble of deep belief networks is reported to improve Android malware detection accuracy and robustness under a single 10% label noise scenario.","lead":"A Peking University team applies selective ensemble learning, using a genetic algorithm to pick a subset of neural networks, for Android malware detection. The method reports higher average precision and recall than a single deep network when 10% of training labels are corrupted, but no variance or code is provided.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central robustness numbers are arithmetically incompatible with a corrupted test set; the paper never states whether the 10% label swap reaches the test partition.","rationale":"The reader's weakest assumption focused on whether 10% random label swapping faithfully simulates real-world noise. That is a legitimate concern, but the more immediate and load-bearing issue is that the paper does not specify whether the test partition itself is corrupted. If it is, the central numeric evidence in Table 2 is not merely optimistic but arithmetically impossible for any generalizing classifier. If it is not, the paper must say so explicitly, because the current wording implies that the modified dataset on which SEdroid achieves 98.3% precision and 98.1% recall includes swapped test labels. This ambiguity directly undermines the strongest claim of robustness. The recommended verdict remains CONDITIONAL rather than REJECT because a corrected experiment with clean test labels could plausibly support the claim, but the condition is now much sharper: the evaluation protocol must be clarified and the reported numbers reconciled with the noise model. The reader's stated concern about noise realism is adjacent but not identical, hence partial agreement.","tokens_in":4555,"tokens_out":5169,"duration_ms":60455,"concrete_test":"Determine whether the 10% swap was applied before or after the train/test split, and recompute Table 2 under two explicit protocols: (i) clean test labels with corrupted training labels, and (ii) corrupted test labels. If protocol (ii) is used, compare against the analytical ceiling of 90% precision/recall; any reported value above that demonstrates leakage or test-set overfitting. A one-line clarification of the split order plus the recomputed table settles the concern.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1 says a \"modified dataset\" is created by swapping 10% of malicious and benign apps, and that SEdroid is evaluated on both datasets; the 60/20/20 split is described after this modification. If the swap is applied to the test partition, which is the natural reading, then a perfect true-label classifier cannot exceed 90% precision and recall on a balanced test set: among true malicious apps 10% carry benign labels and vice versa, so precision = recall = 0.45 / 0.5 = 0.90. Table 2 reports SEdroid at 98.3% precision and 98.1% recall on the modified dataset. These numbers are impossible under that reading. The text never disambiguates between corrupting training labels only and corrupting the whole dataset, and the reported metrics are not accompanied by the variance across the 30 repeated experiments. Since the entire robustness claim rests on Table 2, the evaluation protocol is the load-bearing weakness: either the test set was clean and the text is misleading, or the metrics are computed against corrupted test labels and reflect fitting the noise rather than detecting malware.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEdroid, an Android malware detector that builds an ensemble of deep belief network (DBN) component learners trained on bootstrap samples and then uses a genetic algorithm to select a subset of these learners by a fitness function combining accuracy and diversity. The evaluation uses a dataset of 16,000 Android apps (8,000 malicious and 8,000 benign) and compares SEdroid against SVM, DroidDeep, and an ensemble of SVM classifiers on an original dataset and on a modified dataset in which 10% of malicious and benign labels are swapped. The paper reports that SEdroid achieves 98.3% precision and 98.1% recall on the modified dataset, versus 93.8% and 94.3% for DroidDeep on average, and interprets this as evidence of the superiority and robustness of selective ensembling.","tokens_in":4793,"tokens_out":4234,"duration_ms":47575,"significance":"If the reported results were sound, the paper would make a useful contribution by showing that selective ensemble learning can improve Android malware detection under label noise. The general idea of jointly optimizing accuracy and diversity with a genetic algorithm is reasonable, and the use of 30 repeated experiments is a good instinct. However, the central robustness claim is not adequately supported as presented: the evaluation protocol for the modified dataset is ambiguous and appears arithmetically inconsistent with the reported numbers, and the paper lacks the statistical detail, baseline comparisons, and hyperparameter information needed to confirm that selective ensembling is the source of the improvement. The contribution is therefore promising but unverified.","major_comments":[{"comment":"The evaluation protocol for the modified dataset is ambiguous and, under the natural reading, arithmetically inconsistent. Section 3.1 states that 10% of malicious and benign apps are swapped to create a modified dataset and that SEdroid is evaluated on both datasets, with the 60/20/20 split described afterward. If the swap is performed before splitting, then the test partition contains corrupted labels, and no true-label classifier can exceed 90% precision and recall on a balanced test set: with 10% of each class relabeled, precision = recall = 7,200/8,000 = 0.90. Table 2 reports SEdroid at 0.983 precision and 0.981 recall on the modified dataset, which is impossible under that reading. The manuscript never states whether the swap reaches the test partition or how the metrics are computed relative to the corrupted labels. Please clarify the protocol and, if the test set is corrupted, recompute the reported metrics on clean test labels; as written, the main robustness result is not supported.","section":"Section 3.1 and Table 2"},{"comment":"The 30 repeated experiments are summarized only by worst, best, and average values for DroidDeep and by a single average for SEdroid. No standard deviation, confidence interval, or paired significance test is reported. Given that DroidDeep's best precision and recall are 0.967 and 0.960, respectively, and SEdroid's average is 0.983 and 0.981, the claimed superiority could be within run-to-run variation. Please report the dispersion of SEdroid's results across the 30 runs and a paired statistical comparison against DroidDeep.","section":"Section 3.3"},{"comment":"The genetic algorithm is not specified enough to reproduce the experiments or to judge whether the selection process is responsible for the reported improvement. The manuscript omits the population size, the number of iterations, crossover and mutation probabilities, the selection strategy, and the number N of bootstrap component learners. In addition, the fitness function F(x) = Accuracy x D in Eq. (2) uses accuracy on the training data, but the paper does not state whether the validation split is used to guide selection or to avoid overfitting to the noisy training labels. These details are essential for a claim about robustness.","section":"Section 2.3 and Algorithm 1"},{"comment":"There is no comparison with a full ensemble baseline that does not perform selection. Table 1 reports an ensemble of SVM learners and SEdroid, which is an ensemble of DBN learners, but there is no 'ensemble of all DBN learners' row. Consequently, the improvement over DroidDeep could be due to ensembling itself rather than to the selective mechanism, especially because Table 1 already shows that ensembling SVM improves over single SVM. The paper should directly compare SEdroid against its own full ensemble with all component learners included.","section":"Section 3.3 and Table 1"},{"comment":"The noise model is limited to a single level of label swapping, and the structure of the simulated noise is not analyzed. Real-world mislabeling may be correlated with application features, concentrated in particular malware families, or present at rates different from 10%, and the paper provides no evidence that the reported advantage persists under such conditions. The conclusion that SEdroid is robust to 'low-quality datasets' in general is broader than what the current experiment can support.","section":"Section 3.1 and Section 3.3"}],"minor_comments":[{"comment":"The text refers to 'Fig.6', but the manuscript contains only Figure 1; please correct the cross-reference.","section":"Section 3.3"},{"comment":"In Eq. (3), the symbol N is used for both the number of component learners and the normalization factor in the diversity measure, which is confusing; please clarify and define the range of the outer sum explicitly.","section":"Equation (3)"},{"comment":"The column headers 'Precision(M)' and 'Recall(M)' are not defined; please state explicitly that M denotes the malware class and describe whether the reported metrics are per-class or macro-averaged values.","section":"Table 1 and Table 2"},{"comment":"Several references have incomplete or inconsistent bibliographic information, such as reference [9], which lists both 'Sarnoff Symposium, 2016 IEEE' and the year 2017; please unify the reference format.","section":"References"},{"comment":"The manuscript would benefit from language editing; for example, Section 3.2 contains the phrase 'almostly correct', and several sentences in the introduction and conclusion are awkwardly constructed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central table's internal consistency is the key issue. If the modified dataset's test partition is indeed corrupted, the reported SEdroid numbers cannot be correct, and the authors would need to rerun the evaluation with clean test labels or with a clearly specified training-only corruption protocol. This is fixable within the scope of the manuscript, so I am not recommending rejection, but the current version should not be accepted as is. The paper is very short, and the experimental section needs substantially more detail on the experimental setup and statistical analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing you should know before reading SEdroid: the headline numbers in Table 2 don't survive contact with the protocol description. The authors say they swap 10% of malicious and benign labels to build the modified dataset, then split 60/20/20, and report SEdroid at 98.3% precision, 98.1% recall on that modified dataset. If the swap reaches the test partition, which is the natural reading, a perfect true-label classifier scores at most 90% on a balanced set: 450 of 500 true positives carry the malicious label, and 450 of 500 predicted positives are truly malicious. So 98.3% is impossible against corrupted test labels. If the swap is only in the training set, the numbers become plausible, but the text never says that, and 'evaluated on both datasets' suggests otherwise. That ambiguity is load-bearing because the whole robustness claim rests on it.\n\nWhat's genuinely there: this is the first application of selective ensemble learning to Android malware detection, as far as the citations go. The fitness function multiplies an ensemble's accuracy by a pairwise Euclidean-distance diversity term, which is a sensible adaptation of Zhou's many-could-be-better-than-all idea. The 16k-app dataset is a real asset, and comparing against DroidDeep is a fair baseline. The method description is coherent, and the idea of making detectors robust to label noise is worth pursuing.\n\nThe soft spots beyond the test-set ambiguity: no variance reported despite 30 repeats; no full-ensemble ablation, so you can't tell if selection helps or hurts; genetic algorithm and DBN hyperparameters are missing; only one noise level (10%) is tested; no artifacts. These are fixable. What worries me more is that the paper draws a strong 'superiority and robustness' conclusion from an evaluation protocol that is not pinned down.\n\nIf the intended protocol is training-only corruption, this is a modest but legitimate empirical contribution, and a referee could move it forward with a clarification and some ablation. As written, I wouldn't cite the robustness numbers. I'd still send it to peer review—the question is real and the flaw is fixable—but it needs major revision before the claims can be trusted.","headline":"The robustness claim in Table 2 is arithmetically incompatible with the stated 10% test-set label swap, so the paper's central evidence doesn't hold as written.","tokens_in":5267,"tokens_out":4072,"would_cite":false,"duration_ms":44761,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEdroid uses selective ensemble learning to keep Android malware detection at 98.3% precision even when 10% of training labels are flipped.","keywords":["Android malware detection","selective ensemble learning","genetic algorithm","deep belief network","label noise","robustness","majority voting","static analysis"],"falsifier":"Re-run the same 30-run comparison with label noise correlated to app families, or at noise levels of 5% and 20%, or with naturally noisy labels from multiple antivirus scanners; if SEdroid's average precision and recall fall to DroidDeep's level or below in any of these conditions, the robustness claim is falsified.","tokens_in":4379,"feed_emoji":"🛡️","tokens_out":4834,"duration_ms":47979,"temperature":0.7,"pith_summary":"This paper proposes SEdroid, an Android malware detector designed to stay accurate when training labels are noisy. Instead of training one deep network on possibly mislabeled data, SEdroid trains many networks on bootstrap samples and uses a genetic algorithm to choose a subset whose majority vote is both accurate and diverse. On a dataset of 16,000 apps where 10% of malicious/benign labels were deliberately swapped, SEdroid achieves an average precision of 98.3% and recall of 98.1%, compared to the single deep network baseline DroidDeep's 93.8% and 94.3%. The authors present this as evidence that selective ensembling makes Android malware detection more robust to imperfect real-world datasets.","feed_headline":"SEdroid keeps 98% accuracy even when 10% of training labels are wrong","feed_subtitle":"Selective ensembling with a genetic algorithm outperforms a single deep network on noisy Android malware data.","key_machinery":"The load-bearing mechanism is the selective ensemble: many component deep belief networks are trained on bootstrap samples, a 0/1 weight vector chooses which components vote, and the weights are evolved by a genetic algorithm whose fitness function is $F(x) = \\text{Accuracy} \\times D$, where $D$ is the average pairwise Euclidean distance between component predictions. The diversity factor prevents the ensemble from collapsing into identical networks, while the accuracy factor keeps the selected subset useful. The final detector is a majority vote over the selected components.","core_discovery":"SEdroid's central claim is that a selective ensemble of deep belief networks, with component selection driven by a genetic algorithm that maximizes accuracy times diversity, is more robust to training-label noise than a single deep network. The paper supports this by comparing average performance over 30 runs on a dataset with 10% of labels swapped: DroidDeep degrades to 93.8% precision and 94.3% recall, with its worst run at 89.7% and 90.3%, while SEdroid holds 98.3% and 98.1% on average. On the clean dataset, the selective ensemble also improves over single-component classifiers, which the authors attribute to the diversity term in the fitness function.","pith_inferences":["The same selective-ensemble recipe likely transfers to other malware-detection feature sets, such as dynamic API call sequences or graph embeddings, since nothing in the argument depends on the three specific static feature types.","The paper only tests one noise level; a natural extension would vary the swap rate to see whether SEdroid's advantage grows or shrinks as noise becomes more or less severe.","The diversity term is computed on final predictions, not on internal representations; testing diversity based on hidden-layer activations or gradients might yield even stronger ensembles under label noise.","If real-world mislabeling is correlated with app families or specific features, SEdroid's advantage could change; measuring it on naturally noisy multi-scanner labels would be a useful stress test."],"forward_implications":["Android malware detectors could be trained from automatically collected, uncurated app datasets without a full manual audit, since 10% label noise no longer degrades accuracy sharply.","The accuracy-times-diversity fitness gives a practical recipe for choosing ensemble members in other security classifiers where label noise is expected.","A selective ensemble can recover performance that a single deep network loses under noise, suggesting diversity is a usable robustness lever, not just an accuracy booster.","Because the genetic algorithm selects a subset of trained networks, robustness does not require paying the full ensemble's compute or memory cost at inference."],"supporting_citations":[{"why":"Supplies the selective ensemble principle that a subset of learners can outperform all, which is the core architecture of SEdroid.","marker":"[5]"},{"why":"Provides the genetic-algorithm-based search and the accuracy-plus-diversity fitness idea used to select component learners.","marker":"[8]"},{"why":"Defines DroidDeep, the single deep-network baseline whose degraded performance under label noise SEdroid is compared against.","marker":"[9]"},{"why":"Supports the claim that ensembles of neural networks improve generalization, motivating the ensemble approach.","marker":"[6, 7]"},{"why":"Introduces deep belief networks to Android malware detection, the component-learner architecture used in SEdroid.","marker":"[2]"}],"fun_headline_variants":["SEdroid: 98% accuracy with 10% wrong labels","Genetic algorithm makes malware detector noise-proof","Robust malware detection: selective ensemble wins","SEdroid shrugs off label noise, keeps 98% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The robustness claim rests on the assumption that randomly swapping 10% of the malicious and benign labels mimics real-world dataset noise; if real mislabeling is correlated with app families or features, or occurs at a different rate, SEdroid's reported advantage may not hold.","fun_headline_variants_meta":{"raw":{"variants":["SEdroid: 98% accuracy with 10% wrong labels","Genetic algorithm makes malware detector noise-proof","Robust malware detection: selective ensemble wins","SEdroid shrugs off label noise, keeps 98% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2532,"prompt_tokens":835,"completion_tokens":1697,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":1631}},"tokens_in":451,"tokens_out":1697,"duration_ms":12931,"temperature":1.0,"reasoning_tokens":1631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:39:24.030788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 30-run comparison with label noise correlated to app families, or at noise levels of 5% and 20%, or with naturally noisy labels from multiple antivirus scanners; if SEdroid's average precision and recall fall to DroidDeep's level or below in any of these conditions, the robustness claim is falsified.","supporting_citations":[{"cited_title":"Ensembling neural networks: Many could be better than all","cited_arxiv_id":null,"evidence_quote":"Supplies the selective ensemble principle that a subset of learners can outperform all, which is the core architecture of SEdroid."},{"cited_title":"Design and implementation of a genetic-based algorithm for data mining","cited_arxiv_id":null,"evidence_quote":"Provides the genetic-algorithm-based search and the accuracy-plus-diversity fitness idea used to select component learners."},{"cited_title":"Droiddeeplearner: Identifying android malware using deep learning","cited_arxiv_id":null,"evidence_quote":"Defines DroidDeep, the single deep-network baseline whose degraded performance under label noise SEdroid is compared against."},{"cited_title":"Deepflow: Deep learning- based malware detection by mining android application for abnormal usage of sensitive data","cited_arxiv_id":null,"evidence_quote":"Introduces deep belief networks to Android malware detection, the component-learner architecture used in SEdroid."}],"review_version":1}