{"id":"abb18277-bd5b-4e89-aa18-e6f8c2cc4b67","arxiv_id":"1908.04473","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A silhouette-clustering label flipping attack and two semi-supervised defenses (LSD, CSD) are proposed, with claimed accuracy gains over KSSD on Android malware datasets, but the CSD algorithm is not implementable as written.","lead":"This paper proposes a new label flipping attack based on silhouette clustering and two semi-supervised defenses to correct poisoned labels in Android malware detection. The defenses are tested on three datasets and claim up to 19% higher accuracy than a prior KNN-based defense.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CSD and LSD are not executable as written: shape mismatches and an undefined voting step make the claimed 19% accuracy advantage untestable.","rationale":"The reader's verdict of REJECT is well supported, but the single most load-bearing issue is not primarily the clean-validation assumption stated in the reader's weakest_assumption. The more fundamental problem is that the two proposed defenses, as specified in Algorithms 2 and 3, are not executable: they pair feature matrices and label vectors of incompatible sizes and invoke an undefined voting procedure. If an algorithm cannot run as published, then no experimental curve in Fig. 6 or Table 2 can be interpreted as evidence for the paper's central claim that LSD and CSD outperform KSSD and GANX. This is an internal-consistency failure rather than a disagreement with the field's consensus, so it directly undermines the empirical conclusion. The clean-validation issue is real and could be raised after the algorithms are corrected, but it is secondary to the fact that the defenses are not precisely defined. I therefore keep the reader's REJECT verdict without modification; the paper would need corrected algorithms, a defined voting rule, and re-run experiments with statistical support before the claim could be reconsidered.","tokens_in":25747,"tokens_out":6231,"duration_ms":61261,"concrete_test":"Take a small random split with n_train=120 and n_validation=40. Execute Algorithm 2 exactly as written: fit label spreading, label propagation, and the CNN with X_train paired with Y_validation, then run Algorithm 3 exactly as written: compute rand_score, mutual_info_score, homogeneity_score, and fowlkes_mallows_score on X_validation paired with YCorrected (length 120). Record the resulting exceptions, typically 'Found input variables with inconsistent numbers of samples.' Then run the authors' GitHub code from reference [31] on the same split; if it produces outputs without these errors, the public code must use different input/label pairings than the paper specifies, which would show that the reported experiments do not test the stated algorithms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that LSD and CSD are well-defined, runnable algorithms whose outputs can be compared with KSSD and GANX. That condition fails. In Algorithm 3, line 1 sets X to X_validation (with n_validation rows), while line 3 sets YCorrected to labels predicted for X_train (with n_train rows). Lines 4-7 then compute Rand Index, Mutual Information, Homogeneity, and Fowlkes-Mallows Index from X and YCorrected; all of these clustering metrics require the two inputs to have matching row counts. The loop in lines 8-16 repeats the mismatch: X_temp has n_validation+1 rows, but YCorrected still has n_train labels, so the comparison at line 11 is undefined. Algorithm 2 has a related problem: line 1 sets X to X_train, line 2 sets Y to Y_validation, and lines 3-10 fit label spreading, label propagation, and the CNN on these mismatched lengths. Moreover, line 11 calls an undefined Voting function with an uninitialized YCorrected and never uses Poisoned Y_train, which is the very input the defense is supposed to correct. Consequently, the accuracy and FPR numbers reported in Fig. 6 and Table 2 cannot be attributed to the algorithms as specified in the paper. Even before considering the clean-validation assumption that the reader flagged, the published defenses lack a well-defined implementation, so the empirical superiority claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a silhouette-clustering-based label flipping attack (SCLFA) against Android malware classifiers, and two defenses built on semi-supervised learning: LSD (label-based, using label propagation, label spreading, and a CNN with a voting step) and CSD (clustering-based, using four clustering metrics and validation data to decide whether each training sample is properly labeled). The authors report that LSD and CSD improve accuracy and lower false-positive rate relative to the KSSD and GANX baselines across three Android datasets (Drebin, Contagio, Genome) and three feature types (API, Intent, Permission), claiming up to 19% higher accuracy and about 15% lower FPR for CSD. The paper also includes computational-complexity comparisons. The central empirical claim is that the two proposed defenses are effective and superior to prior defenses, but the algorithms as specified are not executable, which calls the reported results into question.","tokens_in":26036,"tokens_out":4519,"duration_ms":43536,"significance":"The threat model (label flipping with a silhouette-based selection of samples to flip) is reasonable and the intended approach of using validation data to correct poisoned labels is a direction worth exploring. The paper also makes a concrete baseline comparison and provides runtime measurements. However, the claims of superiority are not assessable because the two central algorithms contain shape mismatches and an undefined voting function, so the reported accuracy and FPR numbers in Figures 4-6 and Tables 2-3 cannot be reproduced from the manuscript. The authors provide a GitHub link for source code, but no code is available in the manuscript and the link in reference [31] appears malformed. There are no machine-checked proofs or theoretical guarantees; the contribution is purely empirical, and the empirical evidence is undermined by the algorithmic inconsistencies.","major_comments":[{"comment":"The CSD algorithm is not executable as written. Line 1 sets X to X_validation (which has n_validation rows), while line 3 sets YCorrected to the labels predicted for X_train (which has n_train rows). Lines 4-7 then compute Rand Index, Mutual Information, Homogeneity, and Fowlkes-Mallows Index from X and YCorrected; all four metrics require the two input clusterings to have the same number of elements, but n_validation and n_train are generally different. The loop in lines 8-16 repeats the mismatch: X_temp has n_validation+1 rows, yet YCorrected still contains n_train entries, so the comparison at line 11 is undefined. Consequently, the accuracy and FPR values reported for CSD in Fig. 6 and Table 2 cannot be attributed to the algorithm as specified in the paper.","section":"Section 4.3.2, Algorithm 3"},{"comment":"The LSD algorithm is also not executable as specified. Line 1 sets X to X_train and line 2 sets Y to Y_validation, and lines 3-10 fit label spreading, label propagation, and a CNN on (X, Y); this is only meaningful when n_train equals n_validation, and even then it pairs training features with validation labels, which is semantically incorrect. Additionally, line 11 calls an undefined Voting function with an argument YCorrected that has never been assigned a value on any previous line, and the input Poisoned_Y_train is never used. Thus the voting step that is essential to the LSD method is not defined, and the empirical comparison for LSD is unsupported.","section":"Section 4.3.1, Algorithm 2"},{"comment":"The headline claim that CSD 'has approximately 19% higher Accuracy than the KSSD and has about 15% lower FPR compared to the KSSD' is not supported by any statistical analysis. Figures 4-6 and Tables 2-3 report point estimates without error bars, confidence intervals, or significance tests, and no number of independent runs is stated. Given the variability typical of malware-detection datasets, the comparison may not be robust, and the absence of variance reporting is a load-bearing gap for a paper whose contribution is empirical.","section":"Section 7, Conclusion"}],"minor_comments":[{"comment":"The threshold comparison S <= 0.1 in line 11 is a hand-chosen parameter with no sensitivity analysis; the manuscript should state how this threshold was selected and how the results depend on it.","section":"Section 4.3.2, Algorithm 3"},{"comment":"The description of the voting step in the text (Section 4.3.1) says the ensemble votes among 'LP output, Label Spreading, CNN predicted labels and poisoned labels,' but Algorithm 2 passes YCorrected (uninitialized) and none of the inputs explicitly as poisoned labels; the intended voting procedure should be formalized.","section":"Section 4.3.1, Algorithm 2"},{"comment":"The loop header 'for i = 1≤m' should read 'for i = 1 to m'.","section":"Algorithm 4, line 1"},{"comment":"The header of Table 2 is garbled, with repeated 'FNR' and 'AUC' labels; the column structure should be cleaned so that each WoFS/WFS pair is clearly labeled for each dataset.","section":"Table 2"},{"comment":"The x-axis labels DS1, DS2, DS3, -F, and -S are not defined in the caption; the caption says DS1=Drebin, DS2=Contagio, DS3=Genome, but this should be stated in the figure caption itself.","section":"Fig. 6"},{"comment":"The GitHub URL contains spaces ('mshojafar / sourcecodes') and is likely broken; please provide a working link and verify that the source code is publicly accessible.","section":"Reference [31]"},{"comment":"The phrase 'flip the labels by adding some perturbation of existing labels' is confusing; label flipping changes the class value, not a feature perturbation, and should be phrased accordingly.","section":"Section 3.2"},{"comment":"The statement 'we assign an interval [-1,1] for each sample' is imprecise; the silhouette value is computed, not assigned, and the definition should say that the silhouette value is used as a criterion for flipping.","section":"Section 4.2, Definition 1"}],"recommendation":"reject","confidential_remarks":"The paper has been accepted in what appears to be a journal (Neural Computing and Applications?), but the manuscript I reviewed contains two central algorithms that cannot be executed as written. This is a reproducibility-blocking issue, not merely an editorial one. A major revision could potentially fix the pseudocode, but the empirical claims would then need to be regenerated from corrected implementations, and the statistical analysis would need to be redone. Given the scope of the new experiments required, I recommend rejection in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper is not ready for serious refereeing. The defense algorithms LSD and CSD, as written in Algorithms 2 and 3, cannot be executed. In Algorithm 2, the label spreading and propagation models are fit on X_train with Y_validation—mismatched lengths—and the Voting function is called with uninitialized YCorrected; Poisoned Y_train, the thing the defense is supposed to correct, is never used. Algorithm 3 computes Rand Index, Mutual Information, Homogeneity, and Fowlkes-Mallows between an n_validation-by-d feature matrix and an n_train-length predicted label vector. The set sizes don't match, and those metrics compare label assignments, not features versus labels. The threshold 0.1 is arbitrary. This is not a minor implementation detail; it means the headline 19% accuracy improvement over KSSD, and the lower FPR claims, have no paper-level support.\n\nI want to give credit where it's due. The problem—defending Android malware classifiers against label flips—is real. The authors set up a sensible comparison across three datasets, three feature families, and with/without feature selection, against KSSD and GANX. The attack SCLFA, silhouette-guided flipping, is a reasonable extension of prior label-flipping attacks, even if not a big leap. The idea of using semi-supervised label propagation and spreading as part of an ensemble is not crazy. The paper also openly discusses some limitations, such as computational cost and static binary features. All of that is worth something.\n\nThe soft spots are load-bearing, though. No error bars anywhere; the numbers in Fig. 6 and Table 2 could be within noise. The validation set is assumed clean, but the paper never says what happens if the attacker can poison it. The GANX baseline is from the authors' own prior work, which is fine, but the description of Algorithm 5 is sketchy enough that I can't tell what was actually run.\n\nIf you want a negative teaching example of how not to present algorithms, this works. But as a scientific contribution, the empirical claims are untestable from the text, and no amount of light editing short of rewriting the algorithms and redoing the experiments would fix it. I'd desk-reject in its current form and invite a corrected resubmission if the authors can supply runnable code and clear pseudo-code. I would not cite it.","headline":"The defense algorithms as written cannot execute, so the claimed 19% accuracy gain over KSSD is untestable.","tokens_in":26532,"tokens_out":4168,"would_cite":false,"duration_ms":37905,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semi-supervised defenses LSD and CSD can correct labels flipped by a silhouette-based poisoning attack and beat KSSD, with CSD about 19% more accurate and 15% lower FPR.","keywords":["label flipping attack","data poisoning","semi-supervised defense","Android malware detection","silhouette clustering","convolutional neural network","label propagation","clustering metrics"],"falsifier":"Run SCLFA on any of the three Android datasets, then flip a small fraction of validation labels before applying CSD; if CSD's accuracy drops below KSSD's or below the no-defense baseline, the defense's clean-validation assumption is load-bearing. A direct reproducibility check is also possible: on Drebin API features with 300 random-forest-selected features, CSD should reproduce roughly the reported 19% accuracy gain and 15% FPR reduction over KSSD.","tokens_in":25562,"feed_emoji":"🛡️","tokens_out":6787,"duration_ms":62352,"temperature":0.7,"pith_summary":"This paper tries to establish that two semi-supervised defense algorithms, LSD and CSD, can correct labels flipped by a new poisoning attack built from silhouette clustering, and that they do this more accurately than the existing KNN-based KSSD defense and a GAN-based defense. The target is Android malware detection in IoT settings, with binary features for API calls, permissions, and intents. A sympathetic reader should care because label-flipping attacks need little attacker capability and can quietly destroy a malware classifier, and the paper offers a practical correction mechanism that needs only a small validation set. Its headline quantitative claim is that CSD reaches roughly 19% higher accuracy than KSSD and about 15% lower false-positive rate.","feed_headline":"A clustering defense beats label-flipping attacks by 19% accuracy","feed_subtitle":"Semi-supervised label correction restores Android malware detection after silhouette-based poisoning.","key_machinery":"The load-bearing objects are the SCLFA attack and the two defenses. SCLFA runs K-means with two clusters on the training data, computes each sample's silhouette value, and flips labels wherever the silhouette value is at most zero, meaning the sample looks as though it belongs in the other cluster. LSD is an ensemble: validation data trains label propagation, label spreading, and a three-layer Conv-1D CNN, and each flipped training label is replaced by the majority vote of those three predictions and the original poisoned label. CSD trains the CNN on validation data, computes the four clustering metrics on the validation partition, then adds each training sample one by one and accepts its CNN-predicted label only if the sum of absolute metric changes stays within 0.1. The CNN itself is a fixed architecture with three Conv-1D layers of 16, 32, and 64 filters, kernel and stride of 2, max-pooling, and a dense sigmoid output.","core_discovery":"The paper's central claim is that a poisoning attack can be built by clustering training samples and flipping the labels of those with non-positive silhouette values, and that two CNN-based semi-supervised defenses can undo most of the damage. LSD trains label propagation, label spreading, and the CNN on a clean validation set and re-labels each training sample by majority vote among these three predictions and the poisoned label. CSD instead uses four clustering-quality metrics (Rand index, mutual information, homogeneity, Fowlkes-Mallows index) computed on validation data as a baseline, and keeps each training sample's label only if adding that sample to the validation set changes the summed metrics by at most 0.1. The paper reports that across the Drebin, Contagio, and Genome datasets and API, intent, and permission feature sets, both defenses outperform KSSD and GANX, with CSD the strongest and roughly 19% more accurate than KSSD while lowering FPR by about 15%.","pith_inferences":["The paper treats the validation set as safe; the natural next attack is to poison validation labels, which would likely break both defenses and force a defense that does not trust any fixed reference partition.","The CSD acceptance threshold of 0.1 is a hand-set constant; varying it should produce a trade-off between correcting too many good labels and leaving flipped labels in place, and a data-dependent threshold might do better.","Because the features are binary and sparse, the clustering-metric shortcut is cheap; for dense or continuous features the same four metrics may not scale, so the defense's practical reach depends on feature representation.","SCLFA flips only samples that silhouette analysis already regards as mis-clustered; an attacker who flips high-confidence, correctly clustered samples may stress the defenses differently and would be a stronger test."],"forward_implications":["A defender facing a silhouette-based label-flip attack can recover most of the classifier's accuracy without discarding the poisoned training set.","CSD offers a practical accuracy-versus-speed trade: it is slower than KSSD but much more accurate, and clearly faster than LSD while maintaining the higher accuracy.","Random-forest feature selection down to 300 features preserves or improves the defenses' advantage while cutting runtime sharply in many configurations.","The defenses hold across all three Android datasets and three feature types, so the relabeling mechanism is not tied to a single data distribution."],"supporting_citations":[{"why":"Supplies the KNN-based KSSD defense that LSD and CSD are compared against and must outperform.","marker":"[26]"},{"why":"Supplies the GAN-based defense GANX used as the second comparison baseline.","marker":"[30]"},{"why":"Provides the Drebin dataset used in the evaluation.","marker":"[4]"},{"why":"Provides the Genome Android malware dataset.","marker":"[18]"},{"why":"Provides the Contagio dataset.","marker":"[1]"},{"why":"Defines the Rand index used in CSD's clustering-metric voting.","marker":"[27]"},{"why":"Defines the Fowlkes-Mallows index used in CSD.","marker":"[14]"},{"why":"Defines the homogeneity metric used in CSD.","marker":"[16]"},{"why":"Defines mutual information used in CSD.","marker":"[3]"}],"fun_headline_variants":["New defenses fix label-flipping attacks on Android malware","Clustering attack on malware labels foiled by CNN defenses","Malware label-flipping countered with 19% accuracy gain","Silhouette-based label attack meets its match in CSD","Semi-supervised defenses restore Android malware detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The defenses assume the validation data has clean, trustworthy labels and is representative enough to train the propagators and the CNN and to serve as a clustering baseline; if an attacker can flip labels in the validation set, the relabeling and the 0.1 threshold comparisons are corrupted.","fun_headline_variants_meta":{"raw":{"variants":["New defenses fix label-flipping attacks on Android malware","Clustering attack on malware labels foiled by CNN defenses","Malware label-flipping countered with 19% accuracy gain","Silhouette-based label attack meets its match in CSD","Semi-supervised defenses restore Android malware detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000482,"raw_usage":{"total_tokens":2415,"prompt_tokens":1011,"completion_tokens":1404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1338}},"tokens_in":627,"tokens_out":1404,"duration_ms":11449,"temperature":1.0,"reasoning_tokens":1338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:41:10.881350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SCLFA on any of the three Android datasets, then flip a small fraction of validation labels before applying CSD; if CSD's accuracy drops below KSSD's or below the no-defense baseline, the defense's clean-validation assumption is load-bearing. A direct reproducibility check is also possible: on Drebin API features with 300 random-forest-selected features, CSD should reproduce roughly the reported 19% accuracy gain and 15% FPR reduction over KSSD.","supporting_citations":[{"cited_title":"In: Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the KNN-based KSSD defense that LSD and CSD are compared against and must outperform."},{"cited_title":"Can Machine Learning Model with Static Features be Fooled: an Adversarial Machine Learning Approach","cited_arxiv_id":"1904.09433","evidence_quote":"Supplies the GAN-based defense GANX used as the second comparison baseline."},{"cited_title":"In: Ndss, vol","cited_arxiv_id":null,"evidence_quote":"Provides the Drebin dataset used in the evaluation."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides the Genome Android malware dataset."},{"cited_title":"http://contagiominidump.blogspot","cited_arxiv_id":null,"evidence_quote":"Provides the Contagio dataset."},{"cited_title":"Journal of the American Statistical as- sociation 66(336), 846–850 (1971)","cited_arxiv_id":null,"evidence_quote":"Defines the Rand index used in CSD's clustering-metric voting."},{"cited_title":"In: 2019 13th European Conference on Antennas and Propagation (EuCAP), pp","cited_arxiv_id":null,"evidence_quote":"Defines the Fowlkes-Mallows index used in CSD."},{"cited_title":"IEEE Transactions on Image Processing 14(3), 360–369 (2005)","cited_arxiv_id":null,"evidence_quote":"Defines the homogeneity metric used in CSD."},{"cited_title":"https://nlp.stanford.edu/IR-book/ html/htmledition/mutual- information- 1.html (2020)","cited_arxiv_id":null,"evidence_quote":"Defines mutual information used in CSD."}],"review_version":1}