{"id":"8fcd647e-f8f0-47ef-8dce-cbd3cf418b84","arxiv_id":"2502.08679","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A genetic-algorithm feature augmentation method is added to ANN, CNN, and RNN malware classifiers, with reported accuracy gains on a 2020 dataset, but the claimed concept drift handling is not supported by temporal train/test evaluation.","lead":"This paper combines deep learning classifiers with genetic-algorithm-generated variations of API call n-grams and reports accuracy gains on an older malware dataset. The experiments do not separate training and test periods, so the claimed concept drift handling is not actually measured.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper compares Dataset 2 with and without GA-mutated features but never introduces temporal drift; the reported gains are equally explained by static feature augmentation, so the concept-drift claim is not established.","rationale":"I agree with the reader's weakest-assumption assessment. The paper's strongest claim in Section 6.2 is that using concept drift in the malware dataset improves testing accuracy and reduces loss, indicating an effective concept-drift malware classifier. For that claim to hold, the experimental manipulation must vary the temporal data-generating distribution, and it does not. Tables 11 and 12 share the same Dataset 2; the only treatment is the genetic-algorithm mutation of API n-gram features. Concept drift is formally defined in Section 2.1 as a change from P_t(X,y) to P_{t+1}(X,y) at successive time steps, but no time steps appear in the evaluation. The GA fitness function in Phase 5 is edit distance to a target string, optimizing similarity to fixed strings rather than responsiveness to distributional shift. Thus the independent variable is feature-set composition, not drift. Section 8's internal and conclusion validity statements explicitly acknowledge that the concept-drift representation and the evaluation metrics may not capture real-world drift effectiveness. A temporal train/test split plus a control for feature-count augmentation is the minimal design that would test the claim. Because the central claim is untested and the available comparison cannot distinguish drift handling from static feature augmentation, the rejection stands. My read does not change the reader's verdict, so I recommend 'UNCHANGED.'","tokens_in":13080,"tokens_out":4827,"duration_ms":47978,"concrete_test":"Run a strict temporal validation: split Dataset 2 by collection date (for example, train on samples seen before a date T and test on samples seen after T), with and without the GA-mutated features, and with all mutant generation restricted to the training partition. Include a control in which an equal number of randomly selected n-grams (not GA-mutated) are added as features under the same random-split setting. The concept-drift claim survives only if the GA variant beats both the no-GA baseline and the random-feature control in the temporal setting but not in a shuffled-split setting; otherwise the improvement is feature augmentation, not drift handling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim requires that the 10,500 GA-mutated API n-gram features simulate concept drift, so that the accuracy improvement in Table 12 over Table 11 demonstrates an adaptive classifier. This equivalence is untested. Tables 11 and 12 are both labeled Dataset 2 (Table 10), a single static collection dated January to May 2020; the paper reports no temporal split, no train-on-old/test-on-new evaluation, and no drift detector. The only stated difference between the two tables is the presence of the GA-generated feature set. Under the paper's own definition in Section 2.1, concept drift requires P_t(X,y) vs. P_{t+1}(X,y) at successive time steps, but no time steps are used in the evaluation. The GA procedure in Phase 5 changes API n-gram strings via edit-distance-based mutation; it produces alternative features, not a future malware distribution, and Section 8 concedes the approach 'may not fully capture the complexity and dynamics of concept drift in real-world malware datasets.' The observed gains in Table 12 (roughly 0.6 to 2.3 percentage points in accuracy, with CNN validation accuracy changing only slightly) are therefore equally consistent with static feature augmentation on a fixed distribution. No error bars, repeated runs, or significance tests are reported, so the paper has not established drift handling as the cause of the improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid deep-learning and genetic-algorithm framework for malware classification from API call n-grams, with the stated goal of handling concept drift. The method extracts unigrams, bigrams, and trigrams from Cuckoo Sandbox reports, applies term-frequency filtering, and then generates additional features by mutating existing API n-grams with a genetic algorithm. The GA selects 10,500 mutants by an edit-distance fitness score, and these are added to the feature set. The authors evaluate ANN, CNN, and RNN classifiers on two datasets collected from VirusShare (Dataset 1: January–June 2023; Dataset 2: January–May 2020), comparing results without and with the GA-generated features. They report that including the GA features on Dataset 2 improves accuracy and reduces loss, and they attribute this improvement to concept-drift handling. The paper also includes a comparison with two related works and a brief threats-to-validity section.","tokens_in":13347,"tokens_out":3013,"duration_ms":28978,"significance":"If the central claim were valid, a method that demonstrably improves malware classification under concept drift by augmenting API n-gram features would be of practical interest to the malware-detection community. The paper also makes a concrete attempt to combine evolutionary feature generation with deep classifiers, and it provides a detailed data-collection pipeline. However, the significance is severely limited by the experimental design: the comparison claimed to demonstrate concept-drift handling is conducted on a single static dataset with no temporal or distributional shift, so the reported gains are equally consistent with ordinary feature augmentation. The GA fitness function is underspecified, and the RNN results indicate a non-learning model, further weakening the empirical claims. The authors do provide a public code link and a dataset-availability statement, but no machine-checked proofs or reproducible experiments beyond what is described.","major_comments":[{"comment":"The central claim that \"utilising concept drift in the malware dataset can lead to improved testing accuracy and reduced loss\" is not supported by the experiments. Tables 11 and 12 both report results on Dataset 2 (Table 10), a single static collection from January to May 2020. The only reported difference between the two conditions is the presence of the GA-generated features; there is no temporal split, no train-on-old/test-on-new evaluation, and no distributional shift between the two settings. Under the paper's own definition in Section 2.1, concept drift is a change between successive time steps P_t(X,y) and P_{t+1}(X,y), but no time steps are used in the evaluation. The improvements in Table 12 over Table 11 are therefore equally explained by static feature augmentation on a fixed distribution, and the concept-drift claim is not established.","section":"Section 6.2, Tables 11 and 12"},{"comment":"The GA fitness function is underspecified in a way that directly affects the validity of the feature-selection step. Equation (1) defines Fitness(individual) as the number of mismatched characters against a \"target string,\" but the manuscript never defines what the target string is, how it is chosen per class, or how it relates to the malware families. The selection of the \"top 1500\" features per class based on this fitness score is therefore not reproducible and may introduce arbitrary class-specific bias. Moreover, the fitness score is computed on the same malware corpus that is later used for training and testing, so the feature-selection process is circular with respect to the reported accuracy gain.","section":"Section 5, Phase 5 and Equation (1)"},{"comment":"The RNN results are not interpretable as a meaningful comparison. Table 7 shows that the RNN's loss and accuracy remain nearly constant (loss around 1.7–1.9, accuracy around 0.39) across all epochs, which strongly suggests the model is not learning. The architecture in Table 3 uses an input shape of (88972, 1), treating all 88,972 features as a single time step, which is not a meaningful sequence representation for API call n-grams. The RNN rows in Tables 11 and 12 show slightly different but still near-constant values, so any \"improvement\" attributed to the GA for the RNN is not credible as evidence of concept-drift handling.","section":"Section 6.2, Tables 7 and 11/12"},{"comment":"No error bars, repeated runs, or statistical significance tests are reported. The accuracy differences between Table 11 and Table 12 are small (e.g., CNN training accuracy improves from 0.9250 to 0.9314 and validation accuracy from 0.9430 to 0.9459), and the ANN validation accuracy improves from 0.8946 to 0.9184 while its training loss slightly increases from 0.6250 to 0.6314. Without variance estimates or multiple trials, the claim of \"significant improvements\" is not justified, and the observed differences could be within run-to-run variability.","section":"Section 6.2, Tables 11 and 12"},{"comment":"The manuscript itself concedes that the concept-drift handling approach \"may not fully capture the complexity and dynamics of concept drift in real-world malware datasets\" (Section 8, External Validity). This admission directly undermines the abstract and Section 6.2's conclusion that the GA-augmented features demonstrate adaptability to concept drift. Since the evaluation never introduces a temporal or distributional shift, this limitation is not a secondary caveat but a statement that the central phenomenon the paper claims to address has not been measured.","section":"Section 8, Threats to Validity"}],"minor_comments":[{"comment":"Typographical errors include \"handeling\" (abstract), \"handelling\" (Section 2.3), and \"paramount are dynamic environments\" (Section 2.1). These should be corrected.","section":"Abstract and Section 2.3"},{"comment":"The manuscript refers to Figure 5 as \"Mutated Features\" but the figure appears after Figure 3 and the caption is uninformative; the figure numbering and captions should be revised for clarity.","section":"Section 5, Phase 5"},{"comment":"The row numbering in Table 7 is inconsistent: rows 5 and 6 are labeled \"3\" and \"4\" again after rows 3 and 4 have already used those numbers. The epoch values also skip from 60 to 80 but the labels repeat.","section":"Table 7"},{"comment":"Equation (1) defines the fitness as a sum over positions i of the target string, but it is not specified how the fitness is computed when the individual and the target string have different lengths, which is likely given that API n-gram strings vary in length. This should be clarified.","section":"Equation (1)"},{"comment":"The text states that \"two lakhs of a diverse set of malware samples\" were collected, but the reported datasets contain only about 22,000 samples each. The discrepancy between the claimed collection size and the actual dataset sizes should be explained.","section":"Section 6.1 and Table 5"},{"comment":"Table 13 compares the proposed work with only two prior studies, and the comparison is not quantitative because the datasets and evaluation protocols differ. The table would be more informative if it included a direct performance comparison or a clear statement that no direct comparison is possible.","section":"Section 7 and Table 13"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands: the paper's headline claim about concept drift handling is not measured by the experiments. Tables 11 and 12 differ only in feature augmentation on the same static dataset, so the improvement cannot be attributed to drift adaptation. The GA fitness function is under-specified, and the RNN results suggest a non-learning baseline, which further weakens the empirical support. In addition, the comparison with related work is superficial, and the threats-to-validity section concedes the main limitation without addressing it. This is a load-bearing flaw that cannot be fixed by minor revision; it would require a new experimental design with temporal evaluation, which is beyond the current manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe concept-drift claim doesn't survive contact with the experiment. Tables 11 and 12 both use Dataset 2, a fixed collection from Jan–May 2020; the only difference is the GA-mutated feature set. That makes the gains a feature-augmentation effect, not evidence of drift adaptation. The paper's own Section 8 concedes the mutation approach 'may not fully capture' real-world drift, which is closer to the truth.\n\nCredit where it's due: the pipeline is described clearly (Cuckoo sandbox -> API n-grams -> TF filtering -> ANN/CNN/RNN), the dataset is respectable (22k samples), and the authors honestly list validity threats. The GA mutation idea is a legitimate extension, though Table 13 already credits Fernando and Komninos [7] with the same combination of GA and API-call drift handling.\n\nThe soft spots are load-bearing. No temporal split, no train-on-old/test-on-new, no drift detector. The fitness function in Eq. (1) is an edit distance to a 'target string,' but the target is never defined. GA parameters (mutation rate, crossover, selection) are absent, as are error bars or repeated runs. The RNN's accuracy is pinned at 0.39 across epochs, which says the RNN isn't learning – yet its Table 12 'improvement' is still reported without comment. The observed accuracy gains (0.6–2.3 pp) are exactly what you'd expect from adding 10,500 extra features to a fixed corpus.\n\nWho's this for? Someone wanting a straightforward API n-gram baseline on 2020-era malware, not anyone looking for evidence on concept drift. If the authors rerun with a proper temporal split and fully specify the GA, there's a salvageable empirical study here. As it stands, I'd send it to review only with a strong directive on the drift evaluation; the current version isn't publishable. I'd rather see a revised version than a desk rejection, because the core idea – GA-generated features as a proxy for drift – is testable and the authors seem willing to engage.\n\nRecommendation: send to peer review with clear instructions, but expect major revision or likely rejection if the drift evidence doesn't materialize.","headline":"The concept-drift claim is unsupported because Tables 11 and 12 compare static feature sets on the same fixed dataset; the GA-augmented features show a feature-engineering gain, not drift adaptation.","tokens_in":13894,"tokens_out":3585,"would_cite":false,"duration_ms":34957,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding genetic-algorithm mutants of API n-gram features raises malware classification accuracy on data from a different time period.","keywords":["concept drift","malware classification","genetic algorithm","API call sequences","n-gram features","deep learning","CNN","feature mutation"],"falsifier":"Train the models on a dataset from one period (e.g., 2023) and test on a later period (e.g., 2024), with and without the 10,500 GA-mutated features. If the accuracy gain over the un-augmented model disappears on this genuinely future test set, the GA mutation is not handling concept drift; it is only helping on the particular older test set used in the paper.","tokens_in":12813,"feed_emoji":"🧬","tokens_out":7038,"duration_ms":59301,"temperature":0.7,"pith_summary":"This paper tries to show that a deep-learning malware classifier can handle concept drift without retraining on newly collected data, by using a genetic algorithm to create mutated API n-gram features and add them to the training corpus. On a malware dataset released in early 2020, adding 10,500 such mutants raised ANN accuracy from 91.1% to 93.4% and CNN accuracy from 92.5% to 93.1%, with lower validation loss, compared to the same networks trained without the mutants. The authors take this as evidence that the mutated features emulate drift in malware behavior, letting the model adapt to a distribution it never saw during training. If correct, the approach offers a lightweight way to refresh malware classifiers against evolving threats.","feed_headline":"Mutating API features boosts malware classification on drifted data","feed_subtitle":"Adding 10,500 genetic-algorithm mutants lifted ANN and CNN accuracy on a 2020 malware dataset.","key_machinery":"The carrying mechanism is the genetic algorithm operating on n-gram strings in Phase 5 of the framework. It mutates sub-parts of API sequence features, defines a fitness score as the number of positions in which the mutant differs from a target string (a Hamming/edit distance), and uses that score to select the most distant mutants as new features. These selected mutants are appended to the original corpus, so the deep networks train on both the original 2023-era features and synthetic variants, which the paper argues allows the classifier to recognize samples from an older, drifted distribution.","core_discovery":"The central discovery claim is that GA-generated mutations of API n-gram sequences constitute an effective concept-drift handling mechanism. Starting from n-grams extracted from sandbox API call reports, the algorithm mutates the secondary and tertiary parts of the sequence strings while leaving the primary API name untouched, scores each mutant by its edit distance to a target string, and selects the top 1,500 mutants per malware family (10,500 total) to augment the existing feature corpus. Retraining the same ANN, CNN, and RNN architectures with these added features on a 2020 malware dataset produced higher accuracy and lower loss than training without them, which the paper presents as evidence of adaptability to concept drift.","pith_inferences":["An implicit, testable extension would be to check whether the GA-generated mutant n-grams actually occur in malware released after the training period; the paper does not show that the synthetic strings match real drifted behavior.","The comparison between two fixed datasets does not by itself establish drift handling; stronger evidence would come from a time-ordered evaluation where the model trains on one period and is tested on a later period, with and without mutants.","The fitness function rewards distance from a target string, which may select for highly unusual strings rather than representative ones; an alternative would be to generate mutants from drift-detection signals such as a validation-window performance drop."],"forward_implications":["If the GA-mutated features truly emulate drift, then a model trained on current features plus mutants should generalize to past or future distributions without collecting new samples.","The method adds only about 1% more features (10,500 of 88,972), so drift handling can be achieved at a marginal feature-cost increase.","The edit-distance fitness selection offers a principled way to choose which mutants to retain, favoring strings most different from the originals.","The effect appeared in both ANN and CNN architectures, suggesting the augmentation may transfer to other deep classifiers or feature-based detectors."],"supporting_citations":[{"why":"Supplies the genetic-algorithm approach to mining concept-drifted data streams that the paper adapts to malware n-gram features.","marker":"[3]"},{"why":"Provides the edit-distance fitness score formula used to select the 10,500 mutant features.","marker":"[19]"},{"why":"Source of the malware samples and the two temporal datasets used in the experiments.","marker":"[17]"},{"why":"Defines concept drift and the taxonomy of drift types that frames the paper's motivation.","marker":"[2]"},{"why":"Establishes that malware detectors degrade over time due to data-space drift, motivating the need for drift handling.","marker":"[4]"},{"why":"Baseline concept-drift-adaptive ransomware detection framework compared in Section 7, the closest related work using API calls and genetic algorithms.","marker":"[7]"}],"fun_headline_variants":["Genetic algorithm mutations improve malware classification on drifted data","API sequence mutations via GA boost malware detection under concept drift","Deep learning with GA-mutated API features adapts to malware drift","Mutating API n-grams helps deep learning models handle concept drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that mutating strings of existing API n-gram features produces features that represent how malware behavior drifts over time; if that equivalence is false, the experiments measure feature augmentation on a fixed distribution rather than drift handling.","fun_headline_variants_meta":{"raw":{"variants":["Genetic algorithm mutations improve malware classification on drifted data","API sequence mutations via GA boost malware detection under concept drift","Deep learning with GA-mutated API features adapts to malware drift","Mutating API n-grams helps deep learning models handle concept drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000333,"raw_usage":{"total_tokens":1764,"prompt_tokens":773,"completion_tokens":991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":389,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":389,"tokens_out":991,"duration_ms":7403,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T05:58:08.679924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the models on a dataset from one period (e.g., 2023) and test on a later period (e.g., 2024), with and without the 10,500 GA-mutated features. If the accuracy gain over the un-augmented model disappears on this genuinely future test set, the GA mutation is not handling concept drift; it is only helping on the particular older test set used in the paper.","supporting_citations":[{"cited_title":"Mining data streams with concept drifts using genetic algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the genetic-algorithm approach to mining concept-drifted data streams that the paper adapts to malware n-gram features."},{"cited_title":"Genetic algorithms for credit scoring: Alternative fitness function performance comparison","cited_arxiv_id":null,"evidence_quote":"Provides the edit-distance fitness score formula used to select the 10,500 mutant features."},{"cited_title":"https://virusshare.com/","cited_arxiv_id":null,"evidence_quote":"Source of the malware samples and the two temporal datasets used in the experiments."},{"cited_title":"Learning under concept drift: A review","cited_arxiv_id":null,"evidence_quote":"Defines concept drift and the taxonomy of drift types that frames the paper's motivation."},{"cited_title":"Is it overkill? analyzing feature-space con- cept drift in malware detectors","cited_arxiv_id":null,"evidence_quote":"Establishes that malware detectors degrade over time due to data-space drift, motivating the need for drift handling."},{"cited_title":"Fesad ran- somware detection framework with machine learning using adap- tion to concept drift","cited_arxiv_id":null,"evidence_quote":"Baseline concept-drift-adaptive ransomware detection framework compared in Section 7, the closest related work using API calls and genetic algorithms."}],"review_version":1}