{"id":"4b910634-3cf4-4f49-9961-df516203cbcc","arxiv_id":"2501.10956","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Stacking header and section models gives a small accuracy gain over single models, but the gain lacks statistical validation and the dataset is small and imbalanced.","lead":"This paper tests whether malware classifiers improve when separate models are trained on the header and body sections of Windows PE files and then combined. The best combined model reaches 99.30% accuracy versus 98.35% for the best single model, on a small five-family dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed multimodal advantage rests on ~4 test-set samples from one fixed 80/20 split, with no confidence interval, significance test, or multiple-comparison control; the abstract's conclusion is not supported at the reported precision.","rationale":"The reader's conditional verdict is sensible, and their weakest assumption names both stacking leakage and the absence of significance testing. I agree with both but would reorder them: the lack of statistical grounding is the load-bearing issue, because the entire abstract claim reduces to a four-sample difference. Leakage in the stacking step could explain part of the gain, but even if the probabilities are out-of-fold, a gap of 0.0095 on 423 samples is within normal split-to-split variation. The best-of-nine selection makes this worse: reporting the maximum of nine correlated accuracy estimates inflates the expected gap under the null. The class imbalance reinforces the point: 93% of the data are two families, so accuracy is not a sensitive metric for the minority families, and a four-sample difference can be a single-family artifact. The paper's own Section 5 limitation statement concedes that the dataset is too small or not diverse enough to quantify effectiveness. A concrete re-analysis with bootstrap or McNemar testing and out-of-fold stacking would settle the matter; absent that, the central claim should remain conditional. I therefore keep the reader's verdict unchanged. No ad hominem is intended; the flaw is in the evidence, not the authors.","tokens_in":13755,"tokens_out":6522,"duration_ms":73708,"concrete_test":"Obtain the 423 test-set predictions and labels from the current pipeline (or rerun the described pipeline), then: (1) compute the best-baseline versus best-of-nine-multimodal accuracy difference and run a paired bootstrap 95% confidence interval along with McNemar's exact test on discordant pairs; and (2) rerun the Section 4.5 meta-SVM using out-of-fold probabilities generated inside each training split, repeating the 80/20 split 100 times. If the confidence interval includes zero, p is at or above 0.05, or the median out-of-fold gain falls below roughly 0.005, the claim that training separate models on distinct PE parts is advantageous is not established at the paper's reported precision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that the best multimodal model (accuracy 0.9930) outperforms the best baseline (SVM on sections, 0.9835). With a 20% test split of 2,114 samples, this 0.0095 gap is about four test samples. The paper reports a single split, no repeated runs, no confidence interval, and no paired significance test. It also selects the best of nine multimodal combinations without any multiple-comparison adjustment, so part of the gap can reflect winner's curse. On this dataset, two families (Zbot, 1,238; ZeroAccess, 734) account for 93% of samples; overall accuracy therefore mostly reflects performance on two classes, and a four-sample shift could be driven by one family. Section 5 itself acknowledges that a larger, more diverse dataset is needed 'to more accurately quantify the effectiveness' of the approach. The Section 4.5 stacking description also never states that the probability features for the meta-SVM are out-of-fold; if they are training-set probabilities, the meta-feature distribution differs between train and test and can bias the comparison. Both issues attach directly to the central claim, but the statistical fragility alone is decisive: even a perfectly leak-free rerun could easily produce a gap of this size by chance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies SVM, LSTM, and CNN models to features extracted from PE headers, PE sections, and entire PE files, and then trains an SVM on the concatenated output probabilities of one header-based and one section-based model for each of nine combinations. On a 2,114-sample Malicia subset with five malware families, it reports that the best multimodal models, (LSTM, CNN)→SVM and (CNN, CNN)→SVM, reach accuracies of 0.9929 and 0.9930, respectively, versus 0.9835 for the best baseline (SVM on sections). From this the abstract concludes that it can be advantageous to train separate models on distinct parts of Windows PE files.","tokens_in":14015,"tokens_out":2548,"duration_ms":31559,"significance":"If the reported comparison were statistically robust, this would be a modest but useful empirical data point: probability stacking of simple models yields about a one-percent accuracy gain, halving the error rate, on a public dataset. The paper is clearly written, the experimental pipeline is transparent, and the use of an external dataset (Malicia) is a strength. However, the headline gain corresponds to roughly four test samples from a single 80-20 split, with no confidence intervals, repeated seeds, or significance tests, and the stacking protocol in Section 4.5 does not state whether out-of-fold probabilities were used. As presented, the evidence is too fragile to support the abstract's conclusion.","major_comments":[{"comment":"The central comparative claim rests on a single 80-20 split with no repeated runs, confidence intervals, or significance tests. The reported gap between the best multimodal accuracy (0.9930) and the best baseline (0.9835) is about four test samples out of roughly 423, and with Zbot and ZeroAccess accounting for 93% of the dataset (Figure 7), the difference could be driven by one family. The authors should provide repeated stratified cross-validation or bootstrap confidence intervals, and a paired significance test such as McNemar's test for the comparison of interest, with multiple-comparison control over the nine multimodal combinations.","section":"Section 4.5 and Figure 11"},{"comment":"The manuscript does not state whether the probability vectors used as features for the final SVM were produced out-of-fold or on the same training data used to fit the component models. If the latter, the meta-SVM is trained on optimistically biased features, and part of the reported 1% gain could be an artifact. The authors must specify the protocol and, if it is not out-of-fold, rerun the experiments with out-of-fold or nested cross-validation.","section":"Section 4.5, stacking protocol"},{"comment":"The conclusion acknowledges that testing on a larger, more diverse, and more challenging dataset is needed 'to more accurately quantify the effectiveness' of the approach. This self-identified limitation directly undercuts the abstract's categorical claim that the results 'indicating that it can be advantageous' to train separate models on distinct PE parts. The conclusion should be rephrased to match the strength of the current evidence, and the paper should state the uncertainty associated with the reported accuracies.","section":"Section 5 (Conclusion and Future Work)"},{"comment":"The SVM hyperparameter grid search is described as being performed 'using the header features' (Table 2), but the text does not state whether the selected hyperparameters were then used for all SVM variants (sections, entire file, and the meta-SVM) and whether the grid search was conducted entirely within the training partition. This should be clarified, as tuning on the full dataset or on the test portion would bias the reported accuracies.","section":"Section 4.2, hyperparameter selection"}],"minor_comments":[{"comment":"There is a duplicated word and an unclosed parenthesis in the sentence beginning 'While copious previous research has trained trained a wide variety learning models...'; the sentence should be rewritten for clarity.","section":"Section 1, Introduction"},{"comment":"In the description of the .reloc section, 'session' should be 'section'; the sentence reads 'The .reloc session contains the relocation table...'.","section":"Section 2.1"},{"comment":"Minor typographical and grammatical issues include 'introduces the machine learning that are used' and 'these results demonstrates the efficacy'; these should be corrected.","section":"Section 2.2 and Section 4.4"},{"comment":"The bar charts would be more informative if they included error bars or the number of test samples per family; given the class imbalance, this would help readers judge the reliability of the accuracy differences.","section":"Figure 11 and Appendix Figure A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical comparison that, with proper statistical support, could be a publishable modest contribution. The load-bearing issues are the lack of repeated-resampling evidence and the unspecified stacking protocol; both are fixable within the manuscript's scope. I would not reject on novelty grounds, but the abstract's conclusion should be softened unless the authors provide out-of-fold stacking and a paired significance test."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honestly, this is a textbook stacking experiment wrapped in a PE-file-specific framing. The new piece is splitting the PE file into header and sections and feeding probabilities from separate header/section models into an SVM. That idea is sensible, and the paper is clearly written: dataset, features, architectures, and hyperparameters are all specified well enough to reproduce, and the authors show per-family counts and acknowledge the imbalance. The related work is fair and contextual.\n\nThe problem is the central claim. The best multimodal result is 0.9930 versus 0.9835 for the best baseline SVM. On a test set of roughly 423 samples, that gap is about four samples. With a single 80/20 split, no confidence intervals, no repeated runs, no significance tests, and nine multimodal combinations without multiple-comparison control, the difference is well within noise. The dataset is also dominated by two families (Zbot and ZeroAccess make up 93% of the 2,114 samples), so a four-sample shift could easily be driven by one family. The stacking description in Section 4.5 never says whether the probability features for the meta-SVM are out-of-fold; if they are training-set probabilities, the comparison is optimistically biased. The authors themselves note in Section 5 that a larger dataset is needed to more accurately quantify effectiveness, which is exactly the issue. These are not manufactured flaws; they attach directly to the abstract's conclusion.\n\nWhat the paper does well is present the experiment cleanly and avoid overclaiming in the body text. The method is not novel enough to offset the statistical fragility, and the evidence does not justify the headline improvement. I would not cite this as a demonstration that multimodal PE classification works.\n\nFor a reader, this could serve as a compact example of why single-split accuracy on imbalanced malware data is brittle. For peer review, I would desk reject: the empirical claim is too weak and the technique is textbook. With out-of-fold stacking, confidence intervals, significance tests, and ideally a modern, larger dataset, the authors could turn the idea into a credible workshop paper. But this version does not support the abstract.","headline":"A clean but statistically fragile stacking experiment; the claimed multimodal gain is about four test samples from one split, and the abstract overstates it.","tokens_in":14538,"tokens_out":2957,"would_cite":false,"duration_ms":32669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By training separate models on PE headers and PE sections and feeding their probability outputs to a final SVM, the paper reports malware classification accuracy of 0.9930, up from 0.9835 for the best single-model baseline.","keywords":["malware classification","multimodal machine learning","Windows Portable Executable","PE header and section features","probability vector stacking","support vector machine","LSTM","CNN"],"falsifier":"Compute the final SVM's accuracy using out-of-fold probabilities: generate header- and section-model probabilities on held-out folds, train the stacking SVM only on those held-out predictions, and compare against the reported 0.9930. If the gap over the 0.9835 baseline disappears or reverses, the multimodal gain is an artifact of training the stack on probabilities from the component models' own training data.","tokens_in":13544,"feed_emoji":"🦠","tokens_out":10251,"duration_ms":100404,"temperature":0.7,"pith_summary":"This paper tries to establish that the two structural parts of a Windows PE file—the header and the remaining sections—carry complementary information for malware classification, and that exploiting that structure through a multimodal model yields a genuine accuracy gain. The authors train support vector machine (SVM), long short-term memory (LSTM), and convolutional neural network (CNN) baselines on header features, on section features, and on whole-file features, then concatenate the output probabilities of each header/section pair and train a final SVM on those ten probabilities. Of the nine combinations, the two best multimodal models reach accuracies of 0.9929 and 0.9930, against 0.9835 for the best single-model baseline, which roughly halves the number of misclassified samples. The result matters because it suggests a low-cost improvement path: split a structured binary file by its natural parts, keep the existing classifiers, and let a meta-classifier combine their probability outputs.","feed_headline":"Stacking header and section models lifts malware accuracy to 0.9930","feed_subtitle":"The best multimodal classifier nearly halves the error of the best single model, rising from 0.9835 to 0.9930.","key_machinery":"The central mechanism is a two-stage stacking scheme. In the first stage, each component model outputs a five-class probability vector for a PE file; a header-based model and a section-based model are trained independently. In the second stage, the two probability vectors are concatenated into a ten-dimensional feature vector, and an SVM is trained on those vectors to produce the final classification. The PE file's natural division into headers and sections is what makes the two modalities distinct, and the probability vector is the carrier of information between the stages.","core_discovery":"On the paper's own terms, the central discovery is that a multimodal stacking setup—one model trained on PE-header bytes, one model trained on PE-section bytes, and a final SVM trained on the concatenated output-layer probability vectors of the two—outperforms any single model trained on one part or on the entire file. The best results are (LSTM, CNN) → SVM at 0.9929 and (CNN, CNN) → SVM at 0.9930, versus 0.9835 for the best baseline, which is an SVM trained on section histograms. The paper reads this as evidence that training separate models on distinct parts of PE files can be advantageous: the probability vectors of the component models act as engineered features that preserve information lost when one model sees only part of the file, and the final SVM learns to combine them.","pith_inferences":["A natural extension the paper leaves implicit is testing whether the stacking gain persists on a larger, more balanced PE dataset; with 2,114 samples across five heavily imbalanced families, the reported one-point gain is a point estimate from a single split.","The design implies that the gain should grow when the two component models' errors are complementary and shrink when the two modalities are redundant; this could be tested by measuring error correlation between header and section predictions.","The same probability-vector stacking recipe should transfer to other structured binary formats, such as ELF files, though the paper only demonstrates it on Windows PE files."],"forward_implications":["Training existing classifiers separately on PE headers and PE sections and combining their probabilities with an SVM can improve accuracy by roughly one percentage point over the best single-model baseline.","The benefit is not tied to a single architecture: the two top combinations, (LSTM, CNN) → SVM and (CNN, CNN) → SVM, both beat every baseline.","At these accuracy levels, moving from 0.9835 to 0.9930 means the best multimodal model misclassifies about half as many samples as the best baseline.","The method treats the component models as trainable feature extractors, so any classifier that emits probabilities can be plugged into the second-stage SVM."],"supporting_citations":[{"why":"Supplies the 11,371-sample Malicia dataset and family labels from which the five-family experimental set is drawn; without it the experiments have no data.","marker":"[28]"},{"why":"Introduces the static PE-feature approach (byte histograms, header metadata) that the paper's SVM and header/section feature sets build on.","marker":"[2]"},{"why":"Provides evidence that PE-header-based features alone can classify malware, justifying the header modality.","marker":"[34]"},{"why":"Establishes the end-to-end whole-PE-byte baseline against which the entire-file and section models are compared.","marker":"[32]"},{"why":"Shows CNN classification of PE binary fragments works, supporting the CNN section models.","marker":"[40]"}],"fun_headline_variants":["Stacked header+section models hit 0.9930 malware accuracy","Multimodal stacking on PE files reaches 0.9930 accuracy","From 0.9835 to 0.9930: stacking PE features improves malware detection","Combining PE header and section models lifts accuracy to 0.9930"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central result assumes the probability features fed to the final SVM come from samples that were not used to train the header and section models; if those probabilities were computed on the same training data, the reported accuracy gain could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Stacked header+section models hit 0.9930 malware accuracy","Multimodal stacking on PE files reaches 0.9930 accuracy","From 0.9835 to 0.9930: stacking PE features improves malware detection","Combining PE header and section models lifts accuracy to 0.9930"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2480,"prompt_tokens":890,"completion_tokens":1590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1505}},"tokens_in":506,"tokens_out":1590,"duration_ms":14329,"temperature":1.0,"reasoning_tokens":1505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:47:15.114266+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the final SVM's accuracy using out-of-fold probabilities: generate header- and section-model probabilities on held-out folds, train the stacking SVM only on those held-out predictions, and compare against the reported 0.9930. If the gap over the 0.9835 baseline disappears or reverses, the multimodal gain is an artifact of training the stack on probabilities from the component models' own training data.","supporting_citations":[{"cited_title":"Zubair Rafique, and Juan Caballero","cited_arxiv_id":null,"evidence_quote":"Supplies the 11,371-sample Malicia dataset and family labels from which the five-family experimental set is drawn; without it the experiments have no data."},{"cited_title":"A PE header-based method for malware detection using clustering and deep embedding tech- niques","cited_arxiv_id":null,"evidence_quote":"Provides evidence that PE-header-based features alone can classify malware, justifying the header modality."},{"cited_title":"Nicholas","cited_arxiv_id":null,"evidence_quote":"Establishes the end-to-end whole-PE-byte baseline against which the entire-file and section models are compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows CNN classification of PE binary fragments works, supporting the CNN section models."}],"review_version":1}