{"id":"1e0f43e5-3f6f-435d-95eb-29872b806450","arxiv_id":"2608.12077","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":13,"one_line_summary":"A Random Forest on embeddings from 16 MobileNetV2 models, including Grad-CAM overlays generated with true labels, reaches 0.777 accuracy on 17 malware families, but the overlay representation leaks the answer.","lead":"This paper tested eight ways of turning malware files into images, then used a computer vision technique called Grad-CAM to inspect where image-based malware classifiers look. It reports that explanation quality and classification accuracy usually do not match, and claims a small accuracy improvement on a known benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grad-CAM overlays are generated using the true family label as the target class (Section 3.2), leaking the ground-truth label into the features; the reported 0.777 Random Forest accuracy is therefore not a valid deployment accuracy.","rationale":"In good faith, the paper's central contribution is the benchmark claim that Grad-CAM-derived features can exceed prior malware classification accuracy, and the secondary claim that accuracy and explanation faithfulness diverge. For the benchmark claim to hold, the feature-generation protocol must be usable at inference time: given an unseen binary, one must be able to produce the same features without knowing its family. Section 3.2 breaks this condition: 'The true family label is used as the target class c.' Consequently, the overlay images used as inputs to the overlay CNNs, the HOG-hybrid, and the final Random Forest contain the ground-truth label through the gradient computation. At test time, the true label is used to construct the test input, so the classifier is effectively given the answer. This is not a subtle hyperparameter choice; it is a direct circularity. The reader's weakest_assumption identifies exactly this, and I agree. The paper even flags the protocol explicitly, so it is not a hidden artifact. The missing split in Section 4.6 is a second, independent concern: the text reports 'test accuracy' for the 4096-dimensional Random Forest but never states how the 16,997 samples were divided into training and test sets. If the RF was fitted and evaluated on embeddings extracted from the same samples, the number is further inflated. This is flagged as a limitation or an omitted detail; the manuscript does not address it. The XAI faithfulness/stability analysis (Tables 5 and 6) is less affected by the leakage because it evaluates explanation plausibility rather than deployable classification, but the headline comparison of transformation accuracies is contaminated. I am not arguing that the authors intended deception; the issue is in the experimental protocol. The concrete test—regenerating overlays with predicted classes and reporting a clean split—would settle the matter. If the 0.777 result requires true labels at inference, it cannot stand as a benchmark for malware classification.","tokens_in":20002,"tokens_out":5471,"duration_ms":43487,"concrete_test":"Regenerate all Grad-CAM overlay images for the test partition using the predicted class from the corresponding CNN (rather than the true label), re-extract the 256-dimensional embeddings, concatenate the 4096-dimensional feature vector, and retrain the Random Forest under a clearly stated stratified split. If the accuracy drops substantially below 0.777—or below the 0.751 prior benchmark—the headline result is an artifact of label leakage. As a negative control, also generate overlays with a randomly chosen target class and show that accuracy falls to near chance for those features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Abstract, Table 9) is a Random Forest test accuracy of 0.777 on 4096-dimensional features that concatenate CNN embeddings from original and Grad-CAM overlay images (Section 4.6). Section 3.2 states: 'The true family label is used as the target class c, so that the explanation corresponds to the ground-truth class.' Thus every overlay image used to train the overlay CNNs (Table 3), the CNN-HOG-XGBoost hybrid (Table 4), and the 256/512/4096-dim embeddings is constructed with the ground-truth family label. At inference time the test overlay is also generated with the test sample's true label, so the input features carry the label information the classifier is supposed to predict. This is label leakage: the reported accuracy does not describe a usable malware classifier, since a deployed system would not know the family in advance. The second claim, that accuracy and explanation faithfulness do not coincide (Sections 4.5 and 5), inherits the same leakage in its accuracy component. A separate concern is that Section 4.6 does not state the train/test split used for the 4096-dimensional Random Forest; without a clean split, the accuracy could be further inflated. Both issues must be resolved before the benchmark claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares eight binary-to-image transformations for malware family classification on a 17-class, 17,000-sample dataset. It trains MobileNetV2 CNNs on original and Grad-CAM overlay images, extracts handcrafted features from Grad-CAM heatmaps, builds hybrid CNN-HOG-XGBoost pipelines, and concatenates CNN embeddings into 256-, 512-, and 4096-dimensional feature vectors. The headline result is a 0.777 test accuracy for a Random Forest on the 4096-dimensional embeddings, which the authors claim exceeds a prior benchmark of 0.751. The paper also evaluates faithfulness and stability of Grad-CAM explanations, compares Grad-CAM with HiResCAM, and concludes that accuracy and explanation faithfulness do not coincide across the eight transformations.","tokens_in":20403,"tokens_out":5311,"duration_ms":48458,"significance":"If valid, the paper would offer a useful systematic comparison of image transformations for malware classification and an empirical study of Grad-CAM behavior in this domain. The faithfulness/stability analysis and the HiResCAM comparison are of interest, and the authors disclose practical details such as fixed seeds and three excluded degenerate heatmaps. However, the central classification benchmark is invalid because the Grad-CAM overlays are generated using the true family label as the target class, so the input features encode the label the classifier is supposed to predict. This undermines the headline accuracy claims and the accuracy component of the accuracy-versus-faithfulness finding.","major_comments":[{"comment":"Section 3.2 states: 'The true family label is used as the target class c, so that the explanation corresponds to the ground-truth class.' The resulting overlay images are then used as inputs to the overlay CNNs (Table 3), the CNN-HOG-XGBoost hybrid (Table 4), and the 256/512/4096-dimensional embeddings (Tables 8 and 9). Because the test overlays are also generated using each test sample's true family label, the input features carry the label information the classifier is supposed to predict. The reported test accuracies, including the 0.777 in the Abstract and Table 9, therefore do not describe a deployable classifier; they measure label-conditioned representations. This also invalidates the accuracy component of the 'accuracy and explanation faithfulness do not coincide' conclusion in Sections 4.5 and 5.","section":"3.2, 4.3-4.6, Tables 3, 4, 8, 9"},{"comment":"The 4096-dimensional experiment does not specify the train/test split used for the final Random Forest. The text reports only that samples present across all 16 models are included (16,997 samples) and that a Random Forest with 500 trees is trained. Since Section 3.1 constructs per-transformation stratified splits separately, it is unclear how these splits are reconciled when samples are pooled across all transformations. Without a clean single split, the 0.777 accuracy could be further inflated even beyond the leakage problem identified in the previous comment.","section":"4.6, Table 9"}],"minor_comments":[{"comment":"The Abstract reports a previous benchmark of 0.750, while Sections 4.6 and Table 11 report 0.751 from Agrawal et al. [2]; the abstract should be corrected to 0.751 or the discrepancy should be explained.","section":"Abstract"},{"comment":"In the first paragraph of Section 6, 'There are several promising directions for extending the work considered in this chapter, We showed that combining HOG descriptors...' should read 'chapter. We showed' with a period instead of a comma.","section":"Section 6"},{"comment":"Overlay-image CNN accuracies appear with different values for the same transformations across Tables 3, 4, and 7 (e.g., Grayscale 0.566, 0.565, and 0.602). The paper should clarify which of these runs correspond to the same or different training conditions, since the current presentation makes the discrepancies difficult to interpret.","section":"Tables 3, 4, and 7"}],"recommendation":"reject","confidential_remarks":"The faithfulness/stability evaluation and the HiResCAM comparison are potentially salvageable, but the headline benchmark claim is invalid because the Grad-CAM overlays used as inputs are generated with the true family label. Fixing this requires re-running the overlay-based experiments with a non-leaky protocol, which would likely change all of the central results. The current manuscript cannot be accepted as a valid benchmark contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the headline number does not survive contact with the paper's own protocol. The 0.777 Random Forest result is built from Grad-CAM overlays generated using the true family label as the target class — stated plainly in Section 3.2, and true at test time as well. Those features encode the answer, so Tables 3, 4, 8, and 9 describe a label-conditioned representation, not a deployable classifier. The same leak runs through the 256-dim and 512-dim fusion results and the classical-ML-on-CAM-features experiment in Table 2. The reader's rejection is fair on this point.\n\nWhat is worth keeping is the XAI comparison. The systematic faithfulness (deletion/insertion AUC, concentration, drop-top) and stability (Spearman, SSIM, top-K overlap) evaluation across eight transformations is the first of its kind for malware imagery as far as I know, and the finding that accuracy and faithfulness do not coincide is genuinely interesting — Entropy Hilbert as the balanced default is plausible. The HiResCAM result is a clean, well-explained negative: at MobileNetV2's 7x7 final feature map the two methods are numerically nearly identical. The reporting is honest about seeds, splits, and retraining variation, which I appreciate.\n\nSoft spots beyond the leak. First, Section 4.6 never states the split used for the 4096-dim Random Forest; Table 10 suggests 150-sample test blocks per family, but the protocol should be explicit. Second, the faithfulness evaluation also targets the true class, which is fine for correct predictions but distorts the metric on misclassified samples; since error rates differ across transformations, the rankings are not fully like-for-like. Using the predicted class, or restricting to correctly classified samples, would tighten this. Third, small inconsistencies: Bigram Polar is \"ranked seventh\" on CNN accuracy but sits second in Table 1; the abstract says 0.750 for the prior benchmark while the body says 0.751; and the \"same\" overlay CNN experiment gives 0.604 in Table 3 but 0.624 in Table 7. Minor, but they add to the impression that the experimental narrative needs a careful pass.\n\nWho is this for? Someone working on XAI evaluation for image-based malware analysis will get value from the faithfulness/stability apparatus and the HiResCAM observation — but they should not cite the 0.777 number. The fix is clear: regenerate overlays without test labels (predicted-class CAMs, or drop the overlay-as-input classification claims entirely) and report the fusion split.\n\nI'd send it to referees — it deserves a serious one — but I'd expect major revision at best, and the classification claim as stated should be withdrawn or re-run.","headline":"The headline 0.777 accuracy is invalid: the Grad-CAM overlays are generated with the true family label as target, so the features leak the answer — but the faithfulness/stability comparison across transformations is a genuinely useful contribution that deserves a serious referee.","tokens_in":20837,"tokens_out":7152,"would_cite":false,"duration_ms":59203,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining CNN embeddings from original malware images and their Grad-CAM overlays achieves 0.777 test accuracy across 17 malware families, beating the previous 0.751 benchmark, and that accuracy and explanation…","keywords":["image-based malware classification","Grad-CAM","explainable AI","MobileNetV2","Random Forest","malware family classification","faithfulness metrics","binary-to-image transformations"],"falsifier":"Regenerate the overlay dataset with the model's predicted class as the Grad-CAM target, retrain the 4096-dimensional Random Forest, and compare test accuracy to 0.777; a substantial drop would show the headline result depends on label leakage into feature generation.","tokens_in":19815,"feed_emoji":"🛡️","tokens_out":6731,"duration_ms":58560,"temperature":0.7,"pith_summary":"This paper tries to establish two things: that Grad-CAM heatmaps of malware images carry enough family-discriminative structure to be used as classification features, and that the choice of binary-to-image transformation affects accuracy and explanation quality in different ways. On the first point, it reports that a Random Forest trained on a 4096-dimensional vector of MobileNetV2 embeddings, built from original images and Grad-CAM overlays across eight transformations, reaches 0.777 test accuracy on 17 malware families, above the previous best of 0.751 on the same dataset. On the second point, it reports that the most faithful explanations come from Bigram Polar, a mid-tier accuracy transformation, while the most accurate transformation, Entropy Hilbert, is also second-most faithful and therefore the best single compromise. A sympathetic reader would care because the result suggests explanations are not just a diagnostic add-on but a usable signal for classification, and because the accuracy/faithfulness split warns against choosing an image representation by accuracy alone.","feed_headline":"Fusing Grad-CAM overlays pushes malware family accuracy to 0.777","feed_subtitle":"A Random Forest over combined original and heatmap embeddings beats the prior best on the same 17-family dataset.","key_machinery":"The load-bearing object is the Grad-CAM heatmap, $L^c_{\\mathrm{Grad\\text{-}CAM}} = \\mathrm{ReLU}\\bigl(\\sum_k \\alpha_k^c A^k\\bigr)$, where $A^k$ is the last convolutional activation map of MobileNetV2 and $\\alpha_k^c$ is the spatially averaged gradient of the class logit with respect to $A^k$. The heatmap is generated for every sample and transformation, and the true family label is used as the target class. Three uses carry the argument: overlaying the heatmap on the image to create a new input representation, extracting fifty handcrafted statistical, shape, and texture features from the raw heatmap for classical classifiers, and using the CNN's 256-dimensional penultimate-layer embedding from each original and overlay model. The final model concatenates all sixteen 256-dimensional embeddings into a 4096-dimensional vector and classifies it with Random Forest. Faithfulness is quantified as insertion AUC minus deletion AUC from pixel-perturbation experiments, and stability is quantified by Spearman correlation, SSIM, and top-20% overlap under Gaussian noise.","core_discovery":"The central claim is that a Random Forest trained on concatenated penultimate-layer embeddings from 16 MobileNetV2 models—eight trained on original transformation images and eight on their Grad-CAM overlays—attains 0.777 test accuracy on the 17-family dataset, exceeding the 0.751 benchmark from prior work on the same data. The paper also claims that accuracy and explanation faithfulness do not coincide: Bigram Polar ranks first on faithfulness but seventh on CNN accuracy, whereas Grayscale ranks well on accuracy but seventh on faithfulness, and Entropy Hilbert is the only transformation that is strong on both. In addition, Grad-CAM overlays are claimed to be more than visualizations: for five of the eight transformations, CNNs trained on overlay images match or beat those trained on the original images, with the largest gains for Grayscale and Byteclass Hilbert.","pith_inferences":["The paper leaves implicit that its central comparison would change under a predicted-label protocol: if Grad-CAM targets the model's own predicted class instead of the true label, the overlay and 4096-dimensional accuracies could drop, and that drop would measure how much label information leaks into feature generation.","A natural extension is to add HOG features to the 4096-dimensional vector; the paper shows HOG consistently helps in smaller hybrids, so the same gain may appear at full fusion, though the paper does not test it.","The faithfulness-stability inversion suggests a practical selection rule for security analysts: choose Entropy Hilbert for a single representation that serves both accuracy and human review, and treat Bigram Polar explanations as precise but fragile."],"forward_implications":["Explanations themselves carry family-discriminative signal: a 4096-dimensional Random Forest built from CNN embeddings of original and Grad-CAM overlay images reaches 0.777 test accuracy on 17 families, above the previous 0.751 benchmark on the same dataset.","For five of eight transformations, CNNs trained on Grad-CAM overlay images match or outperform CNNs trained on the original images, so overlays can serve as an input representation rather than only a diagnostic tool.","Accuracy alone is a poor guide to explainability: the most faithful transformation, Bigram Polar, is mid-tier in accuracy, while the most accurate transformation, Entropy Hilbert, is also second-most faithful, making it the best compromise.","Faithfulness and stability are inversely ranked across transformations, so explanation quality is not a single axis; an analyst must choose between explanations that are correct and explanations that are robust to small input changes."],"supporting_citations":[{"why":"Supplies the eight image transformations, the 17-family dataset setup, and the 0.751 benchmark that the 0.777 result is compared against.","marker":"[2]"},{"why":"Provides the RawMalTF malware dataset with family labels and the 1,000-sample-per-family selection.","marker":"[5]"},{"why":"Defines Grad-CAM, the explanation method whose heatmaps and overlays are the paper's central feature source.","marker":"[35]"},{"why":"Provides HiResCAM, the comparison method used to show that Grad-CAM and HiResCAM coincide at MobileNetV2's final layer.","marker":"[18]"},{"why":"Provides HOG features used in the hybrid CNN-HOG-XGBoost pipeline that consistently improves on CNN-only accuracy.","marker":"[15]"},{"why":"Supplies the deletion test protocol used to compute deletion AUC for faithfulness.","marker":"[19]"},{"why":"Supplies the insertion test protocol used to compute insertion AUC for faithfulness.","marker":"[33]"},{"why":"Defines GLCM texture statistics used in the handcrafted Grad-CAM feature vector.","marker":"[21]"},{"why":"Defines local binary patterns used to quantify the texture of Grad-CAM heatmaps.","marker":"[32]"}],"fun_headline_variants":["Grad-CAM overlays push malware classification to 0.777","Random Forest on Grad-CAM features ups malware accuracy","In malware images, accuracy and faithfulness diverge","Grad-CAM hybrid tops 17-family malware accuracy benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the true family label is available when generating each Grad-CAM overlay, because the explanation is computed with the ground-truth class as target; if a deployed system must produce overlays without knowing the label, the reported 0.777 accuracy describes a different, untested procedure.","fun_headline_variants_meta":{"raw":{"variants":["Grad-CAM overlays push malware classification to 0.777","Random Forest on Grad-CAM features ups malware accuracy","In malware images, accuracy and faithfulness diverge","Grad-CAM hybrid tops 17-family malware accuracy benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1740,"prompt_tokens":951,"completion_tokens":789,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":721}},"tokens_in":567,"tokens_out":789,"duration_ms":8047,"temperature":1.0,"reasoning_tokens":721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:17:15.816619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the overlay dataset with the model's predicted class as the Grad-CAM target, retrain the 4096-dimensional Random Forest, and compare test accuracy to 0.777; a substantial drop would show the headline result depends on label leakage into feature generation.","supporting_citations":[{"cited_title":"RISE: Randomized input sampling for explanation of black-box models","cited_arxiv_id":null,"evidence_quote":"Supplies the insertion test protocol used to compute insertion AUC for faithfulness."},{"cited_title":"A comparison of selected image transformation techniques for malware classification","cited_arxiv_id":null,"evidence_quote":"Supplies the eight image transformations, the 17-family dataset setup, and the 0.751 benchmark that the 0.777 result is compared against."},{"cited_title":"Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra","cited_arxiv_id":null,"evidence_quote":"Defines Grad-CAM, the explanation method whose heatmaps and overlays are the paper's central feature source."},{"cited_title":"Histograms of oriented gradients for human detection","cited_arxiv_id":null,"evidence_quote":"Provides HOG features used in the hybrid CNN-HOG-XGBoost pipeline that consistently improves on CNN-only accuracy."},{"cited_title":"Computing and evaluating saliency maps for image classification: a tutorial.Journal of Electronic Imaging, 32(2):020801, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the deletion test protocol used to compute deletion AUC for faithfulness."},{"cited_title":"Haralick, K","cited_arxiv_id":null,"evidence_quote":"Defines GLCM texture statistics used in the handcrafted Grad-CAM feature vector."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines local binary patterns used to quantify the texture of Grad-CAM heatmaps."}],"review_version":1}