{"id":"d322bea0-ddb4-4f60-b711-98cf8b04baff","arxiv_id":"2608.11634","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Saliency-guided cutout does not improve grayscale malware classification in controlled ResNet18 experiments, though low-saliency cutout slightly helps CIFAR-100, indicating domain-dependent value.","lead":"This study tests whether guiding cutout augmentation with saliency maps improves malware image classification. It finds that saliency-guided cutout does not help on malware images, while low-saliency cutout gives a small gain on natural images, so the benefit is domain dependent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random-cutout control may re-sample masks each epoch while CAM-guided masks are static, confounding saliency placement with mask diversity.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the paper's central comparison between saliency-guided and random cutout is meant to isolate placement rule, but Section 3.3 only guarantees static masks for the CAM-guided conditions. The sentence 'The augmented copies are static' appears in the CAM-guided paragraph and is tied to caching for CAM generation, so it is ambiguous whether random cutout is also static. If random re-samples each epoch, then the random control has strictly more mask diversity, which can independently improve regularization and obscure or create differences attributed to saliency. This concern is concrete, checkable in the public code, and affects the paper's main empirical contrast (low/random and high/random paired differences). However, it does not overturn the paper's most robust finding that no-cutout outperforms every cutout variant on RawMal-TF, since that comparison is insensitive to the random mask dynamics. The absence of a held-out test set is a real limitation but is explicitly acknowledged in Section 3.6 and Section 5.5, and the paper frames its claims in terms of validation accuracy. Because the concern is addressable and the reader already issued CONDITIONAL, the verdict should remain unchanged while requesting the clarifying evidence. The public repository and committed runs make the test straightforward, and the paper's transparency is a point in its favor.","tokens_in":14180,"tokens_out":2968,"duration_ms":33033,"concrete_test":"Inspect cutout.py at repository commit cd7f42fe9b3c3b52e291acf795e2dcc7e75bd14f to determine whether the RandomCutout dataset wrapper caches a fixed window per (sample_index, augmentation_index) or samples a fresh random window in __getitem__ on every access. If random re-samples, rerun the RawMal-TF random cutout condition with static masks (using the same per-index seed scheme as the CAM-guided conditions) for M=4, area=10%, seeds 42-44, and recompute the low-saliency minus random paired differences; if the paired effect changes by more than a small fraction of the reported standard deviation, the claimed seed-sensitivity is confounded by mask diversity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that for CAM-guided cutout, 'The augmented copies are static: a sample index and augmentation index map to the same selected window in every epoch,' and that this caching is motivated by CAM generation cost. The text does not specify whether the standard random cutout control also uses a static window per sample/copy index or re-samples a new window each epoch. If random cutout re-samples, the random control enjoys greater mask diversity over training. Then the paired comparisons in Tables 3 and 4 (low-minus-random and high-minus-random) do not isolate the effect of saliency-based placement; they conflate placement rule with temporal mask diversity. This directly affects the paper's secondary claims that low-saliency cutout is seed-sensitive relative to random on RawMal-TF and that high-saliency cutout is generally harmful. The primary negative claim that no-cutout beats all cutout conditions is less affected, because all cutout conditions would share the static/dynamic issue, but the core controlled comparison that motivates the study would be compromised. The public repository makes this concern directly checkable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper experimentally evaluates whether cutout regularization with regions selected by HiResCAM saliency improves image-based malware classification relative to no cutout and standard random cutout. Using ResNet18 with 100 epochs, seeds 42–44, cutout areas of 5%, 10%, 20%, and 30%, and M=4 and M=8 augmented copies, the authors compare four conditions (no cutout, random, low-saliency, high-saliency) on grayscale RawMal-TF images (17 families, 1,000 samples per family) and, for contrast, on CIFAR-100. The paper reports that on RawMal-TF the no-cutout baseline achieves the highest mean best validation accuracy (72.83%±0.16%) and that every cutout condition is lower; low-saliency cutout is seed-sensitive relative to matched random cutout, and high-saliency cutout is generally worse. On CIFAR-100, low-saliency cutout at 10% area improves over the no-cutout baseline (63.51% vs. 62.65%) while high-saliency cutout is consistently harmful. The authors conclude that the value of saliency-guided cutout is domain dependent and that malware images should not be treated as equivalent to natural images.","tokens_in":14319,"tokens_out":4842,"duration_ms":47288,"significance":"If the experimental comparison is sound, this paper provides a useful controlled negative result for CAM-guided cutout in malware classification and a contrasting positive result on CIFAR-100. The study's strengths include within-seed paired comparisons, multiple seeds and cutout areas, a teacher–student setup that avoids circularity (the teacher is trained without cutout and the student is trained on different augmented data), and explicit acknowledgment that three seeds make the statistics exploratory. The public repository and per-run metrics files support reproducibility. The main caveat is that the central matched comparison between saliency-guided and random cutout presumes that the two conditions differ only in how the cutout window is selected; if the random control re-samples masks each epoch while the CAM-guided masks are static, the measured differences conflate placement rule with mask diversity. This issue must be resolved before the secondary conclusions about low- versus high-saliency placement and the CIFAR-100 advantage over random cutout can be considered reliable.","major_comments":[{"comment":"The paper does not state whether the standard random cutout control also uses a static window per sample/copy index or re-samples a new window each epoch. Section 3.3 explicitly says that for CAM-guided cutout \"the augmented copies are static: a sample index and augmentation index map to the same selected window in every epoch,\" but no corresponding statement is given for the random condition. If random cutout re-samples each epoch, then the random control enjoys greater mask diversity over the 100 training epochs, meaning the paired differences in Tables 3 and 4 (low-minus-random and high-minus-random) do not isolate the effect of saliency-based placement; they conflate the placement rule with temporal mask diversity. This directly affects the paper's secondary claims that low-saliency cutout is seed-sensitive relative to random on RawMal-TF, that high-saliency cutout is generally harmful, and that low-saliency cutout improves over random on CIFAR-100. The primary negative claim that no-cutout beats all cutout conditions is less affected, but the controlled comparison that motivates the study is compromised. The authors should state explicitly how the random control is implemented; if it is dynamic, they should either re-run with matched static masks or substantially temper these comparative conclusions.","section":"Section 3.3, Tables 3 and 4"}],"minor_comments":[{"comment":"In the concluding sentence of the introduction, \"the field fo malware analysis\" should read \"the field of malware analysis.\"","section":"Section 1, Introduction"},{"comment":"The sentence \"a model becomes more stronger\" contains a redundant comparative; it should be \"a model becomes stronger.\"","section":"Section 2.1"},{"comment":"The description of candidate-window selection says that a \"specified, copy-specific seed value\" is used, but it does not clarify whether this seed is also epoch-specific or fixed across epochs. This is closely related to the major issue about static versus dynamic masks and should be clarified in the revision.","section":"Section 3.3"},{"comment":"The paper appropriately acknowledges that only validation metrics are available and that no held-out test set is reported. Given this limitation, the abstract and conclusions should perhaps more consistently use the phrase \"validation accuracy\" rather than \"accuracy\" to avoid any impression of test-set generalization, though the current phrasing is mostly careful.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The static/dynamic mask issue is directly checkable in the public repository because the code and committed run folders are available. I recommend asking the authors to verify and report the behavior of the random cutout condition. If the random control is dynamic, the paper's central matched-comparison claims need substantial qualification, and the CIFAR-100 low-saliency advantage over random cutout should not be stated as robust until the comparison is rerun under matched conditions. The paper is otherwise carefully scoped and honestly labeled, so this is a fixable issue rather than a fundamental rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a controlled, honest empirical study of whether HiResCAM-guided cutout improves malware classification. The main finding is negative for malware: on grayscale RawMal-TF, no-cutout beats every cutout condition. On CIFAR-100, low-saliency cutout gives a small real gain. The novelty is moderate—the components are known—but the controlled sweep against matched random cutout and the negative malware result are new. They also ship run artifacts, configs, and code, and they explicitly label every conclusion exploratory. That is credit where due.\n\nWhat is good: the design holds cutout area and augmentation multiplicity fixed, compares low- and high-saliency cutout to same-seed/same-area/M random controls, reports paired differences, and includes a natural-image domain contrast. The limitations section is unusually candid: three seeds, ResNet18 only, validation-only metrics, no held-out test set, no family-level metrics. They do not overclaim; the conclusion is appropriately negative.\n\nSoft spots, in order of severity. First, the stress-test ambiguity is real. Section 3.3 says CAM-guided augmented copies are static because of caching, but never says whether the random cutout control re-samples a new window each epoch or is also static. If random re-samples, the random control has more mask diversity, and the low-minus-random and high-minus-random paired comparisons conflate placement rule with temporal diversity. That directly affects the secondary claims about low-saliency seed-sensitivity on RawMal-TF and high-saliency harmfulness on CIFAR-100. The primary negative claim—no-cutout beats all cutout—is less affected because it does not depend on the random control's diversity. The public repo makes this directly checkable; the authors should specify the random control's behavior and, if needed, match the caching.\n\nSecond, validation-only metrics with three seeds. They acknowledge this, and the paired structure helps, but the absence of a held-out test set means the absolute numbers should not be treated as final. Third, a single architecture. They say this is deliberate, so it is a scope limit rather than a flaw.\n\nMinor: a few typos, and the CIFAR-100 random control deteriorates sharply at large areas, which makes the low-saliency advantage at 20–30% area look partly like a comparison against a weak control. Still, the 10% low-saliency improvement over no cutout is independent of that. The citation pattern looks reasonable, with relevant prior cutout, CAM, and malware-visualization work covered.\n\nVerdict: this deserves a serious referee. The central negative result is supported, and the paper is useful as a domain-dependence caveat for practitioners. I would ask for the random-control clarification and a held-out evaluation before publication, but I would not desk-reject. Worth a maybe for reading group.","headline":"A well-scoped negative result with unusually transparent reporting; the central claim survives the main ambiguity, but the random-cutout control's static/dynamic behavior needs checking before the paired claims can be trusted.","tokens_in":14884,"tokens_out":2670,"would_cite":true,"duration_ms":28845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Saliency-guided cutout does not improve grayscale malware classification.","keywords":["HiResCAM","cutout regularization","saliency-guided augmentation","malware image classification","RawMal-TF","CIFAR-100","ResNet18","data augmentation"],"falsifier":"Run the random-cutout control with exactly the same cached static window per sample and augmentation index used by the CAM-guided conditions; if the low-saliency advantage over random on CIFAR-100 shrinks or reverses, the reported placement effect is partly an artifact of mask diversity rather than of saliency.","tokens_in":1734,"feed_emoji":"🧩","tokens_out":4600,"duration_ms":110431,"temperature":0.7,"pith_summary":"The paper tests whether cutout regularization can be improved by placing the removed square where a saliency map says the image matters least or most. It compares no cutout, random cutout, low-saliency cutout, and high-saliency cutout on grayscale RawMal-TF malware images and CIFAR-100, all with ResNet18 and three seeds. The central result is negative for malware: the no-cutout baseline reaches $72.83\\% \\pm 0.16\\%$ mean best validation accuracy, and every cutout condition lands lower, with high-saliency cutout generally the worst. The same pipeline on CIFAR-100 improves under low-saliency cutout, reaching $63.51\\% \\pm 0.36\\%$ versus $62.65\\% \\pm 0.57\\%$ for no cutout. The paper reads this as evidence that saliency-guided cutout is domain dependent and that malware images cannot be treated as natural images.","feed_headline":"Saliency cutout fails to beat no-cutout baseline","feed_subtitle":"ResNet18 on grayscale RawMal-TF peaks at 72.83% without cutout; CIFAR-100 gains under low-saliency masks.","key_machinery":"The machinery is a HiResCAM (High-Resolution Class Activation Mapping) saliency heatmap produced by a no-cutout ResNet18 teacher and used to place square cutout windows. For each training image, the teacher's predicted-class heatmap is normalized and resized; candidate squares are scored by mean saliency, the lowest- or highest-scoring $10\\%$ form the candidate set, and one window is chosen per copy with a copy-specific seed. The chosen windows are cached, so each sample-index and augmentation-index pair maps to the same static window every epoch. The student is then trained on the original image plus $M$ occluded copies, with validation images never cut out, and every saliency condition is compared against random cutout with identical cutout area and $M$. This design is meant to isolate the placement rule as the only difference between conditions.","core_discovery":"The paper's central claim is that HiResCAM-guided cutout does not improve grayscale malware family classification in the tested controlled setting. On RawMal-TF, the no-cutout ResNet18 baseline attains the highest mean best validation accuracy, $72.83\\% \\pm 0.16\\%$; the best cutout condition is random cutout with $M=4$ and $30\\%$ area at $71.55\\% \\pm 0.45\\%$, and the best low-saliency condition is $71.43\\% \\pm 1.12\\%$. Low-saliency cutout is seed-sensitive relative to matched random cutout, ranging from $-0.56$ to $+0.46$ percentage points across the eight area and $M$ cells, while high-saliency cutout is negative in most cells, with paired deficits from $-0.21$ to $-2.00$ percentage points. On CIFAR-100 the pattern reverses: low-saliency cutout with $M=4$ and $10\\%$ area reaches $63.51\\% \\pm 0.36\\%$ versus $62.65\\% \\pm 0.57\\%$ for no cutout, and low-saliency placement protects large-mask performance, while high-saliency cutout is consistently harmful. The conclusion is that the value of saliency-guided cutout depends on the image domain, so malware-image saliency should not be equated with natural-image object saliency.","pith_inferences":["A tighter random control that caches one static window per sample and augmentation index, exactly like the CAM-guided runs, would test whether the reported low-versus-random differences are inflated by mask diversity; the paper does not state whether the random condition re-samples each epoch.","The teacher's HiResCAM is computed for its predicted class from the final convolutional layer, so using ground-truth labels or intermediate layers could change both low- and high-saliency rankings, an extension the paper leaves open.","Aligning masks with executable structure, such as PE-section boundaries or byte ranges, rather than image-saliency squares, is the paper's own suggestion for future work and could separate the failure of square saliency from the failure of cutout itself."],"forward_implications":["On grayscale RawMal-TF with ResNet18 and 100 epochs, adding cutout in any tested form lowers mean best validation accuracy, so the no-cutout baseline is the strongest setting in this controlled sweep.","Low-saliency cutout is not reliably better than random cutout on malware: its paired advantage ranges from $-0.56$ to $+0.46$ percentage points and changes sign across seeds, while high-saliency cutout is generally worse by up to 2.00 percentage points.","On CIFAR-100, low-saliency cutout improves peak validation accuracy over no cutout at small mask sizes and over random cutout for large masks, whereas high-saliency cutout is worse than random in every tested cell.","The same implementation can produce a positive saliency-guided result, so the negative RawMal-TF outcome is a domain effect rather than a broken pipeline."],"supporting_citations":[{"why":"It defines standard random cutout, which is the primary control condition and the method the saliency-guided variants modify.","marker":"[6]"},{"why":"It defines HiResCAM, the saliency map used to score and select low- and high-saliency cutout windows.","marker":"[8]"},{"why":"It provides the RawMal-TF dataset with malware samples and family labels used in the main experiments.","marker":"[3]"},{"why":"It documents the eight RawMal-TF image transformations, including the grayscale representation the paper selects.","marker":"[2]"},{"why":"It provides the CIFAR-100 dataset used as the natural-image comparison.","marker":"[15]"},{"why":"It supplies the ResNet18 architecture used for teacher, baseline, and student runs.","marker":"[12]"},{"why":"It motivates applying CAM maps to malware visualization and classification, the approach this paper adapts to cutout placement.","marker":"[5]"}],"fun_headline_variants":["Saliency cutout fails to lift malware accuracy","Cutout helps natural images, not malware","HiResCAM cutout: domain-dependent at best","Malware images resist saliency cutout gains"],"cache_read_input_tokens":17024,"weakest_assumption_plain":"The controlled comparison assumes that the CAM-guided and random cutout conditions differ only in where the square is placed, yet the paper specifies that CAM-guided masks are static across epochs and does not say whether the random control re-samples its window each epoch.","fun_headline_variants_meta":{"raw":{"variants":["Saliency cutout fails to lift malware accuracy","Cutout helps natural images, not malware","HiResCAM cutout: domain-dependent at best","Malware images resist saliency cutout gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":3106,"prompt_tokens":1134,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":1913}},"tokens_in":750,"tokens_out":1972,"duration_ms":16964,"temperature":1.0,"reasoning_tokens":1913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:33:08.463019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the random-cutout control with exactly the same cached static window per sample and augmentation index used by the CAM-guided conditions; if the low-saliency advantage over random on CIFAR-100 shrinks or reverses, the reported placement effect is partly an artifact of mask diversity rather than of saliency.","supporting_citations":[{"cited_title":"A Comparison of Selected Image Transformation Techniques for Malware Classification","cited_arxiv_id":"2509.10838","evidence_quote":"It documents the eight RawMal-TF image transformations, including the grayscale representation the paper selects."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"It provides the CIFAR-100 dataset used as the natural-image comparison."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"It supplies the ResNet18 architecture used for teacher, baseline, and student runs."},{"cited_title":"Through the static: Demystifying malware visualization via explainability.Journal of Information Security and Applications, 91:104063, 2025","cited_arxiv_id":null,"evidence_quote":"It motivates applying CAM maps to malware visualization and classification, the approach this paper adapts to cutout placement."}],"review_version":1}