{"id":"51e320d2-451e-4ea3-9a80-4cd1366ac911","arxiv_id":"2411.14029","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Siamese denoising autoencoder with relation-aware embeddings classifies malware from entropy images, improving few-shot accuracy on two datasets, including under NOP obfuscation.","lead":"This paper combines a Siamese neural network with denoising autoencoders to classify malware from entropy images, tested in few-shot settings with and without NOP obfuscation. It is relevant to cyber security because it aims to recognize new malware families from very few examples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The few-shot evaluation may not actually test unseen families: episodes are drawn from the combined train/test pool and augmentation may cross the split, which would invalidate the central zero-day claim.","rationale":"The paper's strongest claim is that the model classifies unseen, obfuscated malware from few examples. The most load-bearing condition for that claim is that training and test episodes are disjoint at the family level and that no augmented variant leaks across the split. The text in Section V-B undermines this condition: episodes are drawn from the \"total set of training and test sets,\" and augmentation is described before the split without a leakage guarantee. I therefore focus on protocol validity rather than the narrower obfuscation-coverage concern identified by the reader. If the protocol is clean, the remaining issues (NOP-only obfuscation, small gains, and lack of significance tests) still warrant a conditional verdict. If the protocol is not clean, the empirical core of the paper does not support the central claim. The verdict should remain conditional, with the added requirement that code or a precise protocol description be provided and the key Table IV rows be rerun under a strictly class-disjoint, split-after-augmentation evaluation.","tokens_in":16588,"tokens_out":9308,"duration_ms":91879,"concrete_test":"Ask for the episode-construction code or an exact protocol description, then run two checks. (1) Verify that the family split is class-disjoint and that every VUW and Malimage episode samples only held-out families for both support and query; (2) verify that augmentation is applied after the split and that no rotated or rescaled copy of a training binary appears in any test episode. Then rerun the Table IV rows for Malimage 2-way 1-shot and VUW 5-way 1-shot under the corrected protocol. If the numbers drop materially, or if VUW 5-way cannot be constructed with held-out families alone, the zero-day generalization claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section V-B states that accuracy is measured \"on random datasets drawn from the total set of training and test sets in each of 20,000 episodes.\" That wording allows episodes to contain training-set classes as well as held-out classes; true few-shot evaluation of zero-day malware must restrict support and query episodes to families never seen during training. The same section applies rotation/rescaling augmentation (\"We increased the image sample size ... using a data augmentation technique\") and only afterwards describes the random split, but it never states that augmented copies of one original binary are confined to one split. If they are not, a test image can be a near-duplicate of a training image, and the reported Malimage accuracies (96.1% and 97.3%) and the claimed robustness gains would be inflated by memorization rather than generalization to unseen families. In addition, the 11-family VUW dataset makes a clean held-out 5-way evaluation questionable under the described split; the paper does not specify how 5-way episodes are formed from a small held-out set. This concern is more fundamental than the NOP-only obfuscation limitation: no amount of denoising can support the central claim if the evaluation protocol does not isolate unseen classes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a few-shot malware-family classifier that combines entropy-image representations, a Siamese denoising autoencoder, and a relation module. The model takes pairs of malware entropy images (one support, one query), encodes them with shared-weight convolutional encoders, reconstructs them with decoders, and computes a relation score for the pair. The authors evaluate on the VUW ransomware dataset (11 families) and the Malimage dataset (25 families) under 2-way/5-way, 1-shot/5-shot episodes, with and without NOP-insertion obfuscation. They report strong results on Malimage (96.1% and 97.3% for 2-way 1-shot and 5-shot) and claim that the model generalizes to previously unseen malware families and is more robust to NOP obfuscation than Prototypical Networks. The paper also includes ablations on the reconstruction-loss weight, linear-layer dimension, pooling method, and the presence of the decoder.","tokens_in":16779,"tokens_out":11707,"duration_ms":102990,"significance":"If the central claims were established, the paper would offer a practically relevant architecture: the use of entropy images as input, a Siamese relation module for few-shot comparison, and a decoder intended to strip obfuscation noise are all sensible building blocks, and the comparison against Prototypical Networks on a well-known public dataset (Malimage) is a useful reference point. The paper also contains some positive experimental elements: it tests two datasets, reports multiple few-shot configurations, and includes an ablation isolating the decoder's contribution. However, the evaluation protocol has serious gaps: the episode-generation description appears to allow training classes in evaluation episodes, the augmentation-before-split ordering is not justified, and only NOP insertion at a single frequency is used for the obfuscation claim. These gaps mean that the paper's headline claims are not yet supported, and the current results should be interpreted with caution.","major_comments":[{"comment":"The sentence in Section V-B stating that accuracy is measured 'on random datasets drawn from the total set of training and test sets in each of 20,000 episodes' is, taken literally, an invalid evaluation of few-shot generalization: if an episode can contain classes from the training split, the model has already seen those classes, so the reported accuracies do not measure classification of previously unseen malware families. The split description in Section V-C ('9/6 classes and 2/5 classes') is also ambiguous. Please specify unambiguously that support and query sets are drawn only from held-out classes in every test episode, and if that is not what was done, rerun the experiments with class-disjoint episodes. Given the small VUW test sets (2 classes for 2-way, 5 classes for 5-way), also explain how 20,000 non-overlapping episodes are constructed or why repeated sampling does not bias the estimates.","section":"V-B, V-C"},{"comment":"The augmentation step (rotations of 90, 180, 270 degrees and rescaling) is described before the random class split, but the paper does not state whether the split unit is the original binary or the augmented image. If augmented views of the same original file can appear on both sides of the split, the test accuracy is inflated by near-duplicate images. Please state explicitly that all augmented variants of a single original sample are assigned to the same split, and ideally provide sample-level identifiers so that disjointness can be verified.","section":"V-B"},{"comment":"The robustness claim is supported only by NOP insertion at frequency 200, and the model is trained and evaluated on the same obfuscation procedure. This evaluates the model's ability to fit a specific noise pattern, not its robustness to obfuscation in general. The conclusion's statement that other obfuscation techniques (Packers/XOR/Base64, register reassignment, code transposition) are future work is an explicit acknowledgment of this limitation, but the abstract and contributions use the broader phrase 'obfuscation techniques.' Please narrow the claims to NOP insertion or add experiments with held-out obfuscation types, including at least one type not seen during training.","section":"V-B, VI, Table IV"},{"comment":"The denoising objective is not implemented as described. Algorithm 1 lists original samples x1_o, x2_o as inputs but never uses them; the reconstruction loss is computed as RC([dn1, en1], [dn2, en2]), which appears to compare the decoder output with the encoder input (the obfuscated sample). Eq. (5), by contrast, states that the target is the original (uncorrupted) sample. If the decoder is trained only to reconstruct the obfuscated input, then the model is not a denoising autoencoder, and the ablation in Section V-F.3 cannot be interpreted as evidence that denoising improves robustness. Please clarify the exact input/target pair used in the reconstruction loss and, if necessary, retrain with clean targets.","section":"Algorithm 1, Eq. (5)"},{"comment":"The claimed advantage over Prototypical Networks is not consistent across the reported results. On VUW with obfuscation, the proposed model is worse than Prototypical in 5-way 1-shot (53.2% vs 65.2%) and 5-way 5-shot (57.3% vs 68.3%). On Malimage without obfuscation, the proposed model is below Prototypical in all 5-way settings (e.g., 83.8% vs 92.3% in 5-way 1-shot), and with obfuscation it is below in 5-way 1-shot (82.9% vs 85.4%). The paper emphasizes the smaller accuracy drops rather than the absolute accuracies, and no significance tests are reported. Please provide a statistical comparison (e.g., paired tests across episodes or repeated splits) and discuss the configurations where the proposed model does not improve over the baseline.","section":"Table IV, V-D, V-E"}],"minor_comments":[{"comment":"The text in Section V-B refers to classes 'Blocal' and 'Newbak' as having very few samples, but these names do not appear in Table I; the table's minimum is Petya with 6 samples. Please correct the inconsistency.","section":"V-A, V-B"},{"comment":"Equation (7) is incomplete: the summation sign and the contrastive-loss terms L_s and L_d in Eq. (8) are not defined. Please provide the full forms and definitions.","section":"IV-D, Eq. (7)"},{"comment":"The learning rate is given as 1e-02 in Table III but as 0.02 in the text, and the batch size entry '19/10-15' in Table III is unclear. Please harmonize the notation.","section":"V-C, Table III"},{"comment":"The text uses 'epochs' and 'episodes' interchangeably (20,000 epochs in V-C vs 20,000 episodes in V-B); since the training is episode-based, please use consistent terminology.","section":"V-B, V-C"},{"comment":"The description of how entropy sequences are converted into a 105x105 image is vague ('concatenated with the full entropy sequences sequentially'); please specify the block size, stride, and normalization used to form the image.","section":"IV-A"},{"comment":"No code, random seeds, or sample-level split definitions are provided, which makes the ambiguities in the episode-generation protocol impossible to resolve from the manuscript; releasing these artifacts would substantially strengthen the paper.","section":"V"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the episode-generation protocol; if the literal reading of Section V-B is correct, the central zero-day claim is not tested and the experiments should be redone. The NOP-only obfuscation is a secondary but important limitation. I believe the paper is potentially salvageable with a careful revision, but the current version is not publishable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable incremental paper on few-shot malware classification, but the central robustness claim is not currently supported by the experiments.\n\nWhat's new: the combination of entropy images, a Siamese denoising autoencoder, and a relation-aware module for few-shot malware classification. Each ingredient exists, but the specific stack is not in the cited prior work, including their own [2]. The ablation study on the reconstruction loss weight and pooling methods is a useful addition. The authors also report comparisons against Relation Networks and Prototypical Networks.\n\nThe problems are in the evaluation. The paper states that accuracy is measured 'on random datasets drawn from the total set of training and test sets.' If that means what it says, test episodes can include classes from the training split, which would break the few-shot protocol and invalidate the zero-day claim. The later description of a class split into 9/6 and 13/12 classes does not resolve the ambiguity. The data augmentation (rotation/rescaling) is described before the split, and the paper never says augmented copies of one original are kept on one side of the split. If they are not, near-duplicates can leak across the boundary and inflate accuracy. These concerns are as serious as the NOP-only obfuscation limitation, and they need to be addressed before the numbers can be taken at face value.\n\nOn obfuscation: the claim of robustness rests on a single technique (NOP insertion at frequency 200). The results are mixed — on VUW the model's obfuscated accuracy is actually below the non-obfuscated baseline and sometimes below the Prototypical network. The Malimage 2-way numbers look better, but the 5-way results are not consistently better. No significance tests, no error bars beyond standard deviation, and no code or data are provided.\n\nThe paper is not a waste of time. The method is plausible and the ablation gives some insight. But the evidence for the central claim is weak. A serious referee could fix this by asking for code, a clear description of episode construction, a leakage-free split, and tests with at least one more obfuscation type. I would not cite this version in its current form.\n\nRecommendation: send it to review — it's the right kind of problem and the authors have done enough groundwork to deserve a chance to fix the protocol.","headline":"Sensible incremental method, but the zero-day robustness claim is undercut by an ambiguous episode-construction protocol and single-obfuscation testing.","tokens_in":17364,"tokens_out":3519,"would_cite":false,"duration_ms":32521,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Siamese denoising autoencoder trained on entropy images, with a learned relation module instead of a fixed distance metric, can classify previously unseen malware families from one or five labelled examples and…","keywords":["malware classification","few-shot learning","Siamese neural network","denoising autoencoder","entropy image","NOP obfuscation","zero-day malware","relation network"],"falsifier":"Run the same 2-way and 5-way few-shot evaluation with other obfuscation techniques, such as packing, XOR, or register reassignment, on Malimage and VUW: if accuracy drops as much as or more than the Prototypical baseline's, the anti-obfuscation claim is specific to NOP insertion rather than general. A complementary check is to train the model on obfuscated images but decode to clean images, then verify on a held-out family that the reconstructed entropy image actually diverges from the obfuscated input at the inserted NOP positions.","tokens_in":16357,"feed_emoji":"🛡️","tokens_out":6286,"duration_ms":52037,"temperature":0.7,"pith_summary":"Using only static analysis, the paper seeks to establish that newly seen malware families can be classified from one or five labelled examples even after junk-code obfuscation, provided the classifier combines entropy images, a Siamese denoising autoencoder, and a learned relation module. The authors argue that existing autoencoder-based detectors treat each sample independently and therefore miss the semantic relationships between malware samples, while fixed distance metrics are too rigid for obfuscated inputs. Their model compares pairs of entropy images through shared-weight branches, reconstructs the clean image from the NOP-obfuscated version in each branch, and scores similarity with a trained relation network. On the Malimage dataset it reports 96.1% accuracy for 2-way 1-shot and 97.3% for 2-way 5-shot, and under NOP insertion its accuracy drops less than the Prototypical Networks baseline, which the paper reads as evidence that the approach helps with anti-obfuscation.","feed_headline":"Relation-aware Siamese net spots obfuscated malware from few samples","feed_subtitle":"Reports 96.1% accuracy from one labeled example and smaller drops under NOP obfuscation than the baseline.","key_machinery":"The central machinery is a Siamese denoising autoencoder with a relation module on top. Each branch is a convolutional encoder-decoder; the encoder compresses an input entropy image into a latent vector while the decoder reconstructs the clean version, and the reconstruction loss is what is meant to strip the noise added by NOP obfuscation. The relation module is a small network that takes the concatenation of two latent embeddings (after mean pooling of support samples) and outputs a similarity score, so the model learns the comparison instead of using a fixed metric. The input representation is the entropy image, a $105\\times105$ grayscale image obtained by computing Shannon entropy over bytecode blocks, which the paper argues preserves structural information and subtle differences that survive junk-code insertion. The total objective is $L = L_r + 0.7\\,L_{mse}$, tying similarity scoring to denoising in a single training episode.","core_discovery":"The central claim is that the relations between malware samples, learned from pairs of entropy images, are a more reliable basis for few-shot classification of unseen and obfuscated malware than matching independent feature vectors. The network is a Siamese denoising autoencoder: two branches with shared weights each contain a convolutional encoder and a decoder, and the decoder is trained to reconstruct the clean entropy image from an input that has had NOP instructions inserted into its bytecode. The latent embeddings produced by the encoders are concatenated and passed through a relation module that outputs a similarity probability, replacing a hand-chosen distance with a learnable comparator; the training loss combines the relation loss with the reconstruction loss using weight $\\lambda=0.7$. The paper reports that on Malimage this reaches 96.1% in 2-way 1-shot and 97.3% in 2-way 5-shot, and that under NOP obfuscation the accuracy drops by 1.8 and 1.6 percentage points compared with drops of 3.7 and 2.0 for Prototypical Networks. It concludes from these results that the model can predict previously unseen malware classes and is more robust to NOP insertion than the baseline.","pith_inferences":["Beyond the paper: if the entropy representation is as invariant as claimed, the same architecture may transfer to other byte-level obfuscations such as packing or XOR, but the paper does not test this.","Beyond the paper: the denoising-plus-relation design is not malware-specific, so it could be tried on other few-shot image tasks where noise is structured rather than Gaussian, such as occluded or watermarked images.","Beyond the paper: the 5-way results on the VUW dataset are substantially lower than the 2-way results, so the advantage may shrink as the number of candidate families grows; testing at 10-way or 20-way episodes would clarify practical zero-day conditions.","Beyond the paper: the reported comparisons do not separately ablate the decoder and the relation module under obfuscation, so an experiment isolating each component would show which one actually carries the anti-obfuscation gain."],"forward_implications":["Unseen malware families can be recognised from one or five labelled examples using entropy images alone, without feature retraining on the new family.","NOP-inserted junk code can be partially absorbed by the denoising branch, since on Malimage the model's obfuscated-versus-clean accuracy drop is smaller than the Prototypical baseline's.","Entropy images carry more discriminative structure than grayscale byte images for few-shot malware classification; the Prototypical baseline also improves when switched to entropy features.","Using a learned relation module and mean pooling of support embeddings outperforms both the Relation Network baseline and the model without data augmentation.","The architecture's accuracy is sensitive to the weight of the reconstruction loss (best at $\\lambda=0.7$) and to the linear embedding width (256 better than 128), giving concrete design guidance."],"supporting_citations":[{"why":"Supplies the Malimage dataset and the grayscale malware-image representation that the paper converts to entropy images.","marker":"[36]"},{"why":"The Prototypical Networks baseline whose accuracy and obfuscation drops the model is compared against.","marker":"[66]"},{"why":"Introduces the Relation Network few-shot idea whose learnable comparator the model adapts into its relation-aware module.","marker":"[62]"},{"why":"Shows denoising autoencoders used to counter obfuscation in traffic data, the approach the paper extends to malware entropy images.","marker":"[5]"},{"why":"Supports the claim that entropy-based features have higher uniqueness and information content, motivating entropy images as inputs.","marker":"[51]"}],"fun_headline_variants":["Relation-aware Siamese net nails few-shot malware ID","Entropy images + Siamese net: robust few-shot malware detection","Few-shot malware classifier uses relation-aware embeddings for obfuscation","Siamese autoencoder learns malware relations, beats baselines in few-shot","Relation-aware Siamese net: 96% accuracy on unseen malware with one sample"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the decoder learns to reconstruct a clean entropy image from a NOP-obfuscated one, so that the latent representation stays stable under obfuscation; only NOP insertion is tested, and the paper itself lists packing, XOR, register reassignment, and code transposition as untested future work.","fun_headline_variants_meta":{"raw":{"variants":["Relation-aware Siamese net nails few-shot malware ID","Entropy images + Siamese net: robust few-shot malware detection","Few-shot malware classifier uses relation-aware embeddings for obfuscation","Siamese autoencoder learns malware relations, beats baselines in few-shot","Relation-aware Siamese net: 96% accuracy on unseen malware with one sample"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3075,"prompt_tokens":1061,"completion_tokens":2014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":1922}},"tokens_in":677,"tokens_out":2014,"duration_ms":14129,"temperature":1.0,"reasoning_tokens":1922,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:35:12.317575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 2-way and 5-way few-shot evaluation with other obfuscation techniques, such as packing, XOR, or register reassignment, on Malimage and VUW: if accuracy drops as much as or more than the Prototypical baseline's, the anti-obfuscation claim is specific to NOP insertion rather than general. A complementary check is to train the model on obfuscated images but decode to clean images, then verify on a held-out family that the reconstructed entropy image actually diverges from the obfuscated input at the inserted NOP positions.","supporting_citations":[{"cited_title":"Prototypical networks for few-shot learning,","cited_arxiv_id":null,"evidence_quote":"The Prototypical Networks baseline whose accuracy and obfuscation drops the model is compared against."},{"cited_title":"Denoising ad- versarial autoencoder for obfuscated traffic detection and recovery,","cited_arxiv_id":null,"evidence_quote":"Shows denoising autoencoders used to counter obfuscation in traffic data, the approach the paper extends to malware entropy images."},{"cited_title":"Entropy based local binary pattern (elbp) feature extraction technique of multimodal biometrics as defence mechanism for cloud storage,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that entropy-based features have higher uniqueness and information content, motivating entropy images as inputs."}],"review_version":1}