{"id":"e8eb6f51-8bbe-4d08-b6a1-637da0599b7b","arxiv_id":"2412.18932","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Converting HMM hidden-state sequences into images and classifying them with a CNN yields 0.9781 accuracy on a 7-family Malicia subset, a 0.0023 gain over the authors' HMM-RF baseline.","lead":"The paper trains one Hidden Markov Model per malware family, converts each sample's hidden-state sequences into 224x224 images, and classifies them with a CNN. It reports 0.9781 accuracy on a seven-family Malicia subset, slightly above its HMM-Random Forest baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed superiority over HMM-RF rests on a single 80/20 split with grid-search hyperparameters selected on that same split; a 0.0023 accuracy margin is within expected split-to-split noise.","rationale":"The reader identified the same load-bearing weakness: single-split evaluation with hyperparameter selection on the test set and no significance testing. My reading of the paper confirms this; Section 5.2 and Table 4 show grid-search selection directly on the split used for Table 5, and the gap over HMM-RF is tiny relative to sample count and class imbalance. I do not see reason to change the CONDITIONAL verdict: the pipeline is coherent and the direction is plausible, but the headline superiority claim requires a repeated-split protocol and significance test. I do not escalate to REJECT because the contribution is an empirical extension and the methodology could be validated with additional rigor.","tokens_in":11748,"tokens_out":3178,"duration_ms":28405,"concrete_test":"Run the identical HMM-CNN and HMM-RF pipelines on 10 random 80/20 splits (or 5-fold cross-validation). For each split, select hyperparameters using only the training fold (nested grid search or a separate validation split), then report mean and standard deviation of accuracy/F1 on the held-out split and a McNemar paired test between HMM-CNN and HMM-RF per split. If the accuracy advantage is not consistently positive, or p>0.05 in most splits, the claimed superiority is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that HMM-CNN significantly outperforms HMM-RF, is supported only by Table 5, where HMM-CNN accuracy is 0.9781 vs HMM-RF 0.9758 and F1 is 0.9778 vs 0.9732. Section 4.2 fixes one 80:20 train/test split, and Section 5.2 then performs a grid search over L, base model, optimizer, learning rate, and loss, selecting the configuration with the best accuracy on that same held-out split. Thus 0.9781 is a maximum over a large hyperparameter grid evaluated on the test set, not an unbiased estimate of generalization; the baselines are not subjected to an equally strong selection protocol. Given the class imbalance (Winwebsec alone accounts for 4360 of about 8054 samples, while Cridex has 74), accuracy is dominated by a few families, and the margin over HMM-RF is 0.23 percentage points. With roughly 1,600 test samples, that difference corresponds to about 4 samples, well within paired-test noise unless a paired significance test is reported. The paper provides no variance estimate, no repeated splits, and no McNemar or similar test. Therefore the abstract's 'superior performance' claim is not established by the experiments as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HMM-CNN, a hybrid malware family classifier that combines Hidden Markov Models with Convolutional Neural Networks. The method trains one HMM per malware family on opcode sequences, extracts hidden-state sequences of length L for each sample under each family-specific HMM, concatenates them into a vector of length 7L, reshapes and zero-pads this vector into a 224x224 image, and classifies the image with a CNN built on a pretrained base model. Experiments are conducted on a seven-family subset of the Malicia dataset with a single 80:20 train-test split. The paper reports HMM-CNN validation accuracy 0.9781 and F1 0.9778, slightly above the authors' prior HMM-Random Forest model (0.9758 and 0.9732), and concludes that HMM-CNN achieves superior performance. Training and testing times for all compared techniques are also reported.","tokens_in":12030,"tokens_out":4891,"duration_ms":42892,"significance":"If the reported result were statistically robust, the contribution would be a modest but potentially useful engineering insight: HMM hidden-state sequences can be embedded as images and classified by a CNN, extending the authors' earlier HMM-RF work. The paper is clearly written and provides useful detail on the dataset, preprocessing, hardware, and hyperparameter candidates, which aids reproducibility. However, the central claim of superiority rests on a single data split and a hyperparameter configuration selected by accuracy on that same held-out partition, with no error bars, repeated splits, or significance tests. The reported margin over HMM-RF is 0.0023 in accuracy, which corresponds to roughly four test samples and is well within plausible split-to-split noise given the class imbalance. As presented, the evidence does not establish the paper's main claim.","major_comments":[{"comment":"The entire comparison rests on one fixed 80:20 split (Section 4.2), and the grid search in Section 5.2 selects hyperparameters by accuracy on that same held-out partition. The reported 0.9781 is therefore a selected maximum over the grid, not an unbiased estimate of generalization. With about 1,611 test samples, the 0.0023 accuracy gap over HMM-RF corresponds to approximately four samples, which is within plausible split-to-split noise unless a paired test is reported. The manuscript provides no variance estimate, no repeated splits, and no McNemar or similar significance test. Please report repeated stratified cross-validation or multiple splits with error bars and a paired significance test, and state explicitly whether hyperparameters were chosen on a separate validation set rather than the final test set.","section":"Section 4.2 and Section 5.2, Table 5"},{"comment":"The hyperparameter search is internally inconsistent: Section 5.1 states that L was tested over {25, 50, 100, 200}, while Table 4 lists L over {56, 112, 224}. This makes the exact selected configuration and the search protocol unreproducible. Please reconcile these values and state exactly which L values were tested, which value was selected, and on which data partition the selection was made.","section":"Section 5.1 and Table 4"},{"comment":"For the baseline models (Word2Vec-LSTM, BERT-LSTM, Random Forest, SVM, HMM-RF, HMM-SVM, CNN), the manuscript does not report hyperparameter values or whether any tuning was performed. If HMM-CNN received a grid search while the baselines used default settings, the comparison in Table 5 is confounded by tuning effort. This is critical because the reported margins are small. Please specify the baseline configurations and, ideally, give the baselines the same tuning budget or otherwise justify that the comparison is fair.","section":"Section 5.3, Table 5"}],"minor_comments":[{"comment":"The phrase 'poses a significant challenges' is ungrammatical; use 'significant challenges' or 'a significant challenge'.","section":"Abstract"},{"comment":"The column heading 'Validation Accuracy' conflicts with the '80:20 train-test split' terminology in Section 4.2; use consistent 'test' terminology throughout.","section":"Section 5.3, Table 5"},{"comment":"The sentence 'The accuracy obtained for the best choice of hyperparameters in Table 4 was 0.9781' does not state whether this is the test accuracy from the single split; please clarify this point explicitly.","section":"Section 5.2"},{"comment":"The three subfigures are said to show 'Accuracy trends for different hyperparameters,' but the text does not specify how each hyperparameter is varied while holding others fixed, nor whether these accuracies are computed on the test split; add a sentence describing the protocol.","section":"Figure 5"},{"comment":"The statement 'this implies an HMM is an Expectation Maximization (EM) technique' is imprecise; Baum-Welch is an EM algorithm used for HMM training, not the HMM itself.","section":"Section 2.1"},{"comment":"Reference [40] contains the URL 'https://www.microsoft.com/enus/wdsi/...'; the path should be 'en-us'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is an incremental empirical study whose main claimed gain over the authors' own prior HMM-RF model is very small. The blocking issue is statistical: a single split, hyperparameter selection on the test set, and no significance testing. I believe this is fixable within the scope of the manuscript by adding repeated cross-validation or multiple splits, a separate validation set for hyperparameter selection, and paired significance tests, so I recommend major revision rather than rejection. The authors should also reconcile the inconsistent L values in Section 5.1 and Table 4, and disclose the baseline configurations to rule out tuning-effort confounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: the paper replaces the random forest in the authors' HMM-RF model with a CNN and reports a hair better accuracy on a well-known malware benchmark. The claimed superiority is not established by the experiments as reported, because the entire comparison rests on one 80:20 split with the hyperparameters selected on that same test set.\n\nWhat's genuinely new is the specific HMM-CNN combination: HMM hidden-state sequences from seven family HMMs are concatenated, reshaped into a square, and padded to 224x224, then fed to a pretrained CNN. That exact pairing doesn't appear in the cited prior work. The paper is clearly written, the pipeline is described in enough detail to reproduce, and the baseline set is reasonable. The authors also report training and testing times, which is useful.\n\nThe soft spots are concentrated in the evaluation. The grid search in Table 4 picks L, base model, optimizer, learning rate, and loss by maximizing accuracy on the same split that produced Table 5. The 0.9781 is a selected maximum, not an unbiased estimate. The margin over HMM-RF is 0.0023 accuracy and 0.0046 F1, which, on roughly 1,600 test samples, is about four samples. No significance test, no repeated splits, no confidence intervals. The class imbalance (Winwebsec alone has 4,360 of ~8,054 samples; Cridex has 74) makes the aggregate numbers dominated by the large families; Cridex accuracy is 0.200. So the abstract's 'superior performance' overstates what the data show.\n\nOne minor contradiction: the introduction says the HMM-hidden-state-as-feature approach has not been used in the malware domain, but the authors' own HMM-RF paper [23] does exactly that with a different classifier. The sentence should be narrowed to the CNN combination. No code or data is released, which also limits verification.\n\nWho this is for: anyone working on opcode-based malware classification, and people interested in evaluation pitfalls in applied ML security papers. The core idea may have legs, but the current evidence doesn't support the headline claim.\n\nRecommendation: deserve a serious referee. The method is plausible and the description is honest, but the evaluation needs a major revision: multiple splits or cross-validation, paired significance testing (e.g., McNemar), and released artifacts. I'd accept it for review with the expectation of heavy revision.","headline":"The HMM-CNN idea is plausible but the reported 0.23 percentage point edge over HMM-RF is within split noise, so the paper's central claim is not yet supported.","tokens_in":12586,"tokens_out":2489,"would_cite":false,"duration_ms":20722,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that turning HMM hidden-state sequences into 224×224 images and classifying them with a CNN outperforms eight compared malware classifiers on a seven-family Malicia subset.","keywords":["malware classification","Hidden Markov Model","convolutional neural network","opcode sequences","Malicia dataset","feature engineering","image-based malware detection","transfer learning"],"falsifier":"Repeat the comparison with 5-fold or 10-fold cross-validation, or with multiple random 80/20 splits, and report per-family accuracy; if HMM-CNN's mean accuracy does not consistently exceed HMM-RF's, or a held-out split reverses the ranking, the central claim is refuted.","tokens_in":11517,"feed_emoji":"🦠","tokens_out":6077,"duration_ms":49384,"temperature":0.7,"pith_summary":"This paper is trying to establish that a hybrid feature-engineering pipeline beats both pure sequence models and earlier HMM hybrids for malware family classification. The pipeline trains one Hidden Markov Model per malware family on disassembled opcode sequences, decodes each sample's hidden state sequence with every trained HMM, concatenates the seven sequences, reshapes the scaled vector into a square, zero-pads it to 224×224, and feeds that image to a CNN. On a seven-family subset of the Malicia dataset the method reports validation accuracy 0.9781 and weighted F1 0.9778, edging out the HMM-Random Forest baseline (0.9758, 0.9732) it was designed to surpass. If the result holds, it would mean that HMM hidden states are a genuinely useful feature representation for static malware classification, not just for the random-forest classifier used in prior work.","feed_headline":"HMM hidden states as images beat malware classifiers","feed_subtitle":"On seven Malicia families, the hybrid hits 0.9781 accuracy, edging out HMM-Random Forest and six other baselines.","key_machinery":"The load-bearing object is the HMM hidden state sequence as an engineered image. For each of the seven families, an HMM with $N=20$ hidden states and $M=426$ observation symbols is trained on concatenated mnemonics; then for a sample, the first $L$ opcodes are scored against each of the seven trained HMMs to produce seven length-$L$ hidden state sequences, which are concatenated into a length-$7L$ vector. This vector is standardized, rearranged into a $\\lceil\\sqrt{7L}\\rceil \\times \\lceil\\sqrt{7L}\\rceil$ matrix, and zero-padded to 224×224 to form the CNN input. The CNN is a pretrained base architecture (ResNet50V2, ResNet101V2, ResNet152V2, DenseNet201, or Xception) with a global average pooling layer, a 1024-unit ReLU dense layer, and a softmax output. The mechanism's work is to turn sequential opcode statistics into a spatial representation that a vision model can separate, and the paper's evidence for this is the accuracy comparison with models that skip the HMM step.","core_discovery":"The central claim is that the hidden state sequences of per-family HMMs, when reorganized as images, carry enough discriminative signal for a CNN to classify malware families better than the alternatives tested on the same data. The authors call this HMM-CNN and report it as the best row in their comparison: validation accuracy 0.9781 and weighted F1 0.9778, versus 0.9758/0.9732 for HMM-RF and 0.9725/0.9727 for a plain CNN trained on opcode-derived images. They also observe that HMM-based features help on a rare family: SecurityShield is classified at 87.5% average accuracy by HMM-based models versus 41.67% by non-HMM models, while the smallest family Cridex remains hard, at 0.200 accuracy in the HMM-CNN confusion matrix.","pith_inferences":["The paper's headline comparison rests on a gap of 0.0023 in accuracy and 0.0046 in F1; a reasonable editorial inference is that this margin is within the noise of a single 80/20 split, and repeated splitting or cross-validation could erase it.","Because the hidden states are categorical labels rather than spatial gradients, the 224×224 image is not a natural image; one testable inference is that a simple classifier on the same 7L hidden-state vector might match the CNN, which would mean the gain comes from the HMM representation rather than the CNN.","The weighted F1 is dominated by the largest families; with Cridex at 0.200 accuracy and only 74 samples, a macro-averaged or per-family comparison would likely show smaller or reversed differences between methods.","A direct ablation the paper does not run—HMM features with a linear model versus HMM features with the CNN—would isolate how much of the gain is due to the CNN and how much to the hidden-state embedding."],"forward_implications":["If the reported numbers hold, HMM hidden states are a reusable static feature representation: they improve accuracy over feeding CNNs raw opcode-derived images (0.9781 vs 0.9725) and over HMM features with a random forest or SVM (0.9781 vs 0.9758 and 0.9757).","The approach inherits CNN test-time efficiency: per-sample testing is 0.0076 seconds, comparable to a plain CNN, despite 48.83 hours of training.","The HMM feature step appears to help rare-class discrimination: SecurityShield samples are classified at 87.5% average accuracy by HMM-based models versus 41.67% by non-HMM models.","The authors suggest the same NLP-inspired feature-engineering recipe could transfer to other sequence-classification domains and to obfuscated malware; those are extensions of the method, not yet demonstrated results."],"supporting_citations":[{"why":"Supplies the HMM-RF model and the seven-family Malicia subset that HMM-CNN is compared against.","marker":"[23]"},{"why":"Provides the Malicia dataset, its family labels, and the preprocessing context for all experiments.","marker":"[26]"},{"why":"Provides the HMM training and hidden-state decoding algorithms (forward-backward and Baum-Welch) on which feature extraction rests.","marker":"[38]"},{"why":"Defines the HMM formalism $\\lambda=(A,B,\\pi)$ used to represent each malware family.","marker":"[11]"},{"why":"Supplies the ResNet family of base CNN architectures tested in the grid search.","marker":"[13]"},{"why":"Supplies DenseNet201, another base architecture whose accuracy is reported.","marker":"[14]"},{"why":"Supplies the Xception base architecture, one of the alternatives in the hyperparameter search.","marker":"[7]"}],"fun_headline_variants":["HMM images sharpen CNN malware detection","Hidden-state images push malware scores past baselines","Hybrid HMM-CNN bests seven malware classifiers","From opcodes to images: HMM-CNN tops malware tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a single 80:20 train/test split, with the grid search selecting hyperparameters on the same test split, yields a stable and unbiased accuracy estimate for a heavily imbalanced dataset.","fun_headline_variants_meta":{"raw":{"variants":["HMM images sharpen CNN malware detection","Hidden-state images push malware scores past baselines","Hybrid HMM-CNN bests seven malware classifiers","From opcodes to images: HMM-CNN tops malware tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1600,"prompt_tokens":896,"completion_tokens":704,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":641}},"tokens_in":512,"tokens_out":704,"duration_ms":6693,"temperature":1.0,"reasoning_tokens":641,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:19:24.861952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the comparison with 5-fold or 10-fold cross-validation, or with multiple random 80/20 splits, and report per-family accuracy; if HMM-CNN's mean accuracy does not consistently exceed HMM-RF's, or a held-out split reverses the ranking, the central claim is refuted.","supporting_citations":[{"cited_title":"A natural language pro- cessing approach to malware classification","cited_arxiv_id":null,"evidence_quote":"Supplies the HMM-RF model and the seven-family Malicia subset that HMM-CNN is compared against."},{"cited_title":"Zubair Rafique, and Juan Caballero","cited_arxiv_id":null,"evidence_quote":"Provides the Malicia dataset, its family labels, and the preprocessing context for all experiments."},{"cited_title":"A revealing introduction to hidden Markov models","cited_arxiv_id":null,"evidence_quote":"Provides the HMM training and hidden-state decoding algorithms (forward-backward and Baum-Welch) on which feature extraction rests."},{"cited_title":"Hidden Markov models","cited_arxiv_id":null,"evidence_quote":"Defines the HMM formalism $\\lambda=(A,B,\\pi)$ used to represent each malware family."},{"cited_title":"Identity map- pings in deep residual networks","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet family of base CNN architectures tested in the grid search."},{"cited_title":"Densely connected convolutional networks","cited_arxiv_id":null,"evidence_quote":"Supplies DenseNet201, another base architecture whose accuracy is reported."},{"cited_title":"Xception: Deep learning with depthwise separable convo- lutions","cited_arxiv_id":null,"evidence_quote":"Supplies the Xception base architecture, one of the alternatives in the hyperparameter search."}],"review_version":1}