{"id":"b3288c76-0596-4372-bbdf-86d0293d8c9e","arxiv_id":"2509.10838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across eight malware-to-image conversions and seven classifiers, grayscale images match or beat more elaborate conversions, supporting the view that conversion details are secondary to model strength.","lead":"This paper compares eight ways of turning malware files into images and tests seven machine-learning models on 17,000 samples from 17 malware families. The simplest conversion, plain grayscale, matches or beats the fancier ones, suggesting the classifier matters more than the conversion trick.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-split, single-run accuracy estimates are too noisy to support the 'conversion strategy is secondary' claim; reported between-conversion gaps are near the standard error.","rationale":"The reader's conditional verdict is appropriate. The paper's qualitative observation that several conversions cluster together and that model choice has a larger visible effect is present in the tables, but it is not quantified against the noise floor. The binomial standard error calculation makes concrete why the missing confidence intervals matter: with roughly 1,700 test samples, some headline differences among conversions are smaller than one standard error, so the central equivalence claim is not yet established. The Spiral leakage is a real, located flaw, but because Spiral is the worst-performing arm it does not overturn the positive 'several conversions are comparable' claim; it still contaminates one comparison and should be fixed or removed. No internal inconsistency was found in the core accuracy tables, and the accuracy-equals-recall property is consistent with a balanced test set. There is no basis for outright rejection, but also no basis for unconditional acceptance. The appropriate disposition remains CONDITIONAL, so the reader's verdict is unchanged.","tokens_in":19889,"tokens_out":7267,"duration_ms":68627,"concrete_test":"Run the full pipeline (8 conversions x 11 model/feature setups plus 4 baselines) on R=10 independent 80/10/10 splits with different seeds, using the same Optuna/grid protocol, and record test accuracy for each configuration. Compute per-configuration means and 95% confidence intervals, then test the central claim by comparing the between-conversion spread within each model/feature group to the between-model spread within each conversion, for instance with paired McNemar tests or a mixed-effects model. If the Grayscale-vs-Hilbert-vs-Byteclass-vs-HIT gap under XGBoost-HOG remains within the confidence interval and the model factor dominates, the claim is supported; if rankings flip across seeds, it is not. Independently, reconstruct Spiral with the Gini ranking computed only on the training split and check whether its accuracy changes materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the observed accuracy differences between conversion techniques are stable signals, not artifacts of a single split or single run. Section 4.1 fixes one 80/10/10 split with random seed 42, and Section 5.2 reports one evaluation per configuration, with no confidence intervals, repeated seeds, or significance tests. On the 1,700-sample test set, a reported accuracy of 0.75 has a binomial standard error of roughly 0.011, and several of the 'comparable' best-accuracy gaps are of this magnitude (XGBoost-HOG: Grayscale 0.7512, Hilbert 0.7424, Byteclass 0.7306, HIT 0.7200; Figure 14 and Tables A.2-A.6). Without repeated runs, one cannot separate a true conversion effect from split luck or optimizer variability. The conclusion that conversion choice is secondary also requires showing that the between-conversion spread is small relative to between-model spread with known noise, which the paper never does. Additionally, the Spiral arm in Section 3.4.7 computes its Gini-based byte ordering on all dataset samples before the split, leaking test-set information into that image construction; this invalidates Spiral as a clean comparison arm, even though Spiral happens to perform worst.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares eight malware-to-image conversion techniques (Grayscale, Byteclass, Hilbert, Entropy, HIT, Spiral, Cartesian bigrams, and Polar bigrams) across seven classifiers (KNN, MLP, SVM, XGBoost, VGG16, InceptionV3, DenseNet121), with HOG and Haralick features for the four non-CNN models. The experiments use 17,000 samples from RawMalTF (17 families, 1000 per family) and an 80/10/10 single split. The main finding is that several conversion techniques yield similar classification accuracy, with Grayscale being the best overall, and that image-based approaches generally outperform byte-histogram baselines. The paper concludes that the specific image conversion strategy is secondary to the inherent strengths of image analysis techniques.","tokens_in":20123,"tokens_out":3848,"duration_ms":34536,"significance":"If the central claim were robust, this would be a practically useful result: it would support defaulting to simple grayscale conversion and focusing research effort on classifier improvements rather than bespoke conversion schemes. The paper is honest and internally consistent: the balanced-class dataset makes accuracy equal recall in every table, which the authors correctly note, and the dataset construction is clearly described. The comparison is broad, covering 92 configurations, and the qualitative ordering of techniques is visible in the reported tables. However, the evidentiary value is currently limited by the single-split, single-run design and the Spiral construction leak, which prevent the main similarity claim from being distinguished from run-to-run noise.","major_comments":[{"comment":"The central claim that conversion techniques 'perform similarly' is not supported by the reported single-run results. Each configuration is evaluated once on a fixed 80/10/10 split (random seed 42), with no confidence intervals, repeated splits, or significance tests. On the 1,700-sample test set, the binomial standard error of a 0.75 accuracy is approximately 0.0105, so the XGBoost/HOG gap between Grayscale (0.7512) and Hilbert (0.7424) is 0.0088—less than one standard error—and the Grayscale-to-HIT gap of 0.0312 is only about three standard errors. Without repeated trials or interval estimates, the observed similarity could equally be an artifact of underpowered measurement. The paper should report mean accuracies over multiple seeds or folds, or at least provide confidence intervals, before claiming that conversion strategy is a secondary factor.","section":"§4.1, §5.2, Fig. 14"},{"comment":"The Spiral image construction is not a clean comparison arm because the byte-ordering is fitted on all dataset samples before the split. Section 3.4.7 states that Random Forest Gini importances are computed 'for all dataset samples,' which means test-set information influences the image representation used for evaluation. This leakage invalidates the Spiral accuracy numbers as an unbiased estimate, and because Spiral is the worst-performing conversion in Figures 11–14, the related claim that baseline histogram features underperform all image-based conversions except Spiral is also suspect. The authors should recompute the Gini ordering within each training fold only (or use nested cross-validation) and report whether the Spiral results change.","section":"§3.4.7, §5.2"},{"comment":"The concluding assertion that 'the simplest (i.e., Grayscale) yielded the best results' is stronger than the evidence permits. In the best-performing model class (XGBoost with HOG features), Grayscale is not statistically separated from Hilbert or Byteclass given the single-run measurement noise discussed above. The authors should temper this claim or support it with inferential statistics showing that the top conversions are genuinely ordered.","section":"§6, Conclusion"}],"minor_comments":[{"comment":"The phrase 'successful applied' should read 'successfully applied.'","section":"§3.2.5, InceptionV3 paragraph"},{"comment":"The sentence 'Each value in this two-dimensional array represent the luminosity of an image pixel' should be 'Each value ... represents the luminosity ...'","section":"§3.4.1"},{"comment":"The entropy construction is underspecified: the text says the entropy value at position i uses a block B=(b_i,...,b_{i+n}) with n=min{255,N-i}, but then states that the result is an array of entropy values x_0,...,x_255. Clarify how the number of window positions relates to the 224×224 canvas and how the block length n is used in the sum.","section":"§3.4.4"},{"comment":"The phrase 'we consider each byte position as a feature' should be 'each byte value as a feature' to avoid confusion with offsets in the executable file.","section":"§3.4.7"},{"comment":"The train:test:validation order is given inconsistently (once as 'train:test:validation' and once as 'train:validation:test'). Also, the statement that accuracy equals recall because the dataset is balanced could explicitly note that the test set is balanced by construction (1000 samples per family).","section":"§4.1, §4.3"},{"comment":"The confusion matrices in Figures B.1–B.8 are very dense and difficult to read at the printed size; consider enlarging or providing a table of per-family recall values alongside them.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical comparison and the topic is within the journal's scope. My main concern is that the single-split, single-run design cannot support the paper's headline claim without uncertainty quantification. The Spiral leakage is a clear methodological flaw, but it is fixable by refitting the byte ordering on training folds. I would not reject the manuscript, but I would require a revised version with repeated runs or confidence intervals and a corrected Spiral evaluation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful benchmark—to my reading, the first systematic side-by-side of eight image conversion techniques on a common 17-family dataset with seven models. The headline claim, that conversion choice is secondary to model choice, is directionally supported by the numbers, but it is stated more strongly than the evidence allows. The paper deserves a serious referee, but it needs repeated runs, a fix for the Spiral leakage, and a more measured conclusion.\n\nWhat's new: the individual techniques are not new, but the systematic comparison is. The authors use a modern dataset (RawMalTF), a balanced 17-family subset, and test grayscale, HIT, entropy, byteclass, Hilbert, spiral, Cartesian, and polar bigrams, plus baseline histogram features. That is a lot of work, and the qualitative conclusion is visible in the tables: model choice moves accuracy by 0.15–0.2, while conversion choice (excluding Spiral) moves it by 0.06–0.09. The claim that several conversions perform comparably is plausible.\n\nSoft spots:\n1. No uncertainty quantification. Every configuration is evaluated once on one 80/10/10 split with seed 42. On a 1,700-sample test set, the standard error of a 0.75 accuracy is roughly 0.011, and several between-conversion gaps are that size. For example, XGBoost-HOG gives Grayscale 0.7512 and Hilbert 0.7424—a gap of 0.0088. The paper reports no repeated seeds, confidence intervals, or significance tests, so we cannot tell whether the ranking of conversions is stable. This does not destroy the main claim, but it undercuts the specific claim that Grayscale is 'best' and that several techniques are truly equivalent.\n2. Spiral data leakage. Section 3.4.7 says the Gini-based byte ordering is computed on 'all dataset samples' before the split. That leaks test-set information into the Spiral image construction, invalidating Spiral as a clean comparison arm. It happens to perform worst, so the leakage likely did not inflate the headline conclusion, but it is a methodological error that should be fixed or acknowledged.\n3. No code is provided, which makes the single-run results harder to verify.\n4. The conclusion that 'image analysis techniques themselves are the key' is a leap. The data show that for this dataset and these models, conversion matters less than model choice. That is useful, but not a general principle.\n\nWho it's for: researchers working on image-based malware classification, especially those choosing a conversion method or benchmarking new ones. It is a practical reference, not a theoretical breakthrough.\n\nRecommendation: send it to peer review. A good referee should ask for repeated runs (or at least a variance estimate), a fix for the Spiral leakage, and a more measured conclusion. The core comparison is worth publishing.","headline":"A useful but statistically thin benchmark; the main claim is plausible but the paper overstates it.","tokens_in":20671,"tokens_out":4456,"would_cite":true,"duration_ms":35360,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Eight malware-to-image conversions perform about equally, so the classifier matters more than the conversion.","keywords":["image-based malware classification","malware-to-image conversion","grayscale conversion","HIT image transformation","entropy images","byte bigram visualization","RawMalTF dataset","transfer learning"],"falsifier":"Re-run the 92 experiments under ten different random seeds and compare the per-configuration accuracy ranges; if for any classifier grayscale and the other top conversions are not separated beyond the run-to-run spread, the paper's core claim is supported, whereas stable, non-overlapping gaps would show that conversion choice does matter.","tokens_in":19668,"feed_emoji":"🦠","tokens_out":11395,"duration_ms":85470,"temperature":0.7,"pith_summary":"This paper asks whether the way an executable is turned into an image matters for malware-family classification. The authors test eight conversion techniques—from plain grayscale to entropy maps, Hilbert curves, byte bigrams, and a hybrid HIT encoding—across seven classifiers, using 17,000 samples from 17 families. They report that several very different conversions achieve roughly comparable accuracy, with the simplest (grayscale) giving the best results overall. The paper concludes that image-based malware classification mostly owes its success to the analysis technique itself, not to the particular image conversion chosen. If this conclusion holds, researchers can safely default to a cheap grayscale conversion and spend effort on classifiers and features.","feed_headline":"Fancy malware-image tricks don't beat plain grayscale","feed_subtitle":"Across 8 conversions and 7 classifiers, the simplest encoding matched or beat elaborate alternatives.","key_machinery":"The load-bearing device is a standardized comparison: every conversion takes the same first 50,176 bytes of each of 17,000 samples, produces a 224×224 image, and is evaluated under the same 80:10:10 train-validation-test split with the same Optuna hyperparameter search (random seed 42). This uniformity isolates the conversion method as the only variable, and the box-and-whisker plots over the eleven model/feature combinations per conversion are what support the claim of near-equivalence. Within this apparatus, the conversion techniques themselves—grayscale, Byteclass, Hilbert, Entropy, HIT, Spiral, Cartesian bigrams, and Polar bigrams—are the objects being compared.","core_discovery":"On the authors' own terms, the discovery is that eight substantially different executable-to-image conversion strategies produce overlapping accuracy ranges across eleven model/feature configurations, and the simplest one, grayscale, is the best or near-best in nearly every setting. The strongest single result is XGBoost with HOG features on grayscale images at 0.7512 accuracy, followed by Hilbert at 0.7424, Byteclass at 0.7306, and HIT at 0.7200, while the best baseline using byte histograms reaches only 0.6906. All image-based conversions except Spiral beat the histogram baseline, which the authors take as evidence that the image-based pipeline carries the discriminative signal. Because the conversions differ so much in how they lay out bytes, the authors argue the comparable results imply that the image analysis techniques themselves, rather than the conversion detail, are what drive the success.","pith_inferences":["One testable extension the authors do not run: randomly permute the byte-to-pixel mapping so local neighborhoods are destroyed, and retrain VGG16; if accuracy collapses, the conversion layout still contributes signal, and if accuracy holds, the image-ness itself is secondary in a way the paper does not isolate.","Because Spiral is the only conversion built from histogram features and the only one that underperforms the histogram baseline, a per-family error analysis could reveal whether the spiral ordering hides or destroys the discriminative byte-value patterns.","A practical corollary the authors leave implicit: ensemble classifiers over multiple conversions (for example, concatenating HOG features from grayscale and Hilbert images) may add little if the representations are redundant; testing this would quantify how much unique information each conversion carries."],"forward_implications":["A simple grayscale conversion can serve as a default for image-based malware classification, since it matched or beat all other tested conversions in accuracy.","Image-based representations outperform byte-histogram baselines across nearly all conversions, so the spatial arrangement of bytes carries information that histograms discard.","Transfer-learning models are not all equal on this task: VGG16 consistently outperformed InceptionV3 and DenseNet121 across every conversion technique.","Because conversion choice is secondary, the wide variety of ad hoc malware-to-image schemes in the literature can be treated as mostly interchangeable, and effort is better spent on classifiers and features.","The histogram-based Spiral images underperform, showing that not every image conversion is equally useful and that basing an image on histogram features forfeits the advantage."],"supporting_citations":[{"why":"Supplies the RawMal-TF dataset with the 17 malware families and the byte histograms used throughout the experiments.","marker":"[5]"},{"why":"Introduced the original grayscale malware-image approach that the Grayscale conversion represents.","marker":"[26]"},{"why":"Provides the Byteclass encoding, the entropy color-channel formulas, and the HIT hybrid image used in the comparison.","marker":"[42]"},{"why":"Contributes the entropy-based image representation that the Entropy conversion draws on.","marker":"[43]"},{"why":"Describes the Cartesian and Polar bigram visualizations used as two of the eight conversions.","marker":"[41]"},{"why":"Inspires the Spiral visualization from a different problem domain that is adapted for malware images.","marker":"[33]"},{"why":"The hilbertcurve library used to rasterize Hilbert and Entropy images onto the 224x224 canvas.","marker":"[17]"},{"why":"Optuna performs the hyperparameter search for the non-CNN models, so all conversion techniques are tuned under the same procedure.","marker":"[2]"}],"fun_headline_variants":["Malware imaging: grayscale matches complex conversion tricks","Eight image conversions, grayscale still the best for malware","Image-based malware: conversion method barely matters","Simple grayscale holds its own against fancy malware images","Malware classification: conversion details less important than analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy numbers, each measured once on a single random 80/10/10 split with one seed, are stable enough that the differences between conversion techniques are real and not just run-to-run noise.","fun_headline_variants_meta":{"raw":{"variants":["Malware imaging: grayscale matches complex conversion tricks","Eight image conversions, grayscale still the best for malware","Image-based malware: conversion method barely matters","Simple grayscale holds its own against fancy malware images","Malware classification: conversion details less important than analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1514,"prompt_tokens":874,"completion_tokens":640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":565}},"tokens_in":490,"tokens_out":640,"duration_ms":5760,"temperature":1.0,"reasoning_tokens":565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:52:09.421243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 92 experiments under ten different random seeds and compare the per-configuration accuracy ranges; if for any classifier grayscale and the other top conversions are not separated beyond the run-to-run spread, the paper's core claim is supported, whereas stable, non-overlapping gaps would show that conversion choice does matter.","supporting_citations":[{"cited_title":"Nataraj, S","cited_arxiv_id":null,"evidence_quote":"Introduced the original grayscale malware-image approach that the Grayscale conversion represents."},{"cited_title":"Nguyen, Tu N","cited_arxiv_id":null,"evidence_quote":"Provides the Byteclass encoding, the entropy color-channel formulas, and the HIT hybrid image used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the entropy-based image representation that the Entropy conversion draws on."},{"cited_title":"Simple binary data visualization.https://martin.varela","cited_arxiv_id":null,"evidence_quote":"Describes the Cartesian and Polar bigram visualizations used as two of the eight conversions."},{"cited_title":"Darknet traffic clas- sification and adversarial attacks using machine learning.Computers & Security, 127:103098, 2023","cited_arxiv_id":null,"evidence_quote":"Inspires the Spiral visualization from a different problem domain that is adapted for malware images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The hilbertcurve library used to rasterize Hilbert and Entropy images onto the 224x224 canvas."},{"cited_title":"Optuna: A next-generation hyperparameter optimization frame- work","cited_arxiv_id":null,"evidence_quote":"Optuna performs the hyperparameter search for the non-CNN models, so all conversion techniques are tuned under the same procedure."}],"review_version":2}