{"id":"8970b542-75f3-4575-a2a1-623f6ce8fe19","arxiv_id":"2505.18613","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new open behavioural ransomware dataset with 64 families and balanced goodware, plus guidelines and a feature selection pipeline that reaches about 98% binary detection accuracy.","lead":"This paper introduces MLRan, a behavioural dataset of 4,880 Windows samples, 2,330 ransomware across 64 families and 2,550 goodware, collected with Cuckoo Sandbox. It also proposes dataset construction guidelines and reports that a logistic regression model with 483 features reaches about 98% binary detection accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98% detection figures rest on feature-selection choices (MI threshold 0.01, RFE 2% / 483 features) made by evaluating on the test set (Table 5, Figure 9, Algorithm 2), violating the no-test-snooping rule the paper itself states in Section 3.4.3; the performance claim is therefore not yet…","rationale":"The paper's contribution is twofold: a new dataset and an empirical demonstration of its utility. The dataset is large, balanced, spans 2006-2024, covers 64 families, and is publicly released with code—these are real strengths and I am not questioning them. The weakest link is the empirical claim that simple models reach ~98% accuracy on this dataset. That claim is load-bearing because it is the stated evidence that the dataset is 'well-suited for building effective ransomware detection models' (Section 1) and it is repeated in the abstract and results. The experimental protocol in Sections 5.5, 6.1, and 6.2 selects feature-selection hyperparameters (the MI threshold and the RFE feature count) by evaluating model performance on the test set. This is exactly the test-snooping bias the authors themselves warn against in Section 3.4.3, citing Arp et al. (2022). Because the test set informed these choices, the reported 98.15% accuracy is an optimistic estimate of how the model would perform on genuinely unseen data. The abstract's higher numbers (98.7% accuracy, 98.9% precision, 98.5% recall) do not match any table in the paper, compounding the concern. The fix is straightforward: reserve the test set, use a validation split derived from the training set for hyperparameter and feature-count selection, and report the single test-set evaluation. Until that is done, the central performance claim should be treated as conditional. I therefore keep the reader's CONDITIONAL verdict; no further adjustment is needed.","tokens_in":44129,"tokens_out":5271,"duration_ms":38493,"concrete_test":"Re-run the MLRan pipeline with a strict protocol: hold out the existing 976-sample test set completely; from the 3,904 training samples create a time-ordered validation split (e.g., earliest 80% of training for fitting, latest 20% for selection). On the validation split, choose the MI threshold and the RFE percentage using only validation balanced accuracy, then retrain on the full training set with the chosen settings and evaluate once on the untouched test set. If the resulting accuracy falls below ~97% (or drops by >1 point from 98.15%), the reported performance claim is not supported and should be re-reported with confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that logistic regression on 483 features yields ~98% accuracy/precision/recall—depends on the feature-selection pipeline being independent of the test set. The paper states that a time-aware 80:20 split is used and that test data should only be used for final evaluation (Section 3.4.3), yet the results show otherwise. In Section 6.1 / Table 5, three filter methods (MI 0.01, MI 0.001, Chi-Square) are compared on the test set and MI(0.01) is chosen because it yields the best trade-off, with performance figures reported on test. In Section 6.2 / Algorithm 2, the RFE feature-count grid (1%–90% of 24,162 features) is evaluated by fitting on the training set, transforming the test set (Lines 15–16), and recording balanced accuracy on the test set (Lines 19–21); Figure 9 then selects the 2% (483-feature) configuration with the highest test balanced accuracy (98.07%). This is test-set snooping as defined in the paper itself: test data is used to select both the MI threshold and the RFE feature count. Consequently, the 98.15% accuracy / 98.18% balanced accuracy in Table 6 (and the abstract's 98.7%/98.9%/98.5%) are optimistic estimates; the magnitude of the bias is unknown but the design does not justify the headline number. The abstract numbers also do not match any table entry, further weakening the reporting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MLRan, a behavioural ransomware dataset of 4,880 Windows samples (2,330 ransomware samples from 64 families spanning locker, crypto, RaaS, and modern types over 2006-2024, plus 2,550 goodware samples from 11 categories). It proposes GUIDE-MLRan, a set of fifteen criteria for constructing and documenting behavioural ransomware datasets, and describes an open-source pipeline for sample collection, Cuckoo Sandbox analysis, feature extraction, and ML evaluation. The authors evaluate several classifiers under a two-stage feature-selection scheme (mutual information filtering followed by recursive feature elimination, reducing 6.4M features to 483) and report binary detection accuracy, precision, and recall up to 98.7%, 98.9%, and 98.5%, together with SHAP/LIME analyses and a misclassification study.","tokens_in":44499,"tokens_out":6287,"duration_ms":50536,"significance":"The dataset itself is a potentially valuable community resource: it is larger and more balanced than most prior behavioural ransomware datasets, spans a long temporal range, covers four ransomware types, provides metadata, and ships public code for collection, parsing, feature selection, and training. GUIDE-MLRan is a useful systematisation of dataset-construction requirements, and the paper is transparent about sandbox hardening and labelling. However, the quantitative ML claims are not currently supported because the feature-selection thresholds and feature count are chosen using test-set performance, contrary to the paper's own no-test-snooping protocol. The headline figures must be re-estimated before the paper can be accepted.","major_comments":[{"comment":"The evaluation protocol uses the test set to make feature-selection decisions. Section 6.1 compares MI(0.01), MI(0.001), and Chi-Square on test-set metrics and selects MI(0.01); Section 6.2, Algorithm 2, Lines 15-21 transforms X_test and computes balanced accuracy on y_test for each RFE feature count, and Figure 9 selects the 2% (483-feature) configuration. This is precisely the \"test snooping\" the paper itself prohibits in Section 3.4.3, where it states that test data should be used only for final evaluation. The binary results in Table 6 are therefore optimistic and do not measure generalization to unseen data. The authors should re-run feature selection inside the training folds (nested cross-validation or a separate validation split), freeze thresholds and feature counts before touching the test set, and then report one final test evaluation.","section":"Section 6.1 (Table 5) and Section 6.2 (Algorithm 2, Figure 9) in light of Section 3.4.3"},{"comment":"The abstract and introduction report \"accuracy, precision and recall of up to 98.7%, 98.9%, 98.5%,\" but no row in Table 5 or Table 6 attains these values; the closest binary logistic-regression row is 98.15% / 98.16% / 98.15% (Table 6). This mismatch weakens the headline claim. The authors should either identify the exact configuration producing the abstract numbers or correct the reported figures to those in the tables.","section":"Abstract and Section 1 vs. Tables 5 and 6"},{"comment":"The paper claims that the two-stage feature selection reduced the feature space from over 6.4 million to 483 \"without accuracy loss.\" Because both the MI threshold and the RFE feature count were selected using test-set performance, there is currently no valid evidence for this claim; the comparison between Table 5 MI(0.01) (logistic regression accuracy 97.54%) and Table 6 after RFE (98.15%) is not a fair evaluation. A nested cross-validation comparison with confidence intervals is needed before the efficiency claim can be assessed.","section":"Section 6.2, Result 2"}],"minor_comments":[{"comment":"The ShieldFS row appears to have the #Ran and #Good columns swapped relative to the text in Section 2, which states 383 ransomware samples and 2,245 goodware applications.","section":"Table 1, ShieldFS row"},{"comment":"The first model is labelled \"XGBoost\" in the tables, while Section 5.3.1 describes LightGBM; the terminology should be aligned.","section":"Tables 5 and 6"},{"comment":"Feature counts are computed from |X_train|, but n_features_to_select is a feature count; the notation should be clarified to refer to the number of features, not the number of training samples.","section":"Algorithm 2, Line 6"},{"comment":"The text says \"improving model efficiency and interoperability\"; \"interpretability\" is presumably intended.","section":"Section 6.2, Result 2"},{"comment":"The hyperparameter grids are listed, but the paper does not state whether hyperparameters were tuned on a validation split or with cross-validation; this should be documented to support the no-snooping claim.","section":"Table 9"},{"comment":"The t-SNE visualisations do not report the perplexity or state whether the projection was fitted on training data only, which is needed for reproducibility.","section":"Figures 13 and 14"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is confirmed by the manuscript itself: Algorithm 2 openly computes test-set balanced accuracy to choose the RFE feature count, and Section 6.1 uses test-set results to select the MI threshold. This is fixable by rerunning the feature-selection and model-selection pipeline with a test set used only once, so I would not reject the paper. The dataset and guidelines have independent value, but the evaluation section needs a substantive rewrite before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution; the 98% detection numbers are not supported as reported. The feature-selection pipeline (MI threshold, RFE feature count) was chosen by looking at the test set, which the paper itself warns against. The abstract's 98.7/98.9/98.5 also don't match Table 6's 98.15/98.16/98.15. This is a data-snooping problem, not a flaw in the dataset itself.\n\nWhat's actually new: MLRan looks like a genuinely useful resource. 4,880 samples, 64 families, balanced goodware, nine behavioural feature groups, spanning 2006–2024, with documented sourcing from EldeRan, MOTIF, MarauderMap, and curated threat intel. The Cuckoo automation and feature-extraction scripts are public, and the curation process is documented in unusual detail: per-source counts, duplicate removal, VirusTotal validation, AVClass family alignment, and sandbox hardening (VMCloak, human/disguise modules). That fills a real gap, since most behavioural ransomware datasets are small or lack goodware.\n\nThe GUIDE-MLRan guidelines are a reasonable systematization of known practices, not a big new idea, but useful for the community. The SHAP/LIME analysis is standard and adds little beyond what the feature-selection results already show, but it's not harmful.\n\nWhere it falls down: the evaluation protocol. Table 5 and Figure 9 select the MI threshold and RFE feature count based on test-set accuracy. Algorithm 2 explicitly transforms the test set for each candidate feature count and records balanced accuracy on it before choosing 2%. That is exactly the test snooping the paper says to avoid in Section 3.4.3. The fix is straightforward: use a validation split or nested CV for FS parameter selection, then evaluate once on the held-out test set. Until then, treat all reported detection metrics as optimistic. Minor issue: labels rely on VirusTotal detection counts, which is noisy, but they did verify non-zero detections and manually aligned family names.\n\nWho this is for: people building or benchmarking ransomware datasets, and anyone teaching data snooping in ML security. It deserves a serious referee because the dataset is valuable and the evaluation flaw is correctable.\n\nRecommendation: send it to peer review, but require a corrected evaluation protocol before the performance claims are accepted.","headline":"The dataset is the real contribution; the 98% detection numbers are not supported because feature-selection choices were made on the test set, violating the paper's own protocol.","tokens_in":45007,"tokens_out":2828,"would_cite":true,"duration_ms":24744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MLRan, a balanced 4,880-sample ransomware dataset, claims 98% binary detection with 483 selected features.","keywords":["ransomware dataset","dynamic analysis","machine learning","feature selection","mutual information","recursive feature elimination","explainable AI","behavioral features"],"falsifier":"Re-run the paper's pipeline with both feature-selection decisions locked inside the training process: split the data time-aware, choose the mutual information threshold and the recursive feature elimination count on training or validation data only, and then evaluate the final model once on the untouched most-recent 20%. If balanced accuracy falls substantially below 98% or varies widely across folds, the reported benchmark is not robust to honest evaluation.","tokens_in":43982,"feed_emoji":"🛡️","tokens_out":11742,"duration_ms":91443,"temperature":0.7,"pith_summary":"MLRan is a new behavioural dataset of 4,880 balanced Windows samples: 2,330 ransomware across 64 families and 2,550 goodware, collected from multiple sample sources and spanning 2006 to 2024, with nine behavioural feature groups. The paper's central claim is that, after two-stage feature selection (mutual information filtering at threshold 0.01 followed by recursive feature elimination), a logistic regression model on just 483 features detects ransomware versus goodware at about 98% accuracy, precision and recall. It also proposes GUIDE-MLRan, a fifteen-criterion checklist for building reproducible behavioural ransomware datasets, and releases its dataset, code and metadata so others can rerun or extend the experiments. A fair reader would care because public behavioural ransomware datasets are small, imbalanced and poorly documented; a large, balanced, benchmarkable dataset with a lightweight high-accuracy baseline is a concrete resource for detection research.","feed_headline":"New ransomware dataset hits 98% accuracy with 483 features","feed_subtitle":"MLRan balances 4,880 Windows samples across 64 families and releases the full detection pipeline for replication.","key_machinery":"The central object is the dataset itself: 4,880 samples analysed in a hardened open-source dynamic-analysis sandbox with simulated user activity, an extended timeout and realistic system configuration, then encoded as binary presence features in nine categories: API calls, registry keys, file operations, directory operations, strings, network activity, system operations, dropped files and signatures. The argument is carried by a two-stage feature-selection pipeline: group-wise mutual information filtering cuts the 6.4-million-feature space to 24,162, and recursive feature elimination then selects 2% of that (483 features), which the paper reports as the point of highest balanced accuracy. The deployed detection model is logistic regression over those 483 binary features.","core_discovery":"The paper's discovery, on its own terms, is that a carefully balanced behavioural dataset at this scale makes high-accuracy ransomware detection easy: no deep model is needed. From 6,467,926 binary presence features extracted from sandbox reports, group-wise mutual information filtering (threshold 0.01) keeps 24,162 features, and recursive feature elimination on a logistic regression estimator selects 483. On a time-aware 80:20 split, with samples ordered by first-seen timestamp and the most recent 20% held out, logistic regression reaches 98.15% accuracy, 98.18% balanced accuracy, 98.16% precision and 98.15% recall for binary classification, producing 12 false positives and 6 false negatives in a 975-sample test set. Explainability analysis using SHAP and LIME shows that the top discriminating features come from API calls, strings, registry and system behaviours, with network features eliminated entirely; distinguishing types and families is harder, with accuracy reaching 87.69% for types and 81.85% for families.","pith_inferences":["Editorial inference: the 98% figures likely overstate deployment performance, because the mutual information threshold (0.01) and the recursive-feature-elimination feature count (2%) appear to be selected using test-set performance curves (Section 6.1, Figure 9), which conflicts with the paper's own no-test-snooping guideline; choosing these hyperparameters on a validation split could lower the re","Editorial inference: the highest-ranked feature is a generic string found at the start of essentially every Windows executable, so part of the binary signal may come from file-type artifacts rather than ransomware behaviour; removing that feature and retraining would show how much of the 98% is genuinely behavioural.","Editorial inference: a more convincing benchmark would run the whole pipeline under nested cross-validation with feature selection inside each fold and then report per-family and per-year performance; the time-aware split is a step in that direction, but the feature-count selection needs to happen before the test set is touched."],"forward_implications":["If the central claims hold, MLRan becomes a public benchmark where binary ransomware detection is solved by a linear model and 483 features, giving future work a strong, cheap baseline to beat.","The two-stage feature-selection recipe, mutual information filtering followed by recursive feature elimination, transfers to other high-dimensional behavioural datasets and cuts training time from thousands of seconds to under a second for the evaluated models.","The explainability results give feature engineers a concrete shortlist: API calls, registry accesses, strings and system operations matter, while network, file and directory features contribute little for binary detection in this environment.","Multiclass results for type and family classification are substantially weaker, so the paper's strength is specifically binary detection; researchers using MLRan for family identification should expect lower accuracy.","Because the full pipeline is released, any team can reproduce the numbers, update the dataset with new families, or stress-test the 483-feature model on independent data."],"supporting_citations":[{"why":"Supplies the Elderan ransomware samples that form one source of MLRan and the binary presence feature representation the dataset adopts.","marker":"Sgandurra et al., 2016"},{"why":"Supplies the MOTIF subset of ground-truth-labelled ransomware samples used in MLRan.","marker":"Joyce, Amlani, Nicholas and Raff (2023)"},{"why":"Supplies the MarauderMap samples that contribute to MLRan's ransomware collection.","marker":"Hou et al. (2024)"},{"why":"Provides the AVClass labelling tool used to assign family names to samples lacking ground-truth labels.","marker":"Sebastián et al. (2016)"},{"why":"Describes the open-source sandbox that generates the behavioural reports from which MLRan features are extracted.","marker":"Liu, Lu and Liu (2014)"},{"why":"Motivates time-aware training and evaluation, the basis of MLRan's chronological 80:20 split.","marker":"Pendlebury et al. (2019)"},{"why":"Provides the data-snooping guidelines the paper cites for keeping test, temporal and selective information separate.","marker":"Arp et al. (2022)"},{"why":"Motivates validating goodware with zero-detection scanning and warns against assuming crawled software is benign.","marker":"Botacin et al. (2021)"},{"why":"Justifies the 120-second execution timeout by showing most malicious behaviour occurs in the first two minutes.","marker":"Küchler et al. (2021)"}],"fun_headline_variants":["Ransomware dataset: 4,880 samples, 64 families, 98% accuracy","MLRan: From 6.4M features to 483 for 98% ransomware detection","Balanced ransomware dataset hits 98% recall without deep learning","New ransomware dataset simplifies detection: 98% accuracy, 483 features","Ransomware detection dataset spans 2006-2024, hits 98% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 98% figures assume the held-out test set never influenced feature-selection choices, yet the paper appears to pick the feature-keeping threshold and the final feature count from curves computed on that same test set.","fun_headline_variants_meta":{"raw":{"variants":["Ransomware dataset: 4,880 samples, 64 families, 98% accuracy","MLRan: From 6.4M features to 483 for 98% ransomware detection","Balanced ransomware dataset hits 98% recall without deep learning","New ransomware dataset simplifies detection: 98% accuracy, 483 features","Ransomware detection dataset spans 2006-2024, hits 98% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1854,"prompt_tokens":1031,"completion_tokens":823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":715}},"tokens_in":647,"tokens_out":823,"duration_ms":6089,"temperature":1.0,"reasoning_tokens":715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:28:07.218836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's pipeline with both feature-selection decisions locked inside the training process: split the data time-aware, choose the mutual information threshold and the recursive feature elimination count on training or validation data only, and then evaluate the final model once on the untouched most-recent 20%. If balanced accuracy falls substantially below 98% or varies widely across folds, the reported benchmark is not robust to honest evaluation.","supporting_citations":[{"cited_title":", author Amlani, D","cited_arxiv_id":null,"evidence_quote":"Supplies the MOTIF subset of ground-truth-labelled ransomware samples used in MLRan."},{"cited_title":", author Lu, S","cited_arxiv_id":null,"evidence_quote":"Describes the open-source sandbox that generates the behavioural reports from which MLRan features are extracted."},{"cited_title":", author Pierazzi, F","cited_arxiv_id":null,"evidence_quote":"Motivates time-aware training and evaluation, the basis of MLRan's chronological 80:20 split."},{"cited_title":", author Quiring, E","cited_arxiv_id":null,"evidence_quote":"Provides the data-snooping guidelines the paper cites for keeping test, temporal and selective information separate."},{"cited_title":", author Ceschin, F","cited_arxiv_id":null,"evidence_quote":"Motivates validating goodware with zero-detection scanning and warns against assuming crawled software is benign."}],"review_version":1}