{"id":"36703bc0-bd1b-4fd0-94bf-bffaab5a7ffe","arxiv_id":"2501.01083","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A batch-incremental CNN-LSTM with attention and parallel LSTM modules detects ransomware in Sysmon logs with a reported F2-score of 99.61% on a private dataset.","lead":"The paper presents iCNN-LSTM, a ransomware detection system that reads Windows Sysmon event logs and updates its neural network in batches to recognize new ransomware without full retraining. It reports a 99.61% F2-score on a private, imbalanced dataset of about 197,000 events, but the dataset and code are not released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Within-batch 80:20 split causes temporal/family leakage; the 99.61% F2 does not establish detection of unseen ransomware.","rationale":"The reader's weakest assumption is that the per-batch 80:20 splits prevent the model from seeing test events resembling training events from the same ransomware family or process; I agree and sharpen this to the same execution/batch: because the 80:20 split is performed inside each temporal batch, test events can be near-duplicates of training events from the same five-minute ransomware run, not merely same-family events across batches. That makes the leakage concern more direct and more likely to inflate the headline F2. I did not find a separate problem that outweighs this one. The reported FP/FN inconsistency is real but secondary, and it does not by itself change the verdict. The paper's positive contribution is a plausible incremental-update architecture and a reasonable motivation for using Sysmon; the weakness is in the evaluation design and the lack of public artifacts, not in the method's internal logic. The proposed test—chronological or family-disjoint evaluation plus duplicate-overlap reporting—would settle whether the 99.61% F2 reflects genuine detection of unseen ransomware. Since the reader already assigned CONDITIONAL, my recommendation is UNCHANGED.","tokens_in":18928,"tokens_out":4143,"duration_ms":41828,"concrete_test":"Re-run the evaluation with a family-disjoint or strictly chronological split: assign each Sysmon event its ransomware family (or benign session) ID and timestamp; train the initial model on the first 40,000 events, update on each batch, but evaluate on the next batch (a future time window) or on held-out families rather than on 20% of the same batch. Also report the family-to-batch distribution and compute the fraction of exact/near-duplicate events (same selected feature tuple) shared between the 80% train and 20% test portions of each batch. If the F2-score drops materially below 99.61% (or the duplicate fraction is high), the reported result is an artifact of within-batch leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the evaluation protocol in §3.6/Algorithm 1: each 10,000-event update batch is split 80:20 and the model is trained and tested on the same batch. The dataset (§3.3) is a temporal stream of Sysmon events from six ransomware families, with each sample executed for five minutes and events forwarded until the ransom note appears. Sysmon events from one execution are highly repetitive and adjacent in the stream, and the selected features (§3.4.2: CallTrace, GrantedAccess, SourceUser, TargetImage, TargetUser, Task) are largely constant for a given attack process. A random 80:20 split within the same batch therefore places near-identical events from the same attack on both sides of the split. The paper does not report which families appear in which batch, so the claim that 'each update incorporates novel samples from various ransomware families not included in the original training dataset' (§3.6) cannot be checked; even if true at batch level, the test 20% of a batch can share an execution with the training 80% of that same batch. Table 7's F2 of 99.61% is thus consistent with intra-execution memorization rather than generalization to new ransomware. A secondary reporting issue reinforces the need for care: the abstract's 0.17% 'false positive rate' matches the FP column of Table 7, while §1 and §4.4.2 call 4.69% the false positive rate, although Table 7 lists 4.69% as false negatives; the claims about 'lowest false positive rate' are inconsistently worded. No code or dataset is released, so the only way to adjudicate is a protocol change that removes within-batch leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes iCNN-LSTM, a batch-based incremental ransomware detection system that classifies Sysmon event streams using a hybrid CNN-LSTM architecture with parallel LSTM branches and attention mechanisms. The authors collect a private dataset of roughly 197,000 Sysmon events (176,130 benign and 20,710 ransomware events from six families), train an initial model on 40,000 events, then update it on fifteen mini-batches of 10,000 events. Each batch is split 80:20, and the model is fine-tuned on the training portion and evaluated on the test portion. The reported results are an F2-score of 99.61%, precision 99.61%, recall 99.62%, FP 0.17%, FN 4.69%, and runtime 195.69 s, which the paper claims are the best among seven CNN/LSTM baselines re-implemented within the same framework. The paper also positions the approach as overcoming the need for full retraining in real-time ransomware detection.","tokens_in":19244,"tokens_out":7914,"duration_ms":65345,"significance":"The paper addresses a real operational need: ransomware detection in endpoint event streams with incremental model updates. If the empirical results were obtained under a leakage-free protocol, the proposed architecture could be a useful contribution. The authors provide a detailed algorithm (Algorithm 1), a realistic lab setup (Windows 11 VMs with Sysmon and internet), and an explicit discussion of limitations (e.g., no degradation detection). However, the central performance claim rests on a private dataset, a within-batch random split that may leak information across train/test, and re-implemented baselines; the evidence for superiority over prior work is therefore not yet convincing. The paper would benefit from releasing code/data or, at minimum, reporting a temporal evaluation with family-level batch composition.","major_comments":[{"comment":"The evaluation protocol in Section 3.6 and Algorithm 1 splits each 10,000-event update batch into an 80% training and 20% test portion. Since the dataset consists of Sysmon event streams from repeated detonations of six ransomware families, and the selected features in Table 3 (e.g., CallTrace, GrantedAccess, TargetImage, SourceUser, TargetUser, Task) are largely constant for a given attack process, a random split within the same batch places near-identical events from the same execution on both sides of the split. Table 2 reports only event counts per batch, not family-level distribution, so the claim in §3.6 that each update incorporates novel ransomware families is unverifiable. Moreover, even if new families appear in a batch, the test 20% can share an execution with the training 80%. The reported F2-score of 99.61% (Table 7) is therefore consistent with intra-execution memorization rather than generalization to unseen ransomware. The authors should re-evaluate using a temporal split (e.g., train on earlier batches and test on a later batch, with a gap), and report the per-batch family distribution.","section":"§3.6 / Algorithm 1"},{"comment":"Section 4.2 states that Optuna was used for hyperparameter optimization, but the paper does not specify whether a separate validation set was used to guide the search. If the Optuna objective was evaluated on the same test portions that later appear in Table 7, the reported metrics are optimistically biased. The authors should clarify the tuning protocol and, ideally, perform tuning on a split that is disjoint from the final test data used for reporting.","section":"§4.2"},{"comment":"The manuscript is internally inconsistent about false positives and false negatives. The abstract reports FP=0.17% and FN=4.69%, while the contributions bullet in Section 1 states a 'low false positive rate of 4.69%'. Section 4.4.2 repeats this and claims 'the lowest false positive rate of 4.69%', but Table 7 lists 4.69% as the false-negative rate of iCNN-LSTM and shows that the lowest FP among all models is 0.16% (Zhang et al., not iCNN-LSTM). These contradictions prevent a reader from determining which error rate is actually being claimed as minimal. All occurrences should be corrected and the comparative statements checked against Table 7.","section":"Abstract; §1; §4.4.2; Table 7"},{"comment":"The comparison in Section 4.4.2 and Tables 6-7 is entirely conducted on the authors' private Sysmon dataset, with prior models re-implemented by the same authors; no code, data, or pre-trained models are provided. As a result, the 'highest F1/F2' and 'fastest runtime' claims cannot be independently reproduced, and the equivalence of the baseline re-implementations to the originals is not assured. I recommend adding a data/code availability statement and, if that is not possible, clearly framing the results as a within-framework architecture comparison with reduced generality claims.","section":"§4.4.2 / Tables 6-7"}],"minor_comments":[{"comment":"Section 2.1 contains a missing citation: 'within the ransomware and malware detection domains []'.","section":"§2.1"},{"comment":"Section 3.3.2 introduces SMOTE, but Algorithm 1 does not indicate where SMOTE is applied; the authors should state explicitly that SMOTE is applied only to the training portion after the 80:20 split, to avoid synthetic samples appearing in the test set.","section":"§3.3.2 / Algorithm 1"},{"comment":"Section 3.6 claims each update contains 'novel samples from various ransomware families', but Table 2 only gives counts; providing the family composition of each batch would make this assertion checkable.","section":"§3.6 / Table 2"},{"comment":"Figure 9's x-axis is labeled in seconds but the described evaluation is per-batch; please clarify the meaning of the time axis.","section":"Figure 9"},{"comment":"Equation (2) has a stray '= 1' at the end of the summation; this appears to be a typesetting error.","section":"Equation (2)"},{"comment":"Several equations (e.g., Equations 4, 6, 10, 13) have inconsistent subscript notation or missing indices; a careful proofread would improve readability.","section":"Various equations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's empirical contribution is currently difficult to audit because the dataset and code are not released and the comparison baselines are re-implementations. Given the strong claims of superiority, I would ask the editors to require at least a detailed appendix on the data split and family distribution, or a release of the artifacts, before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. The paper describes a plausible incremental ransomware detector that watches Sysmon event streams, and it is honest about its limitations. The new piece is the combination: batch-incremental training, a CNN-LSTM with parallel LSTM modules and attention, applied to Sysmon logs. They also re-implement seven prior architectures in the same framework and report runtime alongside accuracy, which is more than most papers in this space do.\n\nThe problem is the evaluation protocol, and it is a big one. Each 10,000-event batch is split randomly 80:20, with the model trained on 80% and tested on the remaining 20%. The dataset is a temporal stream of Sysmon events from repeated detonations of six ransomware families. Events from a single execution are highly repetitive and adjacent in the stream. A random split will place near-identical events from the same attack on both sides of the split, so the reported F2 of 99.61% is consistent with intra-execution memorization rather than generalization to unseen ransomware. The paper says each batch includes novel families but never reports which families land in which batch, so you cannot check that claim. This is a load-bearing flaw: the headline number does not demonstrate what the paper wants it to demonstrate.\n\nThere are also smaller issues. No code or dataset is released, and the baseline models are re-implemented by the authors on the same private data, so the comparison is not externally anchored. Hyperparameters were tuned with Optuna without a separate validation set. And the reporting is sloppy: the contributions section says a false positive rate of 4.69%, but Table 7 lists 4.69% as false negatives; the abstract has it right at 0.17%. That inconsistency alone would need fixing.\n\nWhere the paper does well: the architecture is thoughtfully designed and the runtime comparison is a nice touch. The parallel LSTM claim is testable and worth checking. But as it stands, the central performance claim is not reliable. The fix is conceptually simple: split by time or by execution so a test event never shares an execution with a training event, and report family-level distribution across batches. If they do that and release the artifacts, the result could be meaningful.\n\nWho is this for? Researchers working on endpoint detection and incremental learning. I would not cite the headline numbers, but I might cite the architecture approach. It deserves a serious referee, but it needs major revision before publication.\n\nRecommendation: send it to peer review, but with the explicit expectation that the evaluation protocol gets fixed. A desk reject would be too harsh given the honest engineering and the clarity of the rest of the paper.","headline":"Plausible incremental ransomware detector, but the within-batch split leaks and the headline F2 doesn't demonstrate generalization to new ransomware.","tokens_in":19793,"tokens_out":3033,"would_cite":false,"duration_ms":27633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a batch-incremental CNN-LSTM system that reads Sysmon event logs detects ransomware in a stream with a 99.61% F2-score, 0.17% false positives, and 4.69% false negatives on a highly imbalanced dataset.","keywords":["ransomware detection","deep learning","incremental learning","Sysmon","attention mechanism","CNN-LSTM","imbalanced data","batch learning"],"falsifier":"Re-run the incremental protocol with training and test batches separated by whole ransomware sample (or family), and compare the per-family recall; if the F2-score falls substantially below 99.61%, the reported performance came from split leakage rather than generalization to new variants.","tokens_in":18745,"feed_emoji":"🛡️","tokens_out":6821,"duration_ms":56043,"temperature":0.7,"pith_summary":"The paper claims that a hybrid CNN-LSTM model reading Windows Sysmon event logs can spot ransomware in a continuous stream with an average F2-score of 99.61%, a false-negative rate of 4.69%, and a false-positive rate of 0.17% on a highly imbalanced dataset. Its distinguishing move is batch-based incremental learning: instead of rebuilding the model from scratch, it updates the network every 10,000 events, so the detector can absorb new ransomware variants as they appear. If true, this would give endpoint security a real-time detector that stays current without the cost of full retraining, and that prioritizes catching attacks (recall) over avoiding false alarms.","feed_headline":"Ransomware detector scores 99.61% on Sysmon event stream","feed_subtitle":"Batch-by-batch updates let it catch new variants without full retraining, with false negatives at 4.69%.","key_machinery":"The load-bearing machinery is the parallel-LSTM-plus-attention CNN-LSTM classifier combined with a batch-incremental update loop. Sysmon events are embedded with fastText, filtered by Pearson-correlation feature selection, balanced with SMOTE, and fed into 1D CNN layers that extract local patterns; parallel LSTM modules then process the sequence, and an attention mechanism weights the most informative hidden states before the dense classification layers. This parallel arrangement removes the sequential LSTM bottleneck and is what the paper credits for the improved runtime, while the 10,000-event batch updates give the model its ability to adapt to new ransomware variants without a complete rebuild.","core_discovery":"The central discovery is that an architecture made of parallel LSTM branches with an attention layer, mounted after 1D CNN feature extractors, detects ransomware events in a stream of Sysmon logs more accurately and faster than seven CNN/LSTM baselines re-implemented inside the same incremental framework. On the authors' imbalanced dataset of nearly 200,000 events, the proposed system reached a 99.61% F2-score, 99.62% recall, and 99.61% precision, with a 4.69% false-negative rate, and it did so while reducing runtime compared to sequentially stacked CNN-LSTM models. The batch-incremental updating procedure, which uses 10,000-event mini-batches and SMOTE balancing, kept the F2-score above 99% even when each batch introduced ransomware families absent from the initial training set.","pith_inferences":["If the 80:20 split is done per batch without tracking the originating ransomware sample, the test portion can contain events from the same execution as training events; a family-disjoint holdout would be a stronger test of the 'new variant' claim.","The dataset covers only six families on a single Windows 11 lab build, so the transfer of the reported scores to other families, other OS versions, or production-scale event rates is an open question that the paper does not address.","A natural next step is to measure per-batch inference latency against a real-world Sysmon event rate; the 195.69-second runtime is an end-to-end training-plus-evaluation figure, not a per-event detection latency.","Because SMOTE is applied per batch and features are re-selected by Pearson correlation, the method's long-run behavior under concept drift (e.g., benign software changes) remains to be tested beyond the fixed lab scenario."],"forward_implications":["A deployed detector could be updated after every 10,000-event batch, keeping itself current with new ransomware behaviors without a full retraining cycle.","The 99.61% F2-score is achieved with recall weighted twice as heavily as precision, so the system is deliberately biased toward catching attacks at the cost of more false alarms.","Among the seven CNN/LSTM baselines reimplemented in the same framework, only the proposed architecture reached a false-negative rate below 5% at a runtime of 195.69 seconds.","The parallel LSTM branches and attention mechanism eliminate the sequential processing bottleneck, which is the paper's stated reason for the latency reduction.","The consistency of the F2-score above 99% across all 15 update batches indicates the incremental mechanism does not suffer from catastrophic forgetting in this setting."],"supporting_citations":[{"why":"Supplies the BiLSTM-CRF incremental baseline whose reported F1, recall and precision are used as comparison in Table 5.","marker":"[14]"},{"why":"Provides the batch-incremental bagging baseline from the literature that the paper compares its F1 and F2 scores against.","marker":"[23]"},{"why":"An LSTM architecture re-implemented inside the authors' incremental framework to serve as a baseline in Table 7.","marker":"[44]"},{"why":"A single-layer CNN-LSTM model re-implemented in the framework; a main speed/accuracy comparison point.","marker":"[45]"},{"why":"A single-layer LSTM detector re-implemented in the framework to show the limitation of CNN-only or LSTM-only designs.","marker":"[46]"},{"why":"Multi-layer CNN with attention, re-implemented in the framework; its runtime and F2 result are compared in Table 7.","marker":"[9]"},{"why":"A CNN-LSTM model re-implemented in the framework; one of the closest F2-score rivals.","marker":"[39]"},{"why":"Tri-layer sequentially stacked CNN-LSTM re-implemented in the framework, representing the stacked-architecture competitor.","marker":"[47]"},{"why":"Stacked CNN-LSTM with transpose layers re-implemented in the framework; a comparison point for the parallel design.","marker":"[48]"}],"fun_headline_variants":["Ransomware detector hits 99.61% F2 on Sysmon logs","Batch-updated CNN-LSTM catches ransomware at 99.61% F2","Incremental learning: 99.61% F2 ransomware detection on Sysmon","Sysmon-based ransomware model adapts without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the per-batch 80:20 split keeps test events genuinely unseen, but since all ransomware logs come from repeated lab detonations of only six families, similar events may appear on both sides of the split and inflate the reported F2-score.","fun_headline_variants_meta":{"raw":{"variants":["Ransomware detector hits 99.61% F2 on Sysmon logs","Batch-updated CNN-LSTM catches ransomware at 99.61% F2","Incremental learning: 99.61% F2 ransomware detection on Sysmon","Sysmon-based ransomware model adapts without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3167,"prompt_tokens":932,"completion_tokens":2235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2153}},"tokens_in":548,"tokens_out":2235,"duration_ms":14776,"temperature":1.0,"reasoning_tokens":2153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:35:29.991548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the incremental protocol with training and test batches separated by whole ransomware sample (or family), and compare the per-family recall; if the F2-score falls substantially below 99.61%, the reported performance came from split leakage rather than generalization to new variants.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BiLSTM-CRF incremental baseline whose reported F1, recall and precision are used as comparison in Table 5."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the batch-incremental bagging baseline from the literature that the paper compares its F1 and F2 scores against."},{"cited_title":"Maniath, A","cited_arxiv_id":null,"evidence_quote":"An LSTM architecture re-implemented inside the authors' incremental framework to serve as a baseline in Table 7."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A single-layer CNN-LSTM model re-implemented in the framework; a main speed/accuracy comparison point."},{"cited_title":"Homayoun, A","cited_arxiv_id":null,"evidence_quote":"A single-layer LSTM detector re-implemented in the framework to show the limitation of CNN-only or LSTM-only designs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multi-layer CNN with attention, re-implemented in the framework; its runtime and F2 result are compared in Table 7."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A CNN-LSTM model re-implemented in the framework; one of the closest F2-score rivals."},{"cited_title":"Bensaoud, J","cited_arxiv_id":null,"evidence_quote":"Tri-layer sequentially stacked CNN-LSTM re-implemented in the framework, representing the stacked-architecture competitor."},{"cited_title":"Deivakani, M","cited_arxiv_id":null,"evidence_quote":"Stacked CNN-LSTM with transpose layers re-implemented in the framework; a comparison point for the parallel design."}],"review_version":1}