{"id":"a4e13f0d-616c-4d64-9bdf-c34502654336","arxiv_id":"2501.01089","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SILRAD combines Adaptive Random Forest, PCC feature selection, and ADWIN drift detection to classify Sysmon events as ransomware or benign in a stream, reaching 94.11% MCC.","lead":"This paper presents SILRAD, a system that detects ransomware by continuously learning from Windows Sysmon event logs, reporting 98.89% accuracy on its own 196,840-event test stream. It matters because online learning could remove the retraining delay that leaves systems exposed to new ransomware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation stream is a concatenation of pure ransomware and goodware blocks; the reported MCC may reflect block-level label runs rather than performance on a realistic continuous endpoint stream.","rationale":"The paper is honest about its main data limitation: Section 4.4 says the stream was stitched because VMs had to be rolled back. That honesty is a point in its favour, but it does not remove the problem. The central empirical claim, that SILRAD improves as new ransomware families arrive, is demonstrated only on a stream whose structure is very favourable to online updating. With pure positive blocks, the model is fed hundreds of consecutive same-label events; a prequential classifier will appear to adapt even without a sophisticated drift detector. The absence of block-order details and released code/data means this cannot be checked by a reader. A second weakness reinforces the conditional verdict: the non-incremental baselines in Section 5.1 are trained once on the first two blocks and never updated, so the comparison shows the cost of never retraining, not the cost of periodically retraining from scratch. The method itself is reasonable and built on standard components (ARF, ADWIN, River), so I would not reject it; the evidence simply does not yet establish the deployment claim. The reader's conditional verdict remains appropriate.","tokens_in":15965,"tokens_out":12761,"duration_ms":140855,"concrete_test":"Reproduce the experiment with a mixed stream: for each of the 50 ransomware detonations, replay its captured Sysmon events at their original relative timestamps inside a concurrent goodware session, so malicious and benign events are interleaved, and concatenate these sessions in the same family order used in Section 4.4. Run SILRAD with identical River 0.19.0 hyperparameters and compare the overall MCC and the MCC computed on the first 100 ransomware events of each detonation before any labels from that detonation are used for updating. If the mixed-stream MCC falls materially below 94.11% or the first-block MCC is near chance, the stitched pure-block design is responsible for the reported improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 states that because each ransomware detonation destroys the VM, the evaluation stream was created by stitching separate goodware and ransomware captures into one file. The result is a sequence of long, pure positive blocks (50 ransomware executions, 20,710 events) and long pure negative stretches (176,130 events). ARF is updated after every labeled event, so once a block begins, the model receives a long run of same-class examples and can rapidly tune itself to that family; the rising MCC curve in Fig. 9a is therefore at least partly a label-run artifact. No details are given for block order, family order, block lengths, or the stitching rule, and no code/data are released, so a reader cannot determine whether the reported 98.89% accuracy and 94.11% MCC would survive a realistic stream in which ransomware events are interleaved with benign activity rather than arriving as pure blocks. This concern is load-bearing because the paper's central claim is adaptation to evolving ransomware in a continuous production stream.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SILRAD, an online incremental learning system that detects ransomware from Sysmon event streams. The system uses Adaptive Random Forest (ARF) as the base classifier, Pearson Correlation Coefficient (PCC) for feature selection, and ADWIN for concept drift detection. The authors constructed a custom dataset by stitching Sysmon captures from 50 ransomware detonations (six families, 20,710 events) and benign activity (176,130 events), and report 98.89% accuracy and 94.11% MCC for SILRAD, along with lower memory consumption and faster classification than several competing incremental learners.","tokens_in":16142,"tokens_out":4053,"duration_ms":39203,"significance":"If the reported results are robust, the paper makes a useful contribution by demonstrating that online incremental learning can be applied to Sysmon-based ransomware detection, addressing a real limitation of batch retraining. Strengths include the use of live ransomware samples, a Sysmon-specific feature representation, comparison with multiple incremental and batch classifiers, and reporting of resource consumption alongside predictive performance. However, the evaluation rests on a stitched block-structured stream whose realism is not established, and the feature-count selection is performed on the same data used for reporting. The core idea is promising, but the current evidence is not sufficient to support the paper's central claims about continuous real-world detection.","major_comments":[{"comment":"The evaluation stream is described as a concatenation of separately captured goodware and ransomware Sysmon events, with 'blocks of ransomware activity positioned throughout.' No details are given for block order, family order, block lengths, or the stitching rule. Because the model is updated after every labeled event (Algorithm 1), long runs of same-class events let the classifier rapidly adapt to each family's characteristics; the rising MCC curve in Fig. 9a may therefore reflect label-run structure rather than performance on an interleaved production stream. Please report the exact stream construction, evaluate at least one interleaved or temporally realistic stream, and provide per-block and per-family results.","section":"Section 4.4, Fig. 9a"},{"comment":"The number of features (5) was chosen after evaluating classifier performance on the test data, as Table 3 reports MCC for 5, 10, 15, 20, and 25 features and the text selects the best value. This is a form of test-set selection and can inflate the reported metrics. No separate validation procedure or error bars are given. Please use a held-out validation stream for feature-count selection, or report cross-validation or repeated-run variance, so the headline accuracy and MCC are not optimistically biased.","section":"Section 5.2, Table 3"},{"comment":"The reported metrics for SILRAD are internally inconsistent. Section 5.3 states 'yielded 98.88% accuracy, 94.99% precision, 94.35% recall, an F1-score of 94.67% and a Matthews correlation coefficient (MCC) 94.04%,' while Table 4 and the abstract/conclusion report 98.89% accuracy, 94.87% precision, 94.59% recall, 94.73% F1-score, and 94.11% MCC. Please reconcile these numbers and ensure all tables, text, and the abstract refer to the same experimental run.","section":"Table 4, Section 5.3, Abstract, Conclusion"},{"comment":"The comparison with other incremental classifiers is not fully specified: it is unclear whether ARF, HT, LB, and SRP used the same five features selected by PCC or the full feature set. Since the paper's claims about SILRAD's lower memory and faster classification time are central, the comparison must control for the feature set and hyperparameters across all methods. Please state the exact feature set and hyperparameters used for each classifier.","section":"Table 4, Section 5.3"}],"minor_comments":[{"comment":"The pseudocode is incomplete and contains typographical errors: 'ConvertSentanceToVector' is misspelled, the prediction variable '̈y' is rendered incorrectly, and the algorithm does not specify how the ADWIN window W is used to trigger a model reset.","section":"Algorithm 1"},{"comment":"The scoring function for fastText is mathematically ill-formed: the expression sets a sum of inner products equal to 1, which is not a definition of a score. Please rewrite this as a proper scoring or embedding formula.","section":"Eq. (1)"},{"comment":"The notation for the incremental prediction is confusing: a summation over time is used to define a single prediction. It should be written as y_hat_t = m_{t-1}(x_t), with a separate statement that each prediction is made before the model update.","section":"Eq. (3)"},{"comment":"The box plot shows a distribution of MCC values, but the online evaluation appears to be a single pass over one stream. Please explain how the multiple MCC values were obtained (e.g., repeated runs with different seeds, or per-window measurements).","section":"Fig. 10"},{"comment":"Hyperparameters are only partially reported: ensemble sizes for ARF, LB, and SRP, the number of trees, and the grace period for HT are not fully specified. This limits reproducibility.","section":"Sections 5.2 and 5.3"},{"comment":"'Random Forrest' should be 'Random Forest'. Additionally, the table should indicate which classifiers are online incremental and which are batch to avoid confusion.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is reasonable and the area is relevant, but the evaluation protocol is the main obstacle. The stitched-stream construction and test-set-based feature selection directly affect the validity of the headline numbers, and the internal metric inconsistencies must be resolved. I do not see a fundamental flaw that would require rejection, but the authors should be asked to substantially rework the evaluation and reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nThe paper does something genuinely under-studied: it applies true online incremental learning (Adaptive Random Forest + ADWIN drift detection) to Sysmon event streams for ransomware detection. That combination is new in the cited literature, and the pipeline is sensible: fastText embeddings, PCC-based feature ranking, and MCC as the headline metric on an imbalanced dataset. The authors also document a credible detonation lab (Windows 11 VMs with full internet via VPN gateway), and they are honest that SILRAD is not the most accurate classifier in their comparison; LB and SRP beat it, but SILRAD is faster and uses less memory. That trade-off is worth publishing.\n\nThe soft spots are real, though. The evaluation stream is stitched from 50 pure ransomware detonation blocks (20,710 events) and long benign stretches (176,130 events). The stress-test concern is correct: since ARF updates after every labeled instance, a long same-class block lets the model tune to that family quickly, and the rising MCC curve in Fig. 9a is at least partly a label-run artifact. No block order, family order, or stitching rule is given, and no code or data is released. That makes the headline numbers hard to extrapolate to a live stream where ransomware events are interleaved with benign activity. The authors acknowledge the stitching in Section 4.4 but don't discuss its threat to external validity.\n\nMinor issues: the chosen feature count (5) is selected after looking at test performance (Table 3), no error bars are reported for incremental runs, and Eq. (1) looks garbled. The citation pattern is fine; prior incremental works (Li, Darem, Roy & Chen) are fairly characterized as batch-update approaches.\n\nThis is a competent application paper for researchers working on streaming malware detection or endpoint log analysis. It's not a breakthrough, but the evaluation flaw is instructive. Send it to peer review—a good referee can push for a more realistic stream simulation or an ablation with shuffled/interleaved blocks, plus code and data. As it stands, a conditional accept with major revisions.\n\nBest,","headline":"A credible online-learning ransomware detector on Sysmon logs, but the stitched evaluation stream makes the headline numbers hard to trust for real deployments.","tokens_in":26,"tokens_out":3240,"would_cite":true,"duration_ms":61983,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A streaming Sysmon-based ransomware detector achieves 98.89% accuracy and adapts to new families without full retraining.","keywords":["ransomware detection","incremental learning","online learning","concept drift detection","sysmon","adaptive random forest","feature selection","imbalanced data"],"falsifier":"Deploy SILRAD on natural continuous Sysmon logs collected from production Windows endpoints (or an instrumented sandbox with interleaved benign and malicious activity), feed it a holdout set of ransomware families not seen in training, and check whether the MCC stays above 94% and whether ADWIN triggers on genuine drift; if accuracy drops below the reported range or the drift detector fires spuriously when the stream is not stitched into blocks, the central claim would be contradicted.","tokens_in":15769,"feed_emoji":"🛡️","tokens_out":9224,"duration_ms":69076,"temperature":0.7,"pith_summary":"This paper claims that ransomware can be detected continuously, as it happens, by feeding Windows Sysmon security events into an online incremental learning model that updates itself without full retraining. The authors build SILRAD, which uses an Adaptive Random Forest classifier, selects a handful of the most predictive Sysmon features via Pearson correlation, and triggers model updates via ADWIN concept-drift detection. On a stitched stream of nearly 200,000 Sysmon events from six ransomware families, SILRAD reports 98.89% accuracy and a Matthews Correlation Coefficient of 94.11%, and its performance improves as new ransomware families appear. The authors argue that this closes the 'training gap' that leaves systems vulnerable between batch retraining cycles, and that it does so with lower memory and faster classification than competing incremental methods.","feed_headline":"98.89% accuracy: ransomware detector adapts to new strains","feed_subtitle":"Detects new ransomware families in live Sysmon streams without retraining the model from scratch.","key_machinery":"The load-bearing mechanism is the combination of three components: (1) Adaptive Random Forest (ARF), an ensemble of decision trees that each update incrementally and vote on streaming instances; (2) ADWIN (Adaptive Windowing), a drift detector that keeps a variable-length window of recent errors and shrinks the window—and signals a model reset—when the mean error of two sub-windows differs beyond a confidence-based bound; (3) Pearson Correlation Coefficient (PCC) feature selection, which ranks Sysmon log fields by linear correlation with the ransomware/benign label so the model uses only the most predictive features. The paper's contribution is the specific assembly of these pieces on Sysmon events, plus a lab setup that detonates live ransomware in Windows virtual machines and forwards events to a central logging server for feature extraction via fastText embeddings.","core_discovery":"SILRAD's central claim is that ransomware behaviour is learnable from a live Sysmon event stream with an online incremental classifier, provided the model can adapt to concept drift. Using only five Sysmon features—TargetObject, Task, CallTrace, ParentImage, and IntegrityLevel—the Adaptive Random Forest engine, coupled with ADWIN drift detection, detects ransomware from six modern families with 98.89% accuracy and a 94.11% MCC on an imbalanced stream in which ransomware is the minority class. The authors further show that, in contrast to batch-trained models (kNN, Naive Bayes, Random Forest) whose MCC collapses as new families are introduced, SILRAD's MCC stays above 90% and even improves over time (Fig. 9a), because ADWIN resets the model when the stream statistics change. They position this as the first online incremental ransomware detector over Sysmon streams, rather than batch or mini-batch updates.","pith_inferences":["Because the pipeline is not ransomware-specific, the same ARF+ADWIN+PCC design could be transferred to other endpoint telemetry (PowerShell logs, ETW events) where concept drift is expected.","The reported accuracy depends on the stitched stream; a natural next experiment is to run SILRAD on raw continuous Sysmon logs from an instrumented network to see whether ADWIN still triggers correctly when benign and malicious events interleave naturally rather than in blocks.","Since SILRAD's MCC is slightly below Leveraging Bagging and Streaming Random Patches but with a much smaller resource footprint, tuning ARF hyperparameters or adding a second drift detector could close that accuracy gap.","The dominance of TargetObject and CallTrace suggests that ransomware detection is driven by object-access and process-ancestry patterns; testing how the top-five feature ranking shifts across a wider set of ransomware families would clarify its generality."],"forward_implications":["Traditional batch classifiers (kNN, Naive Bayes, Random Forest) drop to MCC values below 35% when new ransomware families appear, while SILRAD stays above 94% MCC.","Five Sysmon features—TargetObject, Task, CallTrace, ParentImage, and IntegrityLevel—carry almost all of the detection signal, so a lightweight agent can monitor endpoints in real time.","SILRAD consumes less memory and classifies faster than Adaptive Random Forest, Hoeffding Tree, Leveraging Bagging, and Streaming Random Patches, while maintaining an MCC over 94%.","The improvement of SILRAD's MCC over time as new families are introduced (Fig. 9a) indicates that ADWIN successfully detects and reacts to concept drift in an imbalanced stream."],"supporting_citations":[{"why":"Supplies the Adaptive Random Forest ensemble method that SILRAD uses as its online classification engine.","marker":"[49]"},{"why":"Supplies the ADWIN adaptive windowing algorithm that triggers concept-drift resets in SILRAD.","marker":"[50]"},{"why":"Provides the fastText subword embedding used to vectorize Sysmon log values into model features.","marker":"[46]"},{"why":"Provides the Pearson Correlation Coefficient formula that SILRAD uses to rank and select Sysmon features.","marker":"[47]"},{"why":"Provides the streaming machine learning library used to implement SILRAD and the comparison classifiers.","marker":"[52]"}],"fun_headline_variants":["Incremental ransomware detector hits 98.89% accuracy on live streams","SILRAD: online ransomware detection that adapts without retraining","Sysmon-based model catches new ransomware using drift detection","98.89% accuracy for ransomware detector that updates incrementally"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the stitched sequence of separately captured benign and ransomware Sysmon events faithfully represents the temporal order, interleaving, and noise of a real production endpoint stream, so that the measured drift adaptation and accuracy transfer to live deployment.","fun_headline_variants_meta":{"raw":{"variants":["Incremental ransomware detector hits 98.89% accuracy on live streams","SILRAD: online ransomware detection that adapts without retraining","Sysmon-based model catches new ransomware using drift detection","98.89% accuracy for ransomware detector that updates incrementally"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1487,"prompt_tokens":1022,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":391}},"tokens_in":638,"tokens_out":465,"duration_ms":4211,"temperature":1.0,"reasoning_tokens":391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:35:12.620523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy SILRAD on natural continuous Sysmon logs collected from production Windows endpoints (or an instrumented sandbox with interleaved benign and malicious activity), feed it a holdout set of ransomware families not seen in training, and check whether the MCC stays above 94% and whether ADWIN triggers on genuine drift; if accuracy drops below the reported range or the drift detector fires spuriously when the stream is not stitched into blocks, the central claim would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Adaptive Random Forest ensemble method that SILRAD uses as its online classification engine."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ADWIN adaptive windowing algorithm that triggers concept-drift resets in SILRAD."},{"cited_title":"doi:10.1007/978-1-4419-9863-7_ 372","cited_arxiv_id":null,"evidence_quote":"Provides the Pearson Correlation Coefficient formula that SILRAD uses to rank and select Sysmon features."},{"cited_title":"Montiel, M","cited_arxiv_id":null,"evidence_quote":"Provides the streaming machine learning library used to implement SILRAD and the comparison classifiers."}],"review_version":1}