{"id":"8c26e75d-96f7-42ee-b865-d3c555ebc5ee","arxiv_id":"2412.04259","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"SCADE uses BM25 and log-entropy rarity scoring plus Isolation Forest context to detect command-line attacks, claiming over 98% SNR with no labeled data.","lead":"SCADE is a command-line anomaly detection framework that blends global rarity scoring with local user and asset context to flag potentially malicious commands without labeled data. The paper claims near-perfect signal-to-noise ratios from a red team exercise, but provides no quantitative evidence, baselines, data, or code to support the claims.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The local 5-day baseline cannot filter the rare-but-benign commands it is designed to filter, so the claimed >98% SNR/TPR is not supported by the method itself.","rationale":"The paper's central operational claim is near-perfect signal-to-noise ratio (SNR) in detecting red-team activity while suppressing false positives from authorized administrative tasks. The authors themselves identify rare-but-benign admin commands as the main false-positive risk and assign the local analysis layer the job of filtering them. That layer, however, is purely frequency-based over a 5-day window. Any command whose recurrence period exceeds five days is invisible in the baseline and will look exactly like a novel malicious command. There is no intent model, no policy knowledge, and no longer-term normalcy model in S9/S10 that could recognize a once-a-year maintenance task. Thus the claimed false-positive minimization is not just unmeasured; it is contradicted by the architecture's stated inputs. This is a load-bearing correctness risk because the headline 98%+ numbers depend on exactly this filtering step. The reader's verdict already notes the short 5-day history as part of the weakest assumption; we sharpen it to a direct internal inconsistency and provide a concrete falsifiable check. Even if the private red-team evaluation were released, it would need to include rare-but-authorized administrative commands in the test set before the SNR claim could be credited.","tokens_in":13112,"tokens_out":7685,"duration_ms":81901,"concrete_test":"Create 90 days of synthetic PROCESS telemetry in which a legitimate administrator runs a specific command once every 30 days, and an attacker runs a distinct malicious command once on day 60. Run SCADE S1-S10 with the stated 5-day local history and default thresholds. If the once-every-30-days command is flagged as high-severity or is not routed to Benign Positives while the malicious command is a True Positive, the local filter fails to separate authorized rare uses from malicious rare uses, contradicting the claimed >98% SNR. Report the confusion matrix for both classes.","verdict_should_be":"REJECT","load_bearing_attack":"The only mechanism SCADE has for preventing rare-but-legitimate commands from becoming false positives is the local analysis stage (S8-S10). Section S8 says a test command run by an admin once a year is rare but not harmful and that local analysis is used to filter such cases. S9 then builds the local baseline from only five days of historical data, computing per-day counts for the flagged command, asset, and user. A command that legitimately runs once a year is absent from every five-day window, exactly like a one-time malicious command. The Isolation Forest in S10 is trained on these five-day statistics, so it has no feature that distinguishes 'rare because authorized but periodic' from 'rare because malicious.' Both appear as a single execution inconsistent with the recent baseline. Consequently, the architecture cannot deliver the false-positive reduction asserted in the abstract and conclusion; the >98% SNR/TPR claim is not derivable from the described pipeline, independent of the missing evaluation details.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SCADE, an unsupervised command-line anomaly detection framework for large-scale data-center environments. The framework consists of a global analysis stage that tokenizes telemetry payload items into 1- and 2-grams and scores them with BM25 and Log Entropy, followed by dynamic threshold selection, and a local analysis stage that builds per-user, per-asset baselines over five-day windows and applies an Isolation Forest to filter false positives. The authors report a red-team proof of concept claiming 100% signal-to-noise ratio in Section 4.4 and 'above 98%' true positive rate in the abstract and conclusion. The contributions claimed include dual-layer detection, first application of BM25 and Log Entropy to command-line anomaly detection, dynamic thresholding, and scalability to billions of events.","tokens_in":13334,"tokens_out":3407,"duration_ms":34067,"significance":"If the reported performance were substantiated, SCADE would be a useful addition to the unsupervised command-line anomaly detection literature: the combination of global rarity scoring with local contextual baselines is sensible, and the application of BM25 and Log Entropy to command-line payloads is a reasonable novelty. The paper also explicitly targets a real operational constraint, the lack of labeled data, and the proposed architecture is modular and plausible for distributed deployment. However, the manuscript as submitted provides no quantitative evaluation beyond unverifiable summary claims, and two architectural elements—the circular threshold selection and the five-day local baseline—undermine the central false-positive-reduction claim. The contributions are therefore not supported by evidence in the current text, and the paper is not reproducible in its present form.","major_comments":[{"comment":"The evaluation section contains no quantitative evidence: no data volume, event count, node count, duration, red-team injection count, confusion matrix, or false-positive count is reported. The statement that SCADE achieved an SNR of 100% is not accompanied by any computation, and the abstract and conclusion claim 'above 98%' and 'true positive rate above 98%' without reconciling these numbers with the 100% figure. The central performance claim is therefore unverifiable from the manuscript.","section":"§4.4"},{"comment":"The dynamic threshold is computed from the mean and standard deviation of the anomaly scores of the same run that is later used to compute the reported SNR. Because there is no held-out period, separate validation set, or ground-truth-based threshold tuning described, the threshold selection and the reported detection performance are not independent. The reported SNR is consequently not a reliable estimate of performance on new data.","section":"§4.3.2, S7"},{"comment":"The local analysis baseline is built from only five days of historical data, yet S8 explicitly identifies the case of a legitimate command executed once a year as rare but not harmful. Such a command would be absent from every five-day window, exactly like a one-time malicious command. The Isolation Forest in S10 therefore has no feature that distinguishes 'rare because authorized and periodic' from 'rare because malicious,' so the architecture cannot deliver the false-positive reduction asserted in the abstract and conclusion.","section":"§4.3.3, S9"},{"comment":"The IDF formula as written is not the standard BM25 IDF. The expression log((N - n(t) + 0.5) / (n(t) + 0.5 + 1)) (if the '+1' is outside the log) or log((N - n(t) + 0.5) / (n(t) + 0.5) + 1) (if inside) differs from the standard form log((N - n(t) + 0.5) / (n(t) + 0.5) + 1), which guarantees non-negativity. The printed version can produce negative or shifted scores, and because all BM25-based anomaly scores in S5–S8 depend on this value, the scoring is not well defined as presented.","section":"§4.3.2, Eq (2)"},{"comment":"The Log Entropy weight formula is not the standard log-entropy weighting and appears to have the summation in the wrong place. The expression 1 + f(t,d) / (sum_{d in D} f(t,d) · log(|D| / (1 + f(t)))) either omits the required entropy sum over tokens or uses f(t) in two incompatible roles. The paper also does not specify how the per-token Log Entropy weights are aggregated into the payload-item score that enters the anomaly threshold in S7, making the entire Log Entropy path irreproducible.","section":"§4.3.2, S6 and Eq (4)"}],"minor_comments":[{"comment":"The high-severity criterion 'more than two standard deviations' and the medium-severity criterion 'between 1.5 and 2 standard deviations' leave the boundary at exactly 2σ unclassified; the paper should state whether that case is high or medium severity.","section":"§4.3.2, S7"},{"comment":"There is a duplicated definite article in 'the the term frequency' that should be corrected.","section":"§4.3.2, S6"},{"comment":"The term 'high Signal-to-Noise Ratio (SNR) environment' is used to describe a setting with many false positives, whereas the rest of the paper treats high SNR as a desirable property. The terminology should be clarified or reversed for consistency.","section":"§3, item 3"},{"comment":"Reference [25] contains the typo 'First Confrence' and several references lack complete venue or year information; in particular, the ACM template placeholders ('Conference acronym XX', 'June 03–05, 2018') should be replaced with the actual submission venue and date.","section":"References"},{"comment":"The title and the abstract use 'High-Performance System' while the body refers to 'high-computation environments' and 'data centers'; these terms should be aligned to avoid ambiguity.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper appears to be an early draft: the ACM template placeholders, inconsistent performance numbers, and absence of any evaluation detail suggest it is not yet ready for formal review. The architectural concerns about the five-day baseline and circular thresholding are substantive and would require redesign or much stronger evidence to overcome. The lack of any experimental data makes it difficult to assess even the plausibility of the claimed 98%+ true positive rate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SCADE is a clearly written system description of a two-layer command-line anomaly detector. The architecture—global BM25/log-entropy rarity scoring, then local per-user/per-asset Isolation Forest—is a reasonable engineering response to a real problem, and the related work is adequate. That is where the credit ends.\n\nThe evaluation is essentially absent. No data volume, event counts, node counts, duration, baseline, or confusion matrix are reported. Section 4.4 claims 100% SNR, the abstract and conclusion claim >98% SNR and >98% TPR, and none of these numbers are tied to a measurement. Equations (2) and (4) contain typos, and the dynamic threshold in S7 is set from the same run that is later scored, so any reported performance is in-sample.\n\nThe deeper problem is the local-analysis stage. The paper says S8–S10 exist to avoid flagging rare-but-benign commands like an admin test that runs once a year. But the S9 baseline uses only five days of history. A once-a-year command is absent from every five-day window, just like a one-time malicious command. The Isolation Forest in S10 therefore has no feature that distinguishes 'rare because periodic' from 'rare because malicious'. The claimed false-positive reduction cannot be produced by this pipeline.\n\nThe novelty is thin: BM25 and log entropy are standard 1990s IR weights, and applying them to command-line tokens is an incremental transfer. The 'first application' claim is unverified.\n\nWho is this for? Practitioners wanting a rough architecture sketch might glance at it. Researchers should not cite its performance numbers. The paper needs real evaluation and a reworked local-analysis design before it is peer-reviewable. I would desk reject as is, with an invitation to resubmit with numbers.","headline":"A plausible architecture with an unverified performance claim and a local-analysis stage that cannot do what the paper says it does.","tokens_in":13838,"tokens_out":4223,"would_cite":false,"duration_ms":38994,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCADE claims that scoring command-line token rarity with BM25 and log entropy, then checking local user and asset baselines, detects malicious commands at above 98% signal-to-noise without labeled data.","keywords":["anomaly detection","command-line security","living-off-the-land","unsupervised learning","BM25","log entropy","dynamic thresholding","Isolation Forest"],"falsifier":"A red-team test where the injected commands reuse only the most frequent command-line tokens and ordinary admin parameter formats should make the global rarity scores for malicious events overlap with normal events; if SCADE's signal-to-noise still exceeds 98% under that condition, the rarity assumption is not actually load-bearing, and if it drops, the framework's core premise is falsified.","tokens_in":12903,"feed_emoji":"🛡️","tokens_out":6597,"duration_ms":61973,"temperature":0.7,"pith_summary":"SCADE is an unsupervised framework for spotting malicious command-line activity in very large enterprise systems, aimed at attacks that abuse legitimate tools, known as living-off-the-land. It argues that rarity is the usable signal: tokens that rarely appear across millions of command-line events are scored by BM25 and log-entropy statistics, and then re-checked against per-user and per-asset baselines so that rare-but-authorized admin actions are not escalated as threats. The paper reports that in a red-team exercise on Microsoft data-center telemetry, the framework separated injected malicious commands from routine activity with above 98% signal-to-noise and true-positive rate, without labeled training data. If the reported figures hold, the framework offers a label-free, scalable detection path for environments where supervised detectors choke on alert volume.","feed_headline":"Rare-command scoring detects attacks at 98% signal-to-noise","feed_subtitle":"An unsupervised dual-layer engine separates malicious commands from legitimately rare admin tasks in Azure-scale telemetry.","key_machinery":"The engine is the two-layer scoring pipeline. Raw process-creation telemetry (Event ID 4688) is normalized and concatenated into a payload_items field, then tokenized as 1-grams and 2-grams. Two rarity scorers run over the corpus: BM25, computed as $BM25(d)=\\sum_t IDF(t)\\cdot TF(t,d)$, and log entropy, which weights each token by $1+\\frac{f(t,d)}{\\sum_{d\\in D} f(t,d)\\log(|D|/(1+f(t)))}$. Dynamic thresholds based on the mean and standard deviation of recent scores, $1.5\\sigma$ for medium severity and $2\\sigma$ for high severity, select candidate anomalies. The local layer then builds a 5-day per-user, per-asset execution history and applies an Isolation Forest to the resulting statistics, producing the final true-positive versus benign-positive decision. The combination lets the system downgrade globally rare commands that are normal for a particular user or asset, which is what keeps the signal-to-noise ratio high.","core_discovery":"The central claim is that combining a global rarity layer with a local context layer lets an unsupervised detector tell genuine threats from benign anomalies at scale. On the global side, each command-line event is flattened into a concatenated payload string, tokenized into 1-grams and 2-grams, and scored by BM25 and log-entropy; both scores reward tokens that are rare across the whole corpus. A dynamic threshold, set at $1.5\\sigma$ and $2\\sigma$ from the recent mean, flags high- and medium-severity candidates. The local layer then looks at each flagged command's 5-day execution history for the involved user and asset and runs an Isolation Forest over those statistics, reclassifying as a benign positive anything that fits the user's or asset's usual pattern. The authors report 100% signal-to-noise in the evaluation narrative and above 98% signal-to-noise and true-positive rate in the abstract and conclusion, with the true-positive versus benign-positive split being the key differentiator.","pith_inferences":["The rarity principle is transferable: the same BM25-plus-local-baseline pipeline could be applied to PowerShell script blocks, bash history, or Kubernetes exec events, though SCADE only tests process-command telemetry.","A natural ablation study would vary the BM25 $k$ and $b$ parameters and the $\\sigma$ thresholds to see how sensitive the claimed >98% signal-to-noise is to those choices; the paper does not report this.","The 5-day local window is short relative to monthly or quarterly admin tasks; extending it may catch annual or seasonal rare-but-authorized commands that currently risk being mislabeled as true positives.","The paper reports the red-team exercise in narrative form without giving confusion-matrix counts or exact score distributions, so an independent replication on a public dataset would be the decisive test."],"forward_implications":["SCADE can operate without labeled data, so it can be dropped into data-center environments where supervised models fail for lack of ground truth.","The dual-layer split between global rarity and local habit should reduce alert fatigue for security teams, since benign positives are separated from true positives rather than dumped into one queue.","Because both global scorers are simple statistical models, the pipeline scales to billions of events across 18 million nodes in near real time, unlike transformer-based detectors.","The framework can flag compliance issues, not just attacks, as demonstrated by the long-running certification dump processes it uncovered.","The modular design allows future integration of active learning, intent understanding, and additional telemetry without rebuilding the core."],"supporting_citations":[{"why":"Supplies the BM25 probabilistic relevance formula that SCADE adapts to score token rarity in command-line payload items.","marker":"[38]"},{"why":"Supplies the log-entropy weighting scheme used as the second global rarity scorer.","marker":"[2]"},{"why":"Baseline active-learning living-off-the-land detector that SCADE contrasts with for label-free operation.","marker":"[34]"},{"why":"Baseline Markov-chain command-line anomaly detector that motivates the need for a dual-layer alternative.","marker":"[25]"},{"why":"Recent command-line language-model detector that frames the scalability comparison for the global analysis layer.","marker":"[27]"},{"why":"Graph-based parent-child process anomaly detector that informs SCADE's metadata considerations in local analysis.","marker":"[16]"}],"fun_headline_variants":["SCADE: Unsupervised CLI anomaly detection with 98% SNR","SCADE: global rarity meets local context for CLI threat spotting","BM25 and log entropy power SCADE's 98% SNR detection","SCADE spots Living-off-the-Land CLI attacks at 98% SNR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that malicious commands are statistically rare in the token distribution, so rarity alone can flag candidates before the local layer reclassifies them; if an attacker uses only common commands or common parameter patterns, the global layer has nothing to catch.","fun_headline_variants_meta":{"raw":{"variants":["SCADE: Unsupervised CLI anomaly detection with 98% SNR","SCADE: global rarity meets local context for CLI threat spotting","BM25 and log entropy power SCADE's 98% SNR detection","SCADE spots Living-off-the-Land CLI attacks at 98% SNR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001366,"raw_usage":{"total_tokens":5554,"prompt_tokens":974,"completion_tokens":4580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":4502}},"tokens_in":590,"tokens_out":4580,"duration_ms":31519,"temperature":1.0,"reasoning_tokens":4502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:34:28.416203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A red-team test where the injected commands reuse only the most frequent command-line tokens and ordinary admin parameter formats should make the global rarity scores for malicious events overlap with normal events; if SCADE's signal-to-noise still exceeds 98% under that condition, the rarity assumption is not actually load-bearing, and if it drops, the framework's core premise is falsified.","supporting_citations":[{"cited_title":"ProblemChild: Discovering Anomalous Patterns based on Parent-Child Process Relationships","cited_arxiv_id":"2008.04676","evidence_quote":"Graph-based parent-child process anomaly detector that informs SCADE's metadata considerations in local analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the log-entropy weighting scheme used as the second global rarity scorer."},{"cited_title":"Stokes, Jonathan Bar Or, Ke Tian, Farid Tajaddo- dianfar, Joshua Neil, Christian Seifert, Alina Oprea, and John C","cited_arxiv_id":null,"evidence_quote":"Baseline active-learning living-off-the-land detector that SCADE contrasts with for label-free operation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline Markov-chain command-line anomaly detector that motivates the need for a dual-layer alternative."},{"cited_title":"In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.)","cited_arxiv_id":null,"evidence_quote":"Recent command-line language-model detector that frames the scalability comparison for the global analysis layer."}],"review_version":1}