{"id":"103af0cc-1a0b-44b7-8d57-f8fbe8bfd807","arxiv_id":"2607.29390","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Preprocessing and labeling choices in lateral-movement detector benchmarks substantially inflate reported performance and can change which detector looks best.","lead":"This paper shows that seemingly small choices in how benchmark datasets are prepared and labeled can dramatically change the measured performance of lateral-movement detectors. Re-evaluating three popular detectors under more realistic policies, the author finds much lower performance and, for one dataset, a different ranking of methods.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OpTC ranking claim rests on a manually built, unreleased 249-event label set; its sensitivity to label perturbations is untested.","rationale":"The reader's weakest_assumption identifies the manual/hybrid label construction for OpTC as the load-bearing premise. I agree: the paper's most striking and novel conclusion is the OpTC ranking change, and that conclusion is entirely dependent on labels that are not provided, not reproducible from a formal rule, and based on the author's manual judgment. The small size of the positive class (249 LM events) makes the metrics especially sensitive to labeling decisions. Other potential concerns—modified detector implementations, fixed hyperparameters, limited detector selection—are real but secondary: the paper's contribution is an evaluation methodology, not a claim that the detectors themselves are poorly implemented, and the authors do provide code for the preprocessing pipeline. The LANL results, which use official labels, already support the weaker claim of over-optimism. But the stronger claim about ranking reversal on OpTC stands or falls with the label set. Therefore the verdict should remain CONDITIONAL, contingent on the authors releasing the labels and demonstrating that the ranking is robust to reasonable label perturbations. My proposed test directly settles this by measuring label sensitivity or inter-annotator reproducibility.","tokens_in":12036,"tokens_out":3708,"duration_ms":44714,"concrete_test":"Check whether the released code includes the full OpTC label set (event IDs for the 1354 malicious and 244 LM-related flows). If not, request it. Then perform a sensitivity analysis: perturb the LM label set by adding/removing 10% (≈25 events) of the manually added events, using alternative interpretations of the red team report, and recompute Table 6 metrics for all three detectors. If the ordering of Argus vs Euler/Pikachu on AUC or AP changes under any plausible perturbation, the ranking claim is not robust. Alternatively, have a second researcher independently re-derive the label set from the red team report and system logs and compare with the authors' labels, reporting agreement metrics. If agreement is low (<80% on LM-related events), the manual labeling is the decisive, fragile step.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline conclusion—that on OpTC Argus no longer outperforms competitors and the ranking changes—is driven by Table 6, where 'LM only' metrics are computed over just 249 positive events out of 36M. These positives come from the hybrid labeling method in §5.3, which combines process-based taint labels with manual inspection: the author decides which additional flow-start events are 'reliably attributed to lateral movements' and which process-labeled events count as LM versus other malicious activity. The label set is not shipped with the released code, and no inter-annotator agreement or sensitivity analysis is reported. Because AUC and AP over 249 positives are high-variance, a handful of label changes could flip the Argus/Euler/Pikachu ordering. The 'all malicious' set is also partly manual, so the ranking-impact claim depends on the same unverified labels. The broader statement that published results were 'excessively optimistic' is partially supported by LANL, where labels are official, so this concern is specific to the OpTC ranking-change contribution—but that ranking change is a headline result and is not robust until the label set is independently reproduced or shown stable under perturbation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how preprocessing and labeling choices for the LANL and OpTC benchmark datasets affect the evaluation of graph-based lateral movement detectors. It surveys published practices, identifies choices that are unrealistic or lead to unfair comparisons, and proposes its own guidelines. Using these guidelines, it re-evaluates three published detectors—Pikachu, Euler, and Argus—and reports substantially lower AUC and AP than in the original papers. On LANL the relative ranking is preserved, but on OpTC the ranking changes, with Argus no longer outperforming the others. The paper concludes that previously published lateral movement detection results were 'excessively optimistic' and that benchmark methodology must be standardized.","tokens_in":12365,"tokens_out":5139,"duration_ms":59679,"significance":"The topic is important and timely: benchmark evaluation practices in this area are indeed heterogeneous and often poorly documented, and the paper provides a concrete, reproducible framework for improving them. The LANL re-evaluation is credible and uses official labels, and the released code and detailed preprocessing recommendations are useful community assets. However, the OpTC ranking-change claim—one of the two headline conclusions—rests on a manually constructed, unreleased label set. This is a load-bearing weakness: until the labels are released and shown stable under perturbation, or are independently reproduced, the claim that benchmark practices alter detector rankings on OpTC is not fully supported.","major_comments":[{"comment":"The OpTC ranking claim rests on a manually constructed label set. The manuscript states that the author 'manually inspect[s] the system logs' to add flow-start events and uses the red-team report to separate lateral movement from other malicious activity, but it provides no annotation protocol, no inter-annotator agreement, no release of the label set, and no reproducibility statement. The 'LM only' metrics in Table 6 are computed over only 244 positives (Table 4), so the Argus/Euler/Pikachu ordering is highly sensitive to individual label decisions. This is load-bearing for the conclusion that 'the ranking of the three detectors is also impacted.'","section":"§5.3, Table 6"},{"comment":"The ranking change is not confined to the LM-only column; in the 'all malicious' column Argus (AUC 79.75) also falls below Pikachu (86.02) and Euler (84.31). These labels come from the same hybrid manual set, so the ranking-impact conclusion depends on the same unverified labels. A robustness analysis—for example, perturbing a small fraction of the 1,354 positive labels or bootstrapping over label subsets—is needed before claiming that evaluation practices systematically alter detector rankings on OpTC. Without it, the observed ranking change could be an artifact of the label set rather than of detector quality.","section":"§6.2, Table 6 (all-malicious column)"}],"minor_comments":[{"comment":"Typo: 'In constrast' should be 'In contrast'.","section":"§5.3"},{"comment":"The 'duplicates' terminology in the legend is unclear until one reads §4.1; please define it in the caption (the curves are computed after duplicating merged-event scores to the event level).","section":"Figure 3"},{"comment":"The malicious-flow counts in Table 3 (1,359 total, 249 LM-related) differ from Table 4 (1,354 total, 244 LM-related). This is presumably due to flow deduplication described in §4.3, but the reconciliation is left implicit; add an explicit sentence.","section":"Tables 3 and 4"},{"comment":"The standard deviations from 10 runs are reported, but no statistical test is used to support the ranking claims. In particular, in the LM-only AUC column, Euler (52.90±6.82) and Pikachu (50.74±0.81) overlap within one standard deviation; a paired test or confidence interval would strengthen the claim that the ranking changed.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The LANL part of the paper is solid and useful. The OpTC ranking-change claim is the main risk. If the authors can release the exact label set or provide a detailed annotation protocol plus a label-perturbation sensitivity analysis, the paper would be acceptable. I recommend major revision, not rejection, because the issue is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Corentin's paper is a useful, careful benchmarking study. What's genuinely new is the systematic quantification of how preprocessing and labeling choices change measured performance and even rankings for the two most-used datasets in lateral-movement research. The LANL part is the strongest: using official labels and the full test set, all three detectors drop in performance, and the explanation for why previous preprocessing inflated scores is convincing. The survey is useful, the recommendations are sensible, and the event-level vs edge-level analysis is a nice contribution that goes beyond prior replication work.\n\nThe soft spot is exactly where the stress-test note points. The OpTC 'LM only' results, which are the basis for saying Argus no longer outperforms competitors, are computed over 249 positively labeled events hand-picked by the author. The label set is not shipped, and there is no sensitivity analysis or inter-annotator check. With 249 positives, AUC and AP have wide error bars; a few changes in the labels could flip the ordering. This is not a fatal flaw, but it is load-bearing for one of the headline conclusions. The 'all malicious' set is also partly manual, so the ranking impact for OpTC depends on the same unverified labels.\n\nTwo smaller caveats: the detectors' implementations are modified in 'minor' ways, and the effect of those changes is not quantified. And the paper draws conclusions from only three detectors; that's acceptable for a study like this, but the conclusions should be phrased as evidence, not proof.\n\nOverall, I trust the direction: published numbers in lateral movement detection are inflated, and the LANL evidence supports that. The OpTC claim is plausible but needs either the label set released or a robustness check against label perturbations. If that's fixed, this will be a useful reference for anyone evaluating on these datasets. For now, it deserves peer review, and the reviewers should push on the labels.\n\nI'd cite it for the guidelines and LANL results, and I'd bring it to a reading group focused on benchmarking.","headline":"Useful benchmarking study with a solid LANL re-evaluation, but the OpTC ranking reversal rests on a manually built, unreleased label set that needs verification before it becomes a headline.","tokens_in":12745,"tokens_out":3220,"would_cite":true,"duration_ms":33087,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that published lateral movement detection results are systematically inflated by dataset preprocessing and labeling choices, and shows that re-evaluation under fairer policies drops performance and changes detector ranking","keywords":["lateral movement detection","benchmark evaluation","dataset preprocessing","labeling","LANL dataset","OpTC dataset","graph-based detection","reproducibility"],"falsifier":"Independently reproduce the OpTC label set from the red-team report and system logs, or have a second team apply the same hybrid procedure, then re-run Argus, Euler, and Pikachu on the author's preprocessed data; if Argus's AUC returns to near 99.7 on the resulting labels, the paper's ranking change collapses.","tokens_in":11954,"feed_emoji":"📉","tokens_out":4008,"duration_ms":41211,"temperature":0.7,"pith_summary":"This paper claims that how benchmark datasets are preprocessed and labeled is a major, largely overlooked source of bias in lateral movement detection research. It surveys inconsistent practices across studies using the LANL and OpTC datasets, and proposes standardized policies that are realistic for real-world deployment. Re-evaluating three published detectors under these policies, it finds substantially lower performance than originally reported on both datasets. For OpTC, the reported ranking changes: Argus, previously the clear best, performs no better than competitors when only true lateral movements are labeled. The upshot: published comparisons in this area may not be reliable until evaluation practices are standardized.","feed_headline":"Benchmark choices inflate lateral-movement detector results","feed_subtitle":"Standardizing preprocessing and labels on LANL and OpTC cuts reported performance and flips detector rankings.","key_machinery":"The central mechanism is the evaluation pipeline itself: how raw network/authentication events are filtered, split into train/test, merged into graph edges, and labeled. The paper's key contributions are a set of concrete preprocessing recommendations (include all remote LogOn events on LANL; keep the full test period; deduplicate flow-start events and map IPs to hosts on OpTC; compute metrics at the event level rather than the edge level) and a hybrid labeling method for OpTC that combines process-based taint labels with manual inspection of system logs and the red-team report to flag only genuine lateral movements.","core_discovery":"The paper's central discovery is that common preprocessing choices — such as filtering out benign users, using only NTLM authentications, merging events into edges, and labeling all flows from compromised hosts as malicious — artificially inflate detector performance and make cross-paper comparisons unfair. The author re-runs three widely cited detectors (Pikachu, Euler, Argus) on LANL and OpTC using new, operationally grounded preprocessing and labeling policies. On LANL all three drop in AUC and average precision but keep their ranking. On OpTC, with a hybrid label set that separates true lateral movements from other malicious activity, Argus falls to an AUC of 43.23 for lateral-movement-o","pith_inferences":["The same preprocessing/labeling biases likely affect other benchmark datasets and detectors beyond the three studied; a broader replication sweep could reveal similar inflation elsewhere.","The manual labeling effort described for OpTC is hard to reproduce, which points to the need for a community-maintained, versioned label set as a shared resource.","If the ranking change on OpTC holds, it suggests that graph-based detectors may be closer in true capability than citation counts imply, and that future work should re-baseline against a common preprocessing rather than cite original numbers.","The paper's distinction between lateral-movement-only and all-malicious labels could be applied as a standard two-axis evaluation, giving defenders a clearer picture of what a detector actually flags."],"forward_implications":["If the paper's evaluation methodology is adopted, published performance numbers for the three studied detectors on LANL and OpTC cannot be directly compared with future results.","The OpTC detector ranking changes: Argus's reported superiority over Euler and Pikachu is not reproduced under the new labels, so conclusions drawn from the original ranking are unsupported.","The large performance gap between 'all malicious events' and 'lateral movement only' metrics on OpTC means that detectors optimized to catch noisy malicious activity may be much worse at the specific lateral-movement task defenders care about.","The paper's recommendations make evaluation more realistic by removing a priori knowledge of compromised users and benign-account filtering, so new results should reflect harder, more honest conditions.","Fair comparisons require event-level test sets with identical events for all detectors, which implies that future papers should report exactly which events were excluded and how edges were formed."],"fun_headline_variants":["Preprocessing inflates lateral movement detector scores","Benchmark prep distorts lateral movement detector rankings","Fair data prep cuts lateral movement detector performance","Standardized benchmark evals change detector rankings","How benchmark labeling skews lateral movement detection"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the author's manually constructed hybrid labels for OpTC correctly identify which flow-start events are true lateral movements; if those labels are wrong or not reproducible, the drastic performance drops and ranking changes could be artifacts of the label set rather than of detector quality.","fun_headline_variants_meta":{"raw":{"variants":["Preprocessing inflates lateral movement detector scores","Benchmark prep distorts lateral movement detector rankings","Fair data prep cuts lateral movement detector performance","Standardized benchmark evals change detector rankings","How benchmark labeling skews lateral movement detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2475,"prompt_tokens":650,"completion_tokens":1825,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":394,"completion_tokens_details":{"reasoning_tokens":1758}},"tokens_in":394,"tokens_out":1825,"duration_ms":15806,"temperature":1.0,"reasoning_tokens":1758,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:54:38.045954+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently reproduce the OpTC label set from the red-team report and system logs, or have a second team apply the same hybrid procedure, then re-run Argus, Euler, and Pikachu on the author's preprocessed data; if Argus's AUC returns to near 99.7 on the resulting labels, the paper's ranking change collapses.","supporting_citations":[],"review_version":1}