Pith. sign in

REVIEW 2 major objections 4 minor 31 references

On fair and realistic performance evaluations for graph-based lateral movement detectors

T0 review · 2 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that published lateral movement detection results are systematically inflated by dataset preprocessing and labeling choices, and shows that re-evaluation under fairer policies drops performance and changes detector ranking

desk verdict Useful benchmarking study with a solid LANL re-evaluation, but the OpTC ranking reversal rests on a manually built, unreleased label set that needs verification before it becomes a headline. read the letter →

arxiv 2607.29390 v1 pith:TOEZFWH7 submitted 2026-07-31 cs.CR

Corentin Larroche This is my paper · ORCID
classification cs.CR
keywords lateralmovementdetectionbenchmarkevaluationdatasetpreprocessinglabelingLANLOpTCgraph-basedreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that how benchmark datasets are preprocessed and labeled is a major, largely overlooked source of bias in lateral movement detection research. It surveys inconsistent practices across studies using the LANL and OpTC datasets, and proposes standardized policies that are realistic for real-world deployment. Re-evaluating three published detectors under these policies, it finds substantially lower performance than originally reported on both datasets. For OpTC, the reported ranking changes: Argus, previously the clear best, performs no better than competitors when only true lateral movements are labeled. The upshot: published comparisons in this area may not be reliable until evaluation practices are standardized.

What carries the argument

The central mechanism is the evaluation pipeline itself: how raw network/authentication events are filtered, split into train/test, merged into graph edges, and labeled. The paper's key contributions are a set of concrete preprocessing recommendations (include all remote LogOn events on LANL; keep the full test period; deduplicate flow-start events and map IPs to hosts on OpTC; compute metrics at the event level rather than the edge level) and a hybrid labeling method for OpTC that combines process-based taint labels with manual inspection of system logs and the red-team report to flag only genuine lateral movements.

What would settle it

Independently reproduce the OpTC label set from the red-team report and system logs, or have a second team apply the same hybrid procedure, then re-run Argus, Euler, and Pikachu on the author's preprocessed data; if Argus's AUC returns to near 99.7 on the resulting labels, the paper's ranking change collapses.

Watch

Extended reading notes

Core claim

The paper's central discovery is that common preprocessing choices — such as filtering out benign users, using only NTLM authentications, merging events into edges, and labeling all flows from compromised hosts as malicious — artificially inflate detector performance and make cross-paper comparisons unfair. The author re-runs three widely cited detectors (Pikachu, Euler, Argus) on LANL and OpTC using new, operationally grounded preprocessing and labeling policies. On LANL all three drop in AUC and average precision but keep their ranking. On OpTC, with a hybrid label set that separates true lateral movements from other malicious activity, Argus falls to an AUC of 43.23 for lateral-movement-o

Load-bearing premise

The load-bearing assumption is that the author's manually constructed hybrid labels for OpTC correctly identify which flow-start events are true lateral movements; if those labels are wrong or not reproducible, the drastic performance drops and ranking changes could be artifacts of the label set rather than of detector quality.

Editorial extensions

If this is right

  • If the paper's evaluation methodology is adopted, published performance numbers for the three studied detectors on LANL and OpTC cannot be directly compared with future results.
  • The OpTC detector ranking changes: Argus's reported superiority over Euler and Pikachu is not reproduced under the new labels, so conclusions drawn from the original ranking are unsupported.
  • The large performance gap between 'all malicious events' and 'lateral movement only' metrics on OpTC means that detectors optimized to catch noisy malicious activity may be much worse at the specific lateral-movement task defenders care about.
  • The paper's recommendations make evaluation more realistic by removing a priori knowledge of compromised users and benign-account filtering, so new results should reflect harder, more honest conditions.
  • Fair comparisons require event-level test sets with identical events for all detectors, which implies that future papers should report exactly which events were excluded and how edges were formed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same preprocessing/labeling biases likely affect other benchmark datasets and detectors beyond the three studied; a broader replication sweep could reveal similar inflation elsewhere.
  • The manual labeling effort described for OpTC is hard to reproduce, which points to the need for a community-maintained, versioned label set as a shared resource.
  • If the ranking change on OpTC holds, it suggests that graph-based detectors may be closer in true capability than citation counts imply, and that future work should re-baseline against a common preprocessing rather than cite original numbers.
  • The paper's distinction between lateral-movement-only and all-malicious labels could be applied as a standard two-axis evaluation, giving defenders a clearer picture of what a detector actually flags.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies how preprocessing and labeling choices for the LANL and OpTC benchmark datasets affect the evaluation of graph-based lateral movement detectors. It surveys published practices, identifies choices that are unrealistic or lead to unfair comparisons, and proposes its own guidelines. Using these guidelines, it re-evaluates three published detectors—Pikachu, Euler, and Argus—and reports substantially lower AUC and AP than in the original papers. On LANL the relative ranking is preserved, but on OpTC the ranking changes, with Argus no longer outperforming the others. The paper concludes that previously published lateral movement detection results were 'excessively optimistic' and that benchmark methodology must be standardized.

Significance. The topic is important and timely: benchmark evaluation practices in this area are indeed heterogeneous and often poorly documented, and the paper provides a concrete, reproducible framework for improving them. The LANL re-evaluation is credible and uses official labels, and the released code and detailed preprocessing recommendations are useful community assets. However, the OpTC ranking-change claim—one of the two headline conclusions—rests on a manually constructed, unreleased label set. This is a load-bearing weakness: until the labels are released and shown stable under perturbation, or are independently reproduced, the claim that benchmark practices alter detector rankings on OpTC is not fully supported.

major comments (2)
  1. [§5.3, Table 6] The OpTC ranking claim rests on a manually constructed label set. The manuscript states that the author 'manually inspect[s] the system logs' to add flow-start events and uses the red-team report to separate lateral movement from other malicious activity, but it provides no annotation protocol, no inter-annotator agreement, no release of the label set, and no reproducibility statement. The 'LM only' metrics in Table 6 are computed over only 244 positives (Table 4), so the Argus/Euler/Pikachu ordering is highly sensitive to individual label decisions. This is load-bearing for the conclusion that 'the ranking of the three detectors is also impacted.'
  2. [§6.2, Table 6 (all-malicious column)] The ranking change is not confined to the LM-only column; in the 'all malicious' column Argus (AUC 79.75) also falls below Pikachu (86.02) and Euler (84.31). These labels come from the same hybrid manual set, so the ranking-impact conclusion depends on the same unverified labels. A robustness analysis—for example, perturbing a small fraction of the 1,354 positive labels or bootstrapping over label subsets—is needed before claiming that evaluation practices systematically alter detector rankings on OpTC. Without it, the observed ranking change could be an artifact of the label set rather than of detector quality.
minor comments (4)
  1. [§5.3] Typo: 'In constrast' should be 'In contrast'.
  2. [Figure 3] The 'duplicates' terminology in the legend is unclear until one reads §4.1; please define it in the caption (the curves are computed after duplicating merged-event scores to the event level).
  3. [Tables 3 and 4] The malicious-flow counts in Table 3 (1,359 total, 249 LM-related) differ from Table 4 (1,354 total, 244 LM-related). This is presumably due to flow deduplication described in §4.3, but the reconciliation is left implicit; add an explicit sentence.
  4. [Table 6] The standard deviations from 10 runs are reported, but no statistical test is used to support the ranking claims. In particular, in the LM-only AUC column, Euler (52.90±6.82) and Pikachu (50.74±0.81) overlap within one standard deviation; a paired test or confidence interval would strengthen the claim that the ranking changed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: re-evaluations use external ground truth and public detector implementations; manual OpTC labels are a robustness limitation, not a circular dependence.

full rationale

The paper's central contribution is a benchmark methodology: it proposes preprocessing and labeling policies justified by operational realism, then re-evaluates published detectors under those policies. The LANL results use the dataset's official labels (Section 5.2), which are external to this paper and not derived from detector outputs. The OpTC results use a hybrid label set described in Section 5.3, seeded by the process-based labels of Majorczyk et al. [21] and augmented by manual inspection guided by the red-team report. Those labels are constructed from system logs and documented red-team activity, not from the outputs of Pikachu, Euler, or Argus. No parameter is fitted to the evaluation labels, and no equation defines the predicted scores in terms of the ground-truth labels. The detectors are run from their authors' released code with the originally reported hyperparameters (Section 6), so the observed drops in AUC/AP and the OpTC ranking change are empirical outcomes that could in principle have gone the other way. There are no self-citations: the author is not among the referenced authors, and the borrowed process-based labeling idea is adopted as a starting point with explicit stated limitations rather than as an unexamined authority. The manual nature of the 249-event 'LM only' label set and the lack of sensitivity analysis are genuine reproducibility and robustness limitations, and the paper's own error bars (e.g., Euler LM-only AUC 52.90 ± 6.82) already signal instability, but this is a label-quality concern, not a circular argument. No step in the derivation chain reduces by construction to its own inputs, so the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger contains no invented physical entities. The main external inputs are the two public datasets, the official/red-team ground truth, and the three published detectors. The central evaluation depends on the hand-chosen LANL cutoff, the borrowed detector hyperparameters, and the manual OpTC label construction; these are the elements a reader would need to audit.

free parameters (2)
  • LANL train/test cutoff = 41 hours
    Hand-selected in Section 4.2 as training period because it ends before the first malicious event; different cutoffs change test-set size and class imbalance. It is a policy choice, not derived from operational requirements.
  • Detector hyperparameters (Pikachu, Euler, Argus) = Best-performing values reported in original papers; specific values not listed
    Section 6 says hyperparameters are set to the original best values. These numbers were fitted to data by the original authors and are load-bearing for the re-evaluation; if they are not transferable, part of the performance drop could be a hyperparameter artifact.
assumptions (4)
  • domain assumption LANL's official red-team labels are a sufficient ground truth for lateral movement evaluation despite being an unlabeled subset.
    Section 5.2 admits the labels are not explicitly qualified as lateral movement and represent only a subset of red-team events, then uses them as ground truth anyway.
  • ad hoc to paper The OpTC red-team report plus manual inspection of system logs yields correct event-level labels, including events not spawned by malicious processes.
    Section 5.3 constructs the hybrid label set through manual inspection; the accuracy of this process is assumed and not independently verified.
  • domain assumption Event-level metric computation on an identical test set is the correct basis for fair comparison.
    Section 4.1 argues edge-level metrics are contaminated by test-set composition; this is a methodological premise, not a proven theorem.
  • domain assumption Temporal separation with training before the first attack is the realistic evaluation protocol.
    Section 4.2 recommends stopping training before the first malicious event; this assumes a deployment where label timing is known.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On fair and realistic performance evaluations for graph-based lateral movement detectors." pith.science (2026). https://pith.science/paper/TOEZFWH7

@misc{pith2026260729390,
  author       = {Pith},
  title        = {Pith review of: On fair and realistic performance evaluations for graph-based lateral movement detectors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOEZFWH7}},
  note         = {Machine review of arXiv:2607.29390}
}
read the original abstract

Research on lateral movement detection has made significant progress in recent years, spurred by the widespread availability of benchmark datasets that make evaluating detectors practical. However, the exact way in which these benchmark datasets are used varies across the literature: both the preprocessing applied before feeding the data to the detector and the labeling of lateral movement-related events change substantially from one paper to another. We survey preprocessing and labeling methodologies for two popular datasets and demonstrate their impact on the fairness and realism of downstream evaluations. We also propose well-grounded preprocessing and labeling policies for these datasets. Finally, we re-evaluate three widely cited lateral movement detection methods under these new policies; our results differ significantly from those reported in the original papers, further highlighting the critical importance of dataset preprocessing and labeling practices in evaluating lateral movement detectors.

Figures

Figures reproduced from arXiv: 2607.29390 by the authors.

Figure 1
Figure 1. Three ways to turn a sequence of timestamped events into a graph. The width of each edge depends on the number of underlying events. (WMI), and network protocols designed to manage sessions on remote comput￾ers (e.g., RDP or SSH). Both the diversity of methods and the opportunity for attackers to leverage legitimate, frequently used software and protocols make rule-based detection of lateral movement difficult: many… view at source ↗
Figure 2
Figure 2. Influence of snapshot length on the statistical properties of discrete-time graphs: for the same event sequence, different snapshot lengths ∆1 and ∆2 result in edge sets of different size and class imbalance. edge level; that is, after merging all events with the same source, destination, and snapshot index into a single record. As a consequence, the number of benign and malicious test samples depends on snapshot le… view at source ↗
Figure 3
Figure 3. Performance of Argus and Euler on the LANL and OpTC datasets as a function of snapshot length, with the original preprocessing and after duplication of the merged events. The error bars represent the standard deviation over ten runs. Horizontal jitter added for readability. same argument can be made about the exclusion of events involving local, built￾in, and computer accounts: some of the malicious authentications … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Two examples of lateral movements missed by process-based labeling: (left) no system logs are available for the source host; (right) the process handling the network connection is not the malicious process. Both happen on 24 September. 5.3 Recommendations for the OpTC …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 1 canonical work pages

  1. [1]

    Lateral movement, tactic TA0008.https://attack.mitre.org/tactics/TA0008/, accessed: 2025-03-31

    2025

  2. [2]

    8(2), 404–415 (2020)

    Apruzzese, G., Pierazzi, F., Colajanni, M., Marchetti, M.: Detection and threat pri- oritizationofpivotingattacksinlargenetworks.IEEETrans.Emerg.Top.Comput. 8(2), 404–415 (2020)

    2020

  3. [3]

    In: USENIX Security (2022)

    Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., Rieck, K.: Dos and don’ts of machine learning in computer security. In: USENIX Security (2022)

    2022

  4. [4]

    Bouček, V.: Pikachu,https://github.com/vboucek/Pikachu, accessed: 2026-05- 20

    2026

  5. [5]

    In: CNSM (2025)

    Bouček, V., Husák, M.: Replication: Network-based lateral movement detection methods using machine learning. In: CNSM (2025)

    2025

  6. [6]

    In: RAID (2020)

    Bowman, B., Laprade, C., Ji, Y., Huang, H.H.: Detecting lateral movement in enterprise computer networks with unsupervised graph AI. In: RAID (2020)

    2020

  7. [7]

    Corso, G., Stark, H., Jegelka, S., Jaakkola, T., Barzilay, R.: Graph neural networks. Nat. Rev. Methods Primers4(1), 17 (2024)

    2024

  8. [8]

    DARPA: Operationally transparent cyber dataset (2020),https://github.com/ FiveDirections/OpTC-data, accessed: 2026-05-20

    2020

Show all 31 references
  1. [9]

    In: KDD (2016)

    Grover, A., Leskovec, J.: node2vec: Scalable feature learning for networks. In: KDD (2016)

    2016

  2. [10]

    GWU GraphLab: Picodomain (2020),https://github.com/iHeartGraph/ PicoDomain, accessed: 2026-05-20

    2020

  3. [11]

    Ho, G., Dhiman, M., Akhawe, D., Paxson, V., Savage, S., Voelker, G.M., Wagner, D.:Hopper:Modelinganddetectinglateralmovement.In:USENIXSecurity(2021)

    2021

  4. [12]

    Los Alamos Na- tional Laboratory (2015).https://doi.org/10.17021/1179829

    Kent, A.D.: Comprehensive, multi-source cyber-security events. Los Alamos Na- tional Laboratory (2015).https://doi.org/10.17021/1179829

    2015 doi

  5. [13]

    In: Dynamic Networks in Cybersecurity

    Kent, A.D.: Cybersecurity data sources for dynamic network research. In: Dynamic Networks in Cybersecurity. Imperial College Press (2015)

    2015

  6. [14]

    In: SP (2024)

    Khoury, J., Klisura, Ð., Zanddizari, H., Parra, G.D.L.T., Najafirad, P., Bou-Harb, E.: Jbeil: Temporal graph-based inductive learning to infer lateral movement in evolving enterprise networks. In: SP (2024)

    2024

  7. [15]

    King, I.J.: Euler,https://github.com/iHeartGraph/Euler, accessed: 2026-05-20

    2026

  8. [16]

    In: NDSS (2022)

    King, I.J., Huang, H.H.: EULER: Detecting network lateral movement via scalable temporal link prediction. In: NDSS (2022)

    2022

  9. [17]

    ACM Trans

    King, I.J., Huang, H.H.: Euler: Detecting network lateral movement via scalable temporal link prediction. ACM Trans. Priv. Secur.26(3), 35:1–35:36 (2023)

    2023

  10. [18]

    Technometrics64(2), 241–252 (2022)

    Lee, W., McCormick, T.H., Neil, J., Sodja, C., Cui, Y.: Anomaly detection in large- scale networks with latent space models. Technometrics64(2), 241–252 (2022)

    2022

  11. [19]

    In: TrustCom (2020)

    Liu, F., Wen, Y., Wu, Y., Liang, S., Jiang, X., Meng, D.: MLTracer: Malicious logins detection system via graph neural network. In: TrustCom (2020)

    2020

  12. [20]

    In: MILCOM (2018)

    Liu, Q., Stokes, J.W., Mead, R., Burrell, T., Hellen, I., Lambert, J., Marochko, A., Cui, W.: Latte: Large-scale lateral movement detection. In: MILCOM (2018)

    2018

  13. [21]

    In: ACSAC Workshops (2025)

    Majorczyk, F., Pilastre, B., Dijoud, F.: A new hope for DARPA OpTC. In: ACSAC Workshops (2025)

    2025

  14. [22]

    In: CIoT (2024)

    Nikulshin, V., Talhi, C.: Effective IDS under constraints of modern enterprise net- works: revisiting the OpTC dataset. In: CIoT (2024)

    2024

  15. [23]

    In: NOMS (2022) 18 C

    Paudel,R.,Huang,H.H.:Pikachu:Temporalwalkbaseddynamicgraphembedding for network anomaly detection. In: NOMS (2022) 18 C. Larroche

    2022

  16. [24]

    arXiv preprint arXiv:2006.10637 (2020)

    Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., Bronstein, M.: Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 (2020)

    2006 arXiv

  17. [25]

    In: IPCCC (2022)

    Sun, X., Yang, J.: HetGLM: Lateral movement detection by discovering anomalous links with heterogeneous graph neural network. In: IPCCC (2022)

    2022

  18. [26]

    Tang, M., Ge, B.: Social-Hunter: A social heuristics-based approach to early un- veiling unknown malicious logins using valid accounts. Comput. Secur. p. 104269 (2024)

    2024

  19. [27]

    In: ACSAC (2025)

    Wang, C., Zheng, P., Gui, J., Hua, C., Hassan, W.U.: R+R: from claims to crashes: A systematic re-evaluation of graph-based network intrusion detection systems. In: ACSAC (2025)

    2025

  20. [28]

    In: ICICS (2019)

    Wei, R., Cai, L., Yu, A., Meng, D.: AGE: Authentication graph embedding for detecting anomalous login activities. In: ICICS (2019)

    2019

  21. [29]

    Xu, J.: Argus,https://github.com/C0ldstudy/Argus, accessed: 2026-05-20

    2026

  22. [30]

    In: SP (2024)

    Xu, J., Shu, X., Li, Z.: Understanding and bridging the gap between unsupervised network representation learning and security analytics. In: SP (2024)

    2024

  23. [31]

    In: ICICS (2019)

    Zhao, S., Wei, R., Cai, L., Yu, A., Meng, D.: CTLMD: Continuous-temporal lateral movement detection using graph embedding. In: ICICS (2019)

    2019

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.