Pith. sign in

REVIEW 3 major objections 4 minor 60 references

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On three of four audited E3 datasets, a simple allowlist of known executables matches or beats learned intrusion-detection baselines, indicating measured gains often reflect lexical novelty.

desk verdict A serious, mostly reproducible PIDS evaluation with a clean allowlist finding; the Theia exception and the Q-based explanation rest on hyperparameters selected using test labels, so the paper needs a revision before its headline architectural claim is protocol-clean. read the letter →

arxiv 2608.01454 v1 pith:G6YT3CTR submitted 2026-08-02 cs.CR cs.LG

Lorenzo Guerra , Thomas Chapuis , Guillaume Duc , Pavlo Mozharovskyi , Van-Tam Nguyen This is my paper
classification cs.CRcs.LG
keywords provenance-basedintrusiondetectionbenchmarkevaluationE3provenancedatasetssemanticsignalqualitylexicalnoveltyallowlistbaselinealertingvsinvestigationmetricsgraphneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reported results in provenance-based intrusion detection describe the benchmark and evaluation protocol at least as much as the detector architecture. Re-running representative learned systems on an audited subset of the E3 provenance family under a strict protocol — temporally separated test periods, validation-only checkpoint and threshold calibration, multi-seed reporting — it finds that high alerting scores do not translate into process-level investigation utility. On Cadets, FiveDirections, and Trace, a zero-parameter allowlist that flags executables unseen in training matches or beats the learned baselines on several operating-point metrics, suggesting much of the measured performance comes from lexical novelty. On Theia, the only dataset with high semantic signal quality $Q = C \cdot H$ (feature completeness times field entropy), the paper's reference model Theseus clearly improves both ranking and node recovery. A sympathetic reader should take away that architectural claims in this field are interpretable only together with the dataset signal and calibration choices that produced them.

What carries the argument

The load-bearing object is the semantic signal quality diagnostic $Q=C\cdot H$, where $C$ is the fraction of applicable semantic fields (process path, command line, file path, network-flow description) that are populated and $H$ is a corpus-level token-entropy average across those fields. It is paired with an auditable evaluation protocol: shared process-level labels with artifact exclusions, strict temporal splits, validation-only checkpoint and threshold selection, and multi-seed reporting. Theseus, the reference model, is a two-stage architecture: a one-hop E-GraphSAGE encoder over windowed heterogeneous provenance graphs followed by a Transformer autoencoder that reconstructs node embedd

What would settle it

Construct or find a benchmark whose process and file fields are nearly complete and diverse (high $Q$) yet where a zero-parameter executable allowlist still matches the best learned model on node-level recovery; the paper predicts such a dataset would expose architecture differences, so the opposite result would falsify the $Q$-drives-separation claim. A more direct check: on Theia, keep attacks and graph structure fixed but progressively strip semantic fields to lower $Q$; the claim implies Theseus's advantage should shrink and disappear as $Q$ falls.

Watch

Extended reading notes

Core claim

Under one shared protocol, the paper establishes three things. First, alerting success (ADP or precision at a fixed threshold) and investigation utility (AP, F1, MCC at the selected operating point) can diverge sharply: Velox reaches 0.883 precision and 0.870 ADP on Cadets while its F1 is 0.007. Second, a deterministic allowlist over executable names and paths observed in training reproduces much of the fixed-threshold behavior of learned models on Cadets, FiveDirections, and Trace, indicating that lexical novelty rather than richer provenance modeling explains much of the reported gains. Third, the datasets differ in how much usable semantic evidence they expose, quantified by $Q=C\cdot H$;

Load-bearing premise

The load-bearing premise is that $Q=C\cdot H$ measures the signal a richer model can exploit, so the ordering of $Q$ explains which datasets separate architectures; if $Q$ is confounded with dataset size, attack type, graph topology, or the graph-construction choices made here, the explanation for Theia being the exception could be wrong — and the paper itself concedes it cannot fully isolate that contribution.

Editorial extensions

If this is right

  • Reporting only ADP or precision can make a system look capable of forensic investigation when it recovers almost none of the attack footprint.
  • A zero-parameter allowlist should be reported alongside learned baselines as a control, because it captures the novelty signal that appears to drive many results.
  • Dataset-level $Q$ values order the observed architectural separation, so benchmark releases should document feature completeness and field entropy.
  • Richer architecture helps where the data actually carries the signal (Theia AP 0.671 vs best baseline 0.062), so claims about model value require benchmark signal, not just model design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the allowlist result makes a cheap universal control obvious — every new PIDS evaluation should report the executable/path novelty baseline before claiming architectural gains; most current papers do not.
  • Editorial extension: $Q$ as defined depends on the graph-construction pipeline, so it should be reported as a property of the processed benchmark release, not the raw logs; different pipelines could reorder $Q$ and change which dataset appears rich.
  • Editorial extension: if the $Q$ ordering generalizes, benchmark designers could use completeness and entropy as release criteria, discarding or augmenting low-$Q$ datasets before running expensive model comparisons.
  • Editorial extension: the paper's process-level scope conflates benign and malicious actions within one process; edge-level labels, where available, would let the allowlist-vs-learning question be tested at finer granularity and could change the measured divergence between alerting and investigation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a strict evaluation protocol for provenance-based intrusion detection systems, with temporal splits, validation-only checkpoint and threshold calibration, artifact exclusions, and multi-seed reporting, and applies it to four audited DARPA TC E3 datasets plus ATLASv2 as secondary evidence. It compares three learned baselines (Velox, Orthrus, Magic) with a zero-parameter allowlist over executable names/paths seen in training, and introduces a reference model, Theseus, to probe whether additional architectural capacity helps when datasets contain richer semantic signal. The central empirical finding is that on Cadets, FiveDirections, and Trace the allowlist matches or exceeds the learned baselines on key operating-point metrics, suggesting that much of the measured performance reflects lexical novelty. On Theia, Theseus shows substantially stronger ranking and node-level recovery, which the paper attributes to higher semantic signal quality Q = C·H (feature completeness × field entropy). The paper concludes that architectural claims from PIDS evaluations must be interpreted together with benchmark properties and the evaluation protocol.

Significance. If the central allowlist result holds, it is an important and sobering finding for the PIDS evaluation community: a deterministic lexical baseline can match or outperform learned models on three of the four audited datasets, and the shared protocol cleanly demonstrates divergence between alerting and investigation utility. The paper’s methodological contributions are real strengths: public code and artifacts, multi-seed reporting, validation-only threshold calibration, artifact-aware scoring exclusions, and threshold-sensitivity analysis. The Theia exception and the semantic-signal-quality explanation are interesting but are not currently protocol-clean, because the reference model’s configuration appears to be selected on Theia test labels. The broad benchmark-design message is valuable, but the specific claim that richer modeling succeeds when semantic signal is strong is not yet established at the level the abstract suggests.

major comments (3)
  1. [§V-D1, §V-D2, Appendix C-B] The final Theseus configuration appears to be selected using Theia test labels. Table VII chooses the 0.40 attention mask ratio because "test AP is highest" (0.671), and Table VIII selects the reverse network-flow weighting because it yields the "highest test AP and MCC." Appendix C-B states that final configurations were found by "Bayesian optimization targeting AP" without specifying which split the optimization used. If these choices used the held-out test period, the reported Theia advantage (AP 0.671 vs Velox 0.028) is not a clean evaluation under the paper’s own no-look-ahead principle (Section IV-B), and the subsequent semantic-signal-quality explanation (Table V, Section V-B) is partly an artifact of test-set selection. The allowlist finding on Cadets, FiveDirections, and Trace is independent of this issue, but the abstract’s claim that Theia is the exception and the conclusion t
  2. [§V-B, Table V] Even setting aside the test-set selection issue, the semantic-signal-quality explanation rests on a single exceptional dataset: Q has only four data points, and the ordering is compatible with many confounds, including graph-construction choices, the T3-only restriction for Theia, attack structure, and workload differences. The paper itself concedes in Section V that "differences in graph structure, workloads, attacks, and background activity prevent us from fully isolating its contribution," and Section VI reiterates that Q is not a general predictor. Given the hyperparameter-selection problem in the previous comment, the abstract and conclusion should not present Q as a demonstrated explanation of which benchmarks separate architectures. The diagnostic should be described as a hypothesis-generating observation, with a concrete route to falsification (e.g., more datasets with comparable
  3. [§IV-B, Table XIII] The split schedule for Theia and Trace is non-chronological within the development period: for Theia T3, the validation day (Day 10) precedes the training day (Day 11), and for Trace validation Day 10 is used with training Days 9 and 11. The paper discloses this and keeps the pre-test pool disjoint from the test period, so there is no direct test leakage. However, the abstract’s phrase "temporally separated test periods" and the protocol description in Section IV-B may give readers the impression of a monotonic train→validate→test simulation. If the intended regime is not deployment-ordered validation, the text should state this limitation explicitly and explain how checkpoint selection on a validation period that is earlier than part of the training period preserves the deployment-relevant no-look-ahead property.
minor comments (4)
  1. [Abstract and §V-A2] The phrase "matches or exceeds the selected learned baselines on key operating-point metrics" should be made metric-specific. For example, on Cadets the allowlist does not match Velox’s precision (0.429 vs 0.883), but it leads F1 and MCC; on Theia it leads all baselines except Theseus. A short sentence listing which metrics are referred to would prevent overclaiming.
  2. [§V-D1] The sentence "At the reported mask ratio of 0.40, test AP is highest" is in tension with the paper’s protocol. If the reported mask ratio was actually selected on validation data, the text should say so; if it was selected on test data, Table VII should be labeled as a test-set exploration rather than a protocol-compliant choice.
  3. [§V-E, Table IX] ATLASv2 is explicitly presented as corroborative, but the comparison of Theseus and Velox on h2 is mixed (Theseus higher AP on h1, Velox higher AP on h2). This is fine as reported, but the sentence "The fact that Theseus leads in AP only on the higher-entropy host is compatible with the semantic-signal interpretation" should be flagged more clearly as a weak, two-host correlation rather than supporting evidence.
  4. [§IV-D] The notation Q=C·H is introduced as a diagnostic, but later text sometimes refers to it as a property of datasets. Consider renaming it "semantic-signal diagnostic index" or adding a one-line reminder in Section V-B that Q is computed on the processed node tables and therefore reflects graph-construction choices as well as raw data quality.

Circularity Check

1 steps flagged · score 6.0 of 10

Theseus hyperparameters are selected on Theia test AP, so the 'Theia exception' and the Q-based semantic-signal explanation are partly forced by construction; the allowlist finding on the other three datasets remains independent.

  1. fitted input called prediction [Section V-D1 (Table VII), V-D2 (Table VIII), Appendix C-B (Table XVI)]
    "Appendix C-B: 'We developed these configurations through a limited coarse random search followed by Bayesian optimization targeting AP.' Section V-D1: 'At the reported mask ratio of 0.40, test AP is highest at 0.671. ... we use the 0.40 mask ratio because it gives the best threshold-independent ranking performance.' Section V-D2: 'The reported reverse network-flow weighting yields the highest test AP and MCC in this ablation.'"

    Theia is the exception that motivates the semantic-signal-quality explanation (Q=6.72 in Table V), but Theseus's configuration is fitted to Theia test AP. Appendix C-B says 'Bayesian optimization targeting AP' with no stated restriction to validation; V-D1 explicitly chooses mask ratio 0.40 because test AP is highest (0.671); V-D2 chooses the 'Reverse network' weighting because it yields the highest test AP and MCC. These are test-label-driven model selections, not protocol-clean validation choices under the paper's own no-look-ahead rule. The resulting Theia advantage is therefore partly manufactured by the fitting target, and the claim that Theia exposes semantic signal 'that Theseus can exploit' is a fitted input called evidence rather than an independent prediction. The allowlist findi

full rationale

We found one substantive circular step: the Theia exception—the sole dataset where Theseus beats baselines and the basis for the semantic-signal-quality explanation—is produced by a model whose key configuration choices (attention mask ratio, token weighting, structural features) were selected on Theia test AP/MCC, as stated in the ablations and Appendix C-B. Calling that advantage evidence that 'semantic signal' explains architectural separation is a fitted input called a prediction: the selection criterion and the explained outcome are the same test-period metric. This does not affect the allowlist diagnostic on Cadets/FiveDirections/Trace, which is deterministic and independent of any fitted parameter, nor is Q itself circular since the paper explicitly excludes it from training and calibration (Section IV-D). The other citations, including the authors' prior NeurIPS paper, are descriptive rather than load-bearing; no uniqueness theorem or ansatz is imported by self-citation. Score 6: partial circularity concentrated in the Theia/semantic-signal claim.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim (allowlist matches baselines) does not depend on fitted parameters, so the ledger is relatively light. However, the supporting Theseus result depends on several hand-selected hyperparameters (mask ratio, token weighting, structural features) that appear to have been optimized against test performance. The most important domain assumptions are the correctness of the REAPr/Liu labels and the representativeness of the graph abstraction and validation scheme.

free parameters (4)
  • Theseus attention mask ratio = 0.40 on Theia
    Selected to maximize test AP on Theia (Section V-D1, Table VII); if selected on test rather than validation, this is a protocol leak affecting the Theseus-on-Theia result.
  • Theseus token weighting scheme = Reverse network-flow weighting on Theia
    Chosen because it yields the highest test AP/MCC in Table VIII; the selection basis (validation vs test) is not explicitly stated.
  • Structural feature inclusion per dataset = On for Cadets, FiveDirections, Trace; off for Theia; on for ATLASv2
    Chosen per dataset via hyperparameter search targeting AP (Appendix C-B); affects the conclusion that structural features help on low-Q datasets.
  • Theseus hyperparameters (learning rates, dropout, dims, etc.) = See Table XVI
    Found via coarse random search and Bayesian optimization targeting AP (Appendix C-B); the data split used for selection is not stated.
assumptions (5)
  • domain assumption REAPr process-level labels plus Liu et al.'s artifact exclusions correctly identify attack-relevant processes in E3 and ATLASv2
    All metrics are computed against this ground truth; incorrect labels would change scores. Cited from [19], [10].
  • domain assumption The graph abstraction (processes, files, network flows) preserves the causal information needed for PIDS detection and scoring
    The paper evaluates only process nodes but uses all node types in message passing; Section VII-A reports up to 59% unlinkable events, so the abstraction may drop security-relevant information.
  • ad hoc to paper Semantic signal quality Q = C·H over processed node tables captures the signal available for architectural comparison
    New diagnostic introduced in Section IV-D and used to explain cross-dataset differences; it is not independently validated and the paper acknowledges it is not a general predictor.
  • domain assumption Validation-only calibration with the threshold set to the maximum benign validation score is a fair operating point for all systems
    The paper replaces clustering with validation thresholding for Orthrus and uses this policy to compare systems; if the policy disadvantages certain score distributions, comparisons could be unfair.
  • ad hoc to paper Attack-containing validation days used for checkpoint selection do not leak test information even when they interleave with training days
    The paper relaxes temporal ordering (Appendix B-B), e.g., Theia uses Day 10 (attack) for validation and Day 11 for training, so validation contains attacks that occur before some training data; the assumption is that this does not bias checkpoint selection.
invented entities (1)
  • Theseus reference model independent evidence
    purpose: Probe model to test whether added capacity improves detection when semantic signal is rich
    New architecture introduced in Section IV-F; publicly released code provides a falsifiable handle, but its hyperparameter selection has a potential test-set leak (Section V-D1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection." pith.science (2026). https://pith.science/paper/G6YT3CTR

@misc{pith2026260801454,
  author       = {Pith},
  title        = {Pith review of: How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6YT3CTR}},
  note         = {Machine review of arXiv:2608.01454}
}
read the original abstract

Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. On three of the four primary datasets, a simple allowlist built from training executable names and paths matches or exceeds the selected learned baselines on key operating-point metrics, suggesting that much of their measured performance reflects lexical novelty rather than richer provenance modeling. To explain why only some datasets expose architectural differences, we measure semantic signal quality through feature completeness and field entropy. This analysis helps explain why several audited E3 datasets can expose alerting behavior without reliably separating model architectures, while Theia pairs the strongest semantic signal quality with the clearest improvements in ranking and node-level recovery by our reference model. These results show that architectural claims in PIDS should be interpreted together with the benchmark properties and evaluation protocol that produced them.

Figures

Figures reproduced from arXiv: 2608.01454 by the authors.

Figure 1
Figure 1. Theseus, the paper’s reference architecture for richer modeling. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Threshold sensitivity on Theia for frozen Theseus and Velox models. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Feature completeness and field entropy in the evaluated E3 datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Anomaly-score distributions by attack type for Cadets. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Anomaly-score distributions by attack type for FiveDirections. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Anomaly-score distributions by attack type for Trace. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Anomaly-score distributions by attack type for Theia. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Difference in AP after adding structural features ( [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Illustrative example of evaluation scopes. The current process-level [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Threshold sensitivity on DARPA TC E3 for frozen Theseus and Velox models. Curves and annotated ADP values are means over five seeds, and [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 44 canonical work pages

  1. [1]

    Provenance-based in- trusion detection systems: A survey,

    M. Zipperle, F. Gottwalt, E. Chang, and T. Dillon, “Provenance-based in- trusion detection systems: A survey,”ACM Computing Surveys, vol. 55, no. 7, pp. 1–36, 2022

    work page 2022

  2. [2]

    Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,

    F. Dong, S. Li, P. Jiang, D. Li, H. Wang, L. Huang, X. Xiao, J. Chen, X. Luo, Y . Guoet al., “Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communica- tions Security, 2023, pp. 2396–2410

    work page 2023

  3. [3]

    SLEUTH: Real-time attack scenario reconstruction from COTS audit data,

    M. N. Hossain, S. M. Milajerdi, J. Wang, B. Eshete, R. Gjomemo, R. Sekar, S. Stoller, and V . Venkatakrishnan, “SLEUTH: Real-time attack scenario reconstruction from COTS audit data,” in26th USENIX Security Symposium (USENIX Security 17). USENIX Association, 2017, pp. 487–504

    work page 2017

  4. [4]

    HOLMES: Real-time APT detection through correlation of suspicious information flows,

    S. M. Milajerdi, R. Gjomemo, B. Eshete, R. Sekar, and V . Venkatakr- ishnan, “HOLMES: Real-time APT detection through correlation of suspicious information flows,” in2019 IEEE Symposium on Security and Privacy (SP). IEEE, 2019, pp. 1137–1152

    work page 2019

  5. [5]

    Graph neural networks for intrusion detection: A survey,

    T. Bilot, N. El Madhoun, K. Al Agha, and A. Zouaoui, “Graph neural networks for intrusion detection: A survey,”IEEE Access, vol. 11, pp. 49 114–49 139, 2023

    work page 2023

  6. [6]

    KAIROS: Practical intrusion detection and investigation using whole- system provenance,

    Z. Cheng, Q. Lv, J. Liang, Y . Wang, D. Sun, T. Pasquier, and X. Han, “KAIROS: Practical intrusion detection and investigation using whole- system provenance,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024

    work page 2024

  7. [7]

    THREATRACE: Detecting and tracing host-based threats in node level through provenance graph learning,

    S. Wang, Z. Wang, T. Zhou, H. Sun, X. Yin, D. Han, H. Zhang, X. Shi, and J. Yang, “THREATRACE: Detecting and tracing host-based threats in node level through provenance graph learning,”IEEE Transactions on Information Forensics and Security, vol. 17, pp. 3972–3987, 2022. 14

    work page 2022

  8. [8]

    On the reproducibility of provenance-based intrusion detection that uses deep learning,

    T. Abrar, A. Shamail, M. J. Iqbal, A. Ahmed, M. Abdullah, M. Shayan, F. Zaffar, T. Pasquier, D. Eyers, and A. Gehani, “On the reproducibility of provenance-based intrusion detection that uses deep learning,” in Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ser. ACM REP ’25. New York, NY , USA: Association for Computing Machin...

    work page arXiv 2025

Show all 60 references
  1. [9]

    Sometimes simpler is better: A comprehensive analysis of state-of-the-art provenance-based intrusion detection systems,

    T. Bilot, B. Jiang, Z. Li, N. El Madhoun, K. Al Agha, A. Zouaoui, and T. Pasquier, “Sometimes simpler is better: A comprehensive analysis of state-of-the-art provenance-based intrusion detection systems,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 7193– 7212

    2025

  2. [10]

    What we talk about when we talk about logs: Understanding the effects of dataset quality on endpoint threat detection research,

    J. Liu, M. A. Inam, A. Goyal, A. Riddle, K. Westfall, and A. Bates, “What we talk about when we talk about logs: Understanding the effects of dataset quality on endpoint threat detection research,” in2025 IEEE Symposium on Security and Privacy (SP), 2025, pp. 112–129

    2025

  3. [11]

    On the forensic validity of approximated audit logs,

    N. Michael, J. Mink, J. Liu, S. Gaur, W. U. Hassan, and A. Bates, “On the forensic validity of approximated audit logs,” inProceedings of the 36th Annual Computer Security Applications Conference, ser. ACSAC ’20. New York, NY , USA: Association for Computing Machinery, 2020, p...

    2020

  4. [12]

    Shortcut learning in deep neural networks,

    R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,”Nature Machine Intelligence, vol. 2, no. 11, pp. 665–673, 2020

    2020

  5. [13]

    ORTHRUS: Achieving high quality of attribution in provenance-based intrusion detection systems,

    B. Jiang, T. Bilot, N. El Madhoun, K. Al Agha, A. Zouaoui, S. Iqbal, X. Han, and T. Pasquier, “ORTHRUS: Achieving high quality of attribution in provenance-based intrusion detection systems,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 7173– 7192

    2025

  6. [14]

    NoDoze: Combatting threat alert fatigue with automated provenance triage,

    W. U. Hassan, S. Guo, D. Li, Z. Chen, K. Jee, Z. Li, and A. Bates, “NoDoze: Combatting threat alert fatigue with automated provenance triage,” in26th Annual Network and Distributed System Security Sym- posium (NDSS), 2019

    2019

  7. [15]

    Back-Propagating system dependency impact for attack investigation,

    P. Fang, P. Gao, C. Liu, E. Ayday, K. Jee, T. Wang, Y . F. Ye, Z. Liu, and X. Xiao, “Back-Propagating system dependency impact for attack investigation,” in31st USENIX Security Symposium (USENIX Security 22). Boston, MA: USENIX Association, Aug. 2022, pp. 2461–2478

    2022

  8. [16]

    NODLINK: An online system for fine-grained apt attack detection and investigation,

    S. Li, F. Dong, X. Xiao, H. Wang, F. Shao, J. Chen, Y . Guo, X. Chen, and D. Li, “NODLINK: An online system for fine-grained apt attack detection and investigation,” in31st Annual Network and Distributed System Security Symposium (NDSS), 2024

    2024

  9. [17]

    Dos and don’ts of machine learning in computer security,

    D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wress- negger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 3971–3988

    2022

  10. [18]

    Transparent computing engagement 3 data release,

    DARPA, “Transparent computing engagement 3 data release,” https://github.com/darpa-i2o/Transparent-Computing/blob/master/ README-E3.md, 2018, commit:e94c9f2. Accessed: December 2025

    2018

  11. [19]

    REAPr: Recovery every attack process,

    J. Liu, M. A. Inam, A. Goyal, K. Westfall, A. Riddle, and A. Bates, “REAPr: Recovery every attack process,” https://bitbucket.org/sts-lab/ reapr-ground-truth, 2023, commit:e726c01. Accessed: July 2026

    2023

  12. [20]

    ATLASv2: Atlas attack engagements, version 2,

    A. Riddle, K. Westfall, and A. Bates, “ATLASv2: Atlas attack engagements, version 2,” 2023. [Online]. Available: https://arxiv.org/ abs/2401.01341

    2023 arXiv

  13. [21]

    How to effectively trace provenance on windows endpoint detection & response telemetry,

    J. Liu, M. A. Inam, A. Goyal, D. Greenenwald, A. Bates, and S. Chittal, “How to effectively trace provenance on windows endpoint detection & response telemetry,” inWorkshop on Attack Provenance, Reasoning, and Investigation for Security in the Monitored Environment (PRISM’26), 2026

    2026

  14. [22]

    SPADE: Support for provenance auditing in distributed environments,

    A. Gehani and D. Tariq, “SPADE: Support for provenance auditing in distributed environments,” inACM/IFIP/USENIX International Confer- ence on Distributed Systems Platforms and Open Distributed Processing. Springer, 2012, pp. 101–120

    2012

  15. [23]

    Practical whole-system provenance capture,

    T. Pasquier, X. Han, M. Goldstein, T. Moyer, D. Eyers, M. Seltzer, and J. Bacon, “Practical whole-system provenance capture,” inProceedings of the ACM Symposium on Cloud Computing (SoCC ’17). ACM, 2017

    2017

  16. [24]

    Trustworthy Whole- System provenance for the linux kernel,

    A. Bates, D. J. Tian, K. R. Butler, and T. Moyer, “Trustworthy Whole- System provenance for the linux kernel,” in24th USENIX Security Symposium (USENIX Security 15). Washington, D.C.: USENIX Association, Aug. 2015, pp. 319–334

    2015

  17. [25]

    Loggc: garbage collecting audit log,

    K. H. Lee, X. Zhang, and D. Xu, “Loggc: garbage collecting audit log,” inProceedings of the 2013 ACM SIGSAC Conference on Computer and Communications Security, 2013, pp. 1005–1016

    2013

  18. [26]

    High accuracy attack provenance via binary-based execution partition,

    ——, “High accuracy attack provenance via binary-based execution partition,” in20th Annual Network and Distributed System Security Symposium (NDSS), vol. 16, 2013

    2013

  19. [27]

    High fidelity data reduction for big data security dependency analyses,

    Z. Xu, Z. Wu, Z. Li, K. Jee, J. Rhee, X. Xiao, F. Xu, H. Wang, and G. Jiang, “High fidelity data reduction for big data security dependency analyses,” inProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’16. New York, NY , USA: Ass...

    2016

  20. [28]

    Towards a timely causality analysis for enterprise security,

    Y . Liu, M. Zhang, D. Li, K. Jee, Z. Li, Z. Wu, J. Rhee, and P. Mittal, “Towards a timely causality analysis for enterprise security,” in25th Annual Network and Distributed System Security Symposium (NDSS), 2018

    2018

  21. [29]

    W ATSON: Abstracting behaviors from audit logs via aggregation of contextual semantics,

    J. Zeng, Z. L. Chua, Y . Chen, K. Ji, Z. Liang, and J. Mao, “W ATSON: Abstracting behaviors from audit logs via aggregation of contextual semantics,” in28th Annual Network and Distributed System Security Symposium (NDSS), 2021

    2021

  22. [30]

    DeepLog: Anomaly detection and diagnosis from system logs through deep learning,

    M. Du, F. Li, G. Zheng, and V . Srikumar, “DeepLog: Anomaly detection and diagnosis from system logs through deep learning,” inProceedings of the 2017 ACM SIGSAC Conference on Computer and Communica- tions Security. ACM, 2017, pp. 1285–1298

    2017

  23. [31]

    Logbert: Log anomaly detection via bert,

    H. Guo, S. Yuan, and X. Wu, “Logbert: Log anomaly detection via bert,” in2021 international joint conference on neural networks (IJCNN). IEEE, 2021, pp. 1–8

    2021

  24. [32]

    Unicorn: Runtime provenance-based detector for advanced persistent threats,

    X. Han, T. Pasquier, A. Bates, J. Mickens, and M. Seltzer, “Unicorn: Runtime provenance-based detector for advanced persistent threats,” in 27th Network and Distributed System Security Symposium (NDSS), 2020

    2020

  25. [33]

    You are what you do: Hunting stealthy malware via data provenance analysis,

    Q. Wang, W. U. Hassan, D. Li, K. Jee, X. Yu, K. Zou, J. Rhee, Z. Chen, W. Cheng, C. A. Gunter, and H. Chen, “You are what you do: Hunting stealthy malware via data provenance analysis,” in27th Annual Network and Distributed System Security Symposium (NDSS), 2020

    2020

  26. [34]

    MAGIC: Detecting advanced persistent threats via masked graph representation learning,

    Z. Jia, Y . Xiong, Y . Nan, Y . Zhang, J. Zhao, and M. Wen, “MAGIC: Detecting advanced persistent threats via masked graph representation learning,” in33rd USENIX Security Symposium, USENIX Security 2024, 2024

    2024

  27. [35]

    Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,

    M. U. Rehman, H. Ahmadi, and W. U. Hassan, “Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 3552–3570

    2024

  28. [36]

    R-caid: Embedding root cause analysis within provenance-based intrusion detection,

    A. Goyal, G. Wang, and A. Bates, “R-caid: Embedding root cause analysis within provenance-based intrusion detection,” in2024 IEEE Symposium on Security and Privacy (SP), 2024, pp. 3515–3532

    2024

  29. [37]

    The relationship between precision-recall and roc curves,

    J. Davis and M. Goadrich, “The relationship between precision-recall and roc curves,” inProceedings of the 23rd International Conference on Machine Learning, 2006, pp. 233–240

    2006

  30. [38]

    The precision-recall plot is more informa- tive than the ROC plot when evaluating binary classifiers on imbalanced datasets,

    T. Saito and M. Rehmsmeier, “The precision-recall plot is more informa- tive than the ROC plot when evaluating binary classifiers on imbalanced datasets,”PloS one, vol. 10, no. 3, p. e0118432, 2015

    2015

  31. [39]

    The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,

    D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,”BMC genomics, vol. 21, no. 1, p. 6, 2020

    2020

  32. [40]

    TESSERACT: Eliminating experimental bias in malware classification across space and time (extended version),

    Z. Kan, S. McFadden, D. Arp, F. Pendlebury, R. Jordaney, J. Kinder, F. Pierazzi, and L. Cavallaro, “TESSERACT: Eliminating experimental bias in malware classification across space and time (extended version),”

  33. [41]

    SoK: Pragmatic assessment of machine learning for network intrusion detection,

    G. Apruzzese, P. Laskov, and J. Schneider, “SoK: Pragmatic assessment of machine learning for network intrusion detection,” in2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 2023, pp. 592–614

    2023

  34. [42]

    Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging,

    N. Reimers and I. Gurevych, “Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging,” inProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 338–348

    2017

  35. [43]

    Accounting for variance in machine learning benchmarks,

    X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V . V oleti, S. Ebrahimi Kahou, V . Michalski, T. Arbel, C. Pal, G. Varoquaux, and P. Vincent, “Accounting for variance in machine learning benchmarks,” inP...

    2021

  36. [44]

    ATLAS: A sequence-based learning approach for attack investigation,

    A. Alsaheel, Y . Nan, S. Ma, L. Yu, G. Walkup, Z. B. Celik, X. Zhang, and D. Xu, “ATLAS: A sequence-based learning approach for attack investigation,” in30th USENIX Security Symposium (USENIX Security 21). USENIX Association, Aug. 2021, pp. 3005–3022

    2021

  37. [45]

    A new hope for darpa optc,

    F. Majorczyk, B. Pilastre, and F. Dijoud, “A new hope for darpa optc,” in2025 Annual Computer Security Applications Conference Workshops (ACSAC Workshops). IEEE, 2025, pp. 551–561

    2025

  38. [46]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,”The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948

    1948

  39. [47]

    Survivalism: Systematic analysis of Windows malware living- off-the-land,

    F. Barr-Smith, X. Ugarte-Pedrero, M. Graziano, R. Spolaor, and I. Mar- tinovic, “Survivalism: Systematic analysis of Windows malware living- off-the-land,” in2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021, pp. 1557–1574

    2021

  40. [48]

    Self-supervised learning of graph representations for network intrusion detection,

    L. Guerra, T. Chapuis, G. Duc, P. Mozharovskyi, and V .-T. Nguyen, “Self-supervised learning of graph representations for network intrusion detection,” inAdvances in Neural Information Processing Systems, vol. 38. Curran Associates, Inc., 2025, pp. 109 471–109 501. [Online]. A...

    2025

  41. [49]

    E- GraphSAGE: A graph neural network based intrusion detection system for IoT,

    W. W. Lo, S. Layeghy, M. Sarhan, M. Gallagher, and M. Portmann, “E- GraphSAGE: A graph neural network based intrusion detection system for IoT,” inNOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium, 2022, pp. 1–9

    2022

  42. [50]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008

    2017

  43. [51]

    Efficient estimation of word representations in vector space,

    T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,”arXiv preprint arXiv:1301.3781, 2013

    2013 arXiv

  44. [52]

    PIDSMaker: An ml framework for building provenance- based intrusion detection systems,

    T. Bilotet al., “PIDSMaker: An ml framework for building provenance- based intrusion detection systems,” https://github.com/ubc-provenance/ PIDSMaker, 2025, commit:2a73886. Accessed: December 2025

    2025

  45. [53]

    MAGIC: Official implementation,

    Z. Jiaet al., “MAGIC: Official implementation,” https://github.com/ FDUDSDE/MAGIC, 2024, commit:aa0b647. Accessed: December 2025

    2024

  46. [54]

    PyTorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “PyTorch: An imperative style, high-performance deep learning library,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019

    2019

  47. [55]

    Fast graph representation learning with PyTorch Geometric,

    M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” inICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. [Online]. Available: https://arxiv.org/abs/1903.02428

    2019 arXiv

  48. [56]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  49. [57]

    Datasheets for datasets,

    T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, “Datasheets for datasets,”Communications of the ACM, vol. 64, no. 12, pp. 86–92, 2021

    2021

  50. [58]

    The Menlo Report: Ethical principles guiding information and communication technology research,

    D. Dittrich and E. Kenneally, “The Menlo Report: Ethical principles guiding information and communication technology research,” U.S. Department of Homeland Security, Tech. Rep., 2012

    2012

  51. [59]

    Fast memory-efficient anomaly detection in streaming heterogeneous graphs,

    E. Manzoor, S. M. Milajerdi, and L. Akoglu, “Fast memory-efficient anomaly detection in streaming heterogeneous graphs,” inProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY , USA: Association for Comp...

    2016

  52. [2025]

    Available: https://arxiv.org/abs/2402.01359

    [Online]. Available: https://arxiv.org/abs/2402.01359

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.