Pith. sign in

REVIEW 3 major objections 4 minor 97 references

From Alerts to Intelligence: A Novel LLM-Aided Framework for Host-based Intrusion Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes SHIELD, an LLM-aided host intrusion detection pipeline that claims very high precision across three log datasets while also producing tactic-level and story-level attack intelligence.

desk verdict Solid system design and broad evaluation, but the headline numbers don't match the stated default LLM; fix the configuration reporting before believing the claims. read the letter →

arxiv 2507.10873 v1 pith:B5ERCFLG submitted 2025-07-15 cs.CR

Danyu Sun , Jinghuai Zhang , Jiacen Xu , Yu Zheng , Yuan Tian , Zhou Li This is my paper
classification cs.CR
keywords host-basedintrusiondetectionlargelanguagemodelssystemauditlogsmaskedautoencoderattackstorygenerationtacticpredictionnormalbehaviorprofilingSHIELD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SHIELD, a host-based intrusion detection framework that routes audit logs through a multi-stage large-language-model pipeline instead of building a full provenance graph. SHIELD first trains an event-level masked autoencoder on attack-free logs to flag likely attack windows, then asks an LLM to name suspicious command lines, expands those leads into a bounded evidence neighborhood, augments the prompt with deterministic profiles of normal command-line behavior, and finally prompts the LLM to classify entities and events, assign attack tactics, and write a plain-text attack story. The paper's central claim is that this pipeline reaches very high precision, over 0.95 on the three DARPA-E3 sub-datasets and over 0.9 on all ATLASv2 sub-datasets, while consistently outperforming five open-source HIDS on precision and the Matthews correlation coefficient. If true, it would mean an LLM-based text pipeline can deliver both precise detection and human-readable attack reconstruction across heterogeneous host logs, addressing the alert-fatigue and interpretability failures of current systems. The authors also report that every component contributes to the result, that reasoning-oriented LLMs work best, and that the system remains largely stable under injected mimicry events.

What carries the argument

The load-bearing mechanism is the focus-and-expand evidence neighborhood: a bounded subgraph built backward and forward from LLM-named suspicious command lines, capped by a token-budget threshold so the prompt never exceeds the model's context window. Around that core, three support mechanisms do the work: an event-level Masked Autoencoder that scores each event against benign training data to pick attack windows; Deterministic Data Augmentation (DDA), a key-value store of normal command-line frequencies that supplies the LLM with an exact-match benign profile; and a multi-purpose prompt template that forces the LLM to produce a narrative, tactic labels, and IoCs in one structured response, using the redundancy of those outputs as a consistency check. The system deliberately avoids constructing a full provenance graph, treating logs as a text corpus and preserving entity relationships only through subject and object fields in each event.

What would settle it

Build a host-log benchmark from attacks created after the knowledge cutoff of the evaluated LLMs, with ground-truth IoCs that never appeared in pretraining corpora, and run SHIELD unchanged. If precision on that fresh data falls far below the reported levels around 0.95 while old benchmarks still score high, the reported results were memorization rather than generalization; if precision holds, the pipeline's focus-and-expand design is doing the detection work.

Watch

Extended reading notes

Core claim

On its own terms, SHIELD's discovery is that intrusion detection can be reframed as an LLM reading task: instead of representing logs as a graph and learning on nodes, SHIELD keeps logs as text and gives the LLM a small, focused, well-contextualized slice to investigate. The pipeline works by (1) training an event-level Masked Autoencoder on benign events and using its reconstructions with a one-class boundary model to score and select the most anomalous fixed-duration windows, (2) condensing those truncated events and asking the LLM to identify suspicious command lines as attack evidence, (3) expanding the evidence through bounded breadth-first neighborhood construction to a few hundred events that fit the token limit, (4) augmenting the prompt with a deterministic key-value profile of normal command lines so the LLM knows what benign looks like, and (5) issuing a multi-purpose prompt that asks for IoCs, tactic labels, and an attack narrative in one pass. The reported outcome is precision above 0.95 on DARPA-E3, above 0.9 on ATLASv2, and between 0.41 and 0.88 on NL-SD, with perfect tactic precision and story similarity scores around 0.52 to 0.77. The paper reads this as evidence that naive LLM prompting fails but a purpose-built LLM pipeline can combine misuse detection and anomaly detection with outputs at four levels: event, entity, tactic, and story.

Load-bearing premise

The evaluation assumes the LLMs have not memorized the evaluation datasets or their ground-truth indicators of compromise, so the high precision reflects detection rather than retrieval, and the paper's only direct check is asking the models about the attacks through an API.

Editorial extensions

If this is right

  • If SHIELD's precision holds, operators could act on the small set of flagged events instead of triaging thousands of graph nodes, reducing alert fatigue.
  • Multi-level outputs, covering events, entities, tactics, and a story, let a threat hunter reconstruct the campaign from one LLM response rather than stitching entity alerts together.
  • Since DDA and the masked autoencoder only need attack-free training logs, adapting SHIELD to a new environment means retraining those two components without re-engineering the LLM prompts.
  • Reasoning-oriented LLMs matter: the paper's comparisons suggest that weaker reasoners lose precision, so model choice is a first-order parameter for deploying the system.
  • The robustness experiments with injected mimicry edges suggest the narrative and self-consistency steps absorb some adversarial noise, although degradation appears once thousands of fake events are added.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is using the same focus-and-expand recipe for network logs or cloud audit streams, where the bottleneck is also context size rather than graph construction.
  • Because the memorization check only asks the models about the attacks through an API, the cleanest test of generalization would be a benchmark published after the LLMs' training cutoff; that test is not in the paper.
  • The DDA exact-match profiler could be recycled as a standalone explainability tool: it tells an analyst which command lines are rare or novel for a given executable, independent of the LLM verdict.
  • The paper's own case studies show false positives often come from guilt by association, with a benign binary downloaded via a suspicious tool; prompt-level countermeasures that ask the LLM to separate primary malware from tools the malware invoked would be a concrete next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SHIELD, an LLM-aided host-based intrusion detection framework. The pipeline preprocesses raw audit logs into text events, uses a masked-autoencoder plus one-class SVM to select candidate attack windows, extracts 'attack evidence' with an LLM and expands it into a neighborhood graph, augments the resulting prompt with deterministic normal-behavior profiles (DDA), and prompts the LLM for entity/event classification, MITRE ATT&CK tactic labels, and a narrative attack story. The evaluation compares SHIELD with Flash, MAGIC, ORTHRUS, NodLink, and AirTag on DARPA-E3, NL-SD, and ATLASv2, reporting large precision and MCC gains over baselines, plus ablation, hyperparameter, robustness, and cost experiments.

Significance. If the reported results hold for a single, clearly specified configuration, SHIELD would be a meaningful contribution: it would show that a text-based LLM pipeline can match or exceed graph-based HIDS across heterogeneous host logs while also producing analyst-friendly, multi-level output, and it does so with a transparent component-level ablation and a robustness study. The paper's strengths are the breadth of the evaluation, the per-LLM results in the appendix, and the explicit reporting of costs and latencies. However, the headline numbers currently conflate multiple LLM backends, and the main claims are not yet supported as written.

major comments (3)
  1. [§V-A, §V-B, Tables V–VII vs. Tables XIII–XV] The main evaluation tables do not consistently report the stated default LLM (DeepSeek-R1). For example, Table VII ATLASv2-s1 (TP=774, FP=75, TN=1366, FN=147; Pre=0.9117, MCC=0.8013) is exactly the OpenAI-o3-mini row in Table XV, while DeepSeek-R1 on s1 has TP=768, FP=109, TN=1332, FN=153 (Pre=0.8757, MCC=0.7654). Table VII m5 and m6 match Llama3.3-70B (TP=454/877, MCC=0.6946/0.8400), whereas DeepSeek-R1 on m5 and m6 has MCC=0.3137 and 0.1454. In NL-SD, Table VI HW17 and HW20 match Sonar-Reasoning-Pro, not DeepSeek-R1, and in DARPA-E3, Table V TRACE matches OpenAI-o3-mini. Consequently, the averaged precisions in Table I and the Section V-B claim of 'over 0.7 MCC on all ATLASv2 sub-datasets' are not the performance of a single configuration; they represent an upper envelope over four different LLMs. The authors should either rerun all main results under a single default LLM or explicitly label each row with the LLM used and revise the claims to refer to per-LLM performance rather than to 'SHIELD' as a fixed system.
  2. [§V-A (contamination check)] The threat-to-validity discussion is acknowledged, but the check performed is too weak to rule out memorization of the evaluation datasets or their ground-truth IoCs. Asking an LLM through an API whether it knows the attacks and receiving vague answers does not test whether the model can recall specific IoCs, command lines, or attack patterns from pretraining. A concrete test would be to query the LLM with partial or masked ground-truth IoCs (e.g., file names, IPs, process names) and measure whether it can complete them, or to compare detection on original events versus semantically equivalent events with IoCs renamed/obfuscated. Without such a check, the strong detection numbers could partly reflect retrieval rather than generalization, which is a load-bearing ambiguity for the paper's central claim.
  3. [§V-B, Tables XIII–XV] No variance or error bars are reported for any of the LLM-based results, even though LLM outputs are stochastic (temperature, sampling, and API nondeterminism). The reported differences between models are often small at the entity level, such as DeepSeek-R1 and OpenAI-o3-mini on DARPA-E3 TRACE (TP=722 vs. TP=723), and without multiple runs it is impossible to tell whether the observed ranking is stable or within run-to-run noise. The paper should report at least a few repeated runs with mean and standard deviation, or explicitly state the decoding settings and verify that the main rankings are robust.
minor comments (4)
  1. [§I] The introduction refers to 'ND-SL [10]' and 'Othrus [14]'; these should be 'NL-SD' and 'ORTHRUS' to match the rest of the paper.
  2. [§V-E, Table X] The text says 'MAE takes most time (242 seconds)' for DARPA-E3 THEIA, but Table X lists MAE=121s, EvidenceExt=104s, DDA=171s, and Investigation=90s; either the sentence or the table is inconsistent, and the claim that MAE is the most time-consuming component is not supported by the listed numbers.
  3. [§V-A, Figure 4] The notation for the sliding window is inconsistent: the text defines 'window size Wl' while Figure 4 labels the axis 'Window Size Ws'.
  4. [Appendix A] The appendix contains a typo 'DAPRA Transparent Computing' and the reference for Windows Event Tracing is misspelled 'Winodws Event tracing'; both should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SHIELD's detection pipeline is evaluated against external ground-truth benchmarks, and the cited self-works are background only; the main validity risks are data-memorization and table-consistency issues, which are not circularity by construction.

full rationale

I walked the claimed derivation chain: the event-level MAE is trained on benign logs with masked-language-modeling and reconstruction losses (Eq. 1), then an OCSVM boundary and window aggregation produce attack windows; attack evidence is extracted by the LLM, expanded through a provenance neighborhood; DDA profiles are built offline from attack-free training logs; and the final multi-purpose prompt yields entity/event, tactic, and story outputs. Each component's output is compared against external ground truth (DARPA-E3 reports, NL-SD and ATLASv2 repos), and no parameter is fitted to the evaluation set and then renamed as a prediction. The self-citations (AutoAttacker [53], ProGrapher [25], Kairos [47]) appear only as related-work or design acknowledgments and are not load-bearing; no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The paper itself flags the main validity threat, namely that LLMs might have memorized ground-truth documents, and tests it by querying the models; whether that test is adequate is an empirical-data-contamination concern, not a definitional circularity. Likewise, the reviewer-observed mismatch between some main-table rows and the stated default LLM (DeepSeek-R1) is a reporting/consistency issue that undermines the strength of specific claims but does not make any derivation equivalent to its inputs by construction. For these reasons, the paper does not exhibit circular reasoning under the stated criteria.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The free parameters are hand-chosen system hyperparameters, not fitted to the detection targets. The main assumptions are the attack-free training set, the separation of anomalies by OCSVM, the fidelity of the provenance graph for expansion, the reliability of LLM knowledge without dataset memorization, and the correctness of the ground-truth labels.

free parameters (6)
  • Sliding window size Wl = 30 minutes
    Hand-chosen to fit LLM token limits; ablation in Fig. 4 shows sensitivity.
  • Top-k portion k for window score = 10%
    Chosen because attack events are scarce and only the most anomalous events are relevant.
  • Retained time windows C = 3
    Caps the context length for the LLM.
  • Neighborhood expansion threshold TNBR = 500 events
    Bounded by LLM token limit; ablation shows degradation when reduced.
  • Sampling ratio r for DDA = 0.5
    Chosen as a balance between profile diversity and prompt size.
  • MAE masking ratios = 15-30% (encoder), 50-70% (decoder)
    Taken from prior MAE work; not tuned on the target datasets.
assumptions (5)
  • domain assumption Training logs DT r are attack-free.
    SHIELD builds normal behavior profiles and trains the MAE/OCSVM on these logs; contamination would corrupt the anomaly scores.
  • domain assumption OCSVM on MAE embeddings separates malicious events from benign events.
    This is the core of attack window detection (Section IV-B).
  • domain assumption The provenance graph built from preprocessed log fields captures attack dependencies.
    Neighborhood expansion from attack evidence relies on graph edges linking related entities (Section IV-C).
  • domain assumption The LLM has accurate knowledge of attack techniques and has not memorized the test datasets.
    Attack evidence identification and investigation depend on this; the paper's check is weak (Section V-A).
  • domain assumption Ground-truth labels for the three datasets are correct.
    All precision and MCC metrics are computed against these labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Alerts to Intelligence: A Novel LLM-Aided Framework for Host-based Intrusion Detection." pith.science (2026). https://pith.science/paper/B5ERCFLG

@misc{pith2026250710873,
  author       = {Pith},
  title        = {Pith review of: From Alerts to Intelligence: A Novel LLM-Aided Framework for Host-based Intrusion Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5ERCFLG}},
  note         = {Machine review of arXiv:2507.10873}
}
read the original abstract

Host-based intrusion detection system (HIDS) is a key defense component to protect the organizations from advanced threats like Advanced Persistent Threats (APT). By analyzing the fine-grained logs with approaches like data provenance, HIDS has shown successes in capturing sophisticated attack traces. Despite the progresses embarked by the research community and industry, HIDS still frequently encounters backlash from their operators in the deployed environments, due to issues like high false-positive rate, inconsistent outcomes across environments and human-unfriendly detection results. Large Language Models (LLMs) have great potentials to advance the state of HIDS, given their extensive knowledge of attack techniques and their ability to detect anomalies through semantic analysis, anchored by recent studies. Yet, our preliminary analysis indicates that building an HIDS by naively prompting an LLM is unlikely to succeed. In this work, we explore the direction of building a customized LLM pipeline for HIDS and develop a system named SHIELD. SHIELD addresses challenges related to LLM's token limits, confusion of background noises, etc., by integrating a variety of techniques like event-level Masked Autoencoder (MAE) for attack window detection, attack evidence identification and expansion, Deterministic Data Augmentation (DDA) for profiling normal activities, and multi-purpose prompting that guides the LLM to conduct precise and interpretable attack investigations. Extensive experiments on three log datasets (DARPA-E3, NodLink-simulated-data and ATLASv2) show that SHIELD consistently achieves outstanding performance in comparison with 5 representative HIDS. These findings highlight the potential of LLMs as powerful tools for intrusion detection and pave the way for future research in this domain.

Figures

Figures reproduced from arXiv: 2507.10873 by the authors.

Figure 1
Figure 1. The framework of SHIELD. Given a testing log, we first identify attack windows to reduce the context size. We then leverage the internal knowledge of LLMs to generate attack evidence and extract attack-related events (referred to as neighborhood evidence) for fine-grained investigation. Finally, we apply a multi-purpose prompting strategy to query the LLMs, which produces a precise and interpretable attack summary f… view at source ↗
Figure 2
Figure 2. The framework of event-level Masked Autoencoder (MAE) for attack window detection. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Illustration of neighborhood expansion from the attack evidence with [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Impact of key hyperparameters on the entity-level detection perfor [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Effect of adversary mimicry attacks [30] against SHIELD. TABLE X EFFICIENCY AND COSTS. DDA∗ IS PERFORMED OFFLINE ONLY ONCE, BEFORE THE ATTACK INVESTIGATION. MAE, EVIDENCEEXT, DDA AND INVESTIGATION ARE COMPONENTS DESCRIBED IN SECTION IV-B, IV-C, IV-D, AND IV-E RESPECTIV…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

97 extracted references · 48 canonical work pages

  1. [1]

    Fbi releases annual internet crime report,

    Federal Bureau of Investigation, “Fbi releases annual internet crime report,” 2025. [Online]. Available: https://www.fbi.gov/news/press-rel eases/fbi-releases-annual-internet-crime-report

    2025

  2. [2]

    Cyberattacks cost victims more than quadruple the amount hackers pay to execute attacks,

    P. Technologies, “Cyberattacks cost victims more than quadruple the amount hackers pay to execute attacks,” 2024. [Online]. Available: https://global.ptsecurity.com/about/news/cyberattacks-cost-victims-mor e-than-quadruple-the-amount-hackers-pay-to-execute-attacks?utm sou rce=chatgpt.com

    2024

  3. [3]

    Intrusion detection system market size, share, and growth analysis,

    SkyQuest, “Intrusion detection system market size, share, and growth analysis,” https://www.skyquestt.com/report/intrusion-detection-syste m-market

  4. [4]

    Sok: History is a vast early warning system: Auditing the provenance of system intrusions,

    M. A. Inam, Y . Chen, A. Goyal, J. Liu, J. Mink, N. Michael, S. Gaur, A. Bates, and W. U. Hassan, “Sok: History is a vast early warning system: Auditing the provenance of system intrusions,” in 2023 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, 2023, pp. 307–325

    2023

  5. [5]

    Backtracking intrusions,

    S. T. King and P. M. Chen, “Backtracking intrusions,” ACM Transactions on Computer Systems (TOCS) , vol. 23, no. 1, pp. 51–76, 2005

    2005

  6. [6]

    Nodoze: Combatting threat alert fatigue with automated provenance triage,

    W. U. Hassan, S. Guo, D. Li, Z. Chen, K. Jee, Z. Li, and A. Bates, “Nodoze: Combatting threat alert fatigue with automated provenance triage,” in network and distributed systems security symposium , 2019

    2019

  7. [7]

    Transparent computing engagement 3,

    DARPA I2O, “Transparent computing engagement 3,” https://github.c om/darpa-i2o/Transparent-Computing/blob/master/README-E3.md, accessed: 2023-03-13

    2023

  8. [8]

    Nodlink repo,

    PKU-ASAL, “Nodlink repo,” https://github.com/PKU-ASAL/Simulate d-Data, 2023

    2023

Show all 97 references
  1. [9]

    Atlasv2: Atlas attack engagements, version 2,

    A. Riddle, K. Westfall, and A. Bates, “Atlasv2: Atlas attack engagements, version 2,” 2023. [Online]. Available: https://arxiv.org/ab s/2401.01341

    2023 arXiv

  2. [10]

    Nodlink: An online system for fine-grained apt attack detection and investigation,

    S. Li, F. Dong, X. Xiao, H. Wang, F. Shao, J. Chen, Y . Guo, X. Chen, and D. Li, “Nodlink: An online system for fine-grained apt attack detection and investigation,” in Proceedings 2024 Network and Distributed System Security Symposium , ser. NDSS 2024. Internet Society, 2024....

    2024

  3. [11]

    AIRTAG: Towards automated attack investigation by unsupervised learning with log texts,

    H. Ding, J. Zhai, Y . Nan, and S. Ma, “AIRTAG: Towards automated attack investigation by unsupervised learning with log texts,” in 32nd USENIX Security Symposium (USENIX Security 23) . Anaheim, CA: USENIX Association, 2023, pp. 373–390

    2023

  4. [12]

    Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,

    M. Ur Rehman, H. Ahmadi, and W. Ul Hassan, “Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,” in 2024 IEEE Symposium on Security and Privacy (SP), 2024, pp. 3552–3570

    2024

  5. [13]

    {MAGIC}: Detecting advanced persistent threats via masked graph representation learning,

    Z. Jia, Y . Xiong, Y . Nan, Y . Zhang, J. Zhao, and M. Wen, “ {MAGIC}: Detecting advanced persistent threats via masked graph representation learning,” in 33rd USENIX Security Symposium (USENIX Security 24) , 2024, pp. 5197–5214

    2024

  6. [14]

    ORTHRUS: Achieving High Quality of At- tribution in Provenance-based Intrusion Detection Systems,

    B. Jiang, T. Bilot, N. El Madhoun, K. Al Agha, A. Zouaoui, S. Iqbal, X. Han, and T. Pasquier, “ORTHRUS: Achieving High Quality of At- tribution in Provenance-based Intrusion Detection Systems,” in Security Symposium (USENIX Sec’25) . USENIX, 2025

    2025

  7. [15]

    Constructing knowledge graph from cyber threat intelligence using large language model,

    J. Liu and J. Zhan, “Constructing knowledge graph from cyber threat intelligence using large language model,” in 2023 IEEE International Conference on Big Data (BigData) , 2023, pp. 516–521. 13

    2023

  8. [16]

    Ctikg: Llm-powered knowledge graph construc- tion from cyber threat intelligence,

    L. Huang and X. Xiao, “Ctikg: Llm-powered knowledge graph construc- tion from cyber threat intelligence,” in First Conference on Language Modeling, 2024

    2024

  9. [17]

    Towards a scalable ai- driven framework for data-independent cyber threat intelligence infor- mation extraction,

    O. Sorokoletova, E. Antonioni, and G. Col `o, “Towards a scalable ai- driven framework for data-independent cyber threat intelligence infor- mation extraction,” in 2024 2nd International Conference on Foundation and Large Language Models (FLLM) , 2024, pp. 398–406

    2024

  10. [18]

    Fade: Few-shot/zero-shot anomaly detection engine using large vision-language model,

    Y . Li, E. Ivanova, and M. Bruveris, “Fade: Few-shot/zero-shot anomaly detection engine using large vision-language model,” arXiv preprint arXiv:2409.00556, 2024

    2024 arXiv

  11. [19]

    Ad-llm: Benchmarking large language models for anomaly detection,

    T. Yang, Y . Nian, S. Li, R. Xu, Y . Li, J. Li, Z. Xiao, X. Hu, R. Rossi, K. Ding et al. , “Ad-llm: Benchmarking large language models for anomaly detection,” arXiv preprint arXiv:2412.11142 , 2024

    2024

  12. [20]

    Large language models for forecasting and anomaly detec- tion: A systematic literature review,

    J. Su, C. Jiang, X. Jin, Y . Qiao, T. Xiao, H. Ma, R. Wei, Z. Jing, J. Xu, and J. Lin, “Large language models for forecasting and anomaly detec- tion: A systematic literature review,” arXiv preprint arXiv:2402.10350 , 2024

    2024 arXiv

  13. [21]

    Anomaly detection of tabular data using llms,

    A. Li, Y . Zhao, C. Qiu, M. Kloft, P. Smyth, M. Rudolph, and S. Mandt, “Anomaly detection of tabular data using llms,” arXiv preprint arXiv:2406.16308, 2024

    2024 arXiv

  14. [22]

    Summarization is (almost) dead,

    X. Pu, M. Gao, and X. Wan, “Summarization is (almost) dead,” arXiv preprint arXiv:2309.09558, 2023

    2023 arXiv

  15. [23]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024

    2024

  16. [24]

    Unicorn: Runtime provenance-based detector for advanced persistent threats,

    X. Han, T. Pasquier, A. Bates, J. Mickens, and M. Seltzer, “Unicorn: Runtime provenance-based detector for advanced persistent threats,” arXiv preprint arXiv:2001.01525 , 2020

    2001 arXiv

  17. [25]

    {PROGRAPHER}: An anomaly detection system based on provenance graph embedding,

    F. Yang, J. Xu, C. Xiong, Z. Li, and K. Zhang, “ {PROGRAPHER}: An anomaly detection system based on provenance graph embedding,” in 32nd USENIX Security Symposium (USENIX Security 23) , 2023, pp. 4355–4372

    2023

  18. [26]

    Masked au- toencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked au- toencoders are scalable vision learners,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16 000–16 009

    2022

  19. [27]

    Madeline: Continuous and low-cost monitoring with graph-free representations to combat cyber threats,

    W. Song, H. Ding, N. Meng, P. Gao, and D. Yao, “Madeline: Continuous and low-cost monitoring with graph-free representations to combat cyber threats,” in 2024 Annual Computer Security Applications Conference (ACSAC). IEEE, 2024, pp. 874–889

    2024

  20. [28]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in neural information processing systems , vol. 33, pp. 9459–9474, 2020

    2020

  21. [29]

    Enterprise tactics,

    MITRE, “Enterprise tactics,” https://attack.mitre.org/tactics/enterprise/, 2025

    2025

  22. [30]

    Sometimes, you aren’t what you do: Mimicry attacks against provenance graph host intrusion detection systems,

    A. Goyal, X. Han, G. Wang, and A. Bates, “Sometimes, you aren’t what you do: Mimicry attacks against provenance graph host intrusion detection systems,” in 30th Network and Distributed System Security Symposium, 2023

    2023

  23. [31]

    Winodws event tracing,

    Don Marshall, “Winodws event tracing,” https://docs.microsoft.com/e n-us/windows-hardware/drivers/devtest/event-tracing-for-windows--e tw-, 2021

    2021

  24. [32]

    Linux audit,

    Steve Grubb, “Linux audit,” https://linux.die.net/man/8/auditd, 2021

    2021

  25. [33]

    Dtrace on freebsd,

    George V . Neville-Neil, “Dtrace on freebsd,” https://wiki.freebsd.org/D Trace, 2018

    2018

  26. [34]

    Cyber kill chain,

    Lockheed Martin, “Cyber kill chain,” https://www.lockheedmartin.com /en-us/capabilities/cyber/cyber-kill-chain.html, 2022

    2022

  27. [35]

    Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,

    F. Dong, S. Li, P. Jiang, D. Li, H. Wang, L. Huang, X. Xiao, J. Chen, X. Luo, Y . Guo et al. , “Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communica- tion...

    2023

  28. [36]

    Provenance-aware tracing ofworm break-in and contaminations: A process coloring approach,

    X. Jiang, A. Walters, D. Xu, E. H. Spafford, F. Buchholz, and Y .-M. Wang, “Provenance-aware tracing ofworm break-in and contaminations: A process coloring approach,” in Distributed Computing Systems, 2006. ICDCS 2006. 26th IEEE International Conference on . IEEE, 2006, pp. 38–38

    2006

  29. [37]

    High fidelity data reduction for big data security dependency analyses,

    Z. Xu, Z. Wu, Z. Li, K. Jee, J. Rhee, X. Xiao, F. Xu, H. Wang, and G. Jiang, “High fidelity data reduction for big data security dependency analyses,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , 2016, pp. 504–516

    2016

  30. [38]

    Can data provenance put an end to the data breach?

    A. Bates and W. U. Hassan, “Can data provenance put an end to the data breach?” IEEE Security & Privacy, vol. 17, no. 4, pp. 88–93, 2019

    2019

  31. [39]

    Towards a timely causality analysis for enterprise security,

    Y . Liu, M. Zhang, D. Li, K. Jee, Z. Li, Z. Wu, J. Rhee, and P. Mittal, “Towards a timely causality analysis for enterprise security,” inProceed- ings of the 25th Network and Distributed System Security Symposium (NDSS), 2018

    2018

  32. [40]

    Holmes: real-time apt detection through correlation of suspicious information flows,

    S. M. Milajerdi, R. Gjomemo, B. Eshete, R. Sekar, and V . Venkatakrish- nan, “Holmes: real-time apt detection through correlation of suspicious information flows,” in 2019 IEEE Symposium on Security and Privacy (SP), 2019, pp. 1137–1152

    2019

  33. [41]

    Combating dependence ex- plosion in forensic analysis using alternative tag propagation semantics,

    M. N. Hossain, S. Sheikhi, and R. Sekar, “Combating dependence ex- plosion in forensic analysis using alternative tag propagation semantics,” in 2020 IEEE Symposium on Security and Privacy (SP) . IEEE, 2020, pp. 1139–1155

    2020

  34. [42]

    Nodemerge: Template based efficient data reduction for big-data causality analysis,

    Y . Tang, D. Li, Z. Li, M. Zhang, K. Jee, X. Xiao, Z. Wu, J. Rhee, F. Xu, and Q. Li, “Nodemerge: Template based efficient data reduction for big-data causality analysis,” in Proceedings of the 2018 ACM SIGSAC conference on computer and communications security , 2018, pp. 1324– 1337

    2018

  35. [43]

    {SEAL}: Storage- efficient causality analysis on enterprise logs with query-friendly com- pression,

    P. Fei, Z. Li, Z. Wang, X. Yu, D. Li, and K. Jee, “ {SEAL}: Storage- efficient causality analysis on enterprise logs with query-friendly com- pression,” in 30th USENIX security symposium (USENIX Security 21) , 2021, pp. 2987–3004

    2021

  36. [44]

    The case for learned provenance graph storage systems,

    H. Ding, J. Zhai, D. Deng, and S. Ma, “The case for learned provenance graph storage systems,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 3277–3294

    2023

  37. [45]

    Tactical provenance analysis for endpoint detection and response systems,

    W. U. Hassan, A. Bates, and D. Marino, “Tactical provenance analysis for endpoint detection and response systems,” in Proceedings of the IEEE Symposium on Security and Privacy , 2020

    2020

  38. [46]

    Alchemist: Fusing application and audit logs for precise attack provenance without instrumentation,

    L. Yu, S. Ma, Z. Zhang, G. Tao, X. Zhang, D. Xu, V . E. Urias, H. W. Lin, G. Ciocarlie, V . Yegneswaranet al., “Alchemist: Fusing application and audit logs for precise attack provenance without instrumentation,” in NDSS, 2021

    2021

  39. [47]

    Kairos: Practical intrusion detection and investigation using whole-system provenance,

    Z. Cheng, Q. Lv, J. Liang, Y . Wang, D. Sun, T. Pasquier, and X. Han, “Kairos: Practical intrusion detection and investigation using whole-system provenance,” 2023. [Online]. Available: https: //arxiv.org/abs/2308.05034

    2023 arXiv

  40. [48]

    R-caid: Embedding root cause analysis within provenance-based intrusion detection,

    A. Goyal, G. Wang, and A. Bates, “R-caid: Embedding root cause analysis within provenance-based intrusion detection,” in 2024 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, 2024, pp. 257–257

    2024

  41. [49]

    Shadewatcher: Recommendation-guided cyber threat analysis using system audit records,

    J. Zeng, X. Wang, J. Liu, Y . Chen, Z. Liang, T. Chua, and Z. Chua, “Shadewatcher: Recommendation-guided cyber threat analysis using system audit records,” in 2022 IEEE Symposium on Security and Privacy (SP) (SP), 2022

    2022

  42. [50]

    SIGL: securing software installations through deep graph learning,

    X. Han, X. Yu, T. F. J. Pasquier, D. Li, J. Rhee, J. W. Mickens, M. I. Seltzer, and H. Chen, “SIGL: securing software installations through deep graph learning,” in 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021 . USENIX Association, 2021, pp. 2345–2362

    2021

  43. [51]

    Darpa tc engagement 3 ground-truth,

    STS-Lab, “Darpa tc engagement 3 ground-truth,” https://bitbucket.org/ sts-lab/reapr-ground-truth/src/master/darpa-tc-engagement3/, 2023

    2023

  44. [52]

    A comprehensive overview of large lan- guage models (llms) for cyber defences: Opportunities and directions,

    M. Hassanin and N. Moustafa, “A comprehensive overview of large lan- guage models (llms) for cyber defences: Opportunities and directions,” arXiv preprint arXiv:2405.14487 , 2024

    2024 arXiv

  45. [53]

    Autoattacker: A large language model guided system to implement automatic cyber-attacks,

    J. Xu, J. W. Stokes, G. McDonald, X. Bai, D. Marshall, S. Wang, A. Swaminathan, and Z. Li, “Autoattacker: A large language model guided system to implement automatic cyber-attacks,” arXiv preprint arXiv:2403.01038, 2024

    2024 arXiv

  46. [54]

    {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “ {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing,” in 33rd USENIX Security Symposium (USENIX Security 24) , 2024, pp. 847–864

    2024

  47. [55]

    Large language model guided protocol fuzzing,

    R. Meng, M. Mirchev, M. B ¨ohme, and A. Roychoudhury, “Large language model guided protocol fuzzing,” in Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS) , vol. 2024, 2024

    2024

  48. [56]

    Exploring {ChatGPT’s} capabilities on vulnerability management,

    P. Liu, J. Liu, L. Fu, K. Lu, Y . Xia, X. Zhang, W. Chen, H. Weng, S. Ji, and W. Wang, “Exploring {ChatGPT’s} capabilities on vulnerability management,” in 33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 811–828. 14

    2024

  49. [57]

    Raconteur: A knowledgeable, insightful, and portable llm-powered shell command explainer,

    J. Deng, X. Li, Y . Chen, Y . Bai, H. Weng, Y . Liu, T. Wei, and W. Xu, “Raconteur: A knowledgeable, insightful, and portable llm-powered shell command explainer,” arXiv preprint arXiv:2409.02074 , 2024

    2024 arXiv

  50. [58]

    Deeplog: Anomaly detection and diagnosis from system logs through deep learning,

    M. Du, F. Li, G. Zheng, and V . Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, 2017, pp. 1285–1298

    2017

  51. [59]

    Log2vec: A heterogeneous graph embedding based approach for detecting cyber threats within enterprise,

    F. Liu, Y . Wen, D. Zhang, X. Jiang, X. Xing, and D. Meng, “Log2vec: A heterogeneous graph embedding based approach for detecting cyber threats within enterprise,” in Proceedings of the 2019 ACM SIGSAC conference on computer and communications security , 2019, pp. 1777– 1794

    2019

  52. [60]

    {ATTACK2VEC}: Leveraging temporal word embeddings to understand the evolution of cyberattacks,

    Y . Shen and G. Stringhini, “ {ATTACK2VEC}: Leveraging temporal word embeddings to understand the evolution of cyberattacks,” in 28th USENIX Security Symposium (USENIX Security 19), 2019, pp. 905–921

    2019

  53. [61]

    Semantic anomaly detection with large language models,

    A. Elhafsi, R. Sinha, C. Agia, E. Schmerling, I. A. Nesnas, and M. Pavone, “Semantic anomaly detection with large language models,” Autonomous Robots, vol. 47, no. 8, pp. 1035–1055, 2023

    2023

  54. [62]

    tiktoken,

    OpenAI, “tiktoken,” https://github.com/openai/tiktoken, 2025

    2025

  55. [63]

    Nolima: Long-context evaluation beyond literal matching,

    A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S. Yoon, and H. Sch ¨utze, “Nolima: Long-context evaluation beyond literal matching,” arXiv preprint arXiv:2502.05167 , 2025

    2025 arXiv

  56. [64]

    Fundamental limits of prompt compression: A rate-distortion framework for black-box language models,

    A. Nagle, A. Girish, M. Bondaschi, M. Gastpar, A. V . Makkuva, and H. Kim, “Fundamental limits of prompt compression: A rate-distortion framework for black-box language models,” Advances in Neural Infor- mation Processing Systems , vol. 37, pp. 94 934–94 970, 2024

    2024

  57. [65]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” arXiv preprint arXiv:2402.07927 , 2024

    2024 arXiv

  58. [66]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  59. [67]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  60. [68]

    Custos: Practical tamper-evident auditing of operating systems using trusted execution,

    R. Paccagnella, P. Datta, W. U. Hassan, A. Bates, C. Fletcher, A. Miller, and D. Tian, “Custos: Practical tamper-evident auditing of operating systems using trusted execution,” in Network and distributed system security symposium, 2020

    2020

  61. [69]

    Auditing frameworks need resource isolation: A systematic study on the super producer threat to system auditing and its mitigation,

    P. Jiang, R. Huang, D. Li, Y . Guo, X. Chen, J. Luan, Y . Ren, and X. Hu, “Auditing frameworks need resource isolation: A systematic study on the super producer threat to system auditing and its mitigation,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 355–372

    2023

  62. [70]

    Support vector method for novelty detection,

    B. Sch ¨olkopf, R. C. Williamson, A. Smola, J. Shawe-Taylor, and J. Platt, “Support vector method for novelty detection,” Advances in neural information processing systems , vol. 12, 1999

    1999

  63. [71]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  64. [72]

    bert-base-uncased,

    H. Face, “bert-base-uncased,” https://huggingface.co/bert-base-uncased, 2023, accessed: 2025-04-11

    2023

  65. [73]

    Retromae: Pre-training retrieval- oriented language models via masked auto-encoder,

    S. Xiao, Z. Liu, Y . Shao, and Z. Cao, “Retromae: Pre-training retrieval- oriented language models via masked auto-encoder,” arXiv preprint arXiv:2205.12035, 2022

    2022 arXiv

  66. [74]

    Backtracking intrusions,

    S. T. King and P. M. Chen, “Backtracking intrusions,” ACM Trans. Comput. Syst., vol. 23, no. 1, pp. 51–76, 2005

    2005

  67. [75]

    Representative sam- pling for reliable data analysis: theory of sampling,

    L. Petersen, P. Minkkinen, and K. H. Esbensen, “Representative sam- pling for reliable data analysis: theory of sampling,” Chemometrics and intelligent laboratory systems , vol. 77, no. 1-2, pp. 261–277, 2005

    2005

  68. [76]

    ATLAS: A sequence-based learning approach for attack investigation,

    A. Alsaheel, Y . Nan, S. Ma, L. Yu, G. Walkup, Z. B. Celik, X. Zhang, and D. Xu, “ATLAS: A sequence-based learning approach for attack investigation,” in USENIX Security Symposium . USENIX Association, 2021, pp. 3005–3022

    2021

  69. [77]

    Threatrace: Detecting and tracing host-based threats in node level through provenance graph learning,

    S. Wang, Z. Wang, T. Zhou, H. Sun, X. Yin, D. Han, H. Zhang, X. Shi, and J. Yang, “Threatrace: Detecting and tracing host-based threats in node level through provenance graph learning,” IEEE Transactions on Information Forensics and Security , vol. 17, pp. 3972–3987, 2022

    2022

  70. [78]

    Mitigating advanced and persistent threat (apt) damage by reasoning with provenance in large enterprise network (marple) program,

    J. Rao, Y . Chen, R. Sekar, and V . Venkatakrishnan, “Mitigating advanced and persistent threat (apt) damage by reasoning with provenance in large enterprise network (marple) program,” AFRL-RY-WP-TR-2019- 0285, International Business Machines Corporation, Tech. Rep., 2020

    2019

  71. [79]

    Atlasv2,

    sts-lab, “Atlasv2,” https://bitbucket.org/sts-lab/reapr-ground-truth/src/m aster/atlasv2/, 2023

    2023

  72. [80]

    Flash repo,

    DART-Laboratory, “Flash repo,” https://github.com/DART-Laboratory/ Flash-IDS, 2024

    2024

  73. [81]

    Magic repo,

    FDUDSDE, “Magic repo,” https://github.com/FDUDSDE/MAGIC, 2023

    2023

  74. [82]

    Orthrus repo,

    ubc-provenance, “Orthrus repo,” https://github.com/ubc-provenance/or thrus, 2025

    2025

  75. [83]

    Airtag repo,

    dhl123, “Airtag repo,” https://github.com/dhl123/Airtag-2023, 2023

    2023

  76. [84]

    Atlas dataset,

    purseclab, “Atlas dataset,” https://github.com/purseclab/ATLAS, 2020

    2020

  77. [85]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  78. [86]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  79. [87]

    Openai o3-mini,

    OpenAI, “Openai o3-mini,” https://openai.com/index/openai-o3-mini/, 2025

    2025

  80. [88]

    Sonar reasoning pro,

    Perplexity, “Sonar reasoning pro,” https://sonar.perplexity.ai, 2025

    2025

  81. [89]

    all-MiniLM-L6-v2,

    UKPLab, “all-MiniLM-L6-v2,” https://huggingface.co/sentence-transfo rmers/all-MiniLM-L6-v2, 2025

    2025

  82. [90]

    Semeval- 2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation,

    D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia, “Semeval- 2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation,” arXiv preprint arXiv:1708.00055 , 2017

    2017 arXiv

  83. [91]

    (2025) Api pricing

    DeepSeek. (2025) Api pricing. [Online]. Available: https://api-docs.de epseek.com/quick start/pricing

    2025

  84. [92]

    Attacks on deidentification’s defenses,

    A. Cohen, “Attacks on deidentification’s defenses,” in 31st USENIX security symposium (USENIX Security 22) , 2022, pp. 1469–1486

    2022

  85. [93]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, Z. Luo, Z. Feng, and Y . Ma, “Llamafactory: Unified efficient fine-tuning of 100+ language models,” arXiv preprint arXiv:2403.13372 , 2024

    2024 arXiv

  86. [94]

    s1: Simple test-time scaling,

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand `es, and T. Hashimoto, “s1: Simple test-time scaling,” arXiv preprint arXiv:2501.19393 , 2025

    2025 arXiv

  87. [95]

    Persistent pre-training poisoning of llms,

    Y . Zhang, J. Rando, I. Evtimov, J. Chi, E. M. Smith, N. Carlini, F. Tram`er, and D. Ippolito, “Persistent pre-training poisoning of llms,” arXiv preprint arXiv:2410.13722 , 2024

    2024 arXiv

  88. [96]

    Sok: Pragmatic assessment of machine learning for network intrusion detection,

    G. Apruzzese, P. Laskov, and J. Schneider, “Sok: Pragmatic assessment of machine learning for network intrusion detection,” in 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 2023, pp. 592–614

    2023

  89. [97]

    A survey on data-driven network intrusion detection,

    D. Chou and M. Jiang, “A survey on data-driven network intrusion detection,” ACM Computing Surveys (CSUR) , vol. 54, no. 9, pp. 1–36, 2021. APPENDIX A. Description of Datasets We comprehensively evaluate SHIELD and baseline methods on three public datasets, including DARPA-E3,...

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.