Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

MM-AttacKG: A Multimodal Approach to Attack Graph Construction with Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that threat images in cyber threat intelligence reports carry attack information that text alone misses, and that a multimodal large language model pipeline, MM-AttacKG, can extract it and integrate it into text-based…

desk verdict First real attempt to use MLLMs for image-based attack graph construction; worth peer review, but the headline evaluation needs a precision check before the central accuracy claim can stand. read the letter →

arxiv 2506.16968 v1 pith:D4PIDUVW submitted 2025-06-20 cs.CR cs.CY

Yongheng Zhang , Xinyun Zhao , Yunshan Ma , Haokai Ma , Yingxiao Guan , Guozheng Yang , Yuliang Lu , Xiang Wang This is my paper · ORCID
classification cs.CRcs.CY
keywords attackgraphconstructioncyberthreatintelligencemultimodallargelanguagemodelsimageparsingMITREATT&CKinformationextractionAG-LLM-mmdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that images inside cyber threat intelligence (CTI) reports are not redundant decorations: they carry entities, relations, and MITRE ATT&CK techniques that text-based attack graph construction leaves out. To demonstrate this, it builds a five-stage pipeline, MM-AttacKG, that turns image parsing into a question-answering dialogue, filters and iteratively refines the answers, and then integrates the surviving image-derived information into an attack graph produced from report text. On the new AG-LLM-mm dataset, the image-enhanced output reports higher F1 than text-only extractors and than simpler ICL and CoT prompt schemes, with the image contributions concentrated at the entity, relation, and technique levels.

What carries the argument

The carrying mechanism is the image-to-graph enhancement loop. Brainstorming generates general and task-specific questions about each threat image; Extraction answers them using the image plus two textual supports, Image-Aware-Context (what the image shows) and Global-Context (the report outline); Verification filters questions by direct and answer-oriented relevance and refines weak answers through iterative question-led or answer-iteration rounds scored on accuracy, consistency, completeness, and relevance; Integration turns each accepted question-answer pair into a threat enhancement reference that adds nodes, updates relations, or adds techniques to the text-based attack graph.

What would settle it

Have multiple independent cybersecurity analysts annotate the threat images in a sample of AG-LLM-mm for entities, relations, and MITRE techniques, then run MM-AttacKG on the same images and compare the image-derived additions against the independent annotations. If the added items largely fail to match the human labels, the claim of accurate image information identification and graph enhancement is not supported.

Watch

Extended reading notes

Core claim

The central claim, framed as a new application of multimodal LLMs to attack graph construction, is that threat images contain key information absent from the text, and that MM-AttacKG identifies it accurately enough to improve the completeness of the resulting attack graph. The authors formalize the task by extending the text-based atomic event quintuple $(s, a, o, t, p)$ to include image-derived threat enhancement references, and they define three integration operations: node extension, relation update, and technique addition under the MITRE ATT&CK framework. Reported results on AG-LLM-mm show the full pipeline at F1 0.7716 for entities, 0.8147 for relations, and 0.5703 for techniques, and the integration experiments show higher image-derived gain than ICL and CoT prompts across Qwen2.5-VL model sizes.

Load-bearing premise

The evaluation assumes that the human-proofread gold labels and the gain metric measure true attack graph quality; the paper does not report inter-annotator agreement on the labels, and the gain metric counts every image-derived addition as correct with no precision check.

Editorial extensions

If this is right

  • If images are non-redundant with text, then any CTI pipeline that ignores images understates attack paths, and multimodal extraction should become part of standard attack graph construction.
  • The framework's prompt-only design means attack graph enrichment can ride on open-source multimodal LLMs without fine-tuning on labeled image data.
  • The AG-LLM-mm dataset gives the community a first benchmark for evaluating image-derived entities, relations, and TTPs in CTI reports.
  • The verification loop of question filtering plus answer refinement is the component most directly tied to accuracy; ablations show removing it lowers F1 on relation and technique extraction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gain metric counts every image-derived addition as valuable without checking precision, so a direct test of the paper's 'accurately identifies' claim would measure whether a human analyst agrees with each added entity, relation, and technique; the paper does not report that precision.
  • Because the pipeline depends on AttacKG+ for its text graphs, the multimodal claim could be tested by swapping in a different text constructor: if the gains vanish, the benefit may come from the text parser rather than the image module.
  • The question-answer-refine loop resembles general self-consistency methods for multimodal QA, so one could test whether the same verification loop improves other image-to-knowledge tasks, such as extracting indicators of compromise from screenshots or logs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MM-AttacKG, a pipeline for integrating information from images in cyber threat intelligence (CTI) reports into attack graph construction. The framework consists of five modules: text-based graph construction (using AttacKG+), a Brainstorming module that generates general and task-specific questions about a threat image, an Extraction module that answers those questions with MLLM assistance and report context, a Verification module that filters questions and iteratively refines answers based on self-evaluation, and an Integration module that adds image-derived entities, relations, and MITRE techniques to the text-based attack graph. The authors introduce a new dataset, AG-LLM-mm, collected from Cisco Talos and Microsoft reports, and evaluate MM-AttacKG against ICL and CoT prompting baselines using Qwen2.5-VL models. They report that image-enhanced methods outperform text-based extractors on entity, relation, and technique extraction, and that MM-AttacKG achieves higher F1 and larger 'gain' than the baselines. The central claim is that MM-AttacKG accurately identifies key information in threat images and significantly improves multimodal attack graph construction.

Significance. The problem addressed is real and timely: most attack graph construction ignores the visual content of CTI reports, and a systematic method for incorporating threat images could improve threat intelligence analysis. The proposed pipeline is clearly described, and the authors provide prompt templates and a new dataset, which are potentially useful resources. However, the evaluation as presented does not substantiate the central claims. The gain metric counts unverified additions, the verification module uses the same model to judge its own outputs, the annotation procedure lacks documented guidelines and inter-annotator agreement, and no significance testing is reported. If the authors can add independent validation of image-derived additions and reframe the comparisons appropriately, the work could be valuable. As it stands, the paper documents an interesting system but not the claimed accuracy improvement.

major comments (4)
  1. [5.1.1, Table 2] The gold standard for the precision/recall/F1 numbers in Table 2 is not defined. The text says three postgraduate students 'manually proofread and assessed' the data, but no annotation guidelines, no inter-annotator agreement, and no external standard are reported. The row 'Human Anotation-Text' has precision 1.0000 for all three tasks, which indicates it is being used as the reference, not as a rival method; including it as a baseline conflates ground truth with a competing system. Moreover, the text-based methods have no access to images, so their lower recall relative to image-enhanced methods is expected by construction; Table 2 therefore does not support the claim of 'advantages of multimodal integration.' A controlled comparison is needed, e.g., an LLM-based text-only pipeline with the same prompts, or a comparison that reports performance separately on the text-only and image-only portions of the gold standard.
  2. [5.3, Figure 5] The 'gain' metric is defined as the incremental number of entities, relations, and techniques added to the text-based attack graph, with no precision check on the added items. A hallucinated entity, a duplicate of text-derived information, or an irrelevant relation all count as positive gain. The abstract claims that MM-AttacKG 'can accurately identify key information in threat images'; this requires verifying that each image-derived addition is correct, non-duplicative, and grounded in the image, and reporting the precision of those additions. Without such verification, the high gain numbers in Figure 5 could reflect hallucinated rather than genuine threat information, undermining the central claim.
  3. [4.4, 5.5.3, Table 6, Figure 7] The answer refinement evaluation is circular. The same MLLM that produces the answers also rates them on accuracy, consistency, completeness, and relevance using the prompt in Table 6 (Answer Evaluation), and then refines its own answers based on its own suggestions. Figure 7 therefore shows the model agreeing with itself, not an objective improvement in answer quality. To support the claim that the verification module improves answer quality, the authors should use an independent judge (a different model or human raters) and report inter-rater agreement, or compare refined answers against a fixed external reference.
  4. [5.2-5.4, Tables 2-3, Figure 5] No error bars, confidence intervals, or significance tests are reported for any of the main results. Differences such as entity F1 0.7716 vs 0.7623 (Table 3, w/o V) or relation F1 0.8147 vs 0.8099 may be within run-to-run noise, especially with temperature 0.7. The abstract's 'significantly improve' is therefore not supported by statistical evidence. The authors should report variance over multiple runs and, where feasible, paired significance tests for the systems being compared.
minor comments (4)
  1. [5.5.1, Figure 6] The text states that 'monotonicity remains stable and close to [1],' but the definition given is that higher monotonicity indicates more similar questions and thus poorer diversity; a value close to 1 would indicate poor diversity, contradicting the stated conclusion of high diversity. Please clarify whether the figure's values are close to 0 or whether the scale is inverted.
  2. [References [6]-[9]] References [6] through [9] appear to be unrelated to cyber threat intelligence; they discuss breast cancer MRI, ensemble learning, and academic professional track faculty, and use 'APT' in a different sense. These citations should be corrected or removed.
  3. [Throughout] There are several typos and inconsistencies: 'EXTROCTOR' in Section 5.1.2, 'Human Anotation-Text' in Table 2, 'Constrction' in Figure 8, 'we introduces' in Section 6.1, and reference [2] cited for Qwen models without a proper Qwen technical report reference. Please proofread and standardize the notation.
  4. [Data availability] The code and dataset are stated to be released upon acceptance; for a reproducibility-focused evaluation, consider providing at least the annotation guidelines, the list of report sources, and the exact prompts used in an appendix or supplementary material even before acceptance.

Circularity Check

2 steps flagged · score 5.0 of 10

Evaluation is partly circular: the verification module's answer-quality improvements are self-rated by the same MLLM that produced the answers, and the 'gain' metric counts the model's own prompted additions as quality; the central extraction claim retains some independent F1 evidence.

  1. self definitional [Section 4.4 (Verification-Answer Refinement) and Section 5.5.3 / Figure 7]
    "we set up an answer assessment module to evaluate the quality of generated answers... the assessment of answer quality is realized by guiding the LLMs to evaluate from four dimensions: accuracy, consistency, completeness, and relevance... The results show that: (1) As the number of optimization rounds increases, the overall quality rating of the answers consistently improves, demonstrating the effectiveness of the answer refinement module in enhancing answer quality."

    The same LLM that writes and refines the answers also assigns the 'accuracy/consistency/completeness/relevance' scores used as the outcome. The refinement prompt instructs the model to make its answer satisfy those four criteria; the evaluation prompt then asks the same model to rate whether the answer satisfies them. Therefore the improvement shown in Figure 7 is the model approving its own revisions, with no independent scorer, no human gold standard for the answers, and no external measure of threat-information quality. The claimed superiority of the answer-refinement module reduces to self-consistency of one MLLM with its own rubric.

  2. other [Section 5.3 (Performance of Attack Graph Integration), Figure 5; Section 4.5 (Integration)]
    "We evaluated the performance of the attack graphs in terms of entity, relationship, and the gain from utilizing image modality information. Namely, the incremental information of the updated multimodal attack graph over the text-based attack graph at the entity, relation, and technology levels."

    The 'gain' metric is defined as the number of entity/relation/technique items added by the multimodal graph relative to the text graph. Those additions are generated by the integration module, which is explicitly prompted to perform Node Extension, Relation Update, and Technique Addition from its own image-answer parsing. No precision, duplication, or grounding check is applied to the added items. Hence the reported 'gain' is essentially the module counting its own prompted output as quality improvement; the metric does not independently verify that the additions correspond to real information in the image. The abstract's claim that visual information 'significantly improve[s] the quality' is therefore supported by a self-referential output-count rather than by verified accuracy.

full rationale

The framework itself is a pipeline and does not derive a quantity from a fitted parameter; no equation in the paper is circular. The text-based constructor AttacKG+ is prior work by the same authors, but it is code-released and used as a component/baseline, so that citation is not load-bearing for the multimodal claim. The main circularity is in the evaluation: the verification module's quality scores are produced by the same LLM that generates and refines answers, so Figure 7's 'quality improvement' is self-ratings. Additionally, the 'gain' metric in Section 5.3 counts exactly the node/relation/technique additions that the integration module is prompted to produce, with no precision check, so the claimed 'quality improvement' is partially the model's own output by construction. Table 2 does provide precision/recall/F1 against manually assessed labels, which gives the central extraction claim some independent content; hence overall score 5.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about CTI images and LLM visual comprehension, plus a methodological assumption about the quality of the authors' manual annotations. No numeric free parameters are fitted. There are no invented entities.

assumptions (4)
  • domain assumption CTI images contain threat information that is non-redundant with the text and useful for attack graph construction.
    Motivation in Section 1 and Figure 1; the paper does not quantify how often images add non-redundant facts.
  • domain assumption Qwen2.5-VL models can reliably read cybersecurity diagrams, tables, code screenshots, and flowcharts with the given prompts.
    Used throughout Sections 4.2 to 4.4; no calibration against an external benchmark for image comprehension is provided.
  • ad hoc to paper The manually annotated AG-LLM-mm labels and the three-student verification process are a valid gold standard.
    Section 5.1.1 describes manual proofreading and assessment but gives no inter-annotator agreement, no label definition details, and no evidence that the annotators agree with an external standard.
  • ad hoc to paper The hand-written leading questions (Table 5) cover the aspects of threat images that matter for attack graph construction.
    Section 4.1 seeds generation with these questions; their completeness is assumed rather than derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MM-AttacKG: A Multimodal Approach to Attack Graph Construction with Large Language Models." pith.science (2026). https://pith.science/paper/D4PIDUVW

@misc{pith2026250616968,
  author       = {Pith},
  title        = {Pith review of: MM-AttacKG: A Multimodal Approach to Attack Graph Construction with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4PIDUVW}},
  note         = {Machine review of arXiv:2506.16968}
}
read the original abstract

Cyber Threat Intelligence (CTI) parsing aims to extract key threat information from massive data, transform it into actionable intelligence, enhance threat detection and defense efficiency, including attack graph construction, intelligence fusion and indicator extraction. Among these research topics, Attack Graph Construction (AGC) is essential for visualizing and understanding the potential attack paths of threat events from CTI reports. Existing approaches primarily construct the attack graphs purely from the textual data to reveal the logical threat relationships between entities within the attack behavioral sequence. However, they typically overlook the specific threat information inherent in visual modalities, which preserves the key threat details from inherently-multimodal CTI report. Therefore, we enhance the effectiveness of attack graph construction by analyzing visual information through Multimodal Large Language Models (MLLMs). Specifically, we propose a novel framework, MM-AttacKG, which can effectively extract key information from threat images and integrate it into attack graph construction, thereby enhancing the comprehensiveness and accuracy of attack graphs. It first employs a threat image parsing module to extract critical threat information from images and generate descriptions using MLLMs. Subsequently, it builds an iterative question-answering pipeline tailored for image parsing to refine the understanding of threat images. Finally, it achieves content-level integration between attack graphs and image-based answers through MLLMs, completing threat information enhancement. The experimental results demonstrate that MM-AttacKG can accurately identify key information in threat images and significantly improve the quality of multimodal attack graph construction, effectively addressing the shortcomings of existing methods in utilizing image-based threat information.

Figures

Figures reproduced from arXiv: 2506.16968 by the authors.

Figure 1
Figure 1. Motivation for incorporating images into constructing attack graphs is explained as follows. The images in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The atomic event data structure of the attack graph contains three parts: [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The overall framework of MM-AttacKG consists of five modules: (1) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The threat image parsing answer optimization paradigm. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Performance of MM-AttacKG’s threat gain in different prompting schemes and LLM versions. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Brainstorming questions generate evaluation situations. The abscissa represents the monotonicity of the [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: The answer quality distribution changes with the number of refinement rounds. (1) [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Example of multimodal attack graph Constrction (MM-AttacKG). [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: QA Example 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Answer Example 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Iteration Example 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Combine Answer Example 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GARAGE: Characterizing the Automation Boundary in LLM-based Attack Graph Generation

    cs.CR 2026-07 conditional novelty 5.0 of 10

    LLMs with automotive-CTI retrieval reconstruct high-level tactical attack paths for unseen vehicles but fail at implementation-level steps — a measured automation boundary for TARA support.

Reference graph

Works this paper leans on

91 extracted references · 66 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang.AttacKG: Constructing Technique Knowledge Graph from Cyber Threat Intelligence Reports, pages 589–609. 09 2022

    2022

  2. [2]

    Venkatakrishnan

    Kiavash Satvat, Rigel Gjomemo, and V . Venkatakrishnan. Extractor: Extracting attack behavior from threat reports, 04 2021

    2021

  3. [3]

    Threatkg: A threat knowledge graph for automated open-source cyber threat intelligence gathering and management, 12 2022

    Peng Gao, Xiaoyuan Liu, Edward Choi, Sibo Ma, Xinyu Yang, Zhengjie Ji, Zilin Zhang, and Dawn Song. Threatkg: A threat knowledge graph for automated open-source cyber threat intelligence gathering and management, 12 2022

    2022

  4. [4]

    Automated generation of attack graphs using nvd

    Muharrem Aksu, Kemal Bicakci, Mustafa Hadi Dilek, Murat Ozbayoglu, and Emin Tatlı. Automated generation of attack graphs using nvd. pages 135–142, 03 2018

    2018

  5. [5]

    Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources

    Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. pages 103–115, 12 2017. 17 Yongheng Zhang et al

    2017

  6. [6]

    Detection of breast cancer lesions using apt weighted mri: a systematic review.Journal of Translational Medicine, 23, 01 2025

    Ryan Lee, Montek Boparai, and Tim Duong. Detection of breast cancer lesions using apt weighted mri: a systematic review.Journal of Translational Medicine, 23, 01 2025

    2025

  7. [7]

    Improving apt detection with ensemble learning

    Emma Oye, Edwin Frank, and Jane Owen. Improving apt detection with ensemble learning. 12 2024

    2024

  8. [8]

    Reframing the role of academic professional track (apt) faculty

    Karthikeyan Loganathan, Shanna Banda, Lynn Peterson, Carter Tiernan, and Nila Veerabathina. Reframing the role of academic professional track (apt) faculty. 03 2025

    2025

Show all 91 references
  1. [9]

    Advanced persistent threat (apt), 10 2024

    Mostafizur Rahman Masum. Advanced persistent threat (apt), 10 2024

    2024

  2. [10]

    Automatically labeling cyber threat intelligence reports using natural language processing

    Hamza Abdi, Steven R Bagley, Steven Furnell, and Jamie Twycross. Automatically labeling cyber threat intelligence reports using natural language processing. InProceedings of the ACM Symposium on Document Engineering 2023, pages 1–4, 2023

    2023

  3. [11]

    Enhancing cyber threat in- telligence with named entity recognition using bert-crf

    Sheng-Shan Chen, Ren-Hung Hwang, Chin-Yu Sun, Ying-Dar Lin, and Tun-Wen Pai. Enhancing cyber threat in- telligence with named entity recognition using bert-crf. InGLOBECOM 2023-2023 IEEE Global Communications Conference, pages 7532–7537. IEEE, 2023

    2023

  4. [12]

    Attackg+: Boosting attack graph construction with large language models.Computers & Security, 150:104220, 2025

    Yongheng Zhang, Tingwen Du, Yunshan Ma, Xiang Wang, Yi Xie, Guozheng Yang, Yuliang Lu, and Ee-Chien Chang. Attackg+: Boosting attack graph construction with large language models.Computers & Security, 150:104220, 2025

    2025

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...

    2010 arXiv

  7. [15]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  8. [16]

    Mm-forecast: A multimodal approach to temporal event forecasting with large language models

    Haoxuan Li, Zhengmao Yang, Yunshan Ma, Yi Bin, Yang Yang, and Tat-Seng Chua. Mm-forecast: A multimodal approach to temporal event forecasting with large language models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 2776–2785, 2024

    2024

  9. [19]

    AlienVault OTX, 2022

    2022

  10. [20]

    Acing the ioc game: Toward automatic discovery and analysis of open-source cyber threat intelligence

    Xiaojing Liao, Kan Yuan, XiaoFeng Wang, Zhou Li, Luyi Xing, and Raheem Beyah. Acing the ioc game: Toward automatic discovery and analysis of open-source cyber threat intelligence. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 75...

    2016

  11. [22]

    IBM X-Force Exchange, 2022

    2022

  12. [24]

    Semi-automated information extraction from unstructured threat advisories

    Roshni R Ramnani, Karthik Shivaram, and Shubhashis Sengupta. Semi-automated information extraction from unstructured threat advisories. InProceedings of the 10th Innovations in Software Engineering Conference, pages 181–187, 2017

    2017

  13. [25]

    A supervised machine learning based approach for automatically extracting high-level threat intelligence from unstructured sources

    Yumna Ghazi, Zahid Anwar, Rafia Mumtaz, Shahzad Saleem, and Ali Tahir. A supervised machine learning based approach for automatically extracting high-level threat intelligence from unstructured sources. In2018 International Conference on Frontiers of Information Technology (FI...

    2018

  14. [26]

    Using entropy and mutual information to extract threat actions from cyber threat intelligence

    Ghaith Husari, Xi Niu, Bill Chu, and Ehab Al-Shaer. Using entropy and mutual information to extract threat actions from cyber threat intelligence. In2018 IEEE international conference on intelligence and security informatics (ISI), pages 1–6. IEEE, 2018

    2018

  15. [27]

    Cyberentrel: Joint extraction of cyber entities and relations using deep learning.Computers & Security, 136:103579, 2024

    Kashan Ahmed, Syed Khaldoon Khurshid, and Sadaf Hina. Cyberentrel: Joint extraction of cyber entities and relations using deep learning.Computers & Security, 136:103579, 2024

    2024

  16. [28]

    Threat action extraction based on coreference resolution

    Dengtian Mao, Ruike Zhao, Rui He, Pengyi He, Fanghui Ning, and Lingqi Zeng. Threat action extraction based on coreference resolution. InInternational Conference on Database Systems for Advanced Applications, pages 207–221. Springer, 2023

    2023

  17. [29]

    Atdg: An automatic cyber threat intelligence extraction model of dpcnn and bigru combined with attention mechanism

    Bo Cui, Jinling Li, and Wenhan Hou. Atdg: An automatic cyber threat intelligence extraction model of dpcnn and bigru combined with attention mechanism. InInternational Conference on Web Information Systems Engineering, pages 189–204. Springer, 2023. 18 Yongheng Zhang et al

    2023

  18. [30]

    A span-based multivariate information- aware embedding network for joint relational triplet extraction of threat intelligence.Knowledge-Based Systems, 295:111829, 2024

    Wenli Shang, Bowen Wang, Pengcheng Zhu, Lei Ding, and Shuang Wang. A span-based multivariate information- aware embedding network for joint relational triplet extraction of threat intelligence.Knowledge-Based Systems, 295:111829, 2024

    2024

  19. [31]

    Knowcti: Knowledge-based cyber threat intelligence entity and relation extraction.Computers & Security, 141:103824, 2024

    Gaosheng Wang, Peipei Liu, Jintao Huang, Haoyu Bin, Xi Wang, and Hongsong Zhu. Knowcti: Knowledge-based cyber threat intelligence entity and relation extraction.Computers & Security, 141:103824, 2024

    2024

  20. [33]

    Automated threat report classification over multi-source data

    Gbadebo Ayoade, Swarup Chandra, Latifur Khan, Kevin Hamlen, and Bhavani Thuraisingham. Automated threat report classification over multi-source data. In2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC), pages 236–245. IEEE, 2018

    2018

  21. [34]

    Seqmask: Behavior extraction over cyber threat intelligence via multi-instance learning.The Computer Journal, 67(1):253–273, 2024

    Wenhan Ge and Junfeng Wang. Seqmask: Behavior extraction over cyber threat intelligence via multi-instance learning.The Computer Journal, 67(1):253–273, 2024

    2024

  22. [35]

    Threat intelligence att&ck extraction based on the attention transformer hierarchical recurrent neural network.Applied Soft Computing, 122:108826, 2022

    Chenjing Liu, Junfeng Wang, and Xiangru Chen. Threat intelligence att&ck extraction based on the attention transformer hierarchical recurrent neural network.Applied Soft Computing, 122:108826, 2022

    2022

  23. [36]

    Tim: threat context-enhanced ttp intelligence mining on unstructured threat data.Cybersecurity, 5(1):3, 2022

    Yizhe You, Jun Jiang, Zhengwei Jiang, Peian Yang, Baoxu Liu, Huamin Feng, Xuren Wang, and Ning Li. Tim: threat context-enhanced ttp intelligence mining on unstructured threat data.Cybersecurity, 5(1):3, 2022

    2022

  24. [37]

    Multimodal graph analysis of cyber attacks

    Nirnimesh Ghose, Loukas Lazos, Jerzy Rozenblit, and Ronald Breiger. Multimodal graph analysis of cyber attacks. In2019 Spring simulation conference (SpringSim), pages 1–12. IEEE, 2019

    2019

  25. [38]

    Nikolaos Bakalos, Athanasios V oulodimos, Nikolaos Doulamis, Anastasios Doulamis, Avi Ostfeld, Elad Salomons, Juan Caubet, Victor Jimenez, and Pau Li. Protecting water infrastructure from cyber and physical threats: Using multimodal data fusion and adaptive deep learning to mo...

    2019

  26. [39]

    {KnowPhish}: Large language models meet multimodal knowledge graphs for enhancing{Reference-Based} phishing detection

    Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Hoon Wei Lim, and Bryan Hooi. {KnowPhish}: Large language models meet multimodal knowledge graphs for enhancing{Reference-Based} phishing detection. In33rd USENIX Security Symposium (USENIX Security 24), pag...

    2024

  27. [40]

    Ex-action: Automatically extracting threat actions from cyber threat intelligence report based on multimodal learning.Security and Communication Networks, 2021(1):5586335, 2021

    Huixia Zhang, Guowei Shen, Chun Guo, Yunhe Cui, and Chaohui Jiang. Ex-action: Automatically extracting threat actions from cyber threat intelligence report based on multimodal learning.Security and Communication Networks, 2021(1):5586335, 2021

    2021

  28. [41]

    Apt-mmf: An advanced persistent threat actor attribution method based on multimodal and multilevel feature fusion.Computers & security, 144:103960, 2024

    Nan Xiao, Bo Lang, Ting Wang, and Yikai Chen. Apt-mmf: An advanced persistent threat actor attribution method based on multimodal and multilevel feature fusion.Computers & security, 144:103960, 2024

    2024

  29. [42]

    Efficient backdoor defense in multimodal contrastive learning: A token-level unlearning method for mitigating threats.arXiv preprint arXiv:2409.19526, 2024

    Kuanrong Liu, Siyuan Liang, Jiawei Liang, Pengwen Dai, and Xiaochun Cao. Efficient backdoor defense in multimodal contrastive learning: A token-level unlearning method for mitigating threats.arXiv preprint arXiv:2409.19526, 2024

    2024 arXiv

  30. [43]

    Nightshade: Prompt-specific poisoning attacks on text-to-image generative models

    Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng, and Ben Y Zhao. Nightshade: Prompt-specific poisoning attacks on text-to-image generative models. In2024 IEEE Symposium on Security and Privacy (SP), pages 807–825. IEEE, 2024

    2024

  31. [44]

    Jailguard: A universal detection framework for llm prompt-based attacks.arXiv preprint arXiv:2312.10766, 2023

    Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, and Chao Shen. Jailguard: A universal detection framework for llm prompt-based attacks.arXiv preprint arXiv:2312.10766, 2023

    2023 arXiv

  32. [45]

    Fundamentals of generative large language models and perspectives in cyber-defense.arXiv preprint arXiv:2303.12132, 2023

    Andrei Kucharavy, Zachary Schillaci, Loïc Maréchal, Maxime Würsch, Ljiljana Dolamic, Remi Sabonnadiere, Dimitri Percia David, Alain Mermoud, and Vincent Lenders. Fundamentals of generative large language models and perspectives in cyber-defense.arXiv preprint arXiv:2303.12132, 2023

    2023 arXiv

  33. [46]

    Llms perform poorly at concept extraction in cyber-security research literature.arXiv preprint arXiv:2312.07110, 2023

    Maxime Würsch, Andrei Kucharavy, Dimitri Percia David, and Alain Mermoud. Llms perform poorly at concept extraction in cyber-security research literature.arXiv preprint arXiv:2312.07110, 2023

    2023 arXiv

  34. [47]

    Raglog: Log anomaly detection using retrieval augmented generation

    Jonathan Pan, Wong Swee Liang, and Yuan Yidi. Raglog: Log anomaly detection using retrieval augmented generation. In2024 IEEE World Forum on Public Safety Technology (WFPST), pages 169–174. IEEE, 2024

    2024

  35. [48]

    Revolutionizing cyber threat detection with large language models.arXiv preprint arXiv:2306.14263, pages 195–202, 2023

    Mohamed Amine Ferrag, Mthandazo Ndhlovu, Norbert Tihanyi, Lucas C Cordeiro, Merouane Debbah, and Thierry Lestable. Revolutionizing cyber threat detection with large language models.arXiv preprint arXiv:2306.14263, pages 195–202, 2023

    2023 arXiv

  36. [49]

    From text to mitre techniques: Exploring the malicious use of large language models for generating cyber attack payloads.arXiv preprint arXiv:2305.15336, 2023

    PV Charan, Hrushikesh Chunduri, P Mohan Anand, and Sandeep K Shukla. From text to mitre techniques: Exploring the malicious use of large language models for generating cyber attack payloads.arXiv preprint arXiv:2305.15336, 2023

    2023 arXiv

  37. [50]

    Out of the cage: How stochastic parrots win in cyber security environments.arXiv preprint arXiv:2308.12086, 2023

    Maria Rigaki, Ondˇrej Lukáš, Carlos A Catania, and Sebastian Garcia. Out of the cage: How stochastic parrots win in cyber security environments.arXiv preprint arXiv:2308.12086, 2023. 19 Yongheng Zhang et al

    2023 arXiv

  38. [51]

    Llms killed the script kiddie: How agents supported by large language models change the landscape of network threat testing.arXiv preprint arXiv:2310.06936, 2023

    Stephen Moskal, Sam Laney, Erik Hemberg, and Una-May O’Reilly. Llms killed the script kiddie: How agents supported by large language models change the landscape of network threat testing.arXiv preprint arXiv:2310.06936, 2023

    2023 arXiv

  39. [52]

    Building resilient smes: Harnessing large language models for cyber security in australia

    Ben Kereopa-Yorke. Building resilient smes: Harnessing large language models for cyber security in australia. Journal of AI, Robotics & Workplace Automation, 3(1):15–27, 2024

    2024

  40. [53]

    Review the list of existing questions provided and generate new questions that explore different perspectives, details, or contexts within the image

  41. [54]

    Comprehensive Threat Intelligence

  42. [55]

    Generate new questions that explore different perspectives or details of the image based on the information from the knowledge graph

  43. [56]

    A survey on in-context learning.arXiv preprint arXiv:2301.00234, 2022

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  44. [57]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  45. [58]

    Covert Grove

    Xinyun Zhao, Yongheng Zhang, Qingying Zhai, Jinrui Zhang, and Lanlan Qi. A multi-attribute decision-making approach for critical node identification in complex networks.Entropy, 26(12):1075, 2024. A Here we show sample tasks for each stage of the MM-AttacKG runtime: Figure 9 s...

    2024

  46. [59]

    These new questions should help further analyze the image from different perspectives

  47. [60]

    These new questions should be related to cybersecurity or assist in the analysis of cyber threat intelligence

  48. [61]

    What is/are the XXX of/in the image?

    Refer to the format of the given questions, which follows the pattern: "What is/are the XXX of/in the image?", "Where XXX should be replaced with a specific aspect of the image?". Question Answering You are a cybersecurity threat intelligence analyst. Please answer the questio...

  49. [62]

    Your answer must strictly adhere to the content visible in the image when mentioning any entities, objects, and their relationships

  50. [63]

    Your answer must include a topic phrase that is specific to the question

  51. [64]

    Your answer should be a single, concise sentence

  52. [65]

    Do not provide explanations or reasons for uncertainty

    Only provide the direct answer to the question. Do not provide explanations or reasons for uncertainty. Answer Evaluation You are a cybersecurity threat intelligence analyst. Please rate the description based on the following rules:

  53. [66]

    - Consistency: Consistency represents whether the description maintains content relevance to the image information

    Evaluate the description using the following four criteria: - Accuracy: Accuracy represents whether the description accurately answers the question. - Consistency: Consistency represents whether the description maintains content relevance to the image information. - Completene...

  54. [67]

    excellent

    Apply the following rating scale based on the overall quality: - "excellent": The description meets three of the criteria with only minor flaws or imperfections. - "good": The description meets two of the criteria with small deviations or omissions that do not significantly im...

  55. [68]

    If there are statements in the description such as unknown, no details, not mentioned, etc., mark it as "failing"

  56. [69]

    excellent

    Your answer should be a single word: either "excellent", "good", "satisfactory", or "failing". Answer Optimization You are a cybersecurity threat intelligence analyst. Please provide your answers to the following questions again, using the image as a reference, based on the fo...

  57. [70]

    - Consistency: Consistency represents the answer maintains content relevance to the image infor- mation

    Re-answer the questions to ensure the answers meet the following four criteria: - Accuracy: Accuracy represents the answer accurately answers the question. - Consistency: Consistency represents the answer maintains content relevance to the image infor- mation. - Completeness: ...

  58. [71]

    Improve existing unqualified answers (Paradigm 1) or re-answer questions based on suggestions provided (Paradigm 2)

  59. [72]

    26 Yongheng Zhang et al

    Ensure the revised answer differs from the previous unqualified answer (Paradigm 1), or strictly follows the suggestions given (Paradigm 2). 26 Yongheng Zhang et al. Table 7: Prompts of ICL. Extraction of Entities and Relation You are a cyber-security information-extraction sp...

  60. [73]

    Image In-context; 3

    An image; 2. Image In-context; 3. CTI summary; 4. Entity Types & Descriptions; 5. Relation Table. [Example] Input: Image: ... attack path from public Internet→SSH→Public-facing Server1... In-context: This led our responders to identify the occurrence of... CTI summary: CrowdSt...

  61. [74]

    Image context; 3

    CTI image; 2. Image context; 3. CTI summary; 4. MITRE ATT&CK tactics list. [Example] Input: Image: ... attack path from public Internet→SSH→Public-facing Server1... In-context: This led our responders to identify the occurrence of... CTI summary: CrowdStrike’s analysis of the ...

  62. [75]

    New Node Addition: add a new entity to extend an existing triplet

  63. [76]

    New Relationship Addition: link entities from different triplets

  64. [77]

    Technique Addition: tag an existing triplet with a new MITRE ATT&CK technique. [Input]

  65. [78]

    type":"new_node_addition

    CTI image; 2.Knowledge Graph (list of triplets); 3. Entity Types & Descriptions; 4.MITRE ATT&CK techniques list. [Rules] Rule 1: New Node Addition: if the image shows an entity related to a KG triplet, output a JSON list [...] with objects containing: ... description: reason f...

  66. [79]

    Spot entities in the image

  67. [80]

    Map each to the shortest matching provided type

  68. [81]

    Identify clear, active-voice relationships between them

  69. [82]

    Extraction of Techniques You are a cyber-threat intelligence expert

    Form triplets. Extraction of Techniques You are a cyber-threat intelligence expert. Given a CTI image (with optional context/summary) and a list of MITRE ATT&CK tactics, identify the single tactic that best matches the image. [Thinking Process]

  70. [83]

    Inspect the image for attack indicators (flowcharts, logs, interfaces)

  71. [84]

    Identify candidate tactics from the provided list

  72. [85]

    Select the one tactic that most directly reflects what you see

  73. [86]

    Integrated Attack Graph You are a cyber-threat intelligence expert

    Ensure it fits the attack phase implied by the image. Integrated Attack Graph You are a cyber-threat intelligence expert. Given a CTI image, a Knowledge Graph of triplets, Entity Types & Descriptions, and MITRE ATT&CK techniques, extend the graph by discovering:

  74. [87]

    New Node Addition: adding an image-derived entity to an existing triplet

  75. [88]

    New Relationship Addition: linking entities from different triplets

  76. [89]

    [Thinking Process]

    Technique Addition: tagging a triplet with a new MITRE technique. [Thinking Process]

  77. [90]

    Identify triplets strongly correlated with the image

  78. [91]

    Extract and type entities visible in the image

  79. [92]

    For each strong-match triplet: check if an image entity adds a new connection to its subject or object (new node addition)

  80. [93]

    Check if an entity in this triplet should link to an entity in another triplet (new relationship addition)

  81. [94]

    Determine if the image implies a new MITRE technique for the most relevant triplet (technique addition)

  82. [95]

    Assemble JSON outputs

  83. [96]

    technique_id - technique_name

    If none apply, output No Match. [Rules] Rule 1: New Node Addition: if the image shows an entity related to a KG triplet, output a JSON list [...] with objects containing: ... description: reason for adding the node new_node: {id, type, properties:{description}} relationship: {...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.