{"id":"ce3d6a35-2c63-455e-860a-effb5de79873","arxiv_id":"2506.16968","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MM-AttacKG is a pipeline that uses multimodal LLMs to extract threat information from images in cyber threat reports and merge it into text-derived attack graphs, improving entity, relation, and technique coverage.","lead":"This paper builds attack graphs, the step-by-step maps of how a cyberattack happens, by teaching a multimodal language model to read the images inside threat reports in addition to the text. A system like this could give security analysts a fuller picture of an attack faster and with less manual work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation lacks any precision check on image-derived graph additions; the central claim of 'accurate' multimodal extraction is not supported by the reported metrics.","rationale":"The reader identified the same weakest assumption: the evaluation relies on unverified gold labels and a gain metric with no precision check. My reading of the full manuscript confirms this and sharpens it: the 'gain' metric in Section 5.3 and Figure 5 is purely a count of additions, with no correctness filter; the answer quality assessment is performed by the same MLLM being evaluated (Table 6, Answer Evaluation), making the quality signal circular; and the manual verification described in Section 5.1.1 applies to text extraction, not to the image-derived additions. The paper is internally coherent and the framework is described in detail, but the central claim of 'accurate' identification and 'significantly improved quality' is exactly what the evaluation fails to measure. Since this is a fixable evaluation gap rather than a fundamental flaw in the approach, CONDITIONAL is the right verdict: accept only if the authors release the dataset and code, provide a human-verified precision check on image-derived additions, report inter-annotator agreement, and add variance/statistical significance across runs (the paper reports no variance despite setting fixed temperature and seed).","tokens_in":23958,"tokens_out":1725,"duration_ms":15131,"concrete_test":"Manually inspect a random sample of at least 50 image-derived additions from the AG-LLM-mm dataset (entities, relations, and techniques reported as 'gain' in Section 5.3). For each addition, have two independent annotators with cybersecurity expertise check (a) whether the item is actually visible or directly supported by the corresponding threat image, (b) whether the item is already present in the text-based attack graph (i.e., duplicate), and (c) whether the technique ID matches the MITRE ATT&CK definition. Report precision per category and per annotator agreement. If precision of image-derived additions is below, say, 0.8 in any category, the reported gain is not evidence of improved attack graph quality and the central claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MM-AttacKG 'can accurately identify key information in threat images and significantly improve the quality of multimodal attack graph construction.' The paper's own evaluation does not substantiate either the accuracy of identification or the quality of the additions. Section 5.3 measures 'gain' as the raw count of entities, relations, and techniques added by image analysis over the text-based graph. There is no verification that these additions are correct, non-duplicative, or grounded in the image; no precision is computed for the added items. If a substantial fraction of the image-derived additions are hallucinations or duplicates of text-derived information, the reported gain would be misleading: the attack graph would be no more accurate or complete in any meaningful sense, and the central claim that the system 'accurately identifies key information' would be false. The annotation process described in Section 5.1.1 is likewise not a ground-truth check for the image-derived additions: three postgraduate students 'manually proofread and assessed' text extraction, with no guidelines, no inter-annotator agreement, and no external standard, and they did not independently verify that each image-derived entity, relation, or technique is present in the image. The only quality signal for image answers is the self-evaluation by the same MLLM (the Verification module) using the prompt template in Table 6 (Answer Evaluation), which is circular: the model judges its own generations. The paper's own Table 2 shows technique F1 at 0.5703, so even under its own labels, technique identification is frequently wrong; yet the 'gain' in Section 5.3 counts every added technique without any precision filter. This is the load-bearing weakness: the headline experimental quantity (gain) is not validated against any source of truth, so the central claim of accurate and improved multimodal attack graphs is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-AttacKG, a pipeline for integrating information from images in cyber threat intelligence (CTI) reports into attack graph construction. The framework consists of five modules: text-based graph construction (using AttacKG+), a Brainstorming module that generates general and task-specific questions about a threat image, an Extraction module that answers those questions with MLLM assistance and report context, a Verification module that filters questions and iteratively refines answers based on self-evaluation, and an Integration module that adds image-derived entities, relations, and MITRE techniques to the text-based attack graph. The authors introduce a new dataset, AG-LLM-mm, collected from Cisco Talos and Microsoft reports, and evaluate MM-AttacKG against ICL and CoT prompting baselines using Qwen2.5-VL models. They report that image-enhanced methods outperform text-based extractors on entity, relation, and technique extraction, and that MM-AttacKG achieves higher F1 and larger 'gain' than the baselines. The central claim is that MM-AttacKG accurately identifies key information in threat images and significantly improves multimodal attack graph construction.","tokens_in":24245,"tokens_out":4360,"duration_ms":43910,"significance":"The problem addressed is real and timely: most attack graph construction ignores the visual content of CTI reports, and a systematic method for incorporating threat images could improve threat intelligence analysis. The proposed pipeline is clearly described, and the authors provide prompt templates and a new dataset, which are potentially useful resources. However, the evaluation as presented does not substantiate the central claims. The gain metric counts unverified additions, the verification module uses the same model to judge its own outputs, the annotation procedure lacks documented guidelines and inter-annotator agreement, and no significance testing is reported. If the authors can add independent validation of image-derived additions and reframe the comparisons appropriately, the work could be valuable. As it stands, the paper documents an interesting system but not the claimed accuracy improvement.","major_comments":[{"comment":"The gold standard for the precision/recall/F1 numbers in Table 2 is not defined. The text says three postgraduate students 'manually proofread and assessed' the data, but no annotation guidelines, no inter-annotator agreement, and no external standard are reported. The row 'Human Anotation-Text' has precision 1.0000 for all three tasks, which indicates it is being used as the reference, not as a rival method; including it as a baseline conflates ground truth with a competing system. Moreover, the text-based methods have no access to images, so their lower recall relative to image-enhanced methods is expected by construction; Table 2 therefore does not support the claim of 'advantages of multimodal integration.' A controlled comparison is needed, e.g., an LLM-based text-only pipeline with the same prompts, or a comparison that reports performance separately on the text-only and image-only portions of the gold standard.","section":"5.1.1, Table 2"},{"comment":"The 'gain' metric is defined as the incremental number of entities, relations, and techniques added to the text-based attack graph, with no precision check on the added items. A hallucinated entity, a duplicate of text-derived information, or an irrelevant relation all count as positive gain. The abstract claims that MM-AttacKG 'can accurately identify key information in threat images'; this requires verifying that each image-derived addition is correct, non-duplicative, and grounded in the image, and reporting the precision of those additions. Without such verification, the high gain numbers in Figure 5 could reflect hallucinated rather than genuine threat information, undermining the central claim.","section":"5.3, Figure 5"},{"comment":"The answer refinement evaluation is circular. The same MLLM that produces the answers also rates them on accuracy, consistency, completeness, and relevance using the prompt in Table 6 (Answer Evaluation), and then refines its own answers based on its own suggestions. Figure 7 therefore shows the model agreeing with itself, not an objective improvement in answer quality. To support the claim that the verification module improves answer quality, the authors should use an independent judge (a different model or human raters) and report inter-rater agreement, or compare refined answers against a fixed external reference.","section":"4.4, 5.5.3, Table 6, Figure 7"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the main results. Differences such as entity F1 0.7716 vs 0.7623 (Table 3, w/o V) or relation F1 0.8147 vs 0.8099 may be within run-to-run noise, especially with temperature 0.7. The abstract's 'significantly improve' is therefore not supported by statistical evidence. The authors should report variance over multiple runs and, where feasible, paired significance tests for the systems being compared.","section":"5.2-5.4, Tables 2-3, Figure 5"}],"minor_comments":[{"comment":"The text states that 'monotonicity remains stable and close to [1],' but the definition given is that higher monotonicity indicates more similar questions and thus poorer diversity; a value close to 1 would indicate poor diversity, contradicting the stated conclusion of high diversity. Please clarify whether the figure's values are close to 0 or whether the scale is inverted.","section":"5.5.1, Figure 6"},{"comment":"References [6] through [9] appear to be unrelated to cyber threat intelligence; they discuss breast cancer MRI, ensemble learning, and academic professional track faculty, and use 'APT' in a different sense. These citations should be corrected or removed.","section":"References [6]-[9]"},{"comment":"There are several typos and inconsistencies: 'EXTROCTOR' in Section 5.1.2, 'Human Anotation-Text' in Table 2, 'Constrction' in Figure 8, 'we introduces' in Section 6.1, and reference [2] cited for Qwen models without a proper Qwen technical report reference. Please proofread and standardize the notation.","section":"Throughout"},{"comment":"The code and dataset are stated to be released upon acceptance; for a reproducibility-focused evaluation, consider providing at least the annotation guidelines, the list of report sources, and the exact prompts used in an appendix or supplementary material even before acceptance.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has citation hygiene problems beyond the scope of a normal minor revision: references [6]-[9] appear to be placeholder or erroneous entries, and several references are incomplete (e.g., 'Mitre', 'STIX', 'Microsoft'). This suggests the reference list was not carefully checked. Also, the core evaluation gap (no precision check on image-derived additions, circular self-verification) is fixable but will require additional human annotation experiments; the authors should be encouraged to conduct those rather than merely adding caveats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first paper I know of that seriously attempts to use multimodal LLMs to parse threat images for attack graph construction, and the framework is coherent. The evaluation, though, does not yet establish the accuracy claim in the abstract, because the headline 'gain' metric counts additions without checking whether they are correct.\n\nWhat is genuinely new and good: the task definition is real. CTI reports are multimodal, and text-only pipelines leave image information on the table. The AG-LLM-mm dataset, even unreleased, is a useful asset. The pipeline (leading questions, iterative QA, verification, integration) is a reasonable decomposition, and the ablations suggest the context and refinement stages matter. The qualitative case study is illustrative. I give credit for tackling a gap that the existing AttacKG/ThreatKG line ignores.\n\nWhere it falls short: first, ground truth is not established. Three students 'proofread,' but there are no annotation guidelines, no inter-annotator agreement, and no external standard. Table 2 compares text-only systems (no image access) with image-enhanced ones, so higher recall is partly by construction. Second, the gain metric in Section 5.3 counts every added entity, relation, and technique with no precision check. If a chunk of those additions are hallucinations, the 'significant improvement' is an artifact. The paper's own Table 2 shows technique F1 of 0.57, so errors are common even on its own labels. Third, the verification module is circular: the same MLLM grades its own answers on accuracy, consistency, completeness, and relevance, then refines from that self-feedback. Figure 7 shows the model agreeing with itself. Fourth, there are no error bars or significance tests, and code and data are withheld until acceptance. Minor issues: a few references seem unrelated, and the appendix text is mangled, but those do not affect the core argument.\n\nI do not think this should be desk-rejected. The novelty is real, the problem is worth solving, and the fixes are concrete: release the dataset with verified labels, compute precision on image-derived additions, add a text-only ablation of the same pipeline, report variance, and have independent raters judge the image additions. If those land, the central claim becomes testable. As it stands, I would call it a solid system paper with an evaluation that needs heavy revision.","headline":"First real attempt to use MLLMs for image-based attack graph construction; worth peer review, but the headline evaluation needs a precision check before the central accuracy claim can stand.","tokens_in":24848,"tokens_out":3276,"would_cite":true,"duration_ms":34970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that threat images in cyber threat intelligence reports carry attack information that text alone misses, and that a multimodal large language model pipeline, MM-AttacKG, can extract it and integrate it into text-based…","keywords":["attack graph construction","cyber threat intelligence","multimodal large language models","threat image parsing","MITRE ATT&CK","threat information extraction","AG-LLM-mm dataset"],"falsifier":"Have multiple independent cybersecurity analysts annotate the threat images in a sample of AG-LLM-mm for entities, relations, and MITRE techniques, then run MM-AttacKG on the same images and compare the image-derived additions against the independent annotations. If the added items largely fail to match the human labels, the claim of accurate image information identification and graph enhancement is not supported.","tokens_in":23770,"feed_emoji":"🛡️","tokens_out":5241,"duration_ms":51794,"temperature":0.7,"pith_summary":"The paper sets out to show that images inside cyber threat intelligence (CTI) reports are not redundant decorations: they carry entities, relations, and MITRE ATT&CK techniques that text-based attack graph construction leaves out. To demonstrate this, it builds a five-stage pipeline, MM-AttacKG, that turns image parsing into a question-answering dialogue, filters and iteratively refines the answers, and then integrates the surviving image-derived information into an attack graph produced from report text. On the new AG-LLM-mm dataset, the image-enhanced output reports higher F1 than text-only extractors and than simpler ICL and CoT prompt schemes, with the image contributions concentrated at the entity, relation, and technique levels.","feed_headline":"CTI images add attack steps that text-only graphs miss","feed_subtitle":"MM-AttacKG pulls entities, relations, and MITRE techniques from report images and merges them into attack graphs.","key_machinery":"The carrying mechanism is the image-to-graph enhancement loop. Brainstorming generates general and task-specific questions about each threat image; Extraction answers them using the image plus two textual supports, Image-Aware-Context (what the image shows) and Global-Context (the report outline); Verification filters questions by direct and answer-oriented relevance and refines weak answers through iterative question-led or answer-iteration rounds scored on accuracy, consistency, completeness, and relevance; Integration turns each accepted question-answer pair into a threat enhancement reference that adds nodes, updates relations, or adds techniques to the text-based attack graph.","core_discovery":"The central claim, framed as a new application of multimodal LLMs to attack graph construction, is that threat images contain key information absent from the text, and that MM-AttacKG identifies it accurately enough to improve the completeness of the resulting attack graph. The authors formalize the task by extending the text-based atomic event quintuple $(s, a, o, t, p)$ to include image-derived threat enhancement references, and they define three integration operations: node extension, relation update, and technique addition under the MITRE ATT&CK framework. Reported results on AG-LLM-mm show the full pipeline at F1 0.7716 for entities, 0.8147 for relations, and 0.5703 for techniques, and the integration experiments show higher image-derived gain than ICL and CoT prompts across Qwen2.5-VL model sizes.","pith_inferences":["The gain metric counts every image-derived addition as valuable without checking precision, so a direct test of the paper's 'accurately identifies' claim would measure whether a human analyst agrees with each added entity, relation, and technique; the paper does not report that precision.","Because the pipeline depends on AttacKG+ for its text graphs, the multimodal claim could be tested by swapping in a different text constructor: if the gains vanish, the benefit may come from the text parser rather than the image module.","The question-answer-refine loop resembles general self-consistency methods for multimodal QA, so one could test whether the same verification loop improves other image-to-knowledge tasks, such as extracting indicators of compromise from screenshots or logs."],"forward_implications":["If images are non-redundant with text, then any CTI pipeline that ignores images understates attack paths, and multimodal extraction should become part of standard attack graph construction.","The framework's prompt-only design means attack graph enrichment can ride on open-source multimodal LLMs without fine-tuning on labeled image data.","The AG-LLM-mm dataset gives the community a first benchmark for evaluating image-derived entities, relations, and TTPs in CTI reports.","The verification loop of question filtering plus answer refinement is the component most directly tied to accuracy; ablations show removing it lowers F1 on relation and technique extraction."],"supporting_citations":[{"why":"Supplies the text-based attack graph formulation and TTP structure that MM-AttacKG extends.","marker":"[1]"},{"why":"Earlier automatic attack behavior graph extractor used as a baseline for comparison.","marker":"[2]"},{"why":"AttacKG+, the LLM-based text graph constructor that MM-AttacKG uses as its backbone and from which it derives image-aware and global contexts.","marker":"[12]"},{"why":"MITRE ATT&CK matrix, the taxonomy that provides the technique labels used in the technique addition step.","marker":"[32]"},{"why":"STIX 2.0 standard, which supplies the attack graph ontology model used to define atomic threat events.","marker":"[53]"},{"why":"Cisco Talos Intelligence Group, one of the two sources of CTI reports used to build the AG-LLM-mm dataset.","marker":"[54]"},{"why":"Microsoft Security Intelligence Center, the other source of CTI reports for the dataset.","marker":"[55]"},{"why":"In-context learning baseline that MM-AttacKG is compared against in the multimodal extraction and integration experiments.","marker":"[56]"},{"why":"Chain-of-thought prompting baseline used for comparison in the same experiments.","marker":"[57]"}],"fun_headline_variants":["Threat images fill gaps text-only attack graphs miss","Multimodal LLMs add attack steps from CTI report images","Images in CTI reports boost attack graph completeness","Attack graphs gain from visual clues LLMs extract from images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the human-proofread gold labels and the gain metric measure true attack graph quality; the paper does not report inter-annotator agreement on the labels, and the gain metric counts every image-derived addition as correct with no precision check.","fun_headline_variants_meta":{"raw":{"variants":["Threat images fill gaps text-only attack graphs miss","Multimodal LLMs add attack steps from CTI report images","Images in CTI reports boost attack graph completeness","Attack graphs gain from visual clues LLMs extract from images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1876,"prompt_tokens":1004,"completion_tokens":872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":806}},"tokens_in":620,"tokens_out":872,"duration_ms":7641,"temperature":1.0,"reasoning_tokens":806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:14:38.655379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have multiple independent cybersecurity analysts annotate the threat images in a sample of AG-LLM-mm for entities, relations, and MITRE techniques, then run MM-AttacKG on the same images and compare the image-derived additions against the independent annotations. If the added items largely fail to match the human labels, the claim of accurate image information identification and graph enhancement is not supported.","supporting_citations":[{"cited_title":"Attackg+: Boosting attack graph construction with large language models.Computers & Security, 150:104220, 2025","cited_arxiv_id":null,"evidence_quote":"AttacKG+, the LLM-based text graph constructor that MM-AttacKG uses as its backbone and from which it derives image-aware and global contexts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"STIX 2.0 standard, which supplies the attack graph ontology model used to define atomic threat events."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cisco Talos Intelligence Group, one of the two sources of CTI reports used to build the AG-LLM-mm dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Microsoft Security Intelligence Center, the other source of CTI reports for the dataset."}],"review_version":1}