{"id":"f1a7ee48-12d7-4214-b47f-2a167ce2a91a","arxiv_id":"2411.08359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MultiKG automatically extracts and merges attack technique knowledge graphs from CTI reports, audit logs, and static code, and reports higher accuracy than a single-source baseline.","lead":"This paper presents MultiKG, a system that merges cyber threat reports, system audit logs, and attack source code into fine-grained knowledge graphs of attack techniques. It is a candidate building block for automating attack reconstruction and detection in security operations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy claims rest on small author-created ground truth with no inter-annotator agreement; independent re-labeling is needed before the reported F1 can be trusted.","rationale":"I read the paper in good faith: MultiKG is a coherent pipeline that combines audit logs, static code, and CTI reports into technique-level knowledge graphs, and the algorithmic description is detailed enough to be credible as an engineering contribution. The reader's verdict of CONDITIONAL is appropriate. The single most load-bearing weakness is indeed the evaluation ground truth: it is small (10 log-based, 11 CTI, 6 cross-source), author-created, lacks inter-annotator agreement, and for the cross-source comparison is a manual merge of the same graphs the system merges. No code, data, labels, or prompts are released, so the reported accuracy numbers cannot be independently checked. This concern does not invalidate the system's design, but it does mean the headline quantitative claims should not be taken at face value. The concern is addressable—release the artifacts and have independent annotators re-label a held-out set—so CONDITIONAL, not REJECT, remains the right verdict. My analysis does not change the reader's verdict, so I mark the verdict as UNCHANGED.","tokens_in":24910,"tokens_out":3069,"duration_ms":31184,"concrete_test":"Recruit two independent security analysts who were not involved in the paper to re-annotate the 10 log-based techniques, the 11 CTI reports, and the 6 cross-source technique graphs using a pre-registered annotation schema for entities, relationships, and types, without access to MultiKG outputs or the authors' labels. Compute Cohen's kappa between the two annotators and recompute the F1 scores in Tables 4, 5, and 7 against each annotator's labels separately. If kappa is below 0.6, or if the recomputed F1 scores drop by more than 5 percentage points relative to the reported values, the reported accuracy is not robust to labeling subjectivity and should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of MultiKG—accurate, complete, unified technique-level knowledge graphs—is supported almost entirely by accuracy figures computed against manually constructed ground truth: 10 log-based techniques (Table 4, Section 6.2.1), 11 CTI reports (Table 5), and manually completed cross-source merges for 6 techniques (Table 7, Section 6.2.3). The labels were produced by the authors, who executed the attacks and know the expected behavior, and no annotation schema, annotator count, or inter-annotator agreement measure is reported. For the cross-source comparison, the ground truth is a manual merge of the same input graphs that the algorithm merges; if the authors' expectations shaped their manual merge, the near-perfect recall values in Table 7 are inflated. Because the labels, prompts, and code are not released, the numbers are unfalsifiable as presented. This is load-bearing because every positive result in RQ1 and RQ3—93.8% entity accuracy, 91.4% relationship accuracy, and roughly double AttacKG's F1—is measured against this ground truth. The engineering contribution is plausible, but the quantitative claims are not yet established independently of the authors' own judgment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiKG, a fully automated framework that constructs attack technique knowledge graphs by combining three data sources: CTI reports parsed with GPT-4, dynamic audit logs collected via ETW, and static code analyzed with abstract syntax trees. The framework builds separate technique-level graphs for each source, aggregates multiple same-source graphs using a same-level BFS merging rule, and then merges cross-source graphs into a unified technique knowledge graph. The authors evaluate the system on 1,015 Atomic Red Team procedures and 9,006 MITRE ATT&CK descriptions, reporting 93.8% entity accuracy and 91.4% relationship accuracy on 10 log-based executions, 86.4% entity accuracy on 11 CTI reports, compression ratios for aggregation, and an F1 score roughly twice that of AttacKG on a 6-technique cross-source comparison.","tokens_in":25123,"tokens_out":8238,"duration_ms":67842,"significance":"If the reported quantitative results hold, MultiKG would be a useful contribution: it is the first system presented here to integrate audit logs, static code, and CTI reports into a unified, technique-level attack knowledge graph, and the algorithmic pipeline is described in enough detail to be reimplemented. Concrete strengths include the large crawled dataset (1,015 procedures, 9,006 report descriptions), the concrete algorithm pseudocode for extraction and merging, and the case studies that illustrate downstream reconstruction and detection value. However, the demonstrated significance is currently limited by an evaluation that rests on small, author-created ground truth with no inter-annotator agreement, no released labels or code, and a selective AttacKG comparison. The engineering contribution is plausible, but the accuracy claims are not yet independently established.","major_comments":[{"comment":"All RQ1 accuracy claims (93.8% entity, 91.4% relationship accuracy on 10 log-based graphs; 86.4% entity accuracy on 11 CTI reports) are computed against ground truth that the authors themselves created, with no annotation schema, no annotator count, no inter-annotator agreement measure, and no release of the labeled graphs, the GPT-4 prompt, or the code. As presented, the numbers are unfalsifiable, and this is load-bearing because every positive result in RQ1 is measured against these labels. Please release the annotation instructions and gold graphs, add at least a second annotator with an agreement statistic, or otherwise provide an external audit of the labels.","section":"6.2.1, Tables 4-5"},{"comment":"The comparison with AttacKG is reported for only 6 of the 65 matched techniques, and the 6 are not shown to be representative of the full set. The abstract's claim that MultiKG's F1 is 'roughly twice as high' as AttacKG is based entirely on these 6 techniques (F1 0.959/0.958 vs 0.545/0.372 for nodes/edges). Without the full 65-technique comparison or a statistically justified sample, the headline cross-source claim is not demonstrated. Additionally, the ground truth for this table is a manual cross-source merge of the same graphs that MultiKG merges, creating a risk that the gold standard implicitly aligns with the method's behavior.","section":"6.2.3, Table 7"},{"comment":"The same-level merging rule—merging nodes that share the same BFS level and type/process name—is justified only by the assertion that 'similar nodes follow similar calling relationships, share the same calling logic and are more likely to appear at the same level of the calling chain.' No quantitative validation of this assumption is provided, yet it is load-bearing for the aggregation module and thus for the compression ratios and the cross-source F1 scores. Please add an analysis or ablation showing that same-level merging preserves gold-standard nodes and edges across the labeled techniques, or compare it against content-only merging.","section":"5.1, Algorithm 4"},{"comment":"The cross-source merging algorithm uses a content-similarity threshold in FindMatchingNode/ContentSimilarity, but the threshold value is never specified and no sensitivity analysis is reported. The merging decisions, and therefore the Table 7 results, depend critically on this threshold. Please report the exact threshold and show how node/edge F1 varies with it on the labeled set.","section":"5.2, Algorithm 5"},{"comment":"The effectiveness of the aggregation module is measured only through compression/retention rates. Compression alone does not demonstrate correctness: it could also reflect over-merging that discards discriminative detail. The paper does not check whether the aggregated graphs preserve the gold-standard nodes and edges from the labeled set. The table also contains internal inconsistencies: T1090.003 and T1615 have identical before/after CTI counts (59 entities, 52 edges before; 10/20 after) but report retention percentages of 83.051/61.538 and 16.949/38.462 respectively, and the text's average compression ratios of 47.41% and 44.45% do not match any aggregation of the table's values.","section":"6.2.2, Table 6"}],"minor_comments":[{"comment":"The GPT-4 parsing prompt is described as 'well-designed' but is never shown; without the prompt and model version, the CTI extraction component is not reproducible.","section":"4.3.2"},{"comment":"The statement that the system was 'evaluated using 1,015 real attack techniques and 9,006 attack intelligence entries' is misleading because the accuracy evaluation uses only 10 log executions and 11 CTI reports; the larger numbers are dataset statistics rather than evaluation instances.","section":"Abstract and 6.1"},{"comment":"The header 'Overall Precious' should read 'Overall Precision'.","section":"Table 4"},{"comment":"The column header 'Event Nmae' contains a typo and should read 'Event Name'.","section":"Table 2"},{"comment":"The paper repeatedly uses 'CIT report' instead of 'CTI report' (e.g., in Section 4.3 and Section 6.2.3); please correct these typos.","section":"Throughout"},{"comment":"The full cross-source comparison is said to show that AttacKG has 43.76% of the average number of nodes and 76.46% of the edges of MultiKG, but only a 15-technique sample is shown in Table 8; the complete results should be provided in a supplementary table.","section":"6.2.3"}],"recommendation":"major_revision","confidential_remarks":"The central concern from the stress-test is valid: the quantitative claims are supported by a small, author-created, non-released ground truth. The paper would be considerably strengthened by releasing the annotated graphs, the GPT-4 prompt, and the code, and by reporting inter-annotator agreement. The selective 6-of-65 AttacKG comparison is also not sufficient for the abstract's headline claim. That said, the system design is well-motivated and the algorithmic contributions are described concretely, so major revision rather than rejection is appropriate. The fit for cs.CR is fine, though the paper would benefit from clarifying that the 1,015/9,006 numbers are dataset sizes, not evaluation sizes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the thing to know: MultiKG is a credible systems paper that tries to do something genuinely new—build attack technique knowledge graphs from three sources (CTI reports, audit logs, static code) and merge them. The single-source ingredients are known (LLM parsing like AttacKG+, provenance graph mining, AST analysis), but the three-way combination is new relative to the cited baselines, and the architecture is sensible. The evaluation, however, rests on a small ground truth set created by the authors themselves, with no inter-annotator agreement and no released code, data, or prompts. That is a real soft spot, and it means the headline numbers (93.8% entity accuracy, 91.4% relationship accuracy, roughly 2x AttacKG F1) are not yet independently established. It is not a fatal flaw—the method is clear, and the authors disclose that the labels are manual—but it is a load-bearing one.\n\nWhat I like: the decomposition is clean. Logs give real execution topology; static code fills missing nodes; CTI provides variant coverage. The merging heuristics (same-level merging, content-similarity for leaves, cross-source BFS with content-similarity matching) are reasonable, and the compression statistics show the aggregation does something. The comparison to AttacKG on 6 techniques, while small, suggests an advantage that is plausible given the extra sources.\n\nSoft spots, in order: (1) Ground truth. Ten log executions and eleven CTI reports, all hand-labeled by the authors who know what the system should produce, with no inter-annotator agreement or annotation schema. For the cross-source merging, the ground truth is the authors' manual merge of the same graphs. This makes the accuracy numbers unfalsifiable as presented. (2) AttacKG comparison: only 6 of 65 matched techniques are shown in Table 7; the rest appears as aggregate node/edge counts. Need the full per-technique breakdown. (3) \"Fully automated\" overclaim—there is manual environment setup, and the case studies are illustrative, not a detection evaluation. (4) No release.\n\nI disagree with the harshest reading that the method is circular—the extraction itself has no fitted parameters tuned to the labels, so the construction claims are not circular. The issue is measurement, not fitting.\n\nBottom line: this is a paper for security researchers working on CTI and attack investigation. The idea deserves a serious referee. If I were the editor, I'd send it to review and push for release of code/prompts/labels, an independent annotation effort, and the full per-technique comparison. With those, the numbers could be credible.","headline":"Genuinely new three-source attack graph pipeline, but the accuracy claims need independent labels and a fuller AttacKG comparison before they can be trusted.","tokens_in":25713,"tokens_out":2254,"would_cite":true,"duration_ms":21937,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MultiKG builds a unified attack-technique knowledge graph from CTI reports, audit logs, and static code, claiming 93.8% log entity accuracy and roughly twice the cross-source F1 score of AttacKG.","keywords":["attack knowledge graph","threat intelligence aggregation","provenance graph","audit log analysis","static code analysis","large language model","MITRE ATT&CK","attack reconstruction"],"falsifier":"Ask two or more independent analysts who have not seen MultiKG's output to annotate a held-out set of attack executions and CTI reports using the same entity and relationship schema, measure their agreement, and recompute precision and recall against those labels; if inter-annotator agreement is low or the independently measured F1 scores fall well below the reported 93.8% and 86.4% figures, the accuracy claims are not reproducible.","tokens_in":24663,"feed_emoji":"🛡️","tokens_out":8813,"duration_ms":76706,"temperature":0.7,"pith_summary":"MultiKG is a fully automated pipeline that turns three kinds of threat data into one attack-technique knowledge graph: natural-language cyber threat intelligence (CTI) reports, system audit logs captured while an attack actually runs, and the attack's source code. The paper's central claim is that these sources complement one another — reports provide coverage of technique variants, logs provide the true execution topology, and static code analysis restores nodes and edges that log filtering drops — so merging them produces graphs that are more complete and more accurate than graphs built from reports alone. In the authors' evaluations, MultiKG reaches 93.8% entity and 91.4% relationship accuracy on log-based graphs, 86.4% entity accuracy on CTI graphs, and roughly twice AttacKG's F1 score on cross-source merging. If the claim holds, security teams could automatically maintain a fine-grained, technique-level knowledge base that supports attack reconstruction and detection.","feed_headline":"One graph unifies attack knowledge from logs, code, and reports","feed_subtitle":"MultiKG claims 93.8% log entity accuracy and double the F1 of prior graph builders when merging sources","key_machinery":"The central mechanism is a three-lane extraction pipeline with two merging stages. Lane one reconstructs a dynamic provenance subgraph from ETW audit logs by following the process chain from a known starting process ID, then filters event types and removes whitelisted non-attack nodes. Lane two parses the attack script's abstract syntax tree to recover file, registry, and process entity names, matching them against the audit log to supplement missing nodes and edges. Lane three uses a large language model (GPT-4) with a purpose-built prompt to extract entities, entity types, and relationships from the technique descriptions in MITRE ATT&CK. Same-source graphs are merged by hierarchical level and content similarity; cross-source merging treats the log-plus-code graph as the base graph and the CTI graph as the additional graph, traverses the additional graph in breadth-first order, and merges a node only when type and content similarity match, otherwise attaching the node to its parent. A final generalization step replaces user-specific paths with wildcards and merges similar leaf nodes, producing a compact technique graph that spans variants.","core_discovery":"The discovery is that attack knowledge from text, execution logs, and code can be converted separately into technique-level graphs and then merged into a single graph that is finer-grained than any single-source representation. MultiKG builds a 'base graph' from audit logs enriched by static code analysis, then folds in LLM-parsed CTI report graphs using breadth-first traversal and content-similarity matching; edges with no counterpart in the base graph are preserved by attaching them to their parent node. The paper reports that aggregation compresses log-based graph nodes by about 47% on average and CTI graph nodes by about 70%, while the cross-source merged graphs achieve node and edge F1 scores of 95.9% and 95.8% against manual ground truth — roughly twice the corresponding scores for AttacKG. The intended payoff is that downstream security tasks, including attack reconstruction, variant detection, and TTP-based detection, can consume these graphs directly.","pith_inferences":["Beyond the paper: because cross-source merging matches graphs by MITRE technique number, reports describing attacks at a coarser or finer granularity than ATT&CK would need additional alignment logic to be absorbed correctly.","Beyond the paper: the CTI lane's quality is tied to the particular LLM and prompt used, so replacing GPT-4 and re-running the 11-report evaluation would directly test how much of the reported accuracy comes from the model rather than the framework.","Beyond the paper: the base-graph/additional-graph merge is source-agnostic in principle, so the same design could be extended to other telemetry sources such as network flows or endpoint detection data, provided nodes share comparable type and content similarity."],"forward_implications":["Technique-level knowledge graphs can be built automatically from heterogeneous sources instead of relying on manual CTI summarization.","Attack reconstruction can operate at technique granularity, because the merged graph contains the real execution path from logs plus variant details from reports and code.","Detectors that match against TTP heuristics can draw on a richer and more generalized knowledge base, potentially improving coverage of attack variants.","Same-source aggregation compresses redundant nodes and edges by roughly 47% to 70%, keeping the final graphs compact enough for practical matching and review."],"supporting_citations":[{"why":"MITRE ATT&CK is the taxonomy and data source for the 282 techniques and 9,006 procedure descriptions that define the CTI lane's inputs.","marker":"[30]"},{"why":"Atomic Red Team provides the 1,015 executable attack procedures and their source code, which drive both the audit-log executions and the static-code analysis.","marker":"[3]"},{"why":"ETW is the Windows event-tracing mechanism the authors' collector uses to capture the audit logs from which dynamic graphs are built.","marker":"[9]"},{"why":"AttacKG is the technique-aware, graph-structure baseline whose node and edge F1 scores MultiKG is compared against in the cross-source merging evaluation.","marker":"[50]"},{"why":"Poirot is the prior CTI-to-audit-log alignment approach whose single-source limitation motivates the multi-source design.","marker":"[28]"},{"why":"ThreatRaptor is one of the earlier CTI graph construction systems that MultiKG extends by adding log and code sources.","marker":"[11]"},{"why":"Brown's few-shot learning result is cited as the reason an LLM can parse CTI reports without a large security-specific training set.","marker":"[6]"}],"fun_headline_variants":["MultiKG fuses logs, code, and reports into one attack graph","LLM-driven attack graph hits 95.9% F1 from three sources","Cross-source attack graph doubles F1 of prior builders","Aggregated attack knowledge graph: MultiKG tops AttacKG","From logs, code, and text: MultiKG builds unified attack graph"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy figures rest on ground truth that the authors themselves hand-labeled — ten attack technique executions for the log-based graphs and eleven CTI reports for the report-based graphs, with the cross-source ground truth obtained by manually merging the same graphs the algorithm merges — so if those labels are incomplete, mistaken, or biased toward what MultiKG produces, the reported precision and recall overstate real performance.","fun_headline_variants_meta":{"raw":{"variants":["MultiKG fuses logs, code, and reports into one attack graph","LLM-driven attack graph hits 95.9% F1 from three sources","Cross-source attack graph doubles F1 of prior builders","Aggregated attack knowledge graph: MultiKG tops AttacKG","From logs, code, and text: MultiKG builds unified attack graph"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2067,"prompt_tokens":967,"completion_tokens":1100,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1007}},"tokens_in":583,"tokens_out":1100,"duration_ms":9255,"temperature":1.0,"reasoning_tokens":1007,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:39:22.445326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask two or more independent analysts who have not seen MultiKG's output to annotate a held-out set of attack executions and CTI reports using the same entity and relationship schema, measure their agreement, and recompute precision and recall against those labels; if inter-annotator agreement is low or the independently measured F1 scores fall well below the reported 93.8% and 86.4% figures, the accuracy claims are not reproducible.","supporting_citations":[{"cited_title":"MITRE ATTCK","cited_arxiv_id":null,"evidence_quote":"MITRE ATT&CK is the taxonomy and data source for the 282 techniques and 9,006 procedure descriptions that define the CTI lane's inputs."},{"cited_title":"Atomic Red Team","cited_arxiv_id":null,"evidence_quote":"Atomic Red Team provides the 1,015 executable attack procedures and their source code, which drive both the audit-log executions and the static-code analysis."},{"cited_title":"Event Tracing for Windows (ETW)","cited_arxiv_id":null,"evidence_quote":"ETW is the Windows event-tracing mechanism the authors' collector uses to capture the audit logs from which dynamic graphs are built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AttacKG is the technique-aware, graph-structure baseline whose node and edge F1 scores MultiKG is compared against in the cross-source merging evaluation."},{"cited_title":"POIROT: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Hunting","cited_arxiv_id":"1910.00056","evidence_quote":"Poirot is the prior CTI-to-audit-log alignment approach whose single-source limitation motivates the multi-source design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ThreatRaptor is one of the earlier CTI graph construction systems that MultiKG extends by adding log and code sources."}],"review_version":1}