{"id":"2ab64014-82a8-4540-ab75-f09d462bb022","arxiv_id":"2608.13050","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"In an AI-judged comparison on nine cyber threat reports, GraphRAG produced detection plans that appeared more resilient to rotating attacker indicators than standard vector search, but the judge's scores broke the paper's own scoring rules.","lead":"This paper compared two AI retrieval systems for turning cyberattack reports into detection plans: Microsoft GraphRAG's knowledge-graph search and standard text similarity search. It reports that GraphRAG plans keep working after the obvious attack clues are changed, though the scoring AI's own rules contradict part of that result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline durability metric is internally inconsistent: Table 6 scores Plan C detection_durability 8/10 while Table 7 reports 29% survival, which the paper's own rubric caps at 3. The counting procedure behind Table 7 is undocumented and no detection query is executed.","rationale":"The reader's weakest assumption identifies the unvalidated LLM judge and the absence of real telemetry execution; I agree that this is a serious problem. My stress test sharpens it into a concrete internal inconsistency: Table 6, Table 7, and the Section 5.3 rubric cannot all be correct. Plan C's 29% survival triggers a hard ceiling of ≤3 on detection_durability and ioc_resistance, yet Table 6 reports 8 and 9 respectively. This is not a matter of external consensus or subjective scoring; the paper violates its own published evaluation protocol on the exact metric that carries the headline claim. That makes the reported 100% versus 29% durability gap unverifiable even from the manuscript's own evidence, independent of any skepticism about LLM-as-judge. I do not challenge the authors' intent or the plausibility of the architectural hypothesis—GraphRAG retrieving relationship neighborhoods could plausibly yield more behavioral detections—but the current paper does not provide a trustworthy measurement of that hypothesis. The concrete test would settle whether the contradiction is a reporting error or a substantive flaw in the counting procedure, and it would also establish whether the durability numbers survive actual query execution rather than prose inspection.","tokens_in":18082,"tokens_out":2609,"duration_ms":28077,"concrete_test":"Take the APT28 V2 Plan C (Naive RAG) output and independently enumerate every detection in Sections 2 and 3 of the plan. Substitute rotated values for all campaign IPs, domains, and file hashes, then execute each Splunk SPL or Sentinel KQL query against a small simulated telemetry corpus containing both original and rotated indicators, with two security analysts independently labeling each query as firing or not firing after rotation. Compare the resulting survival fraction with Table 7's 4/14. If the true survival is below 30%, the rubric's own ceiling requires detection_durability ≤ 3 and ioc_resistance ≤ 3; any higher score means the evaluation pipeline does not implement its stated rubric. Also re-run the judge on the same plan under the six-step counting procedure and check whether the emitted per-criterion scores violate the floors and ceilings.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim (RQ2, Section 6.4) is that after full rotation of campaign IPs, domains, and hashes, GraphRAG plans retain 100% of detections while Naive RAG retains 29%. This claim rests entirely on Table 7's survival counts, but the paper never states who counted those detections, under what rule 'would still fire' was decided, or how a detection in the GraphRAG Global empty-fallback plan (a 75-character error string that still receives rubric floors) could be counted as surviving. More damaging, the supporting scores contradict the rubric the paper defines in Section 5.3. Table 6 gives Plan C (Naive RAG) detection_durability = 8/10, while Table 7 reports 29% survival and Section 5.3 sets a hard ceiling of '≤3 if <30% survive.' The same contradiction affects ioc_resistance: Table 6 gives Plan C 9/10, while Section 5.3 caps ioc_resistance at ≤3 when fewer than 30% survive rotation. These are not calibration quibbles; they indicate that the judge transcript, the rubric, and Table 7 were not produced under the same counting procedure. If Plan C's Tier 2 scores are recomputed under the stated ceilings, Plan C loses at least 5 points on detection_durability and at least 6 points on ioc_resistance, changing the grand totals and undermining the claim that the breadth experiment is 'approximately tied' and that Tier 2 cleanly discriminates between pipelines. Additionally, the word 'firing' in the abstract and Section 6.4 implies execution against telemetry, but no query is run against logs or simulated data; a detection 'survives rotation' only because an LLM judge or the authors say so. The durability gap may be real, but the instrument used to measure it is demonstrably inconsistent with its own thresholds, so the headline 100% versus 29% is not an established fact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares Microsoft GraphRAG (Local and Global Search) with Naive vector RAG for converting CTI reports into threat-hunting plans, using a locally hosted generation model and an LLM-as-Judge with a ten-criterion rubric. The central claim is that GraphRAG produces plans that are more durable under IOC rotation: a deep-dive on an APT28 report reports 100% survival of GraphRAG detections versus 29% for Naive RAG, and a nine-report breadth experiment is said to confirm a consistent Pyramid-of-Pain advantage for GraphRAG even where grand totals are close. The paper also reports a large V1-to-V2 prompt effect and characterizes two pipeline failure modes.","tokens_in":18399,"tokens_out":9271,"duration_ms":96314,"significance":"If the durability result were sound, it would be a useful contribution to CTI operationalization: much prior work measures extraction accuracy or rule validity rather than resilience to IOC rotation. The paper is also commendably explicit about its failure modes and about the strong effect of the generation prompt. However, the central measurement is not currently credible: the rubric defined in the paper contradicts the reported scores, and the survival counts are undocumented and are not validated against telemetry. As written, the headline 100%-versus-29% gap is as likely to be an artifact of the evaluation instrument as a property of the retrieval architecture.","major_comments":[{"comment":"The rubric in §5.3 states that detection_durability and ioc_resistance are capped at ≤3 when fewer than 30% of detections survive full IOC rotation. Table 7 reports that Plan C (Naive RAG) has 4/14 surviving detections, i.e., 29%, yet Table 6 awards Plan C 8/10 on detection_durability and 9/10 on ioc_resistance. Applying the stated caps lowers Plan C's Tier 2 score from 31 to at most 20 and its grand total from 78 to at most 67, which changes the reported 'close' deep-dive comparison. This is a load-bearing internal inconsistency in the headline evidence, and the manuscript cannot be correct as written.","section":"§5.3, Table 6, Table 7"},{"comment":"The survival counts that drive RQ2 are not accompanied by any documented counting procedure. The text does not state who classified each detection as surviving, what defines a detection for enumeration, how 'would still fire' was determined under complete IOC rotation, or whether the six-step judge procedure described in §5.3 produced Table 7. No detection query is executed against real or simulated logs, so the word 'firing' in the abstract and §6.4 overstates what was measured. An LLM's opinion that a detection 'would still fire' needs validation against at least a documented adversarial simulation or human expert counting before it can support the paper's central claim.","section":"§6.4, Table 7, §5.3"},{"comment":"The V2 generation prompt (Appendix A, rules 3 and 5) requires every detection to carry a self-assigned Pyramid tag and instructs the model that L1–L3 detections expire while L4–L7 survive, and it forces Section 2 to be labelled 'Primary' and Section 3 to be labelled 'Fragile'. The survival measurement is therefore entangled with the prompt contract: the judge may simply be reading durability labels that the generation model was required to write, rather than independently assessing whether the detection logic survives rotation. No ablation or human validation is reported that separates retrieval-architecture effects from prompt-mandated self-labeling.","section":"§A, §6.2, §6.4"},{"comment":"The rubric's floors, ceilings, and the 30%/60% thresholds were calibrated on pilot runs of the same APT28 advisory that is then used as the headline deep-dive (§5.3, §6.1). This makes the evaluation instrument dependent on the test instance. A conclusion based on a rubric fitted to that instance needs a holdout or sensitivity analysis before the reported 100%-versus-29% number can be interpreted as evidence about the retrieval architecture.","section":"§5.3, §6.1"},{"comment":"The breadth experiment's win counts are computed from best-of-runs maxima over two runs and over Plans A and B for GraphRAG, while many cells are 16/100 silent-failure floors or 0/100 judge parsing failures. As reported, the 'approximately tied' grand-total comparison is not a comparison between two fixed pipelines, and the claimed Tier 2 advantage for GraphRAG on the breadth set is not established with this aggregation. Per-run and per-plan scores with failure-aware aggregation and sensitivity analysis should be reported.","section":"§5.4, Table 8"}],"minor_comments":[{"comment":"The text says the corpus is drawn from four vendors (CrowdStrike, Cyble, EclecticIQ, Mandiant) plus one open-source advisory, but Table 3 lists only reports from CrowdStrike, Cyble, and EclecticIQ; the text and table should be reconciled.","section":"§5.1, Table 3"},{"comment":"The term 'firing' should be replaced with language that matches the actual measurement, such as 'judged as likely to fire', unless detection queries are actually executed against telemetry.","section":"§6.4, Abstract"},{"comment":"The limitations section acknowledges the single-judge-run issue, but the paper would benefit from reporting multiple judge runs or at least a variance estimate, since the deep-dive and breadth conclusions rest entirely on this one judge's scores.","section":"§7.3"}],"recommendation":"reject","confidential_remarks":"The research question is well motivated and the authors are unusually candid about failure modes, but the central empirical evidence is internally inconsistent: the rubric's own ceilings contradict the reported scores in Table 6 versus Table 7, and the survival counting procedure is not documented. Re-running the evaluation with a consistent rubric and a validated survival protocol is necessary before the headline claim can be published; this is beyond a local revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time, but the headline claim doesn't hold as presented.\n\nThe new thing: this is the first evaluation I've seen that scores generated hunting plans by Pyramid-of-Pain durability under IOC rotation, and the setup is unusually tidy — same generation model, same prompt, same judge, only retrieval changes. The local pipeline is reproducible, the failure modes (Global silent fallback, judge JSON parsing) are documented honestly, and the V1→V2 prompt-ablation result is a real finding: a harder prompt moved all three back-ends by 55–74 points, dwarfing the retrieval effect. That alone is useful for anyone building CTI-to-detection automation. The literature review is broad and accurate, with close precedents cited rather than buried.\n\nThe soft spots are load-bearing. The stress-test confirmed the internal inconsistency: Section 5.3 caps detection_durability and ioc_resistance at ≤3 when fewer than 30% of detections survive rotation, yet Table 6 gives Plan C 8 and 9 on those criteria while Table 7 reports 29% survival. Those two criteria alone would cost Plan C 11 points under the stated ceilings, changing the grand-total picture. The 'firing' language in Table 7 and the abstract implies execution, but no query is run against logs or simulated telemetry; survival is an undocumented LLM-judge count. The rubric was also calibrated on pilot runs of the same APT28 advisory used for the deep-dive, which adds circularity to the comparison. The breadth experiment is approximately tied on grand totals, and the discriminating story relies on a secondary metric plus best-of-runs 'GraphRAG family' scoring — defensible choices, but they don't rescue the primary evidence.\n\nThe citation pattern looks clean, and the paper is clearly the product of careful thinking, not a sloppy drive-by. But a central empirical claim that contradicts its own rubric is not something I would treat as established. The right move is to send it to peer review anyway: the pipeline, the metric, and the prompt-sensitivity finding deserve referees. The authors should be pushed hard to pre-register a primary metric, validate the judge against human experts, run detections against at least simulated telemetry, document the survival-counting procedure, and fix the Table 6/Table 7 contradiction. If those changes happen, this becomes a solid contribution.\n\nFor you: read Section 5.3, 6.2–6.5, and Appendix A. That's where the real substance and the real problems both live.","headline":"A seriously built pipeline and a genuinely new durability metric, but the headline 100%-vs-29% result is not established because the judge scores contradict the paper's own rubric ceilings.","tokens_in":19063,"tokens_out":1626,"would_cite":false,"duration_ms":19561,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding a cyberattack report into a knowledge-graph retriever instead of a vector-similarity retriever produces hunting plans whose detections survive adversary infrastructure rotation, with 100% survival versus 29% on the deep-dive report.","keywords":["Cyber Threat Intelligence","Retrieval-Augmented Generation","GraphRAG","Knowledge Graphs","Pyramid of Pain","LLM-as-Judge","Threat Hunting","IOC rotation"],"falsifier":"Run the deep-dive APT28 plans against simulated or recorded endpoint and network telemetry with every IP, domain, and file hash rotated, and count how many detections actually fire; if the Naive RAG plan fires on a comparable share of its detections, or the GraphRAG plan loses significant detections, the reported 100% versus 29% gap would not be a property of the retrieval back-ends.","tokens_in":17785,"feed_emoji":"🛡️","tokens_out":8653,"duration_ms":74754,"temperature":0.7,"pith_summary":"This paper asks whether the way a detection-generation system retrieves evidence from a cyber threat report determines how long the resulting detections stay useful. It compares two retrieval back-ends held to the same generation model and the same generation prompt: vector-similarity search over text chunks (Naive RAG) versus retrieval from a knowledge graph of the report's actors, malware, infrastructure, and their relationships (GraphRAG). On one APT28 advisory, every detection in the GraphRAG plan still fired after all IPs, domains, and file hashes in the report were rotated, while the Naive RAG plan retained only 29% of its detections. Across nine real reports, the two systems landed close on total quality scores, but the GraphRAG plans consistently sat at higher, harder-to-evade levels of the Pyramid of Pain. A sympathetic reader should care because automated detection plans built on the bottom pyramid levels expire within hours of report publication, which is precisely the gap this comparison targets.","feed_headline":"GraphRAG keeps 100% of detections alive when IOCs rotate","feed_subtitle":"Vector retrieval loses 71% of its detections to the same rotation; graph retrieval reaches durable behaviour.","key_machinery":"The load-bearing mechanism is GraphRAG Local Search over a report-derived knowledge graph: at indexing time entities (actor, malware, technique, vulnerability, domain, ip, file, sector) and their relationships are extracted and clustered into communities, and at query time the seed entities for a hunting question pull in their relationship neighbourhood, the text units describing those relationships, and the community summary that names a behavioural chain. That multi-entity neighbourhood is what lets the generator verbalise a TTP-level detection such as Outlook spawning mshta.exe then powershell.exe followed by a beacon, instead of latching onto the loudest single indicators. Naive RAG, by contrast, retrieves the top-8 cosine-similar text chunks, which for network-first CTI prose are usually the chunks naming network observables, anchoring the plan at L4 with an L1–L3 tail. The Pyramid of Pain framework supplies the evaluation axis: seven levels ordered by the adversary's cost to evade, from hashes and IPs up to TTPs.","core_discovery":"The central claim is that retrieval architecture, not the language model or the prompt, determines the Pyramid-of-Pain resilience of an automatically generated threat hunting plan. Holding the generation model and the hardened generation prompt fixed and varying only the retrieval step, the paper reports that GraphRAG Local Search produces plans whose detections cluster at L5–L7 (host artefacts, tool fingerprints, TTPs) and survive a complete rotation of campaign IOCs at 100%, while Naive RAG produces plans that cluster at L4 with a long L1–L3 fragile tail and retain only 29% of detections under the same rotation. The paper also finds that the wording of the generation prompt moves total quality far more than the retrieval back-end does, but that only graph retrieval lifts the plan into the durable top of the pyramid. The intended upshot is that knowledge-graph-aware retrieval is the architecturally correct foundation for turning vendor reports into SOC-deployable hunting plans.","pith_inferences":["Extending beyond the paper: if the survival gap reproduces under real telemetry, graph-retrieval benefits may appear for other security text-to-artefact tasks, such as YARA or Sigma rule generation, where the durable signal is relational rather than lexical.","Extending beyond the paper: rotating tool names, registry paths, and network artefacts in addition to IOCs would test whether L5–L7 detections truly persist; the paper's mechanism predicts GraphRAG plans should survive a broader rotation while Naive RAG plans degrade further.","Extending beyond the paper: holding the prompt fixed and varying only retrieval neighbourhood size would isolate whether detection durability tracks the number of retrieved relationships, separating graph structure from retrieval coverage.","Extending beyond the paper: because the prompt change moved total scores far more than the retrieval change, RAG comparisons in security that do not control prompt strictness may misattribute quality gains to the retrieval architecture."],"forward_implications":["Detection plans generated through graph-based retrieval should keep firing after public indicators go stale, extending operational life from hours to weeks or longer.","SOCs deploying automated CTI-to-detection pipelines should make graph-based Local Search the primary retrieval back-end and keep Global Search behind an output-length fallback, since sparse reports silently degrade it.","Because the hardened prompt moved every plan by tens of points, the contract imposed on the generator matters at least as much as retrieval; comparing retrieval architectures without such a contract will understate or mask architectural differences.","The evidence suggests the GraphRAG-versus-Naive gap is not about total plan quality but about composition: near-equal grand totals hide a systematic difference in which detections remain after rotation, so SOCs should evaluate on durability, not aggregate scores."],"supporting_citations":[{"why":"Supplies the GraphRAG indexing and retrieval implementation whose Local and Global search modes are the treatment condition.","marker":"[15]"},{"why":"Defines the Pyramid of Pain seven-level framework that structures the rubric and the IOC-rotation survival analysis.","marker":"[6]"},{"why":"Provides the APT28 LayeredMesh advisory used for the deep-dive case study and the 100% versus 29% rotation survival numbers.","marker":"[14]"},{"why":"Defines Retrieval-Augmented Generation, the baseline architecture (Naive RAG) that the paper compares against.","marker":"[27]"},{"why":"Gives prior evidence that graph-based retrieval beats vector RAG on multi-hop CTI questions, the closest architectural precedent.","marker":"[18]"},{"why":"Supplies the local judge model that scores plans against the ten-criterion rubric.","marker":"[16]"}],"fun_headline_variants":["GraphRAG survives IOC rotation, Naive RAG falls to 29%","Why GraphRAG keeps threats alive when IOCs rotate","Knowledge graph retrieval reaches top of Pyramid of Pain","For durable hunting plans, GraphRAG beats vector search","GraphRAG: 100% detection retention after IOC rotation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire survival comparison rests on trusting the language-model judge's counts of which detections would still fire after IOC rotation, because no detection query was ever run against real or simulated telemetry and the counting procedure's reliability is not established.","fun_headline_variants_meta":{"raw":{"variants":["GraphRAG survives IOC rotation, Naive RAG falls to 29%","Why GraphRAG keeps threats alive when IOCs rotate","Knowledge graph retrieval reaches top of Pyramid of Pain","For durable hunting plans, GraphRAG beats vector search","GraphRAG: 100% detection retention after IOC rotation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1399,"prompt_tokens":1043,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":659,"tokens_out":356,"duration_ms":3487,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:50:02.414628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the deep-dive APT28 plans against simulated or recorded endpoint and network telemetry with every IP, domain, and file hash rotated, and count how many detections actually fire; if the Naive RAG plan fires on a comparable share of its detections, or the GraphRAG plan loses significant detections, the reported 100% versus 29% gap would not be a property of the retrieval back-ends.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Pyramid of Pain seven-level framework that structures the rubric and the IOC-rotation survival analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the APT28 LayeredMesh advisory used for the deep-dive case study and the 100% versus 29% rotation survival numbers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Retrieval-Augmented Generation, the baseline architecture (Naive RAG) that the paper compares against."},{"cited_title":"Beyond RAG for Cyber Threat Intelligence: A Systematic Evaluation of Graph-Based and Agentic Retrieval","cited_arxiv_id":"2604.11419","evidence_quote":"Gives prior evidence that graph-based retrieval beats vector RAG on multi-hop CTI questions, the closest architectural precedent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the local judge model that scores plans against the ten-criterion rubric."}],"review_version":1}