{"id":"f664990f-9e91-4991-9897-494dbbcd64c1","arxiv_id":"2510.11974","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A 691-question benchmark (CTIARENA) shows LLMs need retrieval over heterogeneous cyber-threat-intelligence sources and that domain-specific retrieval beats generic RAG; the attached abstract describes a different 1,860-question benchmark (CTIConnect).","lead":"This preprint introduces a benchmark for testing AI assistants on cyber-threat-intelligence questions drawn from structured vulnerability databases and security blogs. The body reports that retrieval-augmented models beat closed-book models and that security-specific retrieval beats generic search, but the attached abstract describes a different, larger benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract and full text describe different benchmarks, leaving the paper's central claims unsupported.","rationale":"The reader's verdict of REJECT is well-aligned with the most load-bearing concern: the abstract and full text describe two different benchmarks, so the paper's central claim as stated cannot be verified. The reader's stated weakest_assumption focuses on ground-truth label correctness, which is important but secondary; the deeper issue is that the paper does not even present the artifact (CTIConnect) that the abstract advertises. I share the reader's conclusion that the paper should be rejected as submitted, but I identify the abstract/body mismatch as the primary load-bearing concern rather than label correctness. A simple textual check will settle whether this is a genuine incoherence or a metadata artifact; if the mismatch is confirmed, no empirical result in the body can rescue the abstract's claims. My recommendation is therefore UNCHANGED, since it preserves the reader's REJECT verdict.","tokens_in":26363,"tokens_out":2193,"duration_ms":20379,"concrete_test":"Perform a literal string search of the full text (excluding the abstract and references) for the identifiers 'CTIConnect', '1,860', 'temporal split', 'retrieve-then-rerank', and 'IRCoT'. If none of these terms appear in the body, then the abstract's central claims have no supporting derivation. Independently, attempt to download the dataset from the provided GitHub URL at a pinned commit; if the repository contains no dataset snapshot or commit hash, re-evaluation of the benchmark is impossible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper as submitted is internally incoherent: the arXiv abstract defines CTIConnect with 1,860 expert-verified QA pairs, temporal splits spanning 2008–2025, and comparisons against retrieve-then-rerank and IRCoT. The full text, however, defines CTIARENA with 691 QA pairs and, in §4, evaluates only closed-book, vanilla RAG, CSKG-guided RAG, and query-expanded RAG. The abstract's central claims—that CTIConnect is the first heterogeneous multi-source CTI benchmark and that domain-specific retrieval outperforms retrieve-then-rerank and IRCoT—have no supporting derivation anywhere in the body. No table or section reports 1,860 QA pairs, temporal splits, or IRCoT/retrieve-then-rerank results. This is not a disagreement with consensus; it is an internal inconsistency that prevents verification of the stated central claim. Even if the full text's CTIARENA results are internally sound, the paper presents two different artifacts, and the abstract's empirical findings cannot be checked. The GitHub link provides no commit hash or data snapshot, further blocking re-evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, as submitted under the CTIConnect title and abstract, actually contains a full text that presents a different benchmark, CTIARENA. The body defines 691 expert-verified QA pairs across nine tasks grouped into structured, unstructured, and hybrid categories, constructed via a three-stage pipeline (seed correlations, template-guided LLM synthesis, LLM-human curation), and evaluates ten LLMs under closed-book, vanilla RAG, CSKG-guided RAG, and query-expanded RAG. The reported findings are that closed-book performance is low on structured mappings, retrieval augmentation largely closes the structured gap, and the two CTI-specific retrieval variants improve hybrid/unstructured tasks relative to vanilla RAG, with some caveats for weaker models. The abstract, however, claims 1,860 QA pairs, temporal splits spanning 2008-2025, and comparisons against retrieve-then-rerank and IRCoT, none of which appear in the body.","tokens_in":26589,"tokens_out":11161,"duration_ms":92202,"significance":"If the CTIARENA results are reproducible, the benchmark is a useful step: no existing CTI benchmark combines nine tasks over structured, unstructured, and hybrid sources in a retrieval-augmented setup; the evaluation spans ten models; and the proposed CSKG-guided and query-expanded RAG are concrete, domain-tailored retrieval strategies. The paper also documents a human-verification pipeline and makes a credible effort to ground QA pairs in authoritative correlations. However, the current version is internally incoherent: the front matter advertises a different benchmark with unverifiable claims, the dataset is not released, inter-annotator agreement is not reported, and the 'consistently outperforms' statements are not supported by the tables. These issues must be resolved before the contribution can be fully assessed.","major_comments":[{"comment":"The arXiv abstract defines CTIConnect with 1,860 expert-verified QA pairs, temporal splits 2008-2025, and comparisons against retrieve-then-rerank and IRCoT. The full text defines CTIARENA with 691 QA pairs and, in §4, evaluates only closed-book, vanilla RAG, CSKG-guided RAG, and query-expanded RAG. No section or table reports 1,860 pairs, temporal splits, or the two generic baselines. This is load-bearing: the paper's stated central claim cannot be checked. Please either substantiate the CTIConnect claims with data and experiments, or retitle/rewrite the front matter to match the CTIARENA content actually presented.","section":"Abstract vs. full text"},{"comment":"The text claims CSKG-guided RAG 'consistently outperforms' vanilla RAG on unstructured tasks and query-expanded RAG 'consistently outperforms' vanilla RAG on hybrid tasks. Tables III and IV contradict this: e.g., GPT-5 CSC vanilla 0.721 > CSKG 0.671; Gemini-2.5-Pro CSC vanilla 0.612 > CSKG 0.609; Claude-3.5-Haiku MLA vanilla 0.443 > CSKG 0.410; Phi-4 VCA vanilla 0.480 > expansion 0.360; Gemini-2.5-Flash VCA vanilla 0.600 > expansion 0.540. Domain-specific gains are present in many cells but are not consistent. Report averages, per-task significance tests, or error bars, and soften the claim accordingly.","section":"§4.2, Tables III–IV"},{"comment":"The evaluation protocol says structured and hybrid tasks use exact string matching and report accuracy, but Table II's header labels all numbers as F1-score. Please clarify the metric. Additionally, §4.2(1) says the best closed-book ATD score is 0.99, but Table II shows no closed-book ATD value near 0.99; the largest is about 0.06-0.09. This appears to be a typo that inverts CB and KW values, making the quantitative narrative difficult to interpret.","section":"§4.1 vs. Table II"},{"comment":"The query-expanded RAG evaluation prompt instructs the model to 'answer based on the provided ATT&CK / CWE evidence, or use your internal knowledge if you're confident' and 'You don't have to select from the provided CWEs if you have a better answer.' This allows closed-book parametric knowledge to substitute for retrieval, so improvements attributed to query expansion may instead reflect prior knowledge. The main-text description of query-expanded RAG should be reconciled with this prompt, and the setup should be reported as a fallback/hybrid RAG rather than pure retrieval augmentation.","section":"§D.3, RAG Expansion prompt; §4.1"},{"comment":"The benchmark's validity rests on the seed correlations, BlogCluster/B2F annotations, and the Stage 3 LLM-human curation. No inter-annotator agreement or post-hoc error audit is reported, and the GitHub link has no commit hash or data snapshot. In addition, GPT-5 is used both as an evaluated model and as the automatic judge for unstructured answers (§4.1), and GPT-5 is also used in data generation/filtering (§3.2). This creates a risk of circularity for the open-ended-task results. Please release the benchmark data with versioned code, report annotator agreement and remaining label-error rate, and consider using a judge model that is not among the evaluated systems.","section":"§3.2 Stages 1–3; §4.1"}],"minor_comments":[{"comment":"The dataset sizes for unstructured tasks (60/60/30) are very small, and no confidence intervals are reported for the GPT-5 judge scores. Please state variance or bootstrap intervals.","section":"§3.3.2"},{"comment":"The retrieval hyperparameters (top-k=5, similarity threshold 0.5, CSKG entity-overlap ≥0.6) are said to be chosen after exploratory trials; a sensitivity analysis would strengthen the benchmark.","section":"§4.1"},{"comment":"The unstructured-task columns show only KW values, with no CB columns. Since the text says closed-book is not evaluated for unstructured tasks, this should be stated explicitly in the table caption.","section":"Table II"},{"comment":"There are several small inconsistencies, e.g., 'compaign' in §3.3.2 and the mixed use of 'F1' vs. 'accuracy' terminology. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The abstract/full-text mismatch is severe and should be the first thing the editor asks the authors to fix: either the CTIConnect claims must be substantiated or the paper must be retitled and restated around CTIARENA. The remaining issues (metric inconsistency, overclaimed consistency, fallback-to-internal-knowledge prompt, lack of data/agreement statistics) are fixable but would require a substantial revision. I do not see evidence of misconduct, but the current version cannot be accepted as-is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere’s the one-sentence take: the full text of arXiv:2510.11974 is a genuinely useful, well-structured CTI RAG benchmark — but the abstract on the arXiv page describes a different artifact, and that mismatch is severe enough that the paper as submitted can’t be verified.\n\nWhat’s actually new and good: CTIARENA is the first multi-source, heterogeneous CTI benchmark for retrieval-augmented LLMs that I know of, covering nine tasks across structured (CVE/CWE/CAPEC/ATT&CK mappings), unstructured (campaign/actor/malware synthesis), and hybrid (narrative-to-taxonomy attribution) categories. The three-stage construction pipeline (seed correlation annotation, template-constrained QA synthesis, LLM-judge plus human cross-verification) is described in unusual detail, with prompt templates in the appendix. The experiments are internally consistent: closed-book LLMs are terrible at structured mappings, knowledge injection saturates those tasks, and domain-specific retrieval (CSKG-guided RAG, query-expanded RAG) beats vanilla RAG on hybrid and unstructured tasks. The error analysis is concrete and credible — semantic drift, retrieval-but-not-used, and unsupported correctness are real failure modes, and quantifying them at 8%, 15%, and 3% is the kind of detail that makes this paper worth reading. The related-work positioning against CTIBench and SEvenLLM is fair and accurate.\n\nThe soft spots, in order of severity:\n\n1. The abstract/body mismatch. The arXiv metadata claims CTIConnect with 1,860 QA pairs, temporal splits 2008–2025, and comparisons against retrieve-then-rerank and IRCoT. The body defines CTIARENA with 691 QA pairs and evaluates only closed-book, vanilla RAG, CSKG-guided RAG, and query-expanded RAG. No table reports 1,860 pairs, no temporal split appears anywhere, and IRCoT/rerank results are absent. This is not a minor framing issue — the abstract’s central empirical claims have no supporting derivation in the body.\n\n2. Verification is blocked. No dataset release, no commit hash, no data snapshot. The GitHub link alone isn’t enough to check the 691 pairs or the human verification claims.\n\n3. Smaller issues. Per-task sample sizes are small (e.g., 30 for MLA, 50 for ATA) with no error bars. GPT-5 is used to generate, filter, judge open-ended answers, and run the proposed query-expansion method — that’s a circularity concern, not a fatal one, but it deserves a sensitivity analysis.\n\nOverall: the body’s benchmark and experiments are real work and the paper shows clear thinking. The abstract/body inconsistency is load-bearing and must be fixed before the results can be trusted. If the authors reconcile the two artifacts — or explicitly drop the unsupported claims — this is a solid contribution to the CTI-LLM community.\n\nRecommendation: send it to peer review, but with the explicit instruction that the abstract must be aligned with the body and the data released. The underlying work is worth referee time.\n\nBest,\n[You]","headline":"The body of the paper is a solid, detailed CTI RAG benchmark (CTIARENA, 691 QA pairs), but the arXiv abstract describes a different benchmark (CTIConnect, 1,860 pairs, temporal splits, rerank/IRCoT baselines) that the body never delivers; that mismatch blocks verification of the headline claims.","tokens_in":27107,"tokens_out":846,"would_cite":false,"duration_ms":10686,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM cyber-threat analysis is gated by knowledge access, not reasoning: CTIARENA shows closed-book CTI mappings score near zero, jump to near-perfect once authoritative sources are injected, and CTI-tailored retrieval beats generic RAG.","keywords":["cyber threat intelligence","LLM benchmark","retrieval-augmented generation","knowledge-augmented LLM","CVE-CWE-CAPEC-ATT&CK","threat report synthesis","entity-centric retrieval","multi-source reasoning"],"falsifier":"Independently re-annotate a random sample of the 691 QA pairs with fresh experts who never saw the original labels and check agreement: if it is too low to sustain the headline margins (roughly 0.08 to 0.26 points for query-expanded RAG on hybrid tasks and CSKG-guided RAG on unstructured tasks), the rankings could be label noise. A second check: in structured tasks, withhold the gold entry from the injected candidate set; near-perfect accuracy should collapse if grounding, not parametric memory, is what closes the gap.","tokens_in":26218,"feed_emoji":"🛡️","tokens_out":16546,"duration_ms":117431,"temperature":0.7,"pith_summary":"This paper builds CTIARENA, which it presents as the first benchmark for evaluating LLMs on heterogeneous, multi-source cyber threat intelligence under knowledge-augmented settings: 691 expert-verified QA pairs spanning nine tasks that map CVE, CWE, CAPEC, and ATT&CK taxonomies, synthesize vendor reports and blogs, and bridge the two. Its central claim is that the cross-source semantic gap manifests differently across task categories, so no single retrieval recipe serves all of CTI. Closed-book models are nearly useless on structured mappings, yet the same models saturate to near-perfect accuracy once authoritative entries are injected, showing structured CTI reasoning is a grounding problem rather than a reasoning problem. Hybrid tasks demand query expansion that decomposes narratives into atomic security behaviors, while unstructured tasks demand entity-centric retrieval over a security knowledge graph instead of embedding similarity; the bottleneck shifts between retrieval infrastructure and evidence utilization depending on the task. If this holds, CTI copilots should be engineered around domain-tailored retrieval layers before model scale.","feed_headline":"Retrieval, not scale, unlocks LLM threat analysis","feed_subtitle":"A nine-task benchmark shows closed-book LLMs near zero on CTI mappings, near perfect once given authoritative sources.","key_machinery":"Two mechanisms carry the argument. First, the three-stage construction pipeline: seed correlations from authoritative sources (CVE-CWE-CAPEC-ATT&CK mappings, adversary-centered blog clusters, blog-to-framework phrase alignments) become template-constrained QA pairs, then pass through an LLM judge and human expert verification. Second, two CTI-specific retrieval strategies: query-expanded RAG, which decomposes a narrative query into atomic security behaviors and retrieves per behavior before aggregating; and CSKG-guided RAG, which links reports through shared entities in a cyber security knowledge graph, keeping reports with at least 0.6 entity overlap. Against vanilla embedding retrieval, bo","core_discovery":"CTIARENA's contribution is a benchmark and a diagnosis. The benchmark folds five CTI sources into 691 expert-verified QA pairs across nine tasks: structured mappings, unstructured report synthesis, and hybrid attribution of free-text behaviors to CWE and ATT&CK entries. The diagnosis: LLMs fail closed-book because cross-source correlations are long-tail enumerated knowledge that pretraining rarely absorbs; once authoritative entries are injected, structured accuracy saturates near 1.0 for all ten models. Query-expanded RAG beats vanilla semantic RAG on hybrid tasks; entity-centric retrieval over a security knowledge graph does the same on unstructured tasks. Residual errors trace to semantic","pith_inferences":["Editorial inference: the near-saturation of structured tasks implies the benchmark measures the retriever-plus-corpus at least as much as the model on those tasks; reporting structured accuracy as a function of retrieved-candidate coverage would let teams tune corpora rather than prompts.","Editorial inference: the unsupported-correctness finding could be converted into an automatic trust audit, requiring models to cite the exact evidence span behind each answer; the paper motivates this step but does not build it.","Editorial inference: the three-category design could be carried to other fragmented knowledge domains where authoritative enumerations and narrative sources coexist, such as medical or legal reference stacks; the claim that the bottleneck shifts by task category is testable there.","Editorial note on the manuscript: the front-matter abstract describes the benchmark as CTIConnect with 1,860 QA pairs and reports comparisons against retrieve-then-rerank and IRCoT baselines, while the body describes CTIARENA with 691 QA pairs and only vanilla-RAG baselines; this page follows the body."],"forward_implications":["Structured CTI reasoning should be treated as a retrieval and grounding problem: authoritative entry injection lifts all ten models to near-perfect accuracy, so further gains from larger closed-book models will be marginal.","Hybrid retrieval should decompose narrative queries into atomic CTI behaviors before matching taxonomy entries; this consistently beat vanilla semantic RAG, most sharply for the strongest models (GPT-5 ATA 0.742 to 0.900, VCA 0.600 to 0.760).","Unstructured tasks need entity-centric retrieval over a security knowledge graph, yet even with the right evidence, cross-report synthesis stays the hard part: actor profiling and malware lineage scores remain low, so evidence utilization, not just retrieval, needs work.","Retrieval strategies are not model-agnostic: the query expansion that helps stronger models can destabilize smaller ones (the paper reports Phi-4 losing 12 percent accuracy with query-expanded RAG), so augmentation design must be matched to the model's ability to follow decomposition instructions.","CTI copilots should pair LLMs with source-grounded retrieval rather than parametric memorization, and should audit answers against retrieved evidence, since models sometimes produce correct answers while fabricating the supporting justification (3 percent of predictions, up to 7 percent for one model)."],"fun_headline_variants":["Retrieval, not model scale, powers LLM threat analysis","Benchmark: LLMs without retrieval near zero on CTI tasks","Domain retrieval beats generic RAG for cyber intelligence","CTIConnect: Retrieval is the missing key for LLM CTI","Retrieval-augmented LLMs: New benchmark reveals retrieval wins"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Every reported number inherits the assumption that the 691 gold answers are correct and complete: the seed correlations, blog clusters, and blog-to-framework annotations are taken as ground truth, and the LLM-judge-plus-expert filter is assumed to have removed all hallucinated or mislabeled QA pairs, with no inter-annotator agreement and no dataset release reported for independent audit.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval, not model scale, powers LLM threat analysis","Benchmark: LLMs without retrieval near zero on CTI tasks","Domain retrieval beats generic RAG for cyber intelligence","CTIConnect: Retrieval is the missing key for LLM CTI","Retrieval-augmented LLMs: New benchmark reveals retrieval wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1666,"prompt_tokens":828,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":747}},"tokens_in":572,"tokens_out":838,"duration_ms":7367,"temperature":1.0,"reasoning_tokens":747,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:00:42.028666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random sample of the 691 QA pairs with fresh experts who never saw the original labels and check agreement: if it is too low to sustain the headline margins (roughly 0.08 to 0.26 points for query-expanded RAG on hybrid tasks and CSKG-guided RAG on unstructured tasks), the rankings could be label noise. A second check: in structured tasks, withhold the gold entry from the injected candidate set; near-perfect accuracy should collapse if grounding, not parametric memory, is what closes the gap.","supporting_citations":[],"review_version":1}