{"id":"bac82070-8a15-4c46-85ed-f04390905635","arxiv_id":"2505.09261","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper proposes an LLM-driven, dual-layer 'situational knowledge' memory for extracting MITRE ATT&CK techniques from threat reports, claiming an 11% higher F1 than GPT-4o, though the baseline evaluation is incomplete.","lead":"This paper presents a system that turns MITRE ATT&CK definitions and labeled example sentences into a searchable memory of situational contexts, which an LLM uses to classify cyberattack techniques in text. The authors report large F1 gains over GPT-4o, but the comparison is incomplete and the evaluation details are missing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported F1 gain may reflect training/test overlap: memory generation consumes labeled sentences from the same dataset used for evaluation, and the paper never defines a disjoint split.","rationale":"The reader's weakest assumption is exactly the point I would stress: the paper never establishes that the labeled sentences used for memory generation are disjoint from the evaluation sentences. Section 3.3.1 describes using labeled target sentences and contextually similar sentences to synthesize SKR entries, and Section 4 says evaluation uses the same dataset from [5] but gives no split or overlap analysis. Since retrieval is by semantic similarity, any test sentence that contributed to memory construction can be effectively looked up, so the comparison against zero-shot GPT-4o is not apples-to-apples. This is not a disagreement with the method's plausibility; the dual-layer SKR idea is reasonable and could be a useful contribution. But the empirical evidence as written cannot support the claimed superiority. There are additional problems: the abstract says 18% over GPT-4o while the contributions say 11%; the GPT-4o baseline is explicitly partial; and no code, prompts, or data are released. These reinforce the verdict without changing it. I therefore agree with the reader's REJECT verdict and identify the missing disjoint-split guarantee as the single most load-bearing concern.","tokens_in":8788,"tokens_out":2842,"duration_ms":30339,"concrete_test":"Re-run the evaluation on the Nguyen et al. (2024) dataset with memory generation strictly restricted to the training split: build all SKR entries only from training sentences, evaluate only on the official test split, and report the train/test sizes and any sentence-level overlap. Then recompute the Table 1 and Table 2 SKR2 F1 scores, and run GPT-4o on the full test set. If SKR2 F1 drops to within noise of the 0.45 GPT-4o result, the headline claim is unsupported; if it remains near 0.63, the concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the SKR memory system materially improves TTP classification over GPT-4o. This claim depends on the memory being built from labeled evidence that is disjoint from the evaluation data. Section 3.3.1 says the memory is initialized with 'Target Sentences from CTI Report' and that 'the initialization use all Target Sentences as the Contextually Similar Sentences,' while Section 4 says the framework is evaluated 'on the dataset provided by [5]' without reporting any train/test split or overlap check. If labeled sentences from the test portion were used to generate or retrieve SKR entries, Stage 1 can act as literal nearest-neighbor memorization: the retrieved state/action descriptions were synthesized from the very sentence being classified. GPT-4o and Deepseek-v3 receive no additional information, so the comparison would be between a system with access to the answer distribution and zero-shot baselines. The paper also states that the GPT-4o result 'use part of the test set, we will finish soon,' which further undermines the 18% headline improvement. Because the method only outperforms baselines when retrieval is trained on separate evidence, the missing split description is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SKR (Situational Knowledge Representation), a dual-layer memory structure for TTP extraction from CTI reports. Layer 1 stores a generalized \"situational context\" used for retrieval, and Layer 2 stores technique-specific \"manifestations\" used for classification and disambiguation. An LLM generates, updates, and applies these SKR instances; the framework first retrieves candidate techniques and then performs a verification step. The evaluation uses a dataset from Nguyen et al. [5] and reports that Qwen2.5-32B with SKR2 achieves Technique F1 0.63 versus GPT-4o F1 0.45 on the procedures dataset. The paper claims improved accuracy, standardization, transparency, and explainability compared with zero-shot baselines.","tokens_in":8961,"tokens_out":5535,"duration_ms":55752,"significance":"If validated, the idea of converting abstract ATT&CK definitions into a structured, evolvable, dual-layer memory is a useful contribution to TTP extraction and to the broader problem of standard-driven NLP. The two-layer design explicitly separates retrieval text from discriminative classification guidance, which is a sensible response to the verbosity and overlap problems of the official standard. The proposed framework also has practical value as a post-hoc standardization layer for other extraction systems. However, the significance is conditional on the evaluation being sound; the current evidence is undermined by an undocumented train/test split, an explicitly incomplete GPT-4o baseline, and a comparison that confounds the memory structure with access to labeled examples.","major_comments":[{"comment":"The manuscript never defines a train/test split. Memory initialization uses \"all Target Sentences as the Contextually Similar Sentences\" (Section 3.3.1), and evaluation is performed \"on the dataset provided by [5]\" (Section 4) without reporting split sizes or an overlap check. If the labeled sentences used to generate SKR entries include the evaluation sentences, Stage 1 retrieval can return a memory entry synthesized from the very sentence being classified, making the reported F1 gains an artifact of memorization rather than generalization. The authors must specify which sentences are used for memory generation, which are used for evaluation, and demonstrate that the two sets are disjoint.","section":"Section 3.3.1 and Section 4"},{"comment":"The evaluation note states: \"Notice, the 4o result use part of the test set, we will finish soon.\" This means the headline comparison against GPT-4o is explicitly incomplete, and the abstract's claim of an 18% F1 improvement over GPT-4o is not supported by a completed experiment. Furthermore, the abstract says 18% while the contributions section and conclusion say 11%; these numbers must be reconciled and based on a fully evaluated baseline.","section":"Section 4, Table 1 note"},{"comment":"The central comparison confounds the SKR memory structure with access to labeled training evidence. GPT-4o and Deepseek-v3 receive \"no additional information,\" whereas SKR1 and SKR2 are built from labeled sentences from the same dataset. To attribute the F1 gain to the SKR representation, the evaluation needs a retrieval-augmented baseline that uses the same labeled examples as retrieved raw text without the SKR state/action structure, as well as an ablation that uses the SKR memory but omits part of the labeled evidence. Without such controls, the reported gain may reflect the additional labeled signal rather than the proposed knowledge representation.","section":"Section 4, Tables 1 and 2"},{"comment":"No error bars, multiple runs, or statistical significance tests are reported for any of the F1 scores. Since the framework is LLM-based and involves retrieval and prompting, results are likely sensitive to random sampling and to the chosen top-k retrieval parameter. The paper should report variance across at least several runs or an appropriate significance test before claiming superiority over baselines such as Deepseek-v3, whose F1 0.46 is close to the reported procedures-dataset result.","section":"Section 4, Tables 1 and 2"}],"minor_comments":[{"comment":"\"the a new knowledge\" is a grammatical error; the sentence should be revised.","section":"Abstract"},{"comment":"\"Hybird\" should be \"Hybrid\" in the figure caption and in the text near Figure 2.","section":"Section 3, Figure 1 caption"},{"comment":"\"All the result use the same model without memtioned\" contains typos and unclear wording; it should read something like \"All results use the same model unless mentioned otherwise.\"","section":"Section 4"},{"comment":"Memory Optimization is described as adding new actions while preserving states, but no experiment isolates the contribution of the optimization step from the initial generation step; a clarifying ablation would strengthen the paper.","section":"Section 3.3.2"},{"comment":"The limitations list acknowledges that the framework is data-driven and depends on labeled examples, but it does not mention the train/test separation issue, which is the most immediate risk to the reported results.","section":"Section 5"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to be an early preprint: the evaluation section explicitly says the main baseline result is partial and \"we will finish soon,\" and the abstract/contributions give inconsistent improvement numbers. The missing train/test split is a load-bearing validity issue: if the memory generation uses the same labeled sentences as the evaluation, the central claim is not supported. I would recommend rejection, with the possibility of resubmission after a complete evaluation with a documented disjoint split, a finished GPT-4o comparison, and appropriate baseline ablations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper has a plausible idea—use an LLM to generate dual-layer state/action memory entries from official ATT&CK definitions plus labeled examples, then retrieve and classify. But the empirical claims are not supported as written, and the likely reason is a load-bearing evaluation flaw: memory generation appears to consume labeled sentences from the same dataset used for evaluation, and the paper never describes a disjoint train/test split. If test sentences went into memory synthesis, Stage 1 is not classification; it is nearest-neighbor memorization.\n\nWhat is genuinely new: the two-layer SKR structure—broad \"situational context\" for retrieval and technique-specific \"manifestation\" for discrimination—is a thoughtful answer to the mismatch between semantic similarity and contextual relevance. The two-stage pipeline, with the second stage able to reclassify the outputs of other systems, is also a reasonable architectural choice. And the paper is honest in its limitations section: it admits the framework is still data-driven and that the knowledge representation needs refinement.\n\nNow the soft spots, in proportion. The biggest is the missing split. Section 3.3.1 says initialization uses \"all Target Sentences\" as contextually similar sentences; Section 4 says evaluation is on \"the dataset provided by [5]\" with no overlap check. The stress-test note gets this exactly right. Second, the GPT-4o baseline is explicitly partial (\"we will finish soon\") and receives no additional evidence, while SKR is given labeled examples for memory construction—so the comparison is apples to oranges. Third, the numbers are internally inconsistent: the abstract claims an 18% boost, the contributions say 11%, and the tables show still different gaps. Fourth, there are no error bars, no code, no data, no prompts, and only five references, so neither reproducibility nor scholarly grounding is established. These are not minor oversights; they undermine the central quantitative conclusion.\n\nWho this is for: researchers working on standard-driven TTP extraction might find the SKR representation a useful conceptual seed, but not as evidence for a deployable system. A reader should treat the method description as the contribution and the evaluation as an illustration at best.\n\nRecommendation: this deserves peer review only if the authors can first fix the evaluation—report the split, verify non-overlap, finish the GPT-4o run, align the reported gains, and release code/data. The idea is not silly, and with a clean comparison it could become a solid application paper. As it stands, I would not cite the quantitative results, but I would not dismiss the approach outright. Send it to review with a strong demand for a corrected evaluation; reject if the authors cannot provide one.\n\nYours.","headline":"The SKR memory idea is genuinely interesting, but the evaluation likely leaks test labels into memory generation, so the empirical claims do not hold as written.","tokens_in":9543,"tokens_out":3460,"would_cite":false,"duration_ms":33724,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that converting MITRE ATT&CK definitions into a dual-layer, evolvable memory of situational contexts and technique differentiators lets an LLM classify TTPs from threat reports more accurately and more consistently than…","keywords":["MITRE ATT&CK","TTP extraction","Situational Knowledge Representation","evolvable memory","large language models","retrieval-augmented generation","cyber threat intelligence","standard-driven classification"],"falsifier":"Check the dataset from [5] for sentence-level overlap between the memory-generation inputs and the test set. If any test sentence, or a near-duplicate, was used as a target sentence or as one of the contextually similar sentences in the memory-generation step, rerun the SKR2 pipeline on a strictly disjoint split; if its F1 drops from 0.63 toward the official-definition baseline of 0.47 on the procedures dataset, the claimed advantage is memorization, not standard-driven generalization.","tokens_in":8557,"feed_emoji":"🛡️","tokens_out":5196,"duration_ms":47184,"temperature":0.7,"pith_summary":"The paper is trying to establish that the verbosity and overlap of official MITRE ATT&CK definitions are the bottleneck in automated TTP extraction, and that replacing them with machine-generated, dual-layer knowledge fixes it. The proposed framework builds an evolvable memory: each entry has a Layer-1 situational context, such as communication with C2 using encoded subdomains, and Layer-2 technique manifestations that say what distinguishes, say, T1132 Data Encoding from T1071 Application Layer Protocol. An LLM creates these entries from labeled sentences plus official definitions, retrieves them for classification, and then re-checks the classification against the contrastive Layer-2 details. On the paper's evaluation, the two-stage variant reaches a Technique F1 of 0.63 on the procedures dataset and 0.35 on the expert dataset, compared with 0.45 and 0.15 for GPT-4o; the abstract reports an 18% gain while the contributions say 11%. A sympathetic reader would care because the mechanism gives every assignment a traceable rationale, which is exactly what standard-driven threat-intelligence pipelines lack.","feed_headline":"Dual-layer memory beats GPT-4o at extracting ATT&CK techniques","feed_subtitle":"Situation/action memory lifts technique F1 from 0.45 to 0.63 versus GPT-4o on the procedures benchmark.","key_machinery":"The central object is the Situational Knowledge Representation (SKR), a structured memory entry of the form state to a map from TTP IDs to manifestations. The state is a technique-agnostic description of an attack scenario; the manifestation is a concise, contrastive description of how one technique appears inside that scenario. This dual-layer object carries the argument because it converts the official standard's long definitions into short, retrieval-friendly and classification-ready units, and it is what the LLM generates, retrieves, updates, and prunes. Also load-bearing is the two-stage extraction pipeline: Stage 1 retrieves the k most similar states and asks the LLM to classify; Stage 2 retrieves the Layer-2 manifestations for the candidate technique and re-evaluates the assignment using contrastive information.","core_discovery":"On the paper's own terms, the central discovery is that separating retrieval text from classification guidance within a single knowledge representation solves a contradiction that hurts direct standard-based prompting. Retrieval wants a broad scenario description that matches many phrasings; classification wants narrow discriminative details. The Situational Knowledge Representation splits these into a state field and an action map, so the same memory entry can both find the right neighbourhood of techniques and tell them apart. The paper claims this representation, when generated, refined, and forgotten through an LLM-driven memory lifecycle, outperforms strong baselines: SKR2, using Qwen2.5-32B with generation plus verification, achieves Technique F1 of 0.63 versus GPT-4o's 0.45 on the procedures dataset, and 0.35 versus 0.15 on the expert dataset. It further claims the second verification stage can re-label the outputs of other TTP extraction systems, making the framework a standardization layer rather than only a standalone classifier.","pith_inferences":["The state/action split is not specific to ATT&CK; the same pattern could be applied to any verbose, overlapping taxonomy where retrieval and discrimination pull in opposite directions.","The paper's abstract quotes an 18% F1 boost while the contributions quote 11%; the tables support 0.63 versus 0.45 on the procedures dataset, so replication reports should state which dataset and comparison produce each number.","Because the memory entries are human-readable strings, the generated memory itself could be audited or edited by analysts before deployment, turning the framework into a human-in-the-loop knowledge base rather than an end-to-end black box.","A direct testable extension would be to freeze the memory and swap the underlying LLM; if performance transfers across models, the knowledge representation rather than the model is doing the work."],"forward_implications":["TTP assignment no longer has to begin from the raw standard; a small generated memory can carry the actionable version of the standard.","Because Stage 2 can take an external classification as input, the framework can be used to re-check and standardize outputs from existing TTP extraction systems without retraining them.","Memory update and forgetting give a concrete path to track ATT&CK framework revisions: new technique manifestations can be added to existing states, and stale entries can be pruned by their classification utility.","If the reported F1 holds, a general-purpose open LLM such as Qwen2.5-32B can beat a frontier commercial model on this task when supplied with the right memory."],"supporting_citations":[{"why":"supplies the labeled CTI dataset used for memory generation and evaluation; the framework's quantitative claims stand or fall on this corpus.","marker":"[5]"},{"why":"establishes direct prompting of GPT-3.5 and Bard for ATT&CK mapping as a baseline approach the paper improves on.","marker":"[3]"},{"why":"provides evidence that fine-tuning modern LLMs on CTI extraction yields mixed gains, motivating the memory-based alternative.","marker":"[4]"}],"fun_headline_variants":["Split retrieval/classification memory lifts ATT&CK F1 over GPT-4o","Evolvable dual-layer memory improves TTP extraction accuracy","LLM-generated memory re-labels ATT&CK techniques better than GPT-4o","Situation-action memory: 0.63 vs 0.45 F1 for ATT&CK techniques"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the labeled sentences used to generate memory entries are disjoint from the sentences used in testing, but the paper never describes this split; if the same sentences appear on both sides, the reported F1 gains would largely be memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["Split retrieval/classification memory lifts ATT&CK F1 over GPT-4o","Evolvable dual-layer memory improves TTP extraction accuracy","LLM-generated memory re-labels ATT&CK techniques better than GPT-4o","Situation-action memory: 0.63 vs 0.45 F1 for ATT&CK techniques"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":2077,"prompt_tokens":1047,"completion_tokens":1030,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":942}},"tokens_in":663,"tokens_out":1030,"duration_ms":9376,"temperature":1.0,"reasoning_tokens":942,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:35:49.606073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the dataset from [5] for sentence-level overlap between the memory-generation inputs and the test set. If any test sentence, or a near-duplicate, was used as a target sentence or as one of the contextually similar sentences in the memory-generation step, rerun the SKR2 pipeline on a strictly disjoint split; if its F1 drops from 0.63 toward the official-definition baseline of 0.47 on the procedures dataset, the claimed advantage is memorization, not standard-driven generalization.","supporting_citations":[{"cited_title":"Noise contrastive estimation-based matching framework for low-resource security attack pattern recognition,","cited_arxiv_id":null,"evidence_quote":"supplies the labeled CTI dataset used for memory generation and evaluation; the framework's quantitative claims stand or fall on this corpus."}],"review_version":1}