{"id":"474914a2-dc90-43cd-a7a6-9358d9fd7022","arxiv_id":"2505.03147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-step pipeline using GPT-3.5 summarisation and a retrained, rebalanced SciBERT model raises F1 scores for MITRE ATT&CK technique extraction compared with baseline TRAM.","lead":"This paper tests automated ways to map cyber threat reports to MITRE ATT&CK attack techniques, comparing Llama2, TRAM, and a proposed two-step LLM-plus-SciBERT pipeline. The best configuration raises F1 by about seven percentage points over baseline TRAM, though missing ablations and error bars leave the size of the gain uncertain.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed F1 gain from GPT-3.5-augmented retraining is not separable from label leakage or test-set contamination because the paper never specifies how generated summaries/sentences were labeled, split, or verified.","rationale":"The most load-bearing condition for the central claim is that the GPT-3.5 component only reduces verbosity and rebalances the training distribution without inserting label information or overlapping the test distribution. The manuscript describes the pipeline at a high level and gives no prompt, no generated-data examples, no label-verification procedure, and no train/test split specification. This matters because a model that sees paraphrases of test reports at train time, or summaries that explicitly name techniques, can achieve high F1 without learning a general extraction capability. Configuration 4's gain is also reported against the original SciBERT baseline rather than against Configuration 3, so the contribution of rebalancing/retraining is not isolated; nevertheless, label/test leakage is the more fundamental threat to the headline result. I credit the authors for releasing code and using public datasets, and the concern is not an accusation of wrongdoing; it is a gap in evidence that a direct reproducibility audit can close. The reader already captured this as the weakest assumption, and the CONDITIONAL verdict is appropriately calibrated; no verdict change is needed.","tokens_in":6943,"tokens_out":7936,"duration_ms":73762,"concrete_test":"Audit the released repository and reproduce Configuration 4 with this check: extract every GPT-3.5-generated training sentence, verify its label against the source-report ground truth, and compute its maximum ROUGE-L and 5-gram overlap with every held-out test sentence. Then retrain the SciBERT model after removing all generated sentences whose ROUGE-L to any test sentence exceeds 0.3. If the median F1 over the top-50 techniques drops by more than 3 points, the reported gain is due to train/test overlap rather than to the proposed method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Configuration 4 (§3.3) combines GPT-3.5 report summarisation, GPT-3.5 augmentation of underrepresented techniques, and downsampling, then reports a median +7-point F1 gain over the baseline and per-technique F1 values up to 0.92 (Table 4). The central premise is that the synthetic summaries/sentences preserve the true ATT&CK labels and are never seen at evaluation time. The paper gives no generation prompt, no label-consistency check, and no description of how the augmented set was split from the test reports. Reports were only stripped of technique IDs and links (§3.2), so GPT-3.5 text that names or paraphrases techniques can inject discriminative label cues into the input. If the same reports were summarised once and then used both to generate training sentences and to form the test set, the F1 improvement is a train/test overlap artifact. Additionally, Figure 2 compares the retrained model against the baseline, not against Configuration 3, so the marginal effect of rebalancing/retraining is not isolated; Configuration 3 already benefits from summarisation. The 'selected techniques' in Table 4 and the note 'selection is based on best performance settings' further weaken attribution. None of this proves leakage, but the burden is on the paper to rule it out because the headline number depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates automated extraction of MITRE ATT&CK attack techniques from CTI reports. It compares four configurations: zero-shot Llama2 variants (7B, 13B, 70B) as standalone extractors; TRAM with pretrained SciBERT; TRAM with GPT-3.5-based report summarisation; and a two-step pipeline in which GPT-3.5 summarises reports, underrepresented techniques are augmented with GPT-3.5-generated data, overrepresented techniques are downsampled, and SciBERT is retrained on the rebalanced set. Using two ground-truth datasets (AEL and ATD), the paper reports that retraining the rebalanced model yields a median F1 increase of about seven percentage points over the baseline and per-technique F1 values above 0.90, with a best value of 0.9231 for T1070.004. The main contributions claimed are a comprehensive evaluation of existing tools and a novel summarisation-plus-retraining pipeline to mitigate class imbalance and overfitting.","tokens_in":7217,"tokens_out":2867,"duration_ms":26515,"significance":"If the reported gains are real, the pipeline is a practically useful step toward automated CTI technique extraction, and the released code supports reproducibility. The paper addresses a genuine problem (manual analysis of verbose, domain-specific threat reports) and compares against established tools (TRAM) and external ground-truth annotations, which is a strength. However, the central empirical claim currently rests on an incomplete experimental design: the configuration comparisons conflate several interventions, the generation and splitting of the GPT-3.5-augmented training data are underspecified, and the model/threshold selection appears to use test-set performance. The standalone LLM evaluation is based on six reports only. These gaps are fixable, and the paper would be substantially strengthened by an ablation, a clear train/test split description, and uncertainty quantification.","major_comments":[{"comment":"","section":"§3.3 and §4.2, Figure 2"},{"comment":"The paper does not specify how GPT-3.5-generated training sentences and summaries were labelled, how label consistency was verified, or how the augmented training set was separated from the test reports. The preprocessing only strips technique IDs and links (§3.2), so generated or summarised text that names or paraphrases techniques can inject discriminative cues into the input. If any generated data overlaps with test reports, the F1 improvement is a train/test leakage artifact rather than a genuine capability gain. Please provide the generation prompt, a label-consistency check, and an explicit description of the data split and decontamination procedure.","section":"§3.3, Configuration 4; §4.2, Table 4"},{"comment":"The text states that the 25%/75% confidence setting for Configuration 3 'is based on best performance settings' and that Table 4 reports results for the 'best-performing retrained SciBERT model.' Selecting confidence thresholds and the final model based on test-set performance leaks test information into the evaluation and can inflate reported F1 values. The authors should fix thresholds/configuration a priori, use a validation split for model selection, or report the full configuration grid with uncertainty measures, so that the reported numbers are honest estimates of expected performance.","section":"§3.3 and §4.2 (confidence thresholds, model selection)"},{"comment":"The standalone LLM evaluation uses only six short AEL reports, and Table 3 reports aggregate precision, recall, and F1 without error bars or significance tests. The paper's conclusion that Llama2-7B outperforms Llama2-70B rests on a sample too small to support it. In addition, RQ1 is phrased broadly ('How effective are standalone vanilla LLMs in CTI extraction?') but no LLM results are reported on the ATD dataset. A larger evaluation or a clear statement of the scope limitation is needed.","section":"§4.1, Tables 1–3"}],"minor_comments":[{"comment":"The exact prompts for Llama2 zero-shot extraction and for GPT-3.5 summarisation/augmentation are not included in the paper, which makes the reported results hard to reproduce even with the code link.","section":"§3.3 / reproducibility"},{"comment":"The criterion for selecting the techniques shown in Table 4 is not stated; please clarify whether they were chosen as examples, by best performance, or by some other rule.","section":"§4.2, Table 4"},{"comment":"The name 'aCTIon' is introduced without definition or a citation; adding one or two sentences to explain the naming convention would improve clarity.","section":"§3.3, Configuration 3"},{"comment":"Capitalisation is inconsistent ('Llama2' vs 'LLama2', 'SciBert' vs 'SciBERT'); please standardise.","section":"Throughout"},{"comment":"The paper says 'We only report the F1-score in this work due to space constraints,' yet Table 4 includes precision and recall; please reconcile this statement with the presented data.","section":"§4, evaluation metrics"},{"comment":"References [1] and [5] appear to describe the same work and should be merged or disambiguated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is already accepted at WWW '25, per the footnote, which may partly explain the compressed evaluation and the 'due to space constraints' remark. For a journal version, the experimental rigor needs to meet a higher bar: the leakage/contamination concern around GPT-3.5-augmented data is load-bearing, and the lack of an ablation between summarisation and retraining makes the central attribution argument incomplete. I would not reject; these are fixable with careful additional experiments and reporting. The authors should also consider whether the retraining pipeline generalises beyond the top-50 techniques, since the current scope is tied to TRAM's label set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the two-step pipeline is a reasonable engineering contribution, but the paper does not give the reader the one detail needed to trust its headline number—how the GPT-3.5-generated training sentences were kept out of the test set. The stress-test note is right that this is a burden-shifting situation. I don't see evidence of deliberate leakage, but the paper needs to rule it out.\n\nWhat's genuinely new: combining LLM summarisation with LLM-augmented rebalancing and a retrained SciBERT is a clean, simple idea that isn't in the cited prior work. The results on the two public datasets (ATD and AEL) show that the combination can push F1 above 0.9 for some techniques, which would be useful for SOC triage. The evaluation of vanilla Llama2 models is honest and useful: even the 70B model sits below 0.3 F1, which justifies the need for specialised pipelines. The code is linked, which is a plus.\n\nWhere the paper wobbles: first, no ablations. The reader cannot say whether the +7 F1 comes from summarisation, from rebalancing, or from retraining alone. The figure compares the final config to the pretrained baseline, not to the summarisation-only config. Second, thresholds and the 'best-performing' retraining setup are selected on the same test data, and there are no error bars, so the reported numbers are likely optimistic. Third, and most important, the generation step is opaque: no prompt, no label verification, no description of how the augmented sentences were split. If a generated sentence paraphrases a technique name, the model can exploit that cue; if the underlying report appears in the test set, the gain is an artifact. The paper needs to describe the split and ideally show that results hold on a held-out corpus.\n\nThe small standalone-LLM evaluation (six short reports) is a minor concern, given that the main results are on the larger ATD set.\n\nWho this is for: practitioners who want a working recipe for ATT&CK mapping, and researchers who want a cautionary example of how easy it is to under-specify leakage. I'd send it to peer review—the idea deserves scrutiny and the evidence could be fixed with a proper ablation and a clearer data-handling description—but as it stands the central claim is not yet established.\n\nRecommendation: have the authors run a clean split with no shared reports across train/test, add an ablation (summarisation only, rebalancing only, retrained only), report confidence intervals, and stop selecting thresholds on the test set.","headline":"Plausible and useful LLM+retraining pipeline, but the headline performance gain is not credible until the authors show how the generated data stayed out of the test set.","tokens_in":7741,"tokens_out":4061,"would_cite":true,"duration_ms":36192,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-step LLM pipeline—GPT-3.5 summarisation followed by a retrained, rebalanced SciBERT classifier—raises median F1 for MITRE ATT&CK technique extraction by about seven percentage points, with several techniques exceeding 0.90 F1.","keywords":["Cyber Threat Intelligence","MITRE ATT&CK","attack technique identification","large language models","SciBERT","text summarisation","class imbalance","TRAM"],"falsifier":"Train the same SciBERT model on a rebalanced dataset that uses only downsampling and original sentences, with no GPT-3.5-generated examples, and compare F1 on the same test split; if the seven-point gain disappears, the LLM augmentation stage is not doing the work claimed. Separately, measure n-gram or embedding overlap between the GPT-3.5-generated training sentences and the test-set sentences; high overlap would indicate contamination and would invalidate the reported scores.","tokens_in":6755,"feed_emoji":"🛡️","tokens_out":7090,"duration_ms":61215,"temperature":0.7,"pith_summary":"The paper sets out to automate the mapping of web-sourced cyber threat intelligence (CTI) reports to MITRE ATT&CK attack techniques, a task human analysts currently do by hand. It claims that the main obstacle is not classifier architecture but training data: the existing TRAM pipeline caps out near 0.4 F1 because common techniques dominate and the model overfits. The proposed fix is a two-step pipeline in which GPT-3.5 first summarises verbose reports, and a SciBERT model is then retrained on a rebalanced dataset where underrepresented techniques are augmented with GPT-3.5-generated sentences and overrepresented ones are downsampled. On the Attack-Technique-Dataset this raises the median F1-score (a combined precision-recall measure) by about seven percentage points over the baseline, and several techniques reach F1 above 0.90. If this holds, automated CTI extraction could become reliable enough to reduce alert fatigue in security operations centres.","feed_headline":"Two-step LLM pipeline lifts attack-technique F1 to 0.92","feed_subtitle":"Summarising reports with GPT-3.5 then retraining SciBERT on rebalanced data beats the baseline by seven F1 points.","key_machinery":"The load-bearing mechanism is the two-step TRAM-based pipeline: (i) an LLM (GPT-3.5) summarises each CTI report to strip away verbose narrative and keep attack-relevant sentences, and (ii) a SciBERT model—a transformer language model pretrained on scientific text—is retrained on a rebalanced Attack-Technique-Dataset, where rare techniques are augmented with GPT-3.5-generated sentence examples and frequent techniques are downsampled. The rebalancing and retraining step is what carries the reported F1 gain; summarisation alone produces only a modest improvement.","core_discovery":"The central claim is that rebalancing and retraining, not a new architecture, is what unlocks accurate technique identification. Baseline TRAM with the original pretrained SciBERT reaches an F1-score of just over 0.4; adding GPT-3.5 report summarisation alone improves it only slightly. The full configuration summarises each report with GPT-3.5 and then feeds the sentences to a SciBERT model retrained on a rebalanced version of the top-50-technique dataset, and this configuration reports a median F1 increase of roughly seven percentage points, with scores up to 0.92; for instance T1070.004 reaches 0.9231 and T1566.001 reaches 0.9109. The paper also reports that standalone zero-shot Llama2 models, including the 70B variant, perform poorly on this task, with the best F1 around 0.27, which motivates the need for a specialised two-step pipeline.","pith_inferences":["If the augmentation gain is genuine rather than a label-leakage artifact, LLM-generated sentences could substitute for expensive expert annotation of rare techniques, an implication the paper does not develop.","The same summarise-then-rebalance recipe is likely transferable to other threat taxonomy tasks, such as tactics or CAPEC patterns, because the machinery is not tied to the top-50 technique list.","A controlled comparison against non-LLM data augmentation (e.g., back-translation or synonym replacement) would isolate whether the gain comes from the LLM's domain-appropriate phrasing or simply from having a more balanced training set.","The weak zero-shot Llama2 results suggest that instruction-tuned or fine-tuned LLMs, rather than vanilla base models, are the more promising route for LLM-based extraction; the paper leaves that direction to future work."],"forward_implications":["Summarising verbose reports before classification improves extraction even without retraining, confirming that report verbosity is a real obstacle for CTI extraction.","Retraining the classifier on a rebalanced dataset accounts for the main performance jump: median F1 rises by about seven percentage points over the original TRAM baseline.","With the retrained model, several of the top-50 ATT&CK techniques are identified at F1 above 0.90, including T1070.004 at 0.9231 and T1566.001 at 0.9109.","Standalone open-source LLMs with zero-shot prompting are insufficient for technique identification; the best Llama2 configuration reaches only about 0.27 F1.","Underrepresented techniques and false positives remain problem cases, so further progress depends on handling rare classes and reducing overgeneration."],"supporting_citations":[{"why":"Supplies the SciBERT pretrained model that the retrained classifier is built on and that TRAM uses by default.","marker":"[2]"},{"why":"Defines TRAM, the base extraction method and baseline whose performance the paper's pipeline is compared against.","marker":"[4]"},{"why":"Provides the AEL ground-truth dataset of concise attack reports used to evaluate standalone Llama2 models.","marker":"[3]"},{"why":"Provides the ATD ground-truth dataset of longer reports on which the main rebalancing, retraining, and F1 comparisons are run.","marker":"[12]"},{"why":"Supplies the Llama2 7B, 13B, and 70B models evaluated for zero-shot CTI extraction in RQ1.","marker":"[21]"}],"fun_headline_variants":["LLM-summarised reports boost attack detection F1 by 7 points","Retrained SciBERT plus GPT-3.5 summaries beat baseline CTI mapper","Rebalanced data and LLM summaries lift technique F1 to 0.92","GPT-3.5 summaries and retrained SciBERT improve threat report mapping","Zero-shot Llama2 fails, but two-step LLM pipeline hits 0.92 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gain rests on the assumption that the GPT-3.5-generated training sentences and summaries preserve the same ground-truth ATT&CK technique labels as the original reports and do not leak test-set content into training; if the generated data simply repeats the labels the model is meant to predict, or contaminates the evaluation split, the F1 improvement would be an artifact rather than a real capability gain.","fun_headline_variants_meta":{"raw":{"variants":["LLM-summarised reports boost attack detection F1 by 7 points","Retrained SciBERT plus GPT-3.5 summaries beat baseline CTI mapper","Rebalanced data and LLM summaries lift technique F1 to 0.92","GPT-3.5 summaries and retrained SciBERT improve threat report mapping","Zero-shot Llama2 fails, but two-step LLM pipeline hits 0.92 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3418,"prompt_tokens":931,"completion_tokens":2487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2378}},"tokens_in":547,"tokens_out":2487,"duration_ms":17312,"temperature":1.0,"reasoning_tokens":2378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:46.348322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same SciBERT model on a rebalanced dataset that uses only downsampling and original sentences, with no GPT-3.5-generated examples, and compare F1 on the same test split; if the seven-point gain disappears, the LLM augmentation stage is not doing the work claimed. Separately, measure n-gram or embedding overlap between the GPT-3.5-generated training sentences and the test-set sentences; high overlap would indicate contamination and would invalidate the reported scores.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines TRAM, the base extraction method and baseline whose performance the paper's pipeline is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AEL ground-truth dataset of concise attack reports used to evaluate standalone Llama2 models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ATD ground-truth dataset of longer reports on which the main rebalancing, retraining, and F1 comparisons are run."}],"review_version":1}