Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A two-step LLM pipeline—GPT-3.5 summarisation followed by a retrained, rebalanced SciBERT classifier—raises median F1 for MITRE ATT&CK technique extraction by about seven percentage points, with several techniques exceeding 0.90 F1.

desk verdict Plausible and useful LLM+retraining pipeline, but the headline performance gain is not credible until the authors show how the generated data stayed out of the test set. read the letter →

arxiv 2505.03147 v1 pith:GFQ7AU4U submitted 2025-05-06 cs.CR

classification cs.CR
keywords CyberThreatIntelligenceMITREATT&CKattacktechniqueidentificationlargelanguagemodelsSciBERTtextsummarisationclassimbalanceTRAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to automate the mapping of web-sourced cyber threat intelligence (CTI) reports to MITRE ATT&CK attack techniques, a task human analysts currently do by hand. It claims that the main obstacle is not classifier architecture but training data: the existing TRAM pipeline caps out near 0.4 F1 because common techniques dominate and the model overfits. The proposed fix is a two-step pipeline in which GPT-3.5 first summarises verbose reports, and a SciBERT model is then retrained on a rebalanced dataset where underrepresented techniques are augmented with GPT-3.5-generated sentences and overrepresented ones are downsampled. On the Attack-Technique-Dataset this raises the median F1-score (a combined precision-recall measure) by about seven percentage points over the baseline, and several techniques reach F1 above 0.90. If this holds, automated CTI extraction could become reliable enough to reduce alert fatigue in security operations centres.

What carries the argument

The load-bearing mechanism is the two-step TRAM-based pipeline: (i) an LLM (GPT-3.5) summarises each CTI report to strip away verbose narrative and keep attack-relevant sentences, and (ii) a SciBERT model—a transformer language model pretrained on scientific text—is retrained on a rebalanced Attack-Technique-Dataset, where rare techniques are augmented with GPT-3.5-generated sentence examples and frequent techniques are downsampled. The rebalancing and retraining step is what carries the reported F1 gain; summarisation alone produces only a modest improvement.

What would settle it

Train the same SciBERT model on a rebalanced dataset that uses only downsampling and original sentences, with no GPT-3.5-generated examples, and compare F1 on the same test split; if the seven-point gain disappears, the LLM augmentation stage is not doing the work claimed. Separately, measure n-gram or embedding overlap between the GPT-3.5-generated training sentences and the test-set sentences; high overlap would indicate contamination and would invalidate the reported scores.

Watch

Extended reading notes

Core claim

The central claim is that rebalancing and retraining, not a new architecture, is what unlocks accurate technique identification. Baseline TRAM with the original pretrained SciBERT reaches an F1-score of just over 0.4; adding GPT-3.5 report summarisation alone improves it only slightly. The full configuration summarises each report with GPT-3.5 and then feeds the sentences to a SciBERT model retrained on a rebalanced version of the top-50-technique dataset, and this configuration reports a median F1 increase of roughly seven percentage points, with scores up to 0.92; for instance T1070.004 reaches 0.9231 and T1566.001 reaches 0.9109. The paper also reports that standalone zero-shot Llama2 models, including the 70B variant, perform poorly on this task, with the best F1 around 0.27, which motivates the need for a specialised two-step pipeline.

Load-bearing premise

The reported gain rests on the assumption that the GPT-3.5-generated training sentences and summaries preserve the same ground-truth ATT&CK technique labels as the original reports and do not leak test-set content into training; if the generated data simply repeats the labels the model is meant to predict, or contaminates the evaluation split, the F1 improvement would be an artifact rather than a real capability gain.

Editorial extensions

If this is right

  • Summarising verbose reports before classification improves extraction even without retraining, confirming that report verbosity is a real obstacle for CTI extraction.
  • Retraining the classifier on a rebalanced dataset accounts for the main performance jump: median F1 rises by about seven percentage points over the original TRAM baseline.
  • With the retrained model, several of the top-50 ATT&CK techniques are identified at F1 above 0.90, including T1070.004 at 0.9231 and T1566.001 at 0.9109.
  • Standalone open-source LLMs with zero-shot prompting are insufficient for technique identification; the best Llama2 configuration reaches only about 0.27 F1.
  • Underrepresented techniques and false positives remain problem cases, so further progress depends on handling rare classes and reducing overgeneration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the augmentation gain is genuine rather than a label-leakage artifact, LLM-generated sentences could substitute for expensive expert annotation of rare techniques, an implication the paper does not develop.
  • The same summarise-then-rebalance recipe is likely transferable to other threat taxonomy tasks, such as tactics or CAPEC patterns, because the machinery is not tied to the top-50 technique list.
  • A controlled comparison against non-LLM data augmentation (e.g., back-translation or synonym replacement) would isolate whether the gain comes from the LLM's domain-appropriate phrasing or simply from having a more balanced training set.
  • The weak zero-shot Llama2 results suggest that instruction-tuned or fine-tuned LLMs, rather than vanilla base models, are the more promising route for LLM-based extraction; the paper leaves that direction to future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper evaluates automated extraction of MITRE ATT&CK attack techniques from CTI reports. It compares four configurations: zero-shot Llama2 variants (7B, 13B, 70B) as standalone extractors; TRAM with pretrained SciBERT; TRAM with GPT-3.5-based report summarisation; and a two-step pipeline in which GPT-3.5 summarises reports, underrepresented techniques are augmented with GPT-3.5-generated data, overrepresented techniques are downsampled, and SciBERT is retrained on the rebalanced set. Using two ground-truth datasets (AEL and ATD), the paper reports that retraining the rebalanced model yields a median F1 increase of about seven percentage points over the baseline and per-technique F1 values above 0.90, with a best value of 0.9231 for T1070.004. The main contributions claimed are a comprehensive evaluation of existing tools and a novel summarisation-plus-retraining pipeline to mitigate class imbalance and overfitting.

Significance. If the reported gains are real, the pipeline is a practically useful step toward automated CTI technique extraction, and the released code supports reproducibility. The paper addresses a genuine problem (manual analysis of verbose, domain-specific threat reports) and compares against established tools (TRAM) and external ground-truth annotations, which is a strength. However, the central empirical claim currently rests on an incomplete experimental design: the configuration comparisons conflate several interventions, the generation and splitting of the GPT-3.5-augmented training data are underspecified, and the model/threshold selection appears to use test-set performance. The standalone LLM evaluation is based on six reports only. These gaps are fixable, and the paper would be substantially strengthened by an ablation, a clear train/test split description, and uncertainty quantification.

major comments (4)
  1. [§3.3 and §4.2, Figure 2]
  2. [§3.3, Configuration 4; §4.2, Table 4] The paper does not specify how GPT-3.5-generated training sentences and summaries were labelled, how label consistency was verified, or how the augmented training set was separated from the test reports. The preprocessing only strips technique IDs and links (§3.2), so generated or summarised text that names or paraphrases techniques can inject discriminative cues into the input. If any generated data overlaps with test reports, the F1 improvement is a train/test leakage artifact rather than a genuine capability gain. Please provide the generation prompt, a label-consistency check, and an explicit description of the data split and decontamination procedure.
  3. [§3.3 and §4.2 (confidence thresholds, model selection)] The text states that the 25%/75% confidence setting for Configuration 3 'is based on best performance settings' and that Table 4 reports results for the 'best-performing retrained SciBERT model.' Selecting confidence thresholds and the final model based on test-set performance leaks test information into the evaluation and can inflate reported F1 values. The authors should fix thresholds/configuration a priori, use a validation split for model selection, or report the full configuration grid with uncertainty measures, so that the reported numbers are honest estimates of expected performance.
  4. [§4.1, Tables 1–3] The standalone LLM evaluation uses only six short AEL reports, and Table 3 reports aggregate precision, recall, and F1 without error bars or significance tests. The paper's conclusion that Llama2-7B outperforms Llama2-70B rests on a sample too small to support it. In addition, RQ1 is phrased broadly ('How effective are standalone vanilla LLMs in CTI extraction?') but no LLM results are reported on the ATD dataset. A larger evaluation or a clear statement of the scope limitation is needed.
minor comments (6)
  1. [§3.3 / reproducibility] The exact prompts for Llama2 zero-shot extraction and for GPT-3.5 summarisation/augmentation are not included in the paper, which makes the reported results hard to reproduce even with the code link.
  2. [§4.2, Table 4] The criterion for selecting the techniques shown in Table 4 is not stated; please clarify whether they were chosen as examples, by best performance, or by some other rule.
  3. [§3.3, Configuration 3] The name 'aCTIon' is introduced without definition or a citation; adding one or two sentences to explain the naming convention would improve clarity.
  4. [Throughout] Capitalisation is inconsistent ('Llama2' vs 'LLama2', 'SciBert' vs 'SciBERT'); please standardise.
  5. [§4, evaluation metrics] The paper says 'We only report the F1-score in this work due to space constraints,' yet Table 4 includes precision and recall; please reconcile this statement with the presented data.
  6. [References] References [1] and [5] appear to describe the same work and should be merged or disambiguated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical F1 comparison on external ground-truth datasets (AEL/ATD), not a derivation that reduces to its inputs.

full rationale

The paper's central claim, a median F1-score increase of approximately seven percentage points from retraining SciBERT on a rebalanced, GPT-3.5-augmented dataset, is evaluated against externally annotated ground-truth datasets (AEL and ATD, Section 3.2). The F1-scores in Table 4 are computed on test-set sentences with ground-truth technique labels, so the reported numbers are not obtained by construction from the training objective or from the generated augmentation data. The note in Section 3.3 that a configuration 'selection is based on best performance settings' and the Table 4 caption's 'selected techniques' indicate post-hoc selection of operating points and reporting subsets, which can inflate reported numbers, but this is a methodological reporting weakness rather than circularity: no fitted parameter is renamed as a prediction, and no equation reduces the result to its input. The self-citations in the introduction and future-work sections ([1], [5], [7], [13], [16], [18], [19], [20]) are not load-bearing for the extraction pipeline or the F1 comparison; they support motivation and are not invoked to forbid alternatives or to justify the central result. No uniqueness theorem, ansatz, or known result is smuggled in via citation. The paper does not provide generation prompts, label-consistency checks, or explicit descriptions of how the augmented training data was split from the test reports, which is a legitimate reproducibility and possible-leakage concern, but the manuscript itself contains no quoted step where the claimed prediction is equivalent to the input by definition. Under the rule that circularity must be demonstrated by specific reduction rather than suspected, the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on assumptions about label-preserving LLM augmentation and leak-free splits. No new physical or conceptual entities are introduced; LLM-generated training sentences are data artifacts, not theoretical inventions.

free parameters (3)
  • Confidence thresholds for TRAM/SciBERT = 25% and 80% for baseline; 25% and 75% for aCTIon; best-performing setting reported
    Thresholds were selected based on best performance rather than fixed a priori, which can inflate reported F1 on the test set (Section 3.3, Configuration 3; Section 4.2).
  • Best retraining configuration = retraining with 5-fold cross-validation, reported as best
    Three training settings were tried and the best is reported in Table 4, so the headline number is a post-hoc selection (Section 3.3, Configuration 4).
  • Top-50 technique restriction = 50 most prevalent techniques
    Techniques outside the top 50 are excluded to align with TRAM's training data, limiting evaluation to common techniques (Section 3.2).
assumptions (4)
  • domain assumption AEL and ATD ground-truth annotations are accurate and complete.
    Used as ground truth in Section 3.2; no inter-annotator agreement or validation is reported.
  • domain assumption The preprocessed reports and augmented LLM data do not leak into the test set.
    Required for the validity of the F1 improvement in Section 4.2; the paper does not describe a data-contamination check.
  • domain assumption GPT-3.5 summaries and generated sentences preserve ATT&CK technique labels.
    The augmentation step in Section 3.3 assumes generated text is label-preserving; no manual quality audit is reported.
  • domain assumption 5-fold cross-validation splits are i.i.d. and representative of future reports.
    The performance estimate relies on this assumption for generalization claims in Sections 3.3 and 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using Large Language Models." pith.science (2026). https://pith.science/paper/GFQ7AU4U

@misc{pith2026250503147,
  author       = {Pith},
  title        = {Pith review of: Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFQ7AU4U}},
  note         = {Machine review of arXiv:2505.03147}
}
read the original abstract

This work evaluates the performance of Cyber Threat Intelligence (CTI) extraction methods in identifying attack techniques from threat reports available on the web using the MITRE ATT&CK framework. We analyse four configurations utilising state-of-the-art tools, including the Threat Report ATT&CK Mapper (TRAM) and open-source Large Language Models (LLMs) such as Llama2. Our findings reveal significant challenges, including class imbalance, overfitting, and domain-specific complexity, which impede accurate technique extraction. To mitigate these issues, we propose a novel two-step pipeline: first, an LLM summarises the reports, and second, a retrained SciBERT model processes a rebalanced dataset augmented with LLM-generated data. This approach achieves an improvement in F1-scores compared to baseline models, with several attack techniques surpassing an F1-score of 0.90. Our contributions enhance the efficiency of web-based CTI systems and support collaborative cybersecurity operations in an interconnected digital landscape, paving the way for future research on integrating human-AI collaboration platforms.

Figures

Figures reproduced from arXiv: 2505.03147 by the authors.

Figure 1
Figure 1. Our evaluation methodology uses four configurations. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance of TRAM using different Configura [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 16 canonical work pages

  1. [1]

    Mohan Baruwal Chhetri, Shahroz Tariq, Ronal Singh, Fatemeh Jalalvand, Cecile Paris, and Surya Nepal. 2024. Towards human-ai teaming to mitigate alert fatigue Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using LLMs in security operations centres. ACM Transactions on Internet Technology 24, 3 (2024), 1–22

  2. [2]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)

  3. [3]

    Center for Threat-Informed Defense. 2024. Adversary Emulation Li- brary. https://github.com/center-for-threat-informed-defense/adversary_ emulation_library Accessed: 2024-06-12

  4. [4]

    Center for Threat-Informed Defense. 2024. TRAM: Threat Report ATT&CK Map- ping. https://github.com/center-for-threat-informed-defense/tram Accessed: 2024-06-12

  5. [5]

    M Baruwal Chhetri, S Tariq, R Singh, F Jalalvand, C Paris, S Nepal, AA Vali, S Azizi, M Shojafar, S Saryazdi, et al. 2024. Internet Technology. ACM Transactions on 24, 3 (2024)

  6. [6]

    ClearSky Cyber Security. 2016. Operation DustySky . Technical Re- port. https://www.clearskysec.com/wp-content/uploads/2016/01/Operation% 20DustySky_TLP_WHITE.pdf Accessed: 2024-06-12

  7. [7]

    Jessica Irons, Patrick Cooper, Melanie McGrath, Shahroz Tariq, and Andreas Duenser. 2024. Towards a criteria-based approach to selecting human-AI inter- action mode. arXiv preprint arXiv:2411.07406 (2024)

  8. [8]

    Fatemeh Jalalvand, Mohan Baruwal Chhetri, Surya Nepal, and Cecile Paris. 2024. Alert Prioritisation in Security Operations Centres: A Systematic Survey on Criteria and Methods. Comput. Surveys 57, 2 (2024), 1–36

Show all 22 references
  1. [9]

    Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022. AttacKG: Con- structing technique knowledge graph from cyber threat intelligence reports. In European Symposium on Research in Computer Security . Springer, 589–609

  2. [10]

    Dongliang Mu, Alejandro Cuevas, Limin Yang, Hang Hu, Xinyu Xing, Bing Mao, and Gang Wang. 2018. Understanding the reproducibility of crowd-reported security vulnerabilities. In 27th USENIX Security Symposium (USENIX Security 18). 919–936

  3. [11]

    National Institute of Standards and Technology (NIST). 2024. Threat Intelligence - Glossary Term. https://csrc.nist.gov/glossary/term/threat_intelligence Accessed: 2024-06-12

  4. [12]

    NewBee119. 2024. Attack Technique Dataset. https://github.com/NewBee119/ Attack-Technique-Dataset Accessed: 2024-06-12

  5. [13]

    Cécile Paris and Andrew Reeson. 2024. What’s the Secret to Making Sure AI Does Not Steal Your Job? Work with It, Not Against It. In The Conversation on Work, Ian O. Williamson (Ed.). Johns Hopkins University Press, Baltimore, 177–181. First published in The Conversation on Nov...

  6. [14]

    Paweł Pawlinski, Przemylaw Jaroszewski, Piotr Kijewski, Lukasz Siewierski, Pawel Jacewicz, Przemyslaw Zielony, and Radoslaw Zuber. 2014. Actionable information for security incident response. European Union Agency for Network and Information Security, Heraklion, Greece (2014)

  7. [15]

    Md Rayhanur Rahman, Rezvan Mahdavi Hezaveh, and Laurie Williams. 2023. What are the attackers doing now? Automating cyberthreat intelligence extrac- tion from text on pace with the changing threat landscape: A survey. Comput. Surveys 55, 12 (2023), 1–36

  8. [16]

    Emma Schleiger, Claire Mason, Claire Naughtin, Andrew Reeson, and Cecile Paris. 2024. Collaborative Intelligence: A scoping review of current applications. Applied Artificial Intelligence 38, 1 (2024), 2327890

  9. [17]

    BinHui Tang, JunFeng Wang, Zhongkun Yu, Bohan Chen, Wenhan Ge, Jian Yu, and TingTing Lu. 2022. Advanced Persistent Threat intelligent profiling technique: A survey. Computers and Electrical Engineering 103 (2022), 108261

  10. [18]

    Shahroz Tariq, Mohan Baruwal Chhetri, Surya Nepal, and Cecile Paris. 2025. Alert Fatigue in Security Operations Centres: Research Challenges and Opportunities. ACM Comput. Surv. 57, 9, Article 224 (April 2025), 38 pages. https://doi.org/10. 1145/3723158

  11. [19]

    Shahroz Tariq, Mohan Baruwal Chhetri, Surya Nepal, and Cecile Paris. 2024. A2C: A Modular Multi-stage Collaborative Decision Framework for Human-AI Teams. arXiv preprint arXiv:2401.14432 (2024)

  12. [20]

    Shahroz Tariq, Ronal Singh, Mohan Baruwal Chhetri, Surya Nepal, and Cecile Paris. 2025. Bridging Expertise Gaps: The Role of LLMs in Human-AI Collabora- tion for Cybersecurity. arXiv (2025)

  13. [21]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

  14. [22]

    Trend Micro. 2021. Cybersecurity Tool Sprawl Drives Plans to Outsource Detection and Response. https://newsroom.trendmicro.com/2021-10-12- Cybersecurity-Tool-Sprawl-Drives-Plans-to-Outsource-Detection-and- Response Accessed: 2024-06-12

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.