REVIEW 4 major objections 5 minor 1 cited by
Cyber-Attack Technique Classification Using Two-Stage Trained Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that a two-stage training recipe with similarity-selected auxiliary data improves Macro-F1 by 5 to 9 percentage points for classifying cyber threat intelligence sentences into MITRE ATT&CK techniques, while keeping…
desk verdict A competent empirical recipe for low-resource classification, but the similarity-selection step may leak test-set information and needs clarification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a two-stage fine-tuning procedure with similarity-based auxiliary data selection. For each minority class, the method computes the cosine similarity between each auxiliary sentence embedding and each primary sentence embedding from a pretrained BERT, and adds the top auxiliary sentences until the class reaches k samples (k=10). The augmented set is used in the first training stage, and the second stage continues training on the unmodified primary data. The selection step limits distribution shift introduced by the auxiliary data, and the second stage steers the model back toward the primary distribution while preserving the rare-class supervision acquired in the first stage. This recipe is model-agnostic and is demonstrated with CTI-BERT, a BERT re-pretrained from scratch on a cybersecurity corpus, as well as with general and other security-domain BERT variants.
What would settle it
Run the proposed DP + DA_sim → DP recipe on the original unpruned TRAM dataset without dropping the 191 low-resource classes, and compare Macro-F1 against a primary-data-only baseline; if the two-stage model fails to beat the baseline on that full benchmark, the paper's central claim is falsified.
Extended reading notes
Core claim
The paper discovers that the combination of cosine-similarity-based auxiliary selection and two-stage training (denoted DP + DA_sim → DP) yields consistent Macro-F1 gains on the TRAM benchmark after it is deduplicated and pruned to the 73 classes having at least three sentences. Adding all auxiliary data (DP + DA) raises Macro-F1 but lowers Micro-F1 because the MITRE descriptions are out-of-distribution relative to CTI sentences; continuing training on primary data alone recovers Micro-F1 and pushes Macro-F1 higher. The proposed recipe confines augmentation to minority classes (fewer than ten sentences) and selects the most similar auxiliary sentences per class by BERT embedding cosine similarity, producing the best Micro-F1 while matching the best Macro-F1 among the compared settings. The claim is stated as a 5 to 9 percentage point Macro-F1 improvement over the primary-data-only baseline (for CTI-BERT, from 47.3 to 56.1 Macro-F1).
Load-bearing premise
The claimed 5 to 9 point Macro-F1 gain is measured only on a rebuilt, pruned version of the TRAM dataset with 73 classes; if the 191 excluded classes or the original duplicated splits matter, the improvement may not extend to the full benchmark.
Editorial extensions
If this is right
- If the central claim is correct, the same two-stage recipe could be applied to other low-resource text classification tasks where same-label auxiliary descriptions exist, such as legal or medical document classification, without needing multi-task training.
- The finding implies that a short continuation pass on in-distribution data is a general corrective for distribution shift introduced by data augmentation, not a trick specific to cybersecurity.
- The ablation results suggest that for domain-specialized models, random auxiliary data can be nearly as effective as similarity-selected data, so the similarity scoring may be unnecessary when the base model is already security-aware.
- The pruned, deduplicated 73-class TRAM split could become a standard evaluation set for sentence-level attack technique classification, replacing the original duplicated splits.
Reading between the lines
- The paper's evaluation drops 191 of the original 264 TRAM classes, so a natural extension is to test the recipe on the full label set to see whether the Macro-F1 gain survives extreme imbalance and classes with no auxiliary match.
- The similarity selection is done with a base BERT; using the final CTI-BERT itself or a dedicated sentence-transformer to rank auxiliary sentences might further improve selection, which the paper does not test.
- For operational use, the sentence-level classifier could be paired with an event extractor to produce structured attack evidence for analysts; the paper stops at classification accuracy.
- Because the second stage uses only primary data, the recipe could be combined with any augmentation scheme on the primary data, such as back-translation, to push performance further.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage fine-tuning method for low-resource, class-imbalanced cyber-attack technique classification. Minority classes in a primary dataset (TRAM CTI sentences) are augmented with cosine-similarity-selected sentences from an auxiliary dataset (MITRE ATT&CK technique descriptions), the model is first fine-tuned on the augmented data, and then fine-tuned further on the primary data alone. Experiments across four language models (BERT, SecBERT, SecureBERT, and a domain-specific CTI-BERT) report Macro-F1 gains of 5 to 9 percentage points over a primary-only baseline while keeping Micro-F1 competitive, based on a preprocessed 73-class subset of TRAM.
Significance. If the result holds, the two-stage recipe is a simple, model-agnostic way to exploit same-label auxiliary data in low-resource settings, with potential application beyond cybersecurity. The paper deserves credit for a clear motivating problem, a straightforward method, and a fairly extensive set of ablations: it compares similar vs. random auxiliary selection, augmentation of minority vs. all classes, and auxiliary data vs. primary-data oversampling, across four model families including three security-domain LMs. The direction of the Macro-F1 improvement is consistent across all four models, and the reported average standard deviation (0.014) suggests moderate stability. However, the evaluation is limited to 73 of the original 264 TRAM classes, no significance tests are reported, and the paper leaves unresolved a potential leakage mechanism in the similarity-based data selection.
major comments (4)
- [Section 2.1, Eq. (1); Section 3.1] The definition of DP_c in Eq. (1) and the dataset description in Section 3.1 leave unresolved whether the cosine similarity for selecting DA_sim is computed against the full preprocessed TRAM data (including the development and test splits) or only the training split. Section 3.1 states "We use the pre-processed TRAM data as DP" after describing a 2:1:1 split, so the literal reading includes test sentences in DP_c. If the selection uses test-set text, the auxiliary sentences added in Stage 1 are chosen to be similar to test sentences, which is a form of feature-side test leakage; it would give DA_sim an advantage over DA_rand and could inflate the reported 5-9 point Macro-F1 gains. The authors must specify the exact subset used and, if test features were used, rerun the experiments with similarity computed only on the training split.
- [Section 3.1, Table 2] The evaluation is restricted to 73 of the original 264 TRAM classes because all classes with fewer than three examples are removed. This pruning excludes the most extreme low-resource classes, which are the primary target of the proposed method, and the paper's abstract and conclusions present the improvement as being on "the TRAM dataset" without this caveat. The authors should report results on the full 264-class set (or a clearly described subsample with all classes) or explicitly limit the claim to the 73-class subset.
- [Section 4, Table 3] Only five-seed averages and the average standard deviation (0.014) are reported; no significance tests are provided. Several key comparisons are small (e.g., CTI-BERT Micro-F1 71.3 vs 70.9 for rows (6) and (4)), so the reported differences may not be statistically reliable. The authors should report per-configuration standard deviations and paired significance tests for the primary comparisons (proposed vs DP-only and proposed vs DP+DA->DP).
- [Section 2.3 and Figure 1] The description of the second stage is inconsistent: Section 2.3 says the model is "trained more on DP only", while the Figure 1 caption says "with only DA_sim at the second stage". The paper also never clarifies whether "DP" as the training data means the entire preprocessed dataset or the DP Train split. This ambiguity is related to the leakage concern and must be corrected.
minor comments (5)
- [Figure 1] The caption says "then with only DA_sim at the second stage", which should read "then with only DP at the second stage".
- [Section 4.3] The claim that row (6) has "competitive Macro-F1" compared to row (4) is not well supported for SecBERT (48.2 vs 50.4) and CTI-BERT (56.1 vs 58.0); the drops are larger than the average standard deviation and should be addressed.
- [Section 3.2] The text "hyper-parameters are selected base on the validation split" contains a typo; it should be "based on".
- [Section 3.1] The criterion "remove the classes with fewer than three examples" should specify whether this count is based on the full 1,491 sentences or on each split individually.
- [General] The paper does not mention code or data availability; releasing the preprocessing script to reconstruct the 73-class split would aid reproducibility.
Circularity Check
No significant circularity: the two-stage training gains are empirical and replicated across base models; the only self-referential element is the self-cited CTI-BERT, which is not load-bearing for the central claim.
full rationale
The paper's central claim is empirical: adding same-label MITRE descriptions to rare classes and then continuing training on TRAM improves Macro-F1. The result is measured against a held-out test split (Table 3) and is replicated with four different base models (BERT, SecBERT, SecureBERT, CTI-BERT), so it does not depend on any equation that reduces to a fitted value. The similarity selection in Eq. (1) is a data-selection heuristic, and the paper's own ablations show it is not even necessary for all models (DA_rand performs comparably for the security-domain models), so the two-stage contribution is not definitionally tied to the similarity criterion. The only self-referential element is CTI-BERT: it is pretrained by the same authors (Park and You, 2023) on a corpus that includes MITRE datasets, and it is used both as the similarity encoder and as one of the fine-tuning bases. This self-citation is not load-bearing for the main two-stage claim, which holds for the non-self-cited BERT baseline as well. One ambiguity worth noting is that Section 3.1 says "We use the pre-processed TRAM data as DP" after defining train/dev/test splits, while Eq. (1) is written over DP_c; if DP_c in Eq. (1) were read as including the test split, the auxiliary selection would leak test features into training. The paper does not state that reading explicitly, and standard practice would restrict DP_c to the training split, so this is a clarity/correctness risk rather than a demonstrated circularity. Overall, no prediction in the paper is equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (2)
- Minority-class threshold k =
10
- Training hyperparameters =
50 epochs, lr=2e-5, batch=16, warmup=1000, weight decay=0.01
assumptions (4)
- domain assumption The primary and auxiliary datasets share the same label set and domain (TRAM sentences and MITRE descriptions both describe ATT&CK techniques).
- domain assumption Cosine similarity of sentence embeddings from a pretrained BERT model is a valid measure of whether a MITRE description will help a TRAM minority class.
- ad hoc to paper The pruned 73-class TRAM split is a fair and representative evaluation environment for attack technique classification.
- domain assumption CTI-BERT's pretraining corpus, which includes MITRE datasets, does not contain the TRAM test sentences or their source reports.
Cite this review
Pith. "Pith review of Cyber-Attack Technique Classification Using Two-Stage Trained Large Language Models." pith.science (2026). https://pith.science/paper/R5E54NV6
@misc{pith2026241118755,
author = {Pith},
title = {Pith review of: Cyber-Attack Technique Classification Using Two-Stage Trained Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/R5E54NV6}},
note = {Machine review of arXiv:2411.18755}
}
read the original abstract
Understanding the attack patterns associated with a cyberattack is crucial for comprehending the attacker's behaviors and implementing the right mitigation measures. However, majority of the information regarding new attacks is typically presented in unstructured text, posing significant challenges for security analysts in collecting necessary information. In this paper, we present a sentence classification system that can identify the attack techniques described in natural language sentences from cyber threat intelligence (CTI) reports. We propose a new method for utilizing auxiliary data with the same labels to improve classification for the low-resource cyberattack classification task. The system first trains the model using the augmented training data and then trains more using only the primary data. We validate our model using the TRAM data1 and the MITRE ATT&CK framework. Experiments show that our method enhances Macro-F1 by 5 to 9 percentage points and keeps Micro-F1 scores competitive when compared to the baseline performance on the TRAM dataset.
Figures
Forward citations
Cited by 1 Pith paper
-
SynthCTI: LLM-Driven Synthetic CTI Generation to enhance MITRE Technique Mapping
A clustering-guided LLM data augmentation pipeline raises macro-F1 for MITRE technique classification, e.g., ALBERT from 0.35 to 0.52 and SecureBERT to 0.66, across two CTI datasets.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Ehsan Aghaei, Xi Niu, Waseem Shadid, and Ehab Al-Shaer. 2023. https://doi.org/10.1007/978-3-031-25538-0_3 SecureBERT: A Domain-Specific Language Model for Cybersecurity , pages 39--56
-
[4]
Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2023. Looking beyond iocs: Automatically extracting attack patterns from external cti. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID '23, page 92–108. Association for Computing Machinery
work page 2023
-
[5]
Robert Bridges, Kelly Huffer, Corinne Jones, Michael Iannacone, and John Goodall. 2017. https://doi.org/10.1109/ICMLA.2017.0-122 Cybersecurity automated information extraction techniques: Drawbacks of current methods, and enhanced extractors . pages 437--442
-
[6]
Francisco Charte, Antonio J. Rivera, María J. del Jesus , and Francisco Herrera. 2015. https://doi.org/https://doi.org/10.1016/j.knosys.2015.07.019 Mlsmote: Approaching imbalanced multilabel learning through synthetic instance generation . Knowledge-Based Systems, 89:385--397
-
[7]
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. Smote: Synthetic minority over-sampling technique. J. Artif. Int. Res., 16(1):321–357
work page 2002
-
[8]
Hannah Chen, Yangfeng Ji, and David Evans. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.426 Finding F riends and flipping frenemies: Automatic paraphrase dataset augmentation using graph theory . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4741--4751, Online. Association for Computational Linguistics
Show all 38 references
-
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 a . Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805
2019 arXiv
-
[10]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 b . https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Assoc...
2019 doi
-
[11]
Demi Guo, Yoon Kim, and Alexander Rush. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.447 Sequence-level mixed sample data augmentation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5547--5552, Online. Associatio...
2020 doi
-
[12]
Jeremy Howard and Sebastian Ruder. 2018. https://doi.org/10.18653/v1/P18-1031 Universal language model fine-tuning for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328--339, Melb...
2018 doi
-
[13]
Ghaith Husari, Ehab Al - Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017. https://doi.org/10.1145/3134600.3134646 Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of CTI sources . In Proceedings of the 33rd Annual Computer Security Applic...
2017
-
[14]
jackaduma. 2022. https://github.com/jackaduma/SecBERT/ Secbert . Accessed: 2023-10-10
2022
-
[15]
Sosuke Kobayashi. 2018. https://doi.org/10.18653/v1/N18-2072 Contextual augmentation: Data augmentation by words with paradigmatic relations . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language ...
2018 doi
-
[16]
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. 2019. https://doi.org/10.18653/v1/N19-1363 Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation . In Proceedings of the 2019 Conference of the North A merican Cha...
2019 doi
-
[17]
Po-Nien Kung, Sheng-Siang Yin, Yi-Cheng Chen, Tse-Hsuan Yang, and Yun-Nung Chen. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.34 Efficient multi-task auxiliary learning: Selecting auxiliary data by feature similarity . In Proceedings of the 2021 Conference on Empirical Me...
2021 doi
-
[18]
Valentine Legoy, Marco Caselli, Christin Seifert, and Andreas Peter. 2020. http://arxiv.org/abs/2004.14322 Automated retrieval of att & ck tactics and techniques for cyber threat reports . CoRR, abs/2004.14322
2020 arXiv
-
[19]
Levenshtein
Vladimir I. Levenshtein. 1965. https://api.semanticscholar.org/CorpusID:60827152 Binary codes capable of correcting deletions, insertions, and reversals . Soviet physics. Doklady, 10:707--710
1965
-
[20]
Xingjian Li, Haoyi Xiong, Haozhe An, Chengzhong Xu, and Dejing Dou. 2020. http://arxiv.org/abs/2007.10252 Xmixup: Efficient transfer learning with auxiliary samples by cross-domain mixup
2020 arXiv
-
[21]
Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022. Attackg: Constructing technique knowledge graph from cyber threat intelligence reports. In Computer Security – ESORICS 2022 - 27th European Symposium on Research in Computer Security, Proceedings, pages 589--609. Spring...
2022
-
[22]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692
2019 arXiv
-
[23]
Amirreza Niakanlahiji, Lida Safarnejad, Reginald Harper, and Bei - Tseng Chu. 2019. https://doi.org/10.1109/BigData47090.2019.9006562 Iocminer: Automatic extraction of indicators of compromise from twitter . In IEEE International Conference on Big Data , pages 4747--4754. IEEE
2019
-
[24]
Youngja Park and Taesung Lee. 2022. https://doi.org/10.18653/v1/2022.emnlp-industry.54 Full-stack information extraction system for cybersecurity intelligence . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: EMNLP 2022 - Industry Tra...
2022 doi
-
[25]
Youngja Park and Weiqiu You. 2023. https://doi.org/10.18653/v1/2023.emnlp-industry.12 A pretrained language model for cyber threat intelligence . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 113--122, Singapor...
2023 doi
- [26]
-
[27]
Carl Sabottke, Octavian Suciu, and Tudor Dumitras. 2015. https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/sabottke Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits . In 24th USENIX Secu...
2015
-
[28]
o zde G \
G \"o zde G \"u l S ahin and Mark Steedman. 2018. https://doi.org/10.18653/v1/D18-1545 Data augmentation via dependency tree morphing for low-resource languages . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5004--5009, Bruss...
2018 doi
-
[29]
Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. https://api.semanticscholar.org/CorpusID:6473756 A vector space model for automatic indexing . Commun. ACM, 18:613--620
1975
- [30]
-
[31]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1009 Improving neural machine translation models with monolingual data . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...
2016 doi
-
[32]
Hyejin Shin, WooChul Shim, Saebom Kim, Sol Lee, Yong Goo Kang, and Yong Ho Hwang. 2021. https://doi.org/10.1145/3442381.3449797 \# twiti: Social listening for threat intelligence . In WWW '21: The Web Conference , pages 92--104. ACM / IW3C2
2021
-
[33]
Wiem Tounsi and Helmi Rais. 2017. https://doi.org/10.1016/j.cose.2017.09.001 A survey on technical threat intelligence in the age of sophisticated cyber attacks . Computers & Security, 72
2017 doi
-
[34]
Katja Tuma, G \"u l Çalikli, and Riccardo Scandariato. 2018. Threat analysis of software systems: A systematic literature review. J. Syst. Softw., 144:275--294
2018
-
[35]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Neural Informa...
2017
-
[36]
Jason Wei and Kai Zou. 2019. https://doi.org/10.18653/v1/D19-1670 EDA : Easy data augmentation techniques for boosting performance on text classification tasks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International...
2019 doi
-
[37]
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. https://proceedings.neurips.cc/paper/2020/file/44feb0096faa8326192570788b38c1d1-Paper.pdf Unsupervised data augmentation for consistency training . In Advances in Neural Information Processing Systems, volume ...
2020
-
[38]
Dauphin, and David Lopez-Paz
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. https://openreview.net/forum?id=r1Ddp1-Rb mixup: Beyond empirical risk minimization . In International Conference on Learning Representations
2018
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.