REVIEW 3 major objections 5 minor 27 references
This paper establishes that a supervised classifier trained on a curated gold set of 1,207 expert-mapped CVEs roughly doubles recall@5 over a zero-shot baseline, and that LLM-generated labels at about 0.39 agreement with experts provide no
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 02:02 UTC pith:YUQHASWB
load-bearing objection A rigorous small-data CVE-to-ATT&CK pipeline with a genuinely useful gold dataset and an honest evaluation-noise post-mortem, but the headline LLM-expansion null is confounded by vocabulary growth and needs a projection check before it is taken as established. the 3 major comments →
Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is a null result with a positive core: a multi-label classifier trained on a small, curated gold set of 1,207 expert-mapped CVEs is the best available artifact, and adding LLM-labeled rows at about 0.39 agreement with expert labels does not help. Under the corrected protocol, gold-only recall@5 is 0.673±0.019; adding 297 LLM rows drops it to 0.655 and 984 rows to 0.651, while macro-F1—the proxy for rare-technique coverage—falls from 0.177 to 0.151 at the largest size. The same corrected protocol shows that every metric improves monotonically as more curated rows are added, so the classifier is label-quality bound rather than data bound. The paper also identifies the mechani
What carries the argument
Three components carry the argument. First, a provenance-tiered gold dataset that keeps expert labels strictly separate from weak automatically derived labels, so training targets are never contaminated. Second, a multi-label transformer classifier with inverse-frequency loss weighting and a parent-technique vocabulary. Third, a validate-before-expand labeling pipeline that measures the LLM's agreement with expert mappings before any expansion, and—decisively—a corrected evaluation protocol that selects checkpoints on a validation split rather than the test split. The corrected protocol is the mechanism that stabilizes the three-verdict history into a reproducible null result.
Load-bearing premise
The expert-annotated mappings are treated as correct and exhaustive ground truth for training targets and the fixed test yardstick; the paper itself concedes that a technique absent from a record is not necessarily a true negative, so systematic gaps—especially among rare techniques—would bias the agreement figures, the recall numbers, and the observed macro-F1 degradation.
What would settle it
Re-run the decisive expansion contrast under the corrected protocol with the same 984 LLM-labeled rows; if recall@5 on the untouched test split exceeds the gold-only mean by more than two standard errors, the null verdict fails. Alternatively, have a fresh expert panel re-annotate the 121 held-out CVEs; if the revised gold labels add many rare techniques the original experts missed, the yardstick itself is biased and the 0.39 agreement and macro-F1 drop become unreliable.
If this is right
- A supervised model trained on roughly 1,000 curated examples roughly doubles recall@5 over a zero-shot semantic-similarity baseline and improves every ranking metric.
- LLM labels at about 0.39 expert agreement provide no reliable ranking gain at expansion sizes 100–984 CVEs; at the largest size, macro-F1 drops by about 0.04, so rare techniques get worse coverage.
- The deterministic CWE→CAPEC→ATT&CK derivation chain is not usable as a training target: table-expansion artifacts stamp a few spurious techniques onto half the corpus, with a median of eight techniques per CVE.
- The classifier is label-quality bound, not data bound: nested growth of curated rows improves every metric, while LLM rows flatten or degrade them.
- Evaluation noise from checkpoint selection on a small test split can invert conclusions; multi-seed reporting alone is insufficient, and a dedicated validation split plus replication is required.
Where Pith is reading between the lines
- The paper's three-verdict history, taken further, implies that any small-data label-expansion study should adopt validation-split checkpoint selection and independent replication before reporting a positive effect; otherwise sign flips may be misread as findings.
- A testable extension suggested by the paper's own mechanism: restricting LLM expansion to high-confidence slots, or stratifying sampled CVEs by weakness class, may clear the agreement bar and produce gains the random-sample expansion did not.
- Because the scaling curve is still rising at 972 curated rows, a coordinated community dataset of reviewed mappings—with provenance and rationale—is a more credible route to improvement than further LLM labeling, though the paper only calls for such an effort.
- The same corrected protocol could be reused to benchmark human-in-the-loop silver labeling, where expert review of a subset of LLM rows might yield cheap labels that clear the quality threshold.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a reproducible pipeline for mapping CVE vulnerability descriptions to MITRE ATT&CK Enterprise techniques. The authors assemble a gold dataset of 1,207 CVEs from MITRE CTID expert mappings, train a multi-label RoBERTa classifier with a 57-parent-technique vocabulary, and benchmark it against a zero-shot embedding-similarity baseline. They then study whether LLM-generated labels (at ~0.39 micro-F1 agreement with experts) can extend the gold training set. After a three-round experimental investigation, they conclude that LLM expansion provides no reliable improvement at any size tested and degrades rare-technique coverage at ~1,000 added CVEs, and attribute an earlier apparent gain to evaluation noise from checkpoint selection on a small test split. The paper also quantifies noise in the CWE-to-CAPEC-to-ATT&CK derivation chain and shows a corrected protocol with validation-split checkpoint selection.
Significance. If the results hold, the paper provides a strong, reproducible argument that cheap LLM labels at this agreement level are not a substitute for expert curation in CVE-to-ATT&CK mapping, and it delivers a valuable methodological caution about small-data multi-seed comparisons and checkpoint selection. The manuscript is unusually rigorous for a small-data ML study: matched training conditions, five seeds per configuration, an independent replication, a nested scaling sweep, a second-encoder robustness check, a corrected protocol with re-runs, and fully released datasets, code, and trainer logs. These strengths make the paper a useful reference for practitioners building vulnerability-to-technique classifiers and for researchers designing label-expansion studies.
major comments (3)
- [§6.3 / Table 6 / §7] The central negative claim—LLM expansion degrades rare-technique coverage (macro-F1 0.177→0.151 in Table 6)—is confounded by label-vocabulary growth. The paper states (§7) that "folding LLM rows into training also enlarges the label vocabulary, so the union and gold-only models are not graded on an identical label space." New techniques introduced by LLM rows that never appear in the gold test split are guaranteed zero-F1 classes and can enter top-k rankings, mechanically depressing macro-F1 and recall@5. For example, adding 10 such classes to a 57-class space would lower a uniform macro-F1 of 0.177 to ~0.151, almost exactly the reported drop. The dismissal as "inherent to the treatment" is not satisfactory because the label-quality question can be isolated by projecting LLM labels onto the gold-only 57-parent vocabulary (or freezing the classification head) before training. The authors
- [Figure 4 / §6.3] The "Gold labels scale; LLM labels do not" contrast is not drawn on equal terms. The gold-scaling arm freezes the label vocabulary and test set, while the LLM arm does not. The solid and dashed series in Figure 4 compare apples to oranges: the LLM arm's output space is larger, so its lower macro-F1 and recall@5 could be partly an artifact of the added zero-test-positive classes. The conclusion "the classifier is label-quality bound, not data bound" is therefore not yet supported by this overlay. The same projection-to-gold-vocabulary control needed for Table 6 should be applied here before the two curves are compared.
- [Table 5 / scaling sweep] The expansion-size sweep does not report how the label vocabulary changes with N (100, 300, 600, 984). Because the min-example threshold of five is applied to the union training data, the number of output classes is a moving target across the sweep. Without knowing the vocabulary size at each N—or holding the vocabulary fixed—the monotonic declines in macro-F1 and recall@5 cannot be attributed to label quality rather than to an expanding output space that adds only zero-test-positive classes. Please report vocabulary size per N and, ideally, re-run the sweep with the vocabulary projected to the gold-only set.
minor comments (5)
- [§6.2] The bullet list says "Two findings emerge" but then presents three numbered findings. Change to "Three findings emerge" or merge two of them.
- [Table 3 caption] The caption says "Retained as a cautionary example." It would help to explicitly state that this table was measured under the pre-correction protocol (test-split checkpoint selection), so readers do not confuse it with the corrected numbers of record.
- [Listing 1] The caption states the output is abridged to the top five rows, but the text below refers to ranks 17 and 32. Consider noting in the caption that the full ranking is available in the released logs, or showing the tail ranks in the listing itself.
- [§1 / Pre-preprint notice] The phrase "Pre-preprint notice" is unusual. If this is a preprint server requirement, fine; otherwise consider simplifying to "Notice".
- [Reference formatting] Reference [2] contains a raw URL broken across lines with a DOI after it. Add a stable DOI or URL formatting to avoid a broken hyperlink.
Circularity Check
No significant circularity: the gold test yardstick is external (CTID) and held out; the LLM-expansion null was empirically reversed twice before stabilizing; the paper's own admitted label-vocabulary mismatch is a disclosed validity threat, not a by-construction derivation.
full rationale
The derivation chain is self-contained against external, independently produced evidence and its central null result is an out-of-sample empirical finding, not a construction. (i) Training targets and the test yardstick are expert CTID mappings that the authors explicitly say they did not author: 'We do not annotate any CVE ourselves — our contribution at this stage is purely to fetch, normalise, merge, and split existing expert mappings.' The 121-CVE test split is fixed and 'left entirely untouched' when LLM rows are folded into the training split, so no headline metric is scored on data used to fit it. (ii) The decisive LLM-expansion verdict was not forced: the same comparison first looked harmful (Table 3), then helpful (Table 4), and only an independent 984-row replication plus a size sweep produced the null — a trajectory that could have gone the other way, and in fact did at the five-seed stage. (iii) The checkpoint-selection bias is disclosed and corrected: 'checkpoint selection now uses a validation split carved from the gold training data (the test split is evaluated exactly once, for the final report),' and the decisive contrast (Table 6) is re-run under that corrected protocol. (iv) Self-citations (VulnTrain [2], VLAI [3], GCVE [13]) are tooling or adjacent-task work, not load-bearing; the roberta-base choice that 'follows our earlier VLAI model' is explicitly stress-tested against ModernBERT (Table 7) and the conclusion replicates. The one factor that deserves scrutiny is the manuscript's own admitted limitation in §7: 'folding LLM rows into training also enlarges the label vocabulary, so the union and gold-only models are not graded on an identical label space.' Because new output classes with no gold-test positives can mechanically depress macro-F1 and occupy top-k slots, part of the headline 'macro-F1 decreases by 0.04' / recall@5 decline could reflect vocabulary growth rather than label noise, and the abstract's causal attribution 'the root cause is evaluation noise' is an interpretation layered on that comparison. This is a disclosed validity threat to the causal attribution, not a by-construction equivalence: no equation in the paper forces the union model to be worse, the paper answers the practical question it explicitly poses, and the ModernBERT replication keeps the direction stable. A fully vocabulary-frozen LLM arm (projecting LLM labels onto the 57-parent gold vocabulary) would isolate label quality from added classes; its absence
Axiom & Free-Parameter Ledger
free parameters (5)
- Minimum technique-example threshold =
5 (default)
- Per-label positive-weight clipping ratio =
unspecified (clipped)
- Validation split fraction (corrected protocol) =
0.1
- LLM labeler configuration =
qwen3.5:122b, assertive prompt, single call
- Training hyperparameters =
lr 1e-5, 40 epochs, AdamW, seeds 42-46
axioms (5)
- domain assumption CTID expert mappings are valid ground truth for the mapping task
- domain assumption The gold test split (121 CVEs, 119 in-vocabulary) is a stable yardstick
- domain assumption Revoked-by normalization of ATT&CK identifiers preserves semantic equivalence across versions
- standard math Standard ML machinery works as expected on CVE text
- domain assumption The zero-shot SMET-style baseline faithfully represents prior unsupervised mapping approaches
read the original abstract
We present a reproducible pipeline for mapping Common Vulnerabilities and Exposures (CVEs) to MITRE ATT&CK Enterprise techniques from free-text vulnerability descriptions. Rather than relying on the CWE->CAPEC->ATT&CK derivation chain, whose table-expansion artifacts we quantify, we train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings. The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline and improves every ranking metric. We then investigate whether LLM-assisted labeling can extend the gold dataset. Initial experiments suggest contradictory conclusions: a single run indicates degraded performance, while averaging over five random seeds suggests a small gain. However, an independent replication and an expansion-size study (100--984 additional CVEs) show that the apparent improvement is an evaluation artifact. LLM-generated labels, with approximately 0.39 agreement with expert annotations, provide no reliable improvement at any expansion size and reduce rare-technique coverage at around 1,000 added CVEs (macro-F1 decreases by 0.04). The root cause is evaluation noise. Selecting checkpoints on a small test split effectively maximizes over many noisy evaluations, producing recall@5 differences of up to 0.05 between otherwise identical runs. Using a corrected protocol based on validation-split checkpoint selection, the gold-only model achieves recall@5 of (0.673 \pm 0.019), and repeating the decisive experiment confirms the null result for LLM expansion. A final scaling study shows that additional expert-curated data consistently improves performance, whereas LLM-labeled data does not, indicating that the classifier is limited by label quality rather than dataset size. All datasets, models, code, and training logs are publicly released.
Figures
Reference graph
Works this paper leans on
-
[1]
Basel Abdeen, Ehab Al-Shaer, Anoop Singhal, Latifur Khan, and Kevin W. Hamlen. SMET: Semantic mapping of CVE to ATT&CK and its application to cybersecurity. InData and Applications Security and Privacy XXXVII (DBSec 2023), Lecture Notes in Computer Science, pages 243–260. Springer, 2023. 17
2023
-
[2]
GPU efficiency in VLAI model training
C ´edric Bonhomme. GPU efficiency in VLAI model training. Technical report,https://www. vulnerability-lookup.org/files/news/2025/12/GPU_Efficiency_VLAI_Model_ Training_2025.pdf, 2025. DOI: 10.13140/RG.2.2.24233.04963
arXiv 2025
-
[3]
C ´edric Bonhomme and Alexandre Dulaunoy. VLAI: A RoBERTa-based model for automated vulnerability severity classification.arXiv preprint arXiv:2507.03607, 2025
Pith/arXiv arXiv 2025
-
[4]
Mapping ATT&CK to CVE for impact.https://github.com/ center-for-threat-informed-defense/attack_to_cve, 2021
Center for Threat-Informed Defense. Mapping ATT&CK to CVE for impact.https://github.com/ center-for-threat-informed-defense/attack_to_cve, 2021. MITRE Engenuity Center for Threat-Informed Defense
2021
-
[5]
TRAM: Threat report ATT&CK mapper.https://github.com/ center-for-threat-informed-defense/tram, 2023
Center for Threat-Informed Defense. TRAM: Threat report ATT&CK mapper.https://github.com/ center-for-threat-informed-defense/tram, 2023
2023
-
[6]
ATT&CK mappings explorer.https:// center-for-threat-informed-defense.github.io/mappings-explorer/, 2024
Center for Threat-Informed Defense. ATT&CK mappings explorer.https:// center-for-threat-informed-defense.github.io/mappings-explorer/, 2024. In- cludes CVE (KEV) to ATT&CK mappings
2024
-
[7]
CodeCarbon: Estimate and track carbon emissions from machine learning computing
CodeCarbon contributors. CodeCarbon: Estimate and track carbon emissions from machine learning computing. https://github.com/mlco2/codecarbon, 2024
2024
-
[8]
vulnerability-attack-technique-classification- roberta-base (Revision 3b98a1d)
Computer Incident Response Center Luxembourg. vulnerability-attack-technique-classification- roberta-base (Revision 3b98a1d). Hugging Face,https://huggingface.co/CIRCL/ vulnerability-attack-technique-classification-roberta-base, 2026. DOI: 10.57967/hf/9623
-
[9]
vulnerability-attack-techniques-llm-scaling (Revision 0bf633c)
Computer Incident Response Center Luxembourg. vulnerability-attack-techniques-llm-scaling (Revision 0bf633c). Hugging Face,https://huggingface.co/datasets/CIRCL/ vulnerability-attack-techniques-llm-scaling, 2026. DOI: 10.57967/hf/9622
-
[10]
vulnerability-attack-technique-classification-roberta-base-llm-expanded (Revi- sion 9db8c37)
Computer Incident Response Center Luxembourg and C ´edric Bonhomme. vulnerability-attack-technique-classification-roberta-base-llm-expanded (Revi- sion 9db8c37). Hugging Face,https://huggingface.co/CIRCL/ vulnerability-attack-technique-classification-roberta-base-llm-expanded,
-
[11]
vulnerability-attack- techniques (Revision c2b5da0)
Computer Incident Response Center Luxembourg and C ´edric Bonhomme. vulnerability-attack- techniques (Revision c2b5da0). Hugging Face,https://huggingface.co/datasets/CIRCL/ vulnerability-attack-techniques, 2026. DOI: 10.57967/hf/9621
-
[12]
Known exploited vulnerabilities catalog.https: //www.cisa.gov/known-exploited-vulnerabilities-catalog, 2026
Cybersecurity and Infrastructure Security Agency (CISA). Known exploited vulnerabilities catalog.https: //www.cisa.gov/known-exploited-vulnerabilities-catalog, 2026
2026
-
[13]
Alexandre Dulaunoy. GCVE: A decentralized model for vulnerability identification, publication, and operational enrichment.arXiv preprint arXiv:2606.00856, 2026
Pith/arXiv arXiv 2026
-
[14]
CVE2CAPEC: Mapping CVE to CWE, CAPEC and MITRE ATT&CK.https://github.com/ Galeax/CVE2CAPEC, 2024
Galeax. CVE2CAPEC: Mapping CVE to CWE, CAPEC and MITRE ATT&CK.https://github.com/ Galeax/CVE2CAPEC, 2024
2024
-
[15]
CVE2ATT&CK: BERT-based map- ping of CVEs to MITRE ATT&CK techniques.Algorithms, 15(9):314, 2022
Octavian Grigorescu, Andreea Nica, Mihai Dascalu, and Razvan Rughinis ¸. CVE2ATT&CK: BERT-based map- ping of CVEs to MITRE ATT&CK techniques.Algorithms, 15(9):314, 2022
2022
-
[16]
Erik Hemberg, Jonathan Kelly, Michal Shlapentokh-Rothman, Bryn Reinstadler, Katherine Xu, Nick Rutar, and Una-May O’Reilly. Linking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform configurations for cyber hunting.arXiv preprint arXiv:2010.00533, 2020
Pith/arXiv arXiv 2010
-
[17]
RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019. 18
Pith/arXiv arXiv 1907
-
[18]
Ollama: Get up and running with large language models locally.https://ollama.com, 2026
Ollama. Ollama: Get up and running with large language models locally.https://ollama.com, 2026
2026
-
[19]
Sentence-BERT: Sentence embeddings using siamese BERT-networks
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019
2019
-
[20]
Strom, Andy Applebaum, Doug P
Blake E. Strom, Andy Applebaum, Doug P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. MITRE ATT&CK: Design and philosophy. Technical Report MP180360R1, The MITRE Corporation, 2020.https://attack.mitre.org
2020
-
[21]
CAPEC: Common attack pattern enumeration and classification.https://capec
The MITRE Corporation. CAPEC: Common attack pattern enumeration and classification.https://capec. mitre.org, 2026
2026
-
[22]
CVE: Common vulnerabilities and exposures.https://www.cve.org, 2026
The MITRE Corporation. CVE: Common vulnerabilities and exposures.https://www.cve.org, 2026
2026
-
[23]
CWE: Common weakness enumeration.https://cwe.mitre.org, 2026
The MITRE Corporation. CWE: Common weakness enumeration.https://cwe.mitre.org, 2026
2026
-
[24]
MITRE ATT&CK stix data (attack-stix-data).https://github.com/ mitre-attack/attack-stix-data, 2026
The MITRE Corporation. MITRE ATT&CK stix data (attack-stix-data).https://github.com/ mitre-attack/attack-stix-data, 2026. Enterprise ATT&CK; accessed 2026
2026
-
[25]
MiniLM: Deep self-attention distil- lation for task-agnostic compression of pre-trained transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distil- lation for task-agnostic compression of pre-trained transformers. InAdvances in Neural Information Processing Systems (NeurIPS), 2020
2020
-
[26]
Benjamin Warner, Antoine Chaffin, Benjamin Clavi ´e, Orion Weller, Oskar Hallstr¨om, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference....
Pith/arXiv arXiv 2024
-
[2026]
DOI: 10.57967/hf/9624
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.