Pith. sign in

REVIEW 1 major objections 5 minor 19 references

Multi-Class vs. Multi-Label BERT for CVE-to-CWE Mapping: How Taxonomy Structure Shapes the Errors

T0 review · 1 major / 5 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CWE taxonomy, not the model, drives classification errors

desk verdict Solid empirical study: taxonomy structure, not encoder choice, drives CVE-to-CWE errors. One methodological soft spot in the threshold validation. read the letter →

arxiv 2607.07573 v1 pith:MFEGFZXN submitted 2026-07-08 cs.LG cs.CR

Ana Schwengber Kelm , Christian Bockermann , Jörg Frochte This is my paper
classification cs.LGcs.CR
keywords CVECWEmulti-classclassificationmulti-labeltransformerBERTerroranalysistaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares two ways of training BERT-style models to automatically assign weakness categories to vulnerability reports: a single-answer formulation (multi-class) and a multiple-answer formulation (multi-label). Across three encoder variants and three label-space sizes, multi-class consistently outperforms multi-label, though the gap shrinks from 21 to 2 percentage points as the number of classes is reduced from 83 to 25. The paper's central finding comes from the error analysis: when the three encoders' confusion patterns are compared, they are nearly identical (Pearson r > 0.92), and the dominant misclassifications fall along parent-child or sibling relationships in the CWE hierarchy. This suggests that the residual errors are driven by the structure of the taxonomy itself—where adjacent nodes are textually near-identical—rather than by limitations of any particular model. A relaxed evaluation that credits the classifier for landing on the correct branch of the CWE tree, even if the exact node is wrong, raises macro-F1 from roughly 81% to 90%, indicating that standard strict metrics substantially understate how useful the classifiers are for practical triage.

What carries the argument

The central mechanism is the CWE hierarchy itself: a tree of weakness types where broad 'Pillar' categories and specific 'Variant' nodes coexist as valid classification targets. The paper shows that this mixed granularity creates systematic confusion at hierarchy-adjacent nodes, and introduces a hierarchy-relaxed macro-F1 metric that credits within-family predictions to reveal the true branch-level accuracy.

What would settle it

If the error patterns were found to diverge significantly across encoders (e.g., Pearson r < 0.70), or if dominant misclassifications did not correspond to parent-child or sibling relationships in the CWE tree, the claim that taxonomy structure drives errors would collapse.

Watch

Extended reading notes

Core claim

The dominant misclassification patterns in automated CVE-to-CWE mapping are structurally identical across three different transformer encoders (Pearson r > 0.92) and concentrate along parent-child and sibling edges in the CWE hierarchy. This means the errors are a property of the taxonomy's design, not the model. When evaluation forgives within-family confusions, macro-F1 rises from ~81% to ~90%, showing that classifiers are reaching the correct region of the taxonomy far more often than exact-match metrics suggest.

Load-bearing premise

The hierarchy-relaxed evaluation relies on manually defined family groupings of CWE nodes. The authors acknowledge this is a heuristic: alternative groupings would yield different scores, though a sensitivity analysis across four grouping variants produces relaxed macro-F1 between 84% and 91%, which they take as evidence of robustness.

Editorial extensions

If this is right

  • Automated vulnerability triage pipelines can rely on classifiers that reach ~90% branch-level accuracy, which is operationally sufficient for routing CVEs to the correct mitigation team even when the exact CWE identifier is wrong.
  • Improving classification on the hardest pairs (e.g., CWE-77/78) likely requires hierarchy-aware training objectives or better annotation guidelines, not a better encoder.
  • The default sigmoid threshold of 0.5 is suboptimal for multi-label CWE classification; a threshold of 0.9 nearly closes the gap to multi-class, suggesting multi-label models systematically over-predict at moderate confidence levels.
  • Domain-adaptive pretraining (CySecBERT) provides statistically significant gains primarily in the multi-label setting where per-class supervision is sparse, but the benefit diminishes under aggressive full fine-tuning in the multi-class setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. This paper compares multi-class (MC) and multi-label (ML) BERT formulations for CVE-to-CWE mapping using three encoders (BERT Base, SecureBERT, CySecBERT) across three nested label spaces (83, 47, 25 classes). The main findings are: (1) MC outperforms ML on macro-F1 across all settings, with the gap narrowing from ~21pp to ~2pp as the label space shrinks; (2) threshold tuning on the ML side closes the gap at 25 classes; (3) dominant misclassification patterns follow the CWE hierarchy and are shared across all three encoders (Pearson r > 0.92); and (4) a hierarchy-relaxed evaluation raises macro-F1 to ~90%, suggesting strict metrics understate branch-level quality. The paper is well-structured and addresses a genuine gap in the literature by directly comparing the two formulations under matched conditions.

Significance. The paper provides a careful empirical comparison that is missing from prior work, which typically commits to one formulation without examining the trade-off. The confusion-matrix correlation analysis across encoders (r > 0.92) is a strong and falsifiable finding that points to taxonomy structure as the primary error driver. The matched-data ablation (MC without undersampling) is a commendable control that strengthens the formulation-effect claim. The sensitivity analysis of family groupings (four variants, 84–91% range) is a reasonable robustness check on the hierarchy-relaxed metric. The practical implications for vulnerability triage pipelines are clearly articulated.

major comments (1)
  1. [§5, Threshold sensitivity and gap closure (leave-one-seed-out validation)] The leave-one-seed-out threshold validation does not properly separate threshold selection from test evaluation. Each of the five seeds produces an independent 80/20 stratified split of the same 249,867 records. For a given held-out seed, the probability that any test instance also appears in at least one of the other four seeds' test sets is 1 − (0.8)^4 ≈ 0.59. Thus roughly 59% of the held-out evaluation instances are present in the data used to select τ. The procedure therefore cannot rule out that τ = 0.9 is tuned to systematic properties of the shared test distribution rather than being a generalizable model property. This is load-bearing for the claim (Table 7, abstract) that threshold tuning 'closes the gap' between MC and ML. The fact that τ = 0.9 is selected in every fold provides some robustness evidence, and the single-parameter nature of the threshold limits overfitting risk,但
minor comments (5)
  1. [§4, Dataset] The paper states that 'fewer than 1%' of records have multiple CWE labels. Given that this is the primary motivation for the ML formulation, it would help to report the exact count and briefly discuss how this affects the reliability of ML evaluation.
  2. [§4, Evaluation] The family groupings are listed inline (e.g., 'Memory/Buffer: CWE-119, 120, 121, 122, 125, 787') but the full mapping for all families is not provided. A supplementary table would improve reproducibility.
  3. [Table 1] The SGD result (0.4% macro-F1, 81 dead classes) is striking but only briefly discussed. A sentence clarifying whether this is a known interaction or specific to this task would help readers.
  4. [Figure 2(b)] The threshold sensitivity plot shows macro-F1 increasing monotonically with τ up to 0.9. It would be useful to show values beyond 0.9 to confirm that performance does not continue to increase, which would strengthen the claim that 0.9 is an optimum rather than an endpoint.
  5. [§6, Limitations] The paper acknowledges that the threshold analysis is post-hoc on the test set, but the leave-one-seed-out procedure is presented as addressing this concern. Given the overlap issue (see major comment), the limitations paragraph should be revised to accurately characterize what the validation does and does not demonstrate.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and the constructive assessment. The referee raises one major methodological concern regarding potential leakage between threshold-selection and test-evaluation data in our leave-one-seed-out validation procedure. We agree this is a valid concern and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: The leave-one-seed-out threshold validation does not properly separate threshold selection from test evaluation. Each of the five seeds produces an independent 80/20 stratified split of the same 249,867 records. For a given held-out seed, the probability that any test instance also appears in at least one of the other four seeds' test sets is 1 − (0.8)^4 ≈ 0.59. Thus roughly 59% of the held-out evaluation instances are present in the data used to select τ. This procedure therefore cannot rule out that τ = 0.9 is tuned to systematic properties of the shared test distribution rather than being a generalizable model property. This is load-bearing for the claim (Table 7, abstract) that threshold tuning 'closes the gap' between MC and ML.

    Authors: The referee is correct. Our leave-one-seed-out procedure rotates which seed is held out, but because each seed draws an independent 80/20 split from the same 249,867 records, the four seeds used for threshold selection include in their test sets a substantial fraction of instances that also appear in the held-out seed's test set. The referee's calculation (approximately 59% overlap) is accurate, and we concede that this means the procedure cannot fully rule out optimistic bias in the selected threshold. We will revise the manuscript to address this in two ways. First, we will add an explicit acknowledgment of this limitation in the threshold-sensitivity discussion (Section 5) and in the Limitations section, noting that the leave-one-seed-out design does not guarantee strict separation between threshold-selection and evaluation data when splits are drawn independently from the same pool. Second, we will re-run the threshold validation using a proper held-out partition: we will split the dataset into a fixed threshold-validation set (20%) and an evaluation set (80%), with the five random seeds applied only within the evaluation set. Threshold τ will be selected on the validation set and then applied to all five seed-based evaluation splits. This ensures zero overlap between the data used to select τ and the data used to report macro-F1. We note that several factors already limit the overfitting risk the referee is concerned about: (1) τ is a single scalar parameter, which severely limits the capacity to overfit; (2) the selected value (τ = 0.9) is consistent across all five folds, which would be unlikely if the selection were driven by noise in overlapping test instances; and (3) the threshold-sensitivity curve (Figure 2b) shows monotonic improvement from τ = 0.3 to τ = 0 revision: yes

  2. Referee: The fact that τ = 0.9 is selected in every fold provides some robustness evidence, and the single-parameter nature of the threshold limits overfitting risk, but [the comment appears truncated].

    Authors: We agree with the referee's partial acknowledgment that the single-parameter nature of the threshold and the consistency of τ = 0.9 across folds provide some robustness evidence. As noted in our response above, we will strengthen this by running a properly separated validation. We expect the result to hold — the monotonic relationship between τ and macro-F1, combined with the structural explanation (ML systematically overpredicts at moderate confidence levels because fewer than 1% of records carry multiple CWE labels, so a stricter threshold better matches the true label sparsity), suggests the effect is systematic rather than an artefact of data overlap. Nevertheless, we will report whatever the properly separated validation yields, and if the gap does not fully close under strict separation, we will adjust the claim in the abstract and Table 7 accordingly. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: empirical study with independently measured inputs and outputs

full rationale

This paper is a purely empirical comparison of multi-class vs. multi-label BERT formulations for CVE-to-CWE mapping. The central claims rest on independently measured confusion matrices, externally defined CWE taxonomy, and standard evaluation metrics. There are no self-citations in the reference list, no theoretical derivation chain, and no step where a predicted quantity is defined in terms of the quantity it claims to predict. The threshold-tuning result (τ=0.9 closes the MC–ML gap) is standard post-hoc hyperparameter optimization; while the skeptic correctly identifies a test-set overlap concern in the leave-one-seed-out validation, this is a methodology/correctness risk (potential overfitting), not circularity—the threshold is not defined in terms of the macro-F1 result it produces. The hierarchy-relaxed metric uses manually defined family groupings drawn from the external CWE taxonomy, not from the classifier's own outputs, so it is not self-definitional. The paper is self-contained against external benchmarks (CWE hierarchy, three independent encoders) and contains no circular derivation steps.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper does not invent new entities. It uses existing models (BERT, SecureBERT, CySecBERT), an existing dataset (MITRE CVE/CWE), and standard evaluation metrics. The free parameters are standard hyperparameters and data processing choices, not new theoretical constructs. The axioms are domain assumptions about the data and evaluation, not new physical or mathematical postulates.

free parameters (3)
  • Sigmoid threshold (τ) = 0.9
    The threshold for the multi-label setting is optimized post-hoc on the test set and validated via leave-one-seed-out, effectively a parameter fitted to the data.
  • Label frequency threshold = ≥100, ≥205, ≥570
    These thresholds determine the three nested label spaces (83, 47, 25 classes) and are chosen empirically to study the effect of label-space size.
  • Undersampling cap = 2000
    The cap for the four most frequent classes in the multi-class setting is a manually chosen parameter to address class imbalance.
assumptions (3)
  • domain assumption The first listed CWE in a CVE record is the 'primary' target for the multi-class setting.
    The paper states: 'For MC, each record is mapped to its first listed CWE... This follows the annotation order provided in the CVE record and treats the first label as the primary target.' This is a domain assumption about the semantics of the annotation order.
  • ad hoc to paper The manually defined family groupings (e.g., Memory/Buffer, Injection) accurately represent the CWE hierarchy for the purpose of relaxed evaluation.
    The paper states: 'The family groupings are defined manually based on the CWE hierarchy... This remains a heuristic operationalisation.' The relaxed metric depends entirely on this grouping.
  • domain assumption Macro-F1 is the appropriate primary metric for this task.
    The paper states: 'We report macro-F1 as the primary metric because it weights all classes equally and is therefore most sensitive to minority-class performance.' This is a standard but contestable choice in imbalanced classification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Class vs. Multi-Label BERT for CVE-to-CWE Mapping: How Taxonomy Structure Shapes the Errors." pith.science (2026). https://pith.science/paper/MFEGFZXN

@misc{pith2026260707573,
  author       = {Pith},
  title        = {Pith review of: Multi-Class vs. Multi-Label BERT for CVE-to-CWE Mapping: How Taxonomy Structure Shapes the Errors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFEGFZXN}},
  note         = {Machine review of arXiv:2607.07573}
}
abstract

Assigning Common Weakness Enumeration (CWE) categories to Common Vulnerabilities and Exposures (CVE) records remains an important but largely manual step in vulnerability analysis. We study this task as a text classification problem and compare two modelling choices: a \emph{multi-class} formulation that predicts a single CWE per CVE and a \emph{multi-label} formulation that allows multiple assignments. Three transformer encoders (BERT Base, SecureBERT, and CySecBERT) are evaluated on three nested label spaces (83, 47, and 25 classes). Multi-class training achieves higher macro-F1 across all settings, although the gap to multi-label narrows from 21 to 2 percentage points as the label space shrinks. Post-hoc threshold optimisation on the multi-label side closes this gap on the 25-class setting. Confusion analysis shows that the dominant misclassification patterns follow the CWE hierarchy and are shared across all three encoders (Pearson $r > 0.92$), which suggests that the error structure is driven more by taxonomy design than by encoder choice. A hierarchy-relaxed evaluation that forgives within-family confusions raises macro-F1 from ${\sim}$81\% to ${\sim}$90\%, indicating that strict metrics understate branch-level classifier quality. CySecBERT achieves the strongest results overall, with statistically significant gains concentrated in the multi-label setting.

Figures

Figures reproduced from arXiv: 2607.07573 by the authors.

Figure 1
Figure 1. A CVE mapped to a CWE taxonomy node. The hierarchy mixes abstraction levels: CWE-707 (“Pillar”) and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Macro-F1 across label spaces: MC (solid) outperforms ML (dashed) throughout; the gap (annotated) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Zoomed confusion matrix for CySecBERT (MC, 25 classes, averaged over 5 seeds), showing the 10 most [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [1]

    Aghaei, E., Al-Shaer, E., Shadid, W., Niu, X.: Automated CVE analysis for threat prioritization and impact prediction (2023)

    work page 2023

  2. [2]

    Aghaei, E., Niu, X., Shadid, W., Al-Shaer, E.: SecureBERT: A domain-specific language model for cybersecurity (2022)

    work page 2022

  3. [3]

    Aghaei, E., Shadid, W., Al-Shaer, E.: ThreatZoom: Hierarchical Neural Network for CVEs to CWEs Classification, pp. 23–41. Springer (2020). https://doi.org/10.1007/978-3-030-63086-7_2

    work page doi:10.1007/978-3-030-63086-7_2 2020

  4. [4]

    Bayer, M., Kuehn, P., Shanehsaz, R., Reuter, C.: CySecBERT: A domain-adapted language model for the cybersecurity domain (2022)

    work page 2022

  5. [5]

    DeepStrike: Vulnerabilities statistics 2025: Record CVEs, zero-days & exploit speed (oct 2025), https:// deepstrike.io/blog/vulnerability-statistics-2025

    work page 2025

  6. [6]

    Proceedings of NAACL-HLT pp

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT pp. 4171–4186 (2019)

    work page 2019

  7. [7]

    Dodiya, B., Singh, U.K., Gupta, V .: Trend analysis of the cve classes across cvss metrics. Int. J. Comput. Appl 183(33), 23–30 (2021)

    work page 2021

  8. [8]

    ML BERT for CVE-to-CWE MappingA PREPRINT

    Gururangan, S., Marasovi ´c, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., Smith, N.A.: Don’t stop pretraining: Adapt language models to domains and tasks (2020) 8 MC vs. ML BERT for CVE-to-CWE MappingA PREPRINT

    work page 2020

Show all 19 references
  1. [9]

    IEEE Security & Privacy 7(3), 68–71 (2009)

    Howard, M.: Improving software security by eliminating the cwe top 25 vulnerabilities. IEEE Security & Privacy 7(3), 68–71 (2009)

    2009

  2. [10]

    In: Advances in Neural Information Processing Systems

    Kunstner, F., Yadav, R., Milligan, A., Schmidt, M., Bietti, A.: Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. In: Advances in Neural Information Processing Systems. vol. 37 (2024)

    2024

  3. [11]

    CoRRabs/1907.11692(2019), http://arxiv

    Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V .: RoBERTa: A robustly optimized BERT pretraining approach. CoRRabs/1907.11692(2019), http://arxiv. org/abs/1907.11692

    1907 arXiv

  4. [12]

    In: Advances on Broad-Band Wireless Computing, Communication and Applications

    Na, S., Kim, T., Kim, H.: A study on the classification of common vulnerabilities and exposures using naïve bayes. In: Advances on Broad-Band Wireless Computing, Communication and Applications. pp. 657–662. Springer (2017)

    2017

  5. [13]

    nist.gov/general/cve-process

    National Institute of Standards and Technology: NVD – CVEs and the NVD process (2024), https://nvd. nist.gov/general/cve-process

    2024

  6. [14]

    In: Kim, Y ., Miyaji, A., Tibouchi, M

    Simonetto, S., Oosteven, R., {Van Ede}, T., Bosch, P., Jonker, W.: What matters most in vulnerabilities? key term extraction for cve-to-cwe mapping with llms. In: Kim, Y ., Miyaji, A., Tibouchi, M. (eds.) Cryptology and Network Security. pp. 467–492. Lecture Notes in Computer ...

    2026 doi

  7. [15]

    Applied and Computational Engineering71, 107–112 (2024)

    Su, J., Wu, Y .: Refining CVE-to-CWE mapping with enhanced attention in BERT-based models. Applied and Computational Engineering71, 107–112 (2024). https://doi.org/10.54254/2755-2721/71/20241647

    2024 doi

  8. [16]

    The MITRE Corporation: About CWE (2024),https://cwe.mitre.org/about/index.html

    2024

  9. [17]

    The MITRE Corporation: CVE overview (2024),https://www.cve.org/About/Overview

    2024

  10. [18]

    The MITRE Corporation: CWE usage guidance (2024), https://cwe.mitre.org/documents/cwe_usage/ guidance.html

    2024

  11. [19]

    Tiwari, H.: Advancing vulnerability classification with BERT: A multi-objective learning model (2025) 9

    2025

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.