{"id":"fdd65707-3719-4d42-9a92-344ca78d6aab","arxiv_id":"2607.07573","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Multi-class BERT outperforms multi-label BERT for CVE-to-CWE mapping, but the gap narrows with fewer classes and threshold tuning, while dominant errors follow the CWE hierarchy across all models.","lead":"This paper compares single-label vs. multi-label BERT classifiers for mapping cybersecurity vulnerabilities (CVEs) to weakness categories (CWEs), finding that errors are driven by taxonomy structure rather than model choice. A smart generalist might read it to understand how the design of a classification system's label set can constrain or confuse automated text analysis.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The leave-one-seed-out threshold validation does not properly separate threshold selection from test evaluation because the five 80/20 splits share substantial test-set overlap (~59% of any held-out seed's test instances appear in the other four seeds' test sets).","rationale":"The reader identified the threshold-tuning concern as secondary but focused on the family groupings as the weakest assumption. I think the threshold validation issue is more load-bearing because it affects a quantitative claim (MC≈ML after tuning) that frames the paper's narrative, whereas the family-grouping concern is already addressed by the sensitivity analysis showing 84–91% across grouping variants. However, the central claim about error structure (Pearson r>0.92, hierarchy-related confusions) does not depend on the threshold tuning — it is based on MC confusion matrices directly. The threshold concern affects the secondary claim of MC–ML equivalence but not the core finding about taxonomy-driven errors. The paper also transparently acknowledges the post-hoc nature of the threshold analysis. For these reasons, the CONDITIONAL verdict remains appropriate: the threshold concern is real and worth flagging, but it does not overturn the paper's main contribution. The reader's focus on family groupings is a reasonable but less critical concern given the sensitivity analysis already performed. I rate agreement as 'partial' because the reader correctly identified the threshold issue but did not recognize the severity of the test-set overlap problem in the leave-one-seed-out procedure.","tokens_in":9634,"tokens_out":6859,"duration_ms":426777,"concrete_test":"Re-run the 25-class CySecBERT ML threshold analysis with a proper three-way split: 80% train, 10% validation, 10% test (using a single seed). Select τ on the validation set only, then evaluate on the held-out test set. Repeat for all 5 seeds. If the validated τ shifts away from 0.9 or the test-set macro-F1 drops below 80.5, the 'gap closes' claim weakens and the MC–ML equivalence is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper claims that post-hoc threshold tuning closes the MC–ML gap on 25 classes (Table 7: ML rises from 79.2 to 81.2, matching MC at 81.2). To defend against test-set overfitting, they run leave-one-seed-out validation: for each of 5 seeds, select τ on the other 4 seeds' test sets, evaluate on the held-out seed. However, each seed produces an independent 80/20 stratified split of the same 249,867 records. Two independent 20% test sets share ~20%×20%=4% of the total data; across 4 other seeds, the probability that a given instance in the held-out test set also appears in at least one of the other 4 test sets is 1−(0.8)^4 ≈ 0.59. So roughly 59% of the held-out test instances are present in the data used to select τ. This is not equivalent to proper held-out validation. The threshold is effectively selected on a superset that contains the majority of the evaluation instances. While the paper reports τ=0.9 selected in every fold (which provides some robustness evidence), the overlap means the leave-one-seed-out procedure cannot rule out that the threshold is tuned to systematic properties of the shared test distribution rather than a generalizable model property. This matters because the 'gap closes' finding is a key part of the narrative framing the subsequent error analysis as comparing roughly equivalent MC and ML formulations. If the threshold is mildly overfit, the true ML performance at a properly validated threshold could be lower, reopening the gap.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper compares multi-class (MC) and multi-label (ML) BERT formulations for CVE-to-CWE mapping using three encoders (BERT Base, SecureBERT, CySecBERT) across three nested label spaces (83, 47, 25 classes). The main findings are: (1) MC outperforms ML on macro-F1 across all settings, with the gap narrowing from ~21pp to ~2pp as the label space shrinks; (2) threshold tuning on the ML side closes the gap at 25 classes; (3) dominant misclassification patterns follow the CWE hierarchy and are shared across all three encoders (Pearson r > 0.92); and (4) a hierarchy-relaxed evaluation raises macro-F1 to ~90%, suggesting strict metrics understate branch-level quality. The paper is well-structured and addresses a genuine gap in the literature by directly comparing the two formulations under matched conditions.","tokens_in":9901,"tokens_out":2517,"duration_ms":121356,"significance":"The paper provides a careful empirical comparison that is missing from prior work, which typically commits to one formulation without examining the trade-off. The confusion-matrix correlation analysis across encoders (r > 0.92) is a strong and falsifiable finding that points to taxonomy structure as the primary error driver. The matched-data ablation (MC without undersampling) is a commendable control that strengthens the formulation-effect claim. The sensitivity analysis of family groupings (four variants, 84–91% range) is a reasonable robustness check on the hierarchy-relaxed metric. The practical implications for vulnerability triage pipelines are clearly articulated.","major_comments":[{"comment":"The leave-one-seed-out threshold validation does not properly separate threshold selection from test evaluation. Each of the five seeds produces an independent 80/20 stratified split of the same 249,867 records. For a given held-out seed, the probability that any test instance also appears in at least one of the other four seeds' test sets is 1 − (0.8)^4 ≈ 0.59. Thus roughly 59% of the held-out evaluation instances are present in the data used to select τ. The procedure therefore cannot rule out that τ = 0.9 is tuned to systematic properties of the shared test distribution rather than being a generalizable model property. This is load-bearing for the claim (Table 7, abstract) that threshold tuning 'closes the gap' between MC and ML. The fact that τ = 0.9 is selected in every fold provides some robustness evidence, and the single-parameter nature of the threshold limits overfitting risk,但","section":"§5, Threshold sensitivity and gap closure (leave-one-seed-out validation)"}],"minor_comments":[{"comment":"The paper states that 'fewer than 1%' of records have multiple CWE labels. Given that this is the primary motivation for the ML formulation, it would help to report the exact count and briefly discuss how this affects the reliability of ML evaluation.","section":"§4, Dataset"},{"comment":"The family groupings are listed inline (e.g., 'Memory/Buffer: CWE-119, 120, 121, 122, 125, 787') but the full mapping for all families is not provided. A supplementary table would improve reproducibility.","section":"§4, Evaluation"},{"comment":"The SGD result (0.4% macro-F1, 81 dead classes) is striking but only briefly discussed. A sentence clarifying whether this is a known interaction or specific to this task would help readers.","section":"Table 1"},{"comment":"The threshold sensitivity plot shows macro-F1 increasing monotonically with τ up to 0.9. It would be useful to show values beyond 0.9 to confirm that performance does not continue to increase, which would strengthen the claim that 0.9 is an optimum rather than an endpoint.","section":"Figure 2(b)"},{"comment":"The paper acknowledges that the threshold analysis is post-hoc on the test set, but the leave-one-seed-out procedure is presented as addressing this concern. Given the overlap issue (see major comment), the limitations paragraph should be revised to accurately characterize what the validation does and does not demonstrate.","section":"§6, Limitations"}],"recommendation":"minor_revision","confidential_remarks":"The stress-test concern about test-set overlap in the leave-one-seed-out validation is valid and should be addressed in revision. However, the paper's central contribution—the error-structure analysis showing taxonomy-driven errors shared across encoders—is independent of the threshold finding and stands on its own. The threshold gap-closure result is a secondary finding; even if the validation is imperfect, the paper's main claims are not undermined. I recommend minor revision with a requirement to accurately characterize the limitations of the validation procedure."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful reading and the constructive assessment. The referee raises one major methodological concern regarding potential leakage between threshold-selection and test-evaluation data in our leave-one-seed-out validation procedure. We agree this is a valid concern and will revise the manuscript accordingly.","responses":[{"response":"The referee is correct. Our leave-one-seed-out procedure rotates which seed is held out, but because each seed draws an independent 80/20 split from the same 249,867 records, the four seeds used for threshold selection include in their test sets a substantial fraction of instances that also appear in the held-out seed's test set. The referee's calculation (approximately 59% overlap) is accurate, and we concede that this means the procedure cannot fully rule out optimistic bias in the selected threshold. We will revise the manuscript to address this in two ways. First, we will add an explicit acknowledgment of this limitation in the threshold-sensitivity discussion (Section 5) and in the Limitations section, noting that the leave-one-seed-out design does not guarantee strict separation between threshold-selection and evaluation data when splits are drawn independently from the same pool. Second, we will re-run the threshold validation using a proper held-out partition: we will split the dataset into a fixed threshold-validation set (20%) and an evaluation set (80%), with the five random seeds applied only within the evaluation set. Threshold τ will be selected on the validation set and then applied to all five seed-based evaluation splits. This ensures zero overlap between the data used to select τ and the data used to report macro-F1. We note that several factors already limit the overfitting risk the referee is concerned about: (1) τ is a single scalar parameter, which severely limits the capacity to overfit; (2) the selected value (τ = 0.9) is consistent across all five folds, which would be unlikely if the selection were driven by noise in overlapping test instances; and (3) the threshold-sensitivity curve (Figure 2b) shows monotonic improvement from τ = 0.3 to τ = 0","revision_made":"yes","referee_comment":"The leave-one-seed-out threshold validation does not properly separate threshold selection from test evaluation. Each of the five seeds produces an independent 80/20 stratified split of the same 249,867 records. For a given held-out seed, the probability that any test instance also appears in at least one of the other four seeds' test sets is 1 − (0.8)^4 ≈ 0.59. Thus roughly 59% of the held-out evaluation instances are present in the data used to select τ. This procedure therefore cannot rule out that τ = 0.9 is tuned to systematic properties of the shared test distribution rather than being a generalizable model property. This is load-bearing for the claim (Table 7, abstract) that threshold tuning 'closes the gap' between MC and ML."},{"response":"We agree with the referee's partial acknowledgment that the single-parameter nature of the threshold and the consistency of τ = 0.9 across folds provide some robustness evidence. As noted in our response above, we will strengthen this by running a properly separated validation. We expect the result to hold — the monotonic relationship between τ and macro-F1, combined with the structural explanation (ML systematically overpredicts at moderate confidence levels because fewer than 1% of records carry multiple CWE labels, so a stricter threshold better matches the true label sparsity), suggests the effect is systematic rather than an artefact of data overlap. Nevertheless, we will report whatever the properly separated validation yields, and if the gap does not fully close under strict separation, we will adjust the claim in the abstract and Table 7 accordingly.","revision_made":"yes","referee_comment":"The fact that τ = 0.9 is selected in every fold provides some robustness evidence, and the single-parameter nature of the threshold limits overfitting risk, but [the comment appears truncated]."}],"tokens_in":9488,"tokens_out":851,"duration_ms":152871,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Bottom line: this paper does a clean, controlled comparison of multi-class vs. multi-label formulations for CVE-to-CWE mapping and finds that error patterns are driven by the CWE taxonomy, not by which encoder you pick. That's a useful finding for the cybersecurity NLP community and worth a serious referee's time. The stress-test concern about the leave-one-seed-out threshold validation is real but does not sink the paper. What's actually new and done well: the direct, matched comparison of MC and ML under identical encoders, data, and label-space sizes is something prior work didn't do. The cross-encoder confusion analysis (Pearson r > 0.92 across all three encoders) is a strong result — it shows the errors are structural, not model-specific. The hierarchy-relaxed evaluation is a nice operational contribution, and they back it up with a sensitivity analysis across four grouping variants (84–91% range), which addresses the 'manual groupings' concern more seriously than the reader credited. The matched-data ablation (MC without undersampling) is also good practice — it shows the MC advantage is a formulation effect, not a data-composition artifact. Now the soft spots. The stress-test note is correct: the five 80/20 splits share roughly 59% of test instances across folds, so the leave-one-seed-out procedure doesn't give you a clean separation between threshold selection and evaluation. This means the 'gap closes' finding (Table 7: ML rises to 81.2, matching MC) rests on somewhat leaky validation. That said, two things temper the concern. First, the same threshold (τ=0.9) is selected in every fold, which is a robustness signal even if the folds overlap. Second, the threshold-tuned result is a secondary finding — the primary narrative is about error structure, which doesn't depend on the threshold at all. The hierarchy-relaxed metric uses manually defined family groupings, but the sensitivity analysis across four variants makes this less worrying than it sounds. The paper is honest about all of these limitations in its discussion section, which I appreciate. Who gets value from this: researchers and practitioners working on automated vulnerability classification, especially anyone deciding between MC and ML formulations or designing hierarchy-aware training objectives. The paper doesn't propose a new architecture, but it clarifies where the problems actually live — in the taxonomy, not the model. Recommendation: accept for peer review. The threshold validation issue should be flagged to the authors as a revision request — a proper held-out threshold split would strengthen the claim — but the core findings stand on their own.","headline":"Solid empirical study: taxonomy structure, not encoder choice, drives CVE-to-CWE errors. One methodological soft spot in the threshold validation.","tokens_in":10416,"tokens_out":617,"would_cite":true,"duration_ms":115603,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"CWE taxonomy, not the model, drives classification errors","keywords":["CVE","CWE","multi-class classification","multi-label classification","transformer","BERT","error analysis","taxonomy"],"falsifier":"If the error patterns were found to diverge significantly across encoders (e.g., Pearson r < 0.70), or if dominant misclassifications did not correspond to parent-child or sibling relationships in the CWE tree, the claim that taxonomy structure drives errors would collapse.","tokens_in":9775,"feed_emoji":"🌳","tokens_out":865,"duration_ms":103233,"temperature":0.7,"pith_summary":"This paper compares two ways of training BERT-style models to automatically assign weakness categories to vulnerability reports: a single-answer formulation (multi-class) and a multiple-answer formulation (multi-label). Across three encoder variants and three label-space sizes, multi-class consistently outperforms multi-label, though the gap shrinks from 21 to 2 percentage points as the number of classes is reduced from 83 to 25. The paper's central finding comes from the error analysis: when the three encoders' confusion patterns are compared, they are nearly identical (Pearson r > 0.92), and the dominant misclassifications fall along parent-child or sibling relationships in the CWE hierarchy. This suggests that the residual errors are driven by the structure of the taxonomy itself—where adjacent nodes are textually near-identical—rather than by limitations of any particular model. A relaxed evaluation that credits the classifier for landing on the correct branch of the CWE tree, even if the exact node is wrong, raises macro-F1 from roughly 81% to 90%, indicating that standard strict metrics substantially understate how useful the classifiers are for practical triage.","feed_headline":"CWE taxonomy, not the model, drives classification errors","feed_subtitle":"Three BERT variants make nearly identical mistakes along the CWE hierarchy, suggesting the taxonomy itself is the bottleneck — not the encod","key_machinery":"The central mechanism is the CWE hierarchy itself: a tree of weakness types where broad 'Pillar' categories and specific 'Variant' nodes coexist as valid classification targets. The paper shows that this mixed granularity creates systematic confusion at hierarchy-adjacent nodes, and introduces a hierarchy-relaxed macro-F1 metric that credits within-family predictions to reveal the true branch-level accuracy.","core_discovery":"The dominant misclassification patterns in automated CVE-to-CWE mapping are structurally identical across three different transformer encoders (Pearson r > 0.92) and concentrate along parent-child and sibling edges in the CWE hierarchy. This means the errors are a property of the taxonomy's design, not the model. When evaluation forgives within-family confusions, macro-F1 rises from ~81% to ~90%, showing that classifiers are reaching the correct region of the taxonomy far more often than exact-match metrics suggest.","pith_inferences":[],"forward_implications":["Automated vulnerability triage pipelines can rely on classifiers that reach ~90% branch-level accuracy, which is operationally sufficient for routing CVEs to the correct mitigation team even when the exact CWE identifier is wrong.","Improving classification on the hardest pairs (e.g., CWE-77/78) likely requires hierarchy-aware training objectives or better annotation guidelines, not a better encoder.","The default sigmoid threshold of 0.5 is suboptimal for multi-label CWE classification; a threshold of 0.9 nearly closes the gap to multi-class, suggesting multi-label models systematically over-predict at moderate confidence levels.","Domain-adaptive pretraining (CySecBERT) provides statistically significant gains primarily in the multi-label setting where per-class supervision is sparse, but the benefit diminishes under aggressive full fine-tuning in the multi-class setting."],"fun_headline_variants":["BERT models share the same CWE mistakes — taxonomy is the bottleneck","CWE hierarchy shapes CVE classification errors across all BERT variants","Taxonomy structure, not encoder choice, drives CVE-to-CWE mapping errors","Within-family CWE confusions reveal classifiers reach the right branch 90% of the time","Error patterns in CVE-to-CWE mapping are identical across three BERT encoders"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The hierarchy-relaxed evaluation relies on manually defined family groupings of CWE nodes. The authors acknowledge this is a heuristic: alternative groupings would yield different scores, though a sensitivity analysis across four grouping variants produces relaxed macro-F1 between 84% and 91%, which they take as evidence of robustness.","fun_headline_variants_meta":{"raw":{"variants":["BERT models share the same CWE mistakes — taxonomy is the bottleneck","CWE hierarchy shapes CVE classification errors across all BERT variants","Taxonomy structure, not encoder choice, drives CVE-to-CWE mapping errors","Within-family CWE confusions reveal classifiers reach the right branch 90% of the time","Error patterns in CVE-to-CWE mapping are identical across three BERT encoders"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":673,"prompt_tokens":592,"completion_tokens":81,"prompt_tokens_details":null},"tokens_in":592,"tokens_out":81,"duration_ms":17476,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T06:27:54.583888+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the error patterns were found to diverge significantly across encoders (e.g., Pearson r < 0.70), or if dominant misclassifications did not correspond to parent-child or sibling relationships in the CWE tree, the claim that taxonomy structure drives errors would collapse.","supporting_citations":[],"review_version":1}