{"id":"2b56ac74-0b12-42c4-893f-b9dd3ed4f300","arxiv_id":"2505.10008","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An in-context learning approach that retrieves similar vulnerability examples by fusing code and description similarities improves LLM-based severity assessment over prior baselines.","lead":"This paper tests whether an LLM can judge software vulnerability severity by showing it a few similar examples, combining code similarity with vulnerability description similarity. It reports accuracy gains over existing severity-assessment tools on a dataset of 12,071 C/C++ vulnerabilities, with all code and data shared on GitHub.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The recrawled CVSS v3 severity labels are the sole ground truth for every reported metric, yet Section 4.2 provides no validation against NVD or a second annotator; if these labels are noisy, the RQ1 comparison may shift.","rationale":"The paper’s central claim is an empirical benchmark comparison: SVA-ICL outperforms nine baselines on a 12,071-entry CVSS v3 dataset. The ground truth for every reported metric is the set of severity labels that the authors recrawled in Section 4.2 because MegaVul did not fully capture CVSS v3 ratings. The reader identified the lack of validation of these labels as the weakest assumption, and I agree that this is the most load-bearing issue. If the recrawled labels are systematically wrong, all Accuracy, F1, and MCC numbers in Table 3 and in the ablation studies change, potentially altering the ranking of methods. The paper states the recrawling was done, but it gives no agreement statistics against an independent source (e.g., NVD’s official CVSS v3 data) and no second-annotator check. The Threats to Validity section (6.7) does not even mention this threat, which is a notable omission. The leakage test in Section 6.5 is a useful partial check, but it only covers one baseline and does not address label validity. I also noticed minor reporting inconsistencies (the Introduction’s “at least 7.68/5.34/13.59 pp” vs. RQ1’s “2.53/3.04/3.49 pp”, and Section 4.6’s “60% to 40%” vs. RQ5’s “40% and 60%”), but these are not load-bearing for the central claim. The proposed NVD cross-check would directly test whether label noise can explain the reported advantage. If the advantage persists on NVD-agreeing labels, the paper’s central claim is strengthened; if not, the claim is unsupported. The reader’s CONDITIONAL verdict is therefore appropriate, and my read does not change it.","tokens_in":26158,"tokens_out":13127,"duration_ms":120886,"concrete_test":"Use the NVD CVE API to fetch the current CVSS v3.1 base score and severity for each CVE in the 12,071-entry dataset (matching by CVE ID), map to the four levels used in the paper, and compute Cohen’s kappa between the paper’s labels and the NVD-derived labels. Then re-run the RQ1 evaluation (SVA-ICL vs. MTLM, and all baselines if feasible) on the subset of test entries where the paper labels agree with NVD, or alternatively replace the labels with the NVD-official ones and recompute Table 3. If SVA-ICL’s margins over MTLM fall below the full-set 2.53/3.04/3.49 pp or reverse, the central claim is not robust to label error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result of Table 3 depends entirely on the CVSS v3 severity labels that the authors recrawled in Section 4.2. The text reports that MegaVul “did not fully capture the severity ratings of all vulnerability scores according to the CVSS v3 standard” and that the authors “recrawled the severity ratings and vulnerability scores of all vulnerabilities according to the CVSS v3 standard, along with their descriptions.” No validation of these recrawled labels is provided: no comparison against NVD’s official CVSS v3 base scores, no inter-annotator agreement, and no error-rate estimate. The paper’s Threats to Validity section (6.7) also does not list this as a threat. Because every Accuracy, F1, and MCC value in Table 3 and in the ablation tables is computed against these labels, any systematic error in the recrawling step (e.g., misreading vector strings, using a different CVSS version, or crawling scores published at a different time than NVD’s current entries) directly shifts all reported numbers. The marginal advantage over the second-best baseline (2.53 pp Accuracy, 3.04 pp F1, 3.49 pp MCC) is small enough that label noise of even a few percentage points could change the ranking, especially if the noise is correlated with source-code or description features. The dataset is also a claimed contribution, so label validity matters independently of the method comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SVA-ICL, an in-context learning (ICL) approach to software vulnerability assessment (SVA) that classifies C/C++ vulnerabilities into CVSS v3 severity levels (Critical, High, Medium, Low). It selects demonstrations by fusing source-code similarity (semantic similarity from CodeBERT with BERT-whitening, lexical Jaccard similarity, and syntactic AST-sequence Levenshtein similarity) with vulnerability-description similarity from CoSENT, then prompts DeepSeek-V2 with up to four demonstrations. The authors construct a 12,071-entry dataset from MegaVul, compare with nine baselines, and report higher Accuracy, macro F1, and MCC, together with ablations on fusion ratios, demonstration count, ordering, and choice of LLM.","tokens_in":26348,"tokens_out":5282,"duration_ms":47846,"significance":"If the empirical claims hold, the paper offers a useful contribution: a reproducible few-shot LLM pipeline for severity assessment, an openly shared dataset and code, baseline reruns with hyperparameter tuning, and unusually thorough ablations including multiple random splits and a post-cutoff leakage check. The margins over the strongest baselines are modest (e.g., 2.53 pp Accuracy, 3.04 pp F1, 3.49 pp MCC over MTLM in Table 3), so the credibility of the comparison depends heavily on the validity of the recrawled severity labels and on the consistency of the reported experimental configuration. The paper should be made stronger by directly validating the labels and by clarifying which configuration produced the headline numbers.","major_comments":[{"comment":"The recrawled CVSS v3 severity labels are not validated. The authors state that MegaVul did not fully capture CVSS v3 ratings and that they recrawled severity ratings and scores, but they provide no comparison with NVD official CVSS v3 base scores, no second-annotator agreement, and no error-rate estimate. Section 6.7 does not list label noise as a threat. Since every Accuracy, F1, and MCC value in Table 3 and all ablation tables is computed against these labels, and the advantage over the second-best baseline is small (2.53 pp, 3.04 pp, 3.49 pp), even a small systematic labeling error could change the ranking. As the dataset is a claimed contribution, label validity needs direct evidence, such as a random-sample audit against NVD or another authoritative source, with per-class agreement rates.","section":"Section 4.2, Table 3"},{"comment":"The main experimental configuration is reported inconsistently. Section 4.6 states that the ratio of syntactic to lexical similarity is set to 60% to 40%, but RQ5 concludes that the best setting is 40% syntactic and 60% lexical, and Table 7 reports the headline numbers of 77.07% Accuracy, 67.89% F1, and 63.01% MCC in the 40%/60% row. This discrepancy must be resolved: either Section 4.6 has a typo, or the RQ1/RQ2/RQ3/RQ4 results were generated with a different configuration than stated. Without clarification, the main result is not reproducible.","section":"Section 4.6 vs. Section 5.5 / Table 7"},{"comment":"The data-leakage analysis is under-specified. The post-cutoff subset size is not reported, the choice of July 2023 as DeepSeek-V2's cutoff is not justified, and the vulnerability descriptions in the prompts are CVE/NVD-style text that the model may have seen even if the code was introduced after the cutoff. The argument in Section 6.7 that poor zero-shot performance indicates low leakage probability is not compelling, because ICL demonstrations can still expose memorized label patterns. Please report the size and severity distribution of the post-cutoff subset, and state explicitly which aspects of leakage this check can and cannot rule out.","section":"Section 6.5 / Section 6.7"}],"minor_comments":[{"comment":"The Introduction states that SVA-ICL improves Accuracy, F1, and MCC by at least 7.68 pp, 5.34 pp, and 13.59 pp, but Table 3 shows the improvements over the second-best baseline (MTLM) are 2.53 pp, 3.04 pp, and 3.49 pp. These numbers should be reconciled; the current 'at least' phrasing is inconsistent with the reported table.","section":"Introduction, Section 1"},{"comment":"The notation is inconsistent: Table 7 uses 'LexiSim' while Section 3.1.2 and Eq. (4) use 'LexSim'. Please use one term throughout.","section":"Table 7 / Section 3.1.2"},{"comment":"The text says GPT-4o performs slightly better than DeepSeek-V2, but Table 8 shows GPT-4o has higher F1 and MCC while lower Accuracy. The comparison should be described as mixed, or a statistical test should be added if the claim is intended to be substantive.","section":"Section 6.1, Table 8"},{"comment":"Please provide the number of test samples collected after July 2023 and the exact cutoff date for the DeepSeek-V2 API version used. Without the sample size, the leakage check cannot be interpreted.","section":"Section 6.5"},{"comment":"The paper refers to 'the CVSS v3 standard' without specifying whether CVSS v3.0 or v3.1 is used. Since the severity buckets are the same but vector-string handling may differ, this detail matters for reproducing the label recrawling.","section":"Section 4.2"},{"comment":"The caption lists subplots (a) Accuracy, (b) F1-score, and (c) MCC, but the displayed panels appear to contain all three measures in each panel. Please verify the figure and caption correspondence.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The core empirical result is plausible but hinges on the unvalidated recrawled CVSS v3 labels. I would ask the authors to provide a label audit against an authoritative source (e.g., NVD) before the paper can be considered for acceptance. Also verify the 60/40 vs 40/60 syntactic/lexical ratio inconsistency, because it directly affects the reproducibility of the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read SVA-ICL. Short version: the comparison is run more honestly than most benchmark papers in this area, but the headline numbers all rest on recrawled CVSS v3 severity labels that the authors never validate against NVD or a second annotator. That is the thing to know first. Second, the novelty claim is overstated: the paper's own prior work (refs 25/26) uses the same fusion-based retrieval for comment generation, and SVA-PT already applies a pretrained model to severity prediction.\n\nWhat is genuinely good: baselines re-run with shared code and hyperparameter optimization, ablations for every main design choice (fusion ratio, number and ordering of demonstrations, syntactic/lexical trade-off), three random splits, a random-demonstration control, and a post-cutoff leakage check. That is more than most papers in this space do. The gains over the second-best baseline are modest (2.5-3.5 pp), but they are consistent across splits, which makes the empirical claim reasonably credible.\n\nSoft spots, in proportion. The label recrawl in Section 4.2 is load-bearing: every Accuracy, F1, and MCC in Tables 3-11 is computed against labels that are never checked against an independent source. The stress-test note is right that this is absent from the Threats to Validity section. With margins this small, correlated label noise could plausibly change the ranking. This must be fixed before the numbers can be taken as evidence. There is also an internal inconsistency: Section 4.6 says the syntactic-to-lexical ratio is 60:40, but RQ5 (Table 7) reports the optimum at 40:60. Probably a typo, but it needs reconciling. The abstract reports margins of 7.68/5.34/13.59 pp over the worst baseline while RQ1 uses the second-best baseline for its 2.53/3.04/3.49 pp claim; that discrepancy is confusing and should be clarified. Finally, \"first to utilize LLMs for SVA\" is not supported by the paper's own citations (SVA-PT uses prompt-tuned CodeT5) and should be softened.\n\nThe zero-shot result (MCC 7.14) is weak evidence against leakage; the post-cutoff subset check is the better evidence, and they did it. Credit where due.\n\nWho is this for? Researchers working on retrieval-based in-context learning for security tasks. It is a useful data point, not a proof. Send it to peer review, but insist on label validation, a cleaned-up margin story, and the promised code and data. If the labels check out, this is a solid incremental paper.","headline":"Honest, well-run benchmark for ICL-based vulnerability severity assessment, but every number sits on recrawled CVSS v3 labels that were never validated; referee it, but make label validation a hard requirement.","tokens_in":839,"tokens_out":912,"would_cite":false,"duration_ms":28598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fused code-description retrieval lifts LLM severity scoring to 77%","keywords":["software vulnerability assessment","in-context learning","large language model","information fusion","code similarity","vulnerability description","CVSS v3","demonstration selection"],"falsifier":"Take a random sample of 1,000 entries from the released dataset and compare the paper's recrawled severity labels against the official NVD CVSS v3 base scores; if agreement is below 95%, recompute SVA-ICL's accuracy on the corrected subset to see how much of the reported 77.07% persists. A second check is to run SVA-ICL on vulnerabilities disclosed after DeepSeek-V2's training cutoff with the vulnerability description removed, so only code remains; if accuracy drops sharply, part of the gain comes from the LLM's memorized CVE knowledge rather than retrieval quality.","tokens_in":25826,"feed_emoji":"🛡️","tokens_out":4481,"duration_ms":41946,"temperature":0.7,"pith_summary":"This paper claims that a large language model can assess software vulnerability severity well using in-context learning, provided the demonstration examples are selected by fusing source-code similarity with vulnerability-description textual similarity. On a self-built dataset of 12,071 C/C++ vulnerabilities labeled with CVSS v3 severity, the proposed SVA-ICL approach reports 77.07% accuracy, 67.89% F1-score, and 63.01% MCC, outperforming nine baselines including bimodal models. The key claim is that demonstration quality, not model capacity, drives the gain, so no fine-tuning is needed. A reader should care because this offers a practical, low-cost path to automated vulnerability triage using off-the-shelf LLM APIs.","feed_headline":"Fused code-description retrieval lifts LLM severity scoring to 77%","feed_subtitle":"Picking demonstrations by code and text similarity beats nine baselines without any fine-tuning.","key_machinery":"The load-bearing mechanism is a two-level fused similarity score used for demonstration retrieval: Sim(Va, Vb) = φ × CodeSim(A, B) + (1 − φ) × TextSim(Da, Db), where TextSim comes from a CoSENT text-embedding model, and CodeSim itself blends syntactic similarity (Levenshtein distance over tree-sitter AST sequences) with lexical similarity (Jaccard overlap of code tokens). Semantic retrieval is pre-computed via CodeBERT embeddings transformed by BERT-whitening, so online cost per query is low. This module selects the top-k examples that are then inserted into a structured prompt, which is fed to DeepSeek-V2 with temperature set to zero.","core_discovery":"The central claim is that combining code similarity (semantic, lexical, and syntactic) with vulnerability-description textual similarity, fused at a 70% code to 30% description ratio, yields higher-quality in-context demonstrations than any single modality, and that these demonstrations are what make the LLM accurate at severity assessment. The paper shows the optimal configuration to be four demonstrations arranged in ascending order of similarity to the test input, with code similarity itself blended as 40% syntactic and 60% lexical. Under this configuration, SVA-ICL with DeepSeek-V2 scores 77.07% accuracy, 67.89% F1, and 63.01% MCC, improving on the best baseline (MTLM) by 2.53, 3.04, and 3.49 percentage points respectively. The authors further show that random demonstration selection drops accuracy to 61.75%, while a zero-shot baseline achieves only 44.21% accuracy.","pith_inferences":["If the recrawled severity labels were validated against NVD's official CVSS v3 scores, the gap between SVA-ICL and baselines might shrink; the paper does not provide such validation, so part of the reported gain could be an artifact of label noise inherited by all methods.","A testable extension would strip CVE identifiers and vendor names from the vulnerability descriptions in the test prompts; if accuracy falls substantially, part of the gain comes from the LLM's memorized CVE knowledge rather than from retrieval quality alone.","The same fused-retrieval ICL recipe likely transfers to other structured software tasks such as bug triage, patch prioritization, or risk scoring of configuration files, where multi-modal similarity selection could matter as much as the choice of LLM."],"forward_implications":["Organizations can implement LLM-based severity triage for C/C++ code without fine-tuning, as long as they maintain a historical corpus with precomputed embeddings.","The 70/30 code-to-description fusion ratio indicates that source code carries more severity-relevant signal than natural-language descriptions, a guideline for future multimodal retrieval designs.","Demonstration count (four) and ascending-similarity ordering emerge as tunable transferable hyperparameters for ICL in other security tasks.","The released 12,071-entry CVSS v3 dataset built from MegaVul provides a common benchmark aligned with the newer scoring standard.","If the retrieval mechanism is the true driver, then performance should scale with the quality and coverage of the historical vulnerability repository, making corpus curation a practical lever."],"supporting_citations":[{"why":"Supplies the raw MegaVul dataset of 17,975 C/C++ vulnerability entries that the authors filter, recrawl, and relabel to build their 12,071-entry CVSS v3 dataset.","marker":"[42]"},{"why":"MegaVul is the source corpus; this reference establishes the provenance and structure of the vulnerability code and description pairs.","marker":"Ni et al."},{"why":"CodeBERT provides the semantic embeddings of vulnerability code used for the first-stage semantic similarity retrieval.","marker":"[27]"},{"why":"BERT-whitening is applied to improve isotropy and reduce embedding dimension, which the authors claim boosts retrieval quality.","marker":"[28]"},{"why":"Gao et al. supply the motivation and prior evidence for demonstration selection, ordering, and the four-example setting in ICL for code tasks.","marker":"[23]"},{"why":"Zhao et al. provide the prior smart-contract comment-generation ICL approach whose demo number and ordering settings are adapted here.","marker":"[26]"},{"why":"Le et al. provide the FuncRF and FuncLGBM function-level source-code baselines that SVA-ICL is compared against.","marker":"[4]"},{"why":"Xue et al. provide SVA-PT, the prompt-tuning bimodal baseline that SVA-ICL outperforms by at least 2.53 percentage points in accuracy.","marker":"[38]"},{"why":"Du et al. provide MTLM, the strongest baseline (74.54% accuracy), which SVA-ICL beats by 2.53 pp accuracy, 3.04 pp F1, and 3.49 pp MCC.","marker":"[39]"}],"fun_headline_variants":["Fused ICL prompts push LLM vulnerability scoring to 77%","Code-description fusion boosts LLM vulnerability assessment","LLM severity scoring hits 77% with fused ICL demonstrations","ICL with blended code-text similarities lifts SVA accuracy","No fine-tuning, just fused ICL: LLM SVA hits 77%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recrawled CVSS v3 severity labels used as ground truth are assumed correct, but no independent validation against NVD's official scores or a second annotator is reported, so any label noise propagates directly into every reported accuracy, F1, and MCC figure.","fun_headline_variants_meta":{"raw":{"variants":["Fused ICL prompts push LLM vulnerability scoring to 77%","Code-description fusion boosts LLM vulnerability assessment","LLM severity scoring hits 77% with fused ICL demonstrations","ICL with blended code-text similarities lifts SVA accuracy","No fine-tuning, just fused ICL: LLM SVA hits 77%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2579,"prompt_tokens":1012,"completion_tokens":1567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":1477}},"tokens_in":628,"tokens_out":1567,"duration_ms":9779,"temperature":1.0,"reasoning_tokens":1477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:17:42.780855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 1,000 entries from the released dataset and compare the paper's recrawled severity labels against the official NVD CVSS v3 base scores; if agreement is below 95%, recompute SVA-ICL's accuracy on the corrected subset to see how much of the reported 77.07% persists. A second check is to run SVA-ICL on vulnerabilities disclosed after DeepSeek-V2's training cutoff with the vulnerability description removed, so only code remains; if accuracy drops sharply, part of the gain comes from the LLM's memorized CVE knowledge rather than retrieval quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the raw MegaVul dataset of 17,975 C/C++ vulnerability entries that the authors filter, recrawl, and relabel to build their 12,071-entry CVSS v3 dataset."},{"cited_title":"Gao, X.-C","cited_arxiv_id":null,"evidence_quote":"Gao et al. supply the motivation and prior evidence for demonstration selection, ordering, and the four-example setting in ICL for code tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Zhao et al. provide the prior smart-contract comment-generation ICL approach whose demo number and ordering settings are adapted here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Xue et al. provide SVA-PT, the prompt-tuning bimodal baseline that SVA-ICL outperforms by at least 2.53 percentage points in accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Du et al. provide MTLM, the strongest baseline (74.54% accuracy), which SVA-ICL beats by 2.53 pp accuracy, 3.04 pp F1, and 3.49 pp MCC."}],"review_version":1}