{"id":"1185aa37-f8eb-4c25-9375-660324e3b10a","arxiv_id":"2501.14737","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-agent LLM framework with four communication strategies improves CVSS v3.1 vulnerability assessment over single-agent baselines on a new commit dataset spanning C++, Python, and Java.","lead":"This paper applies teams of large language models that debate each other to score software vulnerabilities using the CVSS standard. The authors built a new multilingual vulnerability dataset and report that the multi-agent approach beats a single model on most metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains appear to come from per-metric best-strategy selection: Table 3's Python ChatGPT EvalSVA row exactly matches the best entries across the four strategies in Table 4, making the comparison to a single agent an oracle-selection artifact.","rationale":"The reader's weakest assumption was label noise in the dataset, which is a valid concern but not the most load-bearing issue for the central claim. The per-metric best-strategy selection is more directly fatal: even with perfect labels, the comparison in Table 3 is biased if 'EvalSVA' is an oracle over communication strategies rather than a fixed method. The exact match between Table 3 and Table 4 for Python+ChatGPT provides strong internal evidence of this selection. This does not necessarily disprove the value of multi-agent deliberation—Preceding One Expert alone appears to beat the single agent on average—but it means the paper's specific headline magnitudes are unsupported and the experimental protocol must be corrected. The reader's CONDITIONAL verdict remains appropriate because the issue is fixable by re-running with a fixed strategy and adding significance tests. The dataset, despite its small manual validation sample, is a potentially useful independent contribution. I therefore recommend CONDITIONAL with a specific condition: demonstrate the improvement with a single fixed communication strategy and report confidence intervals.","tokens_in":15004,"tokens_out":6918,"duration_ms":67067,"concrete_test":"Re-evaluate EvalSVA with a single, pre-registered communication strategy applied to all eight metrics for each language (e.g., 'Preceding One Expert', which is best for most metrics), and report per-metric accuracy/F1 compared with the single-agent baseline with paired bootstrap 95% confidence intervals. Also compute the headline F1 improvement using the average of all four strategies or a strategy selected on a time-based validation split, not the test split. If no fixed strategy yields a statistically significant average F1 improvement over the single agent across the three languages, the central claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The main evidence for EvalSVA's superiority is Table 3, but the 'EvalSVA' rows appear to be constructed by selecting, for each metric, the best-performing communication strategy from Table 4 rather than using one fixed strategy. For Python+ChatGPT, every EvalSVA value in Table 3 exactly matches the maximum of the four strategies in Table 4: AV uses Summarizer Assessment (0.3282/0.2014), AC and PR use Previous Communication (0.4351/0.2510 and 0.5954/0.4761), and Scope, Confidentiality, Integrity, Availability use Preceding One Expert (0.4504/0.4386, 0.5115/0.4151, 0.4733/0.3914, 0.4580/0.3432). The paper never discloses this per-metric selection; it only lists four strategies and then presents 'EvalSVA' without specifying which strategy generated each number. The single-agent baseline is a single fixed prompt, so the comparison is unfair: EvalSVA gets the benefit of choosing the best variant post hoc on the test set. The reported 'improves the F1 by 53.71% for ChatGPT' is therefore likely inflated relative to any single reproducible multi-agent method. Test sets are also tiny (approx. 90 Java, 131 Python, 69 C++ samples based on denominators in Table 3), and no confidence intervals or significance tests are provided, so the robustness of even a fixed-strategy comparison is unknown. This is a correctness risk in the central claim, not a matter of consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EvalSVA, a multi-agent LLM framework for CVSS v3.1 vulnerability assessment from vulnerability-related commits. It constructs a multilingual dataset (C++, Java, Python) based on Mend/NVD labels, evaluates four communication strategies for agents, and compares the multi-agent approach against a single-agent baseline using ChatGPT and GPT-4. The authors report large average improvements in accuracy and F1 (44.12% and 43.29% in the abstract; 53.71% and 32.88% F1 for ChatGPT and GPT-4 in the experiments section) and claim that EvalSVA provides human-like, interpretable assessments. The paper also includes a case study, an analysis of expert counts and communication rounds, and a short limitation section.","tokens_in":15358,"tokens_out":6773,"duration_ms":60874,"significance":"If the reported results are valid, the paper makes a useful practical contribution: it is among the first to apply multi-agent LLM deliberation to CVSS metric prediction, it provides a new multilingual benchmark for the task, and it generates rationales for predictions, which is aligned with the interpretability needs of vulnerability assessment. The dataset itself, if cleaned and properly validated, could be a valuable resource. However, the current evaluation protocol has load-bearing weaknesses: the main EvalSVA results appear to be assembled from different communication strategies per metric rather than from a single reproducible configuration, the ground-truth validation is based on only 20 examples per language, the test sets are small (roughly 90/131/69 commits), and no confidence intervals or significance tests are reported. These issues materially affect the central claim of superiority over single-agent baselines.","major_comments":[{"comment":"The EvalSVA rows in Table 3 do not correspond to any single communication strategy described in the 'Communication Strategy' subsection. For Python+ChatGPT, the AV entry (0.3282/0.2014) matches the Summarizer Assessment row in Table 4, the AC entry (0.4351/0.2510) matches Previous Communication, and the PR, Scope, Confidentiality, Integrity, and Availability entries (0.5954/0.4761, 0.4504/0.4386, 0.5115/0.4151, 0.4733/0.3914, 0.4580/0.3432) match Preceding One Expert. The paper does not state that the EvalSVA row is the per-metric optimum, nor does it provide a selection rule. As written, the comparison against the 'Single' baseline is not a comparison of a well-defined, reproducible multi-agent method, and the headline gains may be inflated by post-hoc per-metric selection of the best strategy. Please specify the exact strategy used for each reported EvalSVA number (or use a principled selection on a validation split) and rerun the comparison accordingly.","section":"Experiments, 'EvalSVA Results' and 'Communication Strategy' (Tables 3–4)"},{"comment":"The manual validation of the dataset uses only 20 randomly selected examples per language, independently checked by two developers with no inter-rater agreement statistic. Since the Mend/NVD labels are the ground truth for every accuracy and F1 computation, a 20-example sample is too small to establish label quality; for C++ the reported 90.0% accuracy is exactly 18 of 20 correct, giving a wide confidence interval. Please report a larger validation sample, per-metric agreement rates, and an inter-rater reliability measure, or otherwise provide a statistically justified bound on label noise.","section":"Table 2, 'Dataset Evaluation'"},{"comment":"The test sets are small (about 90 Java, 131 Python, and 69 C++ commits, as implied by the accuracy denominators in Table 3), and no confidence intervals, standard errors, or significance tests are provided. Many per-metric differences between the single agent and EvalSVA are only a few percentage points, so the claim that EvalSVA 'significantly enhances' SV assessment is not statistically supported. Please add confidence intervals or bootstrap/significance tests, and consider whether the test sets can be enlarged.","section":"Table 3, 'EvalSVA Results'"},{"comment":"The analysis of expert counts and communication rounds selects the optimal values based on performance on the Java dataset (e.g., 'the optimal performance occurring when the number of experts is 2-3' and 'PR exhibits optimal performance after three rounds'). If these selected values are used in the main EvalSVA results in Table 3, the comparison is optimistically biased because the configuration is chosen on the test data. Please use a validation split for any hyperparameter selection, or state clearly that the Table 3 results use a fixed, pre-specified configuration.","section":"Figure 3, 'Expert Numbers and Communication Rounds'"}],"minor_comments":[{"comment":"Table 1 reports 689 C++ commits, while the Abstract, Introduction, and Conclusion all state 699; please make these numbers consistent.","section":"Table 1 vs. Abstract/Introduction/Conclusion"},{"comment":"Table 2 would benefit from reporting the raw agreement counts and an inter-rater reliability statistic (e.g., Cohen's kappa), as well as a confidence interval for each language's accuracy.","section":"Table 2"},{"comment":"The Table 3 caption contains a typo ('Webold' should be 'We bold'), and it is unclear what the boldface indicates when only two methods are shown in that table.","section":"Table 3 caption"},{"comment":"The same work by Yin, Ni, and Wang is cited twice as both 2024a and 2024b; the duplicate citation should be removed.","section":"Related Work / References"},{"comment":"The phrase 'average increases of 72.35% and 49.35%' for the exploitability aspect is ambiguous: it does not state whether these are relative or absolute changes, nor over which languages the average is taken; please clarify.","section":"Experiments, 'EvalSVA Results'"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the idea of multi-agent deliberation for CVSS assessment is worth exploring. However, the current presentation of the EvalSVA rows as a single method is misleading, and the statistical support is weak. I encourage the editor to request the authors provide the exact per-metric protocol, a validation-based hyperparameter selection, and confidence intervals. If the authors can make the evaluation reproducible and transparent, the contribution could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: EvalSVA is a plausible application of multi-agent LLM debate to commit-level CVSS v3.1 assessment, and the multilingual dataset (699 C++, 888 Java, 1310 Python commits) is a genuinely useful contribution if released. The framework itself—multiple LLM agents with different communication strategies—is not new, but the domain transfer and the benchmark are. The case study showing agents revising judgments is nice evidence that the mechanism produces interpretable rationales.\n\nThe soft spot is the central quantitative claim. The paper reports that EvalSVA improves F1 by 53.71% for ChatGPT and 32.88% for GPT-4, but Table 3's \"EvalSVA\" rows appear to be assembled by picking, for each metric, the best-performing communication strategy from Table 4. For Python+ChatGPT, every single number in Table 3 matches the maximum across the four strategies in Table 4: AV uses Summarizer Assessment, AC and PR use Previous Communication, and Scope/Confidentiality/Integrity/Availability use Preceding One Expert. The paper never discloses this per-metric selection; it presents \"EvalSVA\" as if it were one fixed method. The single-agent baseline is a single fixed prompt, so the comparison is not apples-to-apples. This is an oracle-selection artifact, not a reproducible multi-agent method. The stress-test note is right, and it lands squarely on the paper's main evidence.\n\nThe remaining issues are less severe but real: test sets are tiny (roughly 90 Java, 131 Python, 69 C++ commits), there are no confidence intervals or significance tests, and the ablation on expert numbers and communication rounds in Figure 3 is tuned on the test set. The dataset validation is thin—20 hand-checked examples per language, two developers, no inter-rater agreement—so label noise is under-examined. No code or data is released, which makes all of this hard to verify.\n\nOn balance: the dataset and problem framing are worth taking seriously, and the multi-agent idea is worth exploring, but the paper as it stands does not support the strength of its own claims. A serious referee should ask for a fixed strategy, a held-out validation split, released artifacts, and significance testing. That is fixable, so I would not desk-reject it, but the authors need to do substantial work.\n\nRecommendation: send to peer review with a request for major revision; the dataset alone justifies referee time.","headline":"Useful new dataset and a plausible multi-agent LLM application, but the headline gains look like per-metric best-strategy selection on the test set, so the central comparison is not yet supported.","tokens_in":15883,"tokens_out":2697,"would_cite":false,"duration_ms":25177,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A team of large language models that discuss a vulnerability commit predicts CVSS v3.1 severity metrics better than any single model, and does it with human-readable rationales.","keywords":["multi-agent LLM","software vulnerability assessment","CVSS v3.1","commit-level vulnerability","multilingual vulnerability dataset","communication strategies","few-shot LLM","explainable security assessment"],"falsifier":"Randomly sample 100 labeled commits per language and have an independent panel of at least three CVSS-trained assessors re-score them; compute inter-rater agreement with the dataset labels. If agreement falls below about 0.6 (for example, Cohen's kappa or Fleiss' kappa), the reported single-agent versus multi-agent differences could be explained by label noise rather than by the multi-agent discussion mechanism.","tokens_in":14794,"feed_emoji":"🛡️","tokens_out":6108,"duration_ms":51818,"temperature":0.7,"pith_summary":"EvalSVA tries to show that software vulnerability assessment—assigning CVSS v3.1 base metrics such as attack vector, privileges required, and impact to a code change—is performed more accurately by a team of large language models that exchange their judgments than by one model acting alone. The paper builds a multilingual dataset of 2,897 vulnerability-related commits (699 C++, 888 Python, 1,310 Java) with full CVSS v3.1 labels, and tests the team on eight metrics. Reported average gains over the single-agent baseline are 44.12% in accuracy and 43.29% in F1, with the per-backbone F1 gain reaching 53.71% for ChatGPT and 32.88% for GPT-4. If correct, the approach converts a scarce-label, multi-factor security task into a few-shot LLM negotiation that produces both a score and a written rationale.","feed_headline":"Multi-agent LLM debate boosts vulnerability scoring F1 by 53%","feed_subtitle":"A chat among expert AI agents beats a single model on CVSS v3.1 metrics, with written rationales.","key_machinery":"The central object is a multi-agent evaluator team: each LLM instance is a vulnerability expert agent with a task-specific prompt, and the agents operate in rounds under one of four communication strategies—preceding one expert, previous communication, simultaneous assessment, or summarizer assessment. The mechanism that carries the argument is the iterative chat history that lets later agents revise earlier judgments; in the case study, Expert 1 changes a wrong 'High' attack-complexity rating to 'Low' after reading Expert 2's rationale for CVE-2023-46502. The framework also includes an adaptive environment that injects the CVSS v3.1 standard text into the prompts, allowing the system to switch standards without retraining.","core_discovery":"The paper's central claim is that a multi-agent evaluator team, in which several LLM experts assess the same vulnerability-related commit and read one another's answers, predicts CVSS v3.1 metrics closer to human-curated labels than any single LLM does. The authors instantiate this as EvalSVA: each LLM agent has a task-specific prompt, and agents communicate over rounds with one of four strategies. Across attack vector, access complexity, privileges required, user interaction, scope, confidentiality, integrity, and availability, EvalSVA beats the single-agent baseline on average accuracy and F1 for both ChatGPT and GPT-4 in all three languages. The paper also reports a qualitative case study in which an agent revises an initially wrong attack-complexity rating after reading another agent's rationale, and an error analysis that identifies XML-related vulnerabilities (CWE-79, CWE-611) as the most frequently misassessed.","pith_inferences":["The debate mechanism may work best for tasks with a single dominant class and clear textual signals; a testable extension is to apply the protocol to CWE classification and measure whether gains concentrate on the XML-related errors the paper flags.","Because the dataset is validated on only 20 manual examples per language with two annotators and no inter-rater statistic, the headline gains may partially be an artifact of label noise; independent re-annotation of a larger sample would clarify.","The superior 'preceding one expert' strategy suggests that recency of context matters more than accumulating all prior answers; this is a specific, falsifiable claim about how LLM attention degrades over long chat histories."],"forward_implications":["A commit-level vulnerability assessment could be produced hours after a patch is written, rather than weeks after a report is filed, because the method needs only the diff and prompt text.","Organizations with few labeled vulnerabilities can still deploy the system, since the framework uses LLMs in a few-shot mode rather than supervised training on large corpora.","Every assessment outputs a written explanation, so a human reviewer can audit the reasoning against the CVSS v3.1 specification, satisfying the FIRST documentation standard.","The finding that 2-3 experts and task-dependent rounds perform best is a concrete tuning guideline for building similar LLM evaluation panels.","The same prompt-driven design can be pointed at future CVSS versions by editing the standard text injected into the adaptive environment, without retraining."],"supporting_citations":[{"why":"Defines the single-agent baseline method whose accuracy and F1 numbers EvalSVA is measured against.","marker":"Yin, Ni, and Wang 2024a"},{"why":"Supplies the earlier commit-level CVSS assessment approach (DeepCVA) that motivates the dataset and task formulation.","marker":"Le et al. 2021"},{"why":"Provides the Big-Vul dataset used as a C/C++ comparison point in the manual dataset-quality evaluation.","marker":"Fan et al. 2020"},{"why":"Provides the D2A dataset used as another comparison point in the dataset-quality evaluation.","marker":"Zheng et al. 2021"},{"why":"Establishes that vulnerability datasets often have quality problems, which is why the paper performs a manual dataset check.","marker":"Croft, Babar, and Kholoosi 2023"},{"why":"Provides one of the two LLM backbones (GPT-4) used as expert agents.","marker":"OpenAI 2023"},{"why":"Provides the other LLM backbone (ChatGPT) used as expert agents.","marker":"ChatGPT 2022"},{"why":"Supplies evidence that multi-agent debate improves factual consistency, backing the choice of expert numbers and rounds.","marker":"Du et al. 2023"}],"fun_headline_variants":["Multi-agent LLM team improves software vulnerability scoring","EvalSVA: Agent debate yields better CVSS predictions","LLM agents chat to assess vulnerabilities, outperforming solo AI","Team of AI evaluators boosts vulnerability assessment accuracy","Multi-agent reasoning improves software vulnerability metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The CVSS v3.1 labels harvested from Mend and the NVD are accurate enough to serve as ground truth for measuring model quality; the manual check of 20 examples per language by two developers, with no inter-rater agreement statistic, leaves considerable room for label noise.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent LLM team improves software vulnerability scoring","EvalSVA: Agent debate yields better CVSS predictions","LLM agents chat to assess vulnerabilities, outperforming solo AI","Team of AI evaluators boosts vulnerability assessment accuracy","Multi-agent reasoning improves software vulnerability metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1732,"prompt_tokens":990,"completion_tokens":742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":667}},"tokens_in":606,"tokens_out":742,"duration_ms":7844,"temperature":1.0,"reasoning_tokens":667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:07:53.351061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample 100 labeled commits per language and have an independent panel of at least three CVSS-trained assessors re-score them; compute inter-rater agreement with the dataset labels. If agreement falls below about 0.6 (for example, Cohen's kappa or Fleiss' kappa), the reported single-agent versus multi-agent differences could be explained by label noise rather than by the multi-agent discussion mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Big-Vul dataset used as a C/C++ comparison point in the manual dataset-quality evaluation."},{"cited_title":"L.; Buratti, L.; Epstein, E","cited_arxiv_id":null,"evidence_quote":"Provides the D2A dataset used as another comparison point in the dataset-quality evaluation."},{"cited_title":"A.; and Kholoosi, M","cited_arxiv_id":null,"evidence_quote":"Establishes that vulnerability datasets often have quality problems, which is why the paper performs a manual dataset check."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the other LLM backbone (ChatGPT) used as expert agents."}],"review_version":1}