Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A team of large language models that discuss a vulnerability commit predicts CVSS v3.1 severity metrics better than any single model, and does it with human-readable rationales.

desk verdict Useful new dataset and a plausible multi-agent LLM application, but the headline gains look like per-metric best-strategy selection on the test set, so the central comparison is not yet supported. read the letter →

arxiv 2501.14737 v1 pith:7ANUKYAE submitted 2024-12-11 cs.SE cs.AI

Xin-Cheng Wen , Jiaxin Ye , Cuiyun Gao , Lianwei Wu , Qing Liao This is my paper · ORCID
classification cs.SEcs.AI
keywords multi-agentLLMsoftwarevulnerabilityassessmentCVSSv3.1commit-levelmultilingualdatasetcommunicationstrategiesfew-shotexplainablesecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EvalSVA tries to show that software vulnerability assessment—assigning CVSS v3.1 base metrics such as attack vector, privileges required, and impact to a code change—is performed more accurately by a team of large language models that exchange their judgments than by one model acting alone. The paper builds a multilingual dataset of 2,897 vulnerability-related commits (699 C++, 888 Python, 1,310 Java) with full CVSS v3.1 labels, and tests the team on eight metrics. Reported average gains over the single-agent baseline are 44.12% in accuracy and 43.29% in F1, with the per-backbone F1 gain reaching 53.71% for ChatGPT and 32.88% for GPT-4. If correct, the approach converts a scarce-label, multi-factor security task into a few-shot LLM negotiation that produces both a score and a written rationale.

What carries the argument

The central object is a multi-agent evaluator team: each LLM instance is a vulnerability expert agent with a task-specific prompt, and the agents operate in rounds under one of four communication strategies—preceding one expert, previous communication, simultaneous assessment, or summarizer assessment. The mechanism that carries the argument is the iterative chat history that lets later agents revise earlier judgments; in the case study, Expert 1 changes a wrong 'High' attack-complexity rating to 'Low' after reading Expert 2's rationale for CVE-2023-46502. The framework also includes an adaptive environment that injects the CVSS v3.1 standard text into the prompts, allowing the system to switch standards without retraining.

What would settle it

Randomly sample 100 labeled commits per language and have an independent panel of at least three CVSS-trained assessors re-score them; compute inter-rater agreement with the dataset labels. If agreement falls below about 0.6 (for example, Cohen's kappa or Fleiss' kappa), the reported single-agent versus multi-agent differences could be explained by label noise rather than by the multi-agent discussion mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-agent evaluator team, in which several LLM experts assess the same vulnerability-related commit and read one another's answers, predicts CVSS v3.1 metrics closer to human-curated labels than any single LLM does. The authors instantiate this as EvalSVA: each LLM agent has a task-specific prompt, and agents communicate over rounds with one of four strategies. Across attack vector, access complexity, privileges required, user interaction, scope, confidentiality, integrity, and availability, EvalSVA beats the single-agent baseline on average accuracy and F1 for both ChatGPT and GPT-4 in all three languages. The paper also reports a qualitative case study in which an agent revises an initially wrong attack-complexity rating after reading another agent's rationale, and an error analysis that identifies XML-related vulnerabilities (CWE-79, CWE-611) as the most frequently misassessed.

Load-bearing premise

The CVSS v3.1 labels harvested from Mend and the NVD are accurate enough to serve as ground truth for measuring model quality; the manual check of 20 examples per language by two developers, with no inter-rater agreement statistic, leaves considerable room for label noise.

Editorial extensions

If this is right

  • A commit-level vulnerability assessment could be produced hours after a patch is written, rather than weeks after a report is filed, because the method needs only the diff and prompt text.
  • Organizations with few labeled vulnerabilities can still deploy the system, since the framework uses LLMs in a few-shot mode rather than supervised training on large corpora.
  • Every assessment outputs a written explanation, so a human reviewer can audit the reasoning against the CVSS v3.1 specification, satisfying the FIRST documentation standard.
  • The finding that 2-3 experts and task-dependent rounds perform best is a concrete tuning guideline for building similar LLM evaluation panels.
  • The same prompt-driven design can be pointed at future CVSS versions by editing the standard text injected into the adaptive environment, without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The debate mechanism may work best for tasks with a single dominant class and clear textual signals; a testable extension is to apply the protocol to CWE classification and measure whether gains concentrate on the XML-related errors the paper flags.
  • Because the dataset is validated on only 20 manual examples per language with two annotators and no inter-rater statistic, the headline gains may partially be an artifact of label noise; independent re-annotation of a larger sample would clarify.
  • The superior 'preceding one expert' strategy suggests that recency of context matters more than accumulating all prior answers; this is a specific, falsifiable claim about how LLM attention degrades over long chat histories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EvalSVA, a multi-agent LLM framework for CVSS v3.1 vulnerability assessment from vulnerability-related commits. It constructs a multilingual dataset (C++, Java, Python) based on Mend/NVD labels, evaluates four communication strategies for agents, and compares the multi-agent approach against a single-agent baseline using ChatGPT and GPT-4. The authors report large average improvements in accuracy and F1 (44.12% and 43.29% in the abstract; 53.71% and 32.88% F1 for ChatGPT and GPT-4 in the experiments section) and claim that EvalSVA provides human-like, interpretable assessments. The paper also includes a case study, an analysis of expert counts and communication rounds, and a short limitation section.

Significance. If the reported results are valid, the paper makes a useful practical contribution: it is among the first to apply multi-agent LLM deliberation to CVSS metric prediction, it provides a new multilingual benchmark for the task, and it generates rationales for predictions, which is aligned with the interpretability needs of vulnerability assessment. The dataset itself, if cleaned and properly validated, could be a valuable resource. However, the current evaluation protocol has load-bearing weaknesses: the main EvalSVA results appear to be assembled from different communication strategies per metric rather than from a single reproducible configuration, the ground-truth validation is based on only 20 examples per language, the test sets are small (roughly 90/131/69 commits), and no confidence intervals or significance tests are reported. These issues materially affect the central claim of superiority over single-agent baselines.

major comments (4)
  1. [Experiments, 'EvalSVA Results' and 'Communication Strategy' (Tables 3–4)] The EvalSVA rows in Table 3 do not correspond to any single communication strategy described in the 'Communication Strategy' subsection. For Python+ChatGPT, the AV entry (0.3282/0.2014) matches the Summarizer Assessment row in Table 4, the AC entry (0.4351/0.2510) matches Previous Communication, and the PR, Scope, Confidentiality, Integrity, and Availability entries (0.5954/0.4761, 0.4504/0.4386, 0.5115/0.4151, 0.4733/0.3914, 0.4580/0.3432) match Preceding One Expert. The paper does not state that the EvalSVA row is the per-metric optimum, nor does it provide a selection rule. As written, the comparison against the 'Single' baseline is not a comparison of a well-defined, reproducible multi-agent method, and the headline gains may be inflated by post-hoc per-metric selection of the best strategy. Please specify the exact strategy used for each reported EvalSVA number (or use a principled selection on a validation split) and rerun the comparison accordingly.
  2. [Table 2, 'Dataset Evaluation'] The manual validation of the dataset uses only 20 randomly selected examples per language, independently checked by two developers with no inter-rater agreement statistic. Since the Mend/NVD labels are the ground truth for every accuracy and F1 computation, a 20-example sample is too small to establish label quality; for C++ the reported 90.0% accuracy is exactly 18 of 20 correct, giving a wide confidence interval. Please report a larger validation sample, per-metric agreement rates, and an inter-rater reliability measure, or otherwise provide a statistically justified bound on label noise.
  3. [Table 3, 'EvalSVA Results'] The test sets are small (about 90 Java, 131 Python, and 69 C++ commits, as implied by the accuracy denominators in Table 3), and no confidence intervals, standard errors, or significance tests are provided. Many per-metric differences between the single agent and EvalSVA are only a few percentage points, so the claim that EvalSVA 'significantly enhances' SV assessment is not statistically supported. Please add confidence intervals or bootstrap/significance tests, and consider whether the test sets can be enlarged.
  4. [Figure 3, 'Expert Numbers and Communication Rounds'] The analysis of expert counts and communication rounds selects the optimal values based on performance on the Java dataset (e.g., 'the optimal performance occurring when the number of experts is 2-3' and 'PR exhibits optimal performance after three rounds'). If these selected values are used in the main EvalSVA results in Table 3, the comparison is optimistically biased because the configuration is chosen on the test data. Please use a validation split for any hyperparameter selection, or state clearly that the Table 3 results use a fixed, pre-specified configuration.
minor comments (5)
  1. [Table 1 vs. Abstract/Introduction/Conclusion] Table 1 reports 689 C++ commits, while the Abstract, Introduction, and Conclusion all state 699; please make these numbers consistent.
  2. [Table 2] Table 2 would benefit from reporting the raw agreement counts and an inter-rater reliability statistic (e.g., Cohen's kappa), as well as a confidence interval for each language's accuracy.
  3. [Table 3 caption] The Table 3 caption contains a typo ('Webold' should be 'We bold'), and it is unclear what the boldface indicates when only two methods are shown in that table.
  4. [Related Work / References] The same work by Yin, Ni, and Wang is cited twice as both 2024a and 2024b; the duplicate citation should be removed.
  5. [Experiments, 'EvalSVA Results'] The phrase 'average increases of 72.35% and 49.35%' for the exploitability aspect is ambiguous: it does not state whether these are relative or absolute changes, nor over which languages the average is taken; please clarify.

Circularity Check

1 steps flagged · score 6.0 of 10

Table 3's EvalSVA gains are the per-metric best entries of Table 4, so the headline F1 improvement is an oracle-selection artifact rather than a fixed-method prediction.

  1. fitted input called prediction [Experiments, 'EvalSVA Results' Table 3 vs 'Communication Strategy' Table 4 (Python/ChatGPT rows)]
    "EvalSV A significantly enhances the performance of the SV assessment process... the multi-agent-based method improves the F1 by 53.71% for ChatGPT... We experiment with these strategies in Python and the detailed results are described in Table 4... We bold the best-performing communication strategy for each metric."

    For Python/ChatGPT, every EvalSVA cell in Table 3 equals the bolded (best-F1) cell of Table 4 for that metric: AV=Summarizer (0.3282/0.2014), AC=Previous Communication (0.4351/0.2510), PR=Previous Communication (0.5954/0.4761), UI=Summarizer (0.4504/0.4496), and Scope/Conf/Integ/Avail=Preceding One Expert (0.4504/0.4386, 0.5115/0.4151, 0.4733/0.3914, 0.4580/0.3432). The paper never states that Table 3 is produced by per-metric post-hoc selection, and it reports no validation-based strategy choice. Thus the 'EvalSVA' row is an oracle over the four strategies on the same test set, while the baseline is a single fixed prompt. The headline 53.71% F1 improvement is therefore forced by taking the best strategy per metric, not by a reproducible multi-agent method.

full rationale

The only circularity-adjacent step is the construction of Table 3's EvalSVA numbers. The numerical match to Table 4's bolded per-metric best entries is exact for all eight Python/ChatGPT metrics, and the resulting average exceeds every single fixed strategy row in Table 4. Because the paper does not disclose any held-out strategy-selection procedure, the reported EvalSVA result is best understood as per-metric selection on the test set, i.e., a fitted input presented as a prediction. This affects the central claim (53.71% F1 improvement). No other circularity was found: dataset labels come from external sources, the single-agent baseline is an external LLM prompt, and self-citations are not load-bearing. Score is 6 rather than 8 because the multi-agent framework and dataset still have independent content; only the headline comparison reduces by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical entities. The free parameters are design choices (agent count, rounds, strategy, prompts) selected on the test set, and the axioms are the unverified assumptions about label quality, output parsing, and the causal role of deliberation.

free parameters (4)
  • number_of_expert_agents = 2-3
    Figure 3 shows performance peaks at 2-3 experts on the Java test set; this count is selected after seeing test results and is task-dependent.
  • communication_rounds = 1-3
    Optimal rounds are read off test curves (e.g., AV peaks at round 1, PR at round 3); no holdout protocol is reported.
  • communication_strategy_per_task = best strategy per metric (e.g., Preceding One Expert for most scope and impact metrics)
    Table 4 compares four strategies and the main Table 3 does not state which strategy is used, suggesting best-on-test selection.
  • prompt_templates = manually crafted, not disclosed
    Task-specific prompts encode CVSS definitions and likely materially affect outputs; they are not included in the provided text.
assumptions (4)
  • domain assumption CVSS v3.1 labels from Mend and NVD are correct ground truth.
    The dataset is filtered to complete v3.1 labels, but label quality is only checked manually on 20 examples per language with two developers and no inter-rater agreement metric.
  • domain assumption LLM outputs can be reliably parsed into the eight CVSS metric categories.
    No parsing, validation, or rejection procedure for malformed or non-compliant agent answers is described.
  • ad hoc to paper The 20-example manual validation sample per language is representative of dataset quality.
    The sample is small and no sampling strategy, confidence intervals, or developer agreement statistics are given; it is used to claim dataset accuracy of 65-90%.
  • domain assumption Multi-agent discussion improves decision quality rather than merely shifting answers toward the most recent or longest prior response.
    No ablation controls for prompt length, position bias, or model variance; the design assumes deliberation is the cause of any improvement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment." pith.science (2026). https://pith.science/paper/7ANUKYAE

@misc{pith2026250114737,
  author       = {Pith},
  title        = {Pith review of: EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ANUKYAE}},
  note         = {Machine review of arXiv:2501.14737}
}
read the original abstract

Software Vulnerability (SV) assessment is a crucial process of determining different aspects of SVs (e.g., attack vectors and scope) for developers to effectively prioritize efforts in vulnerability mitigation. It presents a challenging and laborious process due to the complexity of SVs and the scarcity of labeled data. To mitigate the above challenges, we introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of SV assessment. Specifically, we propose a multi-agent-based framework to simulate vulnerability assessment strategies in real-world scenarios, which employs multiple Large Language Models (LLMs) into an integrated group to enhance the effectiveness of SV assessment in the limited data. We also design diverse communication strategies to autonomously discuss and assess different aspects of SV. Furthermore, we construct a multi-lingual SV assessment dataset based on the new standard of CVSS, comprising 699, 888, and 1,310 vulnerability-related commits in C++, Python, and Java, respectively. Our experimental results demonstrate that EvalSVA averagely outperforms the 44.12\% accuracy and 43.29\% F1 for SV assessment compared with the previous methods. It shows that EvalSVA offers a human-like process and generates both reason and answer for SV assessment. EvalSVA can also aid human experts in SV assessment, which provides more explanation and details for SV assessment.

Figures

Figures reproduced from arXiv: 2501.14737 by the authors.

Figure 1
Figure 1. Figure (a) presents the vulnerability-related commit of CVE-2023-2954 (Detail 2024a). The code shaded in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Communication strategy for SV assessment. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The impact of expert numbers and communication [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A EvalSVA example presents a discussion process [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

    cs.SE 2026-07 accept novelty 6.0 of 10

    A systematic review of 141 papers derives a three-axis taxonomy of multi-agent debate design (participants, interaction, agreement) and shows the field has converged on a narrow default pattern.

Reference graph

Works this paper leans on

59 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    What is CVSS score

    2023. What is CVSS score. https://debricked.com/blog/what-is-cvss-score/

    work page 2023

  4. [4]

    Common Vulnerability Scoring System (CVSS)

    2024 a . Common Vulnerability Scoring System (CVSS). https://www.first.org/cvss/

    work page 2024

  5. [5]

    Common Vulnerability Scoring System SIG

    2024. Common Vulnerability Scoring System SIG. https://www.first.org/cvss/

    work page 2024

  6. [6]

    Common Vulnerability Scoring System v3.0: Specification Document

    2024 b . Common Vulnerability Scoring System v3.0: Specification Document. https://www.first.org/cvss/v3.0/specification-document

    work page 2024

  7. [7]

    Common Vulnerability Scoring System v3.1: Specification Document

    2024 c . Common Vulnerability Scoring System v3.1: Specification Document. https://www.first.org/cvss/v3.1/specification-document

    work page 2024

  8. [8]

    A Complete Guide to the Common Vulnerability Scoring System Version 2.0

    2024 d . A Complete Guide to the Common Vulnerability Scoring System Version 2.0. https://www.first.org/cvss/v2/guide

    work page 2024

Show all 59 references
  1. [9]

    CWE-611: Improper Restriction of XML External Entity Reference

    2024 a . CWE-611: Improper Restriction of XML External Entity Reference. https://cwe.mitre.org/data/definitions/611.html

    2024

  2. [10]

    CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')

    2024 b . CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting'). https://cwe.mitre.org/data/definitions/79.html

    2024

  3. [11]

    Common Vulnerabilities and Exposures (CVE)

    2024. “ Common Vulnerabilities and Exposures (CVE) ”. https://cve.mitre.org/

    2024

  4. [12]

    Bilge, L.; and Dumitras, T. 2012. Before we knew it: an empirical study of zero-day attacks in the real world. In Yu, T.; Danezis, G.; and Gligor, V. D., eds., the ACM Conference on Computer and Communications Security, CCS'12, Raleigh, NC, USA, October 16-18, 2012 , 833--844. ACM

    2012

  5. [13]

    Bosu, A.; and Carver, J. C. 2012. Peer code review in open source communitiesusing reviewboard. In Murphy - Hill, E. R.; Sadowski, C.; and Markstrum, S., eds., Proceedings of the ACM 4th Annual Workshop on Evaluation and Usability of Programming Languages and Tools, PLATEAU 20...

    2012

  6. [14]

    Chan, C.; Chen, W.; Su, Y.; Yu, J.; Xue, W.; Zhang, S.; Fu, J.; and Liu, Z. 2023. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. CoRR, abs/2308.07201

    2023 arXiv

  7. [15]

    ChatGPT. 2022. ChatGPT. https://chat.openai.com/

    2022

  8. [16]

    A.; and Kholoosi, M

    Croft, R.; Babar, M. A.; and Kholoosi, M. M. 2023. Data Quality for Software Vulnerability Datasets. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 , 121--133. IEEE

    2023

  9. [17]

    Croft, R.; Newlands, D.; Chen, Z.; and Babar, M. A. 2021. An Empirical Study of Rule-Based and Learning-Based Approaches for Static Application Security Testing. In Lanubile, F.; Kalinowski, M.; and Baldassarre, M. T., eds., ESEM '21: ACM / IEEE International Symposium on Empi...

    2021

  10. [18]

    S.; Yang, C.; Zhang, S

    Deng, Y.; Xia, C. S.; Yang, C.; Zhang, S. D.; Yang, S.; and Zhang, L. 2024. Large Language Models are Edge-Case Generators: Crafting Unusual Programs for Fuzzing Deep Learning Libraries. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE...

    2024

  11. [19]

    Detail, C.-.-. 2024 a . https://nvd.nist.gov/vuln/detail/CVE-2023-2954/

    2024

  12. [20]

    Detail, C.-.-. 2024 b . https://nvd.nist.gov/vuln/detail/CVE-2023-46502

    2024

  13. [21]

    Dissanayake, N.; Jayatilaka, A.; Zahedi, M.; and Babar, M. A. 2022. An Empirical Study of Automation in Software Security Patch Management. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022 , 7:1--7:13. ACM

    2022

  14. [22]

    B.; and Mordatch, I

    Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. CoRR, abs/2305.14325

    2023 arXiv

  15. [23]

    Fan, J.; Li, Y.; Wang, S.; and Nguyen, T. N. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Kim, S.; Gousios, G.; Nadi, S.; and Hejderup, J., eds., MSR '20: 17th International Conference on Mining Software Repositories, Seoul, Republic of Kore...

    2020

  16. [24]

    Feutrill, A.; Ranathunga, D.; Yarom, Y.; and Roughan, M. 2018. The Effect of Common Vulnerability Scoring System Metrics on Vulnerability Exploit Delay. In Sixth International Symposium on Computing and Networking, CANDAR 2018, Takayama, Japan, November 23-27, 2018 , 1--10. IE...

    2018

  17. [25]

    Foreman., P. 2019. Vulnerability management. Auerbach Publications

    2019

  18. [26]

    Q.; and Grundy, J

    Fu, M.; Tantithamthavorn, C.; Le, T.; Kume, Y.; Nguyen, V.; Phung, D. Q.; and Grundy, J. C. 2024. AIBugHunter: A Practical tool for predicting, classifying and repairing software vulnerabilities. Empir. Softw. Eng., 29(1): 4

    2024

  19. [27]

    Gao, S.; Wen, X.; Gao, C.; Wang, W.; Zhang, H.; and Lyu, M. R. 2023. What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023 , 761-...

    2023

  20. [28]

    Han, Z.; Li, X.; Xing, Z.; Liu, H.; and Feng, Z. 2017. Learning to Predict Severity of Software Vulnerability Using Only Vulnerability Description. In 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017, Shanghai, China, September 17-22, 2017 ,...

    2017

  21. [29]

    J.; Lam, M

    Huang, J.; Li, E. J.; Lam, M. H.; Liang, T.; Wang, W.; Yuan, Y.; Jiao, W.; Wang, X.; Tu, Z.; and Lyu, M. R. 2024. How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments. CoRR, abs/2403.11807

    2024 arXiv

  22. [30]

    M.; Chatterjee, P.; and Damevski, K

    Imran, M. M.; Chatterjee, P.; and Damevski, K. 2023. Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs. CoRR, abs/2312.09731

    2023 arXiv

  23. [31]

    Karpinska, M.; Akoury, N.; and Iyyer, M. 2021. The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation. In Moens, M.; Huang, X.; Specia, L.; and Yih, S. W., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2...

    2021

  24. [32]

    Khan, S.; and Parkinson, S. 2018. Review into State of the Art of Vulnerability Assessment using Artificial Intelligence. In Parkinson, S.; Crampton, A.; and Hill, R., eds., Guide to Vulnerability Analysis for Computer Networks and Systems - An Artificial Intelligence Approach...

    2018

  25. [33]

    K.; Chen, J.; Zhou, M.; Mensah, S.; and Huang, R

    Kudjo, P. K.; Chen, J.; Zhou, M.; Mensah, S.; and Huang, R. 2019. Improving the Accuracy of Vulnerability Report Classification Using Term Frequency-Inverse Gravity Moment. In 19th IEEE International Conference on Software Quality, Reliability and Security, QRS 2019, Sofia, Bu...

    2019

  26. [34]

    Lamkanfi, A.; Demeyer, S.; Giger, E.; and Goethals, B. 2010. Predicting the severity of a reported bug. In Whitehead, J.; and Zimmermann, T., eds., Proceedings of the 7th International Working Conference on Mining Software Repositories, MSR 2010 (Co-located with ICSE), Cape To...

    2010

  27. [35]

    Le, T. H. M.; and Babar, M. A. 2022. On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models. In 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pittsburgh, PA, USA, May 23-24, 2022 , 621--633. ACM

    2022

  28. [36]

    Le, T. H. M.; Chen, H.; and Babar, M. A. 2023. A Survey on Data-driven Software Vulnerability Assessment and Prioritization. ACM Comput. Surv. , 55(5): 100:1--100:39

    2023

  29. [37]

    Le, T. H. M.; Hin, D.; Croft, R.; and Babar, M. A. 2021. DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021 , ...

    2021

  30. [38]

    Le, T. H. M.; Sabir, B.; and Babar, M. A. 2019. Automated software vulnerability assessment with concept drift. In Storey, M. D.; Adams, B.; and Haiduc, S., eds., Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montre...

    2019

  31. [39]

    Li, G.; Hammoud, H. A. A. K.; Itani, H.; Khizbullin, D.; and Ghanem, B. 2023 a . CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. CoRR, abs/2303.17760

    2023 arXiv

  32. [40]

    Li, Y.; Yadavally, A.; Zhang, J.; Wang, S.; and Nguyen, T. N. 2023 b . Commit-Level, Neural Vulnerability Detection and Assessment. In Chandra, S.; Blincoe, K.; and Tonella, P., eds., Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on t...

    2023

  33. [41]

    Liang, T.; He, Z.; Jiao, W.; Wang, X.; Wang, Y.; Wang, R.; Yang, Y.; Tu, Z.; and Shi, S. 2023. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. CoRR, abs/2305.19118

    2023 arXiv

  34. [42]

    National Vulnerability Database (NVD)

    NIST. 2024. “ National Vulnerability Database (NVD) ”. https://nvd.nist.gov/

    2024

  35. [43]

    N.; Pretschner, A.; and Kulkarni, P

    Ognawala, S.; Amato, R. N.; Pretschner, A.; and Kulkarni, P. 2018. Automatically assessing vulnerabilities discovered by compositional analysis. In Perrouin, G.; Acher, M.; Cordy, M.; and Devroey, X., eds., Proceedings of the 1st International Workshop on Machine Learning and ...

    2018

  36. [44]

    OpenAI. 2023. GPT-4 Technical Report. CoRR, abs/2303.08774

    2023 arXiv

  37. [45]

    Peng, Y.; Wang, C.; Wang, W.; Gao, C.; and Lyu, M. R. 2023. Generative Type Inference for Python. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023 , 988--999. IEEE

    2023

  38. [46]

    D.; Guimard, Q.; Bissyand \' e , T

    Sawadogo, A. D.; Guimard, Q.; Bissyand \' e , T. F.; Kabor \' e , A. K.; Klein, J.; and Moha, N. 2021. Early Detection of Security-Relevant Bug Reports using Machine Learning: How Far Are We? CoRR, abs/2112.10123

    2021 arXiv

  39. [47]

    Smyth, V. 2017. Software vulnerability management: how intelligence helps reduce the risk. Netw. Secur., 2017(3): 10--12

    2017

  40. [48]

    Spanos, G.; and Angelis, L. 2018. A multi-target approach to estimate software vulnerability characteristics and severity scores. J. Syst. Softw., 146: 152--166

    2018

  41. [49]

    Statista. 2024. Number of common IT security vulnerabilities and exposures (CVEs) worldwide from 2009 to 2024 YTD. https://www.statista.com/statistics/500755/worldwide-common-vulnerabilities-and-exposures/

    2024

  42. [50]

    E.; and Iida, H

    Thongtanunam, P.; McIntosh, S.; Hassan, A. E.; and Iida, H. 2015. Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System. In Penta, M. D.; Pinzger, M.; and Robbes, R., eds., 12th IEEE/ACM Working Conference on Mining Software Repositories, ...

    2015

  43. [51]

    Thung, F.; Lo, D.; Jiang, L.; Lucia; Rahman, F.; and Devanbu, P. T. 2012. When would this bug get reported? In 28th IEEE International Conference on Software Maintenance, ICSM 2012, Trento, Italy, September 23-28, 2012 , 420--429. IEEE Computer Society

    2012

  44. [52]

    Wang, P.; Zhou, Y.; Sun, B.; and Zhang, W. 2019. Intelligent Prediction of Vulnerability Severity Level Based on Text Mining and XGBboost. In Eleventh International Conference on Advanced Computational Intelligence, ICACI 2019, Guilin, China, June 7-9, 2019 , 72--77. IEEE

    2019

  45. [53]

    M end bolt

    WhiteSource . 2023. “ M end bolt”. https://www.mend.io/free-developer-tools/

    2023

  46. [54]

    Yamamoto, Y.; Miyamoto, D.; and Nakayama, M. 2015. Text-Mining Approach for Estimating Vulnerability Score. In 4th International Workshop on Building Analysis Datasets and Gathering Experience Returns for Security, BADGERS@RAID 2015, Kyoto, Japan, November 5, 2015, 67--73. IEEE

    2015

  47. [56]

    Yin, X.; Ni, C.; and Wang, S. 2024 b . Multitask-based Evaluation of Open-Source LLM on Software Vulnerability. CoRR, abs/2404.02056

    2024 arXiv

  48. [57]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Le...

    2023

  49. [58]

    L.; Buratti, L.; Epstein, E

    Zheng, Y.; Pujar, S.; Lewis, B. L.; Buratti, L.; Epstein, E. A.; Yang, B.; Laredo, J.; Morari, A.; and Su, Z. 2021. D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis. In 43rd IEEE/ACM International Conference on Software Engineering:...

    2021

  50. [59]

    Zhou, J.; Pacheco, M.; Wan, Z.; Xia, X.; Lo, D.; Wang, Y.; and Hassan, A. E. 2021. Finding A Needle in a Haystack: Automated Mining of Silent Vulnerability Fixes. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, Novem...

    2021

  51. [60]

    K.; Wang, C.; Liu, S.; and Liu, Y

    Zhou, Y.; Siow, J. K.; Wang, C.; Liu, S.; and Liu, Y. 2022. SPI: Automated Identification of Security Patches via Commits. ACM Trans. Softw. Eng. Methodol. , 31(1): 13:1--13:27

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.