Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A user study of 17 professional developers finds that state-of-the-art AI vulnerability detection and repair tools are still not practical for real-world use, plagued by false positives and ill-fitted fixes.

desk verdict Read this for the user study, not for the benchmark numbers; the qualitative findings are solid, but the tuned benchmark claims overreach. read the letter →

arxiv 2412.14306 v3 pith:RTOL6NX2 submitted 2024-12-18 cs.SE cs.CRcs.LG

Benjamin Steenhoek , Kalpathy Sivaraman , Renata Saldivar Gonzalez , Yevhen Mohylevskyy , Roshanak Zilouchian Moghaddam , Wei Le This is my paper · ORCID
classification cs.SEcs.CRcs.LG
keywords AIvulnerabilitydetectionautomatedprogramrepairuserstudyIDEintegrationdeeplearninglargelanguagemodelsfalsepositivesthink-aloud
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the first empirical study in which professional developers use an AI-powered vulnerability detection and repair tool on codebases they own. The authors built DeepVulGuard, a Visual Studio Code extension that combines a CodeBERT-based detector with GPT-4 filtering, explanation, and fix generation, and observed 17 Microsoft developers scanning 24 projects for over 1.7 million lines of code. The central finding is that although such tools show genuine promise, they are not yet practical for everyday development: most alerts were false positives caused by missing context, and three-quarters of proposed security fixes could not be applied as-is. Developers still expressed interest in future use (59%), and the paper draws concrete recommendations for deployment, including background scanning, better context awareness, and customizable fixes. A sympathetic reader would take the paper's main contribution to be evidence that benchmark performance is a poor proxy for in-the-field usefulness.

What carries the argument

The load-bearing object is DeepVulGuard itself, an IDE-integrated tool whose pipeline is the mechanism under test: a fine-tuned CodeBERT multi-task model that predicts whether code is vulnerable, localizes the vulnerable tokens, and classifies the vulnerability type; a GPT-4 prompt-based filter that removes false alarms and writes natural-language explanations; and a GPT-4 fix generator that suggests code changes shown as diffs alongside a chat panel. The user-study machinery is a think-aloud protocol with grounded-theory coding of 11 hours of interviews, which produces the categorized counts of useful alerts and fixes. Together they carry the argument from benchmark validation (80% precision on SVEN, 13% correct fixes on Vul4J) to the real-world usefulness assessment.

What would settle it

A direct falsifier would be a field study with a random sample of ordinary developers (not static-analysis specialists) measuring, for every alert, whether the developer independently confirms a real vulnerability, and for every fix, whether the developer accepts it unchanged; if the accepted-fix proportion exceeds 50% and the false-positive proportion falls below 30%, the paper's 'not yet practical' conclusion would not hold for that population.

Watch

Extended reading notes

Core claim

The paper's central discovery is that state-of-the-art AI vulnerability detection and repair models, embodied in the DeepVulGuard IDE extension, do not yet deliver practical value in real-world development settings. In a think-aloud study with 17 professional Microsoft developers working on their own code, participants rated the tool's alerts at 2.5/5 and its fixes at 2/5 for usefulness, the main causes being a high rate of false positives (51% of examined alerts from missing context, 31% from incorrect pattern recognition) and fixes that were not customized to the codebase (21% non-customized, 21% not addressing the vulnerability, 17% incorrectly inserted). Even so, 59% of participants said they would keep using the extension, and 76% found the vulnerability types relevant. The paper argues that these results expose a gap between benchmark measurements and real-world deployment, and that the field needs more holistic evaluations and context-aware models.

Load-bearing premise

The central claim rests on the assumption that the reactions of 17 Microsoft developers, over half of whom have static-analysis expertise, are representative of professional developers in general when they judge which alerts and fixes are useful.

Editorial extensions

If this is right

  • Benchmark scores such as 80% precision overstate real-world usefulness; deployment studies must be part of evaluating detection and repair models.
  • Detection models need access to calling context and runtime information, not single functions, to reduce false positives.
  • Fix generation must be customized to the developer's codebase, reusing existing utilities and style, or developers will rewrite suggestions rather than apply them.
  • Manual scan triggering is a workflow barrier; background scanning integrated into editing or build/commit hooks is needed.
  • Explanations and confidence scores are valued but must be consistent, concise, and clearly communicate what they mean, or they reduce trust.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors do not draw is that the 59% intent-to-use figure, combined with low usefulness ratings, suggests developers may tolerate imperfect tools when the alternative is no security feedback at all—a threshold worth measuring directly.
  • In our view, if the false-positive rate is driven mainly by missing inter-procedural context, then retrieval-augmented detection that feeds the filter actual call sites and dataflow summaries could sharply reduce false positives, a testable extension.
  • The paper's complaint that execution-based fix benchmarks miss non-customization implies a new evaluation metric: a fix is 'useful' only if a developer accepts it or accepts it with minor edits, which future studies could measure.
  • A larger-scale deployment with telemetry, which the paper's tool supports, could turn the qualitative categories into quantitative thresholds, such as the fraction of alerts a developer dismisses without reading.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents DeepVulGuard, a Visual Studio Code extension that combines a fine-tuned CodeBERT detector with GPT-4-based filtering, explanations, and fix generation, and reports a user study in which 17 professional Microsoft developers used the tool on their own projects. The paper first reports benchmark results on SVEN for detection (80% precision, 32% recall, 46% F1) and on Vul4J for repair (13% correct fixes), then describes the think-aloud study, grounded-theory analysis, and survey results. The main finding is that although the tool was perceived as promising, it is not yet practical for real-world use because of a high false-positive rate and fixes that are often not directly applicable. The paper contributes a released tool and data package, a detailed codebook of user feedback, and concrete recommendations for future AI-based vulnerability detection and repair tools.

Significance. If the findings hold, the paper is a valuable and rare user study of deep-learning-based vulnerability detection and repair in a realistic IDE setting, with a relatively large number of professional developers working on their own production code. The qualitative analysis is rich, the saturation argument is plausible, and the released data and code support reproducibility. The study's direct observations about false positives, fix customization, confidence-score use, explanations, and chat interaction are concrete and actionable. However, the benchmark evidence used to position DeepVulGuard as a representative 'state-of-the-art' tool is weakened by prompt selection on the evaluation sets, so the broader conclusion about the class of AI-powered detection and fix tools is not fully established by the current manuscript.

major comments (3)
  1. [Section II-B/II-C] Section II-B states that the GPT-4 filter prompt was selected by trying several prompts on SVEN and that the fix prompt was selected on an internal dataset and Vul4J; Section II-C then reports precision/recall/F1 on SVEN and the fix rate on Vul4J as evidence that DeepVulGuard has 'promising performance' and 'meets the threshold for acceptable false positives.' Because the same benchmarks were used to select the prompts and to report performance, the numbers are selection-biased and cannot serve as an independent estimate of the tool's capability. This is load-bearing because the abstract generalizes from DeepVulGuard to 'state-of-the-art AI-powered detection and fix tools.' Please either add a held-out benchmark not used in prompt selection, or narrow the paper's claims to this tool with these prompts.
  2. [Section IV-A/Figure 8] The paper reports that only 18% of alerts and 25% of fixes were considered useful, and uses this to conclude that the real-world false-positive rate is higher than on SVEN. However, the 51 alerts and 24 fixes are the subset for which participants provided direct feedback during think-aloud, not a random or complete sample of the 170 alerts and 50 fixes generated. Participants may systematically comment more on false positives or on particularly interesting alerts, so the 18% and 25% figures should not be presented as measured rates. Please report all-alert statistics (e.g., from telemetry) or explicitly label Figure 8 as an illustrative qualitative breakdown.
  3. [Section II-C] The claim that DeepVulGuard 'performs comparably or better than state-of-the-art models' is not supported by the comparisons shown. The F1 comparison to Ding et al. is across different datasets and vulnerability types (C/C++ versus Python), and the fix-rate comparison to APR4Vul and Codex uses small, non-identical intersections (n=24 and n=13) and different evaluation protocols. The 13% Vul4J fix rate, with 42% insertion errors and 37% compile failures, also does not by itself read as 'promising.' Please either provide like-for-like comparisons or moderate the claim to say the tool is comparable in the narrow settings examined.
minor comments (5)
  1. [Section IV-A] The sentence 'One problem was an high rate of false positives' contains a typo and should read 'a high rate.'
  2. [References] References [15] and [16] are the same Christakis and Bird paper; please merge or renumber them.
  3. [Section II-B] Please report how many prompts were tried and the exact selection criterion, not just 'we tried several prompts,' so that the prompt-selection process is reproducible.
  4. [Figure 6] Add a caption explaining how to interpret the stacked bars and what the 'security tooling' categories mean.
  5. [Section IV-B/Figure 12] Report raw comment counts in addition to percentages for the categories in Figure 12, since the percentages are based on small numbers and some categories have very few comments.

Circularity Check

2 steps flagged · score 6.0 of 10

SVEN/Vul4J benchmark numbers are prompt-selection results, so the 'promising performance' claim is partly circular; the user-study conclusion itself is independent.

  1. fitted input called prediction [Section II-B (Filtering and explaining alerts with GPT-4) and Section II-C (Evaluating Detection and Fix Capabilities)]
    "We tried several prompts and evaluated on the SVEN benchmark [23], ultimately selecting the prompt shown in Figure 3. This prompt improved the Precision to an acceptable threshold of 80% [16] while keeping the best Recall. ... Figure 5 shows on the SVEN dataset our model achieved 80% Precision and 32% Recall, with an F1 score of 46%."

    The 80% SVEN precision is the optimization target used to select the GPT-4 filter prompt, so reporting it as evidence that the tool 'meets the threshold for acceptable false positives' and is 'promising' is a selection artifact, not an independent benchmark measure. The real-world user-study observations of high false positives remain valid, but the benchmark-side support for 'promising performance' is forced by the prompt-selection procedure.

  2. fitted input called prediction [Section II-B (Prompting GPT-4 for repair and explanation) and Section II-C (Evaluating Detection and Fix Capabilities)]
    "As with the LLM filter, we iterated on several prompts and chose the best performance on an internal dataset of bugs and Vul4J [9]. ... Among the 24 single-hunk bugs with vulnerability types that our tool handles, our model produced 3 (13%) correct fixes and 2 (8%) partial fixes, which resolved the issue but broke 1-3 other tests. ... These results show that our model performs similarly to SOTA evolution-based automated program repair (APR) tools [8] (13% correct fixes...)."

    The reported 13% correct-fix rate on Vul4J is the best result obtained by iterating prompts on Vul4J itself, so comparing that number to APR/LLM tools as evidence of comparable performance is statistically forced. It is not a held-out evaluation of the fix component.

full rationale

The paper's main user-study finding—that AI-powered detection and fix tools are not yet practical due to false positives and non-applicable fixes—is supported by think-aloud interviews, surveys, and grounded-theory coding of 17 developers' actual usage on their own projects. That qualitative evidence is independent of the benchmark tuning under scrutiny. However, the paper also claims 'promising performance' and uses benchmark numbers to frame DeepVulGuard as representative of state-of-the-art tools. Section II-B explicitly states that the GPT-4 filter prompt was selected by evaluating on SVEN and the fix prompt was selected on Vul4J/internal data, and Section II-C then reports SVEN precision and Vul4J fix rate as validation. Those benchmark figures reduce by construction to the prompt-selection criterion, so the 'promising performance' and 'meets the threshold for acceptable false positives' claims are partially circular. Self-citations [12] and [47] are present but are not the load-bearing basis for the usability conclusions; the circularity is concentrated in the prompt-selected benchmark evidence. Overall, this is partial circularity, not a wholesale collapse of the study's central usability result.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central qualitative findings do not depend on fitted numeric parameters, but the auxiliary benchmark validation depends on hand-chosen LLM prompts tuned on the same benchmarks used for reporting. The user-study conclusions assume that think-aloud self-reports from a single-company sample are valid evidence of usefulness.

free parameters (2)
  • LLM filter prompt (Figure 3) = chosen for best Precision/Recall trade-off on SVEN
    Section II-B: 'We tried several prompts and evaluated on the SVEN benchmark, ultimately selecting the prompt shown in Figure 3.' The SVEN results in Section II-C are then reported as model performance, so the benchmark number is not from a held-out prompt.
  • LLM fix prompt (Figure 4) = chosen for best performance on an internal bug dataset and Vul4J
    Section II-B: 'we iterated on several prompts and chose the best performance on an internal dataset of bugs and Vul4J.' The Vul4J fix rate in Section II-C is then reported as the tool's fix capability.
assumptions (3)
  • domain assumption Think-aloud verbalizations and post-interview survey responses measure real-world usefulness.
    Section III-A: the study's RQ1 conclusions rely on participants' self-reports and think-aloud commentary; there is no independent ground truth for usefulness.
  • domain assumption 17 developers from Microsoft, with 56% static-analysis expertise, are enough to represent professional developers.
    Section VI cites Nielsen and Landauer's estimate that 16 users suffice for usability challenges; the sample is a single organization with snowball sampling.
  • domain assumption CodeBERT model trained mostly on Python data transfers sufficiently to C# and TypeScript for a meaningful study.
    Section IV-A notes C# and TypeScript are only 6% of training data; the authors still use the model to generate the alerts studied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE." pith.science (2026). https://pith.science/paper/RTOL6NX2

@misc{pith2026241214306,
  author       = {Pith},
  title        = {Pith review of: Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RTOL6NX2}},
  note         = {Machine review of arXiv:2412.14306}
}
read the original abstract

This paper presents the first empirical study of a vulnerability detection and fix tool with professional software developers on real projects that they own. We implemented DeepVulGuard, an IDE-integrated tool based on state-of-the-art detection and fix models, and show that it has promising performance on benchmarks of historic vulnerability data. DeepVulGuard scans code for vulnerabilities (including identifying the vulnerability type and vulnerable region of code), suggests fixes, provides natural-language explanations for alerts and fixes, leveraging chat interfaces. We recruited 17 professional software developers at Microsoft, observed their usage of the tool on their code, and conducted interviews to assess the tool's usefulness, speed, trust, relevance, and workflow integration. We also gathered detailed qualitative feedback on users' perceptions and their desired features. Study participants scanned a total of 24 projects, 6.9k files, and over 1.7 million lines of source code, and generated 170 alerts and 50 fix suggestions. We find that although state-of-the-art AI-powered detection and fix tools show promise, they are not yet practical for real-world use due to a high rate of false positives and non-applicable fixes. User feedback reveals several actionable pain points, ranging from incomplete context to lack of customization for the user's codebase. Additionally, we explore how AI features, including confidence scores, explanations, and chat interaction, can apply to vulnerability detection and fixing. Based on these insights, we offer practical recommendations for evaluating and deploying AI detection and fix models. Our code and data are available at https://doi.org/10.6084/m9.figshare.26367139.

Figures

Figures reproduced from arXiv: 2412.14306 by the authors.

Figure 1
Figure 1. Overview of DEEPVULGUARD’s user interface on an example program. (1) An editor alert; (2) Problems menu entry; (3) The explanation of the alert; (4a) Quick fix interaction; (4b) Ignore options; (4c) Fix trigger; (5) Suggested fix; (6) Explanation of the fix suggestion; (7) Accept/Reject buttons. precision, 32% recall, and a 46% F1 score on SVEN [23] for vulnerability detection and fixed 13% of vulnerabilities on the… view at source ↗
Figure 2
Figure 2. An overview of DEEPVULGUARD’s detection workflow. (1) Binary classification into vulnerable/not-vulnerable; (2) Localization; (3) Multi-class classification into one of 27 vulnerability types; (4) Alert and explanation shown to the user. in the editor (1) and actionable entries in the Problems window (2), and a natural-language explanation of the vulnerability is shown in the chat panel (3). The user can use this in… view at source ↗
Figure 3
Figure 3. DEEPVULGUARD’s LLM filter prompt. types of annotations). Then we instructed GPT-4 to confirm whether the vulnerability is present. If the answer is Yes, the alert and explanation are shown to the user; otherwise, the alert is not shown ((4) in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Performance of DEEPVULGUARD’s detection compo￾nent on vulnerabilities from SVEN. To evaluate DEEPVULGUARD’s fix component, we used the Vul4J [9] dataset, which includes executable tests to reproduce security vulnerabilities. We assessed the test results, supple￾mented …
Figure 6
Figure 6. Figure 6: Participant demographics and tool adoption. Where [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Summary of participants’ overall perceptions of D [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: A vulnerability that DEEPVULGUARD successfully found and fixed by adding validation logic to ensure that an attacker cannot redirect the user to a malicious third-party site. credentials. We hypothesize that incorporating references to the calling context and runtime e…
Figure 8
Figure 8. Figure 8: Participant responses to LLM-filtered alerts and LLM [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: An example of a false-positive alert caused by missing [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: An example of a non-customized fix. In this case, [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Participants’ in-use feedback on the aspects of D [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: The relative frequency of features suggested by the [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

    cs.CR 2026-07 conditional novelty 6.0 of 10

    DREA improves repository-level vulnerability detection by coupling an LLM planner that forms hypotheses with a cheap local explorer that gathers cross-file evidence, lifting paired accuracy from 19-26% to 30-42% at mu...

  2. Context-Aware CodeLLM Eviction for AI-assisted Coding

    cs.SE 2025-06 reject novelty 5.0 of 10

    CACE, a context-aware eviction policy, cuts code-model reloads and response latency in self-hosted AI-assistant serving compared with LRU, though its future-demand factor reads the actual test workload.

Reference graph

Works this paper leans on

54 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    https://github.com/features/copilot

    GitHub Copilot. https://github.com/features/copilot

  2. [2]

    https://doi.org/10.6084/ m9.figshare.26367139, 2024

    The data package for our study. https://doi.org/10.6084/ m9.figshare.26367139, 2024

    work page 2024

  3. [3]

    https://en.wikipedia

    List of data breaches - Wikipedia. https://en.wikipedia. org/wiki/List of data breaches, 2024

    work page 2024

  4. [4]

    W. Baziuk. BNR/NORTEL: path to improve product quality, reliability and customer satisfaction. In ISSRE,

    work page

  5. [5]

    A few billion lines of code later: using static analysis to find bugs in the real world

    Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. A few billion lines of code later: using static analysis to find bugs in the real world. Communications of the ACM , 2010. doi: 10.1145/1646353.1646374

    work page arXiv 2010

  6. [6]

    Taking Flight with Copilot: Early insights and opportunities of AI-powered pair- programming tools

    Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowder- milk, and Idan Gazit. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair- programming tools. ACM Queue , 2023. doi: 10.1145/ 3582083

    work page 2023

  7. [7]

    Barry W. Boehm. Software Engineering Economics . Springer Berlin Heidelberg, 2002. ISBN 978-3-642- 59412-0

    work page 2002

  8. [8]

    APR4Vul: an empirical study of automatic program repair techniques on real-world java vulnerabilities

    Quang-Cuong Bui, Ranindya Paramitha, Duc-Ly Vu, Fabio Massacci, and Riccardo Scandariato. APR4Vul: an empirical study of automatic program repair techniques on real-world java vulnerabilities. Empirical Software Engineering. doi: 10.1007/s10664-023-10415-7

    doi:10.1007/s10664-023-10415-7

Show all 54 references
  1. [9]

    D ´ıaz Ferreyra

    Quang-Cuong Bui, Riccardo Scandariato, and Nicol ´as E. D ´ıaz Ferreyra. Vul4J: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques. In MSR, 2022. doi: 10.1145/3524842. 3528482

    2022 doi

  2. [10]

    Automatic program repair as se- mantic suggestions: An empirical study

    Diogo Campos, Andr ´e Restivo, Hugo Sereno Ferreira, and Afonso Ramos. Automatic program repair as se- mantic suggestions: An empirical study. In ICST, 2021. doi: 10.1109/ICST49551.2021.00032

    2021

  3. [11]

    Deep learning based vulnerability detec- tion: Are we there yet? IEEE Transactions on Software Engineering, 2021

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detec- tion: Are we there yet? IEEE Transactions on Software Engineering, 2021. doi: 10.1109/TSE.2021.3087402

    2021

  4. [12]

    Transformer-based vulnerability detection in code at edit- time: Zero-shot, few-shot, or fine-tuning?, 2023

    Aaron Chan, Anant Kharkar, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Alec Helyar, Eslam Kamal, Mohamed Elkamhawy, and Neel Sundaresan. Transformer-based vulnerability detection in code at edit- time: Zero-shot, few-shot, or fine-tuning?, 2023. URL https://arxiv.org/ab...

    2023 arXiv

  5. [13]

    Constructing grounded theory: A practi- cal guide through qualitative analysis

    Kathy Charmaz. Constructing grounded theory: A practi- cal guide through qualitative analysis. Sage, 2006. ISBN 0761973532

    2006

  6. [14]

    DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. In RAID, 2023. doi: 10.1145/3607199. 3607242

    2023 doi

  7. [16]

    What developers want and need from program analysis: an empirical study

    Maria Christakis and Christian Bird. What developers want and need from program analysis: an empirical study. In ASE, 2016. doi: 10.1145/2970276.2970347

    2016

  8. [17]

    Vulnerability de- tection with code language models: How far are we?,

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability de- tection with code language models: How far are we?,

  9. [18]

    Dino Distefano, Manuel F ¨ahndrich, Francesco Logozzo, and Peter W. O’Hearn. Scaling static analyses at facebook. Communications of the ACM , 2019. doi: 10.1145/3338112

    2019 doi

  10. [19]

    Selecting Empirical Methods for Software Engineering Research

    Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. Selecting Empirical Methods for Software Engineering Research. Springer London, 2008. ISBN 978-1-84800-044-5

    2008

  11. [20]

    CodeBERT: A pre-trained model for programming and natural lan- guages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural lan- guages. In Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP Findings 2020 , 2...

    2020 doi

  12. [21]

    LineVul: A transformer-based line-level vulnerability prediction

    Michael Fu and Chakkrit Tantithamthavorn. LineVul: A transformer-based line-level vulnerability prediction. In MSR, 2022. doi: 10.1145/3524842.3528452

    2022

  13. [22]

    AIBugHunter: A Practical Tool for Predicting, Classify- ing and Repairing Software Vulnerabilities, 2023

    Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Yuki Kume, Van Nguyen, Dinh Phung, and John Grundy. AIBugHunter: A Practical Tool for Predicting, Classify- ing and Repairing Software Vulnerabilities, 2023. URL http://arxiv.org/abs/2305.16615. arXiv:2305.16615

    2023 arXiv

  14. [23]

    Large language models for code: Security hardening and adversarial testing

    Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In ACM CCS, 2023. URL https://arxiv.org/abs/2302.05319

    2023 arXiv

  15. [24]

    Humphrey

    Watts S. Humphrey. A Discipline for Software Engi- neering. Addison-Wesley Longman Publishing Co., Inc.,

  16. [25]

    Cost of a data breach 2024

    IBM. Cost of a data breach 2024. https://www.ibm.com/ reports/data-breach, 2024

    2024

  17. [26]

    Reporting Experiments in Software En- gineering

    Andreas Jedlitschka, Marcus Ciolkowski, and Diet- mar Pfahl. Reporting Experiments in Software En- gineering. Springer London, 2008. doi: 10.1007/ 978-1-84800-044-5 8

    2008

  18. [27]

    Why don’t software developers use static analysis tools to find bugs? In ICSE, 2013

    Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. Why don’t software developers use static analysis tools to find bugs? In ICSE, 2013. doi: 10.1109/ICSE.2013.6606613

    2013

  19. [28]

    Make your tools sparkle with trust: The PICSE framework for trust in software tools

    Brittany Johnson, Christian Bird, Denae Ford, Nicole Forsgren, and Tom Zimmermann. Make your tools sparkle with trust: The PICSE framework for trust in software tools. In ICSE SEIP , 2023. doi: 10.1109/ ICSE-SEIP58684.2023.00043

    2023

  20. [29]

    Ren ´e Just, Darioush Jalali, and Michael D. Ernst. De- fects4J: a database of existing faults to enable controlled testing studies for java programs. In ISSTA, 2014. doi: 10.1145/2610384.2628055

    2014

  21. [30]

    The Devastating Business Impacts of a Cyber Breach

    Huangm Keman, Xiaoqing Wang, William Wei, and Stuart Madnick. The Devastating Business Impacts of a Cyber Breach. https://hbr.org/2023/05/ the-devastating-business-impacts-of-a-cyber-breach, 2023

    2023

  22. [31]

    thinking-aloud

    Clayton Lewis. Using the “thinking-aloud” method in cognitive interface design . IBM TJ Watson Research Center, 1982

    1982

  23. [32]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt ¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://...

    2021 arXiv

  24. [33]

    Yi Li, Shaohua Wang, and Tien N. Nguyen. Vulnerability detection with fine-grained interpretations. In FSE, 2021. doi: 10.1145/3468264.3468597

    2021

  25. [35]

    Visual Studio Code - Code Editing

    Microsoft. Visual Studio Code - Code Editing. Rede- fined. https://code.visualstudio.com/, 2024

    2024

  26. [36]

    Using an LLM to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an LLM to help with code understanding. In ICSE, 2024. doi: 10.1145/3597503.3639187

    2024

  27. [37]

    Why you only need to test with 5 users

    Jakob Nielsen. Why you only need to test with 5 users. https://www.nngroup.com/articles/ why-you-only-need-to-test-with-5-users/, 2000

    2000

  28. [38]

    Landauer

    Jakob Nielsen and Thomas K. Landauer. A mathematical model of the finding of usability problems. In CHI, 1993. doi: 10.1145/169059.169166

    1993

  29. [39]

    Trust enhancement issues in program repair

    Yannic Noller, Ridwan Shariffdeen, Xiang Gao, and Ab- hik Roychoudhury. Trust enhancement issues in program repair. In ICSE, 2022. doi: 10.1145/3510003.3510040

    2022

  30. [40]

    NVD - Vulnerability Metrics

    NVD. NVD - Vulnerability Metrics. https://nvd.nist.gov/ vuln-metrics/cvss, 2024

    2024

  31. [41]

    Talking about thinking aloud: Perspectives from interactive think-aloud practitioners

    Liam O’Brien and Stephanie Wilson. Talking about thinking aloud: Perspectives from interactive think-aloud practitioners. Journal of User Experience , 2023

    2023

  32. [42]

    GPT-4 techni- cal report, 2024

    OpenAI, Josh Achiam, Steven Adler, et al. GPT-4 techni- cal report, 2024. URL https://arxiv.org/abs/2303.08774. arXiv: 2303.08774

    2024 arXiv

  33. [43]

    Lessons from building static analysis tools at Google

    Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. Lessons from building static analysis tools at Google. Communications of the ACM, 2018. doi: 10.1145/3188720

    2018 doi

  34. [44]

    Carolyn B. Seaman. Qualitative Methods . Springer London, 2008. doi: 10.1007/978-1-84800-044-5 2

    2008 doi

  35. [45]

    Toward improved deep learning-based vul- nerability detection

    Adriana Sejfia, Satyaki Das, Saad Shafiq, and Nenad Medvidovi´c. Toward improved deep learning-based vul- nerability detection. In ICSE, ICSE ’24, 2024. doi: 10.1145/3597503.3608141

    2024

  36. [46]

    Why can’t Johnny fix vulnerabilities: A usability evaluation of static analysis tools for security

    Justin Smith, Lisa Nguyen Quang Do, and Emerson Murphy-Hill. Why can’t Johnny fix vulnerabilities: A usability evaluation of static analysis tools for security

  37. [47]

    An empirical study of deep learning models for vulnerability detection

    Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. An empirical study of deep learning models for vulnerability detection. In ICSE, 2023. doi: 10.1109/ICSE48619.2023.00188

    2023

  38. [49]

    CWE top 25 most dangerous software weaknesses

    The MITRE Corporation. CWE top 25 most dangerous software weaknesses. https://cwe.mitre.org/top25/, 2024

    2024

  39. [50]

    Investigating and Designing for Trust in AI-powered Code Generation Tools, 2023

    Ruotong Wang, Ruijia Cheng, Denae Ford, and Thomas Zimmermann. Investigating and Designing for Trust in AI-powered Code Generation Tools, 2023. URL http: //arxiv.org/abs/2305.11248. arXiv:2305.11248

    2023 arXiv

  40. [51]

    How do developers really feel about bug fixing? directions for automatic program repair

    Emily Winter, David Bowes, Steve Counsell, Tracy Hall, Sæmundur Haraldsson, Vesna Nowack, and John Woodward. How do developers really feel about bug fixing? directions for automatic program repair. IEEE Transactions on Software Engineering , 2023. doi: 10. 1109/TSE.2022.3194188

    2023

  41. [52]

    How effective are neural networks for fixing security vulnerabilities

    Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. How effective are neural networks for fixing security vulnerabilities. In ISSTA, 2023. doi: 10.1145/3597926. 3598135

    2023 doi

  42. [53]

    How does in-context learning work? A framework for understanding the dif- ferences from traditional supervised learning

    Sang Michael Xie and Sewon Min. How does in-context learning work? A framework for understanding the dif- ferences from traditional supervised learning. https: //ai.stanford.edu/blog/understanding-incontext/, 2022

    2022

  43. [54]

    Large language models for test- free fault localization

    Aidan ZH Yang, Claire Le Goues, Ruben Martins, and Vincent Hellendoorn. Large language models for test- free fault localization. In ICSE, 2024. doi: 10.1145/ 3597503.3623342

    2024

  44. [1995]

    doi: 10.1109/ISSRE.1995.497665

    1995

  45. [2020]

    URL https://www

    ISBN 978-1-939133-16-8. URL https://www. usenix.org/conference/soups2020/presentation/smith

  46. [2024]

    arXiv: 2403.18624

    URL https://arxiv.org/abs/2403.18624. arXiv: 2403.18624

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.