{"id":"af0ca75c-14bc-42cf-8f9b-7a1d6aa2bd1b","arxiv_id":"2412.14306","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A user study of 17 developers using an IDE-integrated AI vulnerability scanner on their own projects found high false-positive rates and non-applicable fixes, making the tool not yet practical.","lead":"Researchers built an AI security tool inside Visual Studio Code, had 17 Microsoft developers use it on their own code, and found it still flags too many false positives and suggests fixes that do not fit. The study points to what must change before AI vulnerability scanners are practical in daily development.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark numbers are inflated by selecting prompts on the evaluation sets (SVEN/Vul4J), so the study's leap from DeepVulGuard to 'state-of-the-art tools' is not yet supported.","rationale":"The reader's weakest assumption was participant representativeness (17 Microsoft developers, 56% with static-analysis expertise). That is a legitimate external-validity concern. My analysis identifies a different, internal-validity concern that is at least as load-bearing: the benchmark evidence used to position DeepVulGuard as state-of-the-art is contaminated by prompt selection on the same datasets (SVEN and Vul4J). The paper explicitly says the prompts were chosen by evaluating on these benchmarks, so the reported precision and fix rates are not unbiased estimates. This undermines the bridge from 'DeepVulGuard fails in practice' to 'state-of-the-art AI-powered tools are not practical.' The qualitative findings—users finding many false positives and non-applicable fixes—still support the weaker claim that this tool has serious practical issues. The reader also mentioned prompt selection in the rationale, so there is partial agreement, but it was not identified as the weakest assumption. Since the concern strengthens the case for a conditional verdict rather than overturning it, the verdict remains CONDITIONAL.","tokens_in":16248,"tokens_out":3077,"duration_ms":29802,"concrete_test":"Run a pre-registered evaluation with the exact Figure 3 filter prompt and Figure 4 fix prompt on data withheld before any prompt tuning: for detection, a 20% held-out split of SVEN or an independent dataset such as DiverseVul; for fixes, a fresh split of Vul4J or a different repair benchmark. Record precision, recall, and fix success without further prompt modification. If precision falls below the 80% threshold or fix success diverges materially from 13%, the claim that DeepVulGuard represents SOTA is unsupported, and the central generalization weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's scope is 'state-of-the-art AI-powered detection and fix tools,' which requires DeepVulGuard to be a representative instance. That representativeness rests on the benchmark evidence in Section II-C: 80% precision on SVEN and a 13% fix rate on Vul4J. However, Section II-B states explicitly that the GPT-4 filter prompt was selected by iterating on SVEN, and the fix prompt was selected by iterating on Vul4J and an internal dataset. The reported numbers are therefore tuned on the same benchmarks used to claim 'promising performance' and 'meets the threshold for acceptable false positives.' This is a circularity risk: the real-world drop in precision from 80% to the levels participants saw may reflect prompt overfitting to SVEN's Python, intra-procedural examples rather than a general limitation of SOTA AI tools. The qualitative user feedback is genuine and valuable, but it demonstrates that this particular tool, with these prompts, is not yet practical. Without unbiased benchmark estimates, the broader conclusion about AI-powered tools as a class is not established. The paper's own admission of prompt selection makes this an internal-validity concern, not merely a generalizability one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents DeepVulGuard, a Visual Studio Code extension that combines a fine-tuned CodeBERT detector with GPT-4-based filtering, explanations, and fix generation, and reports a user study in which 17 professional Microsoft developers used the tool on their own projects. The paper first reports benchmark results on SVEN for detection (80% precision, 32% recall, 46% F1) and on Vul4J for repair (13% correct fixes), then describes the think-aloud study, grounded-theory analysis, and survey results. The main finding is that although the tool was perceived as promising, it is not yet practical for real-world use because of a high false-positive rate and fixes that are often not directly applicable. The paper contributes a released tool and data package, a detailed codebook of user feedback, and concrete recommendations for future AI-based vulnerability detection and repair tools.","tokens_in":16479,"tokens_out":7622,"duration_ms":63374,"significance":"If the findings hold, the paper is a valuable and rare user study of deep-learning-based vulnerability detection and repair in a realistic IDE setting, with a relatively large number of professional developers working on their own production code. The qualitative analysis is rich, the saturation argument is plausible, and the released data and code support reproducibility. The study's direct observations about false positives, fix customization, confidence-score use, explanations, and chat interaction are concrete and actionable. However, the benchmark evidence used to position DeepVulGuard as a representative 'state-of-the-art' tool is weakened by prompt selection on the evaluation sets, so the broader conclusion about the class of AI-powered detection and fix tools is not fully established by the current manuscript.","major_comments":[{"comment":"Section II-B states that the GPT-4 filter prompt was selected by trying several prompts on SVEN and that the fix prompt was selected on an internal dataset and Vul4J; Section II-C then reports precision/recall/F1 on SVEN and the fix rate on Vul4J as evidence that DeepVulGuard has 'promising performance' and 'meets the threshold for acceptable false positives.' Because the same benchmarks were used to select the prompts and to report performance, the numbers are selection-biased and cannot serve as an independent estimate of the tool's capability. This is load-bearing because the abstract generalizes from DeepVulGuard to 'state-of-the-art AI-powered detection and fix tools.' Please either add a held-out benchmark not used in prompt selection, or narrow the paper's claims to this tool with these prompts.","section":"Section II-B/II-C"},{"comment":"The paper reports that only 18% of alerts and 25% of fixes were considered useful, and uses this to conclude that the real-world false-positive rate is higher than on SVEN. However, the 51 alerts and 24 fixes are the subset for which participants provided direct feedback during think-aloud, not a random or complete sample of the 170 alerts and 50 fixes generated. Participants may systematically comment more on false positives or on particularly interesting alerts, so the 18% and 25% figures should not be presented as measured rates. Please report all-alert statistics (e.g., from telemetry) or explicitly label Figure 8 as an illustrative qualitative breakdown.","section":"Section IV-A/Figure 8"},{"comment":"The claim that DeepVulGuard 'performs comparably or better than state-of-the-art models' is not supported by the comparisons shown. The F1 comparison to Ding et al. is across different datasets and vulnerability types (C/C++ versus Python), and the fix-rate comparison to APR4Vul and Codex uses small, non-identical intersections (n=24 and n=13) and different evaluation protocols. The 13% Vul4J fix rate, with 42% insertion errors and 37% compile failures, also does not by itself read as 'promising.' Please either provide like-for-like comparisons or moderate the claim to say the tool is comparable in the narrow settings examined.","section":"Section II-C"}],"minor_comments":[{"comment":"The sentence 'One problem was an high rate of false positives' contains a typo and should read 'a high rate.'","section":"Section IV-A"},{"comment":"References [15] and [16] are the same Christakis and Bird paper; please merge or renumber them.","section":"References"},{"comment":"Please report how many prompts were tried and the exact selection criterion, not just 'we tried several prompts,' so that the prompt-selection process is reproducible.","section":"Section II-B"},{"comment":"Add a caption explaining how to interpret the stacked bars and what the 'security tooling' categories mean.","section":"Figure 6"},{"comment":"Report raw comment counts in addition to percentages for the categories in Figure 12, since the percentages are based on small numbers and some categories have very few comments.","section":"Section IV-B/Figure 12"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid user-study contribution and the data release is a plus. The main risk is that the benchmark claims are used to generalize beyond the specific tool; if the authors can provide a held-out evaluation or reframe the claims, I would support publication. The paper may be better positioned in a software-engineering or security venue where qualitative user studies are valued."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the user study, not for the benchmark numbers. The paper is the first user study of an AI vulnerability detection-and-fix tool with professional developers working on code they own, and that part is done carefully enough to be worth taking seriously. The qualitative result—most alerts are false positives because of missing context, and most fixes are not directly applicable because they ignore the developer's existing codebase—is specific, concrete, and consistent across the think-aloud sessions and the grounded-theory analysis. The authors also release their tool and data, which makes the work reproducible and useful for future studies.\n\nThe strongest part is the design recommendations: background scanning, alert suppression, concise explanations with annotations, and separating confidence from severity are all grounded in observed user behavior rather than speculation. The 59% willingness to keep using the tool despite a 2.5/5 usefulness rating is the sort of nuance you do not get from a benchmark.\n\nThe soft spot is the benchmark evidence. Section II-B says the GPT-4 filter prompt was selected by trying several prompts on SVEN, and the fix prompt was selected on Vul4J plus an internal dataset. Then Section II-C reports 80% precision on SVEN and 13% fix rate on Vul4J as if these were independent measurements. They are not. The stress-test note is exactly right: those numbers are tuned, not evaluated. The claim that DeepVulGuard \"meets the threshold for acceptable false positives\" is therefore not established. That matters because the abstract and introduction generalize from DeepVulGuard to \"state-of-the-art AI-powered tools.\" The user study itself shows this particular tool, with these prompts, is not yet practical. That is still a useful finding, and likely the same problems would appear with any similar tool, but the evidence in this paper cannot support the broader class-level claim.\n\nOther weaknesses are smaller and mostly acknowledged: 17 developers from one company, 56% with static-analysis expertise, snowball recruitment, and a tool whose feature set changed mid-study. These are standard limitations for an exploratory study, and the authors are transparent about them.\n\nWho should read it: anyone building or evaluating AI vulnerability tools, and anyone designing user studies for developer tools. It deserves a serious referee. My recommendation is to send it to review with a request that the benchmark section be rewritten to cleanly separate prompt tuning from held-out evaluation, or to drop the benchmark overclaims entirely.","headline":"Read this for the user study, not for the benchmark numbers; the qualitative findings are solid, but the tuned benchmark claims overreach.","tokens_in":17019,"tokens_out":2557,"would_cite":true,"duration_ms":22436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A user study of 17 professional developers finds that state-of-the-art AI vulnerability detection and repair tools are still not practical for real-world use, plagued by false positives and ill-fitted fixes.","keywords":["AI vulnerability detection","automated program repair","user study","IDE integration","deep learning","large language models","false positives","think-aloud study"],"falsifier":"A direct falsifier would be a field study with a random sample of ordinary developers (not static-analysis specialists) measuring, for every alert, whether the developer independently confirms a real vulnerability, and for every fix, whether the developer accepts it unchanged; if the accepted-fix proportion exceeds 50% and the false-positive proportion falls below 30%, the paper's 'not yet practical' conclusion would not hold for that population.","tokens_in":16081,"feed_emoji":"🛡️","tokens_out":5334,"duration_ms":43579,"temperature":0.7,"pith_summary":"This paper reports the first empirical study in which professional developers use an AI-powered vulnerability detection and repair tool on codebases they own. The authors built DeepVulGuard, a Visual Studio Code extension that combines a CodeBERT-based detector with GPT-4 filtering, explanation, and fix generation, and observed 17 Microsoft developers scanning 24 projects for over 1.7 million lines of code. The central finding is that although such tools show genuine promise, they are not yet practical for everyday development: most alerts were false positives caused by missing context, and three-quarters of proposed security fixes could not be applied as-is. Developers still expressed interest in future use (59%), and the paper draws concrete recommendations for deployment, including background scanning, better context awareness, and customizable fixes. A sympathetic reader would take the paper's main contribution to be evidence that benchmark performance is a poor proxy for in-the-field usefulness.","feed_headline":"AI vulnerability tools not ready for real-world devs","feed_subtitle":"17 developers on their own code rate alerts 2.5/5 and fixes 2/5; 59% would still keep using the tool.","key_machinery":"The load-bearing object is DeepVulGuard itself, an IDE-integrated tool whose pipeline is the mechanism under test: a fine-tuned CodeBERT multi-task model that predicts whether code is vulnerable, localizes the vulnerable tokens, and classifies the vulnerability type; a GPT-4 prompt-based filter that removes false alarms and writes natural-language explanations; and a GPT-4 fix generator that suggests code changes shown as diffs alongside a chat panel. The user-study machinery is a think-aloud protocol with grounded-theory coding of 11 hours of interviews, which produces the categorized counts of useful alerts and fixes. Together they carry the argument from benchmark validation (80% precision on SVEN, 13% correct fixes on Vul4J) to the real-world usefulness assessment.","core_discovery":"The paper's central discovery is that state-of-the-art AI vulnerability detection and repair models, embodied in the DeepVulGuard IDE extension, do not yet deliver practical value in real-world development settings. In a think-aloud study with 17 professional Microsoft developers working on their own code, participants rated the tool's alerts at 2.5/5 and its fixes at 2/5 for usefulness, the main causes being a high rate of false positives (51% of examined alerts from missing context, 31% from incorrect pattern recognition) and fixes that were not customized to the codebase (21% non-customized, 21% not addressing the vulnerability, 17% incorrectly inserted). Even so, 59% of participants said they would keep using the extension, and 76% found the vulnerability types relevant. The paper argues that these results expose a gap between benchmark measurements and real-world deployment, and that the field needs more holistic evaluations and context-aware models.","pith_inferences":["One implication the authors do not draw is that the 59% intent-to-use figure, combined with low usefulness ratings, suggests developers may tolerate imperfect tools when the alternative is no security feedback at all—a threshold worth measuring directly.","In our view, if the false-positive rate is driven mainly by missing inter-procedural context, then retrieval-augmented detection that feeds the filter actual call sites and dataflow summaries could sharply reduce false positives, a testable extension.","The paper's complaint that execution-based fix benchmarks miss non-customization implies a new evaluation metric: a fix is 'useful' only if a developer accepts it or accepts it with minor edits, which future studies could measure.","A larger-scale deployment with telemetry, which the paper's tool supports, could turn the qualitative categories into quantitative thresholds, such as the fraction of alerts a developer dismisses without reading."],"forward_implications":["Benchmark scores such as 80% precision overstate real-world usefulness; deployment studies must be part of evaluating detection and repair models.","Detection models need access to calling context and runtime information, not single functions, to reduce false positives.","Fix generation must be customized to the developer's codebase, reusing existing utilities and style, or developers will rewrite suggestions rather than apply them.","Manual scan triggering is a workflow barrier; background scanning integrated into editing or build/commit hooks is needed.","Explanations and confidence scores are valued but must be consistent, concise, and clearly communicate what they mean, or they reduce trust."],"supporting_citations":[{"why":"Supplies the 20% false-positive tolerance threshold used to define whether the tool's precision is acceptable, and its real-world deficit.","marker":"[15]"},{"why":"Provides the CodeBERT multi-task detection method and the large CodeQL-labeled training dataset that DeepVulGuard is built on.","marker":"[12]"},{"why":"The SVEN benchmark used to validate detection precision and recall before the user study.","marker":"[23]"},{"why":"The Vul4J dataset used to evaluate the fix model's ability to produce correct repairs.","marker":"[9]"},{"why":"The prior controlled study with six developers that motivated the present real-world study and shaped expectations for features like localization and quick fixes.","marker":"[22]"},{"why":"Design guidelines for actionable alert placement and workflow integration, used to interpret participants' reactions to the tool.","marker":"[46]"},{"why":"Establishes developer needs for IDE integration, quick fixes, and configurable rulesets, framing the feature-request analysis.","marker":"[27]"}],"fun_headline_variants":["AI vuln tools rated 2/5 on real code by 17 devs","Study: AI security tools not practical despite benchmarks","False positives and bad fixes sink AI vuln tools","AI vulnerability detection not ready for real code, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the reactions of 17 Microsoft developers, over half of whom have static-analysis expertise, are representative of professional developers in general when they judge which alerts and fixes are useful.","fun_headline_variants_meta":{"raw":{"variants":["AI vuln tools rated 2/5 on real code by 17 devs","Study: AI security tools not practical despite benchmarks","False positives and bad fixes sink AI vuln tools","AI vulnerability detection not ready for real code, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2159,"prompt_tokens":1054,"completion_tokens":1105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":1035}},"tokens_in":670,"tokens_out":1105,"duration_ms":9344,"temperature":1.0,"reasoning_tokens":1035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:20:19.394286+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier would be a field study with a random sample of ordinary developers (not static-analysis specialists) measuring, for every alert, whether the developer independently confirms a real vulnerability, and for every fix, whether the developer accepts it unchanged; if the accepted-fix proportion exceeds 50% and the false-positive proportion falls below 30%, the paper's 'not yet practical' conclusion would not hold for that population.","supporting_citations":[{"cited_title":"AIBugHunter: A Practical Tool for Predicting, Classifying and Repairing Software Vulnerabilities","cited_arxiv_id":"2305.16615","evidence_quote":"The prior controlled study with six developers that motivated the present real-world study and shaped expectations for features like localization and quick fixes."},{"cited_title":"Why can’t Johnny fix vulnerabilities: A usability evaluation of static analysis tools for security","cited_arxiv_id":null,"evidence_quote":"Design guidelines for actionable alert placement and workflow integration, used to interpret participants' reactions to the tool."}],"review_version":1}