REVIEW 4 major objections 6 minor 20 references
Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Ensemble of three GPT-4o prompts fixes 26 of 42 Java vulnerabilities.
desk verdict Useful empirical data on prompt context for GPT-4o vulnerability repair, but the headline Top-3 ensemble rate is an in-sample selection artifact until validated on a holdout. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the composition of prompt variants, each feeding the model a different context bundle: the vulnerable method and class header with other methods' signatures redacted, optional added documentation comments, standardized CWE descriptions, raw CVE advisory text, GPT-generated repair guidelines, and manually extracted project-specific hints. These variants are evaluated by a unit-test oracle from the benchmark, and the ensemble strategy simply takes the union of fixes that pass under any of the top prompts' three runs. That union operation, rather than any single prompt, is what produces the headline 61.9% repair rate.
What would settle it
Re-run the 42-vulnerability experiment with the original, unmodified unit tests and a blinding protocol for the four manually inspected cases (VUL4J-16, 53, 63, 65), and tally per-prompt results. If the five relaxed cases (VUL4J-5, 13, 39, 41, 48) and the four manual cases are where the CVE-enabled prompts gain most of their edge, or if the 26-fix ensemble rate falls to or below the 45.2% single-prompt reproduction, the central claim is refuted.
Extended reading notes
Core claim
The paper evaluates a baseline prompt and nine additional context-enhanced prompt variants, running each with GPT-4o three times on a dataset of 42 reproducible Java vulnerabilities and checking the generated candidates with the benchmark's unit tests. It reports that GPT-4o's per-run average with the original prompt was 11.9% below GPT-4, yet it fixed 19 distinct vulnerabilities versus 17 across three runs. Adding raw CVE descriptions was the best single change, raising distinct fixes to 19; the combination of CVE-derived repair guidelines with manually extracted repository context reached 20 distinct fixes and the highest per-run counts of 18, 14, and 17. Using the top three prompts together, GPT-4o fixed 26 distinct vulnerabilities (61.9%), while using all ten prompts fixed 30 (71.4%), a modest gain the authors judge not worth the extra cost. The paper's answer to its second research question is that diverse prompts are jointly more effective than any one prompt.
Load-bearing premise
The whole comparison rests on the assumption that the relaxed and manually inspected test cases classify fixed-versus-not-fixed the same way they would under the original, unmodified oracle; if those adjustments systematically favor outputs from prompts that mention CVE or manual hints, the reported prompt ranking and the 61.9% ensemble rate would be artifacts of the oracle rather than measures of repair ability.
Editorial extensions
If this is right
- Including a raw CVE description in the prompt is the single cheapest reliable improvement tested, so teams that already have advisory text should add it before trying longer instructions.
- Prompt length and verbosity are not decisive; short prompts perform as well as long ones, so cost-sensitive repair pipelines can prefer compact prompts.
- Project-specific code context alone does not help and can hurt, but it becomes valuable when combined with external vulnerability knowledge, implying context retrieval should pair code with CVE-style guidance.
- Generating several candidates with different prompts raises distinct-fix coverage to 61.9% from 45.2% for the same model, and using all ten prompts reaches 71.4% with diminishing returns.
- A newer model is not automatically a better repairer per run; averaged across runs GPT-4o was 11.9% worse than GPT-4, despite fixing more distinct vulnerabilities overall.
Reading between the lines
- Because the 'at least once' metric is a recall-style union across runs, deployment would need a way to pick among candidate patches without a unit-test oracle; the 61.9% number is an upper envelope, not a per-output success rate.
- The advantage of raw CVE text over GPT-generated hints may reflect the model recognizing specific CVE identifiers from training; this can be tested by using post-training CVEs or anonymized advisories, where memorization is unavailable.
- The paper did not compare prompt ensembling against simply sampling more outputs from one prompt, so it remains open whether diversity of prompts or diversity of samples drives the gain.
- The manually extracted context is a ceiling for what automatic retrieval could hope to achieve; the 21-of-42 cases identified as benefiting from extra project context suggest a concrete target for a retrieval-based repair system, but the paper itself does not show automatic retrieval reaches that ceiling.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how different contextual information affects GPT-4o's zero-shot vulnerability repair performance on 42 Java vulnerabilities from the Vul4J dataset. It replicates a prior GPT-4 baseline, evaluates nine new prompts (varying CWE/CVE information, documentation comments, manually extracted code context, prompt length) with three runs each, and reports that CVE-related information improves repair rates, that combining generated CVE guidelines with manually extracted context gives the best single-prompt performance, and that a Top-3 ensemble of prompts repairs 26 of 42 (61.9%) vulnerabilities at least once, exceeding the GPT-4 baseline (40.4%) and the GPT-4o reproduction (45.2%). The paper concludes that ensemble prompting is a promising zero-shot strategy.
Significance. If validated, the ensemble result would be practically useful: it suggests a cheap, zero-shot way to increase LLM repair rates without fine-tuning. The study's strengths include a substantial evaluation effort (1,134 generated fixes), a transparent threat-to-validity section, and the publication of prompts in an online appendix. The CVE-information finding is plausible and worth pursuing. However, the headline claims are currently not supported by the experimental design: the Top-3 ensemble is selected on the same 42 cases on which it is then scored, no statistical tests back any 'significant' statement, and the GPT-4 versus GPT-4o comparison uses different evaluation pipelines. These are load-bearing issues that must be addressed before the conclusions can be accepted.
major comments (4)
- [Section 4 (Top-3 ensemble paragraph)] The Top-3 prompts are chosen after observing their performance on the same 42 vulnerabilities used to compute the 61.9% ensemble rate. The reported 26/42 is therefore an in-sample maximum over prompt subsets, not an unbiased estimate of any pre-specified ensemble strategy. With nine prompt variants, the best subset is expected to outperform the average prompt even if all prompts are equally effective. Additionally, the ensemble uses 9 generations (3 prompts x 3 runs) while the baseline uses 3 generations of one prompt, and the 'at least once' metric mechanically increases with the number of generated candidates. The comparison needs either a held-out split, cross-validation, or a matched-sample baseline (e.g., 9 runs of the single best prompt or of the baseline prompt) to support the ensemble claim.
- [Section 4.2 and answer to RQ2] The text repeatedly uses the word 'significant' ('CVE information significantly outperforms', 'significantly higher than using individual prompts', 'not significantly better') without reporting any statistical test. With only 42 vulnerabilities and multiple correlated runs, the authors should apply a paired or repeated-measures test (e.g., McNemar's test or Wilcoxon signed-rank) with multiple-comparison correction before making significance claims.
- [Section 3.1.1] The evaluation oracle is not applied uniformly across conditions. Tests were relaxed for VUL4J-5, 13, 39, 41, and 48; manual inspection was used for VUL4J-16, 53, 63, and 65; and manual verification was triggered only when GPT-4 succeeded but GPT-4o did not or when a compilation error occurred. These asymmetric interventions can systematically favor certain prompts over others. The authors should report whether the manual or relaxed cases are balanced across the compared prompt conditions and repeat the main analysis excluding these cases as a sensitivity check.
- [Section 5] The paper acknowledges that the GPT-4 and GPT-4o evaluations used different pipelines, leaving 'the possibility of false negative results in the case of GPT-4o.' This undermines the abstract's claim that 'GPT-4o performed 11.9% worse on average than GPT-4.' The authors should either re-evaluate the GPT-4 outputs with the same automated oracle or present the comparison only as observed per-run counts with the pipeline mismatch clearly disclosed in the abstract and results.
minor comments (6)
- [Table 1] The row 'GPT-4o At least once 27 27 27 30' is confusingly labeled; it appears to be the union rate across all prompts rather than a per-run row. Rename it (e.g., 'All 10 prompts (union over runs)') and clarify the header.
- [Throughout] The dataset name is rendered inconsistently as 'Vul4J' and 'VUL4J'; pick one spelling and use it consistently.
- [Table 1] In the row 'Short + CVE 18 14 17 20Hints + Manual', there is a missing space between '20' and 'Hints'; fix the formatting.
- [Section 4.3] The statement that 'forbidding the model to use certain approaches was also ineffective' and that explicit instructions were required is not supported by quantitative data; add evidence or soften the claim to an anecdotal observation.
- [Front matter] The line 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' appears to be a leftover template; remove it.
- [Section 4.3] For the VUL4J-18 example, the paper says the necessary fix is 'explicitly stated in the documentation comment provided,' but it is not clear which prompt or experimental condition provided that comment; clarify the setup.
Circularity Check
The Top-3 prompt set is selected after seeing results on the same 42 vulnerabilities used for scoring, so the headline 61.9% ensemble rate is an in-sample maximum presented as an independent improvement.
-
fitted input called prediction
[Section 4, 'Answer to RQ2' (Table 1 and accompanying text)]
"Although no prompt could consistently outperform others, we examined aggregating high-performing prompts. Using our Top-3 prompts (leading to nine total executions across all vulnerabilities), we achieved 26 (61.9%) successful repairs in at least one run, exceeding both the original baseline and its GPT-4o reproduction (17 (40.4%) and 19 (45.2%) respectively)."
The 'Top-3' prompts are chosen after observing the per-prompt results on the same 42 vulnerabilities (Table 1: 'Using our Top-3 best performing prompts'), so the reported 61.9% is the union of the three best observed configurations, i.e., an in-sample maximum over prompt subsets. No held-out split, cross-validation, or pre-registered ensemble strategy is presented. The comparison is also confounded because the ensemble uses 9 generations (3 prompts × 3 runs) while the baseline uses 3 generations of a single prompt, and 'at least once' mechanically grows with extra samples. Thus the central RQ2 claim—that an ensemble of prompts outperforms individual prompts—reduces to post-hoc selection on the evaluation set rather than an independent prediction.
full rationale
The paper's per-prompt results (CVE information helps, prompt length matters little, manual context alone is insufficient) are genuine empirical observations and are not circular. The baseline replication of the authors' prior GPT-4 study is also legitimate: the comparison re-executes the same prompts on the same subset, and the cited prior work supplies published responses that are re-evaluated, so this is not load-bearing self-citation. The main circularity concern is confined to the headline ensemble claim. The Top-3 prompts are selected after inspecting their success on the same 42 vulnerabilities used to score the ensemble, making the 61.9% 'at least once' rate an in-sample maximum over the nine tested prompts rather than an unbiased estimate of any specified ensemble strategy. The paper itself notes the baseline used only three generations, acknowledging the sample-count confound. The test-relaxation and manual-inspection oracle adjustments are a validity threat, not a circularity, and are explicitly disclosed in the threats-to-validity section. Overall, the empirical comparisons among fixed prompts are self-contained, but the central RQ2 advantage reduces by construction to post-hoc selection, warranting a partial circularity score of 6.
Assumptions & free parameters
assumptions (4)
- domain assumption Vul4J unit tests, after manual relaxation for five cases and manual inspection for four cases, correctly classify a generated patch as a valid fix.
- domain assumption The manually extracted repository context is representative of what an automated retrieval system would supply and does not encode the fix itself.
- domain assumption CVE and CWE descriptions are not already memorized by GPT-4o in a way that makes the measured improvement a training-data leakage artifact.
- domain assumption Three generations at temperature 0.01 are sufficient to estimate each prompt's success rate.
Cite this review
Pith. "Pith review of Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study." pith.science (2026). https://pith.science/paper/PXHEKYXW
@misc{pith2026250611561,
author = {Pith},
title = {Pith review of: Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/PXHEKYXW}},
note = {Machine review of arXiv:2506.11561}
}
read the original abstract
Recent advancements in large language models (LLMs) have shown promise for automated vulnerability detection and repair in software systems. This paper investigates the performance of GPT-4o in repairing Java vulnerabilities from a widely used dataset (Vul4J), exploring how different contextual information affects automated vulnerability repair (AVR) capabilities. We compare the latest GPT-4o's performance against previous results with GPT-4 using identical prompts. We evaluated nine additional prompts crafted by us that contain various contextual information such as CWE or CVE information, and manually extracted code contexts. Each prompt was executed three times on 42 vulnerabilities, and the resulting fix candidates were validated using Vul4J's automated testing framework. Our results show that GPT-4o performed 11.9\% worse on average than GPT-4 with the same prompt, but was able to fix 10.5\% more distinct vulnerabilities in the three runs together. CVE information significantly improved repair rates, while the length of the task description had minimal impact. Combining CVE guidance with manually extracted code context resulted in the best performance. Using our \textsc{Top}-3 prompts together, GPT-4o repaired 26 (62\%) vulnerabilities at least once, outperforming both the original baseline (40\%) and its reproduction (45\%), suggesting that ensemble prompt strategies could improve vulnerability repair in zero-shot settings.
Reference graph
Works this paper leans on
-
[2]
Sondos Mahmoud Bsharat, Aidar Myrzakhan, and Zhiqiang Shen. 2024. Prin- cipled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4. arXiv:2312.16171 [cs.CL] https://arxiv.org/abs/2312.16171
arXiv 2024
-
[3]
Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. 2022. Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared Towards the Study of Program Repair Techniques. In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR) . 464–468. doi:10.1145/3524842.3528482
arXiv 2022
-
[4]
Common Vulnerabilities and Exposures
CVE 2025. Common Vulnerabilities and Exposures. https://cve.mitre.org/. Ac- cessed: 2025-03-13
work page 2025
-
[5]
CWE 2025. Common Weaknesses Enumeration. https://cwe.mitre.org/. Accessed: 2025-03-13
work page 2025
-
[6]
Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. 2024. Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves. arXiv preprint arXiv:2311.04205 (2024). arXiv:2311.04205 [cs.CL] https://arxiv. org/abs/2311.04205
arXiv 2024
-
[8]
Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. 2024. VulRAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level Retrieval-Augmented Generation. arXiv preprint arXiv:2406.11147 (2024). https://arxiv.org/abs/2406. 11147
arXiv 2024
-
[9]
Mohamad Fakih, Rahul Dharmaji, Halima Bouzidi, Gustavo Quiros Araya, Oluwatosin Ogundare, and Mohammad Abdullah Al Faruque. 2025. LLM4CVE: Enabling Iterative Automated Vulnerability Repair with Large Language Mod- els. arXiv preprint arXiv:2501.03446 (2025). arXiv:2501.03446 [cs.SE] https: //arxiv.org/abs/2501.03446
arXiv 2025
-
[10]
Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. AC/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th international conference on mining software repositories . 508–512
2020
Show all 20 references
-
[11]
Tan Khang Le, Saba Alimadadi, and Steven Y. Ko. 2024. A Study of Vulnera- bility Repair in JavaScript Programs with Large Language Models. In Compan- ion Proceedings of the ACM Web Conference 2024 (WWW ’24) . ACM, 666–669. doi:10.1145/3589335.3651463
2024
-
[12]
Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large Language Models Understand and Can be Enhanced by Emotional Stimuli. arXiv:2307.11760 [cs.CL] https: //arxiv.org/abs/2307.11760
2023 arXiv
-
[13]
Pei Liu, Bo Lin, Yihao Qin, Cheng Weng, and Liqian Chen. 2024. T-RAP: A Template-guided Retrieval-Augmented Vulnerability Patch Generation Approach. In Proceedings of the 15th Asia-Pacific Symposium on Internetware (Macau, China) (Internetware ’24). Association for Computing M...
2024
-
[14]
Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Bren- dan Dolan-Gavitt. 2023. Examining Zero-Shot Vulnerability Repair with Large Language Models. In 2023 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, Los Alamitos, CA, USA, 2339–2356. doi:...
2023
-
[15]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. https: //api.semanticscholar.org/CorpusID:160025533
2019
-
[16]
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv:2402.07927 [cs.AI] https: //arxiv.org/abs/2402.07927
2024 arXiv
-
[17]
Zoltán Ságodi, Gábor Antal, Bence Bogenfürst, Martin Isztin, Péter Hegedűs, and Rudolf Ferenc. 2024. Reality Check: Assessing GPT-4 in Fixing Real- World Software Vulnerabilities. In 28th International Conference on Evalua- tion and Assessment in Software Engineering (EASE ’24...
2024
-
[18]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837
2022
-
[19]
Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. 2023. How Effective Are Neural Networks for Fixing Security Vulnerabilities. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analys...
2023
-
[20]
Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. 2024. Prompt-Enhanced Software Vulnerability Detection Using ChatGPT. InICSE Com- panion 2024 (Lisbon, Portugal) (ICSE-Companion ’24). Association for Computing Machinery, New York, NY, USA, 276–277. do...
2024
-
[21]
Jian Zhang, Chong Wang, Anran Li, Wenhan Wang, Tianlin Li, and Yang Liu. 2024. VulAdvisor: Natural Language Suggestion Generation for Software Vulnerability Repair. In ASE ’24 (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1932–1944. d...
2024
-
[22]
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic Chain of Thought Prompting in Large Language Models.arXiv preprint arXiv:2210.03493 (2022). arXiv:2210.03493 [cs.CL] https://arxiv.org/abs/2210.03493 Received 20 February 2007; revised 12 March 2009; accepted...
2022 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.