REVIEW 4 major objections 5 minor 27 references
Real-world ChatGPT code is frequently insecure, and users rarely ask for secure code — even when the model produces vulnerable or hallucinated output.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
2026-08-03 18:36 UTC pith:CK4X3352
load-bearing objection Real-world WildChat data confirms LLM-generated code is often insecure, but the paper's distinctive user-intent claim rests on an unvalidated classifier. the 4 major comments →
WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that, at scale, code generated by ChatGPT during real user interactions exhibits frequent, exploitable security weaknesses: 20.61% of conversations involving hash functions triggered weakness rules (mostly MD5/SHA1 or unauthenticated crypto), 14.85% of C/C++ snippets used unsafe memory functions, about 29.4% of 1,203 regexes were flagged as ReDoS-vulnerable by static detectors, and 14.4% of Python imports were hallucinated. The same analysis of the English-language subset finds that users almost never request security: the 'Secure Coding' intent category received only 21 initial labels out of tens of thousands of conversations, and when code was buggy users rarel
What carries the argument
The construction pipeline that turns real conversation logs into measurable outcomes: extract code blocks from 82,843 conversations that contain AI-generated code, validate programming-language labels with language-specific compilers and linters, apply 648 regex-based security rules mapped to CWE entries, run four regular-expression denial-of-service detectors, filter import statements against package registries, and classify user intents with a zero-shot model. This pipeline is what converts raw conversation logs into the reported vulnerability rates and intent distributions.
Load-bearing premise
The claim that users rarely ask for security rests on an unvalidated zero-shot classifier; if the classifier misses security-related phrasings, the near-zero 'Secure Coding' rate could be a measurement artifact rather than a true reflection of user intent.
What would settle it
Randomly sample 500 English coding conversations, have two humans label the initial and follow-up queries for security intent, and compare the human 'secure coding' rate with the 21/34,478 classifier rate; a significantly higher human rate would falsify the paper's user-awareness claim. Alternatively, a data-flow analysis showing most flagged regexes never handle untrusted input would falsify the ReDoS prevalence claim.
If this is right
- Copying ChatGPT code into production without review will import the measured vulnerability rates — roughly one ReDoS-flagged regex in three, and one in seven Python imports hallucinated in the affected snippets.
- Prompting users to ask for security is unlikely to work: even users who start with a secure-coding intent rarely sustain the topic in follow-ups.
- The memory-safety findings imply that even short C/C++ fragments are risky; assembling larger programs from separately generated fragments may expose use-after-free and double-free bugs that don't appear in small snippets.
- Because users won't ask for security, the burden shifts to automated post-generation scanning and default-safe model behavior.
Where Pith is reading between the lines
- The user-intent result depends on an unvalidated zero-shot classifier; a human audit of a sample of queries could show the true 'secure coding' rate is higher than 0.06 percent, so the 'users don't ask' finding should be treated as provisional.
- Package hallucination names that appear rarely could be pre-registered by attackers as malicious packages, turning a correctness bug into a supply-chain attack vector.
- The paper's ReDoS rate may overcount because regexes that never process untrusted input are safe; a follow-up study that traces data flow from user input to the regex would bound the true exploitability rate.
- If newer models or domain-specific tools are tested with the same pipeline, the difference in vulnerability rates would reveal whether the problem is model capability or user behavior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an empirical study of real ChatGPT-generated code extracted from the WildChat dataset. It constructs a code corpus (82,843 conversations containing code), performs syntax checking on six languages, applies OpenGrep security rules for hash functions, SQL injection, RNG, deserialization, and memory safety, runs four ReDoS detectors on extracted regexes, checks Python/JavaScript imports for package hallucinations, and finally classifies English user queries into intent categories using zero-shot NLI. Headline numbers include 29.4% ReDoS-vulnerable regexes, 14.85% of C/C++ snippets using unsafe memory functions, 20.61% of hash-function conversations triggering weakness rules, and about 14.4% of Python imports hallucinated. The paper concludes that LLM-generated code is frequently insecure and that users rarely request security-focused code.
Significance. If the measurements are sound, this is a valuable complement to earlier synthetic-prompt studies: it uses authentic user–ChatGPT interactions, externalizes vulnerability detection to third-party scanners, and provides a public dataset for reuse. The honest treatment of known over- and undercounting in the security analysis is a strength. However, the distinctive user-intent contribution is currently supported only by an unvalidated zero-shot classifier, and the ReDoS aggregation table is internally inconsistent as printed. The paper is therefore significant in scope but not yet reliable in its strongest claims.
major comments (4)
- [§5.2, Table 10] The conclusion that users rarely ask about security rests entirely on zero-shot classification with bart-large-mnli using hand-defined categories. No validation set, accuracy, precision/recall, human agreement, or error analysis is reported. The 'Secure Coding' category receives only 21 initial labels (Table 10) and 6 follow-up labels (Section 5.4). If the NLI model has low recall for paraphrases such as 'make this safe' or 'is this vulnerable to attacks?', the headline observation in the abstract and Section 5.2 could be a measurement artifact. This needs to be supported by a manually annotated sample with inter-annotator agreement, or the claim should be explicitly downgraded to a classifier-based observation.
- [§4.2 and abstract] The paper states it evaluates code 'with respect to correctness', but the correctness analysis in Section 4.2 is only a syntax check using py_compile, eslint, javac, gcc, php -l, and Roslyn. Syntactic validity is not functional or semantic correctness; a snippet can parse and still be wrong. Please replace the correctness terminology with 'syntactic validity' throughout, or provide an additional functional correctness evaluation (e.g., testing a sample of snippets against assertions).
- [§4.4, Table 9] The aggregation of ReDoS results is not reproducible from Table 9. For C/CPP, SafeRegex alone reports 40 vulnerable regexes out of 80, but the Total Vuln. column reports 36; for Java, SafeRegex reports 160 out of 226, while Total Vuln. reports 47; for Python, SafeRegex reports 309 out of 753, while Total Vuln. reports 212. If Total Vuln. is a union of the tools, it cannot be smaller than the per-tool counts; if it is an intersection, a manually confirmed subset, or something else, that definition and the supporting counts must be given. As printed, the 29.4% headline rate is not defensible from the table.
- [§6] The Discussion says the detection 'likely overcounts some issues' while also 'undercounting others', and then states that 'our reported rates should be considered lower bounds'. With both directions of error acknowledged, the lower-bound conclusion does not follow unless the overcounting cases were all manually removed or quantified. Please clarify which rates are overcounts, which are undercounts, and why the net claim of 'lower bounds' is warranted for the reported percentages.
minor comments (5)
- [Title/Abstract] The full-text title is 'WildCode: An Empirical Analysis of Code Generated by ChatGPT', while the arXiv listing is 'WildCode Revisited: A Comprehensive Empirical Study...'. Please unify these and ensure the submitted title matches the manuscript.
- [Table 2] The columns 'Labelled by ChatGPT' and 'Labelled by Model' are confusing: for Python, 'Labelled by ChatGPT' equals the total number of code snippets (60,451), and the relationship between these two columns and the final valid-snippet count is not explained. Clarify the denominator used for syntax-error percentages.
- [Table 3] The table caption says 'total 177,732 rows', but the sum of the five displayed error categories is 170,516. Either other categories exist and are omitted, or the total is inconsistent.
- [§5.4] The text says 1,562 conversations were flagged by OpenGrep and 1,214 were in English, but the preceding Section 4 reports vulnerability counts per category (e.g., hash, SQL, memory, deserialization) that do not obviously sum to 1,562. Please explain whether the 1,562 figure comes from a broader rule set and reconcile the numbers.
- [§3] Minor typo: 'it’s scale' should be 'its scale'.
Circularity Check
No significant circularity: security findings are external measurements; user-intent classifier is unvalidated but not circular.
full rationale
The central security claims are measured directly from WildChat code snippets using third-party scanners and manual review (OpenGrep, SafeRegex, Rescue, ReDoSHunter, Revealer), with rates such as 20.61% weak hash usage, 14.85% unsafe C/C++ memory functions, and 29.4% ReDoS-prone regexes reported as empirical findings rather than as predictions derived from fitted parameters. The paper's self-citations to its earlier synthetic study [12] are contextual comparisons (e.g., "In our previous paper [12], ChatGPT exhibited particular difficulty with memory corruption vulnerabilities in C/C++ programs") and are not used to define or compute the measured quantities. The user-intent analysis does depend on an unvalidated zero-shot classifier (bart-large-mnli) with hand-defined categories, and the paper itself acknowledges that "zero-shot classification, while scalable, may miss nuanced intents" (Section 6). This is a measurement-validity threat, not a circular reduction: the classifier is an external model applied to natural-language queries, and the paper's conclusion about the rarity of 'Secure Coding' labels is not assumed in the category definitions. No equation, fitted parameter, or self-citation chain forces the reported results to equal their inputs by construction, so no specific circular step can be exhibited.
Axiom & Free-Parameter Ledger
axioms (6)
- domain assumption OpenGrep regex rules are a meaningful proxy for the presence of security vulnerabilities in the analyzed code.
- domain assumption bart-large-mnli zero-shot classification correctly assigns user queries to the predefined intent categories.
- domain assumption WildChat conversations are representative of real-world ChatGPT usage for code generation.
- domain assumption The programming-language identification model's 95% accuracy and the language-specific syntax checkers suffice to label code snippets correctly.
- domain assumption The four ReDoS detectors (SafeRegex, Rescue, ReDoSHunter, Revealer) provide usable signals for regex vulnerability.
- domain assumption Removing code snippets and applying langdetect correctly identifies English conversations.
Cite this review
Pith. "Pith review of WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code." pith.science (2026). https://pith.science/paper/CK4X3352
@misc{pith2026251204259,
author = {Pith},
title = {Pith review of: WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code},
year = {2026},
howpublished = {\url{https://pith.science/paper/CK4X3352}},
note = {Machine review of arXiv:2512.04259}
}
read the original abstract
LLM models are increasingly used to generate code, but the quality and security of this code are often uncertain. Several recent studies have raised alarm bells, indicating that such AI-generated code may be particularly vulnerable to cyberattacks. However, most of these studies rely on code that is generated specifically for the study, which raises questions about the realism of such experiments. In this study, we perform a large-scale empirical analysis of real-life code generated by ChatGPT. We evaluate code generated by ChatGPT both with respect to correctness and security and delve into the intentions of users who request code from the model. We further performed an experiment to evaluate the effectiveness of common prompt engineering strategies using real-life prompts. Our study supports earlier research that employed synthetic queries and produced proof that LLM-generated code is frequently insufficient in terms of security. Additionally, we observe that users don't ask many questions about the security characteristics of the code they ask LLMs to provide.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2505.02133 (2025)
Ashrafi, N., Bouktif, S., Mediani, M.: Enhancing llm code generation: A system- atic evaluation of multi-agent collaboration and runtime debugging for improved accuracy, reliability, and latency. arXiv preprint arXiv:2505.02133 (2025)
Pith/arXiv arXiv 2025
-
[2]
https://aws.amazon.com/blogs/devops/introducing-amazon- codewhisperer-dashboard-and-cloudwatch-metrics/ (2024)
AWS DevOps Blog: Introducing amazon codewhisperer dashboard and cloudwatch metrics. https://aws.amazon.com/blogs/devops/introducing-amazon- codewhisperer-dashboard-and-cloudwatch-metrics/ (2024)
2024
-
[3]
arXiv preprint arXiv:2409.16526 (2024)
Bai, W., Xuan, K., Huang, P., Wu, Q., Wen, J., Wu, J., Lu, K.: Apilot: Navigating large language models to generate secure code by sidestepping outdated api pitfalls. arXiv preprint arXiv:2409.16526 (2024)
Pith/arXiv arXiv 2024
-
[4]
arXiv preprint arXiv:2502.06039 (2025)
Bruni, M., Gabrielli, F., Ghafari, M., Kropp, M.: Benchmarking prompt engi- neering techniques for secure code generation with gpt models. arXiv preprint arXiv:2502.06039 (2025)
Pith/arXiv arXiv 2025
-
[5]
ArXivabs/2410.01026(2024), https://api.semanticscholar.org/CorpusID:273026291
Etsenake, D., Nagappan, M.: Understanding the human-llm dynamic: A litera- ture survey of llm use in programming tasks. ArXivabs/2410.01026(2024), https://api.semanticscholar.org/CorpusID:273026291
Pith/arXiv arXiv 2024
-
[6]
IEEE Transactions on Software Engineering (2024)
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., Lahiri, S.K.: Llm-based test- driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering (2024)
2024
-
[7]
ACM Transactions on Software Engineering and Methodology (2025)
Fu, Y., Liang, P., LI, Z., SHAHIN, M., YU, J., CHEN, J.: Security weaknesses of copilot-generated code in github projects: An empirical study. ACM Transactions on Software Engineering and Methodology (2025)
2025
-
[8]
https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the- developer-experience/ (2025), accessed: 2025-05-26
GitHub: Survey reveals ai’s impact on the developer experience. https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the- developer-experience/ (2025), accessed: 2025-05-26
2025
-
[9]
In: 2023 ACM SIGSAC Conference on Computer and Communi- cations Security
He, J., Vechev, M.: Large language models for code: Security hardening and ad- versarial testing. In: 2023 ACM SIGSAC Conference on Computer and Communi- cations Security. pp. 1865–1879 (2023)
2023
-
[10]
arXiv preprint arXiv:2205.02544 (2022)
Hiesgen, R., Nawrocki, M., Schmidt, T.C., Wählisch, M.: The race to the vulner- able: Measuring the log4j shell incident. arXiv preprint arXiv:2205.02544 (2022)
Pith/arXiv arXiv 2022
-
[11]
ACM Transactions on Software Engineering and Methodology (2024)
Huang, D., Zhang, J.M., Bu, Q., Xie, X., Chen, J., Cui, H.: Bias testing and mit- igation in llm-based code generation. ACM Transactions on Software Engineering and Methodology (2024)
2024
-
[12]
Khoury, R., Avila, A.R., Brunelle, J., Camara, B.M.: How secure is code gener- ated by chatgpt? In: 2023 IEEE international conference on systems, man, and cybernetics (SMC). pp. 2445–2451. IEEE (2023)
2023
-
[13]
https://github.com/jkutner/saferegex (2018)
Kutner, J.: Saferegex. https://github.com/jkutner/saferegex (2018)
2018
-
[14]
In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
Li, D., Yan, M., Zhang, Y., Liu, Z., Liu, C., Zhang, X., Chen, T., Lo, D.: Cosec: On- the-fly security hardening of code llms via supervised co-decoding. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 1428–1439 (2024) WildCode: An Empirical Analysis of Code Generated by ChatGPT 21
2024
-
[15]
In: 30th USENIX Security Symposium
Li, Y., Chen, Z., Cao, J., Xu, Z., Peng, Q., Chen, H., Chen, L., Cheung, S.C.: ReDoSHunter: A combined static and dynamic approach for regular expression DoS detection. In: 30th USENIX Security Symposium. pp. 3847–3864 (2021)
2021
-
[16]
Advances in Neural Information Processing Systems36, 21558–21572 (2023)
Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems36, 21558–21572 (2023)
2023
-
[17]
In: 2021 IEEE Symposium on Security and Privacy (SP)
Liu, Y., Zhang, M., Meng, W.: Revealer: Detecting and exploiting regular expres- sion denial-of-service vulnerabilities. In: 2021 IEEE Symposium on Security and Privacy (SP). pp. 1468–1484. IEEE (2021)
2021
-
[18]
In: 2024 on ACM SIGSAC Conference on Computer and Communications Security
Nazzal, M., Khalil, I., Khreishah, A., Phan, N.: Promsec: Prompt optimization for secure generation of functional source code with large language models (LLMs). In: 2024 on ACM SIGSAC Conference on Computer and Communications Security. pp. 2266–2280 (2024)
2024
-
[19]
owasp.org/www- community/attacks/Regular_expression_Denial_of_Service_-_ReDoS (2025), accessed: 2025-08-04
OWASP Foundation: Regular expression denial of service (redos). owasp.org/www- community/attacks/Regular_expression_Denial_of_Service_-_ReDoS (2025), accessed: 2025-08-04
2025
-
[20]
ACM Transactions on Software Engineering and Methodology32(1), 1–45 (2023)
Sayar, I., Bartel, A., Bodden, E., Le Traon, Y.: An in-depth study of java deserial- ization remote-code execution exploits and vulnerabilities. ACM Transactions on Software Engineering and Methodology32(1), 1–45 (2023)
2023
-
[21]
In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
Shen, Y., Jiang, Y., Xu, C., Yu, P., Ma, X., Lu, J.: Rescue: crafting regular expres- sion dos attacks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. p. 225–235. ASE ’18, Association for Com- puting Machinery, New York, NY, USA (2018)
2018
-
[22]
In: 34th USENIX Security Symposium
Spracklen, J., Wijewickrama, R., Sakib, A.N., Maiti, A., Viswanath, B.: We have a package for you! a comprehensive analysis of package hallucinations by code generating LLMs. In: 34th USENIX Security Symposium. pp. 3687–3706 (2025)
2025
-
[23]
arXiv preprint arXiv:2305.14752 (2023)
Tihanyi, N., Jain, R., Charalambous, Y., Ferrag, M.A., Sun, Y., Cordeiro, L.C.: A new era in software security: Towards self-healing software via large language models and formal verification. arXiv preprint arXiv:2305.14752 (2023)
Pith/arXiv arXiv 2023
-
[24]
Tony, C., Ferreyra, N.E.D., Mutas, M., Dhiff, S., Scandariato, R.: Prompting tech- niques for secure code generation: A systematic investigation (2024)
2024
-
[25]
arXiv preprint arXiv:2304.10778 (2023)
Yetiştiren, B., Özsoy, I., Ayerdem, M., Tüzün, E.: Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint arXiv:2304.10778 (2023)
Pith/arXiv arXiv 2023
-
[26]
arXiv preprint arXiv:2504.15867 (2025)
Zeng, B., Zhang, Q., Zhou, C., Go, G., Jiang, Y., Shi, H.: Inducing vulnerable code generation in llm coding assistants. arXiv preprint arXiv:2504.15867 (2025)
Pith/arXiv arXiv 2025
-
[27]
arXiv preprint arXiv:2405.01470 (2024)
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., Deng, Y.: Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470 (2024)
Pith/arXiv arXiv 2024
This paper was first reviewed by deepseek-v4-flash on August 3, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.