Pith. sign in

REVIEW 4 major objections 5 minor 27 references

Real-world ChatGPT code is frequently insecure, and users rarely ask for secure code — even when the model produces vulnerable or hallucinated output.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review

2026-08-03 18:36 UTC pith:CK4X3352

load-bearing objection Real-world WildChat data confirms LLM-generated code is often insecure, but the paper's distinctive user-intent claim rests on an unvalidated classifier. the 4 major comments →

arxiv 2512.04259 v2 pith:CK4X3352 submitted 2025-12-03 cs.CR cs.SE

WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code

classification cs.CR cs.SE
keywords LLM-generated codesoftware securityChatGPTreal-world conversationsReDoS vulnerabilitiescode hallucinationsuser intentempirical study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes 82,843 authentic ChatGPT conversations that contain generated code, and measures how often that code is insecure. It finds that roughly 29% of regular expressions are flagged as ReDoS-vulnerable, 14.85% of C/C++ snippets use unsafe memory functions, and about 14.4% of Python imports are hallucinated package names. A parallel analysis of user intent finds that security is essentially absent from user queries: fewer than two dozen initial requests mention secure coding. The paper's central claim is that synthetic studies were right — LLM-generated code is frequently insecure in real interactions — and that users do not compensate by asking for security.

Core claim

The paper's central claim is that, at scale, code generated by ChatGPT during real user interactions exhibits frequent, exploitable security weaknesses: 20.61% of conversations involving hash functions triggered weakness rules (mostly MD5/SHA1 or unauthenticated crypto), 14.85% of C/C++ snippets used unsafe memory functions, about 29.4% of 1,203 regexes were flagged as ReDoS-vulnerable by static detectors, and 14.4% of Python imports were hallucinated. The same analysis of the English-language subset finds that users almost never request security: the 'Secure Coding' intent category received only 21 initial labels out of tens of thousands of conversations, and when code was buggy users rarel

What carries the argument

The construction pipeline that turns real conversation logs into measurable outcomes: extract code blocks from 82,843 conversations that contain AI-generated code, validate programming-language labels with language-specific compilers and linters, apply 648 regex-based security rules mapped to CWE entries, run four regular-expression denial-of-service detectors, filter import statements against package registries, and classify user intents with a zero-shot model. This pipeline is what converts raw conversation logs into the reported vulnerability rates and intent distributions.

Load-bearing premise

The claim that users rarely ask for security rests on an unvalidated zero-shot classifier; if the classifier misses security-related phrasings, the near-zero 'Secure Coding' rate could be a measurement artifact rather than a true reflection of user intent.

What would settle it

Randomly sample 500 English coding conversations, have two humans label the initial and follow-up queries for security intent, and compare the human 'secure coding' rate with the 21/34,478 classifier rate; a significantly higher human rate would falsify the paper's user-awareness claim. Alternatively, a data-flow analysis showing most flagged regexes never handle untrusted input would falsify the ReDoS prevalence claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Copying ChatGPT code into production without review will import the measured vulnerability rates — roughly one ReDoS-flagged regex in three, and one in seven Python imports hallucinated in the affected snippets.
  • Prompting users to ask for security is unlikely to work: even users who start with a secure-coding intent rarely sustain the topic in follow-ups.
  • The memory-safety findings imply that even short C/C++ fragments are risky; assembling larger programs from separately generated fragments may expose use-after-free and double-free bugs that don't appear in small snippets.
  • Because users won't ask for security, the burden shifts to automated post-generation scanning and default-safe model behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The user-intent result depends on an unvalidated zero-shot classifier; a human audit of a sample of queries could show the true 'secure coding' rate is higher than 0.06 percent, so the 'users don't ask' finding should be treated as provisional.
  • Package hallucination names that appear rarely could be pre-registered by attackers as malicious packages, turning a correctness bug into a supply-chain attack vector.
  • The paper's ReDoS rate may overcount because regexes that never process untrusted input are safe; a follow-up study that traces data flow from user input to the regex would bound the true exploitability rate.
  • If newer models or domain-specific tools are tested with the same pipeline, the difference in vulnerability rates would reveal whether the problem is model capability or user behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports an empirical study of real ChatGPT-generated code extracted from the WildChat dataset. It constructs a code corpus (82,843 conversations containing code), performs syntax checking on six languages, applies OpenGrep security rules for hash functions, SQL injection, RNG, deserialization, and memory safety, runs four ReDoS detectors on extracted regexes, checks Python/JavaScript imports for package hallucinations, and finally classifies English user queries into intent categories using zero-shot NLI. Headline numbers include 29.4% ReDoS-vulnerable regexes, 14.85% of C/C++ snippets using unsafe memory functions, 20.61% of hash-function conversations triggering weakness rules, and about 14.4% of Python imports hallucinated. The paper concludes that LLM-generated code is frequently insecure and that users rarely request security-focused code.

Significance. If the measurements are sound, this is a valuable complement to earlier synthetic-prompt studies: it uses authentic user–ChatGPT interactions, externalizes vulnerability detection to third-party scanners, and provides a public dataset for reuse. The honest treatment of known over- and undercounting in the security analysis is a strength. However, the distinctive user-intent contribution is currently supported only by an unvalidated zero-shot classifier, and the ReDoS aggregation table is internally inconsistent as printed. The paper is therefore significant in scope but not yet reliable in its strongest claims.

major comments (4)
  1. [§5.2, Table 10] The conclusion that users rarely ask about security rests entirely on zero-shot classification with bart-large-mnli using hand-defined categories. No validation set, accuracy, precision/recall, human agreement, or error analysis is reported. The 'Secure Coding' category receives only 21 initial labels (Table 10) and 6 follow-up labels (Section 5.4). If the NLI model has low recall for paraphrases such as 'make this safe' or 'is this vulnerable to attacks?', the headline observation in the abstract and Section 5.2 could be a measurement artifact. This needs to be supported by a manually annotated sample with inter-annotator agreement, or the claim should be explicitly downgraded to a classifier-based observation.
  2. [§4.2 and abstract] The paper states it evaluates code 'with respect to correctness', but the correctness analysis in Section 4.2 is only a syntax check using py_compile, eslint, javac, gcc, php -l, and Roslyn. Syntactic validity is not functional or semantic correctness; a snippet can parse and still be wrong. Please replace the correctness terminology with 'syntactic validity' throughout, or provide an additional functional correctness evaluation (e.g., testing a sample of snippets against assertions).
  3. [§4.4, Table 9] The aggregation of ReDoS results is not reproducible from Table 9. For C/CPP, SafeRegex alone reports 40 vulnerable regexes out of 80, but the Total Vuln. column reports 36; for Java, SafeRegex reports 160 out of 226, while Total Vuln. reports 47; for Python, SafeRegex reports 309 out of 753, while Total Vuln. reports 212. If Total Vuln. is a union of the tools, it cannot be smaller than the per-tool counts; if it is an intersection, a manually confirmed subset, or something else, that definition and the supporting counts must be given. As printed, the 29.4% headline rate is not defensible from the table.
  4. [§6] The Discussion says the detection 'likely overcounts some issues' while also 'undercounting others', and then states that 'our reported rates should be considered lower bounds'. With both directions of error acknowledged, the lower-bound conclusion does not follow unless the overcounting cases were all manually removed or quantified. Please clarify which rates are overcounts, which are undercounts, and why the net claim of 'lower bounds' is warranted for the reported percentages.
minor comments (5)
  1. [Title/Abstract] The full-text title is 'WildCode: An Empirical Analysis of Code Generated by ChatGPT', while the arXiv listing is 'WildCode Revisited: A Comprehensive Empirical Study...'. Please unify these and ensure the submitted title matches the manuscript.
  2. [Table 2] The columns 'Labelled by ChatGPT' and 'Labelled by Model' are confusing: for Python, 'Labelled by ChatGPT' equals the total number of code snippets (60,451), and the relationship between these two columns and the final valid-snippet count is not explained. Clarify the denominator used for syntax-error percentages.
  3. [Table 3] The table caption says 'total 177,732 rows', but the sum of the five displayed error categories is 170,516. Either other categories exist and are omitted, or the total is inconsistent.
  4. [§5.4] The text says 1,562 conversations were flagged by OpenGrep and 1,214 were in English, but the preceding Section 4 reports vulnerability counts per category (e.g., hash, SQL, memory, deserialization) that do not obviously sum to 1,562. Please explain whether the 1,562 figure comes from a broader rule set and reconcile the numbers.
  5. [§3] Minor typo: 'it’s scale' should be 'its scale'.

Circularity Check

0 steps flagged

No significant circularity: security findings are external measurements; user-intent classifier is unvalidated but not circular.

full rationale

The central security claims are measured directly from WildChat code snippets using third-party scanners and manual review (OpenGrep, SafeRegex, Rescue, ReDoSHunter, Revealer), with rates such as 20.61% weak hash usage, 14.85% unsafe C/C++ memory functions, and 29.4% ReDoS-prone regexes reported as empirical findings rather than as predictions derived from fitted parameters. The paper's self-citations to its earlier synthetic study [12] are contextual comparisons (e.g., "In our previous paper [12], ChatGPT exhibited particular difficulty with memory corruption vulnerabilities in C/C++ programs") and are not used to define or compute the measured quantities. The user-intent analysis does depend on an unvalidated zero-shot classifier (bart-large-mnli) with hand-defined categories, and the paper itself acknowledges that "zero-shot classification, while scalable, may miss nuanced intents" (Section 6). This is a measurement-validity threat, not a circular reduction: the classifier is an external model applied to natural-language queries, and the paper's conclusion about the rarity of 'Secure Coding' labels is not assumed in the category definitions. No equation, fitted parameter, or self-citation chain forces the reported results to equal their inputs by construction, so no specific circular step can be exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The paper contributes a measurement, not a derivation; it introduces no fitted parameters and no new theoretical entities. Everything rests on the validity of the WildChat sampling frame, the third-party scanners, and the unvalidated intent classifier, all listed above.

axioms (6)
  • domain assumption OpenGrep regex rules are a meaningful proxy for the presence of security vulnerabilities in the analyzed code.
    Section 4.3: all security rates (hash, SQLi, RNG, deserialization, memory) are computed from OpenGrep rule matches; the paper admits these both overcount (safe contexts) and undercount (memory issues).
  • domain assumption bart-large-mnli zero-shot classification correctly assigns user queries to the predefined intent categories.
    Section 5.2: the intent distribution (including the near-absence of 'Secure Coding') is generated by this model with no reported validation set, accuracy, or human agreement.
  • domain assumption WildChat conversations are representative of real-world ChatGPT usage for code generation.
    Section 3: the paper treats WildChat as 'real-life' data; if WildChat's opt-in collection biases the user population or conversation types, the generalization to all ChatGPT code generation fails.
  • domain assumption The programming-language identification model's 95% accuracy and the language-specific syntax checkers suffice to label code snippets correctly.
    Section 3: up to 5% label error is acknowledged, and PHP labeling is entirely unusable (all 1,935 model-labeled PHP snippets are actually shell snippets).
  • domain assumption The four ReDoS detectors (SafeRegex, Rescue, ReDoSHunter, Revealer) provide usable signals for regex vulnerability.
    Section 4.4: the union of four tools is used to estimate 'about a third' of regexes are vulnerable, with no false-positive rate analysis; the paper calls the result a 'baseline indication'.
  • domain assumption Removing code snippets and applying langdetect correctly identifies English conversations.
    Section 3: the English-only intent subset (34,478 conversations) is created this way; language misclassification would bias the user-intent results.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code." pith.science (2026). https://pith.science/paper/CK4X3352

@misc{pith2026251204259,
  author       = {Pith},
  title        = {Pith review of: WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CK4X3352}},
  note         = {Machine review of arXiv:2512.04259}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM models are increasingly used to generate code, but the quality and security of this code are often uncertain. Several recent studies have raised alarm bells, indicating that such AI-generated code may be particularly vulnerable to cyberattacks. However, most of these studies rely on code that is generated specifically for the study, which raises questions about the realism of such experiments. In this study, we perform a large-scale empirical analysis of real-life code generated by ChatGPT. We evaluate code generated by ChatGPT both with respect to correctness and security and delve into the intentions of users who request code from the model. We further performed an experiment to evaluate the effectiveness of common prompt engineering strategies using real-life prompts. Our study supports earlier research that employed synthetic queries and produced proof that LLM-generated code is frequently insufficient in terms of security. Additionally, we observe that users don't ask many questions about the security characteristics of the code they ask LLMs to provide.

Figures

Figures reproduced from arXiv: 2512.04259 by Abdelwahab Hamou-Lhadj, Alexander Da Re, Kobra Khanmohammadi, Nicholas Rebelo Melo, Pooria Roy, Raphael Khoury, Wilfried Patrick Konan.

Figure 1
Figure 1. Figure 1: Dataset generation pipeline In the WildChat conversations, code snippets delimited by triple backticks usually begin with a programming language tag, followed by the code itself. However, these tags are often missing or incorrectly specified, so they cannot be relied on. To handle snippets without valid annotations, we applied a program￾ming language identification model.8 to automatically classify the cod… view at source ↗
Figure 2
Figure 2. Figure 2: Hallucination detection pipeline We focus our analysis on Python and JavaScript due to their widespread adoption and the strong ecosystem support provided by package repositories, PyPI for Python and NPM for JavaScript. These repositories allow for system￾atic validation of third-party modules. Our pipeline for detecting hallucinated modules, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of programming languages in initial code generation vs. user [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Predicted category distribution for user queries. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of conversation length grouped by initial and followup mes [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Predicted Category for User Query in Conversations with Buggy Codes [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 9 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2505.02133 (2025)

    Ashrafi, N., Bouktif, S., Mediani, M.: Enhancing llm code generation: A system- atic evaluation of multi-agent collaboration and runtime debugging for improved accuracy, reliability, and latency. arXiv preprint arXiv:2505.02133 (2025)

  2. [2]

    https://aws.amazon.com/blogs/devops/introducing-amazon- codewhisperer-dashboard-and-cloudwatch-metrics/ (2024)

    AWS DevOps Blog: Introducing amazon codewhisperer dashboard and cloudwatch metrics. https://aws.amazon.com/blogs/devops/introducing-amazon- codewhisperer-dashboard-and-cloudwatch-metrics/ (2024)

  3. [3]

    arXiv preprint arXiv:2409.16526 (2024)

    Bai, W., Xuan, K., Huang, P., Wu, Q., Wen, J., Wu, J., Lu, K.: Apilot: Navigating large language models to generate secure code by sidestepping outdated api pitfalls. arXiv preprint arXiv:2409.16526 (2024)

  4. [4]

    arXiv preprint arXiv:2502.06039 (2025)

    Bruni, M., Gabrielli, F., Ghafari, M., Kropp, M.: Benchmarking prompt engi- neering techniques for secure code generation with gpt models. arXiv preprint arXiv:2502.06039 (2025)

  5. [5]

    ArXivabs/2410.01026(2024), https://api.semanticscholar.org/CorpusID:273026291

    Etsenake, D., Nagappan, M.: Understanding the human-llm dynamic: A litera- ture survey of llm use in programming tasks. ArXivabs/2410.01026(2024), https://api.semanticscholar.org/CorpusID:273026291

  6. [6]

    IEEE Transactions on Software Engineering (2024)

    Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., Lahiri, S.K.: Llm-based test- driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering (2024)

  7. [7]

    ACM Transactions on Software Engineering and Methodology (2025)

    Fu, Y., Liang, P., LI, Z., SHAHIN, M., YU, J., CHEN, J.: Security weaknesses of copilot-generated code in github projects: An empirical study. ACM Transactions on Software Engineering and Methodology (2025)

  8. [8]

    https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the- developer-experience/ (2025), accessed: 2025-05-26

    GitHub: Survey reveals ai’s impact on the developer experience. https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the- developer-experience/ (2025), accessed: 2025-05-26

  9. [9]

    In: 2023 ACM SIGSAC Conference on Computer and Communi- cations Security

    He, J., Vechev, M.: Large language models for code: Security hardening and ad- versarial testing. In: 2023 ACM SIGSAC Conference on Computer and Communi- cations Security. pp. 1865–1879 (2023)

  10. [10]

    arXiv preprint arXiv:2205.02544 (2022)

    Hiesgen, R., Nawrocki, M., Schmidt, T.C., Wählisch, M.: The race to the vulner- able: Measuring the log4j shell incident. arXiv preprint arXiv:2205.02544 (2022)

  11. [11]

    ACM Transactions on Software Engineering and Methodology (2024)

    Huang, D., Zhang, J.M., Bu, Q., Xie, X., Chen, J., Cui, H.: Bias testing and mit- igation in llm-based code generation. ACM Transactions on Software Engineering and Methodology (2024)

  12. [12]

    Khoury, R., Avila, A.R., Brunelle, J., Camara, B.M.: How secure is code gener- ated by chatgpt? In: 2023 IEEE international conference on systems, man, and cybernetics (SMC). pp. 2445–2451. IEEE (2023)

  13. [13]

    https://github.com/jkutner/saferegex (2018)

    Kutner, J.: Saferegex. https://github.com/jkutner/saferegex (2018)

  14. [14]

    In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

    Li, D., Yan, M., Zhang, Y., Liu, Z., Liu, C., Zhang, X., Chen, T., Lo, D.: Cosec: On- the-fly security hardening of code llms via supervised co-decoding. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 1428–1439 (2024) WildCode: An Empirical Analysis of Code Generated by ChatGPT 21

  15. [15]

    In: 30th USENIX Security Symposium

    Li, Y., Chen, Z., Cao, J., Xu, Z., Peng, Q., Chen, H., Chen, L., Cheung, S.C.: ReDoSHunter: A combined static and dynamic approach for regular expression DoS detection. In: 30th USENIX Security Symposium. pp. 3847–3864 (2021)

  16. [16]

    Advances in Neural Information Processing Systems36, 21558–21572 (2023)

    Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems36, 21558–21572 (2023)

  17. [17]

    In: 2021 IEEE Symposium on Security and Privacy (SP)

    Liu, Y., Zhang, M., Meng, W.: Revealer: Detecting and exploiting regular expres- sion denial-of-service vulnerabilities. In: 2021 IEEE Symposium on Security and Privacy (SP). pp. 1468–1484. IEEE (2021)

  18. [18]

    In: 2024 on ACM SIGSAC Conference on Computer and Communications Security

    Nazzal, M., Khalil, I., Khreishah, A., Phan, N.: Promsec: Prompt optimization for secure generation of functional source code with large language models (LLMs). In: 2024 on ACM SIGSAC Conference on Computer and Communications Security. pp. 2266–2280 (2024)

  19. [19]

    owasp.org/www- community/attacks/Regular_expression_Denial_of_Service_-_ReDoS (2025), accessed: 2025-08-04

    OWASP Foundation: Regular expression denial of service (redos). owasp.org/www- community/attacks/Regular_expression_Denial_of_Service_-_ReDoS (2025), accessed: 2025-08-04

  20. [20]

    ACM Transactions on Software Engineering and Methodology32(1), 1–45 (2023)

    Sayar, I., Bartel, A., Bodden, E., Le Traon, Y.: An in-depth study of java deserial- ization remote-code execution exploits and vulnerabilities. ACM Transactions on Software Engineering and Methodology32(1), 1–45 (2023)

  21. [21]

    In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

    Shen, Y., Jiang, Y., Xu, C., Yu, P., Ma, X., Lu, J.: Rescue: crafting regular expres- sion dos attacks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. p. 225–235. ASE ’18, Association for Com- puting Machinery, New York, NY, USA (2018)

  22. [22]

    In: 34th USENIX Security Symposium

    Spracklen, J., Wijewickrama, R., Sakib, A.N., Maiti, A., Viswanath, B.: We have a package for you! a comprehensive analysis of package hallucinations by code generating LLMs. In: 34th USENIX Security Symposium. pp. 3687–3706 (2025)

  23. [23]

    arXiv preprint arXiv:2305.14752 (2023)

    Tihanyi, N., Jain, R., Charalambous, Y., Ferrag, M.A., Sun, Y., Cordeiro, L.C.: A new era in software security: Towards self-healing software via large language models and formal verification. arXiv preprint arXiv:2305.14752 (2023)

  24. [24]

    Tony, C., Ferreyra, N.E.D., Mutas, M., Dhiff, S., Scandariato, R.: Prompting tech- niques for secure code generation: A systematic investigation (2024)

  25. [25]

    arXiv preprint arXiv:2304.10778 (2023)

    Yetiştiren, B., Özsoy, I., Ayerdem, M., Tüzün, E.: Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint arXiv:2304.10778 (2023)

  26. [26]

    arXiv preprint arXiv:2504.15867 (2025)

    Zeng, B., Zhang, Q., Zhou, C., Go, G., Jiang, Y., Shi, H.: Inducing vulnerable code generation in llm coding assistants. arXiv preprint arXiv:2504.15867 (2025)

  27. [27]

    arXiv preprint arXiv:2405.01470 (2024)

    Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., Deng, Y.: Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470 (2024)

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.