Pith sign in

REVIEW 3 major objections 2 minor 49 references

Reviewed by Pith at T0; open to challenge.

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Prompt engineering does not reduce the rate of security flaws in LLM-generated code but does change which flaws appear.

2026-06-30 15:15 UTC pith:C5NTXMP5

load-bearing objection WA-0CoT shifts CWE distributions without cutting overall vulnerability rates, but the chi-square claims rest on an unexamined detection and labeling process. the 3 major comments →

arxiv 2605.24298 v1 pith:C5NTXMP5 submitted 2026-05-22 cs.CR cs.AIcs.LG

An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods

Mohammed Kharma , Ahmed Sabbah , Mohammad Alkhanafseh , Mohammad Hammoudeh , David Mohaisen This is my paper
classification cs.CR cs.AIcs.LG
keywords LLM code generationcode securityprompt engineeringCWE categoriesempirical evaluationvulnerability analysischi-square testsprogramming languages
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests five large language models generating code in Java, C++, C, and Python under multiple prompting strategies, including a new weaknesses-aware zero-shot chain-of-thought method that adds CWE security context. Statistical tests show no meaningful drop in how often vulnerabilities occur or how dense they are. The prompts do shift the mix of specific weakness categories, and those shifts differ by programming language. The authors conclude that prompt changes alone cannot make the generated code reliably safer.

Core claim

The empirical evaluation reveals that no prompting method, including the introduced weaknesses-aware zero-shot chain-of-thought strategy, achieves statistically significant reductions in the frequency or density of vulnerabilities in code generated by LLMs across Java, C++, C, and Python. Prompting does, however, systematically shift the distribution of specific CWE weakness categories, with the nature of these shifts depending on the target programming language.

What carries the argument

The weaknesses-aware zero-shot chain-of-thought (WA-0CoT) prompting strategy that adds CWE mappings for security context, paired with chi-square tests on vulnerability counts and category distributions.

Load-bearing premise

Vulnerabilities in the generated code can be reliably detected and classified into CWE categories the same way no matter which model, language, or prompt is used.

What would settle it

A re-run of the same code samples through a different automated scanner or a manual review by security experts that produces different vulnerability counts or different CWE distributions.

If this is right

  • Prompting strategies change the types of weaknesses that appear in LLM code without lowering their total number or density.
  • The effect of any given prompt on weakness types depends on the programming language being generated.
  • Security-aware prompting by itself cannot be relied on to produce fewer vulnerabilities overall.
  • Evaluations of LLM code security must account for differences across both models and languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Combining prompting with later static analysis or manual review may still be needed to catch remaining issues.
  • Model-specific or language-specific prompt tuning could be tested as a way to steer weakness distributions toward less severe categories.
  • The results point toward exploring training data changes or post-processing steps rather than prompt design alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript presents an empirical evaluation of security properties in code generated by five LLMs across four languages (Java, C++, C, Python) under multiple prompting methods, including a proposed weaknesses-aware zero-shot chain-of-thought (WA-0CoT) strategy that incorporates CWE mappings. Chi-square tests are reported to show no statistically significant reductions in vulnerability frequency or density across prompting methods. Prompting strategies are found to systematically alter the compositional distribution of CWE categories, with effects varying by language. The authors conclude that prompt engineering alone is insufficient to reliably reduce overall vulnerability levels.

Significance. If the empirical results hold after methodological clarification, the work would indicate that security-aware prompting can shift weakness types but does not reduce their prevalence, providing evidence against over-reliance on prompt engineering for secure code generation and emphasizing language- and model-aware design. The multi-model, multi-language scope is a strength if detection is reliable.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (Results): The chi-square test outcomes for vulnerability frequency and density are presented without sample sizes, degrees of freedom, exact p-values, effect sizes, or power analysis. This prevents assessment of whether the 'no statistically significant reductions' claim is supported or whether null results reflect underpowered tests.
  2. [§3] §3 (Methodology): No description is given of the vulnerability detection method (static analysis tool, manual review, or hybrid), inter-rater reliability, false-positive rates by language, or validation of CWE mappings for incomplete or multi-weakness code. This process is load-bearing for all frequency, density, and distributional claims.
  3. [§4] §4 (Results): The reported systematic shifts in CWE compositional distributions across prompting methods (including WA-0CoT) and languages rest on the same uncharacterized detection and labeling pipeline; systematic bias or noise varying by condition would render both the null frequency results and the positive distribution results uninterpretable.
minor comments (2)
  1. [Abstract and §3] The abstract and methods could explicitly list the five LLMs (including versions and sampling parameters) and the exact prompting templates used.
  2. [§4] Figure or table captions for CWE distributions should include the total number of generated samples per condition to allow readers to gauge the scale of the compositional analysis.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thorough and constructive review. The comments correctly identify gaps in statistical reporting and methodological transparency that limit interpretability of the results. We will revise the manuscript to address these points directly.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Results): The chi-square test outcomes for vulnerability frequency and density are presented without sample sizes, degrees of freedom, exact p-values, effect sizes, or power analysis. This prevents assessment of whether the 'no statistically significant reductions' claim is supported or whether null results reflect underpowered tests.

    Authors: We agree that these details are required for proper evaluation of the null results. In the revised manuscript we will report the exact sample sizes (number of code samples per model-language-prompting combination), degrees of freedom, exact p-values, effect sizes (Cramér's V), and a post-hoc power analysis for each chi-square test on frequency and density. This will allow readers to assess whether the tests were adequately powered. revision: yes

  2. Referee: [§3] §3 (Methodology): No description is given of the vulnerability detection method (static analysis tool, manual review, or hybrid), inter-rater reliability, false-positive rates by language, or validation of CWE mappings for incomplete or multi-weakness code. This process is load-bearing for all frequency, density, and distributional claims.

    Authors: We acknowledge that §3 lacks a sufficiently detailed account of the detection pipeline. We will add a new subsection describing the vulnerability detection method (including any static-analysis tool and manual-review protocol), inter-rater reliability statistics, language-specific false-positive rates, and the procedure used to assign and validate CWE labels on incomplete or multi-weakness snippets. These additions will make the frequency, density, and distributional results traceable to a characterized process. revision: yes

  3. Referee: [§4] §4 (Results): The reported systematic shifts in CWE compositional distributions across prompting methods (including WA-0CoT) and languages rest on the same uncharacterized detection and labeling pipeline; systematic bias or noise varying by condition would render both the null frequency results and the positive distribution results uninterpretable.

    Authors: This observation is correct and is addressed by the methodological expansion described above. In the revision we will also add a brief discussion (in §4 or the limitations section) of how any residual detection noise or condition-specific bias could affect the chi-square tests on CWE distributions versus overall frequency, thereby clarifying the conditions under which the distributional shifts remain interpretable. revision: yes

Circularity Check

0 steps flagged

Purely empirical study with no derivation chain or self-referential predictions

full rationale

The paper reports an empirical evaluation of LLM-generated code security using chi-square tests on observed vulnerability frequencies, densities, and CWE category distributions across models, languages, and prompting methods. No equations, derivations, fitted parameters, or predictions are presented; the central claims are direct statistical summaries of counted outcomes from generated code samples. No self-citation load-bearing steps, ansatz smuggling, or renaming of known results appear in the reported analysis. The study is self-contained against external benchmarks via standard hypothesis testing on independent experimental outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The study rests on the standard statistical assumption that chi-square tests are appropriate for categorical vulnerability counts and on the established CWE taxonomy for classifying weaknesses; no free parameters or new entities are introduced.

axioms (1)
  • domain assumption Chi-square test is appropriate for determining whether prompting methods produce statistically significant differences in vulnerability counts
    Invoked to support the claim of no significant reductions across methods.

pith-pipeline@v0.9.1-grok · 5752 in / 1332 out tokens · 49647 ms · 2026-06-30T15:15:00.222985+00:00 · methodology

0 comments
read the original abstract

The growing use of Large Language Models (LLMs) for automated code generation has enhanced software development efficiency, but often at the cost of security. Generated code frequently overlooks critical concerns, leaving it vulnerable to issues such as weak encryption and improper input validation. To investigate this problem, we present a comprehensive empirical evaluation of the security quality of LLM-generated code across five LLMs and four programming languages (Java, C++, C, and Python), examining the impact of multiple prompt engineering methods. We introduce a weaknesses-aware zero-shot chain-of-thought (WA-0CoT) prompting strategy that enriches prompts with security context using CWE mappings to guide model reasoning. Our empirical analysis, supported by chi-square tests, finds no statistically significant reductions in vulnerability frequency or density across prompt methods. However, prompting strategies, including WA-0CoT, systematically influence the compositional distribution of CWE categories, with effects varying by programming language. These findings suggest that while security-aware prompting alters the structure of generated weaknesses, prompt engineering alone is insufficient to reliably reduce overall vulnerability levels. The results highlight the importance of language-aware and model-aware prompt design when evaluating the security properties of LLM-generated code.

Figures

Figures reproduced from arXiv: 2605.24298 by Ahmed Sabbah, David Mohaisen, Mohammad Alkhanafseh, Mohammad Hammoudeh, Mohammed Kharma.

Figure 1
Figure 1. Figure 1: Methodology evaluation pipeline. description in the prompt but does not have labeled data for training on par￾ticular input-output pairs. The model then utilizes its prior knowledge to make predictions based on the provided prompt for the new task. Given an LLM f with parameters θ, a zero-shot prompt is Y = f(X; θ) where X is a natural language prompt describing the task, Y is the generated output, and θ r… view at source ↗
Figure 2
Figure 2. Figure 2: The proposed weaknesses-aware chain-of-thought method pipeline. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The task description classification template. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The code generation (chain-of-thought) template. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The zero-shot code generation template. Zero-shot. In this method, we use the original task description with a general security-related context instruction, as shown in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Radar view of average vulnerabilities (axes: Total, Blocker, High) with [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Stacked totals (High + Blocker) per LLM grouped by prompt method. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Chi-squared test results for each LLM per programming language for both [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages · 1 internal anchor

  1. [1]

    JetBrains : Developer tools for professionals and teams

    --- (2024 a ) C++ programming---the state of developer ecosystem in 2023 infographic. JetBrains : Developer tools for professionals and teams. https://www.jetbrains.com/lp/devecosystem-2023/cpp/, (Accessed on 09/07/2024)

    work page 2024

  2. [2]

    https://claude.ai/new, (Accessed on 09/05/2024)

    --- (2024 b ) Claude. https://claude.ai/new, (Accessed on 09/05/2024)

    work page 2024

  3. [3]

    https://www.sonarsource.com/products/sonarqube/, (Accessed on 05/12/2024)

    --- (2024 c ) Code quality, security & static analysis tool with SonarQube . https://www.sonarsource.com/products/sonarqube/, (Accessed on 05/12/2024)

    work page 2024

  4. [4]

    https://www.codewars.com/, (Accessed on 09/09/2024)

    --- (2024 d ) Codewars - achieve mastery through coding practice and developer mentorship. https://www.codewars.com/, (Accessed on 09/09/2024)

    work page 2024

  5. [5]

    https://cwe.mitre.org/about/new_to_cwe.html, (Accessed on 05/12/2024)

    --- (2024 e ) CWE - new to CWE . https://cwe.mitre.org/about/new_to_cwe.html, (Accessed on 05/12/2024)

    work page 2024

  6. [6]

    https://cwe.mitre.org/data/index.html, [Online; accessed 2024-12-16]

    --- (2024 f ) CWE list version 4.16. https://cwe.mitre.org/data/index.html, [Online; accessed 2024-12-16]

    work page 2024

  7. [7]

    https://edabit.com/, (Accessed on 09/09/2024)

    --- (2024 g ) Edabit // learn to code with 10,000+ interactive challenges. https://edabit.com/, (Accessed on 09/09/2024)

    work page 2024

  8. [8]

    https://gemini.google.com/, (Accessed on 05/12/2024)

    --- (2024 h ) Gemini - chat to supercharge your ideas. https://gemini.google.com/, (Accessed on 05/12/2024)

    work page 2024

  9. [9]

    https://github.com/features/copilot, (Accessed on 05/12/2024)

    --- (2024 i ) Github copilot---your ai pair programmer. https://github.com/features/copilot, (Accessed on 05/12/2024)

    work page 2024

  10. [10]

    https://openai.com/index/gpt-4/, (Accessed on 05/12/2024)

    --- (2024 j ) GPT -4 | OpenAI . https://openai.com/index/gpt-4/, (Accessed on 05/12/2024)

    work page 2024

  11. [11]

    https://mistral.ai/, (Accessed on 05/12/2024)

    --- (2024 k ) Mistral ai | frontier ai in your hands. https://mistral.ai/, (Accessed on 05/12/2024)

    work page 2024

  12. [12]

    https://www.orientsoftware.com/blog/most-popular-programming-languages/, (Accessed on 05/12/2024)

    --- (2024 l ) Most popular programming languages in 2024 & beyond. https://www.orientsoftware.com/blog/most-popular-programming-languages/, (Accessed on 05/12/2024)

    work page 2024

  13. [13]

    https://www.perplexity.ai/, (Accessed on 09/05/2024)

    --- (2024 m ) Perplexity. https://www.perplexity.ai/, (Accessed on 09/05/2024)

    work page 2024

  14. [14]

    https://pieces.app/, (Accessed on 05/12/2024)

    --- (2024 n ) Pieces for developers | ai-enabled developer productivity. https://pieces.app/, (Accessed on 05/12/2024)

    work page 2024

  15. [15]

    https://leetcode.com/problemset/, (Accessed on 09/09/2024)

    --- (2024 o ) Problems - leetcode. https://leetcode.com/problemset/, (Accessed on 09/09/2024)

    work page 2024

  16. [16]

    https://www.tabnine.com/, (Accessed on 05/12/2024)

    --- (2024 p ) Tabnine ai coding assistant | private, personalized, protected. https://www.tabnine.com/, (Accessed on 05/12/2024)

    work page 2024

  17. [17]

    https://codeql.github.com/codeql-query-help/codeql-cwe-coverage/, (Accessed on 2025-07-04)

    --- (2025 a ) Codeql cwe coverage. https://codeql.github.com/codeql-query-help/codeql-cwe-coverage/, (Accessed on 2025-07-04)

    work page 2025

  18. [18]

    https://exercism.org/, (Accessed on 2025-07-04)

    --- (2025 b ) Exercism. https://exercism.org/, (Accessed on 2025-07-04)

    work page 2025

  19. [19]

    An C, Zhang J, Zhong M, Li L, Gong S, Luo Y, Xu J, Kong L (2024) Why does the effective context length of llms fall short? CoRR abs/2410.18745

    work page arXiv 2024

  20. [20]

    https://huggingface.co/blog/starcoder, accessed: 2024-05-14

    BigCode Project (2024) Starcoder: A state-of-the-art llm for code. https://huggingface.co/blog/starcoder, accessed: 2024-05-14

    work page 2024

  21. [21]

    (2020) Language models are few-shot learners

    Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, et al. (2020) Language models are few-shot learners. In: NeurIPS, pp 159:1--159:25

    work page 2020

  22. [22]

    A Survey on Evaluation of Large Language Models

    Chang Y, Wang X, Wang J, Wu Y, Zhu K, Chen H, Yang L, Yi X, Wang C, Wang Y, Ye W, Zhang Y, Chang Y, Yu PS, Yang Q, Xie X (2023) A survey on evaluation of large language models. CoRR abs/2307.03109

    work page Pith review arXiv 2023

  23. [23]

    IEEE Trans Dependable Secur Comput 22(4):3602--3615, doi:10.1109/TDSC.2025.3535218, ://doi.org/10.1109/TDSC.2025.3535218

    Choi S, Mohaisen D (2025) Attributing chatgpt-generated source codes. IEEE Trans Dependable Secur Comput 22(4):3602--3615, doi:10.1109/TDSC.2025.3535218, ://doi.org/10.1109/TDSC.2025.3535218

    work page doi:10.1109/tdsc.2025.3535218 2025

  24. [24]

    Choi S, Alkinoon A, Alghuried A, Alghamdi A, Mohaisen D (2025) Attributing chatgpt-transformed synthetic code. In: 45th IEEE International Conference on Distributed Computing Systems, ICDCS 2025, Glasgow, United Kingdom, July 21-23, 2025, IEEE , pp 89--99, doi:10.1109/ICDCS63083.2025.00018, ://doi.org/10.1109/ICDCS63083.2025.00018

    work page doi:10.1109/icdcs63083.2025.00018 2025

  25. [25]

    Large Language Models for Software Engineering: A Systematic Literature Review,

    Hou X, Zhao Y, Liu Y, Yang Z, Wang K, Li L, Luo X, Lo D, Grundy JC, Wang H (2023) Large language models for software engineering: A systematic literature review. CoRR abs/2308.10620, 2308.10620

    work page arXiv 2023

  26. [26]

    In: Findings of the Association for Computational Linguistics (ACL), pp 1049--1065

    Huang J, Chang KCC (2023) Towards reasoning in large language models: A survey. In: Findings of the Association for Computational Linguistics (ACL), pp 1049--1065

    work page 2023

  27. [27]

    IEEE Transactions on Dependable and Secure Computing pp 1--15, doi:10.1109/TDSC.2026.3672745, ://doi.org/10.1109/TDSC.2026.3672745

    Kharma MF, Choi S, Alkhanafseh M, Mohaisen D (2026) Security and quality in llm-generated code: a multi-language, multi-model analysis. IEEE Transactions on Dependable and Secure Computing pp 1--15, doi:10.1109/TDSC.2026.3672745, ://doi.org/10.1109/TDSC.2026.3672745

    work page doi:10.1109/tdsc.2026.3672745 2026

  28. [28]

    Khoury R, Avila AR, Brunelle J, Camara BM (2023) How secure is code generated by chatgpt? In: IEEE SMC

    work page 2023

  29. [29]

    In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)

    Kim G, Baldi P, McAleer S (2023) Language models can solve computer tasks. In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)

    work page 2023

  30. [30]

    Lin J, Mohaisen D (2025) From large to mammoth: A comparative evaluation of large language models in vulnerability detection. In: 32nd Annual Network and Distributed System Security Symposium, NDSS 2025, San Diego, California, USA, February 24-28, 2025, The Internet Society, ://www.ndss-symposium.org/ndss-paper/from-large-to-mammoth-a-comparative-evaluati...

    work page 2025

  31. [31]

    In: Findings of the Association for Computational Linguistics (ACL), pp 2824--2835, doi:10.18653/v1/2022.findings-acl.222

    Logan RL, Balazevic I, Wallace E, Petroni F, Singh S, Riedel S (2022) Cutting down on prompts and parameters: Simple few-shot learning with language models. In: Findings of the Association for Computational Linguistics (ACL), pp 2824--2835, doi:10.18653/v1/2022.findings-acl.222

    work page doi:10.18653/v1/2022.findings-acl.222 2022

  32. [32]

    In: Advances in Neural Information Processing Systems (NeurIPS)

    Madaan A, Tandon N, Gupta P, Hallinan S, Gao L, Wiegreffe S, Alon U, Dziri N, Prabhumoye S, Yang Y, Gupta S, Majumder BP, Hermann K, Welleck S, Yazdanbakhsh A, Clark P (2023) Self-refine: Iterative refinement with self-feedback. In: Advances in Neural Information Processing Systems (NeurIPS)

    work page 2023

  33. [33]

    Large Language Models: A Survey

    Minaee S, Mikolov T, Nikzad N, Chenaghlu M, Socher R, Amatriain X, Gao J (2024) Large language models: A survey. CoRR abs/2402.06196, 2402.06196

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    Perry N, Srivastava M, Kumar D, Boneh D (2023) Do users write more insecure code with ai assistants? In: ACM CCS

    work page 2023

  35. [35]

    (2019) Language models are unsupervised multitask learners

    Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I, et al. (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9

    work page 2019

  36. [36]

    In: Proceedings of the 32nd USENIX Security Symposium, pp 2205--2222

    Sandoval G, Pearce H, Nys T, Karri R, Garg S, Dolan-Gavitt B (2023) Lost at c: A user study on the security implications of large language model code assistants. In: Proceedings of the 32nd USENIX Security Symposium, pp 2205--2222

    work page 2023

  37. [37]

    In: 30th USENIX Security Symposium, pp 1559--1575

    Schuster R, Song C, Tromer E, Shmatikov V (2021) You autocomplete me: Poisoning vulnerabilities in neural code completion. In: 30th USENIX Security Symposium, pp 1559--1575

    work page 2021

  38. [38]

    In: ACM MSR4P&S

    Siddiq ML, Santos JCS (2022) Securityeval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques. In: ACM MSR4P&S

    work page 2022

  39. [39]

    CoRR abs/2311.00889

    Siddiq ML, Santos JCS (2023) Generate and pray: Using SALLMS to evaluate the security of LLM generated code. CoRR abs/2311.00889

    work page arXiv 2023

  40. [40]

    Díaz Ferreyra, Markus Mutas, Salem Dhif, and Ric- cardo Scandariato

    Tony C, Ferreyra NED, Mutas M, Dhif S, Scandariato R (2025) Prompting techniques for secure code generation: A systematic investigation. ACM Trans Softw Eng Methodol 34(8):225:1--225:53, doi:10.1145/3722108, ://doi.org/10.1145/3722108

    work page doi:10.1145/3722108 2025

  41. [41]

    CoRR abs/2302.07248

    Vasconcelos H, Bansal G, Fourney A, Liao QV, Vaughan JW (2023) Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in ai-powered code completions. CoRR abs/2302.07248

    work page arXiv 2023

  42. [42]

    In: Advances in Neural Information Processing Systems

    Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser , Polosukhin I (2017) Attention is all you need. In: Advances in Neural Information Processing Systems

    work page 2017

  43. [43]

    In: Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)

    Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi EH, Le QV, Zhou D (2022) Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)

    work page 2022

  44. [44]

    CoRR abs/2312.05275, 2312.05275

    Wu F, Zhang Q, Bajaj AP, Bao T, Zhang N, Wang R, Xiao C (2023) Exploring the limits of chatgpt in software security applications. CoRR abs/2312.05275, 2312.05275

    work page arXiv 2023

  45. [45]

    (2023).Evaluating the code quality of AI-assisted code generation tools

    Yetistiren B, \" O zsoy I, Ayerdem M, T \" u z \" u n E (2023) Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. CoRR abs/2304.10778

    work page arXiv 2023

  46. [46]

    Progressive-Hint Prompting Improves Reasoning in Large Language Models

    Zheng C, Liu Z, Xie E, Li Z, Li Y (2023) Progressive-hint prompting improves reasoning in large language models. CoRR abs/2304.09797, doi:10.48550/ARXIV.2304.09797, 2304.09797

    work page Pith review doi:10.48550/arxiv.2304.09797 2023

  47. [47]

    In: International Conference on Learning Representations (ICLR), OpenReview.net

    Zhou D, Sch \" a rli N, Hou L, Wei J, Scales N, Wang X, Schuurmans D, Cui C, Bousquet O, Le QV, Chi EH (2023) Least-to-most prompting enables complex reasoning in large language models. In: International Conference on Learning Representations (ICLR), OpenReview.net

    work page 2023

  48. [48]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all ...

    work page

  49. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or or or or FUNCTION ...

    work page