Pith. sign in

REVIEW 2 cited by

Bugs in Large Language Models Generated Code: An Empirical Study

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.08937 v2 pith:HIKU56J3 submitted 2024-03-13 cs.SE cs.AI

Florian Tambon , Arghavan Moradi Dakhel , Amin Nikanjam , Foutse Khomh , Michel C. Desmarais , Giuliano Antoniol This is my paper
classification cs.SEcs.AI
keywords codebugspatternsgeneratedgenerationllm-generatedllmscharacteristics
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) for code have gained significant attention recently. They can generate code in different programming languages based on provided prompts, fulfilling a long-lasting dream in Software Engineering (SE), i.e., automatic code generation. Similar to human-written code, LLM-generated code is prone to bugs, and these bugs have not yet been thoroughly examined by the community. Given the increasing adoption of LLM-based code generation tools (e.g., GitHub Copilot) in SE activities, it is critical to understand the characteristics of bugs contained in code generated by LLMs. This paper examines a sample of 333 bugs collected from code generated using three leading LLMs (i.e., CodeGen, PanGu-Coder, and Codex) and identifies the following 10 distinctive bug patterns: Misinterpretations, Syntax Error, Silly Mistake, Prompt-biased code, Missing Corner Case, Wrong Input Type, Hallucinated Object, Wrong Attribute, Incomplete Generation, and Non-Prompted Consideration. The bug patterns are presented in the form of a taxonomy. The identified bug patterns are validated using an online survey with 34 LLM practitioners and researchers. The surveyed participants generally asserted the significance and prevalence of the bug patterns. Researchers and practitioners can leverage these findings to develop effective quality assurance techniques for LLM-generated code. This study sheds light on the distinctive characteristics of LLM-generated code.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation

    cs.SE 2025-07 reject novelty 6.0 of 10

    An empirical study of 1,023 CoT-code pairs shows that 76.4% of LLM-generated CoTs are low quality and that CoT correctness does not guarantee code correctness.

  2. A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code

    cs.SE 2025-11 conditional novelty 5.0 of 10

    A probabilistic score of code-smell propensity in LLM output is validated, used in a causal analysis, and shown to drop when prompts explicitly discourage known smells.

Pith tools