Pith. sign in

REVIEW 12 cited by

Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.10778 v2 pith:L5FSLFGW submitted 2023-04-21 cs.SE

Burak Yetiştiren , Iş{i}k Özsoy , Miray Ayerdem , Eray Tüzün This is my paper
classification cs.SE
keywords codeamazoncodewhisperercopilotgithubchatgptgenerationtools
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Context: AI-assisted code generation tools have become increasingly prevalent in software engineering, offering the ability to generate code from natural language prompts or partial code inputs. Notable examples of these tools include GitHub Copilot, Amazon CodeWhisperer, and OpenAI's ChatGPT. Objective: This study aims to compare the performance of these prominent code generation tools in terms of code quality metrics, such as Code Validity, Code Correctness, Code Security, Code Reliability, and Code Maintainability, to identify their strengths and shortcomings. Method: We assess the code generation capabilities of GitHub Copilot, Amazon CodeWhisperer, and ChatGPT using the benchmark HumanEval Dataset. The generated code is then evaluated based on the proposed code quality metrics. Results: Our analysis reveals that the latest versions of ChatGPT, GitHub Copilot, and Amazon CodeWhisperer generate correct code 65.2%, 46.3%, and 31.1% of the time, respectively. In comparison, the newer versions of GitHub CoPilot and Amazon CodeWhisperer showed improvement rates of 18% for GitHub Copilot and 7% for Amazon CodeWhisperer. The average technical debt, considering code smells, was found to be 8.9 minutes for ChatGPT, 9.1 minutes for GitHub Copilot, and 5.6 minutes for Amazon CodeWhisperer. Conclusions: This study highlights the strengths and weaknesses of some of the most popular code generation tools, providing valuable insights for practitioners. By comparing these generators, our results may assist practitioners in selecting the optimal tool for specific tasks, enhancing their decision-making process.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 68 citations worldwide. Full citation record

  1. Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Generative UI tools' stated design rationales are not fully implemented in about 25% of cases, with functional requirements failing most often.

  2. Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Prompt language affects LLM code generation, but English is not consistently best: Chinese prompts improve Python correctness on CoderEval, while quality and lexicon effects vary by model and programming language.

  3. Developers' Experience with Generative AI -- First Insights from an Empirical Mixed-Methods Field Study

    cs.HC 2025-12 conditional novelty 6.0 of 10

    Moderate single-mode Copilot use improved developer efficiency and reduced workload, while combined or excessive use diminished these benefits, and chat use improved task completion.

  4. WildCode Revisited: A Comprehensive Empirical Study on the Security of LLM-Generated Code

    cs.CR 2025-12 conditional novelty 6.0 of 10

    Using real ChatGPT conversation logs, code generated by the model is frequently insecure and users rarely request security-related code.

  5. Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity

    cs.SE 2025-08 conditional novelty 6.0 of 10

    Across 500k Python/Java samples, AI-generated functions are shorter, simpler, and trigger more security findings, while human functions carry more complexity and maintainability warnings.

  6. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models

    cs.SE 2025-07 conditional novelty 6.0 of 10

    4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.

  7. Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis

    cs.SE 2025-07 conditional novelty 6.0 of 10

    In a benchmark of LLM-generated C code for graph analysis, Claude Sonnet 4 Extended produced the most correct, fastest, and most memory-efficient implementations, beating human baselines on triangle counting.

  8. Eradicating the Unseen: Detecting, Exploiting, and Remediating a Path Traversal Vulnerability across GitHub

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A single vulnerable Node.js path traversal pattern was found in 1,756 GitHub projects, most rated critical, and the authors' automated pipeline produced patches, disclosures, and evidence that LLMs have learned the pattern.

  9. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  10. Do Generative AI Tools Ensure Green Code? An Investigative Study

    cs.SE 2025-06 conditional novelty 5.0 of 10

    An early study of three AI coding tools found that generated code frequently fails to follow sustainable coding practices, especially for GitHub Copilot.

  11. Secure Code Generation at Scale with Reflexion

    cs.CR 2025-11 conditional novelty 4.0 of 10

    Three-round reflexion prompting raises LLM-generated code security from 70.74% to 79.43% as measured by the ICD static detector on the Instruct Prime benchmark.

  12. Quality Assessment of Python Tests Generated by Large Language Models

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A comparative study of Python test suites generated by GPT-4o, Amazon Q, and LLama 3.3 found 151 execution errors and 512 test smells, with assertion failures and low-cohesion tests most common.

Pith tools