Pith. sign in

REVIEW 4 cited by

Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.12596 v2 pith:DB6JYEYL submitted 2023-07-24 cs.SE

Yue Liu , Thanh Le-Cong , Ratnadira Widyasari , Chakkrit Tantithamthavorn , Li Li , Xuan-Bach D. Le , David Lo This is my paper · ORCID
classification cs.SE
keywords codechatgptchatgpt-generatedissuesprogramsqualityprogrammingtasks
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We systematically study the quality of 4,066 ChatGPT-generated code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is three folds. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT's self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of AI models like ChatGPT.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Detecting LLM-generated Code with Subtle Modification by Adversarial Training

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeGPTSensor+, trained with adversarial samples that combine identifier renaming and structure transformation, is substantially more robust to subtle modifications of LLM-generated code than the original CodeGPTSensor.

  2. Turning the Tide: Repository-based Code Reflection

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.

  3. From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models

    cs.SE 2025-05 conditional novelty 5.0 of 10

    Prompt bias lessons derived from an LLM's own error explanations improve its clone detection F1 score by up to 10.81% on two benchmarks.

  4. Multi-Agent Collaboration for Multilingual Code Instruction Tuning

    cs.CL 2025-02 reject novelty 5.0 of 10

    A multi-agent collaboration framework for generating multilingual code instruction data is introduced, but the reported evaluations show the fine-tuned model underperforming its own base instruct model.

Pith tools