REVIEW 10 cited by
Comparing Human and LLM Generated Code: The Jury is Still Out!
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Comparing Human and LLM Generated Code: The Jury is Still Out!
read the original abstract
Much is promised in relation to AI-supported software development. However, there has been limited evaluation effort in the research domain aimed at validating the true utility of such techniques, especially when compared to human coding outputs. We bridge this gap, where a benchmark dataset comprising 72 distinct software engineering tasks is used to compare the effectiveness of large language models (LLMs) and human programmers in producing Python software code. GPT-4 is used as a representative LLM, where for the code generated by humans and this LLM, we evaluate code quality and adherence to Python coding standards, code security and vulnerabilities, code complexity and functional correctness. We use various static analysis benchmarks, including Pylint, Radon, Bandit and test cases. Among the notable outcomes, results show that human-generated code recorded higher ratings for adhering to coding standards than GPT-4. We observe security flaws in code generated by both humans and GPT-4, however, code generated by humans shows a greater variety of problems, but GPT-4 code included more severe outliers. Our results show that although GPT-4 is capable of producing coding solutions, it frequently produces more complex code that may need more reworking to ensure maintainability. On the contrary however, our outcomes show that a higher number of test cases passed for code generated by GPT-4 across a range of tasks than code that was generated by humans. That said, GPT-4 frequently struggles with complex problem-solving that involve in-depth domain knowledge. This study highlights the potential utility of LLMs for supporting software development, however, tasks requiring comprehensive, innovative or unconventional solutions, and careful debugging and error correction seem to be better developed by human programmers. We plot an agenda for the software engineering community.
Forward citations
Cited by 10 Pith papers
-
AIP: A Graph Representation for Learning and Governing Agent Skills
AIP models skills as graphs of discrete steps connected by typed I/O edges under a validated schema, raising agent mean reward from 0.60 to 0.71 and pass rate from 53% to 67% on 27 SkillsBench tasks while enabling nod...
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human code, with humans handling the majority of changes that are mostly feature extensions rather than bug fixes.
-
Do AI Coding Agents Log Like Humans? An Empirical Study
AI agents modify logging less often than humans in 58.4% of repositories but produce higher log density when they change it; explicit logging instructions are rare (4.7%) and ignored 67% of the time, with humans perfo...
-
How to Compare the Security of Code Written by Humans to LLM-generated Code
Proposes and open-sources an automated framework for species-fair security comparisons of human-written, LLM-generated, and hybrid code, validated via feasibility study.
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
-
Can LLMs Produce Better Object-Oriented Designs than Human-Involved Development?
Comparative case study on a postgraduate Java assignment finds PureAI and PostAI projects simpler with lower code smell density than PreAI but show oversimplification and weaker responsibility separation.
-
The Readability Spectrum: Patterns, Issues, and Prompt Effects in LLM-Generated Code
LLM-generated code matches human-written code in overall readability but exhibits different issue patterns, and prompt engineering has limited impact on improving it.
-
An Evaluation of Large Language Models for Detection of Malicious Python Packages
LLMs detect malicious PyPI packages reasonably at package level but identify specific malicious code indicators poorly, a 'granularity gap' that limits them to triage.
-
How to Compare the Security of Code Written by Humans to LLM-generated Code
Proposes and validates via feasibility study an open-source automated framework for reproducible, species-fair security comparisons of human-written, LLM-generated, and hybrid code.
-
At What Cost? Software Developers' Well-Being in the Age of GenAI
The paper proposes shifting GenAI research in software engineering from narrow performance metrics to also include developer well-being, social context, and sustainable productivity.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.