REVIEW 8 cited by
Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
read the original abstract
As AI code assistants become increasingly integrated into software development workflows, understanding how their code compares to human-written programs is critical for ensuring reliability, maintainability, and security. In this paper, we present a large-scale comparison of code authored by human developers and three state-of-the-art LLMs, i.e., ChatGPT, DeepSeek-Coder, and Qwen-Coder, on multiple dimensions of software quality: code defects, security vulnerabilities, and structural complexity. Our evaluation spans over 500k code samples in two widely used languages, Python and Java, classifying defects via Orthogonal Defect Classification and security vulnerabilities using the Common Weakness Enumeration. We find that AI-generated code is generally simpler and more repetitive, yet more prone to unused constructs and hardcoded debugging, while human-written code exhibits greater structural complexity and a higher concentration of maintainability issues. Notably, AI-generated code also contains more high-risk security vulnerabilities. These findings highlight the distinct defect profiles of AI- and human-authored code and underscore the need for specialized quality assurance practices in AI-assisted programming.
Forward citations
Cited by 8 Pith papers
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human code, with humans handling the majority of changes that are mostly feature extensions rather than bug fixes.
-
Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability
A repository-level AGENTS.md governance framework with commit-level AI disclosure, session logging, pre-commit gates, and mandatory human review is demonstrated on two new TMAP8 validation cases.
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
-
Insights into Security-Related AI-Generated Pull Requests
AI-generated security pull requests frequently contain a small set of recurring weaknesses, with many flawed ones merged and rejections driven by process factors rather than technical issues.
-
Towards an Agent Operating System - Lessons from Classical and Cloud OS
Agentic systems need an Agent-OS of thirteen OS/Cloud-OS-derived primitives with precise platform-enforceable semantics before portable production adoption is possible.
-
Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering
Specification-Driven Development, formalized as the SGRM reference model, is argued to be the necessary governance layer for enterprise AI-native software engineering, with the strongest quantitative evidence still un...
-
Reliability of AI Bots Footprints in GitHub Actions CI/CD Workflows
Large-scale analysis of AI bot PRs shows Copilot and Codex achieve the highest CI/CD success rates but more frequent AI contributions correlate with reduced workflow reliability.
-
Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability
Proposes guidance for responsible AI use in scientific software development under NQA-1 standards, illustrated with TMAP8 V&V cases to ensure accountability and auditability.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.