{"id":"106facdf-4c59-42e8-a838-434123ceaf51","arxiv_id":"2508.21634","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 500k Python/Java samples, AI-generated functions are shorter, simpler, and trigger more security findings, while human functions carry more complexity and maintainability warnings.","lead":"This study compares Python and Java functions written by humans with code generated by ChatGPT, DeepSeek-Coder, and Qwen-Coder across more than 500,000 samples. It finds AI code is shorter and simpler, but triggers more security-related findings, while human code shows more complexity and maintainability issues.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human and AI code differ in task format and length, so raw static-analysis counts may reflect context, not authorship.","rationale":"The reader's weakest assumption and the concern raised here coincide: the human and AI corpora are not comparable units because of integration context and length differences, and raw static-analysis counts are treated as directly interpretable. This is load-bearing for the central claim because the headline conclusions—AI code has more unused constructs, hardcoded debugging, and high-risk vulnerabilities—are all drawn from those raw counts. The proposed concrete test would disentangle authorship from task format by adding a human baseline that faces the same isolated-generation task. If the differences disappear, the paper's conclusions would need to be substantially qualified; if they persist, the central claim gains real support. The reader's conditional verdict already captures this uncertainty, so the verdict should remain UNCHANGED. The concern is not an internal contradiction, but a threat to construct validity that the paper acknowledges yet does not resolve.","tokens_in":17393,"tokens_out":5749,"duration_ms":74112,"concrete_test":"Construct a human-written baseline from the same isolated docstring+signature tasks, e.g., human solutions to LeetCode/AtCoder/contest prompts, and run the identical Pylint/PMD/Semgrep pipeline. Compare defect/CWE profiles and length-adjusted rates (per 100 LOC) between these human-isolated functions and the LLM outputs. If human isolated code shows the same unused-argument, System.out.println, and CWE-78 rates as the AI code, the reported differences are task-format artifacts; if it resembles the GitHub human profile, the confound is refuted. As part of the same check, manually validate a stratified random sample of 100 Semgrep findings per author/CWE; if true-positive rates differ across authors, raw security counts must be reweighted before concluding AI code is more vulnerable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of RQ1–RQ3 is that human functions mined from CodeSearchNet and LLM outputs generated from the same docstrings are equivalent units. Section IV constructs the dataset exactly that way, and Section V compares raw Pylint/PMD/Semgrep counts without normalizing for integration context or code length. This premise is insecure in two concrete ways. First, human functions are embedded in 16,928 real repositories, with imports, classes, call sites, and project conventions, while AI functions are isolated snippets produced from a docstring and signature. Defects such as unused-argument (up to 213k for Qwen in Fig. 1) and System.out.println/printStackTrace are natural consequences of generating a standalone function with no surrounding context, not necessarily evidence of an author-specific defect profile. Second, human functions average 12.72–13.38 NLOC while AI functions average 4.47–8.42 NLOC (Table V); raw violation counts are not volume-adjusted, and no stratified or regression controls are reported. The security claim is similarly exposed: Semgrep fires on syntactic patterns in isolated snippets (e.g., CWE-78, CWE-798), and no random sample is manually validated for true positives. Section VI acknowledges these static-analysis and context limits but does not test whether they change the conclusions. Because every comparison in Tables III–V and Figures 1–2 depends on this equivalence, the reported 'author' effect is confounded with task format and code length.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale comparison of human-written and AI-generated code (ChatGPT, DeepSeek-Coder, Qwen-Coder) across Python and Java, using Pylint/PMD for defect detection, Semgrep for security vulnerabilities, and structural complexity metrics (NLOC, CCN, tokens, unique tokens). Defects are mapped to Orthogonal Defect Classification (ODC) and vulnerabilities to CWE. The dataset pairs 507,044 docstrings with human functions from CodeSearchNet/HMCorp and LLM-generated implementations. The central claims are that AI code is simpler and more repetitive, with more unused constructs and hardcoded debugging, while human code is more complex and maintainability-challenged; and that AI code contains more high-risk vulnerabilities. The paper includes a replication package and documents the rule-to-ODC mapping.","tokens_in":17770,"tokens_out":2369,"duration_ms":29962,"significance":"If the results are valid, the paper is significant: it provides a large-scale, cross-language, multi-model characterization of AI- versus human-written code quality and security, with practical implications for QA practices and for the design of static analysis tools. Strengths include the scale (over 500k samples, ~17k repositories), the use of two standard taxonomies (ODC and CWE), the multiple LLMs, and the public replication artifacts. The documented two-stage consensus mapping of tool rules to ODC is a positive methodological feature. However, the central comparison is currently confounded by task format and code length, so the headline conclusions are not yet established.","major_comments":[{"comment":"The central unit of comparison is not equivalent. Human functions are mined from 16,928 real repositories with surrounding imports, classes, call sites, and project conventions, while AI functions are generated from docstrings and signatures as isolated snippets. Table V shows human functions average 12.72–13.38 NLOC versus 4.47–8.42 for AI functions. Raw defect counts are not volume-adjusted, and no stratified analysis or regression control by length/complexity is reported. Since the prompts include the original signature, rules such as unused-argument (up to 213,264 for Qwen in Python) may be triggered by prompt design rather than by an author-specific defect propensity. This confound affects every comparison in Tables III–V and Figures 1–2, including the abstract's claim that AI code is 'more prone to unused constructs'. I recommend reporting defect densities per function and per NLOC","section":"§IV and §V-A, Tables III–V"},{"comment":"The security comparison uses Semgrep's syntactic pattern matching on isolated snippets, with no manual validation of true positives. High-risk CWEs such as CWE-78 (OS Command Injection), CWE-798 (Hardcoded Credentials), and CWE-489 (Active Debug Code) may be over-reported when functions are generated standalone from docstrings that contain examples or configuration-like text. The claim that AI code contains 'more high-risk security vulnerabilities' depends on these raw counts. The authors should validate a random sample of the flagged vulnerabilities (stratified by CWE and author) and report precision; otherwise the observed differences may reflect the generation task rather than a stable security profile.","section":"§V-B, Table IV and Figure 2"},{"comment":"The handling of 'incorrect' samples needs to be made precise and its effect on the comparisons assessed. For DSC, the paper reports 20,783 incorrect Python samples and 34,382 incorrect Java samples, mostly empty or unparseable outputs, while humans have 3,712 and 245. These samples are excluded from defect counts, so the reported 'defective sample' percentages are not directly comparable across authors. The paper should state whether the percentages in Table III are computed over all samples or only parseable ones, and should analyze sensitivity to the exclusion of incorrect outputs (e.g., by treating empty outputs as a distinct outcome).","section":"§III-A and §V-A, Table III"},{"comment":"The threats section acknowledges the static-analysis and context limitations, but it does not test whether they change the conclusions. Specifically, the internal-validity paragraph focuses on generation consistency and does not address the human/AI task-format asymmetry. Since the confound is load-bearing, the paper needs a concrete robustness analysis (e.g., comparing only functions of similar length, or re-running the defect analysis on a context-matched subsample) rather than a general acknowledgment.","section":"§VI, Threats to Validity"}],"minor_comments":[{"comment":"Figure 1 captions are identical for (a) and (b) and do not state the language; the language is only in the subplot labels. Please make the captions self-contained.","section":"§V-A, Fig. 1"},{"comment":"The table uses blue/red for best/worst scores, but these colors are not accessible in grayscale and are not described in the text. Consider adding symbols or a separate column.","section":"§V-A, Table III"},{"comment":"The meaning of 'Avg. Code Len.' is ambiguous (characters? tokens? lines?). Define it in the table caption or text.","section":"§IV, Table II"},{"comment":"The text says DSC Python has 19,612 empty predictions, while Table III reports 20,783 incorrect samples for DSC Python. Clarify the relationship between 'empty' and 'incorrect' and make the numbers consistent.","section":"§V-A"},{"comment":"The ODC mapping table (Table I) would benefit from an explicit statement of inter-rater agreement (e.g., Cohen's kappa) rather than only 'disagreements were resolved through discussion'.","section":"§III-A"},{"comment":"The unique-token metric is computed per author across the entire corpus, but the text describes it as a per-function metric. Clarify the aggregation unit in the metric definition (§III-C).","section":"§V-C, Table V"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the largest, most thoroughly documented study of its kind, and the descriptive material is worth having. But the central comparison is not as clean as the tables suggest. Human functions come from real repositories with surrounding context and average 12–13 lines; AI functions are generated from docstrings alone and run 4–8 lines. Since every defect count is raw and unnormalized, the 'author' effect is entangled with length and task format. That is a fixable problem, and the authors are honest enough to acknowledge most of it in Section VI.\n\nWhat is genuinely new: applying ODC and CWE across 285k Python and 222k Java samples from three models, with hundreds of static-analysis rules manually mapped to a common taxonomy. That is real work, and the replication package (code, data, rule mappings) is a genuine asset. The qualitative profiles—AI code simpler but full of unused arguments and hardcoded debugging; human code more complex but maintainability-heavy—are plausible and consistent with smaller studies. The security result (AI code triggers more high-risk CWEs per sample) is striking but rests on Semgrep pattern matches, not validated exploits.\n\nSoft spots, in order of severity. First, no normalization for length. Unused-argument and System.out.println counts are naturally higher in standalone functions with no callers; a per-line or stratified analysis would show whether the profile differences survive. Second, no statistical tests; differences are stated from raw counts. Third, the 'high-risk vulnerability' claim is only as strong as Semgrep's true positive rate; a small manual validation of random samples would help. Fourth, only one model (ChatGPT) was sampled from actual assistant usage; the other two were generated specifically for this study, so the comparison is really 'human corpus vs prompt-with-docstring generation', not 'human vs assistant in the wild'.\n\nNone of this sinks the paper. The descriptive profiles are still informative, and the authors flag most limitations themselves. But as it stands, the abstract's claim that AI code contains more high-risk vulnerabilities is oversold. With normalization, subset validation, and significance tests, this could become a solid reference.\n\nWho it is for: anyone building QA tooling, studying LLM code generation, or designing evaluations of AI assistants. It deserves a serious referee: the scale and documentation justify the time, and the confounds are addressable in revision. I would accept it for review and ask for those revisions.","headline":"The biggest, best-documented human-vs-AI code quality comparison to date, but the headline claims are confounded by length and context and need normalization before they fully hold up.","tokens_in":18191,"tokens_out":2361,"would_cite":true,"duration_ms":26194,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI-generated code is simpler and more repetitive than human code, yet more prone to unused constructs, leftover debug code, and high-severity security vulnerabilities such as command injection and hardcoded credentials.","keywords":["AI code generation","Orthogonal Defect Classification","Common Weakness Enumeration","security vulnerabilities","static analysis","code complexity","large language models","Python and Java code quality"],"falsifier":"Re-run the same static-analysis pipeline on length-matched and complexity-matched subsets of human and AI functions, for example only functions of 4–8 lines with cyclomatic complexity 1–2, or human functions stripped of their surrounding project context. If the defect-profile and vulnerability-rate gaps shrink to statistical insignificance, the authorship effect is an artifact of comparing short isolated snippets with longer contextual functions; if they persist, the paper's conclusion survives.","tokens_in":17345,"feed_emoji":"⚠️","tokens_out":6988,"duration_ms":76237,"temperature":0.7,"pith_summary":"This paper asks whether code written by human developers and code produced by large language models differ in ways that matter for software quality, and answers yes. It compares more than 500,000 Python and Java functions—human functions mined from open-source repositories against functions that three LLMs generated from the same docstrings—and classifies every static-analysis finding using two standard taxonomies, one for defects and one for security weaknesses. The paper finds that AI-generated functions are shorter, less complex, and more repetitive, yet they carry more unused constructs, leftover debug code, and high-risk vulnerabilities such as command injection and hardcoded credentials. Human functions are longer and more complex, with a defect profile dominated by maintainability, algorithmic, and exception-handling issues. If the comparison holds, AI-assisted development needs its own quality-assurance and security-review practices rather than a direct transplant of the ones built for human code.","feed_headline":"AI-generated code has more high-risk security flaws","feed_subtitle":"A 500k-function study finds LLMs write simpler but riskier code, repeating flaws like command injection and hardcoded secrets.","key_machinery":"The paired-generation design carries the argument: each human function's own docstring and signature become the prompt for three LLMs, so every author produces an answer to the same specification. On top of that, two normalization layers make the counts comparable across languages and authors: Orthogonal Defect Classification maps tool-specific rule violations from Python and Java static analyzers into a common set of defect types, and the Common Weakness Enumeration maps security-rule hits into a common catalog of weakness types. Without these layers, raw static-analyzer counts would not be commensurable.","core_discovery":"On the paper's own terms: author identity leaves a systematic fingerprint on code. When defects are normalized through Orthogonal Defect Classification (a taxonomy sorting bugs by type—assignment, algorithm, interface, checking, timing, structure) and security findings through the Common Weakness Enumeration, AI-generated code clusters in the assignment and class-structure categories (unused arguments, unused variables, placeholder classes), while human code clusters in algorithm, interface, and checking categories (complex control flow, protected-member access, weak exception handling). AI code triggers high-severity CWE entries—OS command injection, uncontrolled resource consumption, hardc","pith_inferences":["Since AI functions average roughly half the lines of human functions, per-line or per-token vulnerability density could be even higher than the per-sample differences reported; computing rates normalized by length would sharpen the comparison.","The massive unused-argument counts across all three models suggest LLMs preserve prompt signatures without binding parameters to the body; varying signatures while holding docstrings fixed could test whether this is a generation-strategy artifact.","The human baseline comes from mature open-source projects, so its maintainability-heavy profile partly reflects accumulated project debt; sampling first-commit or context-free human snippets might narrow—or widen—the gap and isolate the authorship effect.","Static-analysis rule sets may need reweighting for AI code: rules that dominate human code can swamp counts, while rules that matter most for AI code (unused constructs, debug output) are underweighted in typical quality dashboards."],"forward_implications":["AI-generated code will not 'look like' human code to existing quality gates: the same rules will fire on different categories, so teams cannot assume human-calibrated thresholds transfer.","Security review of AI-generated code should prioritize injection, resource-consumption, hardcoded-secret, and information-exposure weaknesses, which are the categories where the gap is largest.","Because AI functions are shorter and simpler, complexity-based quality metrics will systematically understate their risk; simple code is not automatically safe code.","Model choice matters: the three LLMs differ sharply in syntax failure rate, defect profile, and vulnerability rate, so 'AI-generated code' is not a single category for QA purposes.","The quality gap is wider in Java than in Python, suggesting that language strictness amplifies the defects LLMs introduce."],"supporting_citations":[{"why":"Supplies the Orthogonal Defect Classification taxonomy used to normalize tool-specific defect rules across languages and authors.","marker":"[19]"},{"why":"Supplies the paired human-ChatGPT dataset that this study extends into a four-author comparison.","marker":"[39]"},{"why":"Supplies the human functions and docstrings mined from public repositories that anchor the dataset.","marker":"[40]"},{"why":"One of the three LLMs whose outputs are generated from the same docstrings and analyzed.","marker":"[4]"},{"why":"Another LLM that provides alternative implementations in both languages.","marker":"[15]"},{"why":"Python static analyzer whose rule violations are mapped to ODC defect types.","marker":"[16]"},{"why":"Java static analyzer whose rule violations are mapped to ODC defect types.","marker":"[17]"},{"why":"Security-oriented analyzer whose findings are mapped to CWE entries.","marker":"[18]"},{"why":"Taxonomy used to classify and compare security vulnerability types across authors.","marker":"[32]"},{"why":"Reference list used to mark which detected weaknesses are most dangerous.","marker":"[35]"}],"fun_headline_variants":["AI code: simpler, but higher-risk security flaws","500k-function study: AI code has more high-severity vulnerabilities","LLM-written code prone to injection and hardcoded secrets","AI-generated code repeats security flaws more often","Study: AI code less complex, but more dangerous"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The comparison treats a human function extracted from a real repository, with surrounding project context, as the same unit as an AI function generated as an isolated snippet from a docstring, and compares raw violation counts without adjusting for the large length and complexity differences; if context or length drives violation rates, the authorship effect is confounded.","fun_headline_variants_meta":{"raw":{"variants":["AI code: simpler, but higher-risk security flaws","500k-function study: AI code has more high-severity vulnerabilities","LLM-written code prone to injection and hardcoded secrets","AI-generated code repeats security flaws more often","Study: AI code less complex, but more dangerous"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2363,"prompt_tokens":710,"completion_tokens":1653,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":1573}},"tokens_in":454,"tokens_out":1653,"duration_ms":10931,"temperature":1.0,"reasoning_tokens":1573,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:05:51.947313+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same static-analysis pipeline on length-matched and complexity-matched subsets of human and AI functions, for example only functions of 4–8 lines with cyclomatic complexity 1–2, or human functions stripped of their surrounding project context. If the defect-profile and vulnerability-rate gaps shrink to statistical insignificance, the authorship effect is an artifact of comparing short isolated snippets with longer contextual functions; if they persist, the paper's conclusion survives.","supporting_citations":[{"cited_title":"Orthogonal defect classification- a concept for in-process measurements,","cited_arxiv_id":null,"evidence_quote":"Supplies the Orthogonal Defect Classification taxonomy used to normalize tool-specific defect rules across languages and authors."},{"cited_title":"Distinguishing llm-generated from human-written code by contrastive learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the paired human-ChatGPT dataset that this study extends into a four-author comparison."},{"cited_title":"PyLint - Python linter,","cited_arxiv_id":null,"evidence_quote":"Python static analyzer whose rule violations are mapped to ODC defect types."},{"cited_title":"Source Code Analyzer,","cited_arxiv_id":null,"evidence_quote":"Java static analyzer whose rule violations are mapped to ODC defect types."},{"cited_title":"Common Weakness Enumeration,","cited_arxiv_id":null,"evidence_quote":"Taxonomy used to classify and compare security vulnerability types across authors."},{"cited_title":"Top 25 Most Dangerous Software Weaknesses,","cited_arxiv_id":null,"evidence_quote":"Reference list used to mark which detected weaknesses are most dangerous."}],"review_version":1}