{"id":"78a5137c-20e1-45bb-add3-c5630babc0f0","arxiv_id":"2501.16857","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"On 72 Python tasks, GPT-4 code passed 87.3% of tests versus 54.9% for one student's code, but was more complex and showed more severe security issues.","lead":"The paper compares Python code written by one university student with GPT-4 output on 72 tasks using static analysis and tests. It finds GPT-4 code passes more tests but is more complex and has more severe security issues, while the human code handles a few domain-heavy tasks better.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Functional-correctness comparison is confounded by self-authored tests: if the test suite was written with knowledge of the human solutions, the 87.3% vs. 54.9% pass-rate gap and the 'complex task' sub-analyses reflect interface mismatch, not capability differences.","rationale":"The reader's weakest assumption was that one fourth-year student is not a representative sample of human programmers. That is a valid external-validity concern. My stress-test identifies a more immediate internal-validity threat: the functional-correctness measurements may be unfair if the test suite was authored with knowledge of the human solution. This threat is load-bearing because the paper's headline quantitative result (GPT-4 passes 87.3% vs. human 54.9%) and its qualitative conclusion (humans beat GPT-4 on complex, knowledge-heavy tasks) both rest on test-pass rates. Even if the single student were representative, biased tests would invalidate the comparison. The paper deserves credit for using standard tools (Pylint, Radon, Bandit, Pytest), reporting effect sizes and p-values, and acknowledging several limitations, including the single-student issue. However, the test-author independence issue is unacknowledged and is not addressed by any of the reported statistical tests. The proposed independent-test audit would settle whether the observed gaps are real or artifacts of the evaluation setup. Since the concern is unresolved and central, the appropriate disposition remains rejection: the paper's central claims are not currently supported by the evidence as presented, though a revised version with an independent, released test suite could change that assessment.","tokens_in":12954,"tokens_out":3861,"duration_ms":38058,"concrete_test":"Recruit at least three independent programmers with no access to any human or LLM solution. Give them only the 72 task descriptions (or a stratified sample including the five highlighted complex tasks) and ask each to write a pytest suite solely from those descriptions, specifying expected interfaces and edge cases. Run the original human code and the GPT-4 code against all three independent suites and make the code and tests public. If GPT-4's pass rate on the highlighted complex tasks is not clearly below the human's, or if the overall pass-rate gap shrinks materially, the central claim about LLM struggles on complex domain-knowledge tasks fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the test-pass rate being a neutral measure of functional correctness. Section III.A says a fourth-year student 'developed the code for the 72 coding tasks' and that 'test files were created for the human- and LLM-generated code,' but it never states that the tests were written independently from the human solutions, by multiple authors, or from task specifications alone. If the same student or the authors, having seen the student's code, wrote the tests, then the tests will encode the human's function signatures, return formats, and edge-case assumptions. The overall pass-rate gap (87.3% vs. 54.9%) and, more importantly, the Section IV.D sub-analyses used to support 'GPT-4 frequently struggles with complex problem-solving' are then systematically biased against the LLM. On the five highlighted tasks (TSP/quantum optimization, SIR simulation, LLM interface, GCP sentiment, anagram debugging), the space of valid designs is large, so interface mismatch is especially likely. The conclusion in Section VII that LLM-generated code 'struggles' on such tasks is therefore not established by the reported evidence. Section VI acknowledges the single-student limitation but does not address test-author bias, which is an internal-validity threat rather than merely a generalizability threat. Because the dataset and scripts are not public, this confound cannot currently be audited.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a comparison of code written by one fourth-year computing student and code generated by GPT-4 (April 2024 version) across 72 Python software engineering tasks. Four quality dimensions are evaluated using Pylint, Bandit, Radon, and Pytest: coding standards compliance, security vulnerabilities, code complexity, and functional correctness. The main reported findings are that human code scores somewhat higher on Pylint compliance, that LLM and human code both contain security flaws but LLM code has more high-severity outliers, that GPT-4 code has higher average cyclomatic complexity (5.0 vs. 3.1), and that GPT-4 passes more test cases overall (87.3% vs. 54.9%). The paper further claims, based on a handful of selected tasks, that humans outperform GPT-4 on complex, domain-knowledge-heavy problems, and concludes that LLMs should complement rather than replace human programmers.","tokens_in":13224,"tokens_out":3846,"duration_ms":34902,"significance":"If the results were methodologically airtight, this would be a valuable multi-dimensional, quantitative comparison of human and LLM code generation, complementing prior work that focuses on a single quality dimension or uses subjective evaluations. The study uses standard, well-known static analysis tools and reports statistical tests, and it is commendable that the authors explicitly acknowledge the single-programmer limitation in Section VI. However, the evidence as presented does not establish the paper's central comparative claims because of threats to internal validity (test provenance), an n=1 human sample, unsupported conclusions about XSS, and reliance on a few hand-picked tasks for the 'complex problem-solving' narrative. The paper's strengths are its breadth of metrics and its candid acknowledgment of some limitations, but these do not compensate for the auditability and validity problems in the functional-correctness comparison.","major_comments":[{"comment":"This is a complete sentence.","section":"III.A and IV.D"},{"comment":"This is a complete sentence.","section":"IV.D.2 and VII"},{"comment":"This is a complete sentence.","section":"VII (Conclusion)"},{"comment":"This is a complete sentence.","section":"III.A and VI"}],"minor_comments":[{"comment":"This is a complete sentence.","section":"III.B (Radon)"},{"comment":"This is a complete sentence.","section":"III.A"},{"comment":"This is a complete sentence.","section":"IV.B"},{"comment":"This is a complete sentence.","section":"IV.D.1"},{"comment":"This is a complete sentence.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper addresses an important and timely question, but the current evidence base is not sufficient for publication in a serious journal. The self-authored, non-public test suite is a fundamental internal-validity threat to the main pass-rate comparison, and the n=1 human sample means the 'human vs. LLM' framing overreaches. In addition, the conclusion contains an unsupported XSS claim that does not appear in the results. These issues are not local presentation fixes; they require a different study design or, at minimum, a major re-analysis with independent benchmarks and multiple programmers. If the authors can provide the test suite and demonstrate its independence, a revised manuscript might be considered, but as submitted the central claims are not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a readable, multi-dimensional comparison of GPT-4 and one human's Python code on 72 tasks, but the headline pass-rate comparison is compromised by self-authored tests and n=1 human, so the central conclusion about \"humans\" does not hold as stated.\n\nWhat's new: the specific numbers -- 87.3% vs 54.9% test pass, Radon complexity 5.0 vs 3.1, Bandit severity breakdowns -- are new for this task set and model. The paper does a decent job of combining Pylint, Radon, Bandit, and Pytest in one study, and the static-analysis dimensions (complexity, coding standards) are internally consistent and less dependent on the test-authoring concern. The authors also acknowledge the single-student limitation in Section VI and the one-model, one-language scope, which is more than many papers do.\n\nWhere it gets soft: the functional-correctness comparison is the load-bearing result, and it rests on tests written by the study team. Section III.A says test files were created for both code outputs, but never says they were written from task specs alone, blind to the human solutions. With one fourth-year student as \"human,\" interface mismatch is a real threat: if the tests encode the student's signatures and edge-case assumptions, the 87.3 vs 54.9 gap is partly an artifact. The Section IV.D sub-analyses (TSP, SIR, GCP, anagram) are used to claim GPT-4 \"struggles with complex problem-solving,\" but these are selected examples with small per-task test counts and the same test-bias problem. Also, the conclusion in Section VII says cross-site scripting was \"significantly more common\" in LLM code, but the results section never reports an XSS analysis -- that claim appears unsupported. The dataset is \"available on request\" rather than public, so the confound cannot be audited.\n\nVerdict: the paper is a reasonable pilot study, not a definitive comparison. It would benefit from a larger human sample, independently written tests, and public data. I would send it to peer review with a request for major revision, because the static-analysis results and the task set are useful, and the questions it asks are important. But I would not cite the pass-rate numbers as evidence until the test-authoring issue is fixed.\n\nRecommendation: engage with it as a working paper, not as settled evidence.","headline":"A readable multi-dimensional pilot study whose headline pass-rate comparison is undercut by self-authored tests and a single human subject; the static-analysis findings are worth a look, but the central claims about human vs. LLM capability are not established.","tokens_in":13745,"tokens_out":2262,"would_cite":false,"duration_ms":18701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper compares GPT-4-generated Python code with one human programmer across 72 tasks, finding GPT-4 passes more tests but produces more complex code and struggles on domain-heavy problems.","keywords":["large language models","GPT-4","Python code generation","code quality","code security","cyclomatic complexity","functional correctness","human versus LLM comparison"],"falsifier":"Run the same 72 tasks with a panel of programmers of varied experience levels; if the median human pass rate reaches or exceeds GPT-4's 87.3%, the paper's central correctness claim collapses.","tokens_in":12780,"feed_emoji":"🤖","tokens_out":7303,"duration_ms":59080,"temperature":0.7,"pith_summary":"This paper asks whether a leading large language model can write production-quality Python code the way a human programmer does. It compares GPT-4's output against one human programmer's solutions to 72 software engineering tasks, scoring both with the same static-analysis tools and the same test cases. The headline result is mixed: GPT-4 passes far more test cases (87.3% versus 54.9%) and writes more complex code, while the human's code adheres somewhat better to Python coding standards and handles domain-heavy problems more reliably. The authors conclude that LLMs are useful for routine coding but still need human oversight for tasks requiring deep domain knowledge, creative problem-solving, and careful debugging. The study matters because it tests the common promise that LLMs can automate software development, rather than assuming it.","feed_headline":"GPT-4 passes 87.3% of coding tests; humans 54.9%","feed_subtitle":"But human code edges out GPT-4 on coding standards and on tasks needing deep domain knowledge.","key_machinery":"The carrying mechanism is a paired comparison: the same 72 Python tasks are solved by one human programmer and by GPT-4 under zero-shot prompting, then both solution sets are run through four instruments — Pylint (adherence to Python coding standards, scored -10 to 10), Radon (cyclomatic complexity, graded A to F with an average score), Bandit (static security analysis, flagging LOW, MEDIUM, or HIGH severity issues), and Pytest (the same hand-written test cases for each task). The identity that carries the argument is the per-task pairing: because each human and LLM solution is tested against identical test cases and identical static-analysis settings, any difference in pass rate, complexity, or vulnerability counts is attributed to the code generator rather than to task difficulty.","core_discovery":"The paper's central claim is that GPT-4 and a human programmer do not occupy the same quality profile: GPT-4's code is functionally more correct on standard tests (87.3% pass rate versus 54.9%), but it is systematically more complex (average cyclomatic complexity 5.0 versus 3.1) and its security flaws are more often high-severity (60% versus 45% of issues), while the human code scores slightly higher on Pylint adherence to Python standards, though not statistically significantly (p = 0.64). On tasks requiring in-depth domain knowledge, such as a quantum-optimisation Traveling Salesman problem, SIR epidemic simulation, and debugging an anagram checker, the human solutions passed far more tests than GPT-4's. The authors' conclusion is that LLM-generated code is not yet a substitute for human programming; it is a draft that should be reviewed, simplified, and secured before it is trusted.","pith_inferences":["The paper does not test a panel of programmers; a direct follow-up would be to replace the single human with a group of developers of varied seniority and see whether the 87.3%-versus-54.9% pass-rate gap shrinks or reverses.","The zero-shot prompting choice, which the paper deliberately adopted for fairness, may drive part of the complexity gap; iterative or few-shot prompting in real workflows could produce simpler, more maintainable LLM code.","The security findings imply that LLM-assisted development workflows should include automated security scanning as a mandatory gate rather than an optional check, since both sources of code contain vulnerabilities and LLM flaws are more severe.","The domain-heavy failures suggest a testable hypothesis: performance gaps will widen as tasks require more unstated real-world knowledge, and narrow when tasks are fully specified in the prompt."],"forward_implications":["In routine, well-specified coding tasks, GPT-4 can produce functionally correct Python code more often than the human programmer in this study, supporting the use of LLMs as coding assistants for such tasks.","Because GPT-4's code averages 5.0 versus 3.1 on Radon complexity, adopting LLM output without simplification is likely to increase long-term maintenance and testing effort.","Both human and LLM code contain security flaws; LLM code's flaws skew toward high severity, so security review of LLM output is necessary before deployment.","Tasks requiring deep domain knowledge, creative adaptation, or iterative debugging should remain human-led, with LLMs in a supporting role.","The non-significant Pylint gap (p = 0.64) means claims that humans write cleaner code than LLMs are not supported by this dataset; the visible difference is variation, not a reliable average advantage."],"supporting_citations":[{"why":"Supplies the original 72-task Python benchmark dataset that both human and GPT-4 solutions are drawn from.","marker":"[7]"},{"why":"Provides the code-health and wholesomeness framing that motivates the four quality dimensions and the cyclomatic-complexity rationale.","marker":"[23]"},{"why":"Establishes the static-analysis approach for studying code quality that the Pylint, Radon, and Bandit pipeline follows.","marker":"[25]"},{"why":"Underpins the Bandit security scan used to count and classify vulnerabilities in both code sets.","marker":"[6]"},{"why":"Supports using Radon's cyclomatic complexity and Pylint-style metrics as reliable code-quality measures.","marker":"[5]"},{"why":"Justifies the zero-shot prompting design that keeps human and LLM solutions comparably generated.","marker":"[10]"}],"fun_headline_variants":["GPT-4 passes more tests, but humans write cleaner, simpler code","GPT-4 code is more complex and has more severe security flaws","Humans still beat GPT-4 on tasks needing deep domain knowledge","GPT-4 vs humans: no decisive winner in code quality yet","LLMs draft code, but humans still needed for debugging and innovation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study's human side is a single fourth-year computing student, and if that student is not representative of human programmers, every human-versus-LLM difference in the paper is contingent.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 passes more tests, but humans write cleaner, simpler code","GPT-4 code is more complex and has more severe security flaws","Humans still beat GPT-4 on tasks needing deep domain knowledge","GPT-4 vs humans: no decisive winner in code quality yet","LLMs draft code, but humans still needed for debugging and innovation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2783,"prompt_tokens":1059,"completion_tokens":1724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":1632}},"tokens_in":675,"tokens_out":1724,"duration_ms":12479,"temperature":1.0,"reasoning_tokens":1632,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:08:04.572190+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 72 tasks with a panel of programmers of varied experience levels; if the median human pass rate reaches or exceeds GPT-4's 87.3%, the paper's central correctness claim collapses.","supporting_citations":[{"cited_title":"Feng et al","cited_arxiv_id":null,"evidence_quote":"Supplies the original 72-task Python benchmark dataset that both human and GPT-4 solutions are drawn from."},{"cited_title":"Ndukwe et al","cited_arxiv_id":null,"evidence_quote":"Provides the code-health and wholesomeness framing that motivates the four quality dimensions and the cyclomatic-complexity rationale."},{"cited_title":"Meldrum et al","cited_arxiv_id":null,"evidence_quote":"Establishes the static-analysis approach for studying code quality that the Pylint, Radon, and Bandit pipeline follows."},{"cited_title":"Ruohonen et al","cited_arxiv_id":null,"evidence_quote":"Underpins the Bandit security scan used to count and classify vulnerabilities in both code sets."},{"cited_title":"Jin et al., ”Software Code Quality Measurement: Implications from Metric Distributions,” in IEEE Transactions on Software Engineering, vol","cited_arxiv_id":null,"evidence_quote":"Supports using Radon's cyclomatic complexity and Pylint-style metrics as reliable code-quality measures."}],"review_version":1}