Pith. sign in

REVIEW 1 major objections 2 minor 23 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Developer reviews reveal production-readiness problems in LLM code that standard benchmarks miss.

2026-06-30 22:54 UTC pith:Q6JYRBA5

load-bearing objection The paper adds developer surveys to correctness benchmarks for LLM code but the survey reporting is too thin to back the production-readiness claims. the 1 major comments →

arxiv 2605.09059 v2 pith:Q6JYRBA5 submitted 2026-05-09 cs.SE

Evaluating LLM-Generated Code: A Benchmark and Developer Study

Joanna Szych , Anne Schwerk This is my paper
classification cs.SE
keywords LLM code generationcode evaluation methodologydeveloper studycode qualityproduction readinesscode review processbenchmark comparison
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a three-fold evaluation approach for code generated by large language models that goes beyond measuring correctness. It pairs an automated benchmark on a complex multi-level project with code quality checks and structured developer reviews. The authors apply this method to three models and find that developer input surfaces distinct issues around production readiness. These insights cannot be obtained from correctness metrics alone.

Core claim

Reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.

What carries the argument

A custom three-fold evaluation methodology that combines a dedicated correctness benchmark on a complex multi-level computer science project, code quality verification, and developer opinion surveys collected via structured code-review.

Load-bearing premise

The structured developer code-review process produces unbiased, generalizable insights into production readiness that automated metrics cannot capture.

What would settle it

Running the same developer review process on the generated code samples yields no additional findings beyond what the correctness benchmark already reports.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Model selection for real-world coding tasks should incorporate production-readiness signals from human reviewers.
  • Correctness benchmarks alone are insufficient for judging whether generated code is ready for deployment.
  • The three-fold method can be used to compare models on dimensions other than functional accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The methodology could be adapted to evaluate generated code in specific domains such as web services or embedded systems.
  • Future benchmarks might derive quantitative production-readiness scores from aggregated developer review data.
  • Repeating the study with additional models or different project complexities would test the stability of the new findings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper proposes a three-fold evaluation methodology for LLM-generated code that augments standard correctness benchmarks with code quality verification and a structured developer code-review survey. It applies the methodology to compare three models (GPT-4.1, DeepSeek-V3-0324, Claude Opus 4) on a complex multi-level CS project and concludes that developer reviews surface production-readiness findings unreachable by correctness-focused benchmarks alone.

Significance. If the survey component is executed with adequate controls, the work could usefully demonstrate limitations of existing code-generation benchmarks and motivate evaluation practices that incorporate human judgments on maintainability and deployability. The choice of a non-trivial multi-level project is a positive step beyond typical single-function benchmarks.

major comments (1)
  1. [Methodology section (survey subsection) and Results section] The central claim (abstract and results) that developer reviews produce new, generalizable production-readiness insights rests on the survey being unbiased and representative. The manuscript reports neither the number of reviewers, selection criteria, domain experience, blinding procedures, nor inter-rater agreement statistics. Without these details the findings cannot be assessed for bias or external validity.
minor comments (2)
  1. [Abstract] The abstract states the methodology is 'tree-fold' but the body consistently uses 'three-fold'; standardize terminology.
  2. [Evaluation setup] Model names appear as 'GPT-4.1' and 'Claude Opus 4'; confirm exact versions and cite the precise releases used.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address the single major comment below and will revise the paper to improve transparency around the survey component.

read point-by-point responses

  1. Referee: [Methodology section (survey subsection) and Results section] The central claim (abstract and results) that developer reviews produce new, generalizable production-readiness insights rests on the survey being unbiased and representative. The manuscript reports neither the number of reviewers, selection criteria, domain experience, blinding procedures, nor inter-rater agreement statistics. Without these details the findings cannot be assessed for bias or external validity.

    Authors: We agree that these details are required to evaluate bias and external validity. The manuscript as submitted does not report the number of reviewers, selection criteria, domain experience, blinding procedures, or inter-rater agreement. In the revised version we will add a dedicated 'Developer Survey Design' subsection in Methodology that supplies exactly these elements (participant count, recruitment criteria, experience thresholds, blinding protocol, and agreement metric such as percentage agreement or Cohen's kappa). We will also cross-reference the new details when discussing survey results. This change directly supports the central claim without altering the reported findings. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical methodology with independent components

full rationale

The paper describes a three-fold empirical evaluation (correctness benchmark on a multi-level CS project, code quality verification, and structured developer survey) without equations, fitted parameters, derivations, or self-citations that bear load on the central claim. The claim that developer reviews surface production-readiness findings rests on the described process itself rather than reducing to a definition or prior self-result by construction. No load-bearing step matches any enumerated circularity pattern.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The evaluation depends on the domain assumption that developer opinions obtained through structured reviews provide valid and novel information about production readiness beyond automated correctness and quality metrics.

axioms (1)
  • domain assumption Developer reviews through a structured code-review process yield insights into production readiness that are not obtainable from correctness benchmarks alone.
    This assumption underpins the claim that the methodology produces new findings.

pith-pipeline@v0.9.1-grok · 5707 in / 1094 out tokens · 22085 ms · 2026-06-30T22:54:02.052681+00:00 · methodology

0 comments
read the original abstract

Code generation is one of the tasks for which the use of Large Language Models is widely adopted and highly successful. Given this popularity, there are many benchmarks dedicated to code generation that can help select the best model. However, they primarily focus on measuring solution correctness, leaving other aspects, such as code quality and usability, behind. This paper aims to describe a custom tree-fold evaluation methodology for code generated by Large Language Models that bridges this gap. The methodology includes a dedicated correctness benchmark based on a complex multi-level computer science project, code quality verification, and a survey of developers' opinions on generated code samples gathered through a structured code-review process. The proposed methodology's usage and usefulness are demonstrated by evaluating and comparing three general-purpose Large Language Models: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. The results show that reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 7 internal anchors

  1. [1]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Su...

    work page arXiv 2023

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  4. [4]

    2025.The Temperature Parameter | DeepSeek API Docs

    DeepSeek. 2025.The Temperature Parameter | DeepSeek API Docs. https://api- docs.deepseek.com/quick_start/parameter_settings

    work page 2025

  5. [5]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machinery, New York...

    work page doi:10.1145/3597503.3639219 2024

  6. [6]

    Eitan Farchi, Shmulik Froimovich, Rami Katan, and Orna Raz. 2024. Auto- matic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks. arXiv:2410.21071 [cs.SE] https://arxiv.org/abs/2410.21071

    work page arXiv 2024

  7. [7]

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping Language to Code in Programmatic Context. arXiv:1808.09588 [cs.CL] https://arxiv.org/abs/1808.09588

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  8. [8]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real- World GitHub Issues? arXiv:2310.06770 [cs.CL] https://arxiv.org/abs/2310.06770

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv. org/abs/2305.01210

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. arXiv:2306.03091 [cs.CL] https://arxiv.org/abs/2306.03091

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [11]

    Bradley McDanel and Ed Novak. 2025. Designing LLM-Resistant Programming Assignments: Insights and Strategies for CS Educators. InProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1(Pittsburgh, PA, USA)(SIGCSETS 2025). Association for Computing Machinery, New York, NY, USA, 756–762. doi:10.1145/3641554.3701872

    work page doi:10.1145/3641554.3701872 2025

  12. [12]

    Tanha Miah and Hong Zhu. 2024. User Centric Evaluation of Code Genera- tion Tools (Invited Paper). In2024 IEEE International Conference on Artificial Intelligence Testing (AITest). 109–119. doi:10.1109/AITest62860.2024.00022

    work page doi:10.1109/aitest62860.2024.00022 2024

  13. [13]

    2025.Homepage | SonarQube Cloud | Sonar Documentation

    SonarSource. 2025.Homepage | SonarQube Cloud | Sonar Documentation. https: //docs.sonarsource.com/sonarqube-cloud

    work page 2025

  14. [14]

    2025.Software qualities | SonarQube Cloud Documentation

    SonarSource. 2025.Software qualities | SonarQube Cloud Documentation. https: //docs.sonarsource.com/sonarqube-cloud/digging-deeper/software-qualities/

    work page 2025

  15. [15]

    2026.Evaluating LLM-Generated Code: Benchmarking on complex assignment

    Joanna Szych. 2026.Evaluating LLM-Generated Code: Benchmarking on complex assignment. https://github.com/AsiaSzych/Tree_of_Life/

    work page 2026

  16. [16]

    2026.Evaluating LLM-Generated Code: Developer Study

    Joanna Szych. 2026.Evaluating LLM-Generated Code: Developer Study. doi:10. 5281/zenodo.18806359

    work page 2026

  17. [17]

    2025.Building The Tree of Life from Scratch

    Christopher Tralie. 2025.Building The Tree of Life from Scratch. http://nifty. stanford.edu/2025/tralie-phylogenetic-trees/

    work page 2025

  18. [18]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can LLMs Replace Human Evaluators? An Empirical Study of LLM- as-a-Judge in Software Engineering.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA086 (June 2025), 23 pages. doi:10.1145/3728963

    work page doi:10.1145/3728963 2025

  19. [19]

    Wei Wang, Huilong Ning, Gaowei Zhang, Libo Liu, and Yi Wang. 2024. Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM- Supported SE Tasks.Proc. ACM Softw. Eng.1, FSE, Article 32 (July 2024), 23 pages. doi:10.1145/3643758

    work page doi:10.1145/3643758 2024

  20. [20]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Prag- matic Code Generation with Generative Pre-trained Models. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machin...

    work page doi:10.1145/3597503.3623316 2024

  21. [21]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association...

    work page doi:10.18653/v1/2023.emnlp-main.151 2023

  22. [22]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://arxiv.org/abs/2306.05685

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2024. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Bench- marking on HumanEval-X. arXiv:2303.17568 [cs.LG] https://arxiv.org/abs/2303. 17568

    work page arXiv 2024