Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Comparing Human and LLM Generated Code: The Jury is Still Out!

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper compares GPT-4-generated Python code with one human programmer across 72 tasks, finding GPT-4 passes more tests but produces more complex code and struggles on domain-heavy problems.

desk verdict A readable multi-dimensional pilot study whose headline pass-rate comparison is undercut by self-authored tests and a single human subject; the static-analysis findings are worth a look, but the central claims about human vs. LLM capability are not established. read the letter →

arxiv 2501.16857 v1 pith:PN3ETTHG submitted 2025-01-28 cs.SE

Sherlock A. Licorish , Ansh Bajpai , Chetan Arora , Fanyu Wang , Kla Tantithamthavorn This is my paper · ORCID
classification cs.SE
keywords largelanguagemodelsGPT-4PythoncodegenerationqualitysecuritycyclomaticcomplexityfunctionalcorrectnesshumanversusLLMcomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a leading large language model can write production-quality Python code the way a human programmer does. It compares GPT-4's output against one human programmer's solutions to 72 software engineering tasks, scoring both with the same static-analysis tools and the same test cases. The headline result is mixed: GPT-4 passes far more test cases (87.3% versus 54.9%) and writes more complex code, while the human's code adheres somewhat better to Python coding standards and handles domain-heavy problems more reliably. The authors conclude that LLMs are useful for routine coding but still need human oversight for tasks requiring deep domain knowledge, creative problem-solving, and careful debugging. The study matters because it tests the common promise that LLMs can automate software development, rather than assuming it.

What carries the argument

The carrying mechanism is a paired comparison: the same 72 Python tasks are solved by one human programmer and by GPT-4 under zero-shot prompting, then both solution sets are run through four instruments — Pylint (adherence to Python coding standards, scored -10 to 10), Radon (cyclomatic complexity, graded A to F with an average score), Bandit (static security analysis, flagging LOW, MEDIUM, or HIGH severity issues), and Pytest (the same hand-written test cases for each task). The identity that carries the argument is the per-task pairing: because each human and LLM solution is tested against identical test cases and identical static-analysis settings, any difference in pass rate, complexity, or vulnerability counts is attributed to the code generator rather than to task difficulty.

What would settle it

Run the same 72 tasks with a panel of programmers of varied experience levels; if the median human pass rate reaches or exceeds GPT-4's 87.3%, the paper's central correctness claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that GPT-4 and a human programmer do not occupy the same quality profile: GPT-4's code is functionally more correct on standard tests (87.3% pass rate versus 54.9%), but it is systematically more complex (average cyclomatic complexity 5.0 versus 3.1) and its security flaws are more often high-severity (60% versus 45% of issues), while the human code scores slightly higher on Pylint adherence to Python standards, though not statistically significantly (p = 0.64). On tasks requiring in-depth domain knowledge, such as a quantum-optimisation Traveling Salesman problem, SIR epidemic simulation, and debugging an anagram checker, the human solutions passed far more tests than GPT-4's. The authors' conclusion is that LLM-generated code is not yet a substitute for human programming; it is a draft that should be reviewed, simplified, and secured before it is trusted.

Load-bearing premise

The study's human side is a single fourth-year computing student, and if that student is not representative of human programmers, every human-versus-LLM difference in the paper is contingent.

Editorial extensions

If this is right

  • In routine, well-specified coding tasks, GPT-4 can produce functionally correct Python code more often than the human programmer in this study, supporting the use of LLMs as coding assistants for such tasks.
  • Because GPT-4's code averages 5.0 versus 3.1 on Radon complexity, adopting LLM output without simplification is likely to increase long-term maintenance and testing effort.
  • Both human and LLM code contain security flaws; LLM code's flaws skew toward high severity, so security review of LLM output is necessary before deployment.
  • Tasks requiring deep domain knowledge, creative adaptation, or iterative debugging should remain human-led, with LLMs in a supporting role.
  • The non-significant Pylint gap (p = 0.64) means claims that humans write cleaner code than LLMs are not supported by this dataset; the visible difference is variation, not a reliable average advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test a panel of programmers; a direct follow-up would be to replace the single human with a group of developers of varied seniority and see whether the 87.3%-versus-54.9% pass-rate gap shrinks or reverses.
  • The zero-shot prompting choice, which the paper deliberately adopted for fairness, may drive part of the complexity gap; iterative or few-shot prompting in real workflows could produce simpler, more maintainable LLM code.
  • The security findings imply that LLM-assisted development workflows should include automated security scanning as a mandatory gate rather than an optional check, since both sources of code contain vulnerabilities and LLM flaws are more severe.
  • The domain-heavy failures suggest a testable hypothesis: performance gaps will widen as tasks require more unstated real-world knowledge, and narrow when tasks are fully specified in the prompt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a comparison of code written by one fourth-year computing student and code generated by GPT-4 (April 2024 version) across 72 Python software engineering tasks. Four quality dimensions are evaluated using Pylint, Bandit, Radon, and Pytest: coding standards compliance, security vulnerabilities, code complexity, and functional correctness. The main reported findings are that human code scores somewhat higher on Pylint compliance, that LLM and human code both contain security flaws but LLM code has more high-severity outliers, that GPT-4 code has higher average cyclomatic complexity (5.0 vs. 3.1), and that GPT-4 passes more test cases overall (87.3% vs. 54.9%). The paper further claims, based on a handful of selected tasks, that humans outperform GPT-4 on complex, domain-knowledge-heavy problems, and concludes that LLMs should complement rather than replace human programmers.

Significance. If the results were methodologically airtight, this would be a valuable multi-dimensional, quantitative comparison of human and LLM code generation, complementing prior work that focuses on a single quality dimension or uses subjective evaluations. The study uses standard, well-known static analysis tools and reports statistical tests, and it is commendable that the authors explicitly acknowledge the single-programmer limitation in Section VI. However, the evidence as presented does not establish the paper's central comparative claims because of threats to internal validity (test provenance), an n=1 human sample, unsupported conclusions about XSS, and reliance on a few hand-picked tasks for the 'complex problem-solving' narrative. The paper's strengths are its breadth of metrics and its candid acknowledgment of some limitations, but these do not compensate for the auditability and validity problems in the functional-correctness comparison.

major comments (4)
  1. [III.A and IV.D] This is a complete sentence.
  2. [IV.D.2 and VII] This is a complete sentence.
  3. [VII (Conclusion)] This is a complete sentence.
  4. [III.A and VI] This is a complete sentence.
minor comments (5)
  1. [III.B (Radon)] This is a complete sentence.
  2. [III.A] This is a complete sentence.
  3. [IV.B] This is a complete sentence.
  4. [IV.D.1] This is a complete sentence.
  5. [References] This is a complete sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; pass-rate and static-analysis comparisons are direct measurements without fitted parameters, derived equations, or load-bearing self-citations.

full rationale

The paper contains no derivation chain whose conclusion is equivalent to an input. The reported outcomes are direct measurements: Pylint, Radon, Bandit, and Pytest pass rates are observed on 144 code files (72 tasks x human/LLM), and the headline numbers (87.3% vs 54.9%) are empirical aggregates, not quantities fitted to or defined by each other. The study's self-citations ([24], [29], [32], [40]) are background references to prior work by the authors and are not used as evidence for the current comparison, so no load-bearing self-citation exists. The strongest validity concern is that the test files were authored in the study ('Test files were created for the human- and LLM-generated code', Section III.A), so if the tests implicitly encode the human programmer's interface choices, the pass-rate comparison could be biased. That is an experimental-design threat to the benchmark's neutrality, but it is not circularity under the definitions used here: the paper does not define functional correctness in terms of the human solution, nor does it fit a parameter to one side's output and then 'predict' the other. The single fourth-year student also limits generalizability, but a small or unrepresentative sample is not a circular step. No circular step can be exhibited from the text, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central comparison rests on five untested modeling choices: the single human sample, self-authored tests, tool-based quality proxies, task representativeness, and zero-shot prompting as a fair baseline. No free parameters or invented entities are introduced.

assumptions (5)
  • ad hoc to paper One fourth-year computing student's code represents human-generated code.
    Section III.A states a single student developed all 72 human solutions; all human-versus-LLM comparisons inherit this sample assumption.
  • domain assumption Self-authored test cases are a valid and unbiased measure of functional correctness for both code sources.
    Section III.A says tests were created for all 72 tasks by the study team; no external benchmark or independent test oracle is used.
  • domain assumption Pylint, Radon, and Bandit static-analysis outputs are meaningful proxies for coding standards, complexity, and security.
    Section III.B adopts these tools as measures; their outputs are treated as ground truth for the quality dimensions studied.
  • domain assumption The 72 tasks are representative of software engineering work.
    Section III.A sources the tasks from a previous study without sampling justification or task-difficulty calibration.
  • domain assumption Zero-shot prompting is a fair comparison protocol because human solutions were not iterated.
    Section III.A argues iterative prompting would bias the comparison; this fairness assumption is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparing Human and LLM Generated Code: The Jury is Still Out!." pith.science (2026). https://pith.science/paper/PN3ETTHG

@misc{pith2026250116857,
  author       = {Pith},
  title        = {Pith review of: Comparing Human and LLM Generated Code: The Jury is Still Out!},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PN3ETTHG}},
  note         = {Machine review of arXiv:2501.16857}
}
read the original abstract

Much is promised in relation to AI-supported software development. However, there has been limited evaluation effort in the research domain aimed at validating the true utility of such techniques, especially when compared to human coding outputs. We bridge this gap, where a benchmark dataset comprising 72 distinct software engineering tasks is used to compare the effectiveness of large language models (LLMs) and human programmers in producing Python software code. GPT-4 is used as a representative LLM, where for the code generated by humans and this LLM, we evaluate code quality and adherence to Python coding standards, code security and vulnerabilities, code complexity and functional correctness. We use various static analysis benchmarks, including Pylint, Radon, Bandit and test cases. Among the notable outcomes, results show that human-generated code recorded higher ratings for adhering to coding standards than GPT-4. We observe security flaws in code generated by both humans and GPT-4, however, code generated by humans shows a greater variety of problems, but GPT-4 code included more severe outliers. Our results show that although GPT-4 is capable of producing coding solutions, it frequently produces more complex code that may need more reworking to ensure maintainability. On the contrary however, our outcomes show that a higher number of test cases passed for code generated by GPT-4 across a range of tasks than code that was generated by humans. That said, GPT-4 frequently struggles with complex problem-solving that involve in-depth domain knowledge. This study highlights the potential utility of LLMs for supporting software development, however, tasks requiring comprehensive, innovative or unconventional solutions, and careful debugging and error correction seem to be better developed by human programmers. We plot an agenda for the software engineering community.

Figures

Figures reproduced from arXiv: 2501.16857 by the authors.

Figure 1
Figure 1. Comparison of Human vs LLM Pylint Scores in box plots [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Human vs LLM Bandit Scores in a scatter plot [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Human vs LLM Bandit Scores in box plots [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Comparison of Human vs LLM Test Scores in a scatter plot [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Comparison of Human vs LLM Radon Scores in a box plots [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Comparison of Human vs LLM Test Scores in box plots [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    Using a 61-feature readability model, LLM code matches or slightly exceeds human code in readability score, shows distinct issue patterns, and prompt design has limited influence.

  2. Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis

    cs.SE 2025-07 conditional novelty 6.0 of 10

    In a benchmark of LLM-generated C code for graph analysis, Claude Sonnet 4 Extended produced the most correct, fastest, and most memory-efficient implementations, beating human baselines on triangle counting.

  3. Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

    cs.SE 2025-06 accept novelty 6.0 of 10

    An empirical study of five LLM-based program repair agents shows they excel on simple fixes but overfit tests and struggle with test generation, regression selection, and advanced tooling.

  4. An Evaluation of Large Language Models for Detection of Malicious Python Packages

    cs.CR 2026-02 conditional novelty 5.0 of 10

    LLMs detect malicious PyPI packages reasonably at package level but identify specific malicious code indicators poorly, a 'granularity gap' that limits them to triage.

Reference graph

Works this paper leans on

40 extracted references · 26 canonical work pages · cited by 4 Pith papers

  1. [1]

    Devlin et al., ”BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of NAACL-HLT, 2019

    J. Devlin et al., ”BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of NAACL-HLT, 2019

    work page 2019

  2. [2]

    Liawatimena et al., ”Django Web Framework Software Metrics Mea- surement Using Radon and Pylint,” in Journal of Physics: Conference Series, 2021

    S. Liawatimena et al., ”Django Web Framework Software Metrics Mea- surement Using Radon and Pylint,” in Journal of Physics: Conference Series, 2021

    work page 2021

  3. [4]

    Sharma and D

    T. Sharma and D. Spinellis, ”Do We Need Improved Code Quality Metrics?” in Proc. of the 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2018, pp. 1-12

    work page 2018

  4. [5]

    Jin et al., ”Software Code Quality Measurement: Implications from Metric Distributions,” in IEEE Transactions on Software Engineering, vol

    S. Jin et al., ”Software Code Quality Measurement: Implications from Metric Distributions,” in IEEE Transactions on Software Engineering, vol. 39, no. 4, pp. 597-614, 2023

    work page 2023

  5. [6]

    Ruohonen et al

    J. Ruohonen et al. (2020). ”A Large-Scale Security-Oriented Static Analysis of Python Packages in PyPI,” in Journal of Information Security and Applications, vol. 50, pp. 102-115

    work page 2020

  6. [7]

    Feng et al

    Y . Feng et al. (2023). ”Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data,” in Proceedings of the IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), Torino, Italy, pp. 876-885

    work page 2023

  7. [8]

    I. O. Gallegos et al., ”Bias and Fairness in Large Language Models: A Survey,” arXiv preprint arXiv:2309.00770, 2023

    arXiv 2023

  8. [9]

    Yan et al., ”Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review,” 2023

    L. Yan et al., ”Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review,” 2023

    work page 2023

Show all 40 references
  1. [10]

    White et al., ”A Prompt Pattern Catalog to Enhance Prompt Engi- neering with ChatGPT,” arXiv preprint arXiv:2302.11382, 2023

    J. White et al., ”A Prompt Pattern Catalog to Enhance Prompt Engi- neering with ChatGPT,” arXiv preprint arXiv:2302.11382, 2023

    2023 arXiv

  2. [11]

    Hochmair et al

    A. Hochmair et al. (2023). ”Correctness Comparison of ChatGPT-4, Bard, Claude-2, and Copilot for Spatial Tasks,” in Transactions in GIS, Available: https://doi.org/10.48550/arXiv.2401.02404

    2023 doi arXiv

  3. [12]

    Chen et al., ”Capabilities of Large Language Models in Code Documentation,” 2024

    H. Chen et al., ”Capabilities of Large Language Models in Code Documentation,” 2024

    2024

  4. [13]

    Zhang et al., ”Ethical Considerations and Policy Implications for Large Language Models: Guiding Responsible Development and De- ployment,” arXiv preprint arXiv:2308.02678, 2023

    Z. Zhang et al., ”Ethical Considerations and Policy Implications for Large Language Models: Guiding Responsible Development and De- ployment,” arXiv preprint arXiv:2308.02678, 2023

    2023 arXiv

  5. [14]

    Das et al., ”Under the Surface: Tracking Artifactuality in LLM- Generated Data,” arXiv preprint arXiv:2401.14698v2, 2024

    D. Das et al., ”Under the Surface: Tracking Artifactuality in LLM- Generated Data,” arXiv preprint arXiv:2401.14698v2, 2024

    2024 arXiv

  6. [15]

    Radford et al., ”Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems, 2020

    A. Radford et al., ”Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems, 2020

    2020

  7. [16]

    Yao et al

    ‘S. Yao et al. ”Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” p. arXiv:2305.10601doi: 10.48550/arXiv.2305.10601

    doi arXiv

  8. [17]

    Dillmann et al

    M. Dillmann et al. (2024). Evaluation of large language models for assessing code maintainability. arXiv preprint arXiv:2401.12714

    2024 arXiv

  9. [18]

    Fang et al

    C. Fang et al. (2023). Large language models for code analysis: Do llms really do their job?. arXiv preprint arXiv:2310.12357

    2023 arXiv

  10. [19]

    Dvivedi et al

    S. Dvivedi et al. (2023). A Comparative Analysis of Large Lan- guage Models for Code Documentation Generation. arXiv preprint arXiv:2312.10349

    2023 arXiv

  11. [20]

    Van Rossum et al

    G. Van Rossum et al. (2001). PEP 8: Style guide for Python code. Retrieved from https://www.pep8.org

    2001

  12. [21]

    Du, et al

    X. Du, et al. (2024). Evaluating Large Language Models in Class-Level Code Generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). ACM, Article 81, 1–13. https://doi.org/10.1145/3597503.3639219

    2024

  13. [22]

    Chang et al

    Y . Chang et al. (2024). A Survey on Evaluation of Large Lan- guage Models. ACM Trans. Intell. Syst. Technol. 15, 3, Article 39. https://doi.org/10.1145/3641289

    2024 doi

  14. [23]

    Ndukwe et al

    I. Ndukwe et al. (2023). How have views on software quality differed over time? Research and practice viewpoints. Journal of Systems and Software, 195, 111524. 10.1016/j.jss.2022.111524

    2023

  15. [24]

    S. A. Licorish and T. Nishatharan, ”Contextual Profiling of Stack Overflow Java Code Security Vulnerabilities Initial Insights from a Pilot Study,” 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), Hainan, China, 2021, pp....

    2021

  16. [25]

    Meldrum et al

    S. Meldrum et al. (2020). Understanding stack overflow code quality: A recommendation of caution. Science of Computer Programming, 199, 102516. 10.1016/j.scico.2020.102516

    2020

  17. [26]

    Achiam et al

    J. Achiam et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  18. [27]

    Chen et al

    M. Chen et al. (2021). Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  19. [28]

    Zolduoarrati et al

    E. Zolduoarrati et al. (2022). Impact of individualism and collectivism cultural profiles on the behaviour of software developers: A study of stack overflow. Journal of Systems and Software, 192, 111427. 10.1016/j.jss.2022.111427

    2022

  20. [29]

    Zolduoarrati and S

    E. Zolduoarrati and S. Licorish. (2021). On the Value of En- couraging Gender Tolerance and Inclusiveness in Software Engi- neering Communities. Information and Software Technology, 139. 10.1016/j.infsof.2021.106667

    2021

  21. [30]

    Barke et al

    S. Barke et al. (2023). Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang. 7, OOP- SLA1, Article 78, https://doi.org/10.1145/3586030

    2023 doi

  22. [31]

    Tehrani et al

    B. Tehrani et al. (2024). Evaluating Human-AI Partnership for LLM- based Code Migration. In Extended Abstracts of the 2024 CHI Confer- ence on Human Factors in Computing Systems (CHI EA ’24). ACM, Article 133, 1–8. https://doi.org/10.1145/3613905.3650896

    2024

  23. [32]

    S. A. Licorish and S. MacDonell. (2013). Adopting softer approaches in the study of repository data: A comparative analysis, in Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE 2013) (Porto de Galinhas, Brazil. ACM, 24...

    2013

  24. [33]

    S., Wang, Y ., and Zhang, L

    Liu, J., Xia, C. S., Wang, Y ., and Zhang, L. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and ...

    2023

  25. [34]

    Wang, W., Ning, H., Zhang, G., Liu, L., and Wang, Y . 2024. Rocks coding, not development: A human-centric, experimental evaluation of LLM-supported SE tasks. Proc. ACM Joint Euro- pean Software Engineering Conference and Symposium on the Foundations of Software Engineering (E...

    2024 doi

  26. [35]

    Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., and Ma, S. 2020. CodeBLEU: a method for au- tomatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297

    2020 arXiv

  27. [36]

    Nguyen, N., and Nadi, S. 2022. An empirical evaluation of GitHub Copilot’s code suggestions. Proc. 19th Int’l Conf. on Mining Soft- ware Repositories (MSR 2022). ACM, Pittsburgh, PA, USA, 1–5. DOI:https://doi.org/10.1145/3524842.3528470

    2022

  28. [37]

    N., Denny, P., Becker, B

    Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., and Santos, E. A

  29. [38]

    A., Rochan, M., and Roy, C

    Awal, M. A., Rochan, M., and Roy, C. K. 2024. Comparing robust- ness against adversarial attacks in code generation: LLM-generated vs. human-written. arXiv preprint arXiv:2411.10565

    2024 arXiv

  30. [39]

    S. Ahmed. (2024). LLM Prompt Engineering for Beginners: What It Is and How to Get Started. The Deep Hub. https://medium.com/thedeephub/llm-prompt-engineering-for-beginners- what-it-is-and-how-to-get-started-0c1b483d5d4f

    2024

  31. [40]

    I., Licorish, S., Wang, F., & Treude, C

    Arora, C., Sayeed, A. I., Licorish, S., Wang, F., & Treude, C. (2024). Optimizing Large Language Model Hyperparameters for Code Genera- tion. arXiv preprint arXiv:2408.10577

    2024 arXiv

  32. [2023]

    ACM Transactions on Computer-Human Interaction, 31, 1, Article 1 (January 2024), 35 pages

    It’s weird that it knows what I want: Usability and interac- tions with Copilot for novice programmers. ACM Transactions on Computer-Human Interaction, 31, 1, Article 1 (January 2024), 35 pages. DOI:https://doi.org/10.1145/3579590

    2024 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.