Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Current LLMs produce code whose measured readability is at least comparable to, and often higher than, human-written code, but the failures are characteristically AI-shaped, and prompt design is a real yet bounded lever.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:08 UTC pith:LIYRG6S4

load-bearing objection Substantial empirical study of LLM code readability whose headline parity claim rests on an unvalidated proxy model; the qualitative issue patterns and prompt experiments are worth engaging but need revision. the 4 major comments →

arxiv 2605.13280 v2 pith:LIYRG6S4 submitted 2026-05-13 cs.SE cs.AI

Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code

Hengzhi Ye , Fengyuan Ran , Weiwei Xu , Minghui Zhou This is my paper
classification cs.SE cs.AI
keywords code readabilityLLM-generated codeprompt engineeringreadability assessment modelreadability issue patternsAI-assisted programmingempirical software engineeringtechnical debt
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to determine whether LLM-generated code is as readable as human-written code and whether prompt design can be used to improve it. The authors build a multi-feature readability model, generate code from five LLMs across nearly six thousand real-world and competitive programming tasks, and compare against human implementations. Their central claim is that LLM code is statistically at least as readable as human code, but it fails in distinct, AI-specific patterns—over-engineered logic, redundant comments, and unexplained APIs. They also find that function signatures, constraints, and style descriptions matter most in prompts, but that prompt engineering alone is a weak lever. If correct, the work validates AI-assisted coding from a maintainability perspective while exposing a latent readability debt that future tooling must address.

Core claim

The paper claims that, across nearly six thousand matched coding tasks, code generated by five current LLMs receives readability scores statistically higher than human-written code (p<0.001, moderate effect size), and is therefore at least comparable in practical terms. That aggregate parity masks a divergence: poorly rated human code is dominated by missing comments and inconsistent style, while poorly rated LLM code is dominated by excessive complexity and redundant comments, plus three newly identified AI-specific patterns—unknown API usage, redundant variables, and overblanking. On prompts, function signatures, constraints, and style descriptions are the most influential dimensions, yet

What carries the argument

The central instrument is an integrated readability model: a 61-dimensional feature vector combining textual/lexical, structural/formatting, information-theoretic (program), and visual/geometric metrics, reduced by sequential forward selection and scored by an L2-regularized logistic regression trained on 360 human-annotated snippets. The same model supplies every readability score in the paper. A secondary mechanism is the prompt vector—a seven-bit encoding of which design dimensions a prompt includes—which lets the authors isolate the effect of each prompt dimension through random-forest importance, permutation tests, and ablations.

Load-bearing premise

The entire argument rests on treating the readability model's score—trained on 360 snippets from a different distribution and never checked against human readers for LLM-generated or open-source code—as human-perceived readability in the target setting.

What would settle it

Recruit a pre-registered panel of experienced developers to rate a random sample of matched LLM/human code pairs (blind to authorship) and compare their rankings to the model's scores. If human raters do not place LLM code at or above human code, or if the model's scores fail to track ratings on individual pairs, the central parity claim fails. A simpler check: strip comments and re-run the model to see whether it rewards surface regularity that human readers may not.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LLM-generated code can be adopted into workflows without an overall readability penalty; the risk is concentrated in specific, recognizable failure modes rather than average quality.
  • Existing code-review and static-analysis tools, tuned to human-typical issues, need new detectors for excessive complexity, redundant comments, unknown APIs, redundant variables, and overblanking.
  • Adding function signatures, constraints, and style descriptions to prompts is a cheap, partially effective readability fix, but the low explained variance means prompting alone cannot reliably engineer readable code.
  • Readability should join functional correctness as a first-class evaluation and optimization target for code LLMs.
  • Human-written code is more consistent; teams relying on generated code may need style normalization to reduce readability variance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the three new AI-specific patterns are detector-ready: unknown API usage can be flagged by checking imports and comments, redundant variables by dataflow analysis, and overblanking by layout rules. Building these detectors is a direct, testable next step.
  • A human-rating validation study on the same matched pairs would be the natural stress test; if readers do not perceive the parity the model reports, the 'readable as human code' conclusion is an artifact of the proxy.
  • The commission-versus-omission asymmetry—LLMs add clutter, humans leave gaps—may extend beyond readability to other non-functional qualities like security and performance, suggesting AI output often needs subtractive refactoring.
  • Because the prompt study is restricted to single-turn interactions, iterative or conversational prompting—where a developer asks for simplifications—could have a larger effect; testing readability under multi-turn refinement is an open extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether LLM-generated Python code is as readable as human-written code, what readability issues each exhibits, and how prompt design affects generated-code readability. The authors build a 61-feature readability model from four established feature families (TF, BWF, PF, DF), train it on the Dorn dataset, and apply it to 5,869 WoC/LeetCode prompt-code pairs (the abstract says 2,735). They report that LLM-generated code is 'at least comparable' and, in the main analysis, statistically higher in readability than human code (Wilcoxon p<0.001, r=0.398). A thematic analysis of 500 pairs identifies distinct issue patterns, with LLM code more prone to Excessive Complexity, Redundant Comments, Unknown APIs, Redundant Variables, and Overblanking. A controlled prompt experiment with 5,248 prompts across seven dimensions finds function signature, constraints, and style description to be the most influential prompt dimensions, while overall prompt impact is limited (RF R²<0.3).

Significance. If the central quantitative result holds, the paper would provide a useful large-scale baseline for an important and understudied non-functional property of LLM-generated code. The mixed-method design is a genuine strength: the RQ2 thematic analysis uses human annotation with substantial inter-rater agreement (Cohen's Kappa 0.87/0.81), and the prompt-variant experiment is carefully controlled. The scale of the benchmark and the proposed readability issue taxonomy are valuable contributions independent of the exact numerical scores. However, the RQ1 and RQ3 conclusions rest entirely on the validity of the automated readability model, and the manuscript does not validate that model on the target distribution or against human readers of the target code. This is a load-bearing construct-validity gap, not a circularity problem: the model was trained on an external dataset, but its use to compare LLM and human code on a different distribution requires domain calibration that is not provided.

major comments (4)
  1. [§4.3, §5, Appendix A.2] The central RQ1 claim—LLM code is 'statistically higher' in readability (p<0.001, r=0.398 in §5)—is computed from a logistic-regression readability model trained on the 360-snippet Dorn dataset (§4.3) and applied to 5,869 WoC/LeetCode snippets without any validation on the target distribution or against human judgments on that distribution. Appendix A.2 (Eq. A.31) shows the score is a weighted linear combination of features whose weights are learned on Dorn. The features heavily reward regular formatting, alignment, and low layout entropy, which are properties LLM output may exhibit for stylistic rather than human-comprehension reasons. A concrete fix is to collect human readability ratings on a held-out sample of WoC/LeetCode pairs and report correlation/agreement between the model and human judgments, or otherwise demonstrate that the model's preferences transfer. Without this, the qua
  2. [Appendix A.2, Table 4, Figure 2] There is an internal inconsistency in the definition of the readability score. Eq. A.35 defines r(s) as a logistic-sigmoid probability in [0,1], and the text says this 'is used as the final readability prediction.' But Table 4 reports average scores such as 2.15, 1.29, and 0.77, and Figure 2 shows negative scores and values outside [0,1]. If the actual score used in §5 is the raw linear combination before the sigmoid, the manuscript needs to say so explicitly and consistently; if it is the sigmoid probability, the tables and figures are incorrect. This matters because all RQ1/RQ3 statistics are computed on this score, and the reader cannot infer which quantity was tested.
  3. [§3.4, Table 5, Table 6] The issue-pattern results in RQ2 are not auditable as rendered. Table 6 is garbled: the row values for Human_bad, LLM_bad, and Total are run together (e.g., 'Human_bad60 35841 24 22 7 6 3'), making it impossible to verify which numbers correspond to DC, RC, IS, EC, PS, PN, MV, CD, and HC. Table 5 also has an unexplained arithmetic issue: the per-column totals match (72+60=132 etc.), but the row sums are 214 for Human_bad and 169 for LLM_bad, totaling 383, not the 500 sampled pairs described in §3.4. The manuscript should clarify the unit of analysis, how ties/exclusions were handled, and provide the full frequency table for each issue pattern by group.
  4. [§3.5, Table 7] The statistical thresholds for RQ3 are inconsistent. Section 3.5 states that t-tests and permutation tests use p<0.1 to determine relevance, but Table 7's footnote says 'Bold values indicate statistical significance (p<0.05).' The reported p-values happen to be far below both thresholds for the significant dimensions, but the protocol should be stated consistently and the threshold choice justified, especially because RQ3's conclusion about which dimensions are 'most influential' is based on this test.
minor comments (6)
  1. [Abstract vs. §3.1/§3.2.1] The abstract reports '2,735 scenarios' while the full text and §3.1/§3.2.1 consistently report 5,869 effective samples and 3,000 WoC pairs plus 2,869 LeetCode pairs. This is a central factual inconsistency that should be corrected.
  2. [Table 3] The All-features row reads '2577.5% 83.8%'—likely '25 77.5%' with a missing separator. This should be formatted properly.
  3. [§4.3] Typo: 'Ddorn dataset' should be 'Dorn dataset'.
  4. [Table 4] The model is referred to as 'Llama 3.1' in §3.3 and elsewhere, but Table 4 lists 'Llama 3'. Please make consistent.
  5. [§7] The random-forest R² is reported only as '<0.3'. Reporting the exact value and confidence interval would clarify how limited the prompt effects actually are, especially since the R² is used to support the 'overall impact remains limited' claim.
  6. [§3.5] The description of permutation testing says the binary labels of the tested dimension are shuffled, but permutation importance typically shuffles the feature values. Clarify whether the shuffling was applied to the feature column or to the target; the current wording is ambiguous.

Circularity Check

0 steps flagged

No circular derivation: the readability proxy is independently trained on Dorn and applied out-of-sample; RQ2 uses human annotation; RQ3 uses controlled prompt experiments.

full rationale

The central RQ1 comparison is not circular: the readability model is trained on the Dorn dataset (Section 4.3: 'we conducted experiments on the Ddorn dataset... 360 code snippets') and then applied to 5,869 WoC/LeetCode pairs (Section 5). No parameter of that model is fitted to the LLM-vs-human difference it is used to estimate; the comparison is an out-of-sample application of an independently labeled proxy. The proxy's possible mismatch with the target distribution is an external-validity/construct-validity limitation, not a circular reduction—the authors themselves flag it in Section 8.2.1 ('Although our model integrates dimensions from validated prior work, it may not be exhaustive, as individual developer preferences vary'). RQ2 issue patterns come from independent human annotation with reported Cohen's kappa (Section 3.4), not from the model's fitted weights; using the same four families as annotation dimensions is a categorization choice, not a derivation from the outcome. RQ3 is a controlled prompt-variant experiment (Set B, 5,248 prompts) with random forest importance, t-tests, permutation tests, and ablation; no parameter is fitted to the claimed conclusion. The only self-citation ([99], Licoeval) is a related-work reference about license compliance and is not load-bearing. Therefore no specific equation or fitted value reduces to the target claim, and no circular step can be exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. 'Readability debt' and 'latent technical debt' are interpretive metaphors for observed patterns, not independent mechanisms with falsifiable handles. The central load-bearing parameters are the readability-model weights and the prompt-analysis hyperparameters.

free parameters (3)
  • Readability model weights (logistic regression w, b) and SFS-selected features = Not reported; all-features model uses 25 selected features with 77.5% accuracy and 83.8% AUC on Dorn
    The logistic regression trained on Dorn labels produces the scores used for all LLM-vs-human comparisons and prompt-effect analyses. The weights are fitted to data and are not reported or validated on the target distribution.
  • Random forest hyperparameters (n_estimators, max_depth, min_samples_split) = 100, 15, 5
    Chosen by the authors for the RQ3 feature-importance analysis; affects the reported importance ranking and R^2.
  • Permutation importance significance threshold = p < 0.1
    Used in Section 3.5 to declare prompt dimensions relevant; a looser threshold than conventional p<0.05 and inconsistent with Table 7's bolded p<0.05.
axioms (4)
  • domain assumption Code from WoC (Oct 2021 snapshot) and LeetCode solutions before 2022 are human-written.
    Section 3.1 justifies this by the ChatGPT release date, but Codex and GitHub Copilot were already available in 2021, so the baseline may include AI-assisted code.
  • domain assumption The readability model trained on Dorn (360 snippets) yields valid readability measurements for WoC, LeetCode, and LLM-generated code.
    Section 4.3 reports only in-domain 10-fold cross-validation; no external validation on the target distributions or against human ratings.
  • domain assumption The 61 integrated features (TF/BWF/PF/DF) capture code readability, and the logistic-regression probability is a meaningful continuous readability score.
    Section 4 assumes the established metric families are valid and that the model output can be treated as a direct readability measurement.
  • domain assumption Annotator-identified issue patterns from 1,000 sampled pairs (500 WoC + 500 LeetCode) are representative of the full 5,869 scenarios and of LLM code generally.
    Section 3.4 samples 500 pairs per source; generalization beyond the sample and beyond one selected model is assumed.

pith-pipeline@v1.3.0-alltime-deepseek · 23123 in / 15452 out tokens · 144834 ms · 2026-08-04T05:08:12.668411+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are increasingly changing how code is produced, but generated code still requires human review and validation before it can be adapted or integrated into real-world projects. This makes the readability of LLM-generated code a critical concern. Existing studies have mainly focused on functional correctness and task completion of LLM generated code, leaving open questions about whether it is readable, how its readability fails, and to what extent prompt design can improve it. We therefore investigate the readability of LLM-generated code. We first construct a readability assessment model that integrates textual, structural, program, and visual features. Using this model, we compare human-written code with code generated by representative frontier LLMs across 2,735 scenarios derived from World of Code (WoC) and LeetCode. We further characterize readability issue patterns using thematic analysis and examine prompt design associations through controlled prompt-variant experiments. Our results show that current LLMs produce code that is comparable to human-written code in overall readability. However, this aggregate similarity masks distinct issue patterns, including excessive complexity, redundant comments, and unknown API usage. Prompt analysis further shows that function signature, constraints, and style description are most strongly associated with code readability, although the overall role of prompt design remains bounded. These findings reveal latent readability debt in AI-assisted programming, identify prompt design as a lightweight starting point for improving generated-code readability, and motivate automated support for detecting and mitigating readability issues in future development workflows.

Figures

Figures reproduced from arXiv: 2605.13280 by Fengyuan Ran, Hengzhi Ye, Minghui Zhou, Weiwei Xu.

Figure 1
Figure 1. Figure 1: Overview of the methodology cross-verification phase. Any discrepancies were resolved through iterative discussion to a consensus on all final expressions. We fi￾nally got a comprehensive set of 5,869 prompts reflecting diverse real-world development tasks. 3.2.2 Set B: Controlled Experimental Prompts (RQ3). Analyzing how specific prompt dimensions affect readability requires a rigor￾ous control of variabl… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of the readability scores of LLM [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Readability score distribution of code generated by [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Feature importance of the Random Forest regres [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Hitchhiker's Guide to Monoculture

    cs.CY 2026-07 conditional novelty 6.0

    Code written for Kaggle contests has become substantially more similar in syntax since ChatGPT, converging heavily on seed 42, while the semantic diversity of solution approaches has not declined.

Reference graph

Works this paper leans on

114 extracted references · 17 linked inside Pith · cited by 1 Pith paper

  1. [1]

    [n. d.]. LeetCode. https://leetcode.com/problemset/

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shya- mal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    Duaa Alawad, Manisha Panta, Minhaz Zibran, and Md Rakibul Islam. 2019. An empirical study of the relationships between code readability and software complexity.arXiv preprint arXiv:1909.01760(2019)

    Pith/arXiv arXiv 2019

  4. [4]

    André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. 2010. Per- mutation importance: a corrected feature importance measure.Bioinformatics 26, 10 (2010), 1340–1347

    2010

  5. [5]

    Anthropic. [n. d.]. Claude. https://www.anthropic.com/claude/

  6. [6]

    Anysphere. [n. d.]. Cursor. https://www.cursor.com/

  7. [7]

    Ashley. 2024. What is .cursorrule and How to Use It Effectively. https: //medium.com/towards-agi/what-are-cursor-rules-and-how-to-use-them- ec558468d139

    2024

  8. [8]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  9. [9]

    Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and chal- lenges of modern code review. In2013 35th International Conference on Software Engineering (ICSE). IEEE, 712–721

    2013

  10. [10]

    2006.Pattern recognition and machine learning

    Christopher M Bishop and Nasser M Nasrabadi. 2006.Pattern recognition and machine learning. Vol. 4. Springer

    2006

  11. [11]

    O’Reilly Media, Inc

    Dustin Boswell and Trevor Foucher. 2011.The art of readable code. " O’Reilly Media, Inc. "

    2011

  12. [12]

    Leo Breiman. 2001. Random forests.Machine learning45, 1 (2001), 5–32

    2001

  13. [13]

    Kiran Busch, Alexander Rochlitzer, Diana Sola, and Henrik Leopold. 2023. Just tell me: Prompt engineering in business process management. InInternational Conference on Business Process Modeling, Development and Support. Springer, 3–11

    2023

  14. [14]

    Raymond PL Buse and Westley R Weimer. 2008. A metric for software readability. InProceedings of the 2008 international symposium on Software testing and analysis. 121–130

    2008

  15. [15]

    Raymond PL Buse and Westley R Weimer. 2009. Learning a metric for code readability.IEEE Transactions on software engineering36, 4 (2009), 546–558

    2009

  16. [16]

    Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Pater- son, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In2015 ieee 23rd international conference on program comprehension. IEEE, 255–265

    2015

  17. [17]

    Mark Chen. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  18. [18]

    Tristan Coignion, Clément Quinton, and Romain Rouvoy. 2024. A performance study of llm-generated code on leetcode. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 79–89

    2024

  19. [19]

    Martha E Crosby and Jan Stelovsky. 2002. How do we read algorithms? A case study.Computer23, 1 (2002), 25–35

    2002

  20. [20]

    Pablo Roberto Fernandes de Oliveira, Rohit Gheyi, José Aldo Silva da Costa, and Márcio Ribeiro. 2024. Assessing Python Style Guides: An Eye-Tracking Study with Novice Developers. InSimpósio Brasileiro de Engenharia de Software (SBES). SBC, 136–146

    2024

  21. [21]

    George Digkas, Alexander Chatzigeorgiou, Apostolos Ampatzoglou, and Paris Avgeriou. 2020. Can clean new code reduce technical debt density?IEEE Transactions on Software Engineering48, 5 (2020), 1705–1721

    2020

  22. [22]

    Xi Ding, Rui Peng, Xiangping Chen, Yuan Huang, Jing Bian, and Zibin Zheng

  23. [23]

    2011.Software development and professional practice

    John Dooley and John Zukowski. 2011.Software development and professional practice. Springer

    2011

  24. [24]

    Jonathan Dorn. 2012. A general software readability model.MCS Thesis available at (web.eecs.umich.edu/~weimerw/students/dorn-mcs-paper.pdf)5 (2012), 11–14

    2012

  25. [25]

    James L Elshoff and Michael Marcotty. 1982. Improving computer program readability to aid modification.Commun. ACM25, 8 (1982), 512–521

    1982

  26. [26]

    Tom Fawcett. 2006. An introduction to ROC analysis.Pattern recognition letters 27, 8 (2006), 861–874

    2006

  27. [27]

    Louie Giray. 2023. Prompt engineering with ChatGPT: a guide for academic writers.Annals of biomedical engineering51, 12 (2023), 2629–2633

    2023

  28. [28]

    Github. [n. d.]. Copilot. https://github.com/features/copilot

  29. [29]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al . 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024)

    Pith/arXiv arXiv 2024

  30. [30]

    Isabelle Guyon and André Elisseeff. 2003. An introduction to variable and feature selection.Journal of machine learning research3, Mar (2003), 1157– 1182

    2003

  31. [31]

    1977.Elements of Software Science (Operating and program- ming systems series)

    Maurice H Halstead. 1977.Elements of Software Science (Operating and program- ming systems series). Elsevier Science Inc

    1977

  32. [32]

    Ardis Hanson. 2017. Negative case analysis.The international encyclopedia of communication research methods(2017), 1–2

    2017

  33. [33]

    Mohammad Hassany, Jiaze Ke, Peter Brusilovsky, Arun Balajiee Lek- shmi Narayanan, and Kamil Akhuseyinoglu. 2024. Authoring Worked Examples for JAVA Programming with Human AI Collaboration. InProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing. 101–103

    2024

  34. [34]

    Roberta Heale and Dorothy Forbes. 2013. Understanding triangulation in research.Evidence-based nursing16, 4 (2013), 98–98

    2013

  35. [35]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  36. [36]

    Chao Hu, Yitian Chai, Hao Zhou, Fandong Meng, Jie Zhou, and Xiaodong Gu

  37. [37]

    Yuan Huang, Nan Jia, Junhuai Shu, Xinyu Hu, Xiangping Chen, and Qiang Zhou. 2020. Does your code need comment?Software: Practice and Experience 50, 3 (2020), 227–245

    2020

  38. [38]

    InProceedings of the 39th IEEE/ACM International Conference on Auto- mated Software Engineering

    How Effectively Do Code Language Models Understand Poor-Readability Code?. InProceedings of the 39th IEEE/ACM International Conference on Auto- mated Software Engineering. 795–806

  39. [39]

    Ciera Jaspan and Collin Green. 2023. Defining, measuring, and managing technical debt.IEEE Software40, 03 (2023), 15–19

    2023

  40. [40]

    Reza Iranzad and Xiao Liu. 2025. A review of random forest-based feature selection methods for data science education and applications.International Journal of Data Science and Analytics20, 2 (2025), 197–211

    2025

  41. [41]

    Can Jin, Hongwu Peng, Shiyu Zhao, Zhenting Wang, Wujiang Xu, Ligong Han, Jiahui Zhao, Kai Zhong, Sanguthevar Rajasekaran, and Dimitris N Metaxas

  42. [42]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  43. [43]

    John Johnson, Sergio Lubo, Nishitha Yedla, Jairo Aponte, and Bonita Sharif

  44. [44]

    Sungmin Kang, Louis Milliken, and Shin Yoo. 2024. Identifying inaccurate descriptions in llm-generated code comments via test execution.arXiv preprint arXiv:2406.14836(2024)

    Pith/arXiv arXiv 2024

  45. [45]

    Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen

  46. [46]

    From llms to llm-based agents for software engineering: A survey of current, challenges and future.arXiv preprint arXiv:2408.02479(2024)

    Pith/arXiv arXiv 2024

  47. [47]

    Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2006. What’s in a Name? A Study of Identifiers. In14th IEEE international conference on program comprehension (ICPC’06). IEEE, 3–12

    2006

  48. [48]

    Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2007. Effec- tive identifier names for comprehension and memory.Innovations in Systems and Software Engineering3, 4 (2007), 303–318

    2007

  49. [49]

    Valentina Lenarduzzi, Terese Besker, Davide Taibi, Antonio Martini, and Francesca Arcelli Fontana. 2021. A systematic literature review on technical debt prioritization: Strategies, processes, factors, and tools.Journal of Systems and Software171 (2021), 110827

    2021

  50. [50]

    Ron Kohavi et al. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. InIjcai, Vol. 14. Montreal, Canada, 1137–1145

    1995

  51. [51]

    Dawn Lawrie, Henry Feild, and David Binkley. 2006. Syntactic identifier con- ciseness and consistency. In2006 Sixth IEEE International Workshop on Source Code Analysis and Manipulation. IEEE, 139–148. Ye et al

    2006

  52. [52]

    Sherlock A Licorish, Ansh Bajpai, Chetan Arora, Fanyu Wang, and Kla Tan- tithamthavorn. 2025. Comparing Human and LLM Generated Code: The Jury is Still Out!arXiv preprint arXiv:2501.16857(2025)

    Pith/arXiv arXiv 2025

  53. [53]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems36 (2023), 21558–21572

    2023

  54. [54]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM computing surveys 55, 9 (2023), 1–35

    2023

  55. [55]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al . 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161(2023)

    Pith/arXiv arXiv 2023

  56. [56]

    Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. GPT detectors are biased against non-native English writers.Patterns4, 7 (2023)

    2023

  57. [57]

    Yuxing Ma, Chris Bogart, Sadika Amreen, Russell Zaretzki, and Audris Mockus

  58. [58]

    Yuxing Ma, Tapajit Dey, Chris Bogart, Sadika Amreen, Marat Valiev, Adam Tutko, David Kennard, Russell Zaretzki, and Audris Mockus. 2021. World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data.Empirical Software Engineering26 (2021), 1–42

    2021

  59. [59]

    Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba- Nabende. 2023. Prompt engineering in large language models. InInternational conference on data intelligence and cognitive informatics. Springer, 387–402

    2023

  60. [60]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664(2021)

    Pith/arXiv arXiv 2021

  61. [61]

    Michael R Lyu, Baishakhi Ray, Abhik Roychoudhury, Shin Hwei Tan, and Patanamon Thongtanunam. 2024. Automatic programming: Large language models and beyond.ACM Transactions on Software Engineering and Methodology (2024)

    2024

  62. [62]

    Leon Moonen. 2001. Generating robust parsers using island grammars. In Proceedings eighth working conference on reverse engineering. IEEE, 13–22

    2001

  63. [63]

    In2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)

    World of code: an infrastructure for mining the universe of open source VCS data. In2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 143–154

  64. [64]

    OpenAI. [n. d.]. ChatGPT release. https://openai.com/index/chatgpt/

  65. [65]

    2007.Beautiful code Leading programmers explain how they think

    Andy Oram and Greg Wilson. 2007.Beautiful code Leading programmers explain how they think. O’Reilly Media, Inc

    2007

  66. [66]

    Orni Meerbaum-Salant, Michal Armoni, and Mordechai Ben-Ari. 2011. Habits of programming in scratch. InProceedings of the 16th annual joint conference on Innovation and technology in computer science education. 168–172

    2011

  67. [67]

    Prabhaker Mishra, Uttam Singh, Chandra M Pandey, Priyadarshni Mishra, and Gaurav Pandey. 2019. Application of student’s t-test, analysis of variance, and covariance.Annals of cardiac anaesthesia22, 4 (2019), 407–411

    2019

  68. [68]

    Sebastiano Panichella, Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. 2015. Would static analysis tools help developers with code reviews?. In2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 161–170

    2015

  69. [69]

    Delano Oliveira, Reydne Santos, Benedito De Oliveira, Martin Monperrus, Fer- nando Castor, and Fernanda Madeiral. 2024. Understanding Code Understand- ability Improvements in Code Reviews.IEEE Transactions on Software Engineer- ing(2024)

    2024

  70. [70]

    Felix Petersen, Debarghya Mukherjee, Yuekai Sun, and Mikhail Yurochkin

  71. [71]

    Valentina Piantadosi, Fabiana Fierro, Simone Scalabrino, Alexander Serebrenik, and Rocco Oliveto. 2020. How does code readability change during software evolution?Empirical Software Engineering25 (2020), 5374–5412

    2020

  72. [72]

    Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Andrea De Lucia. 2014. Do they really smell bad? a study on developers’ perception of bad code smells. In2014 IEEE International conference on software maintenance and evolution. IEEE, 101–110

    2014

  73. [73]

    Dangfeng Pan, Zhensu Sun, Cenyuan Zhang, David Lo, and Xiaoning Du. 2025. The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget.arXiv preprint arXiv:2508.13666(2025)

    Pith/arXiv arXiv 2025

  74. [74]

    David Ramel. 2023. GitHub Copilot AI Tech Upgraded, Already Generates 61% of Java Code. https://visualstudiomagazine.com/articles/2023/02/15/copilot- upgrade.aspx

    2023

  75. [75]

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models.Advances in neural information processing systems34 (2021), 11054–11070

    2021

  76. [76]

    Rodrigo REBOUÇAS DE ALMEIDA, Christoph TREUDE, and Uirá KULESZA

  77. [77]

    Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares- Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2017. Automatically assessing code understandability: How far are we?. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 417–427

    2017

  78. [78]

    Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshy- vanyk. 2018. A comprehensive model for code readability.Journal of Software: Evolution and Process30, 6 (2018), e1958

    2018

  79. [79]

    Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A simpler model of software readability. InProceedings of the 8th working conference on mining software repositories. 73–82

    2011

  80. [80]

    Vaclav Rajlich and Prashant Gosavi. 2004. Incremental change in object-oriented programming.IEEE software21, 4 (2004), 62–69

    2004

Showing first 80 references.