Pith. sign in

REVIEW 2 major objections 5 minor 85 references

Programmers catch correct LLM assertions far more often than incorrect ones, yet stay equally confident either way.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 06:01 UTC pith:M4O657F3

load-bearing objection Clean controlled evidence that developers accept correct LLM postconditions far more readily than they reject incorrect ones, and that comments do not fix the problem. the 2 major comments →

arxiv 2607.08885 v1 pith:M4O657F3 submitted 2026-07-09 cs.SE

Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions

Zhanna Kaufman , Yuriy Brun , Adithya Murali , Madeline Endres This is my paper
classification cs.SE
keywords LLM-generated assertionspostconditionscode reviewdeveloper judgmentnatural-language explanationsoverconfidencehuman-AI interactionsoftware reliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

As AI tools generate more code, the hard part of software work is shifting from writing to reviewing. One proposed fix is to let models also generate compact reliability artifacts such as postcondition assertions, so developers can check intended behavior without reading every line of implementation. This paper tests whether that review step actually works. In a controlled study of 86 Python programmers, people recognized correct assertions about three-quarters of the time, but spotted incorrect ones only about half the time, while reporting nearly identical high confidence in both cases. Natural-language explanations that accompanied the assertions did not help overall; under-specified explanations even hurt accuracy while raising confidence. A follow-up think-aloud study shows that developers often compare assertion clauses directly to documentation and rely on negative examples mainly after they already suspect a problem. The authors argue that generating reliability artifacts is not enough: tools must also help people evaluate them.

Core claim

The odds of a developer accurately judging a correct LLM-generated postcondition are nearly three times higher than the odds of accurately judging an incorrect one, with no corresponding difference in self-reported confidence. Natural-language explanations provide no overall accuracy gain, and under-specified explanations can reduce accuracy while increasing confidence.

What carries the argument

A balanced corpus of 26 HumanEval functions, each paired with one correct and one incorrect LLM-generated postcondition and five comment-quality conditions (exact, over-specified, under-specified, incorrect, none), scored with mixed-effects models of accuracy, confidence, and response time plus a directed think-aloud analysis of reasoning strategies.

Load-bearing premise

The claim rests on the idea that short, single-function Python postconditions from a filtered HumanEval corpus, judged in a short online survey, stand in for how developers will review real AI-generated reliability artifacts.

What would settle it

Repeat the same accuracy and confidence measures on multi-function or production codebases with modern model-generated assertions, and check whether the correct-versus-incorrect accuracy gap and the under-specified-comment harm disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Simply attaching LLM-generated assertions or comments will not reliably improve code-review quality.
  • Under-specified explanations can create a false sense of understanding while making wrong judgments more likely.
  • Tools that only generate reliability artifacts leave an unexamined human-verification gap.
  • Assertion structures that afford direct clause comparison are easier for developers to evaluate correctly.
  • Future AI-assisted reliability workflows need explicit support for verifying that generated artifacts match intent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same acceptance bias may affect review of LLM-generated tests, formal specs, and repair suggestions, not only postconditions.
  • Interfaces that force developers to produce a counterexample before accepting an assertion could counteract the observed asymmetry.
  • Flagging high-cyclomatic or implication-heavy assertions for extra scrutiny may be a cheap partial mitigation.
  • Confidence-calibration feedback during review could surface the overconfidence the study documents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper reports a controlled experiment with 86 Python programmers (plus a 10-person think-aloud study) on how well developers judge the correctness and completeness of LLM-generated postcondition assertions, and whether natural-language comments of varying quality help. Using a corpus of 26 HumanEval functions with correct/incorrect postconditions and five comment conditions (exact, over-specified, under-specified, incorrect, none), the authors find a large asymmetry: 74% accuracy on correct assertions vs. 49% on incorrect ones (OR = 2.94, p < 0.001), with similarly high confidence in both cases. Comments provide no overall accuracy benefit; under-specified comments reduce accuracy relative to exact comments (OR = 0.58) while raising confidence. Completeness ratings correlate modestly with an external mutation-based metric, and the think-aloud study identifies five recurring reasoning strategies, especially clause comparison and negative examples. The authors conclude that AI-generated reliability artifacts require better support for human evaluation, not only generation.

Significance. If the result holds under the stated conditions, the paper makes a timely and practically important contribution to AI-assisted software reliability. It challenges the common assumption that generating assertions or natural-language explanations will automatically improve code review, and it quantifies a concrete failure mode: developers are overconfident false-acceptors of incorrect specifications. Strengths include pre-registration of the comment hypotheses, mixed-effects models with participant and stimulus random effects, attention checks, a balanced design, a replication package, and qualitative triangulation. The OR = 2.94 asymmetry and the under-specified-comment harm-plus-confidence effect are clear, falsifiable findings that tool builders and SE researchers can act on. External validity is limited by the HumanEval/GPT-3.5 setting, but that limit is acknowledged and does not erase the internal contribution.

major comments (2)
  1. The central correctness asymmetry (RQ1) and the comment-quality effects (RQ2) are load-bearing and well supported by the pre-registered mixed-effects analyses and the 86-participant sample. No major statistical or design flaw overturns those claims under the experimental conditions. The main load-bearing limitation is external validity: Sections 3.1.2 and 9 restrict the corpus to 26 filtered HumanEval functions and GPT-3.5/4 postconditions from Endres et al., with 10 online stimuli per participant. The manuscript already flags this; a revision should more explicitly bound the claim (e.g., 'for compact single-function postconditions of this form') rather than implying broad applicability to multi-function or production reliability artifacts without further evidence.
  2. RQ1, RQ3, and RQ4 are described as exploratory and post-hoc (Section 4). That is acceptable given the pre-registration for RQ2, but the abstract and introduction currently present the correctness asymmetry as a primary result with equal weight to the pre-registered comment findings. Clarify in the abstract and Section 4 which claims were confirmatory vs. exploratory, and avoid over-interpreting the exploratory feature analyses (type checks, implications, etc.) as established construction-pattern effects without stronger controls.
minor comments (5)
  1. Figure 4 confidence distributions are useful; ensure axis labels and the four-case layout remain readable in print, and consider reporting exact mean confidence values in the caption for the four cells.
  2. Section 6.3: the Spearman ρ = 0.18 completeness correlation is statistically significant but small; state the practical magnitude more carefully so readers do not over-read 'can distinguish stronger from weaker specifications.'
  3. Section 3.1.2: briefly note how many candidate comments were discarded during the three-author review, to give a sense of selection for the final 260 stimuli.
  4. Typographical consistency: 'underspecfied' appears once in Section 3.1; standardize to 'under-specified' throughout.
  5. Related work (Section 8) is thorough; a short explicit contrast with TiCoder and Specine on the human-evaluation gap would help position the contribution more sharply.

Circularity Check

0 steps flagged

No significant circularity: empirical accuracy and confidence results rest on independent ground-truth labels and mixed-effects models, not self-definitional or fitted-as-prediction constructions.

full rationale

This is a controlled human-subjects experiment, not a theoretical derivation. The central claims (OR = 2.94 asymmetry between correct vs. incorrect postconditions; no overall accuracy benefit from comments; under-specified comments reduce accuracy while raising confidence) are obtained by measuring participant judgments against independently labeled ground-truth correctness of assertions (correct = no error on any docstring-aligned I/O; incorrect = at least one such error), then fitting mixed-effects models with participant- and stimulus-level random effects. Completeness ratings are correlated with an external mutation-based bug-completeness metric from prior work rather than defined in terms of the ratings themselves. Pre-registration covers the comment-quality hypotheses; RQ1/RQ3/RQ4 analyses are labeled exploratory. Self-citations (e.g., Endres et al. for the postcondition corpus and completeness scores) supply stimuli and an external benchmark; they do not force the accuracy asymmetry or the comment effects by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported to forbid alternatives, and no ansatz is smuggled in via citation. The paper is self-contained against its stated experimental conditions; external-validity limits (HumanEval subset, GPT-3.5/4 postconditions, online survey) are acknowledged but are not circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

Empirical user-study paper; load-bearing premises are standard experimental assumptions plus domain choices about representativeness of HumanEval and LLM-generated postconditions. No free parameters are fitted to produce the central odds ratio; no new physical or mathematical entities are postulated.

axioms (3)
  • domain assumption Participant accuracy and confidence on 10 short online stimuli with HumanEval functions generalize to real developer review of LLM-generated reliability artifacts.
    Stated as a limitation in Section 9; the central claim about poor judgment of incorrect assertions rests on this transfer.
  • domain assumption Ground-truth correctness labels of the filtered postconditions (correct = never raises on valid I/O; incorrect = raises on at least one valid I/O) are accurate and non-trivial.
    Labels inherited and re-validated from Endres et al.; three authors reviewed consistency (Section 3.1.2).
  • standard math Mixed-effects logistic and linear models with participant and stimulus random effects correctly capture the repeated-measures structure.
    Standard statistical practice for the design; packages and significance threshold stated in Section 4.1.

pith-pipeline@v1.1.0-grok45 · 25833 in / 2384 out tokens · 33033 ms · 2026-07-13T06:01:36.567017+00:00 · methodology

0 comments
read the original abstract

Code comprehension and code review are already critically important software engineering tasks, and the rising use of AI code generation tools is only increasing that importance. Generative AI has the possibility of supporting these activities, for example by augmenting code with assertions and natural-language explanations describing code behavior. However, little is known about how effective such support may be. We conduct a controlled experiment with 86 Python programmers and a follow-up think-aloud study to examine developers' ability to assess the correctness and completeness of generated assertions of varying quality, and to investigate how natural-language explanations influence these assessments. While programmers can somewhat accurately judge correct assertions (74% accuracy), they perform poorly when shown incorrect assertions (49% accuracy), despite reporting similar levels of confidence in both judgments. This difference in judgment accuracy is statistically significant (p < 0.001): the odds of a developer accurately judging a correct assertion was nearly three times higher than the odds of accurately judging an incorrect assertion (OR = 2.94). Surprisingly, natural-language explanations of assertions provided no overall benefit. Furthermore, low-quality explanations could impair specification assessment accuracy (p = 0.037, OR = 0.58) while simultaneously increasing developer confidence (p = 0.005, 3.99/5 vs. 4.25/5). Our findings suggest that, contrary to common assumptions, AI assistance may not improve the reliability of code comprehension and review. More broadly, our findings highlight the importance of helping developers evaluate machine-generated reliability artifacts, in addition to generating them.

Figures

Figures reproduced from arXiv: 2607.08885 by Adithya Murali, Madeline Endres, Yuriy Brun, Zhanna Kaufman.

Figure 1
Figure 1. Figure 1: has an example stimulus, including the target function and associated postcondition conditions. Comment Conditions: For each postcondition–function pair, our corpus contained five stimuli: one for each comment condition. In our definitions, the ‘failure’ of an postcondition indicates that the Python assert statement will raise an error. ‘passing’ indicates that the Python assert statement will not error. W… view at source ↗
Figure 1
Figure 1. Figure 1: Example stimuli shown to participants, including the target function and associated postcondition conditions. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example comments corresponding to the correct postcondition in Figure 1 under each comment-quality condition. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Think-aloud participant overview. participants reported between one and ten years of programming ex￾perience, moderate or greater Python proficiency, frequent Python use, and at least some prior experience writing logical assertions and using LLMs. Think-Aloud Participants. To better understand the reasoning pro￾cesses underlying our quantitative findings, we recruited 10 addi￾tional participants for the t… view at source ↗
Figure 5
Figure 5. Figure 5: Functions curated for our qualitative follow-up study. These func [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Two Sankey diagrams showing the portion of responses from [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 12 canonical work pages

  1. [1]

    Replication Package for The impact of comments on the correct under- standing of logical code statements

    2026. Replication Package for The impact of comments on the correct under- standing of logical code statements. https://osf.io/q6d2m/overview?view_only= 442a1b8d82554de0baa4b4083987a6b6

    2026

  2. [2]

    Abid, Natalia Dragan, Michael L

    Nahla J. Abid, Natalia Dragan, Michael L. Collard, and Jonathan I. Maletic. 2015. Using stereotypes in the automatic generation of natural language summaries for C++ methods. In2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 561–565. doi:10.1109/ICSM.2015.7332514

    doi:10.1109/icsm.2015.7332514 2015

  3. [3]

    Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang

  4. [4]

    arXiv:2103.06333 [cs.CL] https://arxiv.org/abs/2103.06333

    Unified Pre-training for Program Understanding and Generation. arXiv:2103.06333 [cs.CL] https://arxiv.org/abs/2103.06333

    Pith/arXiv arXiv

  5. [5]

    Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Atten- tion Network for Extreme Summarization of Source Code. InInternational Con- ference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 2091–2100. https://proceedings.mlr....

    2016

  6. [6]

    Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and chal- lenges of modern code review. In2013 35th international conference on software engineering (ICSE). IEEE, 712–721

    2013

  7. [7]

    Tobias Baum, Kurt Schneider, and Alberto Bacchelli. 2019. Associating working memory capacity and code change ordering with code review performance. Empirical Software Engineering24, 4 (2019), 1762–1798

    2019

  8. [8]

    Ernst, Mauro Pezzè, and Sergio Delgado Castellanos

    Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Sergio Delgado Castellanos. 2018. Translating code comments to procedure specifications(ISSTA 2018). Association for Computing Machinery, New York, NY, USA, 242–253. doi:10.1145/3213846.3213872 10

    doi:10.1145/3213846.3213872 2018

  9. [9]

    Jürgen Börstler and Barbara Paech. 2016. The Role of Method Chains and Com- ments in Software Readability and Comprehension—An Experiment.IEEE Trans- actions on Software Engineering42, 9 (2016), 886–898. doi:10.1109/TSE.2016. 2527791

    doi:10.1109/tse.2016 2016

  10. [10]

    Cindy Candrian and Anne Scherer. 2022. Rise of the machines: Delegating decisions to autonomous AI.Computers in Human Behavior134 (2022), 107308. doi:10.1016/j.chb.2022.107308

    doi:10.1016/j.chb.2022.107308 2022

  11. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  12. [12]

    Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2025. Auto- mated Code Review in Practice. InInternational Conference on Software Engi- neering: Software Engineering in Practice (ICSE-SEIP). 425–436. doi:10.1109/ICSE- SEIP66354.2025.00043

    doi:10.1109/icse- 2025

  13. [13]

    Douglas Curran-Everett. 2018. Explorations in statistics: the log trans- formation.Advances in Physiology Education42, 2 (2018), 343–347. arXiv:https://doi.org/10.1152/advan.00018.2018 doi:10.1152/advan.00018.2018 PMID: 29761718

    work page doi:10.1152/advan.00018.2018 2018

  14. [14]

    A Dunsmore, M Roper, and M Wood. 2000. The role of comprehension in software inspection.Journal of Systems and Software52, 2 (2000), 121–129. doi:10.1016/ S0164-1212(99)00138-7

    2000

  15. [15]

    Eddy, Jeffrey A

    Brian P. Eddy, Jeffrey A. Robinson, Nicholas A. Kraft, and Jeffrey C. Carver. 2013. Evaluating source code summarization techniques: Replication and expansion. InInternational Conference on Program Comprehension. 13–22. doi:10.1109/ICPC. 2013.6613829

    doi:10.1109/icpc 2013

  16. [16]

    Hadeel Eladawy, Claire Le Goues, and Yuriy Brun. 2024. Automated Program Repair, What Is It Good For? Not Absolutely Nothing!. InInternational Confer- ence on Software Engineering(14–20). Lisbon, Portugal, 1017–1029. doi:10.1145/ 3597503.3639095

    arXiv 2024

  17. [17]

    Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri

  18. [18]

    doi:10.1145/ 3660791

    Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?PACMSE1, FSE (2024), 84:1–84:24. doi:10.1145/ 3660791

    2024

  19. [19]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shu- vendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation.IEEE Transactions on Software Engineering50, 9 (Sept. 2024), 2254–2268. doi:10.1109/tse.2024.3428972

    doi:10.1109/tse.2024.3428972 2024

  20. [20]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational L...

    2020

  21. [21]

    doi:10.18653/v1/2020.findings-emnlp.139

    doi:10.18653/v1/2020.findings-emnlp.139 2020

  22. [22]

    Kathrin Figl, Maria Kirchner, Sebastian Baltes, and Michael Felderer. 2025. The Influence of Code Comments on the Perceived Helpfulness of Stack Overflow Posts. 30, 6 (2025). doi:10.1007/s10664-025-10727-w

    work page doi:10.1007/s10664-025-10727-w 2025

  23. [23]

    Emily First, Markus Rabe, Talia Ringer, and Yuriy Brun. 2023. Baldur: Whole- Proof Generation and Repair with Large Language Models. InACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)(6–8). San Francisco, CA, USA, 1229–1241. doi:10.1145/ 3611643.3616243

    arXiv 2023

  24. [24]

    Erin Foster and Ariel Deardorff. 2017. Open Science Framework (OSF).Journal of the Medical Library Association105 (04 2017). doi:10.5195/JMLA.2017.88

    doi:10.5195/jmla.2017.88 2017

  25. [25]

    Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nie, Xin Xia, and Michael Lyu. 2023. Code Structure–Guided Transformer for Source Code Summa- rization.ACM Trans. Softw. Eng. Methodol.32, 1, Article 23 (Feb. 2023), 32 pages. doi:10.1145/3522674

    work page doi:10.1145/3522674 2023

  26. [26]

    Ernst, and Mauro Pezzè

    Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Auto- matic generation of oracles for exceptional behaviors. InInternational Sympo- sium on Software Testing and Analysis (ISSTA). Saarbrücken, Genmany, 213–224. doi:10.1145/2931037.2931061

    doi:10.1145/2931037.2931061 2016

  27. [27]

    Skyler Grandel, Scott Thomas Andersen, Yu Huang, and Kevin Leach. 2026. ComCat: Expertise-Guided Context Generation to Enhance Code Comprehension. 35, 3, Article 82 (Feb. 2026), 30 pages. doi:10.1145/3742475

    work page doi:10.1145/3742475 2026

  28. [28]

    Burak Gülmez. 2026. Code generation with large language models: a survey from neural program synthesis to autonomous software development.Applied Intelligence56, 6 (2026), 200. doi:10.1007/s10489-026-07230-0

    work page doi:10.1007/s10489-026-07230-0 2026

  29. [29]

    Sonia Haiduc, Jairo Aponte, and Andrian Marcus. 2010. Supporting program comprehension with source code summarization. In32nd International Con- ference on Software Engineering - Volume 2(Cape Town, South Africa)(ICSE ’10). Association for Computing Machinery, New York, NY, USA, 223–226. doi:10.1145/1810295.1810335

    work page doi:10.1145/1810295.1810335 2010

  30. [30]

    Ishrak Hayet, Adam Scott, and Marcelo d’Amorim. 2025. ChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance.IEEE Trans. Softw. Eng. 51, 1 (Jan. 2025), 305–319. doi:10.1109/TSE.2024.3519159

    doi:10.1109/tse.2024.3519159 2025

  31. [31]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  32. [32]

    Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis.Qualitative health research15, 9 (2005), 1277–1288

    2005

  33. [33]

    Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. InProceedings of the 26th Conference on Program Comprehension (Gothenburg, Sweden)(ICPC ’18). Association for Computing Machinery, New York, NY, USA, 200–210. doi:10.1145/3196321.3196334

    doi:10.1145/3196321.3196334 2018

  34. [34]

    Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zimmer- mann. 2022. Practitioners’ expectations on automated code comment generation. InProceedings of the 44th International Conference on Software Engineering(Pitts- burgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New York, NY, USA, 1693–1705. doi:10.1145/3510003.3510152

    doi:10.1145/3510003.3510152 2022

  35. [35]

    Yuan Huang, Shaohao Huang, Huanchao Chen, Xiangping Chen, Zibin Zheng, Xiapu Luo, Nan Jia, Xinyu Hu, and Xiaocong Zhou. 2020. Towards automati- cally generating block comments for code snippets.Information and Software Technology127 (2020), 106373. doi:10.1016/j.infsof.2020.106373

    work page doi:10.1016/j.infsof.2020.106373 2020

  36. [36]

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing Source Code using a Neural Attention Model. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Katrin Erk and Noah A. Smith (Eds.). Association for Computational Linguistics, Berlin, Germany, 2073–2083. do...

    doi:10.18653/v1/p16-1195 2016

  37. [37]

    Jasmin Jahić and Ashkan Sami. 2024. State of Practice: LLMs in Software En- gineering and Software Architecture. InInternational Conference on Software Architecture (ICSA). 311–318. doi:10.1109/ICSA-C63560.2024.00059

    doi:10.1109/icsa-c63560.2024.00059 2024

  38. [38]

    Erik Jones, Hamid Palangi, Clarisse Simões Ribeiro, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Hassan Awadallah, and Ece Kamar. 2024. Teaching Language Models to Hallucinate Less with Synthetic Tasks. InThe Twelfth International Conference on Learning Representations. https: //openreview.net/forum?id=xpw7V0P136

    2024

  39. [39]

    Sungmin Kang, Louis Milliken, and Shin Yoo. 2024. Identifying Inaccu- rate Descriptions in LLM-generated Code Comments via Test Execution. arXiv:2406.14836 [cs.SE] https://arxiv.org/abs/2406.14836

    Pith/arXiv arXiv 2024

  40. [40]

    Zhanna Kaufman, Yuriy Brun, Adithya Murali, and Madeline Endres. 2025. The impact of comments on the correct understanding of logical code statements. osf.io/5f73u

    2025

  41. [41]

    Ninus Khamis, René Witte, and Juergen Rilling. 2010. Automatic Quality As- sessment of Source Code Comments: The JavadocMiner. InNatural Language Processing and Information Systems, Christina J. Hopfe, Yacine Rezgui, Elisabeth Métais, Alun Preece, and Haijiang Li (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 68–79

    2010

  42. [42]

    Junaed Younus Khan and Gias Uddin. 2023. Automatic Code Documentation Generation Using GPT-3. InInternational Conference on Automated Software En- gineering(Rochester, MI, USA)(ASE ’22). Association for Computing Machinery, New York, NY, USA, Article 174, 6 pages. doi:10.1145/3551349.3559548

    doi:10.1145/3551349.3559548 2023

  43. [43]

    Shuvendu K Lahiri. 2026. Intent formalization: A grand challenge for reliable coding in the age of AI agents.arXiv preprint arXiv:2603.17150(2026)

    arXiv 2026

  44. [44]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen

  45. [45]

    InInternational Conference on Software Engineering (ICSE)

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. InInternational Conference on Software Engineering (ICSE). IEEE, 919–931

  46. [46]

    Changwen Li, Christoph Treude, and Ofir Turel. 2026. Do comments and expertise still matter? An experiment on programmers’ adoption of AI-generated JavaScript code.Journal of Systems and Software231 (2026), 112634. doi:10.1016/j.jss.2025. 112634

    doi:10.1016/j.jss.2025 2026

  47. [47]

    Hui Li, Zhen Dong, Siao Wang, Hui Zhang, Liwei Shen, Xin Peng, and Dongdong She. 2025. Extracting Formal Specifications From Documents Using LLMS for Test Automation. InInternational Conference on Program Comprehension (ICPC). 1–12. doi:10.1109/ICPC66645.2025.00039

    doi:10.1109/icpc66645.2025.00039 2025

  48. [48]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InAnnual Meeting of the Association for Computational Linguistics. https://api. semanticscholar.org/CorpusID:964287

    2004

  49. [49]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. InThirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=1qvx610Cu7 11

    2023

  50. [50]

    Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, and Yang Liu. 2021. Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. arXiv:2006.05405 [cs.LG] https://arxiv.org/abs/2006.05405

    Pith/arXiv arXiv 2021

  51. [51]

    McBurney and Collin McMillan

    Paul W. McBurney and Collin McMillan. 2014. Automatic documentation genera- tion via source code summarization of method context. InProceedings of the 22nd International Conference on Program Comprehension(Hyderabad, India)(ICPC 2014). ACM, New York, NY, USA, 279–290. doi:10.1145/2597008.2597149

    doi:10.1145/2597008.2597149 2014

  52. [52]

    Vishal Misra, Jakku Sai Krupa Reddy, and Sridhar Chimalakonda. 2020. Is there a correlation between code comments and issues? an exploratory study. InPro- ceedings of the 35th Annual ACM Symposium on Applied Computing(Brno, Czech Republic)(SAC ’20). Association for Computing Machinery, New York, NY, USA, 110–117. doi:10.1145/3341105.3374009

    work page doi:10.1145/3341105.3374009 2020

  53. [53]

    Ernst, and Mauro Pezzè

    Davide Molinelli, Luca Di Grazia, Alberto Martin-Lopez, Michael D. Ernst, and Mauro Pezzè. 2025. Do LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset. InInternational Conference on Automated Software Engineering (ASE). 278–290. doi:10.1109/ASE63991.2025.00031

    doi:10.1109/ase63991.2025.00031 2025

  54. [54]

    Manish Motwani and Yuriy Brun. 2019. Automatically Generating Precise Oracles from Structured Natural Language Specifications. InInternational Conference on Software Engineering(29–31). Montreal, QC, Canada, 188–199. doi:10.1109/ICSE. 2019.00035

    doi:10.1109/icse 2019

  55. [55]

    Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring method specifications from natural language API descriptions. In2012 34th International Conference on Software Engineering (ICSE). 815–825. doi:10.1109/ICSE.2012.6227137

    doi:10.1109/icse.2012.6227137 2012

  56. [56]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting on Association for Computational Linguistics(Philadelphia, Penn- sylvania)(ACL ’02). Association for Computational Linguistics, USA, 311–318. doi:10.3115/1073083.1073135

    doi:10.3115/1073083.1073135 2002

  57. [57]

    Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. 2021. Retrieval Augmented Code Generation and Summarization. InFindings of the Association for Computational Linguistics: EMNLP 2021, Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, ...

    doi:10.18653/v1/2021.findings-emnlp.232 2021

  58. [58]

    Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Annibal, Alec Peltekian, and Yanfang Ye. 2021. CoTexT: Multi-task Learning with Code-Text Trans- former. InProceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), Royi Lachmy, Ziyu Yao, Greg Durrett, Milos Gligoric, Junyi Jessy Li, Ray Mooney, Graham Neubig, Yu Su,...

    doi:10.18653/v1/2021.nlp4prog-1.5 2021

  59. [59]

    Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good vibrations? A qualitative study of co-creation, communication, flow, and trust in vibe coding.arXiv preprint arXiv:2509.12491 (2025)

    Pith/arXiv arXiv 2025

  60. [60]

    2025.RStudio: Integrated Development Environment for R

    Posit team. 2025.RStudio: Integrated Development Environment for R. Posit Software, PBC, Boston, MA. http://www.posit.co/

    2025

  61. [61]

    Siddhartha Prasad, Skyler Austen, Kathi Fisler, and Shriram Krishnamurthi. 2026. Meaningful Human-in-the-Loop Checking of GenAI Synthesis for Restricted Languages. In40th European Conference on Object-Oriented Programming (ECOOP 2026), Vol. 372. 22:1–22:31. doi:10.4230/LIPIcs.ECOOP.2026.22

    work page doi:10.4230/lipics.ecoop.2026.22 2026

  62. [62]

    Kenneth J Rothman. 1990. No adjustments are needed for multiple comparisons. Epidemiology1, 1 (1990), 43–46

    1990

  63. [63]

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. SpecRover: Code Intent Extraction via LLMs. InInternational Conference on Software Engineering (Ottawa, Ontario, Canada)(ICSE ’25). IEEE Press, 963–974. doi:10.1109/ICSE55347. 2025.00080

    doi:10.1109/icse55347 2025

  64. [64]

    Advait Sarkar, Xiaotong, Xu, Neil Toronto, Ian Drosos, and Christian Poelitz. 2024. When Copilot Becomes Autopilot: Generative AI’s Critical Risk to Knowledge Work and a Critical Solution. arXiv:2412.15030 [cs.HC] https://arxiv.org/abs/ 2412.15030

    Pith/arXiv arXiv 2024

  65. [65]

    Rishab Sharma, Fuxiang Chen, and Fatemeh Fard. 2022. LAMNER: code comment generation using character language model and named entity recognition. In International Conference on Program Comprehension(Virtual Event)(ICPC ’22). ACM, New York, NY, USA, 48–59. doi:10.1145/3524610.3527924

    work page doi:10.1145/3524610.3527924 2022

  66. [66]

    Vijay- Shanker

    Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K. Vijay- Shanker. 2010. Towards automatically generating summary comments for Java methods. InInternational Conference on Automated Software Engineering (Antwerp, Belgium)(ASE ’10). ACM, New York, NY, USA, 43–52. doi:10.1145/ 1858996.1859006

    arXiv 2010

  67. [67]

    Stack Overflow. 2025. 2025 Stack Overflow Developer Survey. https://survey. stackoverflow.co/2025 Accessed June 2026

    2025

  68. [68]

    Sean Stapleton, Yashmeet Gambhir, Alexander LeClair, Zachary Eberhart, Westley Weimer, Kevin Leach, and Yu Huang. 2020. A Human Study of Comprehension and Code Summarization. InProceedings of the 28th International Conference on Program Comprehension(Seoul, Republic of Korea)(ICPC ’20). Association for Computing Machinery, New York, NY, USA, 2–13. doi:10....

    doi:10.1145/3387904.3389258 2020

  69. [69]

    Daniela Steidl, Benjamin Hummel, and Elmar Jürgens. 2013. Quality analysis of source code comments.International Conference on Program Comprehension (ICPC)(2013), 83–92. https://api.semanticscholar.org/CorpusID:16657129

    2013

  70. [70]

    Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*icomment: bugs or bad comments?*/.SIGOPS Oper. Syst. Rev.41, 6 (Oct. 2007), 145–158. doi:10.1145/1323293.1294276

    doi:10.1145/1323293.1294276 2007

  71. [71]

    Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. In2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. 260–269. doi:10.1109/ICST.2012.106

    doi:10.1109/icst.2012.106 2012

  72. [72]

    Zhao Tian and Junjie Chen. 2025. Aligning Requirement for Large Language Model’s Code Generation. arXiv:2509.01313 [cs.SE] https://arxiv.org/abs/2509. 01313

    Pith/arXiv arXiv 2025

  73. [73]

    Glassman

    Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. InExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems(New Orleans, LA, USA)(CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Arti...

    doi:10.1145/3491101.3519665 2022

  74. [74]

    Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. Improving automatic source code summarization via deep reinforcement learning. InInternational Conference on Automated Software Engi- neering(Montpellier, France)(ASE ’18). Association for Computing Machinery, New York, NY, USA, 397–407. doi:10.1145/3238147.3238206

    doi:10.1145/3238147.3238206 2018

  75. [75]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Under- standing and Generation. InConference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (Eds.). Association for Computational Lingui...

    doi:10.18653/v1/2021.emnlp-main.685 2021

  76. [76]

    Robert M West. 2022. Best practice in statistics: The use of log transformation.Annals of Clinical Biochemistry59, 3 (2022), 162–165. arXiv:https://doi.org/10.1177/00045632211050531 doi:10.1177/00045632211050531 PMID: 34666549

    work page doi:10.1177/00045632211050531 2022

  77. [77]

    Haoze Wu, Rocky Klopfenstein, Keith Farkas, and Nina Narodytska. 2026. Viverra: Text-to-Code with Guarantees. arXiv:2605.14972 [cs.SE] https://arxiv.org/abs/ 2605.14972

    Pith/arXiv arXiv 2026

  78. [78]

    Hassan, and Shan- ping Li

    Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shan- ping Li. 2018. Measuring Program Comprehension: A Large-Scale Field Study with Professionals.IEEE Transactions on Software Engineering44, 10 (2018), 951–976. doi:10.1109/TSE.2017.2734091

    work page doi:10.1109/tse.2017.2734091 2018

  79. [79]

    Juan Zhai, Yu Shi, Minxue Pan, Guian Zhou, Yongxiang Liu, Chunrong Fang, Shiqing Ma, Lin Tan, and Xiangyu Zhang. 2020. C2S: translating natural language comments to formal program specifications(ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 25–37. doi:10.1145/3368089.3409716

    doi:10.1145/3368089.3409716 2020

  80. [80]

    Hao Zhong, Lu Zhang, Tao Xie, and Hong Mei. 2009. Inferring Resource Specifi- cations from Natural Language API Documentation. InInternational Conference on Automated Software Engineering. 307–318. doi:10.1109/ASE.2009.94

    doi:10.1109/ase.2009.94 2009

Showing first 80 references.