Pith. sign in

REVIEW 4 major objections 5 minor 54 references

Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most LLM-generated chain-of-thought steps for code are low quality, and a correct chain still leads to wrong code 18.5% of the time.

desk verdict The CoT-quality taxonomy is a genuine contribution, but the paper's flagship 18.5% figure is contradicted by its own Table I (it's 23.2%), and the factor percentages don't reconcile across the text, figures, and tables. read the letter →

arxiv 2507.06980 v1 pith:QO5OZPEO submitted 2025-07-09 cs.SE

Binquan Zhang , Li Zhang , Zhiwen Luo , Yuxin Du , Fang Liu , Song Wang , Lin Shi This is my paper
classification cs.SE
keywords chain-of-thoughtcodegenerationlargelanguagemodelsCoTqualitytaxonomyempiricalstudymulti-agentdebateself-repair
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the step-by-step reasoning chains that LLMs produce before writing code are actually trustworthy, and it answers with a large empirical measurement. Across 1,023 chain-of-thought/code pairs from two code-generation benchmarks, the authors find that 76.4% of the CoTs have quality issues; 53.6% of those issues come from external factors such as unclear requirements and missing context, while 40.1% come from internal reasoning errors. The study also shows that CoT quality and code correctness are only loosely coupled: a correct CoT is followed by failing code 18.5% of the time, and a flawed CoT still sometimes produces correct code. The paper further reports that multi-agent debate detects bad CoTs better than a single LLM, and that more detailed feedback improves LLMs' ability to repair their own CoTs. If the measurements are right, they suggest that CoT-based code generation needs both better input requirements and independent verification of the final code.

What carries the argument

The analytic engine is a two-level taxonomy of CoT quality faults, built by expert annotation of 813 failing code samples and applied to all 1,023 pairs. External factors split into Unclear Implementation Details and Missing Contextual Information; internal factors split into Misunderstanding Explicit Requirements, Incomprehension of Implicit Requirements, and Incorrect Planning. The taxonomy carries the argument because every reported percentage—the 53.6% external / 40.1% internal split, the per-dataset distributions, and the cross-tabulation with pass@1—is a count of samples placed in these categories. A second mechanism is the cross-tabulation of CoT correctness against code correctness on the 210 passing-code samples, which produces the paper's key surprise: correct reasoning does not guarantee correct code. Detection and repair experiments then reuse the same taxonomy as the target output (for the multi-agent debate detector) and as the error description (for feedback-driven self-repair).

What would settle it

Re-annotate the same 1,023 CoT-code pairs with annotators who see only the original problem statement and the CoT, without reference solutions or execution results; if the external/internal split (53.6% versus 40.1%) shifts materially, the reported factor distribution is an artifact of outcome knowledge rather than a property of the CoTs themselves.

Watch

Extended reading notes

Core claim

The central discovery is a quantitative account of how often and why LLM-generated chain-of-thought reasoning fails in code generation. On a corpus of 1,023 CoT-code pairs produced by DeepSeek-R1, o1, and Gemini-2.0-Flash-Thinking on CoderEval and SWE-bench-NF, the authors report that 76.4% of CoTs are of low quality. They build a taxonomy in which external factors (unclear implementation details and missing contextual information) account for 53.6% of quality issues and internal factors (misunderstanding explicit requirements, failing to grasp implicit requirements, and incorrect planning) account for 40.1%. The paper also shows that CoT correctness does not determine code correctness: correct CoTs yield failing code in 18.5% of cases, and incorrect CoTs still yield passing code in 3.1% of cases. Based on these findings, the paper argues that CoT quality is an independent failure dimension that needs its own detection and repair mechanisms, which they test through multi-agent debate and feedback-driven self-repair.

Load-bearing premise

The load-bearing premise is that the human annotators, who classified CoT quality while looking at the reference solution and test execution results, produced an unbiased and consistent ground truth about what makes a CoT flawed.

Editorial extensions

If this is right

  • CoT correctness is not a sufficient condition for code correctness: 18.5% of correct CoTs are followed by code that fails tests, so code verification must remain separate from reasoning verification.
  • CoT quality problems mostly originate in the prompt: 53.6% of faulty CoTs trace to external factors such as vague implementation details or missing context, implying better requirement specification could prevent most CoT failures.
  • Flawed CoT does not doom the code: 11.9% of correct code samples have flawed CoTs, and overall an incorrect CoT still yields correct code 3.1% of the time, so models sometimes compensate for reasoning errors.
  • Multi-agent debate detects low-quality CoTs with higher recall than any single model, but at the cost of much lower precision and higher compute, making detection a cost-performance tradeoff.
  • Self-repair is feasible and improves with feedback granularity; detailed error descriptions raise pass@1 over simple feedback on both benchmarks, though absolute gains are small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 18.5% gap between correct CoT and failing code is consistent with CoT being a post-hoc narrative rather than a faithful trace of computation; if so, code correctness should be verified directly rather than through reasoning chains.
  • The taxonomy can be used as training data to build an automated CoT-quality classifier, which would let future work screen large numbers of reasoning chains without human annotation.
  • Because external factors dominate at 53.6%, a cheap testable intervention is to add a requirement-clarification pass before CoT generation; the repair results suggest such clarifications would translate into measurable pass@1 gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper empirically evaluates the quality of LLM-generated chain-of-thought (CoT) reasoning in code generation. Using CoderEval and a SWE-bench 'New Features' subset, the authors collect 1,023 CoT-code pairs from three reasoning LLMs, annotate the 813 failed-code CoTs with a two-level taxonomy (external vs. internal factors), and report conditional relationships between CoT quality and code correctness. They also propose a multi-agent debate detector for low-quality CoTs and study single-model self-repair under three feedback granularities. The headline findings are that 76.4% of CoTs have quality issues, external factors dominate (53.6%), correct CoTs nonetheless lead to erroneous code in 18.5% of cases, and more detailed feedback improves CoT repair.

Significance. If the quantitative results were accurate, this would be a useful empirical contribution: it addresses a real gap in evaluating the quality of intermediate reasoning in code generation, uses three current reasoning models and two benchmarks, proposes a taxonomy, and ships data and scripts for replication. The paper also makes falsifiable quantitative claims rather than stopping at qualitative impressions. However, several central numbers do not survive recomputation from the paper's own tables, and the annotation procedure for the passing-code subset is underspecified. The qualitative direction of the findings is plausible, but the quantitative grounding must be repaired before the contribution can be assessed.

major comments (4)
  1. [Section III.D / Table I / Abstract] The headline 18.5% figure is not supported by Table I. From the table, the total number of correct CoTs is 241 (182 in CoderEval plus 59 in SWE-bench-NF), and the number of correct CoTs paired with failed code is 56 (CoderEval: Gemini 8 + o1 4 = 12; SWE-bench-NF: DeepSeek 14 + Gemini 16 + o1 14 = 44). The conditional probability is therefore 56/241 ≈ 23.2%, not 18.5%. No denominator reported in the paper yields 18.5%, and the same incorrect figure is repeated in the abstract, Section I, Finding 3, and Section VI.A. The companion figures (25/210 ≈ 11.9% and 25/782 ≈ 3.2%) are internally consistent, which indicates the error is specific to the flagship claim but does not diminish its severity.
  2. [Section III.D / Figure 7 / Table II] The aggregate factor percentages are not reconcilable with the benchmark-specific distributions as reported. The abstract and Section III.D state that external factors account for 53.6% and internal factors for 40.1% (totaling 93.7%). However, Figure 7a reports 63.4% external and 34.3% internal for CoderEval, and Figure 7b reports 40.1% external and 48.1% internal for SWE-bench-NF (with an additional unlabeled 'Others 11.8%' category), and no weighting is given that would produce 53.6%/40.1%. The sentence 'we investigate the distribution of these 649 factors' introduces a denominator (649) that is not connected to the 813 failed samples or to the reported percentages. Please state the exact denominator for each percentage, report the multi-label tagging statistics, and recompute the aggregate figures.
  3. [Section III.A.3 / Section III.D] The annotation of CoT quality for the 210 passing-code samples is never described. Section III.A.3 explains the taxonomy construction using 813 failed samples and reference solutions/execution results, but the extension to the passing subset is asserted without specifying who labeled those CoTs, under what instructions, or with what agreement. This matters because Finding 3's conditional probabilities are defined on the union of both subsets. Since the annotators had access to execution outcomes, the protocol also risks outcome bias: knowing that code failed (or passed) may influence the judgment of whether the CoT is 'correct,' which would bias exactly the quantities the paper claims to measure. Please describe the complete annotation protocol, ideally with blind or outcome-independent labeling, and report inter-rater agreement separately for failed and passing subsets.
  4. [Section IV.B / Table III] The claim in Finding 4 that MAD 'significantly outperforms' single-LLM detection is not backed by any significance test or confidence interval. In Table III, MAD variants improve recall substantially, but precision is often lower than single-LLM precision (e.g., MAD OGD has precision 5.49% on CoderEval and 4.85% on SWE-bench-NF, versus 6.25% and 20.91% for DeepSeek-R1), and the F1 differences are modest. The qualitative conclusion may hold, but it should be supported by appropriate statistical comparisons or stated more cautiously as an exploratory trade-off.
minor comments (5)
  1. [Abstract] The phrase 'analyzing 1,023 failed code samples' is inaccurate; 1,023 is the total number of CoT-code pairs and 813 is the number of failed-code samples.
  2. [Section III.A.3] The statement that a Cohen's Kappa of 82% indicates 'perfect agreement' is imprecise; 82% is usually interpreted as substantial or almost perfect agreement, not perfect.
  3. [Table III] The o1 precision entry appears as '0.61%' while the recall entry is '0.61'; use consistent units and avoid percent signs in the numeric cells.
  4. [Section V.A] The repair experiments use pass@1 only, but the text describes the metric as pass@k; align the terminology with the actual evaluation.
  5. [Table II] Because the percentages are given per model, the row and column totals should explicitly state the denominator (number of CoT samples or number of factors) so that the percentages are interpretable.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the paper's core findings are empirical measurements from execution results and human annotation, and its two self-citations are not load-bearing.

full rationale

Circularity review: The central claims are empirical measurements over 1,023 CoT-code pairs, with code correctness determined by benchmark test execution and CoT quality judged by human annotation with reported Cohen's Kappa scores. No equation is fitted and then used as a 'prediction'; the conditional statements in Finding 3 are direct contingency-table proportions. The taxonomy was induced by open card sorting on a random subsample of failed CoTs and then applied to the remaining samples, which is standard qualitative annotation, not a self-definitional loop. Because the annotation allows all four cells (correct CoT/failed code, incorrect CoT/passing code, correct CoT/passing code, incorrect CoT/failed code), 'CoT correct' is not defined as 'code correct,' so the headline conditional claims are not circular in construction. The paper does cite its own prior work: [31] (ClarifyGPT) is used in Section III.B and Section VI.A to support the importance of requirements quality, and [25] (FastFixer) is cited in Section VI.A for interactive program repair. These self-citations are contextual and are not load-bearing for the headline counts, the taxonomy, or the detection/repair results. No uniqueness theorem or ansatz is imported from the authors' prior work. Note for the record: Table I implies P(failed code | correct CoT) = 56/241 ≈ 23.2%, not the reported 18.5%; that is an arithmetic/reporting validity problem, not a circular-derivation problem. Score 2 reflects only the presence of minor, non-load-bearing self-citations; there is no significant circularity in the paper's derivation chain.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is empirical, so there are no fitted mathematical parameters. The central claims rest on a set of domain assumptions about ground truth, annotation reliability, and representativeness of the studied models and benchmarks.

assumptions (4)
  • domain assumption Reference code solutions and test cases are treated as ground truth for code correctness.
    Used throughout to determine pass/fail status and to guide CoT quality annotation (Section III.A.3).
  • domain assumption Human expert annotation, with inter-rater agreement measured by Cohen's Kappa, is a valid measure of CoT quality.
    The taxonomy and all percentages depend on this annotation (Section III.A.3).
  • domain assumption The three selected LLMs and two benchmark subsets are representative of current LLM code generation with CoT.
    Generalization of the findings depends on this (Section III.A.1-2 and threats to validity).
  • ad hoc to paper The taxonomy categories are exhaustive and can be applied consistently; multiple tags are allowed.
    The categories were created by the authors on a sample of 262 cases, then applied to the rest; exhaustiveness is asserted but not proven (Section III.A.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation." pith.science (2026). https://pith.science/paper/QO5OZPEO

@misc{pith2026250706980,
  author       = {Pith},
  title        = {Pith review of: Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QO5OZPEO}},
  note         = {Machine review of arXiv:2507.06980}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive performance in code generation, particularly when augmented with chain-of-thought (CoT) prompting techniques. They break down requirements into intermediate reasoning steps, which act as design rationales to guide LLMs in writing code like human programmers. Thus, the quality of these steps is crucial for ensuring the correctness and reliability of the generated code. However, little is known about the quality of CoT generated by LLMs. To what extent can we trust the thoughts generated by LLMs? How good are they? This paper empirically explores the external and internal factors of why LLMs generate unsatisfactory CoTs by analyzing 1,023 failed code samples on two widely used code generation benchmarks. We also evaluate their impact on code generation performance by analyzing 210 CoT-code pairs and refining the unsatisfied CoTs by prompting LLMs. Our study reveals three key findings: (1) External factors (53.60%), such as unclear requirements and lack of context, mainly affect CoT quality, while internal factors (40.10%) stem from LLMs' misunderstanding prompts. (2) Even when CoTs are correct, 18.5% of the generated code contains errors due to instruction-following issues; conversely, 11.90% of correct code is paired with flawed CoTs. (3) Refining low-quality CoTs is feasible, i.e., LLMs improve when given detailed problem descriptions. These findings highlight key challenges in CoT-based code generation and suggest directions for improving LLM reasoning and reliability.

Figures

Figures reproduced from arXiv: 2507.06980 by the authors.

Figure 1
Figure 1. This framework is categorized into two primary [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 1
Figure 1. Taxonomy of factors influencing CoT quality [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Example of Unclear Implementation Details [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: shows an example of this category. The purpose of this task is to convert a sequence of path fragments or patterns (provided as input to -find) into glob patterns while preserving existing patterns unchanged. For example, the input [‘foo.txt’, ‘pp:root/somedir’] should…
Figure 5
Figure 5. Figure 5: Example of Implicit Requirement Misinterpretation [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Distribution of CoT Quality Factors on CoderEval and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages

  1. [1]

    https://deepmind

    Gemini-2.0-flash-thinking-exp-01-21. https://deepmind. google/technologies/gemini/flash-thinking/

    work page

  2. [2]

    https://platform.openai.com/docs/models/o1/

    Openai. https://platform.openai.com/docs/models/o1/

    work page

  3. [3]

    Let the llms talk: Simu- lating human-to-human conversational qa via zero-shot llm-to-llm interactions

    Zahra Abbasiantaeb, Yifei Yuan, Evangelos Kanoulas, and Mohammad Aliannejadi. Let the llms talk: Simu- lating human-to-human conversational qa via zero-shot llm-to-llm interactions. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 8–17, 2024

    work page 2024

  4. [4]

    Chain-of-thought reasoning in the wild is not always faithful

    Iv ´an Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679 , 2025

    arXiv 2025

  5. [5]

    Is github’s copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering , 28(6):129, 2023

    Owura Asare, Meiyappan Nagappan, and N Asokan. Is github’s copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering , 28(6):129, 2023

    work page 2023

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christo- pher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    arXiv 2020

  7. [7]

    Codet: Code generation with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397, 2022

    arXiv 2022

  8. [8]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021

    arXiv 2021

Show all 54 references
  1. [9]

    Locagent: Graph- guided llm agents for code localization

    Zhaoling Chen, Xiangru Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Ar- man Cohan, and Xingyao Wang. Locagent: Graph- guided llm agents for code localization. arXiv preprint arXiv:2503.09089, 2025

    2025 arXiv

  2. [10]

    Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration

    Siqi Gu, Chunrong Fang, Quanjun Zhang, Fangyuan Tian, and Zhenyu Chen. Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration. arXiv preprint arXiv:2408.03095 , 2024

    2024 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  4. [12]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023

    2023 arXiv

  5. [13]

    Codecot: Tackling code syntax errors in cot reason- ing for code generation, 2024

    Dong Huang, Qingwen Bu, Yuhao Qing, and Heming Cui. Codecot: Tackling code syntax errors in cot reason- ing for code generation, 2024. URL https://arxiv.org/abs/ 2308.08784

    2024 arXiv

  6. [14]

    Adaptive mixtures of local experts

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79–87, 1991

    1991

  7. [15]

    Devanbu, and Emily Morgan

    Kevin Jesse, Toufique Ahmed, Premkumar T. Devanbu, and Emily Morgan. Large language models and simple, stupid bugs. In 2023 IEEE/ACM 20th International Con- ference on Mining Software Repositories (MSR) , pages 563–575, 2023. doi: 10.1109/MSR59073.2023.00082

    2023

  8. [16]

    Self- planning code generation with large language models,

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self- planning code generation with large language models,

  9. [17]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023

    2023 arXiv

  10. [18]

    Grace: Discriminator- guided chain-of-thought reasoning

    Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Grace: Discriminator- guided chain-of-thought reasoning. arXiv preprint arXiv:2305.14934, 2023

    2023

  11. [19]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199–22213, 2022

    2022

  12. [20]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022. URL https://arxiv.org/abs/ 2207.01780

    2022 arXiv

  13. [21]

    Codechain: Towards JOURNAL OF LATEX CLASS FILES, VOL

    Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. Codechain: Towards JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 13 modular code generation through chain of self-revisions with representative sub-modules, 2024. URL https: //arxiv.or...

    2020 arXiv

  14. [22]

    Structured chain-of-thought prompting for code generation

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting for code generation. ACM Transactions on Software Engineering and Methodology , 2023

    2023

  15. [23]

    Codetree: Agent-guided tree search for code generation with large language models

    Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Codetree: Agent-guided tree search for code generation with large language models. arXiv preprint arXiv:2411.04329 , 2024

    2024 arXiv

  16. [24]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023

    2023 arXiv

  17. [25]

    Fastfixer: An efficient and effective approach for repair- ing programming assignments

    Fang Liu, Zhenwei Liu, Qianhui Zhao, Jing Jiang, Li Zhang, Zian Sun, Ge Li, Zhongqi Li, and Yuchi Ma. Fastfixer: An efficient and effective approach for repair- ing programming assignments. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engi...

    2024

  18. [26]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Ling- ming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023. URL https://arxiv.org/abs/2305. 01210

    2023

  19. [27]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Ling- ming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  20. [28]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology , 33(5):1–26, 2024

    2024

  21. [29]

    No need to lift a finger anymore? assessing the quality of code generation by chatgpt, 2024

    Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift a finger anymore? assessing the quality of code generation by chatgpt, 2024. URL https://arxiv.org/abs/2308.04838

    2024 arXiv

  22. [30]

    Bridging code semantic and llms: Semantic chain-of- thought prompting for code generation

    Yingwei Ma, Yue Yu, Shanshan Li, Yu Jiang, Yong Guo, Yuanliang Zhang, Yutao Xie, and Xiangke Liao. Bridging code semantic and llms: Semantic chain-of- thought prompting for code generation. arXiv preprint arXiv:2310.10698, 2023

    arXiv

  23. [31]

    Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification. Proc. ACM Softw. Eng. , 1(FSE), jul 2024. doi: 10.1145/3660810. URL https://doi....

    2024 doi

  24. [32]

    Codegen: An open large language model for code with multi-turn program synthesis, 2023

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis, 2023. URL https://arxiv.org/abs/2203.13474

    2023 arXiv

  25. [33]

    Mutual reasoning makes smaller llms stronger problem-solvers

    Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195, 2024

    2024 arXiv

  26. [34]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozi `ere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, J ´er´emy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan ...

    2024 arXiv

  27. [35]

    Amazon codewhisperer, 2024

    Amazon Web Services. Amazon codewhisperer, 2024. URL https://aws.amazon.com/codewhisperer. Accessed: 2024-09-10

    2024

  28. [36]

    Quality and trust in llm- generated code

    Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Susmit Jha, Prem De- vanbu, and Toufique Ahmed. Quality and trust in llm- generated code. arXiv preprint arXiv:2402.02047 , 2024

    2024 arXiv

  29. [37]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch ¨arli, Sebas- tian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  30. [38]

    Desmarais, and Giuliano Antoniol

    Florian Tambon, Arghavan Moradi Dakhel, Amin Nikan- jam, Foutse Khomh, Michel C. Desmarais, and Giuliano Antoniol. Bugs in large language models generated code: An empirical study, 2024. URL https://arxiv.org/abs/ 2403.08937

    2024 arXiv

  31. [39]

    Code repair with llms gives an exploration-exploitation tradeoff

    Hao Tang, Keya Hu, Jin Zhou, Si Cheng Zhong, Wei- Long Zheng, Xujie Si, and Kevin Ellis. Code repair with llms gives an exploration-exploitation tradeoff. Advances in Neural Information Processing Systems , 37:117954– 117996, 2024

    2024

  32. [40]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu,...

    2023 arXiv

  33. [41]

    Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    2023

  34. [42]

    Expectation vs

    Priyan Vaithilingam, Tianyi Zhang, and Elena L Glass- man. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language mod- els. In Chi conference on human factors in computing systems extended abstracts , pages 1–7, 2022

    2022

  35. [43]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  36. [44]

    Using github copilot to solve simple programming problems

    Michel Wermelinger. Using github copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, pages 172–178, 2023

    2023

  37. [45]

    Agentless: Demystifying llm- based software engineering agents

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm- based software engineering agents. arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  38. [46]

    Demystifying llm-based software en- gineering agents

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm-based software en- gineering agents. Proceedings of the ACM on Software Engineering, 2(FSE):801–824, 2025

    2025

  39. [47]

    Self- evaluation guided beam search for reasoning

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self- evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems , 36:41618– 41650, 2023

    2023

  40. [48]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793 , 2024

    2024 arXiv

  41. [49]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Infor...

    2023

  42. [50]

    Framework for evalu- ating code generation ability of large language models

    Sangyeop Yeo, Yu-Seung Ma, Sang Cheol Kim, Hyungkook Jun, and Taeho Kim. Framework for evalu- ating code generation ability of large language models. ETRI Journal, 46(1):106–117, 2024

    2024

  43. [51]

    Question-analysis prompting improves llm performance in reasoning tasks

    Dharunish Yugeswardeenoo, Kevin Zhu, and Sean O’Brien. Question-analysis prompting improves llm performance in reasoning tasks. arXiv preprint arXiv:2407.03624, 2024

    2024 arXiv

  44. [52]

    Learning-based widget matching for migrating gui test cases

    Yakun Zhang, Wenjie Zhang, Dezhi Ran, Qihao Zhu, Chengfeng Dou, Dan Hao, Tao Xie, and Lu Zhang. Learning-based widget matching for migrating gui test cases. In Proceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering , volume 66 of ICSE ’24 , page 1–...

    2024

  45. [53]

    Toolchain*: Efficient action space navi- gation in large language models with a* search

    Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain*: Efficient action space navi- gation in large language models with a* search. arXiv preprint arXiv:2310.13227, 2023

    arXiv

  46. [2024]

    URL https://arxiv.org/abs/2303.06689

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.