Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hallucinated code falls into four observable failure categories, and existing benchmarks and mitigations line up with them.

desk verdict Useful survey of code hallucination work with a clean structure, but the central taxonomy is not objective enough as written—categories overlap and benchmarks don't map onto it. read the letter →

arxiv 2504.20799 v2 pith:HJ4ZYRMO submitted 2025-04-29 cs.SE cs.AI

Yunseo Lee , John Youngeun Song , Dongsun Kim , Jindae Kim , Mijung Kim , Jaechang Nam This is my paper · ORCID
classification cs.SEcs.AI
keywords codehallucinationgenerationlargelanguagemodelstaxonomybenchmarkmitigationdetectionsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that hallucinated code from large language models is best understood through what is observably wrong with the output, not through which model produced it. It proposes four primary categories: syntactic hallucinations, runtime execution hallucinations, functional correctness hallucinations, and code quality hallucinations, with subcategories covering syntax violations, incomplete generation, API misuse, invalid references, flawed logic, requirement deviation, resource mishandling, security vulnerabilities, and code smells. It then uses this taxonomy to organize seven existing benchmarks and their evaluation metrics, and to compare five mitigation strategies. A sympathetic reader would take away that code hallucination research already has a usable shape, and that the field's next steps are broader benchmarks and mitigation methods that target specific error categories.

What carries the argument

The central object is the observable-characteristic taxonomy of code hallucinations, a four-branch classification tree grounded in what is visibly wrong with generated code rather than in its causes. It does the organizing work of the paper: benchmark datasets are characterized by which hallucination categories they expose, metrics are grouped by whether they measure functional correctness, detection, classification, or mitigation, and mitigation methods are tied to the specific categories they address.

What would settle it

Collect a corpus of code outputs from multiple LLMs on real repository-level tasks, have independent annotators assign each error to exactly one of the four taxonomy categories, and measure inter-annotator agreement; if many errors fit multiple categories or none, the claim that the taxonomy objectively classifies code hallucinations would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that code hallucinations are not one undifferentiated failure mode: they can be classified by the observable characteristics of the emitted code into four primary categories, independent of model architecture or training data. Syntactic hallucinations break the language rules; runtime execution hallucinations fail during execution despite being syntactically valid; functional correctness hallucinations run but do not satisfy the intended task; and code quality hallucinations introduce resource, security, or maintainability risks. The paper further claims that existing benchmarks and mitigation strategies can be meaningfully compared within this framework, and it maps them to the taxonomy to show where coverage exists and where it is missing.

Load-bearing premise

The survey's conclusions rest on the assumption that its keyword search plus snowballing found every relevant code-hallucination study, and that the authors' manual sorting of papers into categories was consistent, with no independent confirmation of that sorting reported.

Editorial extensions

If this is right

  • If the taxonomy holds, hallucination-detection benchmarks can be compared by which of the four categories they exercise, even when their underlying tasks differ.
  • Mitigation methods can be assigned to categories: grammar-guided decoding targets syntactic hallucinations, retrieval-augmented generation targets API and context failures, and requirements clarification targets requirement deviation, suggesting that combined approaches should outperform any single one.
  • Because most surveyed benchmarks are Python, function-level datasets, reported hallucination rates primarily describe that setting and should not be assumed to transfer to repository-level or multi-language development.
  • Evaluation metrics matter per category, since pass@k alone can miss requirement deviation, security vulnerabilities, and code smells that the taxonomy treats as distinct hallucination types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is to use the taxonomy as an annotation scheme for automated program repair, where the error category could predict which repair strategy is most likely to succeed.
  • Because the categories are defined by observable code properties, static analyzers and execution monitors could in principle detect them automatically, making the taxonomy operational rather than purely descriptive.
  • The four categories are not shown to be mutually exclusive; a single snippet can be both syntactically broken and functionally wrong. Adding primary and secondary labels would make the scheme more robust, but that is an extension rather than a claim in the paper.
  • If multi-language and real-repository benchmarks are built along these categories, a testable open hypothesis emerges: hallucination types may shift across languages, with some languages showing more syntactic failures and others more API or requirement failures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This survey targets hallucinations in code generated by CodeLLMs. It proposes a taxonomy of four primary hallucination categories (Syntactic, Runtime Execution, Functional Correctness, and Code Quality) with subcategories, reviews benchmarks and evaluation metrics for code hallucination, analyzes root causes grouped into training-data, trained-model, and prompt issues, and summarizes five mitigation approaches with strengths and limitations. The paper claims that its taxonomy is based on observable error characteristics and is therefore objective, independent of model architecture or training data, and that the surveyed benchmarks and mitigations can be organized within this framework.

Significance. If the taxonomy were made internally consistent, the survey would fill a real gap: existing LLM-hallucination surveys are NLP-oriented, and code-specific surveys are scarcer. The paper usefully collects recent benchmarks (Table 1), metrics (Table 2), a cause tree (Fig. 4), and five mitigation strategies, and it explicitly positions its contribution against prior surveys. These organizational efforts are valuable. However, the central claim of an objective taxonomy is not yet supported because the primary categories overlap, the classification is language/toolchain-dependent in key subcategories, and the surveyed benchmark metrics do not instantiate the taxonomy. The paper is a literature survey with no derived equations or fitted parameters, so circularity is not a concern; the main risk is that the proposed framework is not a well-defined partition. The taxonomy and the benchmark/metric mapping need revision before the central contribution can be accepted.

major comments (4)
  1. [§5.1.2 and §5.4.3] The taxonomy is not a partition as presented. Section 5.1.2 classifies "Incomplete Code Generation" under Syntactic Hallucinations, while Section 5.4.3 lists "incomplete generation" as an example of Code Smell under Code Quality Hallucinations, citing [34, 52, 57]. The same observable output can therefore be assigned to two primary categories in Fig. 3, which contradicts the paper's claim in Section 5 that the taxonomy provides an objective classification regardless of model architecture or training data. The authors should either remove the duplicate or define an explicit decision rule, such as first checking whether the code parses, that assigns every generated output to exactly one primary category.
  2. [§5.2.1 and §5.2.2] The Runtime Execution Hallucination category is defined as code that is syntactically valid but produces runtime errors, yet two of its subcategories are language/toolchain-dependent. Section 5.2.1 lists missing imports under API Knowledge Conflict, and Section 5.2.2 lists undefined variables and invalid references under Invalid Reference Errors. In statically typed languages these are compile-time errors, whereas in Python they are runtime errors. Thus the observable characteristics are not independent of the execution environment, and no decision rule is given for choosing Runtime Execution over Syntactic or Functional Correctness in such cases. The authors need to either restrict the examples to genuine runtime failures or define the classification relative to a specific language/runtime.
  3. [§6.2.1 and Table 2] The survey's own metrics section does not validate the taxonomy. Pass@k and 10@k, as described in Section 6.2.1, treat any failure to pass all test cases as equivalent and therefore cannot distinguish any of the four primary hallucination categories. The benchmarks listed in Table 1 also use their own hallucination-type sets (e.g., CodeHaluEval and CodeMirage), and the paper does not map those types onto the proposed taxonomy. If the taxonomy is meant to enable comparison of benchmarks and mitigations, the authors should provide such a mapping or explicitly acknowledge that the current metrics and benchmarks are not taxonomy-aware.
  4. [§3.2] The literature-search methodology is not sufficiently validated to support the claim of comprehensiveness. The paper reports a Google Scholar keyword search for "hallucination" and "code generation" plus snowballing, but it does not give the exact search query, inclusion/exclusion criteria, screening decisions, or a PRISMA-style flow. It also reports that titles, abstracts, and introductions were manually categorized without any inter-rater reliability check. Given that the survey's conclusions depend on the representativeness of the included papers, this methodological gap weakens the central contribution and should be addressed.
minor comments (4)
  1. [§6.1] The text refers to "Multiple-E" where the benchmark is called MultiPL-E; this typo appears in the paragraph discussing multi-language benchmarks and should be corrected.
  2. [§6.2.1] The definition of 10@k is garbled: "measures the percentage of tasks for which k samples were created per task, and when at least 10 of them passed the test" is not readable. If the intended metric is the fraction of tasks for which at least 10 of k generated solutions pass, that should be stated clearly, or the original definition from [30] should be quoted.
  3. [§8.3] Reference [16] is cited as support for the claim that grammar-guided generation has been widely proposed, but [16] is the llama.cpp GitHub project, which does not appear to be a grammar-guided generation paper; this citation seems mismatched.
  4. [Fig. 1] Figure 1 is hard to interpret: the Venn-diagram counts are not clearly labeled, and the text's category counts do not obviously reconcile with the total number of references (51 classified papers versus 67 references overall). The authors should clarify whether all references were classified and make the figure self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey organizes external literature; the taxonomy is a synthesis of published classifications, not a reduction to its own inputs.

full rationale

This paper is a literature survey. It performs no derivations, fits no parameters, and tests no hypotheses; it categorizes existing studies on code hallucination into a taxonomy of four primary categories, reviews benchmarks and metrics, and surveys mitigation strategies. The taxonomy is constructed from externally published taxonomies (e.g., CodeHaluEval, CodeMirage, Wang et al., Tambon et al.) and is presented as an organizing framework, not as a result derived from first principles. There is no equation whose output equals its input, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain: the authors do not appear to cite their own prior work in any load-bearing role. The skeptical concern about overlapping categories (e.g., incomplete generation appearing under both Syntactic and Code Quality) is a question of classification consistency and empirical validation, not circularity, and the absence of inter-rater validation is a rigor limitation rather than a self-referential dependency. Because the survey's claims are descriptive summaries of external sources, the central content does not reduce to the paper's own assumptions, and no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is a literature survey; it introduces no fitted parameters, no new entities, and relies only on standard assumptions about literature search coverage and manual categorization.

assumptions (2)
  • domain assumption The selected literature (keyword search plus snowballing) is representative of all relevant research on code hallucinations.
    The survey relies on this to make comprehensive claims about taxonomy, benchmarks, and causes; no formal guarantees of coverage are provided (Section 3.2).
  • domain assumption Manual categorization of papers into taxonomy, benchmarking, and mitigation is reliable enough for the reported statistics.
    The authors state titles, abstracts, and introductions were manually reviewed; no inter-rater reliability or validation is reported (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges." pith.science (2026). https://pith.science/paper/HJ4ZYRMO

@misc{pith2026250420799,
  author       = {Pith},
  title        = {Pith review of: Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJ4ZYRMO}},
  note         = {Machine review of arXiv:2504.20799}
}
read the original abstract

Recent technical breakthroughs in large language models (LLMs) have enabled them to fluently generate source code. Software developers often leverage both general-purpose and code-specialized LLMs to revise existing code or even generate a whole function from scratch. These capabilities are also beneficial in no-code or low-code contexts, in which one can write programs without a technical background. However, due to their internal design, LLMs are prone to generating hallucinations, which are incorrect, nonsensical, and not justifiable information but difficult to identify its presence. This problem also occurs when generating source code. Once hallucinated code is produced, it is often challenging for users to identify and fix it, especially when such hallucinations can be identified under specific execution paths. As a result, the hallucinated code may remain unnoticed within the codebase. This survey investigates recent studies and techniques relevant to hallucinations generated by CodeLLMs. We categorize the types of hallucinations in the code generated by CodeLLMs, review existing benchmarks and mitigation strategies, and identify open challenges. Based on these findings, this survey outlines further research directions in the detection and removal of hallucinations produced by CodeLLMs.

Figures

Figures reproduced from arXiv: 2504.20799 by the authors.

Figure 2
Figure 2. Distribution of papers by venue. We categorized the papers into three key dimensions: Taxonomy, Benchmarking, and Mitigation, as shown [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Taxonomy of hallucinations possibly generated by CodeLLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Potential causes of hallucinations by CodeLLMs. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability

    cs.SE 2026-05 conditional novelty 6.0 of 10

    Proposes guidance for responsible AI use in scientific software development under NQA-1 standards, illustrated with TMAP8 V&V cases to ensure accountability and auditability.

  2. A comprehensive taxonomy of hallucinations in Large Language Models

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.

  3. Position Paper: Programming Language Techniques for Bridging LLM Code Generation Semantic Gaps

    cs.SE 2025-07 unverdicted novelty 2.0 of 10

    A position paper arguing that PL techniques, especially formal verification and structure-aware representations, should be deeply integrated into LLM code generation.

Reference graph

Works this paper leans on

68 extracted references · 21 canonical work pages · cited by 3 Pith papers

  1. [1]

    Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. 2024. CodeMirage: Hallucinations in Code Generated by Large Language Models. doi:10.48550/arXiv.2408.08333 arXiv:2408.08333

    arXiv doi:10.48550/arxiv.2408.08333 2024

  2. [2]

    Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu Lahiri, and Sriram Rajamani. 2024. Monitor-guided decoding of code LMs with static analysis of repository context. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  3. [3]

    Miltiadis Allamanis, Sheena Panthaplackel, and Pengcheng Yin. 2024. Unsupervised Evaluation of Code LLMs with Round-Trip Correctness. arXiv preprint arXiv:2402.08699 (2024)

    arXiv 2024

  4. [4]

    Amazon. 2022. What is CodeWhisperer? https://docs.aws.amazon.com/codewhisperer/latest/userguide/what-is-cwspr.html

    work page 2022

  5. [5]

    Anthropic. 2025. Claude 3.7 Sonnet. https://www.anthropic.com/news/claude-3-7-sonnet

    work page 2025

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  7. [7]

    Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. 2023. Purple llama cyberseceval: A secure coding benchmark for language models.arXiv preprint arXiv:2312.04724 (2023)

    arXiv 2023

  8. [8]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Transactions on Software Engineering 49, 7 (Jul...

    arXiv 2023

Show all 68 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  2. [10]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, Yan Liu, Enyu Zhou, Ming Zhang, Yuhao Zhou, Yueming Wu, Rui Zheng, Ming Wen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Xipeng Qiu, Qi Z...

    2024 doi

  3. [11]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861 (2023)

    2023 arXiv

  4. [12]

    Aryaz Eghbali and Michael Pradel. 2024. De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding. doi:10.48550/arXiv.2401.01701 arXiv:2401.01701

    2024 doi arXiv

  5. [13]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering 50, 9 (Sept. 2024), 2254–2268. doi:10.1109/TSE.202...

    2024 doi

  6. [14]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineeri...

    2023

  7. [15]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated Repair of Programs from Large Language Models. In Proceedings of the 45th International Conference on Software Engineering (ICSE ’23) . IEEE Press, Melbourne, Victoria, Australia, 1469...

    2023

  8. [16]

    Georgi Gerganov et al. 2024. llama.cpp: Port of Facebook’s LLaMA model in C/C++. https://github.com/guidance-ai/guidance

    2024

  9. [17]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  10. [18]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8 (Dec. 2024), 220:1–220:79. doi:10.114...

    2024 doi

  11. [19]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. 43, 2, Article 42...

    2025 doi

  12. [20]

    Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, and Varun Kumar. 2024. On Mitigating Code LLM Hallucinations with API Documentation. arXiv preprint arXiv:2407.09726 (2024)

    2024 arXiv

  13. [21]

    Kevin Jesse, Toufique Ahmed, Premkumar T Devanbu, and Emily Morgan. 2023. Large language models and simple, stupid bugs. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 563–575

    2023

  14. [22]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515 (2024)

    2024 arXiv

  15. [23]

    Nan Jiang, Qi Li, Lin Tan, and Tianyi Zhang. 2024. Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code. arXiv preprint arXiv:2410.09997 (2024)

    2024 arXiv

  16. [24]

    Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt?. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) . IEEE, 2445–2451

    2023

  17. [25]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning . PMLR, 18319–18345

    2023

  18. [26]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 21314–21328

    2022

  19. [27]

    Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv preprint arXiv:2404.00599 (2024)

    2024 arXiv

  20. [28]

    Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, and Jing Ma. 2024. MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 736–783

    2024

  21. [29]

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852 (2023)

    2023 arXiv

  22. [30]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

    2022

  23. [31]

    Yifan Li, Ensheng Shi, Dewu Zheng, Kefeng Duan, Jiachi Chen, and Yanlin Wang. 2024. RepoMinCoder: Improving Repository-Level Code Generation Based on Information Loss Screening. In Proceedings of the 15th Asia-Pacific Symposium on Internetware . 229–238

    2024

  24. [32]

    Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li. 2024. A 3-CodGen: A Repository- Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-Aware.IEEE Transactions on Software ...

    2024

  25. [33]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. doi:10.48550/arXiv.2109.07958 arXiv:2109.07958 [cs]

    2022 doi arXiv

  26. [34]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and Evaluating Hallucinations in LLM-Powered Code Generation. doi:10.48550/arXiv.2404.00971 arXiv:2404.00971

    2024 doi

  27. [35]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Proceedings of the 37th International Conference on Neural Information Processing System...

    2024

  28. [36]

    Mingwei Liu, Tianyong Yang, Yiling Lou, Xueying Du, Ying Wang, and Xin Peng. 2023. Codegen4libs: A two-stage approach for library-oriented code generation. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 434–445

    2023

  29. [37]

    Le, and David Lo

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. 2024. Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues. ACM Trans. Softw. Eng. Methodol. 33, 5 (June 2024), 116:1–116:26. doi:10.1145/3643674

    2024 doi

  30. [38]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al

  31. [39]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification. Proc. ACM Softw. Eng. 1, FSE (July 2024), 103:2332–103:2354. doi:10.1145/ 3660810

    2024

  32. [40]

    Wang, and Xi Victoria Lin

    Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023. LEVER: Learning to Verify Language-to- Code Generation with Execution. In Proceedings of the 40th International Conference on Machine Learning (ICML’23, Vol. 202) . JMLR....

    2023

  33. [41]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. doi:10.48550/arXiv.2203.13474 arXiv:2203.13474 [cs]

    2023 doi arXiv

  34. [42]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 754–768. Hallucination by Code Generati...

    2022

  35. [43]

    Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201.11227 (2022)

    2022 arXiv

  36. [44]

    Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. 2024. Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation. arXiv preprint arXiv:2405.15307 (2024)

    2024 arXiv

  37. [45]

    Kia Rahmani, Mohammad Raza, Sumit Gulwani, Vu Le, Daniel Morris, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. 2021. Multi-modal program inference: A marriage of pre-trained language models and component-based synthesis. Proceedings of the ACM on Programming Languages...

    2021

  38. [46]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 doi arXiv

  39. [47]

    Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. arXiv preprint arXiv:2109.05093 (2021)

    2021 arXiv

  40. [48]

    Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. InInternational Conference on Machine Learning . PMLR, 31693–31715

    2023

  41. [49]

    Mohammed Latif Siddiq, Joanna Cecilia da Silva Santos, Sajith Devareddy, and Anna Muller. 2024. Sallm: Security assessment of generated code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops . 54–65

    2024

  42. [50]

    Mohammed Latif Siddiq and Joanna CS Santos. 2022. SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Se...

    2022

  43. [51]

    Joseph Spracklen, Raveen Wijewickrama, AHM Sakib, Anindya Maiti, Bimal Viswanath, and Murtuza Jadliwala. 2024. We have a package for you! a comprehensive analysis of package hallucinations by code generating llms. arXiv preprint arXiv:2406.10279 (2024)

    2024 arXiv

  44. [52]

    Desmarais, and Giuliano Antoniol

    Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Giuliano Antoniol. 2024. Bugs in Large Language Models Generated Code: An Empirical Study. doi:10.48550/arXiv.2403.08937 arXiv:2403.08937

    2024 doi arXiv

  45. [53]

    Yuchen Tian, Weixiang Yan, Qian Yang, Xuandong Zhao, Qian Chen, Wen Wang, Ziyang Luo, Lei Ma, and Dawn Song. 2024. CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification. doi:10.48550/arXiv.2405.00253 arXiv:2405.00253

    2024 doi arXiv

  46. [54]

    Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2024. SynCode: LLM Generation with Grammar Augmentation. doi:10.48550/arXiv.2403.01632 arXiv:2403.01632

    2024 doi arXiv

  47. [55]

    Chong Wang, Jian Zhang, Yebo Feng, Tianlin Li, Weisong Sun, Yang Liu, and Xin Peng. 2024. Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation. arXiv preprint arXiv:2401.06391 (2024)

    2024 arXiv

  48. [56]

    Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. RLCoder: Reinforcement Learning for Repository-Level Code Completion. arXiv:2407.19487 [cs.SE] https://arxiv.org/abs/2407.19487

    2024 arXiv

  49. [57]

    Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. 2024. Where Do Large Language Models Fail When Generating Code? arXiv preprint arXiv:2406.08731 (2024)

    2024 arXiv

  50. [58]

    Brandon T Willard and Rémi Louf. 2023. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702 (2023)

    2023 arXiv

  51. [59]

    Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE ’14) . Association for Computing Machinery,...

    2014

  52. [60]

    Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, et al . 2023. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. arXi...

    2023 arXiv

  53. [61]

    Chen Yang, Yan Liu, and Changqing Yin. 2021. Recent Advances in Intelligent Source Code Generation: A Survey on Natural Language Based Studies. Entropy 23, 9 (Sept. 2021), 1174. doi:10.3390/e23091174

    2021 doi

  54. [62]

    Daoguang Zan, Bei Chen, Yongshun Gong, Junzhi Cao, Fengji Zhang, Bingchao Wu, Bei Guan, Yilong Yin, and Yongji Wang. 2023. Private-library- oriented code generation with large language models. arXiv preprint arXiv:2307.15370 (2023)

    2023 arXiv

  55. [63]

    Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023. Toolcoder: Teach code generation models to use api search tools. arXiv preprint arXiv:2305.04032 (2023)

    2023 arXiv

  56. [64]

    Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2024. Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code. doi:10.48550/arXiv.2311.07989 arXiv:2311.07989 [cs]

    2024 doi arXiv

  57. [65]

    Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, and Zibin Zheng. 2024. LLM hallucinations in practical code generation: Phenomena, mechanism, and mitigation. arXiv preprint arXiv:2409.20550 (2024)

    2024 arXiv

  58. [66]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al . 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowl...

    2023

  59. [67]

    Li Zhong and Zilong Wang. 2024. Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation. Proceedings of the AAAI Conference on Artificial Intelligence 38, 19 (March 2024), 21841–21849. doi:10.1609/aaai.v38i19.30185

    2024 doi

  60. [2021]

    arXiv preprint arXiv:2102.04664 (2021)

    Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)

    2021 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.