Pith. sign in

REVIEW 7 cited by

LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.20550 v2 pith:ZZKHSLO4 submitted 2024-09-30 cs.SE cs.AIcs.CL

LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation

Ziyao Zhang , Yanlin Wang , Chong Wang , Jiachi Chen , Zibin Zheng This is my paper
classification cs.SE cs.AIcs.CL
keywords codegenerationhallucinationsllmsdevelopmentmitigationpracticalresults
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency. Recent large language models (LLMs) based approaches have shown promising results and revolutionized code generation task. Despite the promising performance, LLMs often generate contents with hallucinations, especially for the code generation scenario requiring the handling of complex contextual dependencies in practical development process. Although previous study has analyzed hallucinations in LLM-powered code generation, the study is limited to standalone function generation. In this paper, we conduct an empirical study to study the phenomena, mechanism, and mitigation of LLM hallucinations within more practical and complex development contexts in repository-level generation scenario. First, we manually examine the code generation results from six mainstream LLMs to establish a hallucination taxonomy of LLM-generated code. Next, we elaborate on the phenomenon of hallucinations, analyze their distribution across different models. We then analyze causes of hallucinations and identify four potential factors contributing to hallucinations. Finally, we propose an RAG-based mitigation method, which demonstrates consistent effectiveness in all studied LLMs. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/LLMCodingHallucination

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

    cs.SE 2026-06 unverdicted novelty 7.0

    LibEvoBench benchmark shows LLMs are version-oblivious on evolving APIs, with documentation helping but version specification not.

  2. Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries

    cs.SE 2025-09 unverdicted novelty 7.0

    A study of seven LLMs finds that realistic prompt variations such as one-character misspellings trigger library hallucinations in up to 26% of cases, fabricated names in up to 99%, and time-based prompts in up to 85%,...

  3. EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention

    cs.SE 2025-08 unverdicted novelty 7.0

    EyeMulator augments CodeLLM fine-tuning loss with token weights derived from human eye-tracking scan paths, producing large gains on code translation and summarization across StarCoder, Llama-3.2 and DeepSeek-Coder.

  4. Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability

    cs.SE 2026-05 conditional novelty 6.0

    A repository-level AGENTS.md governance framework with commit-level AI disclosure, session logging, pre-commit gates, and mandatory human review is demonstrated on two new TMAP8 validation cases.

  5. Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code

    cs.SE 2026-05 accept novelty 6.0

    A review of 114 studies creates taxonomies for code and data quality issues, formalizes 18 propagation mechanisms from training data defects to LLM-generated code defects, and synthesizes detection and mitigation techniques.

  6. Securing LLM-Generated Embedded Firmware through AI Agent-Driven Validation and Patching

    cs.CR 2025-09 reject novelty 5.0

    An iterative fuzz-test, static-analysis, and LLM-patch loop with three AI agents is claimed to raise firmware vulnerability remediation from 67.3% to 92.4% in QEMU-run FreeRTOS tasks.

  7. Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability

    cs.SE 2026-05 conditional novelty 4.0

    Proposes guidance for responsible AI use in scientific software development under NQA-1 standards, illustrated with TMAP8 V&V cases to ensure accountability and auditability.