Pith. sign in

REVIEW 10 cited by

CodeMirage: Hallucinations in Code Generated by Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.08333 v2 pith:WUKZ2IMV submitted 2024-08-14 cs.SE cs.AIcs.CL

Vibhor Agarwal , Yulong Pei , Salwa Alamir , Xiaomo Liu This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords codehallucinationsgenerationllmsgeneratedhallucinationdatasetmodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) have shown promising potentials in program generation and no-code automation. However, LLMs are prone to generate hallucinations, i.e., they generate text which sounds plausible but is incorrect. Although there has been a recent surge in research on LLM hallucinations for text generation, similar hallucination phenomenon can happen in code generation. Sometimes the generated code can have syntactical or logical errors as well as more advanced issues like security vulnerabilities, memory leaks, etc. Given the wide adaptation of LLMs to enhance efficiency in code generation and development in general, it becomes imperative to investigate hallucinations in code generation. To the best of our knowledge, this is the first attempt at studying hallucinations in the code generated by LLMs. We start by introducing the code hallucination definition and a comprehensive taxonomy of code hallucination types. We propose the first benchmark CodeMirage dataset for code hallucinations. The benchmark contains 1,137 GPT-3.5 generated hallucinated code snippets for Python programming problems from two base datasets - HumanEval and MBPP. We then propose the methodology for code hallucination detection and experiment with open source LLMs such as CodeLLaMA as well as OpenAI's GPT-3.5 and GPT-4 models using one-shot prompt. We find that GPT-4 performs the best on HumanEval dataset and gives comparable results to the fine-tuned CodeBERT baseline on MBPP dataset. Towards the end, we discuss various mitigation strategies for code hallucinations and conclude our work.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Code-MUE measures Code LLM uncertainty via execution-based Semantic Interaction Graphs and Von Neumann entropy, correlating strongly with functional correctness on eight models.

  2. MemoCoder: Automated Function Synthesis using LLM-Supported Agents

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A multi-agent LLM framework with a memory of past code fixes (MemoCoder) improves Pass@10 and Pass@50 on MBPP, HumanEval, and LiveCodeBench over zero-shot and self-repair baselines by 3 to 14 percentage points.

  3. SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

    cs.SE 2025-07 conditional novelty 6.0 of 10

    SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.

  4. GraphAlignCoder: Aligning Program and Proof Graphs for Code Generation

    cs.SE 2026-08 conditional novelty 5.0 of 10

    GraphAlignCoder fine-tunes a code LLM with graph-aligned, teacher-generated Lean proof-flow supervision and reports improved Pass@1 on LiveCodeBench v6 and BigCodeBench, but the proof obligations are not derived from ...

  5. Securing LLM-Generated Embedded Firmware through AI Agent-Driven Validation and Patching

    cs.CR 2025-09 reject novelty 5.0 of 10

    An iterative fuzz-test, static-analysis, and LLM-patch loop with three AI agents is claimed to raise firmware vulnerability remediation from 67.3% to 92.4% in QEMU-run FreeRTOS tasks.

  6. Conversation Kernels: A Flexible Mechanism to Learn Relevant Context for Online Conversation Understanding

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Conversation Kernels retrieve small tree-neighborhood windows around a post and feed them to RoBERTa, improving Slashdot comment label prediction over standard text-only baselines.

  7. Chain-of-Programming (CoP) : Empowering Large Language Models for Geospatial Code Generation

    cs.SE 2024-11 conditional novelty 5.0 of 10

    A structured prompting pipeline with five stages, a shared information pool, retrieval from geospatial knowledge bases, and iterative debugging improves LLM-generated geospatial code on a self-built benchmark.

  8. Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

    cs.SE 2025-04 conditional novelty 3.0 of 10

    This survey organizes code-LLM hallucinations into four types, reviews 52 related papers, and identifies open challenges in detection and mitigation.

  9. A comprehensive taxonomy of hallucinations in Large Language Models

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.

  10. SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment

    cs.SE 2025-01 reject novelty 2.0 of 10

    A survey-based review concluding that AI coding assistants introduce security vulnerabilities, hallucinated code, and data leak risks requiring developer vigilance.

Pith tools