{"id":"91cff7ea-6158-402c-becb-c8389ea225a6","arxiv_id":"2504.20799","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This survey organizes code-LLM hallucinations into four types, reviews 52 related papers, and identifies open challenges in detection and mitigation.","lead":"This survey sorts the many ways AI code generators produce wrong or unhelpful code into four categories and reviews the benchmarks, causes, and fixes that researchers have proposed. It is a useful map for engineers and researchers who want to know what can go wrong when an LLM writes code and what is being done about it.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Taxonomy's 'objective' claim is undercut by overlapping primary categories (e.g., incomplete generation appears in both Syntactic and Code Quality) and by benchmarks/metrics that do not map onto the four classes.","rationale":"The reader's weakest assumption was about search comprehensiveness and manual categorization consistency. My concern overlaps with the categorization-consistency part but is more specific and more load-bearing: the taxonomy itself is not a disjoint, decision-procedure-ready classification, regardless of how complete the literature search is. Because the survey's stated unique contribution is an objective taxonomy that makes benchmarks and mitigations comparable, this internal overlap and the lack of mapping from benchmarks/metrics to the taxonomy directly threaten the strongest claim. I do not think the paper should be rejected outright: the survey is readable, synthesizes a useful body of work, and its limitations are fixable. However, the objective-classification claim should be softened or the taxonomy revised. Since the reader already returned CONDITIONAL, my analysis leaves that verdict unchanged while supplying a sharper, testable basis for the condition.","tokens_in":15965,"tokens_out":5133,"duration_ms":53183,"concrete_test":"Take the 1,137 CodeMirage and 699 CodeHaluEval samples from Table 1. Have two independent annotators assign each sample to exactly one primary category in Fig. 3 using only the Section 5 definitions, then compute Cohen's kappa. Separately count samples that satisfy the definitions of both §5.1.2 and §5.4.3 (incomplete generation). If kappa is below 0.8 or dual assignment occurs in more than a trivial fraction of samples, the 'objective' classification claim fails and the taxonomy must be revised before benchmarks can be compared within it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (§5, Fig. 3) is that its four-category taxonomy classifies code hallucinations objectively, independent of model architecture or training data, so that benchmarks and mitigations can be compared within it. The load-bearing condition for that claim is that the categories are mutually exclusive and that assignment is unambiguous. That condition is not met. First, §5.1.2 places 'Incomplete Code Generation' under Syntactic Hallucinations because it prevents compilation/execution, while §5.4.3 lists 'incomplete generation' as a Code Smell under Code Quality Hallucinations (citing [34, 52, 57]). The same observable output can therefore be assigned to two primary categories, so Fig. 3 is not a partition. Second, §5.2.1 'API Knowledge Conflict' and §5.2.2 'Invalid Reference Errors' are labeled Runtime Execution Hallucinations, but in statically typed languages missing imports, undefined variables, and bad references are compile-time failures; in Python they are runtime errors. The 'observable characteristics' therefore depend on the language and toolchain, not merely on the output. No decision rule is given for choosing Runtime Execution over Syntactic or Functional Correctness. Third, the survey's own benchmark/metrics section does not instantiate the taxonomy: Table 2's Pass@k and 10@k collapse every failed test into one correctness class and cannot distinguish any of the four categories. CodeHaluEval and CodeMirage use their own hallucination-type sets, and the paper never shows these sets align with Fig. 3. Hence the claim that existing benchmarks and mitigations can be meaningfully compared within the taxonomy is unsupported. The §3.2 paper counts also conflict with the overlap lists, reinforcing that the manual categorization lacks validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey targets hallucinations in code generated by CodeLLMs. It proposes a taxonomy of four primary hallucination categories (Syntactic, Runtime Execution, Functional Correctness, and Code Quality) with subcategories, reviews benchmarks and evaluation metrics for code hallucination, analyzes root causes grouped into training-data, trained-model, and prompt issues, and summarizes five mitigation approaches with strengths and limitations. The paper claims that its taxonomy is based on observable error characteristics and is therefore objective, independent of model architecture or training data, and that the surveyed benchmarks and mitigations can be organized within this framework.","tokens_in":16385,"tokens_out":4774,"duration_ms":50600,"significance":"If the taxonomy were made internally consistent, the survey would fill a real gap: existing LLM-hallucination surveys are NLP-oriented, and code-specific surveys are scarcer. The paper usefully collects recent benchmarks (Table 1), metrics (Table 2), a cause tree (Fig. 4), and five mitigation strategies, and it explicitly positions its contribution against prior surveys. These organizational efforts are valuable. However, the central claim of an objective taxonomy is not yet supported because the primary categories overlap, the classification is language/toolchain-dependent in key subcategories, and the surveyed benchmark metrics do not instantiate the taxonomy. The paper is a literature survey with no derived equations or fitted parameters, so circularity is not a concern; the main risk is that the proposed framework is not a well-defined partition. The taxonomy and the benchmark/metric mapping need revision before the central contribution can be accepted.","major_comments":[{"comment":"The taxonomy is not a partition as presented. Section 5.1.2 classifies \"Incomplete Code Generation\" under Syntactic Hallucinations, while Section 5.4.3 lists \"incomplete generation\" as an example of Code Smell under Code Quality Hallucinations, citing [34, 52, 57]. The same observable output can therefore be assigned to two primary categories in Fig. 3, which contradicts the paper's claim in Section 5 that the taxonomy provides an objective classification regardless of model architecture or training data. The authors should either remove the duplicate or define an explicit decision rule, such as first checking whether the code parses, that assigns every generated output to exactly one primary category.","section":"§5.1.2 and §5.4.3"},{"comment":"The Runtime Execution Hallucination category is defined as code that is syntactically valid but produces runtime errors, yet two of its subcategories are language/toolchain-dependent. Section 5.2.1 lists missing imports under API Knowledge Conflict, and Section 5.2.2 lists undefined variables and invalid references under Invalid Reference Errors. In statically typed languages these are compile-time errors, whereas in Python they are runtime errors. Thus the observable characteristics are not independent of the execution environment, and no decision rule is given for choosing Runtime Execution over Syntactic or Functional Correctness in such cases. The authors need to either restrict the examples to genuine runtime failures or define the classification relative to a specific language/runtime.","section":"§5.2.1 and §5.2.2"},{"comment":"The survey's own metrics section does not validate the taxonomy. Pass@k and 10@k, as described in Section 6.2.1, treat any failure to pass all test cases as equivalent and therefore cannot distinguish any of the four primary hallucination categories. The benchmarks listed in Table 1 also use their own hallucination-type sets (e.g., CodeHaluEval and CodeMirage), and the paper does not map those types onto the proposed taxonomy. If the taxonomy is meant to enable comparison of benchmarks and mitigations, the authors should provide such a mapping or explicitly acknowledge that the current metrics and benchmarks are not taxonomy-aware.","section":"§6.2.1 and Table 2"},{"comment":"The literature-search methodology is not sufficiently validated to support the claim of comprehensiveness. The paper reports a Google Scholar keyword search for \"hallucination\" and \"code generation\" plus snowballing, but it does not give the exact search query, inclusion/exclusion criteria, screening decisions, or a PRISMA-style flow. It also reports that titles, abstracts, and introductions were manually categorized without any inter-rater reliability check. Given that the survey's conclusions depend on the representativeness of the included papers, this methodological gap weakens the central contribution and should be addressed.","section":"§3.2"}],"minor_comments":[{"comment":"The text refers to \"Multiple-E\" where the benchmark is called MultiPL-E; this typo appears in the paragraph discussing multi-language benchmarks and should be corrected.","section":"§6.1"},{"comment":"The definition of 10@k is garbled: \"measures the percentage of tasks for which k samples were created per task, and when at least 10 of them passed the test\" is not readable. If the intended metric is the fraction of tasks for which at least 10 of k generated solutions pass, that should be stated clearly, or the original definition from [30] should be quoted.","section":"§6.2.1"},{"comment":"Reference [16] is cited as support for the claim that grammar-guided generation has been widely proposed, but [16] is the llama.cpp GitHub project, which does not appear to be a grammar-guided generation paper; this citation seems mismatched.","section":"§8.3"},{"comment":"Figure 1 is hard to interpret: the Venn-diagram counts are not clearly labeled, and the text's category counts do not obviously reconcile with the total number of references (51 classified papers versus 67 references overall). The authors should clarify whether all references were classified and make the figure self-contained.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful survey, but its central contribution—the taxonomy—needs substantial revision to become a well-defined classification scheme. The issues in Sections 5 and 6 are load-bearing, not cosmetic. If the authors can resolve the category overlap, add decision rules for language-dependent cases, and map the surveyed benchmarks/metrics onto the taxonomy, the paper could be acceptable. I see no reason to suspect bad faith or circularity; the problem is internal consistency and methodological transparency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a solid, well-organized survey of code hallucination research and it will be a handy reference for the subfield. But the paper's headline claim—that its four-category taxonomy is objective and lets benchmarks and mitigations be compared within it—does not hold up as written. The categories are not mutually exclusive, and the benchmark/metrics section doesn't actually map onto them.\n\nThe genuinely new thing is synthesis: pulling together recent taxonomies (CodeHaluEval, CodeMirage, HalluCode, LMDefects, and a few others) into a coherent four-part structure, plus a compact comparative table of benchmarks and a review of mitigation strategies. That is useful. The writing is clear, the scope is sensible, and the cause and mitigation sections give a newcomer a good map.\n\nSoft spots, in order of seriousness. First, overlap across the primary categories. 'Incomplete code generation' appears under Syntactic Hallucinations (section 5.1.2) and again as a Code Smell under Code Quality (section 5.4.3). Since the same observable output can be filed under two different primary categories, Fig. 3 is not a partition, and the 'objective classification' claim in section 5 is too strong. Second, some subcategories are language- or toolchain-dependent. API Knowledge Conflict and Invalid Reference Errors are called Runtime Execution Hallucinations, but in C++/Java a missing import or undefined variable is a compile-time error; it only becomes a runtime failure in Python or similar. The taxonomy needs a decision rule that holds across languages. Third, the benchmarks and metrics review is not connected back to the taxonomy. Pass@k and 10@k collapse all failures into one correctness class; CodeHaluEval and CodeMirage use their own hallucination-type sets. The paper never shows how those map to Fig. 3, so the claim that existing benchmarks can be meaningfully compared within the framework is unsupported. Fourth, the literature search is a Google Scholar keyword search plus snowballing with no inter-rater validation or protocol details, so 'comprehensive' is an overstatement. Minor issues include the garbled description of 10@k in section 6.2.1 and the 'Multiple-E' typo.\n\nFor whom: anyone new to code hallucination who wants a fast map of benchmarks, taxonomies, and mitigations. It is a survey, not a research result, so the value is organizational.\n\nRecommendation: deserves peer review, not desk reject, because the area is real and the synthesis is useful. But it needs major revision: either weaken the objectivity claim substantially or make the categories disjoint and demonstrate the benchmark mapping. I'd be willing to read a revised version.","headline":"Useful survey of code hallucination work with a clean structure, but the central taxonomy is not objective enough as written—categories overlap and benchmarks don't map onto it.","tokens_in":16833,"tokens_out":2740,"would_cite":true,"duration_ms":26905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hallucinated code falls into four observable failure categories, and existing benchmarks and mitigations line up with them.","keywords":["code hallucination","code generation","large language models","taxonomy","benchmark","mitigation","hallucination detection","software engineering"],"falsifier":"Collect a corpus of code outputs from multiple LLMs on real repository-level tasks, have independent annotators assign each error to exactly one of the four taxonomy categories, and measure inter-annotator agreement; if many errors fit multiple categories or none, the claim that the taxonomy objectively classifies code hallucinations would fail.","tokens_in":15797,"feed_emoji":"💻","tokens_out":4961,"duration_ms":52663,"temperature":0.7,"pith_summary":"This paper argues that hallucinated code from large language models is best understood through what is observably wrong with the output, not through which model produced it. It proposes four primary categories: syntactic hallucinations, runtime execution hallucinations, functional correctness hallucinations, and code quality hallucinations, with subcategories covering syntax violations, incomplete generation, API misuse, invalid references, flawed logic, requirement deviation, resource mishandling, security vulnerabilities, and code smells. It then uses this taxonomy to organize seven existing benchmarks and their evaluation metrics, and to compare five mitigation strategies. A sympathetic reader would take away that code hallucination research already has a usable shape, and that the field's next steps are broader benchmarks and mitigation methods that target specific error categories.","feed_headline":"Four categories sort code hallucinations","feed_subtitle":"A survey maps benchmarks and fixes for syntax, runtime, functional, and quality failures in LLM code.","key_machinery":"The central object is the observable-characteristic taxonomy of code hallucinations, a four-branch classification tree grounded in what is visibly wrong with generated code rather than in its causes. It does the organizing work of the paper: benchmark datasets are characterized by which hallucination categories they expose, metrics are grouped by whether they measure functional correctness, detection, classification, or mitigation, and mitigation methods are tied to the specific categories they address.","core_discovery":"The paper's central claim is that code hallucinations are not one undifferentiated failure mode: they can be classified by the observable characteristics of the emitted code into four primary categories, independent of model architecture or training data. Syntactic hallucinations break the language rules; runtime execution hallucinations fail during execution despite being syntactically valid; functional correctness hallucinations run but do not satisfy the intended task; and code quality hallucinations introduce resource, security, or maintainability risks. The paper further claims that existing benchmarks and mitigation strategies can be meaningfully compared within this framework, and it maps them to the taxonomy to show where coverage exists and where it is missing.","pith_inferences":["A natural extension the paper leaves implicit is to use the taxonomy as an annotation scheme for automated program repair, where the error category could predict which repair strategy is most likely to succeed.","Because the categories are defined by observable code properties, static analyzers and execution monitors could in principle detect them automatically, making the taxonomy operational rather than purely descriptive.","The four categories are not shown to be mutually exclusive; a single snippet can be both syntactically broken and functionally wrong. Adding primary and secondary labels would make the scheme more robust, but that is an extension rather than a claim in the paper.","If multi-language and real-repository benchmarks are built along these categories, a testable open hypothesis emerges: hallucination types may shift across languages, with some languages showing more syntactic failures and others more API or requirement failures."],"forward_implications":["If the taxonomy holds, hallucination-detection benchmarks can be compared by which of the four categories they exercise, even when their underlying tasks differ.","Mitigation methods can be assigned to categories: grammar-guided decoding targets syntactic hallucinations, retrieval-augmented generation targets API and context failures, and requirements clarification targets requirement deviation, suggesting that combined approaches should outperform any single one.","Because most surveyed benchmarks are Python, function-level datasets, reported hallucination rates primarily describe that setting and should not be assumed to transfer to repository-level or multi-language development.","Evaluation metrics matter per category, since pass@k alone can miss requirement deviation, security vulnerabilities, and code smells that the taxonomy treats as distinct hallucination types."],"supporting_citations":[{"why":"Provides the extensive study of wrong generated code whose error classifications anchor the syntactic, runtime, and functional categories.","marker":"[10]"},{"why":"Introduces CodeHaluEval, an execution-based benchmark whose hallucination types and detection metric underpin the taxonomy and the Hallucination Rate metric.","marker":"[53]"},{"why":"Introduces CodeMirage, which supplies artificially hallucinated code snippets and classification metrics used in the halluncination-type category.","marker":"[1]"},{"why":"Introduces HalluCode with valid rate, hallucination recognition accuracy, and mitigation accuracy, forming the metrics section.","marker":"[34]"},{"why":"Supplies empirical failure analysis with categories such as syntax violations, invalid references, and requirement deviations used to build the taxonomy.","marker":"[57]"},{"why":"Provides both a hallucination taxonomy and a retrieval-augmented mitigation approach addressing API and repository-context failures.","marker":"[65]"},{"why":"Presents De-Hallucinator, an iterative grounding mitigation method that retrieves project APIs to reduce hallucination, serving as a mitigation case study.","marker":"[12]"},{"why":"Shows that self-revision with static-analysis feedback mitigates code quality issues, serving as a second mitigation case study.","marker":"[37]"},{"why":"Presents SynCode, a grammar-augmented decoding method targeting syntactic hallucinations, supporting the claim that syntax errors can be constrained at generation time.","marker":"[54]"},{"why":"Presents ClarifyGPT, a requirements-clarification framework that targets hallucinations caused by ambiguous prompts, supporting the prompt-related mitigation analysis.","marker":"[39]"}],"fun_headline_variants":["Four flavors of code hallucinations in LLMs","Code LLMs hallucinate in four distinct ways","A taxonomy for code hallucinations: syntax, runtime, logic, quality","Benchmarks and fixes for LLM code hallucinations, sorted","Spotting and fixing the four types of code hallucination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's conclusions rest on the assumption that its keyword search plus snowballing found every relevant code-hallucination study, and that the authors' manual sorting of papers into categories was consistent, with no independent confirmation of that sorting reported.","fun_headline_variants_meta":{"raw":{"variants":["Four flavors of code hallucinations in LLMs","Code LLMs hallucinate in four distinct ways","A taxonomy for code hallucinations: syntax, runtime, logic, quality","Benchmarks and fixes for LLM code hallucinations, sorted","Spotting and fixing the four types of code hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2663,"prompt_tokens":866,"completion_tokens":1797,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":1719}},"tokens_in":482,"tokens_out":1797,"duration_ms":11988,"temperature":1.0,"reasoning_tokens":1719,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:19:00.946474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a corpus of code outputs from multiple LLMs on real repository-level tasks, have independent annotators assign each error to exactly one of the four taxonomy categories, and measure inter-annotator agreement; if many errors fit multiple categories or none, the claim that the taxonomy objectively classifies code hallucinations would fail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents ClarifyGPT, a requirements-clarification framework that targets hallucinations caused by ambiguous prompts, supporting the prompt-related mitigation analysis."}],"review_version":1}