Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Across 86,726 failed code samples, even the largest LLMs routinely make simple, security-relevant errors, so unassisted AI code generation is not yet reliable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:00 UTC pith:C6ANSEJG

load-bearing objection Large and useful error census for LLM code generation, but the security conclusion overreaches the runtime-symptom labels; worth a serious referee. the 4 major comments →

arxiv 2608.00661 v1 pith:C6ANSEJG submitted 2026-08-01 cs.SE

Unreliable in Practice? A Comprehensive Study of Errors in LLM-Generated Code

Rodrigo Pato Nogueira , Marco Vieira , Jo\~ao R. Campos This is my paper
classification cs.SE
keywords LLM code generationcompilation errorsruntime errorserror taxonomyempirical software engineeringcode reliabilitymemory safetyinput validation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to characterize what actually goes wrong when seven large language models generate code in C++, Java, C, and Rust, using 86,726 samples that failed to compile or crashed during unit testing. It classifies each failure with a validated LLM-based taxonomy and measures how error types vary by language, model, problem difficulty, prompting strategy, and iterative feedback. The central finding is that while error profiles differ sharply across languages and models, every model — including the largest commercial ones — frequently makes trivial mistakes such as missing imports, undeclared variables, and type mismatches. Worse, generated code often omits basic input validation and memory-safety checks, producing out-of-bounds accesses, numeric overflows, and resource exhaustion that can become security vulnerabilities. A sympathetic reader would take this as evidence that current LLMs, used alone, are not reliable enough for security-sensitive or dependability-critical code.

Core claim

The paper's central claim is that LLM-generated code is unreliable in practice in an unassisted setting: across a corpus of 86,726 failed samples from seven models and four compiled languages, no model escapes a substantial share of elementary mechanical errors, and all models systematically neglect input validation and memory-safety safeguards. The authors show that error types concentrate in a small set of categories — missing includes/imports, incompatible parameter types, and undeclared variables for compilation; out-of-bound accesses, incorrect input processing, out-of-memory, and numeric overflow for runtime — and that these patterns are strongly shaped by language semantics: Rust trig

What carries the argument

The load-bearing machinery is a two-part error taxonomy adapted from prior work — one schema for compilation errors (missing import/include, undeclared variable, incompatible parameter types, etc.) and one for runtime errors (out-of-bound, numeric overflow, out-of-memory, recursion error, etc.) — operationalized by a zero-shot LLM classifier (GPT-4.1-mini) that labels each {code + compiler/runtime diagnostic + test summary} sample. The classifier was validated on 600 manually annotated samples, with per-class precision/recall and a confusion-matrix-based correction applied to the full corpus counts. This machinery lets the authors turn raw compiler and runtime diagnostics into measurable pre

Load-bearing premise

The analysis assumes the LLM-based classifier's labels on the 86,726-error corpus are accurate enough that the reported prevalence figures reflect real error rates; the classifier was validated on only 600 manually labeled samples, with near-zero recall for several rare classes and a documented bias toward 'out-of-bound', so systematic mislabeling could shift the per-language and per-model percentages.

What would settle it

Manually label a fresh random sample of several hundred errors from the full corpus — especially from the dominant categories 'Missing import' and 'Out-of-bound' — and compare with the classifier's labels; if per-class precision/recall on this sample diverges substantially from the reported evaluation set, or if a re-run with a corrected classifier changes which categories dominate, the prevalence claims would need revision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Developers using LLM-generated code in security- or dependability-sensitive contexts cannot rely on unassisted generation; human review and validation remain necessary.
  • Because a few error classes dominate — missing imports, incompatible parameter types, out-of-bound accesses, and incorrect input processing — targeted mitigations such as automated import repair, boundary and overflow validation, and input-handling scaffolding could remove a large share of failures with relatively little effort.
  • The same underlying model behavior can appear correct in one language and faulty in another, because numeric overflow panics in Rust but silently corrupts values in C/C++/Java; language-aware validation is therefore needed rather than assuming correctness from the absence of raised exceptions.
  • Iterative feedback is a weak remedy: it resolves local errors like missing imports, but structural errors such as incompatible parameter types, out-of-bounds, and recursion errors persist after feedback.
  • Chain-of-thought prompting can reduce simple compilation mistakes such as missing imports, but it shifts errors toward more complex structural issues and has little effect on the overall runtime failure profile.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension: The concentration of errors in a few classes suggests that cheap pre-commit static checks — import resolution, type checks, and bound analysis — might form an effective safety layer for AI-generated code, a hypothesis the paper does not test but its data support.
  • Extension: The paper's focus on compiled languages and competitive-programming tasks likely understates practical risk in interpreted or multi-file settings, where the same logical errors may surface even later; applying the taxonomy to interpreted languages and incorrect-output cases would test this.
  • Extension: If the classifier's documented bias toward 'out-of-bound' and low recall for rare classes is systematic, the absolute percentages for those rare classes could be underestimated, but the paper's qualitative conclusion — that all models produce many simple mistakes — would likely survive even a large correction.
  • Extension: Benchmark-style pass@k results can hide exactly the failure modes this paper exposes; reporting error-type distributions alongside correctness metrics could give practitioners a more honest picture.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper analyzes 86,726 compilation and runtime errors in LLM-generated code from seven models across four compiled languages (C++, Java, C, Rust). Errors are labeled with an LLM-based classifier using taxonomies adapted from prior work, manually validated on 600 samples. The study reports error distributions by language, model, and problem difficulty; the effect of iterative feedback; and the impact of CoT prompting. The central claim is that, despite variation across languages and models, all LLMs frequently make simple mistakes (e.g., missing imports, undeclared variables) and often omit input-validation or memory-safety checks, leading to reliability/security risks and making current models unreliable for standalone code generation.

Significance. If the results hold, this would be a valuable large-scale empirical characterization of LLM code-generation errors, going beyond prior single-language or small-sample studies. Strengths include the scale and diversity of the corpus (86,726 errors, 7 models, 4 languages), the reuse of an existing taxonomy, the manual validation subset, the analysis of iterative feedback and CoT prompting, and the public pipeline. The headline conclusion about 'unreliable in practice' is plausible and policy-relevant. However, several load-bearing points need to be addressed before the claims are fully supported.

major comments (4)
  1. [IV.B, IV.C, abstract/conclusion] The abstract and conclusion claim that generated code "often omits basic input validation or memory-safety checks" and that this "can lead to overflows, resource exhaustion, or other reliability/security issues." The taxonomy (Table III) and the reported distributions (Tables X, XI) label runtime failure symptoms (e.g., "Out-of-bound", "Incorrect input processing", "Numeric overflow"), not the absence of validation or safety checks. "Incorrect input processing" is defined as "wrong assumptions about the input format," which can be a parser bug, and "Out-of-bound" can be an off-by-one logic error. The paper does not inspect source code to establish that checks were omitted. Section V scopes conclusions to compilation/runtime failures, but the abstract/conclusion use stronger causal language. Please reframe these claims as "frequent runtime failures consistent with weak input handling/boun
  2. [III.B.3, Table V] The classifier's per-class recall is zero for several categories relevant to the paper's claims: runtime "Other" has recall 0.00 for all four languages (support 1-3), and "Out-of-memory" has recall 0.00 for Rust. The bias-correction described in Section III.B.3 is said to use per-class precision/recall, but the full per-class breakdown is "omitted here for space," so the reader cannot verify that the correction leaves trends unchanged. Moreover, the inference in Section IV.B that the low frequency of "Other" indicates good taxonomy coverage is invalid if the classifier never predicts "Other." Please provide the full confusion matrix or per-class precision/recall on the validation set and report the corrected versus raw distributions for all classes.
  3. [III.A.1, Tables VI, IX, X, XIII] The reported corpus size is internally inconsistent. Section III.A.1 states that across the two prompting techniques there are 52,669 compilation and 34,057 runtime errors (total 86,726), with per-language counts such as 9,953 C++ compilation errors. However, Table VI (baseline compilation) sums to 30,109, and Table X (baseline runtime) sums to 14,322. For C++ compilation, Table VI baseline total is 6,706 and Table IX CoT total is 5,909, summing to 12,615, not 9,953. Similarly, Java runtime baseline 4,051 plus CoT 4,090 is 8,141, not 8,279. These discrepancies affect every prevalence percentage and must be reconciled.
  4. [IV.A, IV.B, statistical tests] The two-proportion z-tests and chi-square tests are performed on raw classifier labels, without propagating classifier uncertainty into the hypothesis tests. Given that the classifier's per-class precision/recall varies widely (e.g., recall 0.00 for "Other"), the statistically significant differences reported in Section IV may reflect classifier bias rather than true differences in error distributions. The paper should either propagate the bias-correction into the statistical tests or add a sensitivity analysis showing that the test conclusions are robust to plausible classifier error rates.
minor comments (6)
  1. [Tables VI, X] The captions say "only the results for labels with a frequency of above three percent in any column," but rows such as "Other" and "Uninitialized variable" appear with frequencies below 3% in some columns. Please clarify the selection rule.
  2. [Tables VI, X] The "Total" rows are not the sum of the displayed rows, because categories with low frequencies in all columns are omitted. Add a footnote stating that the Total includes all classified errors.
  3. [Table V] The category numbers in the header are not self-explanatory; consider using category names or providing a legend directly below the table for readability.
  4. [III.A.1] "Across the two prompting techniques" is ambiguous; specify that the two techniques are 0-shot baseline and CoT, and clarify whether GPT-oss:120b is included in the counts.
  5. [Listing 2 caption] The caption says "incomp. param. types"; expand to "Incompatible parameter types" for consistency.
  6. [IV.B] In the text, "C++ (59.9%)" for Out-of-bound differs from Table X's rounded value of 60.0%; standardize rounding across text and tables.

Circularity Check

0 steps flagged

No significant circularity: the study is an empirical measurement pipeline, and no claimed result reduces to its inputs by construction.

full rationale

This is an empirical study, not a derivation, so the main circularity failure modes (prediction-equals-fit, self-definitional taxonomy, imported uniqueness) largely do not apply. The error taxonomy is explicitly inherited from non-overlapping prior work (Liu et al. [11]) and adapted; the paper states it is not proposing a new taxonomy. The LLM classifier is validated against 600 externally human-labeled samples, and the confusion-matrix correction is a sensitivity analysis rather than a fitted parameter renamed as a prediction. Self-citations exist ([6] prior functional-correctness work, [24] the PROBE dataset), but they are background/data inputs, not load-bearing justifications of the prevalence or security conclusions. The abstract's security phrasing (e.g., 'omits basic input validation or memory-safety checks') does go beyond what the symptom-level runtime labels directly establish; the paper's own threats-to-validity section scopes conclusions to compilation/runtime failures. That is an evidentiary strength concern, not circularity: the labels are not defined in terms of that conclusion. The low/zero recall for the 'Other' class and the omitted per-class confusion-matrix breakdown are validity limitations, but they do not make the analysis self-referential. Overall, no circular step can be exhibited, so the paper receives a low score reflecting only minor self-citation presence.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted parameters or ad hoc constants. The decoder settings (temperature 0, top-k 50, top-p 1.0) are fixed for reproducibility, not fitted. The confusion-matrix correction uses precision/recall ratios from the evaluation set, but that is a sensitivity adjustment, not a free parameter. The paper adds taxonomy categories (e.g., 'Missing Public Class', 'Ownership and lifetime error'), but these are descriptive labels, not invented entities. The main assumptions are about classifier reliability, dataset representativeness, and the scope of error types considered.

axioms (4)
  • domain assumption LLM-based classifier labels approximate ground truth for the 86,726 samples
    Central to all prevalence estimates; validated on 600 labeled samples with exact-match accuracy 0.81-0.92, but per-class recall is near zero for some rare classes (Table V), so systematic classifier errors could shift reported percentages.
  • domain assumption The PROBE dataset generation process is representative of 'LLM-generated code' as claimed
    The paper relies on this corpus (own prior work [24]) for all analysis; its selection of models, temperature, prompting, and competitive-programming problems shapes the observed errors. External validity is limited.
  • domain assumption Compilation and runtime errors are a meaningful proxy for code reliability
    The study excludes incorrect outputs and timeouts, which biases the distribution toward syntactic/runtime failure modes and underrepresents logical/algorithmic mistakes. The authors acknowledge this in Section V.
  • domain assumption The adapted taxonomy from Liu et al. adequately covers the error space
    If the taxonomy misses systematic error types, prevalence conclusions would be incomplete. The small 'Other' share is cited as evidence of coverage, but the classifier's low recall for 'Other' weakens this.

pith-pipeline@v1.3.0-daily-deepseek · 19644 in / 13025 out tokens · 109682 ms · 2026-08-04T01:00:11.258432+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are being widely used for coding, with reports indicating that AI now generates an increasing share of production code. Studies show that LLMs can significantly improve developer productivity, yet they still struggle with more complex coding tasks. Just as understanding error modes in human-written code has been central to improving software quality, identifying and characterizing the errors in LLM-generated code is critical for setting realistic expectations and designing mitigation strategies. Prior research has been limited in scope, often focusing on a single language, a small number of problems, or a limited selection of models. As a result, there is still no comprehensive understanding of which errors are common and which are specific to certain models or languages. To address these gaps and develop a deeper understanding of the quality of LLM-generated code, we analyzed a corpus of 86,726 code samples that contained compilation or runtime errors. These samples were generated by seven LLMs across four compiled languages. We classified errors by their underlying causes using an LLM, manually validated these classifications, and performed a comparative analysis. This labeled data is then used to measure error prevalence by model, language, and problem difficulty, to identify common error patterns. Results show that, although error types vary strongly across languages and models, even the largest models frequently make simple mistakes. We also observe that generated code often omits basic input validation or memory-safety checks, which can lead to overflows, resource exhaustion, or other reliability/security issues.

Figures

Figures reproduced from arXiv: 2608.00661 by Jo\~ao R. Campos, Marco Vieira, Rodrigo Pato Nogueira.

Figure 1
Figure 1. Figure 1: Methodology limitations (Table I). Most focus on a single programming language, typically Python, limiting the range of observable errors and generalizability [7]–[10]. Python’s dynamic execu￾tion model masks entire classes of failures related to memory management and undefined behavior, and even extensions to languages such as Java abstract away many system-level error modes common in lower-level language… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 4 linked inside Pith

  1. [1]

    https://web.archive.org/web/20250430014530/https://www.cnbc.c om/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft -code-is-written-by-ai.html

    (2025) Satya nadella says as much as 30% of microsoft code is written by ai. https://web.archive.org/web/20250430014530/https://www.cnbc.c om/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft -code-is-written-by-ai.html. Accessed: 2025-05-05

    arXiv 2025

  2. [2]

    Microsoft cto breaks down how he sees software developer jobs evolving in the next 5 years,

    S. Perkel, “Microsoft cto breaks down how he sees software developer jobs evolving in the next 5 years,” Business Insider, April 2025, https: //www.businessinsider.com/microsoft-cto-ai-generated-code-software-d eveloper-job-change-2025-4

    2025

  3. [3]

    The impact of ai on developer productivity: Evidence from github copilot,

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The impact of ai on developer productivity: Evidence from github copilot,” 2023

    2023

  4. [4]

    The effects of github copilot on computing students’ pro- gramming effectiveness, efficiency, and processes in brownfield coding tasks,

    M. I. H. Shihab, C. Hundhausen, A. Tariq, S. Haque, Y . Qiao, and B. W. Mulanda, “The effects of github copilot on computing students’ pro- gramming effectiveness, efficiency, and processes in brownfield coding tasks,” inProceedings of the 2025 ACM Conference on International Computing Education Research V .1, ser. ICER ’25. New York, NY , USA: Associatio...

    2025

  5. [5]

    Measuring coding challenge competence with apps,

    D. Hendryckset al., “Measuring coding challenge competence with apps,” 2021

    2021

  6. [6]

    Beyond functional correctness: An empirical evaluation of large language models for text- to-code generation,

    R. P. Nogueira, M. Vieira, and J. R. Campos, “Beyond functional correctness: An empirical evaluation of large language models for text- to-code generation,” in2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE), 2025, pp. 264–275

    2025

  7. [7]

    Exploring and evaluating hallucinations in llm-powered code generation,

    F. Liuet al., “Exploring and evaluating hallucinations in llm-powered code generation,”arXiv preprint arXiv:2404.00971, 2024

    arXiv 2024

  8. [8]

    Towards understanding the characteristics of code generation errors made by large language models,

    Z. Wanget al., “Towards understanding the characteristics of code generation errors made by large language models,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2025, pp. 717–717

    2025

  9. [9]

    An empirical study of code generation errors made by large language models,

    D. Songet al., “An empirical study of code generation errors made by large language models,” in7th Annual Symposium on Machine Programming, 2023

    2023

  10. [10]

    Bugs in large language models generated code: An empirical study,

    F. Tambon, A. Moradi-Dakhel, A. Nikanjam, F. Khomh, M. C. Des- marais, and G. Antoniol, “Bugs in large language models generated code: An empirical study,”Empirical Software Engineering, vol. 30, no. 3, p. 65, 2025

    2025

  11. [11]

    No need to lift a finger anymore? assessing the quality of code generation by chatgpt,

    Z. Liu, Y . Tang, X. Luo, Y . Zhou, and L. F. Zhang, “No need to lift a finger anymore? assessing the quality of code generation by chatgpt,” IEEE Transactions on Software Engineering, vol. 50, no. 6, pp. 1548– 1584, 2024

    2024

  12. [12]

    A deep dive into large language model code generation mistakes: What and why?

    Q. Chen, J. Yu, J. Li, J. Deng, J. T. J. Chen, and I. Ahmed, “A deep dive into large language model code generation mistakes: What and why?” arXiv preprint arXiv:2411.01414, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Assessing and analyzing the correctness of github copilot’s code suggestions,

    R. Moet al., “Assessing and analyzing the correctness of github copilot’s code suggestions,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 7, pp. 1–32, 2025

    2025

  14. [14]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2024

    2024

  15. [15]

    Deepseek-v3 technical report,

    A. Liuet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,

    Q. Zhuet al., “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,”arXiv preprint arXiv:2406.11931, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Qwen2.5-coder technical report,

    B. Huiet al., “Qwen2.5-coder technical report,” 2024

    2024

  18. [18]

    Incoder: A generative model for code infilling and synthesis,

    D. Friedet al., “Incoder: A generative model for code infilling and synthesis,” 2023

    2023

  19. [19]

    Evaluating large language models trained on code,

    M. Chenet al., “Evaluating large language models trained on code,” 2021

    2021

  20. [20]

    A large scale study of programming languages and code quality in github,

    B. Ray, D. Posnett, V . Filkov, and P. Devanbu, “A large scale study of programming languages and code quality in github,” inProceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, 2014, pp. 155–165

    2014

  21. [21]

    Orthogonal defect classification-a concept for in- process measurements,

    R. Chillaregeet al., “Orthogonal defect classification-a concept for in- process measurements,”IEEE Transactions on software Engineering, vol. 18, no. 11, pp. 943–956, 1992

    1992

  22. [22]

    Program synthesis with large language models,

    J. Austinet al., “Program synthesis with large language models,” 2021

    2021

  23. [23]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Luet al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” 2021

    2021

  24. [24]

    Probe: Benchmarking code generation in large language models,

    R. P. Nogueira, M. Vieira, and J. R. Campos, “Probe: Benchmarking code generation in large language models,” 2026. [Online]. Available: https://arxiv.org/abs/2607.13820

    Pith/arXiv arXiv 2026

  25. [25]

    Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,

    R. Puriet al., “Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,” 2021

    2021

  26. [26]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,”Biometrics, vol. 33, no. 1, pp. 159–174, 1977

    1977

  27. [27]

    Polyglot: An extensible framework to benchmark code translation with llms,

    M. Vieira, P. A. Shah, B. Shah, and R. Krasniqi, “Polyglot: An extensible framework to benchmark code translation with llms,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025, pp. 2363–2375

    2025