Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Code-generating LLMs fail in four recurring ways—misreading the problem, flawed algorithms, edge cases, and formatting—and solution complexity plays only a small role.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:40 UTC pith:Z75XDQW6

load-bearing objection Useful descriptive failure taxonomy for LLM code generation, but the single-PASS@1 sampling makes the 'consistently failed' task set—and the pattern counts—unstable. the 3 major comments →

arxiv 2511.04355 v1 pith:Z75XDQW6 submitted 2025-11-06 cs.SE cs.LG

Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

Amir Molzam Sharifloo , Maedeh Heydari , Parsa Kazerooni , Daniel Maninger , Mira Mezini This is my paper
classification cs.SE cs.LG
keywords code generationlarge language modelsbenchmark analysisfailure patternsPASS@1complexity metricsHumanEvalBigCodeBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when modern LLMs fail at code-generation tasks, they tend to fail in four identifiable ways: wrong problem mapping, flawed or incomplete algorithm design, edge case mishandling, and formatting mistakes. Analysing 865 tasks from four widely used benchmarks and six LLMs, the authors found 114 tasks that every model failed, measured the static complexity of the reference solutions, and showed that complexity metrics barely predict failures except in one benchmark. The failures, they conclude, stem less from how hard the code is and more from how models interpret the problem and handle boundary conditions—and in some cases from ambiguous benchmark prompts. A sympathetic reader would care because this redirects the field from chasing leaderboard numbers toward fixing specific, addressable weaknesses.

Core claim

The paper establishes a taxonomy of LLM code-generation failure: across 114 tasks that six leading models all failed, four patterns account for the errors—wrong problem mapping (models apply a familiar-but-wrong problem class, e.g., treating a nested-bracket subsequence task as classic balanced brackets), flawed or incomplete algorithm design (correct general approach with missing steps), edge case mishandling (invalid on boundary inputs), and formatting mistakes (correct logic rejected by strict I/O). Quantitative complexity analysis using six static metrics (cyclomatic complexity, data-structure use, function-call frequency, code length, nesting depth, recursion) shows that failures correl

What carries the argument

The analysis rests on a task-level classification of 'consistently failed' tasks (those all six models failed under a single PASS@1 sample each), a static abstract-syntax-tree tool that computes six code-complexity dimensions of reference solutions, and a manual inspection of the failed tasks that assigns each failure to one of four patterns. Statistics (Spearman correlation and linear regression) separate benchmarks where complexity explains failures (LiveCodeBench) from those where it does not.

Load-bearing premise

The study labels a task as 'consistently failed' when each of six models produced one wrong solution under a single PASS@1 sample; if those outputs are stochastic and sampling conditions vary, a lone failed run may be bad luck rather than a systematic weakness, and every pattern count built on that set would be affected.

What would settle it

Re-run the 114 'consistently failed' tasks with each model under multiple samples (e.g., temperature 0.2 and 0.8, several seeds) and check whether they ever pass; if a substantial fraction pass in at least one sample, the consistent-failure label and the pattern counts collapse. Conversely, if nearly all remain failing across samples, the taxonomy is robust. A second check: retrain or prompt-augment a model specifically on edge-case and format-compliance examples; if the four patterns' failure rate does not drop, the taxonomy is not causal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark design should stop treating solution complexity as the main difficulty axis; adding edge cases, strict-format requirements, and problem-mapping traps would better discriminate models.
  • LLM improvement efforts can target the four failure patterns—for example, training on edge-case reasoning and format compliance—rather than relying on generic scale alone.
  • Leaderboard ranks overstate capability differences: many failures are systematic across all models, and on BCB-Hard the ostensibly weaker model succeeded more by reading prompts literally while stronger models over-assumed.
  • Failure-pattern-aware benchmarks could be constructed to expose these weaknesses, as the paper proposes as future work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'over-optimization' observation—strong models adding sensible but unstated assumptions that violate tests—suggests that as models get better at conventional coding, they may fare worse on literal, contrived benchmarks; a testable prediction is that fine-tuning on benchmark-style literal prompts improves leaderboard scores without improving real-world coding.
  • The four-pattern taxonomy likely generalizes beyond function-level benchmarks to repository-level and agentic coding, since problem-mapping and edge-case errors plausibly scale to multi-file edits; extending the analysis to such benchmarks could test this.
  • If benchmarks were redesigned around failure patterns rather than overall pass rates, model rankings might shift substantially, implying that current leaderboards conflate raw capability with prompt/test alignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents an empirical study of code generation failures across four benchmarks (HumanEval, MBPP, LiveCodeBench, BigCodeBench-Hard) using six LLMs. The authors identify 114 tasks that all six models failed on a single PASS@1 run, measure static complexity metrics of ground-truth solutions, and analyze correlations between complexity and failure counts. They find a clear positive correlation for LiveCodeBench (R² up to 0.32, p<10⁻⁶) but weak or non-significant associations for the other benchmarks. Through manual inspection of the 114 consistently failed tasks, they propose four recurring failure patterns: wrong problem mapping, flawed/incomplete algorithm design, edge-case mishandling, and formatting mistakes, along with benchmark-induced ambiguity. The paper releases data and scripts on GitHub.

Significance. If the central taxonomy is robust, the paper provides a useful, actionable characterization of LLM code-generation weaknesses and a clear negative result on the explanatory power of static code complexity. The study is honest about weak correlations and makes its data and scripts publicly available, which supports reproducibility. The proposed failure patterns (especially wrong problem mapping and over-assumption relative to strict tests) are plausible and consistent with qualitative examples. However, the empirical foundation rests on a fragile single-sample evaluation protocol and subjective manual coding, so the quantitative pattern counts and the taxonomy itself need strengthening before the claims can be fully accepted.

major comments (3)
  1. [Section II, Evaluation Procedure] The 'consistently failed' set of 114 tasks is defined by a single PASS@1 sample per model per task, with no reported temperature, random seed, or repetition count. LLM code generation is stochastic; for tasks with true pass rates around 10–30%, the probability that all six models independently fail one sample is substantial (e.g., 0.8^6 ≈ 0.26 for a 20% per-model pass rate). This means the 114-task set is likely contaminated by sampling noise, and every downstream result—the difficulty distribution in Table II, the pattern counts in Table IV, and the qualitative examples in Section IV—inherits that noise. The central claim of four recurring failure patterns cannot be distinguished from single-sample artifacts unless the authors either fix a seed/temperature and justify it, or re-run with multiple samples and show that the consistently failed set is stable. This is the load-bearing issue
  2. [Section IV, Failure Inspection and Table IV] The manual classification of failure patterns has no reported inter-rater reliability or coding protocol. The four categories are introduced after the fact and applied by the authors to model outputs; without multiple annotators and a kappa/agreement statistic, the pattern counts in Table IV are subjective. The examples are illustrative, but the central claim that these 'four recurring patterns' are systematic is proportional to the rigor of the coding process. Please add a coding scheme, at least two independent annotators, and agreement metrics, or explicitly reposition the taxonomy as an exploratory, non-quantified account.
  3. [Section III.B, Correlation Analysis] The complexity analysis is qualified appropriately for most benchmarks, but the BCB-Hard interpretation is internally inconsistent: the text first reports 'moderate R² values (≈0.14–0.15) with borderline p-values (≈0.07–0.09), suggesting weak but potentially meaningful associations', then states 'none reach conventional thresholds for statistical significance'. With 148 tasks, a p-value of 0.07 is not significant by conventional standards, and calling it 'potentially meaningful' without a pre-registered threshold or multiple-comparison correction weakens the RQ2 conclusion. Either apply a correction and state the conclusion strictly, or report confidence intervals to support the interpretation.
minor comments (4)
  1. [General / Section I] Minor typos: 'benchamrks' in Section II, 'T asks' in the y-axis label of Figure 1. Also, the paper refers to 'Table I reports the number of failures' but the table groups counts in a way that is confusing; consider a simpler per-model failure table.
  2. [Section II, Models] The model list omits version/access dates and inference configuration (temperature, max tokens, prompting format). This information is essential for reproducibility, especially given the single-sample protocol. Please add a table with model identifiers, API or checkpoint dates, and sampling parameters.
  3. [Section IV, 'Ambiguous Prompt & Restricted Test'] The ambiguity category is discussed qualitatively but not included in Table IV or quantified. If ambiguity is a major finding (RQ3), it should be coded and counted like the other four patterns, or explicitly labeled as an observation not a measured pattern.
  4. [Section V, Limitations] The limitations section omits the single-sample issue and the lack of inter-rater reliability. Please acknowledge these directly in the limitations, as they are the main threats to the validity of the taxonomy.

Circularity Check

0 steps flagged

No significant circularity: the failure taxonomy and complexity analysis are empirical inductions over independent model outputs and benchmark solution code, not quantities fitted to the targets they are used to explain.

full rationale

The paper's central outputs are (a) a set of 114 tasks classified as 'consistently failed' because all six models failed a single PASS@1 run, and (b) four inductively derived failure patterns obtained by manually inspecting those tasks' model outputs and failing test cases. These are empirical measurements followed by qualitative classification; the four patterns (wrong problem mapping, flawed/incomplete algorithm design, edge-case mishandling, formatting mistakes) are not defined in terms of the failure counts they are invoked to explain, and no parameter is fitted to a subset of data and then reported as a prediction. The complexity analysis in Section III also treats ground-truth solution metrics as independent explanatory variables and reports correlation/regression against observed failure counts; reporting R^2 and p-values on the same dataset is ordinary descriptive analysis, not circular inference. There is no load-bearing self-citation: prior work [10] is cited as related scope motivation, not as the source of the taxonomy or the uniqueness of the classification. The main legitimate concern is methodological rather than circular: reliance on one PASS@1 sample per model without reported temperature or seeds makes the 'consistently failed' set sensitive to sampling noise, and the manual pattern coding lacks reported inter-rater reliability. But sampling noise and measurement reliability are threats to validity, not evidence that a result reduces by construction to its inputs. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The study rests on measurement assumptions rather than derived laws. The most load-bearing are single-sample failure labeling and the validity of static complexity metrics; neither is experimentally calibrated, and both could shift the reported pattern counts.

axioms (4)
  • domain assumption Single PASS@1 generation per task is sufficient to characterize whether a model fails a task.
    Section II Evaluation Procedure states 'Each task was evaluated using a single generated solution per model (PASS@1)', assuming away sampling variance.
  • domain assumption Static complexity metrics (cyclomatic complexity, length, nesting, etc.) are a meaningful proxy for task difficulty.
    Section III-A defines the metrics but provides no validation that they capture the cognitive difficulty of a task.
  • domain assumption The six selected models and benchmark subsets are representative of current LLM code-generation capabilities.
    Section II Models/Benchmarks: Gemini and Grok were excluded due to budget, and MBPP was reduced to the 378-task subset used by MBPP+.
  • domain assumption Manual inspection of model outputs can reliably identify failure patterns.
    Section IV performs qualitative classification of 114 tasks with no inter-rater reliability or coding protocol reported.

pith-pipeline@v1.3.0-alltime-deepseek · 7202 in / 6753 out tokens · 64692 ms · 2026-08-03T23:40:43.787320+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have achieved remarkable success in code generation, and the race to improve their performance has become a central focus of AI research. Benchmarks and leaderboards are increasingly popular, offering quantitative rankings of LLMs. However, they provide limited insight into the tasks that LLMs consistently fail to solve - information that is crucial for understanding current limitations and guiding the development of more capable models. To address this gap, we examined code generation tasks across four popular benchmarks, identifying those that major LLMs are most likely to fail. To understand the causes of these failures, we investigated whether the static complexity of solution code contributes to them, followed by a systematic inspection of 114 tasks that LLMs consistently struggled with. Our analysis revealed four recurring patterns of weaknesses in LLMs, as well as common complications within benchmark tasks that most often lead to failure.

Figures

Figures reproduced from arXiv: 2511.04355 by Amir Molzam Sharifloo, Daniel Maninger, Maedeh Heydari, Mira Mezini, Parsa Kazerooni.

Figure 1
Figure 1. Figure 1: Comparison of benchmark code metrics for tasks, grouped by the number of LLMs that failed them. LiveCodeBench [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses

    cs.LG 2026-06 unverdicted novelty 6.0

    FailureScope clusters evaluation probes by cross-model failure patterns via LOMO to produce stable taxonomies that generalize across single-turn, multi-turn, and adversarial regimes, with reported metrics of Kendall's...

Reference graph

Works this paper leans on

11 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

  2. [2]

    J. Ji, J. Jun, M. Wu, and R. Gelles, ``Cybersecurity risks of ai-generated code,'' Center for Security and Emerging Technology, Tech. Rep., November 2024. [Online]. Available: https://cset.georgetown.edu/wp-content/uploads/CSET-Cybersecurity-Risks-of-AI-Generated-Code.pdf

    2024

  3. [3]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herb...

    Pith/arXiv arXiv 2021

  4. [4]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, ``Program synthesis with large language models,'' 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  5. [5]

    S. Jain, S. B. Kong, E. Zelikman, A. Chen, T. Chen, A. Svyatkovskiy, and C. Sutton, ``Livecodebench: Holistic and contamination-free evaluation of llms for code,'' 2024. [Online]. Available: https://arxiv.org/abs/2404.00699

    Pith/arXiv arXiv 2024

  6. [6]

    T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, B. Hui, N. Muennighoff, D. Lo, D. Fried, X. Du, H. de Vries, and L. V. Werra, ``Bigcodebench: Benchmark...

    Pith/arXiv arXiv 2025

  7. [7]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, ``Swe-bench: Can language models resolve real-world github issues?'' 2024. [Online]. Available: https://arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2024

  8. [8]

    J. Liu, C. S. Xia, Y. Wang, and L. Zhang, ``Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,'' in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS '23. 1em plus 0.5em minus 0.4em Red Hook, NY, USA: Curran Associates Inc., 2023

    2023

  9. [9]

    Z. Yu, Y. Zhao, A. Cohan, and X.-P. Zhang, ``Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation,'' 2024. [Online]. Available: https://arxiv.org/abs/2412.21199

    Pith/arXiv arXiv 2024

  10. [10]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, ``Measuring coding challenge competence with apps,'' 2021. [Online]. Available: https://arxiv.org/abs/2105.09938

    Pith/arXiv arXiv 2021

  11. [11]

    Z. Wang, Z. Zhou, D. Song, Y. Huang, S. Chen, L. Ma, and T. Zhang, ``Towards understanding the characteristics of code generation errors made by large language models,'' in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1em plus 0.5em minus 0.4em Los Alamitos, CA, USA: IEEE Computer Society, May 2025, pp. 2587--2599. [Online]....

    arXiv 2025