REVIEW 4 major objections 6 minor 17 references
Do Code LLMs Understand Design Patterns?
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper attempts to establish that code LLMs systematically misrecognize and misapply object-oriented design patterns, degrading the reliability of generated code.
desk verdict A useful first cut at how code LLMs handle design patterns, with a real classification finding (best accuracy 38.81%) and a generation evaluation whose string-similarity metrics don't support the paper's downstream-reliability claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The evaluation harness is the central mechanism. It consists of 48 manually selected code repositories, 24 in each language, spanning 12 design patterns, with files graded into easy, moderate, and difficult complexity. Three probes are run: design-pattern classification, line completion with missing lines removed, and function generation from natural-language descriptions with or without the pattern name supplied. Generation quality is scored by code similarity and edit similarity: the first measures longest-matching-subsequence overlap between generated and reference code, and the second measures the normalized edit distance needed to turn generated code into the reference. These two proxies convert whether the model follows the design pattern into a number, and the whole argument about reliability rests on them.
What would settle it
Take a sample of low-similarity outputs from the function-generation task and have expert programmers judge whether each implements the intended pattern's structural and behavioral invariants; if most low-similarity outputs are judged pattern-compliant, the paper's conclusion that models conflict with design patterns would be undermined.
Extended reading notes
Core claim
The paper's central claim is that code LLMs exhibit measurable biases in handling design patterns, and that these biases significantly affect the reliability of downstream tasks. On the classification task, the best models top out at 38.81% accuracy across twelve design patterns and two languages; all models degrade from Java to Python and as code complexity rises. Misclassification heatmaps show that Singleton and Factory are systematically over-predicted because their structural cues, such as static methods, single-instance management, and object creation, overlap with patterns such as Facade, Proxy, Command, Abstract Factory, Builder, and Strategy, while Facade is the hardest to recognize because it lacks explicit structural markers. In line completion and function generation, providing the design pattern sometimes improves output similarity and sometimes worsens it, and code similarity and edit similarity diverge often enough that high textual overlap does not guarantee low repair effort. The paper concludes that better training data, clearer pattern distinctions, and improved pattern integration are needed before these models can be trusted in design-pattern-sensitive software workflows.
Load-bearing premise
The evaluation assumes that how close generated code is to one reference implementation tells whether the model correctly applied the design pattern.
Editorial extensions
If this is right
- Model-generated code in pattern-heavy projects will frequently deviate from the intended architecture, so developers should budget for post-editing even when the code is syntactically valid.
- Reporting edit similarity alongside code similarity gives a truer picture of developer effort, since high textual overlap does not always mean little repair work.
- Prompting a model with the design pattern name is not a reliable universal remedy: it helps on some patterns and models and hurts on others.
- Improvements should target the confusion clusters, such as Singleton versus Facade, Proxy, or Command, and the hardest patterns, including Builder, Bridge, and Facade, where current accuracy is lowest.
Reading between the lines
- These results measure textual similarity to one reference solution rather than structural conformance to the pattern, so the generation experiments may understate a model's true ability when a valid implementation is written differently.
- The observed over-prediction of Singleton and Factory suggests a testable intervention: augmenting training data with negative examples of near-miss patterns should raise classification accuracy on Facade and Strategy categories.
- A human or automated behavioral check of generated functions, such as verifying that a Singleton yields one instance or a Factory returns the correct product type, would separate textually different from pattern-violating and sharpen the reliability claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical study of code LLMs' understanding of design patterns. It uses a manually collected dataset of 48 GitHub repositories (24 Python, 24 Java) covering 12 design patterns and evaluates 11 models on three tasks: design-pattern classification, line completion, and function generation. In classification, the best models (GPT-4o and Llama-31-70B) reach 38.81% overall accuracy. In the generation tasks, the paper reports Code Similarity and Edit Similarity between model outputs and a single reference implementation, with and without providing the design-pattern label. The authors conclude that LLMs exhibit design-pattern biases that significantly affect the reliability of downstream tasks and impose a post-processing burden on developers.
Significance. The paper addresses a relevant and under-studied question: whether code LLMs can recognize and follow design patterns. The classification experiment provides direct evidence that current models are poor at recognizing design patterns, which is a useful data point for the community. The evaluation framework covering recognition, completion, and generation is a reasonable structure, and the inclusion of both Java and Python with 12 patterns is a breadth that many earlier studies lack. If the generation-metric validity concern were addressed, the downstream-reliability claim would be significant for practitioners and model developers. However, the current evidence is conditional on accepting that text-similarity to a single reference measures design-pattern conformance, which is not established; without that, the paper's central conclusion is unsupported.
major comments (4)
- [Section III-A-4 and Section III-C] Code Similarity (CS) and Edit Similarity (ES) are computed against one hand-picked reference implementation per task. These metrics measure token/string overlap and edit distance, not whether the generated code correctly implements the design pattern. A valid implementation that uses different identifiers, method decomposition, or statement ordering can score near zero, while an incorrect implementation that happens to resemble the reference text can score high. The paper interprets low CS/ES as evidence that generated code 'conflicts with the required design patterns' and imposes a post-processing burden (Abstract, Section III-C Insights). This inference is load-bearing for the central claim about downstream reliability, but no validation is provided: no human conformance judgment, no static design-pattern analysis, and no multiple acceptable references. Without such validation, the generation results support only the weaker conclusion that model outputs differ from the reference text.
- [Section III-B, Table I] The classification experiment has extremely small per-cell sample sizes (e.g., Java Easy n=14, Java Medium n=7, Python Easy n=9, Python Hard n=14, total 67 files for the 'All' row). No significance tests, confidence intervals, or effect sizes are reported. As a result, claims that GPT-4o and Llama-31-70B outperform other models, and that performance declines from Java to Python and from Easy to Hard, are unsupported: the difference between 71.43% and 64.29% on Java Easy is a single file. The paper should report paired or per-pattern statistical tests, or at minimum confidence intervals, and discuss the statistical power available with these sample sizes.
- [Section III-C, Tables II and III] The comparisons between providing and withholding design-pattern knowledge show inconsistent effects across models and patterns, yet no statistical analysis is provided to determine whether any of the differences are meaningful. For example, in Table III, GPT-4o's Singleton CS with pattern is 36.15 vs 32.77 without, a 3.38-point difference on an unknown number of test items, whereas for Observer the pattern-provided condition is lower. The text acknowledges such exceptions but still concludes that providing design-pattern knowledge improves performance. A proper statistical treatment (e.g., paired tests across items, mixed-effects models with pattern and condition as factors) is needed to support this claim.
- [Section III-A-2/3 and Table I] The experimental setup is under-specified to the point of hampering reproducibility. No prompts are shown, no decoding parameters (temperature, top-p, max tokens) are reported, no number of runs or random seeds are given, and the 'None' entries in Table I are not explained (did the model fail to produce a valid output, or was that condition not run?). For the generation tasks, the process of randomly removing 'three separate lines of effective code' and using GPT-4 to write function descriptions is not described with enough detail to allow replication. These omissions should be corrected.
minor comments (6)
- [Section III-A-4] The sentence 'difflib's SequenceMatcher, which employs the Ratcliff/Obershelp algorithm to finds the Longest Matching Subsequence' contains a grammatical error and is technically imprecise: Ratcliff/Obershelp is not equivalent to exact longest common subsequence, so the description should be corrected.
- [Table I] The 'None' entries in Table I should be defined in the caption. If they indicate that a model failed to produce a classification, that is itself an important result and should be discussed; if they indicate that the condition was not run, that should be stated explicitly.
- [Tables II and III] The caption notation is confusing. In Table II the slash separates Line Completion from Function Generation, while in Table III it separates Code Similarity from Edit Similarity. Please make the column labels unambiguous (e.g., 'CS (LC/FG)' and 'ES (LC/FG)') or add a note in the table header.
- [Section II-B] The text states that existing studies 'often use benchmarks such as CodeXGLUE [10]-[12]', but references [10] (Codex), [11] (CodeT), and [12] (code smells) are not CodeXGLUE papers. Either replace the references with the actual CodeXGLUE citation or revise the claim.
- [Section II-A] Reference [4] is cited to support claims about the LLaMA 3.1 series, but [4] is a paper on Llama Scope (sparse autoencoders), not the LLaMA 3.1 model. The citation should be replaced with the appropriate technical report or model card.
- [Conclusion] The paper would benefit from an explicit threats-to-validity section. The small dataset, single-reference metrics, and lack of statistical testing are important limitations that should be acknowledged in the main text, not merely in the plan for future work.
Circularity Check
No significant circularity: the paper reports direct empirical measurements of LLM classification accuracy and code similarity, with no fitted parameters, derived predictions, or load-bearing self-citations.
full rationale
This paper is an empirical study, not a derivation. It measures three things: design-pattern classification accuracy (Table I), line-completion and function-generation similarity (Tables II and III), and qualitative patterns in misclassification heatmaps. There is no step in which an output is constructed from its own input. The classification accuracy is computed by simple accuracy on manually selected repositories; the generation metrics are Code Similarity and Edit Similarity computed against reference implementations via difflib and edit distance. These are operational choices, and one could question whether string similarity is a valid proxy for design-pattern conformance, but that is a validity concern, not a circularity concern: the metric is not defined in terms of the conclusion, nor is any parameter fitted to the data and then renamed as a prediction. The paper cites several prior works by the authors (Chain-of-Action, Conv-COA, Codev-Bench), but these appear only as background motivation in the introduction and related work, and none of the paper's measurements or conclusions depends on accepting those prior results. No uniqueness theorem is imported, no ansatz is smuggled in via self-citation, and no known result is renamed as a new framework. The claim that LLM biases 'significantly affect the reliability of downstream tasks' is an interpretive leap from the measured low similarity scores, but the measurements themselves are self-contained and externally checkable against the paper's dataset and code. Under the stated rules, this is a non-finding: the honest verdict is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The manual assignment of each selected repository to one of 12 design patterns is a correct ground truth.
- domain assumption Code similarity (difflib SequenceMatcher) and edit similarity are treated as proxies for pattern conformance and code quality.
- domain assumption GPT-4-generated natural-language descriptions of functions are accurate, unbiased stand-ins for the original code's intent.
- domain assumption The three complexity tiers (easy, moderate, difficult) reflect real difficulty differences that generalize beyond the selected files.
Cite this review
Pith. "Pith review of Do Code LLMs Understand Design Patterns?." pith.science (2026). https://pith.science/paper/IKORDNDE
@misc{pith2026250104835,
author = {Pith},
title = {Pith review of: Do Code LLMs Understand Design Patterns?},
year = {2026},
howpublished = {\url{https://pith.science/paper/IKORDNDE}},
note = {Machine review of arXiv:2501.04835}
}
read the original abstract
Code Large Language Models (LLMs) demonstrate great versatility in adapting to various downstream tasks, including code generation and completion, as well as bug detection and fixing. However, Code LLMs often fail to capture existing coding standards, leading to the generation of code that conflicts with the required design patterns for a given project. As a result, developers must post-process to adapt the generated code to the project's design norms. In this work, we empirically investigate the biases of Code LLMs in software development. Through carefully designed experiments, we assess the models' understanding of design patterns across recognition, comprehension, and generation. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Ale- man, F. L., ...,, et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
- [2]
- [3]
-
[4]
He, Z., Shu, W., Ge, X., Chen, L., Wang, J., Zhou, Y ., ... , et al. (2024). Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders. arXiv preprint arXiv:2410.20526
arXiv 2024
-
[5]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023
-
[6]
Pan, Z., Luo, H., Li, M. and Liu, H., 2024. Chain-of-action: Faithful and multimodal question answering through large language models. arXiv preprint arXiv:2403.17359
arXiv 2024
-
[7]
Pan, Z., Luo, H., Li, M. and Liu, H., 2024. Conv-coa: Improving open- domain question answering in large language models via conversational chain-of-action. arXiv preprint arXiv:2405.17822
arXiv 2024
-
[8]
Yi: Open foundation models by 01
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024
arXiv 2024
Show all 17 references
-
[9]
Qwen2 technical report
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024
2024 arXiv
-
[10]
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
2021 arXiv
-
[11]
G., Chen, W
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J. G., Chen, W. (2022). Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397
2022 arXiv
-
[12]
L., Majumder, S
Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., Santos, J. C. (2022, October). An empirical study of code smells in transformer-based code generation techniques. In 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) (pp. 71...
2022
-
[13]
(2024, February)
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y ., et al. (2024, February). Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (pp. 1-12)
2024
-
[14]
Hashtroudi, S., Shin, J., Hemmati, H., Wang, S. (2023). Automated test case generation using code models and domain adaptation. arXiv preprint arXiv:2308.08033
2023 arXiv
-
[15]
B., Cassano, F., Lamy-Poirier, J., Tazi, N., et al
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., et al. (2024). Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173
2024 arXiv
-
[16]
B., Zi, Y ., Muennighoff, N., Kocetkov, D., Mou, C., et al
Li, R., Allal, L. B., Zi, Y ., Muennighoff, N., Kocetkov, D., Mou, C., et al. (2023). Starcoder: may the source be with you!. arXiv preprint arXiv:2305.06161
2023 arXiv
-
[17]
and Li, Y ., 2024
Pan, Z., Cao, R., Cao, Y ., Ma, Y ., Li, B., Huang, F., Liu, H. and Li, Y ., 2024. Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?. arXiv preprint arXiv:2410.01353
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.