Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a four-language smelly-code benchmark, GPT-4.0 is far more precise than DeepSeek-V3 at detecting code smells, though both models miss most of them.

desk verdict The released multilingual dataset is a real contribution, but the paper's headline precision/recall numbers do not reconcile across its own category-level and type-level tables, and the cost analysis is wrong in both arithmetic and pricing facts. read the letter →

arxiv 2504.16027 v1 pith:CEX5PKFG submitted 2025-04-22 cs.SE cs.AIcs.LGcs.PL

Ahmed R. Sadik , Siddhata Govind This is my paper · ORCID
classification cs.SEcs.AIcs.LGcs.PL
keywords codesmelldetectionlargelanguagemodelsGPT-4.0DeepSeek-V3precisionrecallF1multilingualdatasetSonarQubecost-effectiveness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a fair, reproducible way to compare large language models on code smell detection and to show what that comparison yields. On a deliberately identical restaurant-management codebase written in Java, Python, JavaScript, and C++, it finds that GPT-4.0 flags smells with far fewer false positives than DeepSeek-V3, with precision 0.79 versus 0.42 and F1-score 0.54 versus 0.35. Both models nonetheless miss a large share of the annotated smells, with recall 0.41 for GPT-4.0 and 0.31 for DeepSeek-V3, so neither acts as a complete detector on its own. The paper also argues that LLM-based detection is complementary to rule-based tools like SonarQube and that a hybrid workflow is the practical route.

What carries the argument

The load-bearing object is a purpose-built evaluation setup: a restaurant-management system intentionally implemented once in Java, Python, JavaScript, and C++, with known code smells manually annotated to form the ground truth. Model inputs are the cleaned code, with annotations stripped, sent through a standardized prompt asking for detection and categorization; outputs are scored as true positives, false positives, and false negatives against those annotations, then rolled up into precision, recall, and F1 at overall, category, type, and language levels. The code-smell taxonomy, organized into Bloaters, Dispensables, Couplers, Object-Orientation Abusers, and Change Preventers, supplies the categories into which the counts are sorted.

What would settle it

Re-run the benchmark on the released dataset with the exact prompt and publish each model's raw responses alongside a second, independent annotation of the same code; if the second annotation changes the true-positive, false-positive, and false-negative counts materially, or if the visible matching rules penalize DeepSeek-V3's wording while accepting GPT-4.0's, the reported precision gap is an artifact.

Watch

Extended reading notes

Core claim

The central claim is that, on the authors' multilingual smelly-code dataset, GPT-4.0 is substantially more precise than DeepSeek-V3 at detecting and categorizing code smells, while both models have low recall. GPT-4.0 reports fewer false positives across every smell category, reaching perfect precision on hard-to-detect classes such as Change Preventers at the cost of near-zero recall, and it dominates on common structural smells like Large Class, Message Chains, and Inappropriate Intimacy. DeepSeek-V3 occasionally finds smells that GPT-4.0 misses, notably Refused Bequest in Python, but buries them in a much larger number of false positives. The paper further claims that GPT-4.0's per-query cost is higher but its output quality justifies the expense where precision matters, while DeepSeek-V3's fixed complexity-based pricing suits cheap broad screening; neither model yet matches the determinism of SonarQube, so the authors recommend combining LLMs with static analysis.

Load-bearing premise

The headline numbers rest on the assumption that the manually annotated smells in the small restaurant-management dataset are a correct and complete ground truth, and that the standardized prompt and the unshown rules for matching model output to annotations treat both models fairly.

Editorial extensions

If this is right

  • Teams that want few false alarms should prefer GPT-4.0 over DeepSeek-V3 for code-smell screening on this kind of codebase.
  • Because recall is low for both models, a production workflow should pair an LLM with a rule-based analyzer such as SonarQube to catch smells the models miss.
  • Detection quality varies by language and smell type, so per-type or per-language prompts and thresholds are needed rather than one generic setting.
  • GPT-4.0's token-based pricing makes it costlier at scale, while DeepSeek-V3's fixed complexity tiers make it the cheaper bulk screener, accepting more false positives.
  • The released dataset and evaluation matrix give a reusable benchmark for future models, so the comparison can be extended as new LLMs appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The low recall of both models may partly reflect a conservative reporting style: GPT-4.0 seems to emit only high-confidence smell labels, so tuning prompts or temperature to lower the reporting threshold could raise recall, though the effect on precision is unknown.
  • Because the dataset is one small synthetic system, the absolute numbers should not be read as general detection rates; the relative ordering of the models is the more transportable result.
  • The paper's unshown prompt and matching rules are the true experiment; publishing them and the raw model outputs would let others verify whether the precision gap is real or an artifact of how near-miss labels were counted.
  • A natural next test is an ensemble: let DeepSeek-V3 propose candidate smells for high recall, then have GPT-4.0 confirm or reject them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper benchmarks two large language models, GPT-4.0 and DeepSeek-V3, for automated code smell detection on a custom multilingual restaurant-management dataset spanning Java, Python, JavaScript, and C++. It reports precision, recall, and F1-score at three granularities (overall, category, and type), analyzes language-specific performance, compares costs, and contrasts LLM-based detection with SonarQube. The headline empirical claim is that GPT-4.0 achieves substantially higher precision (0.79 vs. 0.42) while both models have low recall (0.41 and 0.31).

Significance. If its measurements were trustworthy, the paper would provide a useful practitioner-oriented comparison of two LLMs for code smell detection across four languages. The released dataset and the three-level evaluation design are genuine assets. However, the paper's central measurements are internally inconsistent: the category-level and type-level counts of true positives, false positives, and false negatives do not reconcile, and the prompt, output-matching rules, repeated runs, and ground-truth validation are all missing. These gaps are load-bearing because every precision, recall, and F1 value in the paper is derived from those TP/FP/FN counts. The paper is therefore not currently reproducible or verifiable, despite the useful dataset and multi-level framing.

major comments (5)
  1. [5.2–5.3, Tables 3 and 6] The category-level and type-level tables are mutually inconsistent for both models. Summing the five category rows of Table 3 for GPT-4.0 gives TP=168, FP=46, FN=291, whereas summing the type-level rows of Table 6 gives TP=197, FP=49, FN=281 — a discrepancy of 29 true positives. For DeepSeek-V3, Table 3 sums to TP=117, FP=121, FN=342, while Table 6 sums to TP=146, FP=195, FN=332 — a 74-false-positive discrepancy. These are integer counts, not rounding artifacts, and both tables are supposed to be different cuts of the same detection results. At least one set of counts is therefore not a reliable measurement, and because all precision/recall/F1 values derive from these counts, the headline comparison in Table 2 is not supported.
  2. [Section 4, 'Detection and Evaluation Methods'] The 'standardized prompt' is never shown, and the rules used to convert model outputs into TP/FP/FN counts are not described. Without the exact prompt and the string or semantic matching procedure, another researcher cannot reproduce the evaluation, and the comparison may be biased if the matching favored one model. This absence is especially serious because the entire ranking of the two models rests on the TP/FP/FN assignment.
  3. [Section 5.1 and Section 8] No repeated runs, confidence intervals, or variance information are reported for either model. LLM outputs are stochastic, and Section 8 itself acknowledges that 'the same input may yield different results across multiple runs.' A single run per model cannot support the claim that GPT-4.0 'achieves significantly higher precision' or any other quantitative ordering without an estimate of run-to-run variability.
  4. [Section 3, Dataset] The ground truth is self-authored: the authors created the dataset, annotated it, and then evaluate against those annotations. No independent validation, inter-rater agreement, or external benchmark is provided. The correctness and completeness of these annotations are foundational to every metric, so the evaluation is anchored to an unvalidated reference rather than an established gold standard.
  5. [Section 7, Cost Analysis] The cost comparison rests on unsupported assumptions: the 5–10 tokens-per-line estimate, the 250-token response length, the 'low complexity' classification of the script, and the claim that DeepSeek-V3 uses complexity-based rather than token-based pricing. These assumptions are not justified by citations or measurements, and the stated DeepSeek-V3 pricing tiers are not sourced. Since cost-effectiveness is one of the paper's contributions, this analysis needs either real pricing data or clearly labeled illustrative assumptions with sensitivity analysis.
minor comments (5)
  1. [Throughout] The model name is used inconsistently as 'DeepSeek' and 'DeepSeek-V3'; the paper should use one name consistently.
  2. [Sections 10.1 and 10.2] Both subsections say 'Table 9 presents a detailed comparison...' but refer to different appendix material; the table numbering and cross-references need to be corrected.
  3. [Table 6 caption] The caption calls the table 'Type-Language Level,' but the table has no language column and appears to aggregate across languages; the caption should be clarified.
  4. [Abstract and Section 7] Describing DeepSeek-V3 as using 'pattern-matching techniques' is inaccurate for an LLM and should be reworded.
  5. [Section 2 and References] There are several formatting errors in references and inline citations, including missing spaces before citation markers such as 'signifyWaseem et al. [2023]Alves et al. [2024].'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark compares blind model outputs to an independently stated ground truth; no fitted parameter or self-citation is load-bearing.

full rationale

The paper's central claim is a direct empirical measurement: GPT-4.0 and DeepSeek-V3 are given cleaned code stripped of the manually annotated smells (Section 4), and their outputs are scored against the curated ground truth via standard TP/FP/FN definitions. Nothing in the model outputs is fed back into the construction of the ground truth or into the evaluation formulas, so there is no 'prediction' that reduces to its input by construction. Precision, recall, and F1 are defined by textbook formulas (Section 4, Table 1), not derived from the dataset itself. The authors' self-citations (Sadik et al. 2023a,b) appear only in the introduction as motivating context, not as evidence supporting the benchmark results, so they are not load-bearing. The ground truth is self-authored and not externally validated, which is a validity/generalizability limitation rather than circularity. Separately, the tables contain arithmetic inconsistencies between category-level and type-level TP/FP/FN counts (e.g., summing Table 6 for GPT-4.0 gives TP=197, FP=49, FN=281, while summing Table 3 categories gives TP=168, FP=46, FN=291); these would be correctness or reproducibility defects, but they are not instances of a prediction being equivalent to its input. Accordingly, no circular step is identifiable from the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper has no fitted scientific constants, but its cost-effectiveness claim rests on three hand-chosen numerical assumptions. The evaluation itself rests on an unvalidated self-authored ground truth and an undisclosed prompt and matching procedure, plus a factually incorrect model of DeepSeek-V3 pricing.

free parameters (3)
  • Token-per-line estimate = 5 to 10 tokens per line
    Section 7 uses 570 to 1,140 tokens for a 114-line Python file to estimate GPT-4.0 cost; not measured.
  • GPT-4.0 response length assumption = 250 output tokens
    Used in the Section 7 cost formula; not justified or measured.
  • DeepSeek-V3 complexity tier = low complexity
    The paper assigns Pizza.py to a low-complexity tier under a pricing model that does not match DeepSeek-V3's actual token-based API pricing.
assumptions (4)
  • domain assumption The manually annotated code smells in the restaurant-management dataset are correct and complete ground truth.
    Sections 3 and 4: all TP/FP/FN counts and precision/recall/F1 values depend on the correctness of these annotations, which are not validated against external tools or experts.
  • domain assumption The 'standardized prompt' produces comparable, parseable outputs from both models.
    Section 4 states a standardized prompt is used, but the prompt text and output parsing or matching rules are not provided; the recall gap could be an artifact of how outputs were matched to ground-truth smells.
  • ad hoc to paper DeepSeek-V3 pricing is complexity-based with fixed tiers.
    Section 7 assumes DeepSeek-V3 charges $0.01 to $0.20 per request based on complexity; DeepSeek-V3 is an LLM with token-based API pricing, so this assumption is factually unsupported.
  • domain assumption The four language implementations are structurally equivalent enough for cross-language comparison.
    Section 3 constructs a generic restaurant system in Java, JavaScript, Python, and C++; differences in method and attribute counts and smell distributions across languages are attributed to language idioms, but equivalence is not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3." pith.science (2026). https://pith.science/paper/CEX5PKFG

@misc{pith2026250416027,
  author       = {Pith},
  title        = {Pith review of: Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CEX5PKFG}},
  note         = {Machine review of arXiv:2504.16027}
}
read the original abstract

Determining the most effective Large Language Model for code smell detection presents a complex challenge. This study introduces a structured methodology and evaluation matrix to tackle this issue, leveraging a curated dataset of code samples consistently annotated with known smells. The dataset spans four prominent programming languages Java, Python, JavaScript, and C++; allowing for cross language comparison. We benchmark two state of the art LLMs, OpenAI GPT 4.0 and DeepSeek-V3, using precision, recall, and F1 score as evaluation metrics. Our analysis covers three levels of detail: overall performance, category level performance, and individual code smell type performance. Additionally, we explore cost effectiveness by comparing the token based detection approach of GPT 4.0 with the pattern-matching techniques employed by DeepSeek V3. The study also includes a cost analysis relative to traditional static analysis tools such as SonarQube. The findings offer valuable guidance for practitioners in selecting an efficient, cost effective solution for automated code smell detection

Figures

Figures reproduced from arXiv: 2504.16027 by the authors.

Figure 1
Figure 1. LLM-based collaboration in software engineering. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Codesmells Taxonomy. In software engineering, maintaining high code quality is essential for developing robust, maintainable, and scalable systems Sadik et al. [2023b]. As software projects evolve, they often accumulate design inefficiencies or anomalies, commonly referred to as code smells Wu et al. [2024]. These code smells serve as indicators of deeper structural issues within the codebase, potentially leading to… view at source ↗
Figure 3
Figure 3. Generic class diagram representing the generic system implemented in Java, JavaScript, Python, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Heatmaps representing various metrics across Java, JavaScript, Python, and C++ implementa [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of GPT-4.0 and DeepSeek-V3 in code smell detection. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of GPT-4.0 vs. DeepSeek at the category level: (a) TP, FP, FN difference heatmap, [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison of GPT-4.0 vs. DeepSeek at the type level: (a) TP, FP, FN difference heatmap, [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Precision difference (GPT-4.0 - DeepSeek) by code smell and language. Red indicates GPT-4.0’s [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Recall difference (GPT-4.0 - DeepSeek) by code smell and language. Higher recall in GPT-4.0 [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: F1-score difference (GPT-4.0 - DeepSeek) by code smell and language. Red areas indicate [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mitigating LLM Sycophancy in Code Smell Detection Using Evidence-Guided Reasoning Prompts

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Sycophancy bias makes LLM code-smell detectors highly prompt-sensitive (DFR up to 72%, FAR >90%); evidence-first EGDP prompting reduces flips to ~12–26% and restores F1.

  2. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions

    cs.SE 2026-02 conditional novelty 6.0 of 10

    Most MCP tool descriptions (97.1%) contain quality smells, and augmenting them improves agent success by a median of 5.85 percentage points at a 67.46% increase in execution steps.

  3. Are We SOLID Yet? An Empirical Study on Prompting LLMs to Detect Design Principle Violations

    cs.SE 2025-09 conditional novelty 6.0 of 10

    LLMs vary sharply in detecting SOLID violations, GPT-4o Mini leads, and no single prompt strategy wins, with accuracy falling as code complexity rises.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages · cited by 3 Pith papers

  1. [1]

    Coding by design: Gpt-4 empowers agile model driven development

    Ahmed R Sadik, Sebastian Brulin, and Markus Olhofer. Coding by design: Gpt-4 empowers agile model driven development. arXiv preprint arXiv:2310.04304 , 2023a. Ahmed R Sadik, Antonello Ceravola, Frank Joublin, and Jibesh Patra. Analysis of chatgpt on source code. arXiv preprint arXiv:2306.00597 , 2023b. Junda He, Christoph Treude, and David Lo. Llm-based m...

    arXiv

  2. [3]

    Evaluating large language models in detecting test smells.arXiv preprint arXiv:2407.19261 ,

    Keila Lucas, Rohit Gheyi, Elvys Soares, Márcio Ribeiro, and Ivan Machado. Evaluating large language models in detecting test smells.arXiv preprint arXiv:2407.19261 ,

    arXiv

  3. [4]

    How propense are large language models at producing code smells? a benchmarking study

    Alejandro Velasco, Daniel Rodriguez-Cardenas, David N Palacio, Luftar Rahman Alif, and Denys Poshyvanyk. How propense are large language models at producing code smells? a benchmarking study. arXiv preprint arXiv:2412.18989 ,

    arXiv

  4. [5]

    Are sonarqube rules inducing bugs? In 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER), pages 501–511

    Valentina Lenarduzzi, Francesco Lomio, Heikki Huttunen, and Davide Taibi. Are sonarqube rules inducing bugs? In 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER), pages 501–511. IEEE,

    work page 2020

  5. [6]

    Code smells

    Refactoring.Guru. Code smells. https://refactoring.guru/refactoring/smells. Accessed: 2025-02-

    work page 2025

  6. [11]

    Token-hungry, yet precise: Deepseek r1 highlights the need for multi-step reasoning over speed in math.arXiv preprint arXiv:2501.18576 ,

    Evgenii Evstafev. Token-hungry, yet precise: Deepseek r1 highlights the need for multi-step reasoning over speed in math.arXiv preprint arXiv:2501.18576 ,

    arXiv

  7. [13]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437 ,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437 ,

    arXiv

  8. [2006]

    Code smells in machine learning systems

    Jiri Gesi, Siqi Liu, Jiawei Li, Iftekhar Ahmed, Nachiappan Nagappan, David Lo, Eduardo Santana de Almeida, Pavneet Singh Kochhar, and Lingfeng Bao. Code smells in machine learning systems. arXiv preprint arXiv:2203.00803 ,

    arXiv

Show all 13 references
  1. [2016]

    Causes, impacts, and detection approaches of code smell: a survey

    Md Shariful Haque, Jeff Carver, and Travis Atkison. Causes, impacts, and detection approaches of code smell: a survey. InProceedings of the 2018 ACM Southeast Conference , pages 1–8,

    2018

  2. [2018]

    Ahmed R. Sadik. Smelly code dataset - python/java/javascript/c++, 2025a. URLhttps://doi.org/10. 5281/zenodo.14989674. Ahmed R. Sadik. Smelly code dataset - python/java/javascript/c++, 2025b. URLhttps://github. com/HRI-EU/SmellyCodeDataset. Accessed: 2025-02-07. Thanis Paiva, A...

  3. [2019]

    Yiyi Tang, Ziyan Xiao, Xue Li, Qiwen Fang, Qingpeng Zhang, Daniel Yee Tak Fong, Francisco Tsz Tsun Lai, Celine Sze Ling Chui, Esther Wai Yin Chan, Ian Chi Kei Wong, et al. Large language model in medical information extraction from titles and abstracts with prompt engineering ...

    2024

  4. [2024]

    Chatgpt as a software development bot: a project-based study.arXiv preprint arXiv:2310.13648 ,

    13 Muhammad Waseem, Teerath Das, Aakash Ahmad, Peng Liang, Mahdi Fehmideh, and Tommi Mikkonen. Chatgpt as a software development bot: a project-based study.arXiv preprint arXiv:2310.13648 ,

    arXiv

  5. [2025]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.