REVIEW 4 major objections 6 minor 2 references
Automatic Identification of Machine Learning-Specific Code Smells
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read MLpylint flags ML code smells at 87.9% precision.
desk verdict A real tool and large-scale prevalence data, but the headline precision numbers are inflated by author-produced labels and no recall measurement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is MLpylint's static analysis pipeline: a runner feeds Python files to an AST-parsing library that builds an abstract syntax tree and infers object types, even for external machine-learning libraries; then a set of checkers, organized in a visitor pattern, applies dedicated detection logic for each smell. Each smell is classified against five criteria: it must have a description, be specifically relevant to Python ML, be statically detectable, have a distinct code pattern, and have a precise, unambiguous pattern. That classification splits smells into definitive 'code smells' and broader 'code smell advices' that require developer judgment.
What would settle it
Ask two independent teams to manually label the same stratified sample of flagged instances from the 160 analyzed repositories, and compare their labels with the authors' labels. If inter-rater agreement is low (for instance, a Cohen's kappa below 0.6) or if the independent labels yield an overall precision materially below 87.9%, the paper's conclusion that the tool reliably identifies code smells would not be supported.
Extended reading notes
Core claim
The central discovery is that a catalogue of 22 ML-specific code smells can be operationalized into statically detectable patterns: 20 of them pass the paper's criteria and are split into 14 unambiguous 'code smells' and 6 broader 'code smell advices.' MLpylint implements these patterns using an abstract syntax tree parser with object inference, and running it over 160 open-source repositories containing 10 million lines of Python yields 5,380 detected smells in about 36 minutes. In a manually validated stratified sample, the tool achieves 100% precision on code smells and 73.6% on advice, giving an overall precision of 87.9%. The paper also reports that practitioners found the tool easy to use, effective, and time-saving.
Load-bearing premise
The reported precision of 87.9% rests on the assumption that the authors' manual review of a stratified sample is a reliable ground truth for whether a flagged pattern is truly a code smell, and no independent ground truth or inter-rater reliability check is reported.
Editorial extensions
If this is right
- MLpylint can be embedded in development workflows such as pre-commit hooks and CI/CD pipelines, allowing ML teams to catch these smells before changes are merged.
- The 14 code smells and 6 advices form a concrete checklist for Python ML code review, validated on a large set of real projects.
- The prevalence data points to common problem areas: uncontrolled randomness appears 1,796 times, and two advisory patterns each appear over 1,000 times across the analyzed projects.
- The tool's speed, about 13 seconds per project, makes it feasible for routine scanning of large codebases.
- The authors report that a plugin is already in progress to integrate the smell checks into an existing Python linting workflow.
Reading between the lines
- If the precision holds beyond the sampled projects, the 20-smell catalogue could serve as a shared benchmark for other ML-specific static analysis tools, giving the field a common reference set for comparison.
- The wide gap between 100% precision on definitive smells and 73.6% on advisory smells suggests a useful trade-off: unambiguous patterns can be automated and trusted, while fuzzy advice still needs human judgment.
- An independent replication using multiple raters for the manual validation would strengthen confidence in the absolute precision figures, since the reported numbers use the authors' own manual labels as ground truth.
- As the smells are tied to external libraries such as NumPy, Pandas, PyTorch, and TensorFlow, extending the tool to additional ML libraries and to other workflow contexts like notebooks is a natural next step implied by the paper's architecture.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents MLpylint, a static analysis tool that detects machine-learning-specific code smells in Python applications, built on Astroid and targeting 20 smells adopted from Zhang et al. The authors follow a Design Science process, run the tool on 160 open-source GitHub repositories, manually validate a stratified sample of detections, and report 100% precision for Code Smells, 73.6% for Code Smell Advice, and an overall precision of 87.9%. They also report a survey of 15 Ericsson practitioners and conclude that the tool is effective and useful.
Significance. If the reported precision and usefulness findings are credible, the paper would provide an evaluated, publicly available tool for an under-served area: static analysis of ML-specific code smells. Strengths include grounding the smell catalogue in external prior work, an unusually large analyzed corpus (160 repositories, over 10 million lines of Python), and packaged artifacts (source code, repository list, and PyPI package). The stratified sampling protocol is a reasonable start for performance evaluation. However, the central effectiveness claim depends on precision numbers derived from author-assigned manual labels, and the usefulness claim rests on a small, single-organization survey without a control condition. The significance is therefore conditional on addressing these validation gaps.
major comments (4)
- [5.1, Tables 6-7] The precision figures are computed against manual labels assigned solely by the authors, with no inter-rater reliability check and no independent adjudication. For CS14, the text explicitly states that whether a seed configuration constitutes a smell "is up to the developer to determine," yet all 219 sampled CS14 instances are counted as true positives. Since CS14 accounts for 62.5% of the CS stratum, this labeling decision alone drives the reported 100% CS precision and substantially inflates the overall 87.9% figure. I recommend independent annotators, an explicit decision rule for ambiguous cases, and a report of inter-rater agreement or a sensitivity analysis that excludes CS14.
- [5.1, Table 8] The RQ3 evaluation reports precision only; no recall or false-negative analysis is provided. A tool that flags only a few easy patterns would show high precision while missing most real smells, so the abstract's claim that "the results indicate the effectiveness" is not supported by precision alone. To support that claim, the authors should report recall on a labeled benchmark or at least compare against dslinter, which is mentioned in Section 6 but never evaluated experimentally.
- [5.2] The usefulness conclusion rests on a survey of 15 Ericsson practitioners who received a demonstration and then used the tool for a month, with no control condition, no validated questionnaire, and no independent analysis of the responses. The open-ended feedback in Table 9 is useful qualitatively, but it includes suggestions for improvement and a request for more data, and it does not by itself establish claims such as Q5 (reducing time and effort). I recommend presenting this as a preliminary feasibility study and tempering the conclusions accordingly.
- [3.1, Table 2] The classification of 20 of the 22 smells into Code Smell and Code Smell Advice is based on the authors' application of criteria CSC1-CSC5, but Table 2 uses symbols (↭, →, -) that are never defined, and no independent expert check of the classification is reported. Because this CS/CSA split is the basis for the tool's checker design and for the separate precision numbers in Tables 4-8, the symbols should be defined and the classification rationale should be made auditable.
minor comments (6)
- [3.2] The manuscript shows a '3.2 Solution Design' heading with no body text before '3.3 Evaluation'; this should either be filled in or removed, as the current text jumps from problem identification to evaluation.
- [Table 2] Add a legend for the symbols used in Table 2; the current marks cannot be interpreted without one.
- [3.3] The repository selection section lists six search queries and criteria C1-C5, but it does not report how many repositories each query returned or how the final 160 were chosen; include the screening numbers for transparency.
- [5.1, Table 3] The metric labeled 'Code Coverage' is the percentage of Python files among all files, not test coverage or AST coverage; consider using a less ambiguous term such as 'Python file share'.
- [5.2, Figure 3] Figure 3 is described but not visible in the submitted text; ensure the figure is included with labeled response scales, because the percentages in Section 5.2 are rounded and cannot be verified otherwise.
- [Throughout] The text contains several apparent OCR artifacts, such as '!eld' for 'field' and 'O"ermann' for 'Öffermann'; these should be corrected in the final version.
Circularity Check
Precision is measured against the same smell definitions used to build the detector, and ambiguous CS14 cases are all counted as true positives, so the 87.9% effectiveness claim is partly self-confirming.
-
self definitional
[Section 3.1, Table 1 (CSC criteria); Section 5.1, Tables 6-8 (validation)]
"CSC5 Precision of Pattern: The code smell exhibits a clear and unambiguous pattern."
The CSC1-CSC5 criteria are used both to select and classify the 20 smells from Zhang et al. and to program the MLpylint checkers. For 'Code Smells', Section 4 defines them as clear, unambiguous patterns whose detection is precise, so a true positive is effectively defined as a match to the pattern the tool was built from. The reported 100% CS precision in Table 8 is therefore a self-consistency check against the tool's own specification. The CSA detections did yield false positives, so the validation is not wholly vacuous, but the headline CS precision and the conclusion that the tool can reliably identify code smells rest on the definitional part of the evaluation.
-
other
[Section 5.1, CS14 discussion and Table 7]
"As a result, it is up to the developer to determine if this constitutes a code smell in their particular scenario, urging a thoughtful evaluation to uphold the standard of code quality."
The paper says the CS14 randomness-seed pattern is ambiguous and that whether it is a smell depends on the developer, yet Table 7 records all 219 sampled CS14 instances as true positives with zero false positives. The manual label for the most frequent smell was resolved in the tool's favor despite the paper's own admitted ambiguity. This shows the validation labels are anchored to the detector's decisions rather than independently adjudicated, which inflates the precision figures.
full rationale
The central effectiveness claim rests on Table 8's precision numbers, which are computed from manual validation of a stratified sample (Tables 6-7). Those labels are not fully independent: the same CSC1-CSC5 criteria used to implement the tool are used to decide whether a flagged pattern is a smell, and the CS14 case shows ambiguous cases were counted as true positives. This makes the precision claim partially circular. However, the smell catalogue itself comes from external prior work (Zhang et al.), no numerical parameter is fitted to the evaluation data, and the usefulness survey and runtime measurements provide non-circular evidence. The self-citation to the first author's thesis [8] is not load-bearing: the load-bearing smell definitions are external. Taking all of this into account, the paper has partial self-definitional circularity in its headline precision validation, but the central claim still has independent content.
Assumptions & free parameters
assumptions (4)
- domain assumption The 22 ML-specific code smells catalogued by Zhang et al. represent the relevant set of Python ML code smells.
- ad hoc to paper Code smell criteria CSC1-CSC5 in Table 1 are sufficient to classify smells as statically detectable and to split them into Code Smell and Code Smell Advice.
- domain assumption Manual labeling by the authors of sampled detections is a reliable ground truth for true and false positives.
- domain assumption Astroid-based static analysis, with external libraries installed in the environment, is sufficient to detect the selected ML-specific code smells.
Cite this review
Pith. "Pith review of Automatic Identification of Machine Learning-Specific Code Smells." pith.science (2026). https://pith.science/paper/T4GMOKCL
@misc{pith2026250802541,
author = {Pith},
title = {Pith review of: Automatic Identification of Machine Learning-Specific Code Smells},
year = {2026},
howpublished = {\url{https://pith.science/paper/T4GMOKCL}},
note = {Machine review of arXiv:2508.02541}
}
read the original abstract
Machine learning (ML) has rapidly grown in popularity, becoming vital to many industries. Currently, the research on code smells in ML applications lacks tools and studies that address the identification and validity of ML-specific code smells. This work investigates suitable methods and tools to design and develop a static code analysis tool (MLpylint) based on code smell criteria. This research employed the Design Science Methodology. In the problem identification phase, a literature review was conducted to identify ML-specific code smells. In solution design, a secondary literature review and consultations with experts were performed to select methods and tools for implementing the tool. We evaluated the tool on data from 160 open-source ML applications sourced from GitHub. We also conducted a static validation through an expert survey involving 15 ML professionals. The results indicate the effectiveness and usefulness of the MLpylint. We aim to extend our current approach by investigating ways to introduce MLpylint seamlessly into development workflows, fostering a more productive and innovative developer environment.
Figures
Reference graph
Works this paper leans on
-
[2017]
https://doi.org/10.1186/ s40411-017-0041-1 [20]Per Runeson and Martin Höst
On the evaluation of code smells and detection tools.Journal of Software Engineering Research and Development5, 1 (10 2017). https://doi.org/10.1186/ s40411-017-0041-1 [20]Per Runeson and Martin Höst. 2009. Guidelines for Conducting and Reporting Case Study Research in Software Engineering.Empirical Softw. Engg.14, 2 (apr 2009), 131–164. https://doi.org/1...
-
[2018]
Automatic Identi!cation of Machine Learning- Speci!c Code Smells
Review of Literature on Software Quality.88 (2018), 32–42. [3]Nicolás Cardozo. 2023. Prevalence of code smells in reinforcement learning projects. https://arxiv.org/abs/2303.10236 [4]Martin Fowler. 2002.Refactoring: improving the design of existing code. 256 pages. 31 https://doi.org/10.1007/3-540-45672-4\{_ [5]J. Gesi, S. Liu, J. Li, I. Ahmed, N. Nagappa...
arXiv 2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.