{"id":"04ff6028-92d6-4d4e-ab7b-0fbbf18739e1","arxiv_id":"2508.02541","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A static analysis tool called MLpylint detects 20 machine learning-specific code smells in Python code with a reported precision of 87.9% on manually checked detections.","lead":"This paper presents MLpylint, a tool that scans Python machine learning code for 20 known coding problems, called code smells, and reports them to developers. The authors tested it on 160 open-source projects and asked 15 industry machine learning engineers whether they found it useful.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Precision claims rest on author-produced labels that count ambiguous CS14 cases as true positives; absent recall and independent adjudication, the 87.9% precision does not establish effectiveness.","rationale":"The paper's central claim is that MLpylint is effective and useful for identifying ML-specific code smells. The quantitative backbone is the precision metrics in Tables 6-8, which are derived from manual validation of a stratified sample. The load-bearing assumption is that those manual labels are reliable ground truth. The paper provides no inter-rater reliability, no independent annotators, and no recall measurement. My reading highlights a concrete internal signal that this assumption is fragile: the discussion of CS14 explicitly states that whether a seed configuration is a smell requires developer judgment, yet all sampled CS14 instances are counted as true positives. That suggests the labeling process was lenient and oriented toward confirming the tool's output, which would inflate precision. This does not mean the tool is worthless; rather, the evidence as presented cannot support the strong effectiveness claim. The proposed test, independent blind annotation with agreement statistics, directly addresses the weakest link and would either substantiate or undermine the reported precision. The reader's verdict of CONDITIONAL is appropriate, so I recommend no change to that verdict.","tokens_in":15608,"tokens_out":2919,"duration_ms":36704,"concrete_test":"Randomly select the same 351 CS and 345 CSA samples from Tables 6-7; have two independent ML practitioners, blind to MLpylint's output and to each other, classify each instance as true/false positive and compute Cohen's kappa; then recompute precision, treating CS14 cases as true positive only if both annotators agree. If kappa < 0.6 or CS/CSA precision drops materially, the reported 87.9% precision and the effectiveness claim are not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central effectiveness claim (Section 5.1, Table 8) is precision-only. Precision is computed by manually validating a stratified sample (Tables 6-7), with the authors as sole annotators and no inter-rater reliability. Section 5.1 explicitly notes for CS14 that whether a seed configuration constitutes a smell \"is up to the developer to determine,\" yet all 219 sampled CS14 instances are labeled true positives. This shows the ground-truth labels are not independent and likely biased toward confirming the tool's detections, inflating the reported 100% CS precision. Additionally, no recall is measured: a tool that flags only easy, correct cases could show high precision while missing most real smells. Consequently, Tables 6-8 do not support the abstract's claim that \"the results indicate the effectiveness and usefulness of the MLpylint.\" Without baseline comparison to dslinter or an independent label set, the central claim is conditionally supported at best.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MLpylint, a static analysis tool that detects machine-learning-specific code smells in Python applications, built on Astroid and targeting 20 smells adopted from Zhang et al. The authors follow a Design Science process, run the tool on 160 open-source GitHub repositories, manually validate a stratified sample of detections, and report 100% precision for Code Smells, 73.6% for Code Smell Advice, and an overall precision of 87.9%. They also report a survey of 15 Ericsson practitioners and conclude that the tool is effective and useful.","tokens_in":15747,"tokens_out":3942,"duration_ms":49086,"significance":"If the reported precision and usefulness findings are credible, the paper would provide an evaluated, publicly available tool for an under-served area: static analysis of ML-specific code smells. Strengths include grounding the smell catalogue in external prior work, an unusually large analyzed corpus (160 repositories, over 10 million lines of Python), and packaged artifacts (source code, repository list, and PyPI package). The stratified sampling protocol is a reasonable start for performance evaluation. However, the central effectiveness claim depends on precision numbers derived from author-assigned manual labels, and the usefulness claim rests on a small, single-organization survey without a control condition. The significance is therefore conditional on addressing these validation gaps.","major_comments":[{"comment":"The precision figures are computed against manual labels assigned solely by the authors, with no inter-rater reliability check and no independent adjudication. For CS14, the text explicitly states that whether a seed configuration constitutes a smell \"is up to the developer to determine,\" yet all 219 sampled CS14 instances are counted as true positives. Since CS14 accounts for 62.5% of the CS stratum, this labeling decision alone drives the reported 100% CS precision and substantially inflates the overall 87.9% figure. I recommend independent annotators, an explicit decision rule for ambiguous cases, and a report of inter-rater agreement or a sensitivity analysis that excludes CS14.","section":"5.1, Tables 6-7"},{"comment":"The RQ3 evaluation reports precision only; no recall or false-negative analysis is provided. A tool that flags only a few easy patterns would show high precision while missing most real smells, so the abstract's claim that \"the results indicate the effectiveness\" is not supported by precision alone. To support that claim, the authors should report recall on a labeled benchmark or at least compare against dslinter, which is mentioned in Section 6 but never evaluated experimentally.","section":"5.1, Table 8"},{"comment":"The usefulness conclusion rests on a survey of 15 Ericsson practitioners who received a demonstration and then used the tool for a month, with no control condition, no validated questionnaire, and no independent analysis of the responses. The open-ended feedback in Table 9 is useful qualitatively, but it includes suggestions for improvement and a request for more data, and it does not by itself establish claims such as Q5 (reducing time and effort). I recommend presenting this as a preliminary feasibility study and tempering the conclusions accordingly.","section":"5.2"},{"comment":"The classification of 20 of the 22 smells into Code Smell and Code Smell Advice is based on the authors' application of criteria CSC1-CSC5, but Table 2 uses symbols (↭, →, -) that are never defined, and no independent expert check of the classification is reported. Because this CS/CSA split is the basis for the tool's checker design and for the separate precision numbers in Tables 4-8, the symbols should be defined and the classification rationale should be made auditable.","section":"3.1, Table 2"}],"minor_comments":[{"comment":"The manuscript shows a '3.2 Solution Design' heading with no body text before '3.3 Evaluation'; this should either be filled in or removed, as the current text jumps from problem identification to evaluation.","section":"3.2"},{"comment":"Add a legend for the symbols used in Table 2; the current marks cannot be interpreted without one.","section":"Table 2"},{"comment":"The repository selection section lists six search queries and criteria C1-C5, but it does not report how many repositories each query returned or how the final 160 were chosen; include the screening numbers for transparency.","section":"3.3"},{"comment":"The metric labeled 'Code Coverage' is the percentage of Python files among all files, not test coverage or AST coverage; consider using a less ambiguous term such as 'Python file share'.","section":"5.1, Table 3"},{"comment":"Figure 3 is described but not visible in the submitted text; ensure the figure is included with labeled response scales, because the percentages in Section 5.2 are rounded and cannot be verified otherwise.","section":"5.2, Figure 3"},{"comment":"The text contains several apparent OCR artifacts, such as '!eld' for 'field' and 'O\"ermann' for 'Öffermann'; these should be corrected in the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and the artifacts are a positive feature. The main risk is that the abstract's effectiveness claim is stronger than the evidence: precision-only validation with author-assigned labels and a small convenience-sample survey. I do not see a load-bearing error that would require rejection, but the validation must be strengthened or the claims carefully scaled back."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the MLpylint paper. The tool is real, public, and reasonably engineered: they take Zhang et al.'s 22 ML code smells, filter to 20 using explicit criteria, implement detection in Astroid, and run it over 160 GitHub repos with 10M LOC. That scale alone puts the prevalence numbers ahead of most prior work. The prevalence map and the practitioner feedback are genuinely new, and the decision to separate unambiguous Code Smells from broad Code Smell Advice is sensible.\n\nThe soft spot is the validation. The headline 87.9% precision (100% CS, 73.6% CSA) comes from manual review of a stratified sample, but the manual labels were produced by the authors with no inter-rater reliability check. The CS14 case is telling: Section 5.1 says whether a seed is set correctly enough \"is up to the developer to determine,\" yet all 219 sampled CS14 instances are counted as true positives. That is a bias toward confirming the tool, and it directly inflates the 100% figure. There is also no recall measurement, so we do not know how many real smells are missed, and no baseline comparison against dslinter, which already automated detection of these smells. The user survey is 15 people from one company after a demo; it supports usefulness as a soft signal, not much more.\n\nNone of this kills the artifact. The criteria contribution and the tool itself are worth having, and the external catalogue from Zhang et al. keeps the circularity burden low. But the paper's abstract and conclusion overstate what the data can support. The \"effectiveness and usefulness\" claim needs recall, independent or IRR-validated labels, and a comparison to existing tooling.\n\nMy take: send it to peer review. It is a solid empirical/engineering paper that needs major revision, not a desk reject. The authors have the tool and the data; the fixes are feasible. Just do not let the current precision numbers stand as they are.","headline":"A real tool and large-scale prevalence data, but the headline precision numbers are inflated by author-produced labels and no recall measurement.","tokens_in":16280,"tokens_out":1747,"would_cite":true,"duration_ms":19997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MLpylint flags ML code smells at 87.9% precision.","keywords":["code smells","machine learning","static code analysis","Python","software quality","technical debt","MLpylint"],"falsifier":"Ask two independent teams to manually label the same stratified sample of flagged instances from the 160 analyzed repositories, and compare their labels with the authors' labels. If inter-rater agreement is low (for instance, a Cohen's kappa below 0.6) or if the independent labels yield an overall precision materially below 87.9%, the paper's conclusion that the tool reliably identifies code smells would not be supported.","tokens_in":15361,"feed_emoji":"🐍","tokens_out":10001,"duration_ms":96986,"temperature":0.7,"pith_summary":"The paper argues that a newly developed static code analysis tool, MLpylint, can automatically detect machine-learning-specific code smells in Python ML applications with high precision. The tool was evaluated on 160 open-source ML projects, reporting 100% precision for definitive code smells and 73.6% for advisory patterns, for an overall precision of 87.9%. A survey of 15 ML professionals found the tool useful for reducing time and effort in identifying ML-specific smells. The claim matters because ML code accumulates technical debt and maintainability problems, and validated tools for ML-specific smells have been scarce.","feed_headline":"MLpylint flags ML code smells at 87.9% precision","feed_subtitle":"Validated on 160 repositories and rated useful by 15 practitioners, it targets technical debt in Python ML code.","key_machinery":"The mechanism that carries the argument is MLpylint's static analysis pipeline: a runner feeds Python files to an AST-parsing library that builds an abstract syntax tree and infers object types, even for external machine-learning libraries; then a set of checkers, organized in a visitor pattern, applies dedicated detection logic for each smell. Each smell is classified against five criteria: it must have a description, be specifically relevant to Python ML, be statically detectable, have a distinct code pattern, and have a precise, unambiguous pattern. That classification splits smells into definitive 'code smells' and broader 'code smell advices' that require developer judgment.","core_discovery":"The central discovery is that a catalogue of 22 ML-specific code smells can be operationalized into statically detectable patterns: 20 of them pass the paper's criteria and are split into 14 unambiguous 'code smells' and 6 broader 'code smell advices.' MLpylint implements these patterns using an abstract syntax tree parser with object inference, and running it over 160 open-source repositories containing 10 million lines of Python yields 5,380 detected smells in about 36 minutes. In a manually validated stratified sample, the tool achieves 100% precision on code smells and 73.6% on advice, giving an overall precision of 87.9%. The paper also reports that practitioners found the tool easy to use, effective, and time-saving.","pith_inferences":["If the precision holds beyond the sampled projects, the 20-smell catalogue could serve as a shared benchmark for other ML-specific static analysis tools, giving the field a common reference set for comparison.","The wide gap between 100% precision on definitive smells and 73.6% on advisory smells suggests a useful trade-off: unambiguous patterns can be automated and trusted, while fuzzy advice still needs human judgment.","An independent replication using multiple raters for the manual validation would strengthen confidence in the absolute precision figures, since the reported numbers use the authors' own manual labels as ground truth.","As the smells are tied to external libraries such as NumPy, Pandas, PyTorch, and TensorFlow, extending the tool to additional ML libraries and to other workflow contexts like notebooks is a natural next step implied by the paper's architecture."],"forward_implications":["MLpylint can be embedded in development workflows such as pre-commit hooks and CI/CD pipelines, allowing ML teams to catch these smells before changes are merged.","The 14 code smells and 6 advices form a concrete checklist for Python ML code review, validated on a large set of real projects.","The prevalence data points to common problem areas: uncontrolled randomness appears 1,796 times, and two advisory patterns each appear over 1,000 times across the analyzed projects.","The tool's speed, about 13 seconds per project, makes it feasible for routine scanning of large codebases.","The authors report that a plugin is already in progress to integrate the smell checks into an existing Python linting workflow."],"supporting_citations":[{"why":"Supplies the catalogue of 22 ML-specific code smells that MLpylint filters down to 20 and implements as checks.","marker":"[27]"},{"why":"Motivates the AST-based static analysis design and documents limitations of existing Python analyzers for external libraries.","marker":"[6]"},{"why":"Documents the prevalence of code smells and dependency issues in open-source ML projects, defining the gap the tool addresses.","marker":"[23]"},{"why":"Establishes the hidden technical debt problem in early ML prototypes that MLpylint is designed to address.","marker":"[1]"},{"why":"Informs the code smell criteria by reviewing definitions and detection tools for code smells.","marker":"[13]"},{"why":"Provides the proportional stratified random sampling method used to select instances for manual validation.","marker":"[14]"},{"why":"Supplies the experimentation guidelines underlying the evaluation and validity discussion.","marker":"[26]"}],"fun_headline_variants":["MLpylint flags 5,380 ML code smells in 36 minutes","20 ML smell patterns detected with 87.9% precision","MLpylint: 5,380 detections, 87.9% precision, 36 minutes","Static analysis tool hits 87.9% precision on ML smells","MLpylint scans 10M lines, finds 5,380 ML smells in 36 min"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported precision of 87.9% rests on the assumption that the authors' manual review of a stratified sample is a reliable ground truth for whether a flagged pattern is truly a code smell, and no independent ground truth or inter-rater reliability check is reported.","fun_headline_variants_meta":{"raw":{"variants":["MLpylint flags 5,380 ML code smells in 36 minutes","20 ML smell patterns detected with 87.9% precision","MLpylint: 5,380 detections, 87.9% precision, 36 minutes","Static analysis tool hits 87.9% precision on ML smells","MLpylint scans 10M lines, finds 5,380 ML smells in 36 min"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001609,"raw_usage":{"total_tokens":6367,"prompt_tokens":866,"completion_tokens":5501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":5391}},"tokens_in":482,"tokens_out":5501,"duration_ms":47712,"temperature":1.0,"reasoning_tokens":5391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:57:23.608480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask two independent teams to manually label the same stratified sample of flagged instances from the 160 analyzed repositories, and compare their labels with the authors' labels. If inter-rater agreement is low (for instance, a Cohen's kappa below 0.6) or if the independent labels yield an overall precision materially below 87.9%, the paper's conclusion that the tool reliably identifies code smells would not be supported.","supporting_citations":[],"review_version":1}