{"id":"d6c3c83d-a865-4eb7-a1e6-9f017e67a034","arxiv_id":"2504.16027","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"On a small synthetic four-language dataset, GPT-4.0 detects code smells with higher precision than DeepSeek-V3, while both models miss most annotated smells and the cost comparison is unreliable.","lead":"The paper benchmarks OpenAI GPT-4.0 and DeepSeek-V3 on detecting code smells in a small synthetic multilingual codebase, finding GPT-4.0 more precise and both models low in recall. The cost analysis contains arithmetic and factual errors, including a mistaken description of DeepSeek-V3 pricing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Category-level Table 3 cannot be reconciled with type-level Table 6: aggregating GPT-4.0 rows gives TP/FP/FN=197/49/281, while Table 3 categories total 168/46/291. This internal inconsistency undermines the precision/recall measurements central to the paper.","rationale":"The reader's rejection is correct, and the category/type reconciliation failure is concrete evidence of the same underlying fragility the reader identified in the ground-truth and output-matching assumptions. I focus on the internal inconsistency because it is checkable directly from the paper's own tables: no external data or additional experiments are needed to see that Table 3 and Table 6 cannot both be accurate. This is more immediately load-bearing than the cost equation error, because the cost analysis is a secondary contribution, whereas the TP/FP/FN counts are the empirical foundation for the central claim that GPT-4.0 achieves higher precision than DeepSeek-V3. That said, the paper does earn some credit: the type-level Table 6 totals do reconcile exactly with the model-level Table 2 for both models, and the language-breakdown tables also sum consistently to Table 6. So the inconsistency is localized to the category-level table, not pervasive across all granularities. Even so, a localized but unexplained 29-TP gap in the category table is enough to call the whole measurement pipeline into question, and the cost-section errors independently support continued rejection. My verdict therefore remains unchanged from the reader's REJECT.","tokens_in":16688,"tokens_out":10814,"duration_ms":90436,"concrete_test":"Recompute category-level totals by aggregating every type row in Table 6 under the taxonomy stated in Section 2, then compare them with Table 3 and Table 2. If the sums do not match exactly, re-run the full detection experiment using the published dataset, the exact prompt, and a specified output-parsing rule, and report per-run TP/FP/FN counts; the headline numbers must be regenerated from a single internally consistent result set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is that the paper's own TP/FP/FN counts do not agree across granularities. Sections 5.2 and 5.3 present category-level and type-level results that should be two aggregations of the same underlying detections. Aggregating Table 6 for GPT-4.0 yields TP=197, FP=49, FN=281, which matches Table 2 (precision 0.80, recall 0.41, F1 0.54). But summing the five category rows in Table 3 for GPT-4.0 gives TP=168, FP=46, FN=291, a gap of 29 true positives (about 15% of all TP) and 10 false negatives. For DeepSeek-V3, Table 6 totals are TP=146, FP=195, FN=332, while Table 3 categories sum to TP=117, FP=121, FN=342, a 29-TP and 74-FP discrepancy. These are not rounding differences; a single set of measurements cannot satisfy both tables. Because the only link from raw model outputs to precision/recall is the TP/FP/FN assignment, this inconsistency means either the category-level numbers or the type-level numbers are not reliable measurements of the models. Until this is explained, Table 2's headline precision/recall comparison is not trustworthy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks two large language models, GPT-4.0 and DeepSeek-V3, for automated code smell detection on a custom multilingual restaurant-management dataset spanning Java, Python, JavaScript, and C++. It reports precision, recall, and F1-score at three granularities (overall, category, and type), analyzes language-specific performance, compares costs, and contrasts LLM-based detection with SonarQube. The headline empirical claim is that GPT-4.0 achieves substantially higher precision (0.79 vs. 0.42) while both models have low recall (0.41 and 0.31).","tokens_in":17013,"tokens_out":4756,"duration_ms":41426,"significance":"If its measurements were trustworthy, the paper would provide a useful practitioner-oriented comparison of two LLMs for code smell detection across four languages. The released dataset and the three-level evaluation design are genuine assets. However, the paper's central measurements are internally inconsistent: the category-level and type-level counts of true positives, false positives, and false negatives do not reconcile, and the prompt, output-matching rules, repeated runs, and ground-truth validation are all missing. These gaps are load-bearing because every precision, recall, and F1 value in the paper is derived from those TP/FP/FN counts. The paper is therefore not currently reproducible or verifiable, despite the useful dataset and multi-level framing.","major_comments":[{"comment":"The category-level and type-level tables are mutually inconsistent for both models. Summing the five category rows of Table 3 for GPT-4.0 gives TP=168, FP=46, FN=291, whereas summing the type-level rows of Table 6 gives TP=197, FP=49, FN=281 — a discrepancy of 29 true positives. For DeepSeek-V3, Table 3 sums to TP=117, FP=121, FN=342, while Table 6 sums to TP=146, FP=195, FN=332 — a 74-false-positive discrepancy. These are integer counts, not rounding artifacts, and both tables are supposed to be different cuts of the same detection results. At least one set of counts is therefore not a reliable measurement, and because all precision/recall/F1 values derive from these counts, the headline comparison in Table 2 is not supported.","section":"5.2–5.3, Tables 3 and 6"},{"comment":"The 'standardized prompt' is never shown, and the rules used to convert model outputs into TP/FP/FN counts are not described. Without the exact prompt and the string or semantic matching procedure, another researcher cannot reproduce the evaluation, and the comparison may be biased if the matching favored one model. This absence is especially serious because the entire ranking of the two models rests on the TP/FP/FN assignment.","section":"Section 4, 'Detection and Evaluation Methods'"},{"comment":"No repeated runs, confidence intervals, or variance information are reported for either model. LLM outputs are stochastic, and Section 8 itself acknowledges that 'the same input may yield different results across multiple runs.' A single run per model cannot support the claim that GPT-4.0 'achieves significantly higher precision' or any other quantitative ordering without an estimate of run-to-run variability.","section":"Section 5.1 and Section 8"},{"comment":"The ground truth is self-authored: the authors created the dataset, annotated it, and then evaluate against those annotations. No independent validation, inter-rater agreement, or external benchmark is provided. The correctness and completeness of these annotations are foundational to every metric, so the evaluation is anchored to an unvalidated reference rather than an established gold standard.","section":"Section 3, Dataset"},{"comment":"The cost comparison rests on unsupported assumptions: the 5–10 tokens-per-line estimate, the 250-token response length, the 'low complexity' classification of the script, and the claim that DeepSeek-V3 uses complexity-based rather than token-based pricing. These assumptions are not justified by citations or measurements, and the stated DeepSeek-V3 pricing tiers are not sourced. Since cost-effectiveness is one of the paper's contributions, this analysis needs either real pricing data or clearly labeled illustrative assumptions with sensitivity analysis.","section":"Section 7, Cost Analysis"}],"minor_comments":[{"comment":"The model name is used inconsistently as 'DeepSeek' and 'DeepSeek-V3'; the paper should use one name consistently.","section":"Throughout"},{"comment":"Both subsections say 'Table 9 presents a detailed comparison...' but refer to different appendix material; the table numbering and cross-references need to be corrected.","section":"Sections 10.1 and 10.2"},{"comment":"The caption calls the table 'Type-Language Level,' but the table has no language column and appears to aggregate across languages; the caption should be clarified.","section":"Table 6 caption"},{"comment":"Describing DeepSeek-V3 as using 'pattern-matching techniques' is inaccurate for an LLM and should be reworded.","section":"Abstract and Section 7"},{"comment":"There are several formatting errors in references and inline citations, including missing spaces before citation markers such as 'signifyWaseem et al. [2023]Alves et al. [2024].'","section":"Section 2 and References"}],"recommendation":"reject","confidential_remarks":"The internal inconsistency between Tables 3 and 6 is decisive in my view: it invalidates the central quantitative comparison, and the missing prompt and matching rules prevent the authors from demonstrating which table is correct. The paper could be reconsidered if the authors supply raw outputs, the exact prompt, the matching protocol, and repeated-run statistics, but those materials would constitute a substantially revised evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper gives you a reusable annotated dataset and a straightforward benchmark of GPT-4.0 vs DeepSeek-V3 on code smell detection. The dataset — a restaurant management system in Java, Python, JavaScript, and C++ with smell annotations — is on GitHub and Zenodo, and the type-level results are internally consistent with the overall Table 2. That is real, citable infrastructure, and the comparison is a legitimate empirical question for practitioners.\n\nThe problem is that the measurements do not hold together. Section 5.2's category-level Table 3 and Section 5.3's type-level Table 6 should be two views of the same detections, but they do not add up. For GPT-4.0, Table 6 totals TP=197, FP=49, FN=281, while Table 3 categories sum to TP=168, FP=46, FN=291. DeepSeek's numbers differ even more: TP=146, FP=195, FN=332 vs TP=117, FP=121, FN=342. These are not rounding errors. Since TP/FP/FN are definitionally additive, at least one of these tables is not a faithful report of what the models produced, and Table 2's precision/recall/F1 are built on that unstable foundation. Until this is explained, the headline comparison is not trustworthy.\n\nThe methodological gaps make this worse: the standardized prompt is not shown, the output-matching rules are undisclosed, and there are no repeated runs or variance figures. The ground truth is self-authored with no external validation. Those are fixable in revision, but they need to be addressed openly.\n\nThe cost-effectiveness section is a separate set of problems. The GPT-4.0 equation as written evaluates to $0.045, not the stated $0.0375. And DeepSeek-V3's API is token-based, not the \"complexity-based\" fixed tiers described here. That mischaracterization breaks the cost comparison, which was a stated goal of the paper.\n\nThe citation pattern is fine; self-citation is justified when the dataset is actually released.\n\nIf this lands on my desk, I would send it to reviewers rather than desk-reject, because the dataset is useful and the comparison question is worth answering. But I would expect major revision: reconcile or explain the tables, disclose the prompt and matching procedure, run repeated trials, and fix the cost analysis. If the authors cannot reconcile the tables, the empirical claims need to be withdrawn or substantially re-scoped.","headline":"The released multilingual dataset is a real contribution, but the paper's headline precision/recall numbers do not reconcile across its own category-level and type-level tables, and the cost analysis is wrong in both arithmetic and pricing facts.","tokens_in":17490,"tokens_out":1935,"would_cite":false,"duration_ms":19324,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a four-language smelly-code benchmark, GPT-4.0 is far more precise than DeepSeek-V3 at detecting code smells, though both models miss most of them.","keywords":["code smell detection","large language models","GPT-4.0","DeepSeek-V3","precision recall F1","multilingual code dataset","SonarQube","cost-effectiveness"],"falsifier":"Re-run the benchmark on the released dataset with the exact prompt and publish each model's raw responses alongside a second, independent annotation of the same code; if the second annotation changes the true-positive, false-positive, and false-negative counts materially, or if the visible matching rules penalize DeepSeek-V3's wording while accepting GPT-4.0's, the reported precision gap is an artifact.","tokens_in":16505,"feed_emoji":"🐛","tokens_out":6281,"duration_ms":50743,"temperature":0.7,"pith_summary":"This paper tries to establish a fair, reproducible way to compare large language models on code smell detection and to show what that comparison yields. On a deliberately identical restaurant-management codebase written in Java, Python, JavaScript, and C++, it finds that GPT-4.0 flags smells with far fewer false positives than DeepSeek-V3, with precision 0.79 versus 0.42 and F1-score 0.54 versus 0.35. Both models nonetheless miss a large share of the annotated smells, with recall 0.41 for GPT-4.0 and 0.31 for DeepSeek-V3, so neither acts as a complete detector on its own. The paper also argues that LLM-based detection is complementary to rule-based tools like SonarQube and that a hybrid workflow is the practical route.","feed_headline":"GPT-4.0 beats DeepSeek at code smells, misses most","feed_subtitle":"Precision 0.79 vs 0.42 on a four-language benchmark; recall 0.41 vs 0.31 leaves many smells undetected.","key_machinery":"The load-bearing object is a purpose-built evaluation setup: a restaurant-management system intentionally implemented once in Java, Python, JavaScript, and C++, with known code smells manually annotated to form the ground truth. Model inputs are the cleaned code, with annotations stripped, sent through a standardized prompt asking for detection and categorization; outputs are scored as true positives, false positives, and false negatives against those annotations, then rolled up into precision, recall, and F1 at overall, category, type, and language levels. The code-smell taxonomy, organized into Bloaters, Dispensables, Couplers, Object-Orientation Abusers, and Change Preventers, supplies the categories into which the counts are sorted.","core_discovery":"The central claim is that, on the authors' multilingual smelly-code dataset, GPT-4.0 is substantially more precise than DeepSeek-V3 at detecting and categorizing code smells, while both models have low recall. GPT-4.0 reports fewer false positives across every smell category, reaching perfect precision on hard-to-detect classes such as Change Preventers at the cost of near-zero recall, and it dominates on common structural smells like Large Class, Message Chains, and Inappropriate Intimacy. DeepSeek-V3 occasionally finds smells that GPT-4.0 misses, notably Refused Bequest in Python, but buries them in a much larger number of false positives. The paper further claims that GPT-4.0's per-query cost is higher but its output quality justifies the expense where precision matters, while DeepSeek-V3's fixed complexity-based pricing suits cheap broad screening; neither model yet matches the determinism of SonarQube, so the authors recommend combining LLMs with static analysis.","pith_inferences":["The low recall of both models may partly reflect a conservative reporting style: GPT-4.0 seems to emit only high-confidence smell labels, so tuning prompts or temperature to lower the reporting threshold could raise recall, though the effect on precision is unknown.","Because the dataset is one small synthetic system, the absolute numbers should not be read as general detection rates; the relative ordering of the models is the more transportable result.","The paper's unshown prompt and matching rules are the true experiment; publishing them and the raw model outputs would let others verify whether the precision gap is real or an artifact of how near-miss labels were counted.","A natural next test is an ensemble: let DeepSeek-V3 propose candidate smells for high recall, then have GPT-4.0 confirm or reject them."],"forward_implications":["Teams that want few false alarms should prefer GPT-4.0 over DeepSeek-V3 for code-smell screening on this kind of codebase.","Because recall is low for both models, a production workflow should pair an LLM with a rule-based analyzer such as SonarQube to catch smells the models miss.","Detection quality varies by language and smell type, so per-type or per-language prompts and thresholds are needed rather than one generic setting.","GPT-4.0's token-based pricing makes it costlier at scale, while DeepSeek-V3's fixed complexity tiers make it the cheaper bulk screener, accepting more false positives.","The released dataset and evaluation matrix give a reusable benchmark for future models, so the comparison can be extended as new LLMs appear."],"supporting_citations":[{"why":"Supplies the precision, recall, and F1 evaluation scheme used to score code smell detectors.","marker":"Paiva et al. [2017]"},{"why":"Provides one release of the annotated multilingual smelly-code dataset used as ground truth.","marker":"Sadik [2025a]"},{"why":"Provides the other release of the annotated multilingual smelly-code dataset used as ground truth.","marker":"Sadik [2025b]"},{"why":"Documents GPT-4 and its token-based pricing model used in the cost analysis.","marker":"Achiam et al. [2023]"},{"why":"Documents DeepSeek-V3 and its complexity-based pricing model used in the cost analysis.","marker":"Liu et al. [2024]"},{"why":"Establishes SonarQube as the static-analysis baseline for the comparison.","marker":"Lenarduzzi et al. [2020]"},{"why":"Prior evidence that LLMs can detect smells, which this benchmark extends to a multilingual dataset.","marker":"Lucas et al. [2024]"},{"why":"Assembles LLMs with expert toolsets for smell detection, a related hybrid direction the conclusion recommends.","marker":"Wu et al. [2024]"}],"fun_headline_variants":["GPT-4 beats DeepSeek on code smells, but recall still low","Code smell bots: GPT-4 precise, DeepSeek cheap, both miss most","Benchmark: GPT-4 beats DeepSeek at code smells, but both low recall","Precision vs price: GPT-4 vs DeepSeek for code smell detection","Code smell face-off: GPT-4 precise, DeepSeek affordable, both low recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers rest on the assumption that the manually annotated smells in the small restaurant-management dataset are a correct and complete ground truth, and that the standardized prompt and the unshown rules for matching model output to annotations treat both models fairly.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 beats DeepSeek on code smells, but recall still low","Code smell bots: GPT-4 precise, DeepSeek cheap, both miss most","Benchmark: GPT-4 beats DeepSeek at code smells, but both low recall","Precision vs price: GPT-4 vs DeepSeek for code smell detection","Code smell face-off: GPT-4 precise, DeepSeek affordable, both low recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3402,"prompt_tokens":942,"completion_tokens":2460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2350}},"tokens_in":558,"tokens_out":2460,"duration_ms":14431,"temperature":1.0,"reasoning_tokens":2350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:47.039549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark on the released dataset with the exact prompt and publish each model's raw responses alongside a second, independent annotation of the same code; if the second annotation changes the true-positive, false-positive, and false-negative counts materially, or if the visible matching rules penalize DeepSeek-V3's wording while accepting GPT-4.0's, the reported precision gap is an artifact.","supporting_citations":[{"cited_title":"Are sonarqube rules inducing bugs? In 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER), pages 501–511","cited_arxiv_id":null,"evidence_quote":"Establishes SonarQube as the static-analysis baseline for the comparison."}],"review_version":1}