{"id":"ee11f429-2086-473a-be73-94f7a37cd6c2","arxiv_id":"2501.12617","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"State-of-the-art DL-based method-name inconsistency detectors drop from about 55-70% positive precision on balanced data to under 1% on a realistic 1:531 imbalanced benchmark.","lead":"An empirical study finds that five deep-learning tools for detecting inconsistent Java method names lose almost all precision (below 1%) when tested on a realistically imbalanced benchmark instead of the perfectly balanced data used in prior work. The authors build a manually filtered benchmark and argue these tools need major improvement before real-world use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Negative-set label noise (4.4% hidden positives) biases the reported natural-ratio precision by an order of magnitude; corrected precision may be ~7% rather than <1%, though the qualitative decline remains.","rationale":"The reader's weakest assumption concerns re-implementation fidelity and fixed similarity thresholds. Those are legitimate secondary concerns, but the more load-bearing issue is the known noise in the negative labels of the benchmark itself. Section 3.2.2 explicitly reports a 4.4% false-positive rate among the consistent methods, and because the test set is 1.3 million methods, this translates to tens of thousands of hidden positives. In an extreme imbalance setting, even small label noise dominates the precision denominator. The paper's headline RQ2 numbers (0.3-0.5% precision) are therefore not valid estimates of real-world precision; a first-order correction using the paper's own stated error rate yields approximately 7% precision for all five approaches. This does not overturn the central qualitative finding that the ratio substantially impacts precision, but it does invalidate the specific 'precision less than 1%' and 'overwhelming false positives' takeaways. The verdict should remain CONDITIONAL, but the revision must either correct the precision estimates for the known label noise or explicitly frame the numbers as upper-bound false-positive rates rather than true precision values. I disagree with the reader's choice of weakest assumption because the threshold/reimplementation concern is more speculative, while the label-noise bias is directly quantified in the manuscript and demonstrably changes the headline result by an order of magnitude.","tokens_in":35027,"tokens_out":10967,"duration_ms":117644,"concrete_test":"Correct the RQ2 precision for negative-label noise: for each approach, let H = 0.044 x 1,296,743, assume recall on H equals the reported recall on labeled positives, and compute corrected precision = (TP_known + rH) / (TP_known + rH + FP_measured - rH). If the corrected values exceed 1% for all five approaches (likely ~7%), the '<1% precision' conclusion is unsupported. A stronger direct check is to manually inspect a random sample of 383 of the model's flagged 'false positives' on NaturalData, measure the actual hidden-positive rate among them, and recompute precision from that sample.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RQ2 precision collapse is the paper's central evidence, but the NaturalData negative set is known to be noisy: Section 3.2.2 samples 383 consistent methods and finds 366/383 = 95.6% truly consistent, i.e., a 4.4% false-positive rate among the 1,296,743 consistent methods. That implies roughly 57,000 hidden inconsistent methods are labeled as consistent. At the 1:531 ratio, every such hidden positive that an approach flags is counted as a false positive, directly deflating the reported precision. Using the paper's own numbers for IRMCC (Table 19: precision 0.3%, recall 70.0%, N_pos = 2,443), the total flagged as inconsistent is about 570,000. If hidden positives are flagged at the same 70% recall, about 39,900 of them are flagged and mislabeled as false positives. Correcting this yields a true precision of roughly 7.3%, not 0.3%. The same correction lifts the other four approaches from 0.3-0.5% to roughly 7%. The authors acknowledge the 4.4% error rate in Section 8.1.2 but never propagate it into RQ2. Thus the specific claim that precision is under 1% and that false positives are 'overwhelming' is an artifact of benchmarking, not a direct property of the models. The qualitative conclusion that precision drops substantially under natural imbalance survives, but the central quantitative evidence is overstated by about an order of magnitude.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs a new benchmark, BenMark, containing 2,443 inconsistent method names and 1,296,743 consistent method names (ratio 1:531), and evaluates five deep-learning-based approaches (CAN, IRMCC, MNIRE, Cognac, GTNM) under within-project, cross-project, balanced, and natural-ratio settings. The central finding is that precision for detecting inconsistent names collapses from roughly 53-56% on the balanced dataset to 0.3-0.5% on the natural-ratio dataset, while recall remains high, leading the authors to conclude that existing approaches are not yet accurate enough for practical use. The paper also performs quantitative and qualitative analyses of where the approaches succeed and fail, and proposes contrastive learning and LLM-based directions for improvement.","tokens_in":35314,"tokens_out":8694,"duration_ms":94341,"significance":"The study targets an important gap: prior evaluations of inconsistent-method-name detection used balanced datasets that do not reflect real-world prevalence, and the paper provides a manually inspected benchmark, 10-fold experiments, and a public replication package. The qualitative observation that generation-based approaches struggle with 'Narrow'-type renames and with similarity metrics is useful and actionable. However, the headline quantitative claim of sub-1% precision is inflated by negative-set label noise, and the RQ3/RQ4 success/failure analyses are based on a sample that contains almost no true positive instances. The qualitative direction of the results is likely robust, but the specific magnitudes and several failure-mode explanations need revision.","major_comments":[{"comment":"The negative-set label noise measured in Section 8.1.2 is not propagated into the RQ2 precision numbers. Manual inspection of 383 sampled 'consistent' methods found 95.6% truly consistent, i.e., about 4.4% of the 1,296,743 negative-labeled methods in NaturalData are actually inconsistent. Using IRMCC's Table 19 values (precision 0.3%, recall 70.0%, N_pos=2,443), the total number of flagged methods is about 570,000. If hidden positives are flagged at the same 70% recall, roughly 39,900 of the approximately 57,000 hidden positives are flagged and counted as false positives, which would lift the corrected precision to about 7.3% rather than 0.3%. Analogous corrections apply to the other four approaches. The paper's takeaway in Section 8.5 (message 4) that precision is below 1% and false positives are 'overwhelming' is therefore overstated by about an order of magnitude. The authors should report precision after adjusting for the measured noise rate, or at least provide a sensitivity analysis; the qualitative decline is likely to survive, but the central quantitative claim needs correction.","section":"Section 8.1.2, Tables 19-23"},{"comment":"The 383-method sample used in RQ3 (and the same sample used for the generation-based analyses in RQ4) is drawn from NaturalData, whose positive prevalence is 1:531 as stated in Section 3.2.4. A random sample of 383 methods is expected to contain fewer than one inconsistent method, so the reported split of 185 successfully identified and 198 failed cases consists almost entirely of true negatives and false positives. Consequently, the conclusions that IRMCC 'works well' on popular F-tokens and simple bodies (Sections 6.1-6.2) and that retrieval failures are caused by poor body representation (Section 6.3) are actually statements about when consistent methods are not falsely flagged, not about when inconsistent names are detected. The GTNM qualitative analysis in Section 7.3 does sample 332 inconsistent names, but the RQ3 and RQ4 quantitative analyses do not. The sample should be stratified to include a meaningful number of positive instances, or the analysis should be explicitly reframed as a false-positive analysis.","section":"Sections 6 and 7, Section 3.2.4"},{"comment":"The comparison between BalancedData and NaturalData keeps the models and similarity thresholds fixed, which is a valid controlled experiment for isolating the effect of prevalence. However, the practical conclusion in Section 8.5 (message 4) that existing approaches 'may not work accurately in the field' depends on the chosen operating point. The thresholds in Tables 5-7 are tuned on validation data as described in Section 3.3, but a deployment would likely choose a threshold on a natural-ratio validation set to target an acceptable precision-recall tradeoff, and the reported sub-1% precision is not necessarily the only achievable operating point. The authors should report precision-recall curves, or at least evaluate a threshold selected specifically on a natural-ratio validation set, before drawing strong deployment conclusions. If such a recalibration still yields very low precision, that would substantially strengthen the paper's practical claim.","section":"Section 5.1, Tables 5-7, Section 8.5"}],"minor_comments":[{"comment":"There is a typo in the sentence reporting Liu et al.'s results: 'precision of 56.8% an d a recall of 84.5%' should read 'and'.","section":"Section 2.2"},{"comment":"The text says 'not a single inconsistent method name can be identified exclusively by any of the five approaches' but then reports that 'only 0.15% inconsistent method names can be identified exclusively by CAN and Cognac, respectively'; this is contradictory and should be clarified.","section":"Section 8.3"},{"comment":"The phrase 'We obtained a perfect agreement (Cohen's kappa coefficient 0.82)' is imprecise because a kappa of 0.82 is conventionally described as 'almost perfect' or 'substantial' rather than 'perfect'; please reword.","section":"Section 3.2.2"},{"comment":"The paper states that 383 methods were randomly sampled from NaturalData, but the analysis later splits them into 185 success and 198 failure cases; given the 1:531 ratio, the authors should state explicitly how many of the sampled methods were actually inconsistent, since this determines the interpretation of the success/failure analysis.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the replication package is a strength. The main risk is that the headline precision figure changes by an order of magnitude once the measured 4.4% negative-set label noise is accounted for; this is fixable by re-analysis and should not be treated as grounds for rejection. The overlap with Kim et al. (2023) is discussed in the related work, and I found no evidence of misattribution or duplicate publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my take on arXiv:2501.12617. The paper is worth reading, but the headline number—precision below 1% on a natural-ratio benchmark—doesn't survive contact with the paper's own label-noise estimate. They report a 4.4% false-positive rate among the 'consistent' methods in NaturalData. That's roughly 57,000 hidden positives in a negative set of 1.3M. With 70% recall, those get flagged and counted as false positives, which deflates the reported precision from around 7% to 0.3%. The stress-test arithmetic checks out. So the claim that the approaches produce 'overwhelming' false positives is overstated by about an order of magnitude. The qualitative conclusion—precision drops a lot under realistic imbalance—still holds, but the paper should report corrected numbers or, better, re-estimate precision after cleaning the negative set.\n\nWhat's genuinely new and good: BenMark is a serious construction—automatic mining plus developer survey and manual inspection, with the consistent set sampled and checked. Evaluating five approaches on balanced vs. natural ratios is the right experiment, and the finding that the methods' precision is highly sensitive to base rate is real and field-relevant. The qualitative analyses of failure modes (representation, similarity calculation, Narrow-type renames) are useful, even if they're drawn from a sample that is almost all consistent methods—so they mostly describe specificity, not detection of inconsistent names.\n\nOther soft spots, in decreasing order: the thresholds are tuned on balanced validation sets and then held fixed for NaturalData; a user could recalibrate to trade recall for precision, and the paper doesn't explore that. The re-implementations may not perfectly match the original systems, though they do tune hyperparameters and share the code. No majority-class baseline is reported; on NaturalData a trivial always-consistent classifier gets 99.8% accuracy, and the paper's best model gets 34.2%, so the accuracy numbers are easy to misread.\n\nI'd send this to review. The benchmark and the base-rate effect are worth having, even though the central quantitative claim needs substantial revision.","headline":"Solid benchmark and a real base-rate effect, but the headline precision collapse is inflated by label noise the authors measured but never propagated.","tokens_in":35842,"tokens_out":3451,"would_cite":true,"duration_ms":35997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Five deep-learning detectors of inconsistent method names fall below 1% precision on a realistic 1:531 benchmark.","keywords":["inconsistent method names","method name generation","deep learning for code","benchmark construction","code quality","software maintenance","program comprehension","empirical software engineering"],"falsifier":"Run the original released implementations of the five approaches on BenMark with their decision thresholds tuned on a validation set that keeps the 1:531 natural ratio, then report precision for inconsistent names; if precision stays above roughly 50% instead of falling below 1%, the paper's central claim is refuted. A second check is to manually re-label a random sample of BenMark's inconsistent and consistent sets and recompute the evaluation with those labels.","tokens_in":34833,"feed_emoji":"🔬","tokens_out":7354,"duration_ms":71746,"temperature":0.7,"pith_summary":"Five deep-learning tools that flag inconsistent method names all pass their original tests and fail a realistic one. The paper builds a new benchmark, BenMark, with 2,443 genuinely inconsistent names among 1,296,743 consistent ones (a 1:531 ratio), and re-runs the tools under that natural ratio. Every tool's F-score for finding the inconsistent names falls from the 55–70% range on balanced data to under 1%, because precision collapses even though recall stays high. The central message is that balanced test sets flatter these systems, and real-world deployment needs far better precision or a different approach.","feed_headline":"Deep-learning name checkers fall below 1% precision on real code","feed_subtitle":"Five detectors look strong on balanced data but at a natural 1:531 ratio they return almost all false positives.","key_machinery":"BenMark is the load-bearing object: a benchmark of 430 Java projects with 2,443 inconsistent and 1,296,743 consistent method names, built by mining rename commits and then manually filtering out false positives using two developer-derived criteria: the old name must misalign with the method body, and the rename must change the name's semantics. The benchmark's natural 1:531 ratio is what exposes the precision collapse. The secondary mechanism is the decision rule shared by generation-based tools: generate a name from the body, compute lexical similarity to the original name, and flag a mismatch below a threshold; when negatives vastly outnumber positives, that rule produces a flood of false positives.","core_discovery":"On its own terms, the paper's discovery is that the ratio of inconsistent to consistent method names in the test data is the dominant factor in how these tools perform. On a balanced test set, the five evaluated approaches (one retrieval-based, four generation-based) reach F-scores between 55% and 70% for inconsistent names, which looks usable. On BenMark, where only 1 in 531 names is inconsistent, the same trained models and fixed similarity thresholds keep most of their recall (from roughly 70% to 96%) but their precision for inconsistent names drops to 0.3–0.5%. The paper traces the failure to two mechanisms: retrieval-based checking retrieves dissimilar bodies because its code representation is weak and its 'similar bodies have similar names' hypothesis fails, while generation-based checking uses lexical overlap between the generated and original names, which misses synonyms and the 'Narrow' renames where the correct name adds a missing sub-token. The paper concludes that the state of the art is not accurate enough for field use and proposes contrastive learning and large language models as the way forward.","pith_inferences":["An implication left implicit is that the benchmark's 1:531 prior should become the default reporting standard for this task; any future claim of deployability should be measured on natural-ratio data, not balanced data.","A testable extension is a ranking-mode evaluation: if the tools output a similarity score, treating the task as top-k retrieval instead of binary classification may salvage some utility even at low precision.","The paper's taxonomy suggests a concrete experiment: measure precision separately for Generalize, Narrow, and Change renames; the qualitative findings predict Narrow will be the hardest, but the authors do not report that split.","Because the consistent names in BenMark were only sampled at 4.4% false-positive rate, a small share of 'consistent' labels may actually be inconsistent; if that noise is not random, it could bias the measured precision downward."],"forward_implications":["Balanced-data F-scores reported by the original papers do not transfer to natural code; the same tools drop to below 1% precision at a 1:531 inconsistent-to-consistent ratio.","None of the five tools is ready for routine use on real Java codebases without strong additional filtering or human review.","Cross-project training performs as well as within-project training, so users can reuse one trained model instead of retraining per project.","Future methods should replace lexical similarity with semantic similarity and must handle 'Narrow' renames, where the generated name is only missing one or a few sub-tokens.","Contrastive learning and LLM-based checking are proposed next directions, but this paper does not implement or evaluate them."],"supporting_citations":[{"why":"Supplies the IRMCC retrieval-based approach and the original balanced dataset (OriginalData) whose construction and 1:1 ratio the paper argues are flawed.","marker":"Liu et al. 2019"},{"why":"Supplies MNIRE, the first generation-based approach, and the generate-then-compare-similarity decision rule with a fixed threshold that the paper evaluates and later criticizes.","marker":"Nguyen et al. 2020"},{"why":"Supplies CAN, the convolutional attentional method-name generation model used as a representative generation-based baseline.","marker":"Allamanis et al. 2016"},{"why":"Supplies Cognac, the generation-based approach with prior sub-token knowledge that is evaluated on BenMark.","marker":"Wang et al. 2021a"},{"why":"Supplies GTNM, the state-of-the-art global-context transformer model used as the strongest generation-based baseline.","marker":"Liu et al. 2022"},{"why":"Earlier empirical study from a code-review perspective that motivates the paper's warning that commit-history renames are unreliable labels for inconsistency.","marker":"Kim et al. 2023"},{"why":"Supports the claim that many commit-history renames are not caused by name-body inconsistency, which is why manual inspection is needed.","marker":"Wen et al. 2020"},{"why":"Supports the same claim about quick remedy commits and the unreliability of rename-based mining for labeling inconsistency.","marker":"Wen et al. 2022"}],"fun_headline_variants":["Real code breaks DL name checkers: precision drops below 1%","1-in-531 test set reveals deep-learning name checkers' false positive flood","Method name detectors fail in the wild: precision <1% on real code","From 70% F1 to near zero: deep-learning name checks don't scale","DL name consistency tools unusable in practice, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the authors' re-implementations of the five tools, with their fixed similarity thresholds and hyperparameters, faithfully represent what the original tools would do in the field; if the original tools would recalibrate their thresholds on imbalanced data, the reported precision collapse could be overstated.","fun_headline_variants_meta":{"raw":{"variants":["Real code breaks DL name checkers: precision drops below 1%","1-in-531 test set reveals deep-learning name checkers' false positive flood","Method name detectors fail in the wild: precision <1% on real code","From 70% F1 to near zero: deep-learning name checks don't scale","DL name consistency tools unusable in practice, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1470,"prompt_tokens":1027,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":344}},"tokens_in":643,"tokens_out":443,"duration_ms":4904,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:59:10.311501+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original released implementations of the five approaches on BenMark with their decision thresholds tuned on a validation set that keeps the 1:531 natural ratio, then report precision for inconsistent names; if precision stays above roughly 50% instead of falling below 1%, the paper's central claim is refuted. A second check is to manually re-label a random sample of BenMark's inconsistent and consistent sets and recompute the evaluation with those labels.","supporting_citations":[{"cited_title":"In: Proceedings of the 44th International Conference on Software Engineering, pp 1294--1306","cited_arxiv_id":null,"evidence_quote":"Supplies GTNM, the state-of-the-art global-context transformer model used as the strongest generation-based baseline."},{"cited_title":"How are We Detecting Inconsistent Method Names? An Empirical Study from Code Review Perspective","cited_arxiv_id":"2308.12701","evidence_quote":"Earlier empirical study from a code-review perspective that motivates the paper's warning that commit-history renames are unreliable labels for inconsistency."},{"cited_title":"Empirical Software Engineering 27(1):1--31, doi:10.1007/s10664-021-10051-z","cited_arxiv_id":null,"evidence_quote":"Supports the same claim about quick remedy commits and the unreliability of rename-based mining for labeling inconsistency."}],"review_version":1}