Pith. sign in

REVIEW 3 major objections 4 minor 91 references

Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Five deep-learning detectors of inconsistent method names fall below 1% precision on a realistic 1:531 benchmark.

desk verdict Solid benchmark and a real base-rate effect, but the headline precision collapse is inflated by label noise the authors measured but never propagated. read the letter →

arxiv 2501.12617 v1 pith:MCY6GHDL submitted 2025-01-22 cs.SE cs.AI

Taiming Wang , Yuxia Zhang , Lin Jiang , Yi Tang , Guangjie Li , Hui Liu This is my paper · ORCID
classification cs.SEcs.AI
keywords inconsistentmethodnamesnamegenerationdeeplearningforcodebenchmarkconstructionqualitysoftwaremaintenanceprogramcomprehensionempiricalengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Five deep-learning tools that flag inconsistent method names all pass their original tests and fail a realistic one. The paper builds a new benchmark, BenMark, with 2,443 genuinely inconsistent names among 1,296,743 consistent ones (a 1:531 ratio), and re-runs the tools under that natural ratio. Every tool's F-score for finding the inconsistent names falls from the 55–70% range on balanced data to under 1%, because precision collapses even though recall stays high. The central message is that balanced test sets flatter these systems, and real-world deployment needs far better precision or a different approach.

What carries the argument

BenMark is the load-bearing object: a benchmark of 430 Java projects with 2,443 inconsistent and 1,296,743 consistent method names, built by mining rename commits and then manually filtering out false positives using two developer-derived criteria: the old name must misalign with the method body, and the rename must change the name's semantics. The benchmark's natural 1:531 ratio is what exposes the precision collapse. The secondary mechanism is the decision rule shared by generation-based tools: generate a name from the body, compute lexical similarity to the original name, and flag a mismatch below a threshold; when negatives vastly outnumber positives, that rule produces a flood of false positives.

What would settle it

Run the original released implementations of the five approaches on BenMark with their decision thresholds tuned on a validation set that keeps the 1:531 natural ratio, then report precision for inconsistent names; if precision stays above roughly 50% instead of falling below 1%, the paper's central claim is refuted. A second check is to manually re-label a random sample of BenMark's inconsistent and consistent sets and recompute the evaluation with those labels.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the ratio of inconsistent to consistent method names in the test data is the dominant factor in how these tools perform. On a balanced test set, the five evaluated approaches (one retrieval-based, four generation-based) reach F-scores between 55% and 70% for inconsistent names, which looks usable. On BenMark, where only 1 in 531 names is inconsistent, the same trained models and fixed similarity thresholds keep most of their recall (from roughly 70% to 96%) but their precision for inconsistent names drops to 0.3–0.5%. The paper traces the failure to two mechanisms: retrieval-based checking retrieves dissimilar bodies because its code representation is weak and its 'similar bodies have similar names' hypothesis fails, while generation-based checking uses lexical overlap between the generated and original names, which misses synonyms and the 'Narrow' renames where the correct name adds a missing sub-token. The paper concludes that the state of the art is not accurate enough for field use and proposes contrastive learning and large language models as the way forward.

Load-bearing premise

The central claim rests on the assumption that the authors' re-implementations of the five tools, with their fixed similarity thresholds and hyperparameters, faithfully represent what the original tools would do in the field; if the original tools would recalibrate their thresholds on imbalanced data, the reported precision collapse could be overstated.

Editorial extensions

If this is right

  • Balanced-data F-scores reported by the original papers do not transfer to natural code; the same tools drop to below 1% precision at a 1:531 inconsistent-to-consistent ratio.
  • None of the five tools is ready for routine use on real Java codebases without strong additional filtering or human review.
  • Cross-project training performs as well as within-project training, so users can reuse one trained model instead of retraining per project.
  • Future methods should replace lexical similarity with semantic similarity and must handle 'Narrow' renames, where the generated name is only missing one or a few sub-tokens.
  • Contrastive learning and LLM-based checking are proposed next directions, but this paper does not implement or evaluate them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit is that the benchmark's 1:531 prior should become the default reporting standard for this task; any future claim of deployability should be measured on natural-ratio data, not balanced data.
  • A testable extension is a ranking-mode evaluation: if the tools output a similarity score, treating the task as top-k retrieval instead of binary classification may salvage some utility even at low precision.
  • The paper's taxonomy suggests a concrete experiment: measure precision separately for Generalize, Narrow, and Change renames; the qualitative findings predict Narrow will be the hardest, but the authors do not report that split.
  • Because the consistent names in BenMark were only sampled at 4.4% false-positive rate, a small share of 'consistent' labels may actually be inconsistent; if that noise is not random, it could bias the measured precision downward.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper constructs a new benchmark, BenMark, containing 2,443 inconsistent method names and 1,296,743 consistent method names (ratio 1:531), and evaluates five deep-learning-based approaches (CAN, IRMCC, MNIRE, Cognac, GTNM) under within-project, cross-project, balanced, and natural-ratio settings. The central finding is that precision for detecting inconsistent names collapses from roughly 53-56% on the balanced dataset to 0.3-0.5% on the natural-ratio dataset, while recall remains high, leading the authors to conclude that existing approaches are not yet accurate enough for practical use. The paper also performs quantitative and qualitative analyses of where the approaches succeed and fail, and proposes contrastive learning and LLM-based directions for improvement.

Significance. The study targets an important gap: prior evaluations of inconsistent-method-name detection used balanced datasets that do not reflect real-world prevalence, and the paper provides a manually inspected benchmark, 10-fold experiments, and a public replication package. The qualitative observation that generation-based approaches struggle with 'Narrow'-type renames and with similarity metrics is useful and actionable. However, the headline quantitative claim of sub-1% precision is inflated by negative-set label noise, and the RQ3/RQ4 success/failure analyses are based on a sample that contains almost no true positive instances. The qualitative direction of the results is likely robust, but the specific magnitudes and several failure-mode explanations need revision.

major comments (3)
  1. [Section 8.1.2, Tables 19-23] The negative-set label noise measured in Section 8.1.2 is not propagated into the RQ2 precision numbers. Manual inspection of 383 sampled 'consistent' methods found 95.6% truly consistent, i.e., about 4.4% of the 1,296,743 negative-labeled methods in NaturalData are actually inconsistent. Using IRMCC's Table 19 values (precision 0.3%, recall 70.0%, N_pos=2,443), the total number of flagged methods is about 570,000. If hidden positives are flagged at the same 70% recall, roughly 39,900 of the approximately 57,000 hidden positives are flagged and counted as false positives, which would lift the corrected precision to about 7.3% rather than 0.3%. Analogous corrections apply to the other four approaches. The paper's takeaway in Section 8.5 (message 4) that precision is below 1% and false positives are 'overwhelming' is therefore overstated by about an order of magnitude. The authors should report precision after adjusting for the measured noise rate, or at least provide a sensitivity analysis; the qualitative decline is likely to survive, but the central quantitative claim needs correction.
  2. [Sections 6 and 7, Section 3.2.4] The 383-method sample used in RQ3 (and the same sample used for the generation-based analyses in RQ4) is drawn from NaturalData, whose positive prevalence is 1:531 as stated in Section 3.2.4. A random sample of 383 methods is expected to contain fewer than one inconsistent method, so the reported split of 185 successfully identified and 198 failed cases consists almost entirely of true negatives and false positives. Consequently, the conclusions that IRMCC 'works well' on popular F-tokens and simple bodies (Sections 6.1-6.2) and that retrieval failures are caused by poor body representation (Section 6.3) are actually statements about when consistent methods are not falsely flagged, not about when inconsistent names are detected. The GTNM qualitative analysis in Section 7.3 does sample 332 inconsistent names, but the RQ3 and RQ4 quantitative analyses do not. The sample should be stratified to include a meaningful number of positive instances, or the analysis should be explicitly reframed as a false-positive analysis.
  3. [Section 5.1, Tables 5-7, Section 8.5] The comparison between BalancedData and NaturalData keeps the models and similarity thresholds fixed, which is a valid controlled experiment for isolating the effect of prevalence. However, the practical conclusion in Section 8.5 (message 4) that existing approaches 'may not work accurately in the field' depends on the chosen operating point. The thresholds in Tables 5-7 are tuned on validation data as described in Section 3.3, but a deployment would likely choose a threshold on a natural-ratio validation set to target an acceptable precision-recall tradeoff, and the reported sub-1% precision is not necessarily the only achievable operating point. The authors should report precision-recall curves, or at least evaluate a threshold selected specifically on a natural-ratio validation set, before drawing strong deployment conclusions. If such a recalibration still yields very low precision, that would substantially strengthen the paper's practical claim.
minor comments (4)
  1. [Section 2.2] There is a typo in the sentence reporting Liu et al.'s results: 'precision of 56.8% an d a recall of 84.5%' should read 'and'.
  2. [Section 8.3] The text says 'not a single inconsistent method name can be identified exclusively by any of the five approaches' but then reports that 'only 0.15% inconsistent method names can be identified exclusively by CAN and Cognac, respectively'; this is contradictory and should be clarified.
  3. [Section 3.2.2] The phrase 'We obtained a perfect agreement (Cohen's kappa coefficient 0.82)' is imprecise because a kappa of 0.82 is conventionally described as 'almost perfect' or 'substantial' rather than 'perfect'; please reword.
  4. [Section 6] The paper states that 383 methods were randomly sampled from NaturalData, but the analysis later splits them into 185 success and 198 failure cases; given the 1:531 ratio, the authors should state explicitly how many of the sampled methods were actually inconsistent, since this determines the interpretation of the success/failure analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical evaluation is self-contained; threshold tuning is standard practice and the lone self-citation is not load-bearing.

full rationale

The paper contains no derivation that reduces to its own inputs. It constructs a benchmark by mining commit histories plus manual inspection, then evaluates five externally published approaches under controlled settings. The thresholds are tuned on a validation split (Section 3.3) and held fixed across BalancedData and NaturalData; this is a standard controlled comparison, not a fitted parameter being renamed as a prediction. The RQ2 precision decline (e.g., IRMCC from 54.4% to 0.3% in Table 19) is a measured outcome of applying fixed models to a more imbalanced test set, and the arithmetic precision = TP/(TP+FP) is used only to explain the observed decline, not to generate the result. The one motivation citation to Liu et al. (2020), which includes two of the present authors, is not load-bearing: the point that small test sets give unreliable estimates is generic, and the benchmark's value does not rest on that citation. The acknowledged 4.4% false-positive rate among consistent labels (Section 8.1.2) is a correctness or validity concern about ground-truth noise, not a circularity, because it does not make any reported quantity equal to its own input. No self-definitional, fitted-input, self-citation-chain, ansatz-smuggling, or renaming pattern is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's central claim depends on the ground-truth labels produced by the authors' manual inspection and on faithful re-implementation of prior tools. These are domain assumptions rather than free parameters for a derivation. No new entities are introduced.

free parameters (1)
  • Baseline hyperparameters (thresholds, k) = MNIRE 0.89, CAN 0.90, Cognac 0.85, GTNM 0.85, IRMCC k=1
    Tuned by the authors on validation data following original papers; they are inputs to the evaluation. The fixed thresholds are load-bearing for the precision-collapse conclusion, since re-tuning is not explored.
assumptions (3)
  • domain assumption Renames in commit history, after manual filtering, identify genuine inconsistent method names.
    Used to construct BenMark (Section 3.2.2). If commits that rename methods for typos, synonyms, or word-order fixes are not fully filtered, labels contain false positives.
  • domain assumption Untouched methods in a snapshot are consistent unless linked to a rename.
    Consistent set in Section 3.2.2 is automatic; only 383 samples were manually audited (95.6% agreement, 4.4% false positive rate), so some consistent labels may be wrong.
  • domain assumption The re-implemented baselines (Section 3.1) faithfully represent the original published approaches.
    DeepName was excluded because its code would not run; the remaining five were re-implemented and tuned by the authors, so deviations could affect the evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?." pith.science (2026). https://pith.science/paper/MCY6GHDL

@misc{pith2026250112617,
  author       = {Pith},
  title        = {Pith review of: Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCY6GHDL}},
  note         = {Machine review of arXiv:2501.12617}
}
read the original abstract

Concise and meaningful method names are crucial for program comprehension and maintenance. However, method names may become inconsistent with their corresponding implementations, causing confusion and errors. Several deep learning (DL)-based approaches have been proposed to identify such inconsistencies, with initial evaluations showing promising results. However, these evaluations typically use a balanced dataset, where the number of inconsistent and consistent names are equal. This setup, along with flawed dataset construction, leads to false positives, making reported performance less reliable in real-world scenarios, where most method names are consistent. In this paper, we present an empirical study that evaluates state-of-the-art DL-based methods for identifying inconsistent method names. We create a new benchmark by combining automatic identification from commit histories and manual developer inspections, reducing false positives. We evaluate five representative DL approaches (one retrieval-based and four generation-based) on this benchmark. Our results show that performance drops substantially when moving from the balanced dataset to the new benchmark. We further conduct quantitative and qualitative analyses to understand the strengths and weaknesses of the approaches. Retrieval-based methods perform well on simple methods and those with popular name sub-tokens but fail due to inefficient representation techniques. Generation-based methods struggle with inaccurate similarity calculations and immature name generation. Based on these findings, we propose improvements using contrastive learning and large language models (LLMs). Our study suggests that significant improvements are needed before these DL approaches can be effectively applied to real-world software systems.

Figures

Figures reproduced from arXiv: 2501.12617 by the authors.

Figure 1
Figure 1. Popularity of F-tokens vs. Success Rate (IRMCC). 6.2 Method Body Complexity Matters From the method body perspective, we measure the complexity of method bodies by two widely used metrics, i.e., LOC and McCabe’s Cyclomatic Com￾plexity (McCabe 1976). We first investigate the LOC of the sampled 383 methods to see how LOC impacts the performance of IRMCC. We find that the average LOC of the success and failure methods … view at source ↗
Figure 2
Figure 2. Distribution of Methods Similarities. We also analyze how the length of method names (i.e., the number of sub-tokens in a method name) impacts the performance of generation-based approaches. For CAN, MNIRE, Cognac, and GT NM, we calculate the length of all the involved methods in N aturalData and conduct the Spearman cor￾relation analysis (Artusi et al. 2002) between the length of method names and their success rate… view at source ↗
Figure 3
Figure 3. LOC VS. Success Rate. 7.2 Method Body Complexity Matters To investigate how the method body complexity impacts the generation-based approaches, we first analyze the lines of code of all the methods in N aturalData (LOC of methods in success ones and failure ones are analyzed separately). Then we manually segment LOC into four groups and explore the average success rate of each group. The analysis results are present… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Complementarity of Five Selected Approaches in Identifying Inconsistent [PITH_FULL_IMAGE:figures/full_fig_p040_4.png]
Figure 5
Figure 5. Figure 5: Overview of the Contrastive Pre-training. [PITH_FULL_IMAGE:figures/full_fig_p043_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 49 canonical work pages

  1. [1]

    In: 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation, IEEE, pp 125--134

    Abebe SL, Haiduc S, Tonella P, Marcus A (2011) The effect of lexicon bad smells on concept location in source code. In: 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation, IEEE, pp 125--134

    2011

  2. [2]

    Abebe SL, Arnaoudova V, Tonella P, Antoniol G, Gueheneuc YG (2012) Can lexicon bad smells improve fault prediction? In: 2012 19th Working Conference on Reverse Engineering, IEEE, pp 235--244

    2012

  3. [3]

    Aghajani E, Nagy C, Bavota G, Lanza M (2018) A large-scale empirical study on linguistic antipatterns affecting apis. In: 2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018, Madrid, Spain, September 23-29, 2018, IEEE Computer Society, pp 25--35, doi:10.1109/ICSME.2018.00012, ://doi.org/10.1109/ICSME.2018.00012

    arXiv 2018

  4. [4]

    Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering 16-21-Nove:281--293, doi:10.1145/2635868.2635883, arXiv:1402.4182v3

    Allamanis M, Barr ET, Bird C, Sutton C (2014) Learning NATURAL coding conventions . Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering 16-21-Nove:281--293, doi:10.1145/2635868.2635883, arXiv:1402.4182v3

    arXiv 2014

  5. [5]

    Allamanis M, Barr ET, Bird C, Sutton C (2015) Suggesting accurate method and class names . 2015 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Proceedings pp 38--49, doi:10.1145/2786805.2786849

    arXiv 2015

  6. [6]

    International Conference on Machine Learning 48:2091--2100

    Allamanis M, Peng H, Sutton C (2016) A convolutional attention network for extreme summarization of source code. International Conference on Machine Learning 48:2091--2100

    2016

  7. [7]

    ACM Comput Surv 51(4):81:1--81:37, doi:10.1145/3212695, ://doi.org/10.1145/3212695

    Allamanis M, Barr ET, Devanbu PT, Sutton C (2018) A survey of machine learning for big code and naturalness. ACM Comput Surv 51(4):81:1--81:37, doi:10.1145/3212695, ://doi.org/10.1145/3212695

    doi:10.1145/3212695 2018

  8. [8]

    CoRR abs/1803.09473, ://arxiv.org/abs/1803.09473, 1803.09473

    Alon U, Zilberstein M, Levy O, Yahav E (2018) code2vec: L earning distributed representations of code. CoRR abs/1803.09473, ://arxiv.org/abs/1803.09473, 1803.09473

    arXiv 2018

Show all 91 references
  1. [9]

    In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, ://openreview.net/forum?id=H1gKYo09tX

    Alon U, Brody S, Levy O, Yahav E (2019) code2seq: Generating sequences from structured representations of code. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, ://openreview.net/forum?id=H1gKYo09tX

    2019

  2. [11]

    IEEE Transactions on Software Engineering 45(12):1170--1188

    Amann S, Nguyen HA, Nadi S, Nguyen TN, Mezini M (2018) A systematic evaluation of static api-misuse detectors. IEEE Transactions on Software Engineering 45(12):1170--1188

    2018

  3. [13]

    Arnaoudova V (2010) Improving source code quality through the definition of linguistic antipatterns. In: Antoniol G, Pinzger M, Chikofsky EJ (eds) 17th Working Conference on Reverse Engineering, WCRE 2010, 13-16 October 2010, Beverly, MA, USA , IEEE Computer Society, pp 285--2...

    2010 doi

  4. [14]

    Empir Softw Eng 21(1):104--158, doi:10.1007/s10664-014-9350-8, ://doi.org/10.1007/s10664-014-9350-8

    Arnaoudova V, Penta MD, Antoniol G (2016) Linguistic antipatterns: W hat they are and how developers perceive them. Empir Softw Eng 21(1):104--158, doi:10.1007/s10664-014-9350-8, ://doi.org/10.1007/s10664-014-9350-8

    2016 doi

  5. [15]

    The International journal of biological markers 17(2):148--151

    Artusi R, Verderio P, Marubini E (2002) Bravais-pearson and spearman correlation coefficients: M eaning, test of hypothesis and confidence interval. The International journal of biological markers 17(2):148--151

    2002

  6. [16]

    IEEE Transactions on Software Engineering 40(7):671--694

    Bavota G, Oliveto R, Gethers M, Poshyvanyk D, De Lucia A (2013) Methodbook: R ecommending move method refactorings via relational topic models. IEEE Transactions on Software Engineering 40(7):671--694

    2013

  7. [17]

    Empir Softw Eng 18(2):219--276, doi:10.1007/s10664-012-9201-4, ://doi.org/10.1007/s10664-012-9201-4

    Binkley DW, Davis M, Lawrie DJ, Maletic JI, Morrell C, Sharif B (2013) The impact of identifier style on effort and comprehension. Empir Softw Eng 18(2):219--276, doi:10.1007/s10664-012-9201-4, ://doi.org/10.1007/s10664-012-9201-4

    2013 doi

  8. [18]

    In: 2009 16th Working Conference on Reverse Engineering, IEEE, pp 31--35

    Butler S, Wermelinger M, Yu Y, Sharp H (2009) Relating identifier naming flaws and code quality: A n empirical study. In: 2009 16th Working Conference on Reverse Engineering, IEEE, pp 31--35

    2009

  9. [19]

    In: 2010 14th European Conference on Software Maintenance and Reengineering, IEEE, pp 156--165

    Butler S, Wermelinger M, Yu Y, Sharp H (2010) Exploring the influence of identifier names on code quality: A n empirical study. In: 2010 14th European Conference on Software Maintenance and Reengineering, IEEE, pp 156--165

    2010

  10. [20]

    Proceedings - International Conference on Software Engineering 2022-May:2327--2339, doi:10.1145/3510003.3510162, 2112.02650

    Chen Q, Lacomis J, Schwartz EJ, Neubig G, Vasilescu B, Goues CL (2022) VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning . Proceedings - International Conference on Software Engineering 2022-May:2327--2339, doi:10.1145/3510003.3510162, 2112.02650

    2022

  11. [21]

    Educational and psychological measurement 20(1):37--46

    Cohen J (1960) A coefficient of agreement for nominal scales. Educational and psychological measurement 20(1):37--46

    1960

  12. [22]

    Sage publications

    Creswell JW, Creswell JD (2017) Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications

    2017

  13. [23]

    In: 2011 international symposium on empirical software engineering and measurement, IEEE, pp 275--284

    Cruzes DS, Dyba T (2011) Recommended steps for thematic synthesis in software engineering. In: 2011 international symposium on empirical software engineering and measurement, IEEE, pp 275--284

    2011

  14. [24]

    Software Quality Journal 14(3):261--282, doi:10.1007/s11219-006-9219-1

    Deissenboeck F, Pizka M (2006) Concise and consistent naming . Software Quality Journal 14(3):261--282, doi:10.1007/s11219-006-9219-1

    2006 doi

  15. [25]

    In: 2015 IEEE 23rd International Conference on Program Comprehension, IEEE, pp 3--3

    Deissenboeck F, Pizka M (2015) Concise and consistent naming: T en years later. In: 2015 IEEE 23rd International Conference on Program Comprehension, IEEE, pp 3--3

    2015

  16. [26]

    Proceedings of the Annual Meeting of the Association for Computational Linguistics 1:6300--6312, doi:10.18653/v1/2022.acl-long.436, 2110.03868

    Ding Y, Buratti L, Pujar S, Morari A, Ray B, Chakraborty S (2022) Towards Learning (Dis)-Similarity of Source Code from Program Contrasts . Proceedings of the Annual Meeting of the Association for Computational Linguistics 1:6300--6312, doi:10.18653/v1/2022.acl-long.436, 2110.03868

    2022 arXiv

  17. [27]

    arXiv preprint arXiv:230407590

    Dong Y, Jiang X, Jin Z, Li G (2023) Self-collaboration code generation via ChatGPT . arXiv preprint arXiv:230407590

    2023

  18. [28]

    (2020) Codebert: A pre-trained model for programming and natural languages

    Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, et al. (2020) Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:200208155

    2020

  19. [29]

    Gethers M, Savage T, Di Penta M, Oliveto R, Poshyvanyk D, De Lucia A (2011) Codetopics: W hich topic am I coding now? In: Proceedings of the 33rd International Conference on Software Engineering, pp 1034--1036

    2011

  20. [30]

    PhD thesis

    Hill E (2010) Integrating natural language and program structure information to improve software search and exploration. PhD thesis

    2010

  21. [31]

    Commun ACM 59(5):122--131, doi:10.1145/2902362, ://doi.org/10.1145/2902362

    Hindle A, Barr ET, Gabel M, Su Z, Devanbu PT (2016) On the naturalness of software. Commun ACM 59(5):122--131, doi:10.1145/2902362, ://doi.org/10.1145/2902362

    2016 doi

  22. [32]

    In: 2017 IEEE 24th International conference on software analysis, evolution and reengineering (SANER), IEEE, pp 217--227

    Hofmeister J, Siegmund J, Holt DV (2017) Shorter identifier names take longer to comprehend. In: 2017 IEEE 24th International conference on software analysis, evolution and reengineering (SANER), IEEE, pp 217--227

    2017

  23. [33]

    In: 12th International Conference on Learning Representations, ICLR 2024, 2308.00352

    Hong S, Zhuge M, Chen J, Zheng X, Cheng Y, Zhang C, Wang J, Wang Z, Yau SKS, Lin Z, Zhou L, Ran C, Xiao L, Wu C, Schmidhuber J (2024) METAGPT: META PROGRAMMING FOR A MULTI-AGENT COLLABORATIVE FRAMEWORK . In: 12th International Conference on Learning Representations, ICLR 2024,...

    2024 arXiv

  24. [34]

    https://spacy.io/ Accessed August 18, 2024

    Honnibal M, Montani I (2024) spacy. https://spacy.io/ Accessed August 18, 2024

    2024

  25. [35]

    In: European Conference on Object-Oriented Programming, Springer, pp 294--317

    H st EW, stvold BM (2009) Debugging method names. In: European Conference on Object-Oriented Programming, Springer, pp 294--317

    2009

  26. [36]

    Proceedings - 2019 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019 pp 602--614, doi:10.1109/ASE.2019.00062

    Jiang L, Liu H, Jiang H (2019) Machine learning based recommendation of method names: H ow far are we . Proceedings - 2019 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019 pp 602--614, doi:10.1109/ASE.2019.00062

    2019

  27. [37]

    Johnson P (2018 a ) Arg! the 9 hardest things programmers have to do,”

    2018

  28. [38]

    Johnson P (2018 b ) Don’t go into programming if you don’t have a good thesaurus

    2018

  29. [39]

    12701v1 [ cs

    Kim K, Zhou XIN, Kim D, Lawall J, Liu KUI, Klein J, Lee J, Lo D (2023) How are We Detecting Inconsistent Method Names ? An Empirical Study from Code Review Perspective arXiv : 2308 . 12701v1 [ cs . SE ] 24 Aug 2023 . arXiv preprint arXiv:230812701v1 1(1):1--22, arXiv:2308.12701v1

    2023 arXiv

  30. [40]

    Empirical Software Engineering 21(2):565--604

    Kim S, Kim D (2016) Automatic identifier inconsistency detection using code dictionary. Empirical Software Engineering 21(2):565--604

    2016

  31. [41]

    IEEE Transactions on software engineering 28(8):721--734

    Kitchenham BA, Pfleeger SL, Pickard LM, Jones PW, Hoaglin DC, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Transactions on software engineering 28(8):721--734

    2002

  32. [42]

    https://cran.r-project.org/web/packages/VennDiagram/ Accessed August 18, 2024

    Language R (2024) Venndiagram. https://cran.r-project.org/web/packages/VennDiagram/ Accessed August 18, 2024

    2024

  33. [43]

    IEEE International Conference on Program Comprehension 2006:3--12, doi:10.1109/ICPC.2006.51

    Lawrie D, Morrell C, Feild H, Binkley D (2006) What's in a name? A study of identifiers . IEEE International Conference on Program Comprehension 2006:3--12, doi:10.1109/ICPC.2006.51

    2006 doi

  34. [44]

    Le QV, Mikolov T (2014) Distributed representations of sentences and documents. In: Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, JMLR.org, JMLR Workshop and Conference Proceedings, vol 32, pp 1188--1196, ://p...

    2014

  35. [45]

    Proceedings - Asia-Pacific Software Engineering Conference, APSEC 2021-December:22 31, doi:10.1109/APSEC53868.2021.00010

    Li K, Wang T, Liu H (2021 a ) NameChecker: Detecting Inconsistency between Method Names and Method Bodies . Proceedings - Asia-Pacific Software Engineering Conference, APSEC 2021-December:22 31, doi:10.1109/APSEC53868.2021.00010

    2021

  36. [46]

    https://github.com/deepname2021icse/DeepName-2021-ICSE Accessed August 18, 2024

    Li Y (2024) Deepname-2021-icse. https://github.com/deepname2021icse/DeepName-2021-ICSE Accessed August 18, 2024

    2024

  37. [47]

    Li Y, Wang S, Nguyen TN (2021 b ) A context-based automated approach for method name consistency checking and suggestion. In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, IEEE , pp 574--586, doi:10.1109/ICSE43902.202...

    2021

  38. [48]

    Liblit B, Begel A, Sweetser E (2006) Cognitive perspectives on the role of naming in computer programs. In: Proceedings of the 18th Annual Workshop of the Psychology of Programming Interest Group, PPIG 2006, Brighton, UK, September 7-8, 2006, Psychology of Programming Interest...

    2006

  39. [49]

    In: 2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM), pp 81--90, doi:10.1109/SCAM.2017.17

    Lin B, Scalabrino S, Mocci A, Oliveto R, Bavota G, Lanza M (2017) Investigating the use of code analysis and NLP to promote a consistent usage of identifiers. In: 2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM), pp 81--90, doi:10...

    2017 doi

  40. [50]

    Lin B, Nagy C, Bavota G, Lanza M (2019) On the impact of refactoring operations on code naturalness. In: Wang X, Lo D, Shihab E (eds) 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019, IEEE ,...

    2019

  41. [51]

    In: 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM), pp 204--215, doi:10.1109/SCAM.2019.00031

    Lin B, Nagy C, Bavota G, Marcus A, Lanza M (2019) On the quality of identifiers in test code. In: 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM), pp 204--215, doi:10.1109/SCAM.2019.00031

    2019

  42. [52]

    In: Proceedings of the 44th International Conference on Software Engineering, pp 1294--1306

    Liu F, Li G, Fu Z, Lu S, Hao Y, Jin Z (2022) Learning to recommend method names with global context. In: Proceedings of the 44th International Conference on Software Engineering, pp 1294--1306

    2022

  43. [53]

    IEEE Transactions on Software Engineering 41(9):887--900, doi:10.1109/TSE.2015.2427831

    Liu H, Liu Q, Liu Y, Wang Z (2015) Identifying renaming opportunities by expanding conducted rename refactorings. IEEE Transactions on Software Engineering 41(9):887--900, doi:10.1109/TSE.2015.2427831

    2015

  44. [54]

    Liu H, Shen M, Zhu J, Niu N, Li G, Zhang L (2020) Deep learning based program generation from requirements text: Are we there yet? IEEE Transactions on Software Engineering 48(4):1268--1289

    2020

  45. [55]

    https://github.com/SerVal-DTF/debug-method-name Accessed August 18, 2024

    Liu K (2024) debug-method-name. https://github.com/SerVal-DTF/debug-method-name Accessed August 18, 2024

    2024

  46. [56]

    Proceedings - International Conference on Software Engineering 2019-May:1--12, doi:10.1109/ICSE.2019.00019

    Liu K, Kim D, Bissyande TF, Kim T, Kim K, Koyuncu A, Kim S, Le Traon Y (2019) Learning to Spot and Refactor Inconsistent Method Names . Proceedings - International Conference on Software Engineering 2019-May:1--12, doi:10.1109/ICSE.2019.00019

    2019

  47. [57]

    arXiv 3(January), 1911.07605

    Lozoya RC, Baumann A, Sabetta A, Bezzi M (2019) Code2Vec: L earning distributed representations of code changes . arXiv 3(January), 1911.07605

    2019 arXiv

  48. [58]

    Neural Networks 16(5-6):555--559, doi:10.1016/S0893-6080(03)00115-1, ://doi.org/10.1016/S0893-6080(03)00115-1

    Matsugu M, Mori K, Mitari Y, Kaneda Y (2003) Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Networks 16(5-6):555--559, doi:10.1016/S0893-6080(03)00115-1, ://doi.org/10.1016/S0893-6080(03)00115-1

    2003 doi

  49. [59]

    IEEE Transactions on software Engineering (4):308--320

    McCabe TJ (1976) A complexity measure. IEEE Transactions on software Engineering (4):308--320

    1976

  50. [60]

    https://github.com/microsoft/nni Accessed August 18, 2024

    Microsoft (2024) Neural network intelligence. https://github.com/microsoft/nni Accessed August 18, 2024

    2024

  51. [61]

    In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013

    Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of...

    2013

  52. [62]

    Springer International Publishing

    Minehisa T, Aman H, Yokogawa T, Kawahara M (2021) A Comparative Study of Vectorization Approaches for Detecting Inconsistent Method Names , vol 985. Springer International Publishing

    2021

  53. [63]

    Journal of Systems and Software 170:110740

    Newman CD, AlSuhaibani RS, Decker MJ, Peruma A, Kaushik D, Mkaouer MW, Hill E (2020) On the generation, structure, and semantics of grammar patterns in source code identifiers. Journal of Systems and Software 170:110740

    2020

  54. [64]

    Nguyen S, Phan H, Le T, Nguyen TN (2020) Suggesting natural method names to check name consistencies. In: ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020, ACM , pp 1372--1384, doi:10.1145/3377811.3380926, ://doi.org/...

    2020

  55. [65]

    Noever D (2023) Can large language models find and fix vulnerable software? arXiv preprint arXiv:230810345

    2023

  56. [66]

    https://bioinfogp.cnb.csic.es/tools/venny/index.html Accessed August 18, 2024

    Oliveros J (2024) Venny. https://bioinfogp.cnb.csic.es/tools/venny/index.html Accessed August 18, 2024

    2024

  57. [67]

    arXiv preprint arXiv:180703748

    Oord Avd, Li Y, Vinyals O (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:180703748

    2018

  58. [68]

    Cengage Learning

    Ott RL, Longnecker MT (2015) An introduction to statistical methods and data analysis. Cengage Learning

    2015

  59. [69]

    IWoR 2018 - Proceedings of the 2nd International Workshop on Refactoring, co-located with ASE 2018 (August):26--33, doi:10.1145/3242163.3242169

    Peruma A, Mkaouer MW, Decker MJ, Newman CD (2018) An empirical investigation of how and why developers rename identifiers . IWoR 2018 - Proceedings of the 2nd International Workshop on Refactoring, co-located with ASE 2018 (August):26--33, doi:10.1145/3242163.3242169

    2018

  60. [70]

    naturalness

    Ray B, Hellendoorn V, Godhane S, Tu Z, Bacchelli A, Devanbu PT (2016) On the "naturalness" of buggy code. In: Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, ACM , pp 428--439, doi:10.1145/2884781.2884848, ...

    2016

  61. [71]

    Empirical software engineering 14:131--164

    Runeson P, H \"o st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14:131--164

    2009

  62. [72]

    Wiley Interdiscip Rev Data Min Knowl Discov 7(2), doi:10.1002/widm.1200, ://doi.org/10.1002/widm.1200

    Scardapane S, Wang D (2017) Randomness in neural networks: A n overview. Wiley Interdiscip Rev Data Min Knowl Discov 7(2), doi:10.1002/widm.1200, ://doi.org/10.1002/widm.1200

    2017 doi

  63. [73]

    Proceedings - International Conference on Software Engineering pp 31--40, doi:10.1145/3196321.3196332

    Schankin A, Berger A, Holt DV, Hofmeister JC, Riedel T, Beigl M (2018) Descriptive compound identifier names improve source code comprehension . Proceedings - International Conference on Software Engineering pp 31--40, doi:10.1145/3196321.3196332

    2018

  64. [74]

    In: 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation, IEEE, pp 67--76

    Singer J, Kirkham C (2008) Exploiting the correspondence between micro patterns and class names. In: 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation, IEEE, pp 67--76

    2008

  65. [75]

    In: Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012

    Snoek J, Larochelle H, Adams RP (2012) Practical bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 20...

    2012

  66. [76]

    22nd International Conference on Program Comprehension, ICPC 2014 - Proceedings pp 271--274, doi:10.1145/2597008.2597797

    Suzuki T, Sakamoto K, Ishikawa F, Honiden S (2014) An approach for evaluating and suggesting method names using N-gram models . 22nd International Conference on Program Comprehension, ICPC 2014 - Proceedings pp 271--274, doi:10.1145/2597008.2597797

    2014

  67. [77]

    J Prog Lang 4(3):143--167

    Takang AA, Grubb PA, Macredie RD (1996) The effects of comments and identifier names on program comprehensibility: A n experimental investigation. J Prog Lang 4(3):143--167

    1996

  68. [78]

    In: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, pp 121--134

    Wang K, Su Z (2020) Blended, precise semantic program embeddings. In: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, pp 121--134

    2020

  69. [79]

    In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 741--753

    Wang S, Wen M, Lin B, Mao X (2021 a ) Lightweight global and local contexts guided method name recommendation with prior knowledge. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineerin...

    2021

  70. [80]

    https://github.com/Michaelll123/EmpiricalStudy-MCC Accessed August 18, 2024

    Wang T (2024) Empiricalstudy-mcc. https://github.com/Michaelll123/EmpiricalStudy-MCC Accessed August 18, 2024

    2024

  71. [81]

    arXiv preprint arXiv:210900859

    Wang Y, Wang W, Joty S, Hoi SC (2021 b ) Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:210900859

    2021

  72. [82]

    IEEE International Conference on Program Comprehension pp 60--71, doi:10.1145/3387904.3389266

    Wen F, Nagy C, Lanza M, Bavota G (2020) An empirical study of quick remedy commits . IEEE International Conference on Program Comprehension pp 60--71, doi:10.1145/3387904.3389266

    2020

  73. [83]

    Empirical Software Engineering 27(1):1--31, doi:10.1007/s10664-021-10051-z

    Wen F, Nagy C, Lanza M, Bavota G (2022) Quick remedy commits and their impact on mining software repositories . Empirical Software Engineering 27(1):1--31, doi:10.1007/s10664-021-10051-z

    2022 doi

  74. [84]

    arXiv preprint arXiv:230400385

    Xia CS, Zhang L (2023) Keep the conversation going: Fixing 162 out of 337 bugs for \ 0.42 each using ChatGPT . arXiv preprint arXiv:230400385

    2023

  75. [85]

    arXiv preprint arXiv:240701489

    Xia CS, Deng Y, Dunn S, Zhang L (2024) Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:240701489

    2024

  76. [86]

    CoRR abs/2106.11872, ://arxiv.org/abs/2106.11872, 2106.11872

    Zhuang D, Zhang X, Song SL, Hooker S (2021) Randomness in neural network training: Characterizing the impact of tooling. CoRR abs/2106.11872, ://arxiv.org/abs/2106.11872, 2106.11872

    2021 arXiv

  77. [87]

    u gner D, Kirschstein T, Catasta M, Leskovec J, G \

    Z \"u gner D, Kirschstein T, Catasta M, Leskovec J, G \"u nnemann S (2021) Language-agnostic representation learning of source code from structure and context. arXiv preprint arXiv:210311318

    2021

  78. [88]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mi...

  79. [89]

    write newline

    " write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...

  80. [90]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  81. [91]

    write newline

    " write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...

  82. [92]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  83. [93]

    write newline

    " write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.