Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Code-comment coherence, a widely studied quality attribute of summaries, turns out to be no better than random selection for pruning code summarization training sets.

desk verdict A solid, honestly-reported negative result: coherence-based pruning doesn't beat random pruning for code summarization, but the random-baseline comparison rests on a single draw per dataset. read the letter →

arxiv 2502.07611 v1 pith:GA4V5EKF submitted 2025-02-11 cs.SE

Antonio Vitale , Antonio Mastropaolo , Rocco Oliveto , Massimiliano Di Penta , Simone Scalabrino This is my paper · ORCID
classification cs.SE
keywords codesummarizationdataqualitycode-commentcoherenceSIDEmetricdatasetfilteringempiricalstudytransformerfine-tuningrandombaseline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether filtering the training examples of a code summarization model by code-comment coherence improves the generated summaries. Using the SIDE metric, which scores how well a comment matches its code, the authors filter the TL-CodeSum and Funcom datasets at five thresholds, retrain CodeT5+ on each filtered set, and evaluate on two human-curated test sets. They find that even the most aggressive filter, which keeps only about half the training instances, leaves summary quality statistically indistinguishable from training on the full data. When they compare the coherence-filtered set with a randomly sampled set of the same size, the two models perform equally well. The upshot is that these datasets contain many examples that add nothing to model performance, and coherence is not the quality attribute that explains which ones matter.

What carries the argument

The load-bearing object is SIDE (Summary alIgnment to coDe sEmantics), a contrastively trained metric that assigns each code-summary pair a score from -1 to 1, with high scores indicating that the summary accurately describes the code. The study uses SIDE as a filter: for thresholds 0.5, 0.6, 0.7, 0.8, and 0.9, it keeps only training instances whose SIDE score meets or exceeds the threshold, producing progressively smaller training sets (down to roughly 50% of the original). It then fine-tunes the CodeT5+ transformer on each filtered set and compares output quality on three human-curated benchmarks using BLEU-4, METEOR, and ROUGE-L, with Wilcoxon signed-rank tests and Cliff's delta to judge whether differences are meaningful. The decisive comparison is between the SIDE-0.9-filtered model and a random baseline that keeps the same number of instances by chance; the equivalence of these two models is what undermines coherence as a selection principle.

What would settle it

A direct test: train the same model on a coherence-filtered half and a random half of a code summarization corpus, then evaluate on a substantially larger human-annotated benchmark (thousands of instances rather than the hundreds used here) across multiple random seeds. If the coherence-filtered model shows a consistent, statistically significant advantage over the random model on that larger set, the paper's no-difference conclusion would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that code-comment coherence, as measured by SIDE, is not a useful criterion for pruning code summarization training sets. A model fine-tuned on the half of the data with the highest SIDE scores is no better than a model fine-tuned on a random half of the same data, and both are statistically indistinguishable from a model trained on the full dataset. This holds across two training corpora (TL-CodeSum and Funcom) and three manually curated test sets: after Holm correction, the Wilcoxon tests show no meaningful differences between the filtered, random, and full-data models, and the Cliff's delta effect sizes are negligible. The authors read this as evidence that state-of-the-art code summarization datasets are bloated with redundant instances, and that future dataset construction should target informativeness and diversity rather than coherence alone.

Load-bearing premise

The manual test sets (218 CoderEval instances and 892 Mastropaolo et al. instances) are large enough and sensitive enough for the Wilcoxon tests and Cliff's delta to detect a practically meaningful improvement in summary quality if one existed; because the paper's conclusion is that there is no difference, low statistical power could mask a real benefit of coherence filtering.

Editorial extensions

If this is right

  • Cutting code summarization training sets in half, even by random selection, does not degrade summary quality, so training time and energy costs can be cut roughly in half without sacrificing effectiveness; the paper reports up to roughly 111 saved hours and an estimated 55% drop in CO2 emissions for the larger dataset.
  • Code-comment coherence, despite being well correlated with human judgment of summary quality, is not the attribute that determines which training instances matter.
  • Current mined datasets are over-redundant: many instances contribute nothing to the learned model, so dataset builders should prioritize diverse and informative examples over sheer volume.
  • Other quality attributes, such as readability or instance diversity, should be tested as selection criteria, since coherence alone does not differentiate useful from useless training data.
  • The finding that random pruning matches coherence-based pruning implies that even simpler dataset reduction strategies than SIDE filtering are viable in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If random pruning is as good as coherence pruning at half the dataset, then the marginal value of any single content-quality metric for selection may be low; a more promising direction could be coverage- or diversity-based selection (for example, coresets or clustering by code structure), which the paper does not test.
  • The result likely extends to larger models, since bigger transformers are even less sensitive to individual noisy instances, but that extrapolation is untested.
  • Because the paper starts from CAT-cleaned datasets, the finding does not address whether coarse structural cleaning (removing getters, commented-out code, and similar) is still necessary; coherence filtering could behave differently on uncleaned data.
  • A natural extension is to find the random-pruning ceiling: how far can the training set shrink before summary quality starts to drop? The paper's data suggest the ceiling lies below the sizes tested, but the study does not locate it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper empirically evaluates whether filtering code summarization training instances by code-comment coherence, measured with SIDE, yields smaller training sets without hurting model effectiveness. The authors fine-tune CodeT5+ 220M on the CAT-cleaned TL-CodeSum and Funcom datasets, using the full sets and five SIDE-threshold subsets (0.5 through 0.9), and evaluate on two manually curated test sets (CoderEval and Mastropaolo et al.) with BLEU-4, METEOR, and ROUGE-L. For RQ2 they compare the most restrictive SIDE0.9 subset with a randomly drawn subset of the same size. The results show no statistically significant differences between the full and filtered training settings in most comparisons, about 50% dataset/token reduction with comparable metrics, and a large estimated saving in training time and CO2 emissions. The main claim is that coherence-based filtering is no better than random selection and that current datasets contain many redundant instances.

Significance. If the results hold, the paper is a useful negative result for the data-centric AI direction in code summarization: it challenges the assumption that coherence-based instance filtering improves downstream effectiveness and redirects attention to alternative quality attributes. The study is well designed in several respects: two large real-world datasets, three manually curated test sets, six training subsets per dataset, multiple evaluation metrics, Holm-corrected Wilcoxon tests with Cliff's delta effect sizes, and a public replication package. The training-time and CO2 estimates are a practical addition. The main caveats are the use of a single CodeT5+ variant, a single random draw for the RQ2 baseline, and reliance on failure-to-reject as evidence of equivalence; these limit the strength of the central negative claim as currently stated.

major comments (3)
  1. [Section III-B (RQ2), Tables III-IV] The RQ2 comparison between SIDE0.9 and Random is based on a single randomly drawn subset of the same size for each dataset. Random subset selection is stochastic, so one draw cannot characterize the distribution of random baselines; if the particular subset happened to be unusually good or poor, the conclusion that coherence filtering and random selection are equivalent could be an artifact. Please sample multiple random subsets (e.g., 10 or more seeds), report the spread of the resulting metric values or an interval for the random baseline, and compare SIDE0.9 against that distribution.
  2. [Section IV-C, Table IV] The paper uses failure to reject the Wilcoxon null hypothesis as evidence that SIDE0.9 and Random (and, in RQ1, full versus filtered training) have comparable effectiveness. With test sets of 218 and 892 instances, these tests may lack power to detect small but practically meaningful differences, and no equivalence margins or confidence intervals for the differences are reported. The single significant result in Table IV (Funcom BLEU-4 on the Mastropaolo test set, p=0.017) shows the tests are not uniformly insensitive. I recommend pre-specifying an equivalence bound (e.g., a two one-sided test on paired metric differences) or at least reporting confidence intervals and a sensitivity or power analysis for the reported tests.
  3. [Section V and answer to RQ1] The interpretation of RQ1 as evidence that 'state-of-the-art datasets contain instances that do not contribute to improving the models' effectiveness' goes beyond what the statistical tests establish. The tests support absence of statistically significant differences, not absence of practically meaningful differences, especially because the raw scores in Table II show non-monotonic relationships with filtering level (e.g., SIDE0.6 is often best while SIDE0.9 is slightly worse). Please either temper the wording or add the equivalence and power analysis described above so that the no-effect conclusion is actually supported.
minor comments (6)
  1. [Figure 3 caption] The caption cites 'Funcom [58]', but the Funcom dataset reference is [9]; reference [58] is a different paper. Please correct the citation.
  2. [Table IV caption] The caption reads 'RESULTS COMPARISON FOR CODESUM AND FUNCOM'; 'CODESUM' is ambiguous and should be 'TL-CodeSum' for consistency with the rest of the paper.
  3. [Table II header] The arrows in 'Full ↑' and 'SIDE0.9 ↓' are not explained; please clarify what they denote in the table caption or in the text.
  4. [Section IV-B] The detailed Wilcoxon signed-rank test results for RQ1 are only in the replication package; including the p-values and effect sizes at least for the main comparisons would make the paper more self-contained.
  5. [Section III-A] The description of CoderEval says '230 Python and 230 Java code generation problems' but the study uses only the Java subset and ends up with 218 instances; this is fine, but the transition from 230 to 218 could be stated more explicitly in the same paragraph.
  6. [Section IV-A] The phrase 'sub-optimal SIDE score (<0.9)' is vague; consider defining what makes a SIDE score sub-optimal for this study, since the paper later shows that filtering at 0.9 is not better than random.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: SIDE is self-cited but externally validated, and the central comparison with random selection is not forced by construction.

full rationale

The paper makes no fitted-parameter-then-prediction move. SIDE is used only as a filter; the test metrics (BLEU, METEOR, ROUGE) are external to SIDE, and the authors explicitly avoid using SIDE for evaluation, noting that doing so 'could therefore be unnaturally biased in favor of models trained on filtered datasets.' The central RQ2 comparison (SIDE0.9 vs one random subset) is a statistical-sensitivity weakness, not circularity: the no-difference conclusion could be affected by sampling noise and low power, but it is not true by definition. The only overlapping-author dependency is the SIDE metric from Mastropaolo et al. [33], whose human-correlation validation is external and falsifiable; thus the citation is real evidence rather than a circular premise. The manually curated test sets include one from the same group [38], but they are independent, human-validated resources. No equation is defined in terms of the target result, and no self-citation chain forces the conclusion.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The only hand-chosen inputs are the SIDE thresholds. The core conclusion relies on standard assumptions about SIDE validity, test-set sensitivity, and model representativeness, all of which are reasonable but limit generalization.

free parameters (1)
  • SIDE threshold values = 0.5, 0.6, 0.7, 0.8, 0.9
    Hand-chosen operating points to test increasing selectivity; they are not fitted to the outcome, and the paper reports results for all of them.
assumptions (4)
  • domain assumption SIDE scores reliably measure code-comment coherence.
    The study filters training data based on SIDE thresholds; if SIDE does not measure coherence, the RQ1/RQ2 conclusions describe SIDE-based selection rather than coherence-based selection generally. This is cited from prior work [33].
  • domain assumption The manual test sets (CoderEval and Mastropaolo et al.) provide valid and sufficiently sensitive ground truth for detecting meaningful changes in summary quality.
    The main equivalence conclusion relies on the ability of these small sets to detect differences; with 218 and 892 instances, statistical power is limited.
  • domain assumption CodeT5+ 220M is a representative model for code summarization.
    Only one model is used; results may not generalize to larger models, though the paper scopes its claims accordingly.
  • domain assumption CAT-cleaned datasets are the appropriate starting point.
    The study builds on CAT-cleaned TL-CodeSum and Funcom, so conclusions apply to already-cleaned datasets, not to raw mined pairs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?." pith.science (2026). https://pith.science/paper/GA4V5EKF

@misc{pith2026250207611,
  author       = {Pith},
  title        = {Pith review of: Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GA4V5EKF}},
  note         = {Machine review of arXiv:2502.07611}
}
read the original abstract

Automated code summarization is a long-standing goal for code comprehension. This task automatically generates documentation using a given method. Deep Learning (DL)-based approaches have been proven beneficial for various software engineering (SE) tasks, including this one. Most state-of-the-art datasets for code summarization are automatically mined from GitHub and, thus, might contain erroneous or sub-optimal examples. Previous work showed that using a simple rule-based approach for removing noisy instances allows for a tangible reduction of the training set size while not reducing the effectiveness of the trained models. Motivated by this finding, we conjecture that it is possible to further reduce the dataset size by removing instances that contain different issues. In this paper, we explore the extent to which code-comment coherence, a specific quality attribute of code summaries, can be used to optimize code summarization datasets. Specifically, we hypothesize that removing incoherent code-comment pairs might positively impact the effectiveness of the models. To do this, we rely on SIDE, a recently introduced metric for code-summary coherence. We examine multiple selectivity levels of training instances from two state-of-the-art datasets (TL-CodeSum and Funcom) and evaluate the resulting models on three manually curated test sets. The results show that even halving the training set sizes does not significantly affect the model's ability to generate summaries. However, when comparing the most restrictive selection strategy with a simpler one that randomly selects the training instances, we observe that the resulting accuracy of the model also does not change. This result suggests that (i) current datasets contain many irrelevant examples, and (ii) different quality attributes should be explored for optimizing code summarization datasets.

Figures

Figures reproduced from arXiv: 2502.07611 by the authors.

Figure 1
Figure 1. Example of a ⟨code, summary⟩ pair exhibiting a misalignment after the coarse-grained filtering by CAT. Modern DL-based approaches based on Transformers [13], [14], [15] rely on transfer learning. First, a basic model is pre￾trained to acquire knowledge of the programming language. Then, several specialized models can be fine-tuned from it to tackle specific tasks. This approach has been proven effective for several … view at source ↗
Figure 2
Figure 2. Distribution of SIDE scores for TL-CodeSum and Funcom training instances. trained on the full training sets. We use Holm’s correction [57] to adjust the p-values for the multiple tests. We reject the null hypothesis (there is no difference between the effectiveness of two given models) if the p-value is lower than 0.05. Finally, we study the Pareto front to analyze the cost-benefit trade-offs between the effectivene… view at source ↗
Figure 3
Figure 3. Pareto front for TL-CodeSum [24] (top) and Funcom [58] (bottom). From left to right, the chart shows the number of tokens for SIDE0.9, SIDE0.8, SIDE0.7, SIDE0.6, SIDE0.5, and the full dataset. sizes, showing minimal improvements. In the Mastropaolo et al. dataset, the three metrics remain constant across all filtering levels, evidencing that increasing dataset size does not meaningfully impact the models’ effectiven… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization

    cs.SE 2026-01 reject novelty 6.0 of 10

    The best token-pruning strategy for code summarization is language-dependent: AST inputs help Java but hurt Python, while function signatures are best for Python.

  2. Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

    cs.SE 2026-01 conditional novelty 5.0 of 10

    Multi-task QLoRA on Qwen2.5-Coder matches or beats single-task QLoRA and full fine-tuning for code generation and Python summarization, but lags in Java-to-C# translation.

  3. CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement

    cs.SE 2025-05 reject novelty 5.0 of 10

    CIDRe is a reference-free, four-part quality score for structured code comments, validated on 840 manually labeled Russian comments and reported to improve comment generation after dataset filtering.

Reference graph

Works this paper leans on

72 extracted references · 42 canonical work pages · cited by 3 Pith papers

  1. [1]

    Do code and comments co- evolve? on the relation between source code and comment changes,

    B. Fluri, M. Wursch, and H. C. Gall, “Do code and comments co- evolve? on the relation between source code and comment changes,” in 14th Working conference on reverse engineering (WCRE 2007) . IEEE, 2007, pp. 70–79

    work page 2007

  2. [2]

    Analyzing the co- evolution of comments and source code,

    B. Fluri, M. Würsch, E. Giger, and H. C. Gall, “Analyzing the co- evolution of comments and source code,” Software Quality Journal , vol. 17, pp. 367–394, 2009

    work page 2009

  3. [3]

    How do developers document database usages in source code?(n),

    M. Linares-Vásquez, B. Li, C. Vendome, and D. Poshyvanyk, “How do developers document database usages in source code?(n),” in 2015 30th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2015, pp. 36–41

    work page 2015

  4. [4]

    A large-scale empirical study on code-comment inconsistencies,

    F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A large-scale empirical study on code-comment inconsistencies,” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 2019, pp. 53–64

    2019

  5. [5]

    A decade of code comment quality assessment: A systematic literature review,

    P. Rani, A. Blasi, N. Stulova, S. Panichella, A. Gorla, and O. Nierstrasz, “A decade of code comment quality assessment: A systematic literature review,” Journal of Systems and Software , vol. 195, p. 111515, 2023

    work page 2023

  6. [6]

    A study of the documentation essential to software maintenance,

    S. C. B. de Souza, N. Anquetil, and K. M. de Oliveira, “A study of the documentation essential to software maintenance,” in Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information , 2005, pp. 68–75

    work page 2005

  7. [7]

    Improved auto- matic summarization of subroutines via attention to file context,

    S. Haque, A. LeClair, L. Wu, and C. McMillan, “Improved auto- matic summarization of subroutines via attention to file context,” in Proceedings of the 17th International Conference on Mining Software Repositories, 2020, pp. 300–310

    work page 2020

  8. [8]

    Deep code comment generation with hybrid lexical and syntactical information,

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation with hybrid lexical and syntactical information,” Empirical Software Engineering, vol. 25, pp. 2179–2217, 2020

    2020

Show all 72 references
  1. [9]

    A neural model for gener- ating natural language summaries of program subroutines,

    A. LeClair, S. Jiang, and C. McMillan, “A neural model for gener- ating natural language summaries of program subroutines,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 795–806

    2019

  2. [10]

    Retrieval-based neural source code summarization,

    J. Zhang, X. Wang, H. Zhang, H. Sun, and X. Liu, “Retrieval-based neural source code summarization,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE 2020) , 2020, pp. 1385–1397

    2020

  3. [11]

    Studying the usage of text-to-text transfer transformer to support code-related tasks,

    A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshy- vanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2...

    2021

  4. [12]

    Code structure–guided transformer for source code summarization,

    S. Gao, C. Gao, Y . He, J. Zeng, L. Nie, X. Xia, and M. Lyu, “Code structure–guided transformer for source code summarization,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 1, pp. 1–32, 2023

    2023

  5. [13]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  7. [15]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  8. [16]

    Code generation as a dual task of code summarization,

    B. Wei, G. Li, X. Xia, Z. Fu, and Z. Jin, “Code generation as a dual task of code summarization,” Advances in neural information processing systems, vol. 32, 2019

    2019

  9. [17]

    Intellicode compose: Code generation using transformer,

    A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering , 2020, pp. 1433–1443

    2020

  10. [18]

    Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  11. [19]

    Improv- ing llm code generation with grammar augmentation,

    S. Ugare, T. Suresh, H. Kang, S. Misailovic, and G. Singh, “Improv- ing llm code generation with grammar augmentation,” arXiv preprint arXiv:2403.01632, 2024

    2024 arXiv

  12. [20]

    Inferfix: End-to-end program repair with llms,

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp....

    2023

  13. [21]

    Neural transfer learning for repairing security vulnerabilities in C code,

    Z. Chen, S. Kommrusch, and M. Monperrus, “Neural transfer learning for repairing security vulnerabilities in C code,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 147–165, 2022

    2022

  14. [22]

    Sequencer: Sequence-to-sequence learning for end- to-end program repair,

    Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end- to-end program repair,” IEEE Transactions on Software Engineering , vol. 47, no. 9, pp. 1943–1959, 2019

    1943

  15. [23]

    An empirical investigation into learning bug-fixing patches in the wild via neural machine translation,

    M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “An empirical investigation into learning bug-fixing patches in the wild via neural machine translation,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ...

    2018

  16. [24]

    Summarizing source code with transferred API knowledge,

    X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred API knowledge,” in Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden . ijcai.org, 2018, pp. 2269– 2275

    2018

  17. [25]

    Deep code comment genera- tion,

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment genera- tion,” in Proceedings of the 26th conference on program comprehension (ICPC 2010), 2018, pp. 200–210

    2010

  18. [26]

    Improved code summarization via a graph neural network,

    A. LeClair, S. Haque, L. Wu, and C. McMillan, “Improved code summarization via a graph neural network,” in Proceedings of the 28th international conference on program comprehension (ICPC 2020), 2020, pp. 184–195

    2020

  19. [27]

    Fair and balanced?: bias in bug-fix datasets,

    C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V . Filkov, and P. T. Devanbu, “Fair and balanced?: bias in bug-fix datasets,” in Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Sympo- sium on Founda...

    2009

  20. [28]

    It’s not a bug, it’s a feature: how misclassification impacts bug prediction,

    K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: how misclassification impacts bug prediction,” in 35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013 . IEEE Computer Society, 2013, pp. 392–401

    2013

  21. [29]

    The missing links: bugs and bug-fix commits,

    A. Bachmann, C. Bird, F. Rahman, P. T. Devanbu, and A. Bernstein, “The missing links: bugs and bug-fix commits,” in Proceedings of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2010, Santa Fe, NM, USA, November 7-11, 2010. ACM, 2010, pp. 97–106

    2010

  22. [30]

    Are we building on the rock? on the importance of data preprocessing for code summarization,

    L. Shi, F. Mu, X. Chen, S. Wang, J. Wang, Y . Yang, G. Li, X. Xia, and Q. Wang, “Are we building on the rock? on the importance of data preprocessing for code summarization,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Fou...

    2022

  23. [31]

    On the origin of hallucinations in conversational models: Is it the datasets or the models?

    N. Dziri, S. Milton, M. Yu, O. Zaiane, and S. Reddy, “On the origin of hallucinations in conversational models: Is it the datasets or the models?” arXiv preprint arXiv:2204.07931 , 2022

    2022 arXiv

  24. [32]

    On the coherence between comments and implementations in source code,

    A. Corazza, V . Maggio, and G. Scanniello, “On the coherence between comments and implementations in source code,” in 2015 41st Euromicro Conference on Software Engineering and Advanced Applications. IEEE, 2015, pp. 76–83

    2015

  25. [33]

    Evaluat- ing code summarization techniques: A new metric and an empirical characterization,

    A. Mastropaolo, M. Ciniselli, M. Di Penta, and G. Bavota, “Evaluat- ing code summarization techniques: A new metric and an empirical characterization,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  26. [34]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  27. [36]

    Codet5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,” arXiv preprint arXiv:2305.07922 , 2023

    2023 arXiv

  28. [37]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

    2024

  29. [38]

    On the robustness of code generation techniques: An empirical study on Github CoPilot,

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota, “On the robustness of code generation techniques: An empirical study on Github CoPilot,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2...

    2023

  30. [39]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  31. [40]

    METEOR: An automatic metric for MT eval- uation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT eval- uation with improved correlation with human judgments,” in Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, ACL , 2005, pp. 65–72

    2005

  32. [41]

    MPNet: masked and permuted pre-training for language understanding,

    K. Song, X. Tan, T. Qin, J. Lu, and T.-Y . Liu, “MPNet: masked and permuted pre-training for language understanding,” ArXiv, vol. abs/2004.09297, 2020. [Online]. Available: https://api.semanticscholar. org/CorpusID:215827489

    2004 arXiv

  33. [42]

    Automatic semantic augmentation of language model prompts (for code summarization),

    T. Ahmed, K. S. Pai, P. Devanbu, and E. Barr, “Automatic semantic augmentation of language model prompts (for code summarization),” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  34. [43]

    Repohyper: Better context retrieval is all you need for repository-level code com- pletion,

    H. N. Phan, H. N. Phan, T. N. Nguyen, and N. D. Bui, “Repohyper: Better context retrieval is all you need for repository-level code com- pletion,” arXiv preprint arXiv:2403.06095 , 2024

    2024 arXiv

  35. [44]

    How important are good method names in neural code generation? a model robustness perspective,

    G. Yang, Y . Zhou, W. Yang, T. Yue, X. Chen, and T. Chen, “How important are good method names in neural code generation? a model robustness perspective,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, pp. 1–35, 2024

    2024

  36. [45]

    Alex- atm 20b: Few-shot learning using a large-scale multilingual seq2seq model,

    S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky et al., “Alex- atm 20b: Few-shot learning using a large-scale multilingual seq2seq model,” arXiv preprint arXiv:2208.01448 , 2022

    2022 arXiv

  37. [46]

    Ul2: Unifying language learning paradigms,

    Y . Tay, M. Dehghani, V . Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, S. Shakeri, D. Bahri, T. Schuster et al., “Ul2: Unifying language learning paradigms,” arXiv preprint arXiv:2205.05131 , 2022

    2022 arXiv

  38. [47]

    Using transfer learning for code- related tasks,

    A. Mastropaolo, N. Cooper, D. N. Palacio, S. Scalabrino, D. Poshy- vanyk, R. Oliveto, and G. Bavota, “Using transfer learning for code- related tasks,” IEEE Transactions on Software Engineering , vol. 49, no. 4, pp. 1580–1598, 2022

    2022

  39. [48]

    Automatic source code summarization with graph attention networks,

    Y . Zhou, J. Shen, X. Zhang, W. Yang, T. Han, and T. Chen, “Automatic source code summarization with graph attention networks,” Journal of Systems and Software , vol. 188, p. 111257, 2022

    2022

  40. [49]

    Automating code-related tasks through transformers: The impact of pre-training,

    R. Tufano, L. Pascarella, and G. Bavota, “Automating code-related tasks through transformers: The impact of pre-training,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2425–2437

    2023

  41. [50]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  42. [51]

    Towards automatically addressing self-admitted technical debt: How far are we?

    A. Mastropaolo, M. Di Penta, and G. Bavota, “Towards automatically addressing self-admitted technical debt: How far are we?” in 2023 38th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2023, pp. 585–597

    2023

  43. [52]

    On the generalizability of deep learning-based code completion across programming language versions,

    M. Ciniselli, A. Martin-Lopez, and G. Bavota, “On the generalizability of deep learning-based code completion across programming language versions,” in Proceedings of the 32nd IEEE/ACM International Confer- ence on Program Comprehension , 2024, pp. 99–111

    2024

  44. [53]

    How the training procedure impacts the performance of deep learning-based vul- nerability patching,

    A. Mastropaolo, V . Nardone, G. Bavota, and M. Di Penta, “How the training procedure impacts the performance of deep learning-based vul- nerability patching,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , ser. EASE ’...

    2024

  45. [54]

    Early stopping-but when?

    L. Prechelt, “Early stopping-but when?” in Neural Networks: Tricks of the trade. Springer, 2002, pp. 55–69

    2002

  46. [55]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” in Break- throughs in statistics: Methodology and distribution . Springer, 1992, pp. 196–202

    1992

  47. [56]

    R. J. Grissom and J. J. Kim, Effect sizes for research: A broad practical approach. Lawrence Erlbaum Associates Publishers, 2005

    2005

  48. [57]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics , pp. 65–70, 1979

    1979

  49. [58]

    Recommendations for datasets for source code summarization,

    A. LeClair and C. McMillan, “Recommendations for datasets for source code summarization,” arXiv preprint arXiv:1904.02660 , 2019

    1904 arXiv

  50. [59]

    Replication Package of

    Anonymous, “Replication Package of ”Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?",” Nov. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.14060451

    2024 doi

  51. [60]

    Quanti- fying the carbon emissions of machine learning,

    A. Lacoste, A. Luccioni, V . Schmidt, and T. Dandres, “Quanti- fying the carbon emissions of machine learning,” arXiv preprint arXiv:1910.09700, 2019

    1910 arXiv

  52. [61]

    On the evaluation of neural code summarization,

    E. Shi, Y . Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” in Proceed- ings of the 44th international conference on software engineering , 2022, pp. 1597–1608

    2022

  53. [62]

    Code to comment

    D. Gros, H. Sezhiyan, P. Devanbu, and Z. Yu, “Code to comment" translation" data, metrics, baselining & evaluation,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 746–757

    2020

  54. [63]

    Findings of the 2019 conference on machine translation (wmt19)

    L. Barrault, O. Bojar, M. R. Costa-Jussa, C. Federmann, M. Fishel, Y . Graham, B. Haddow, M. Huck, P. Koehn, S. Malmasiet al., “Findings of the 2019 conference on machine translation (wmt19).” ACL, 2019

    2019

  55. [64]

    On the importance of building high-quality training datasets for neural code search,

    Z. Sun, L. Li, Y . Liu, X. Du, and L. Li, “On the importance of building high-quality training datasets for neural code search,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1609–1620

    2022

  56. [65]

    Commit message matters: Investigating impact and evolution of commit message quality,

    J. Li and I. Ahmed, “Commit message matters: Investigating impact and evolution of commit message quality,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 806–817

    2023

  57. [66]

    Data quality matters: A case study of obsolete comment detection,

    S. Xu, Y . Yao, F. Xu, T. Gu, J. Xu, and X. Ma, “Data quality matters: A case study of obsolete comment detection,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 781–793

    2023

  58. [67]

    Summarizing source code using a neural attention model,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in 54th Annual Meeting of the Association for Computational Linguistics 2016 . Association for Computational Linguistics, 2016, pp. 2073–2083

    2016

  59. [68]

    A transformer-based approach for source code summarization,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “A transformer-based approach for source code summarization,” arXiv preprint arXiv:2005.00653, 2020

    2005 arXiv

  60. [69]

    Source code summarization in the era of large language models,

    W. Sun, Y . Miao, Y . Li, H. Zhang, C. Fang, Y . Liu, G. Deng, Y . Liu, and Z. Chen, “Source code summarization in the era of large language models,” arXiv preprint arXiv:2407.07959 , 2024

    2024 arXiv

  61. [70]

    /* icomment: Bugs or bad comments?*,

    L. Tan, D. Yuan, G. Krishna, and Y . Zhou, “/* icomment: Bugs or bad comments?*,” in Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles , 2007, pp. 145–158

    2007

  62. [71]

    Automatic detection of outdated comments during code changes,

    Z. Liu, H. Chen, X. Chen, X. Luo, and F. Zhou, “Automatic detection of outdated comments during code changes,” in 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC) , vol. 1. IEEE, 2018, pp. 154–163

    2018

  63. [72]

    Deep code-comment understanding and assessment,

    D. Wang, Y . Guo, W. Dong, Z. Wang, H. Liu, and S. Li, “Deep code-comment understanding and assessment,” IEEE Access, vol. 7, pp. 174 200–174 209, 2019

    2019

  64. [73]

    Code comment inconsistency detection based on confidence learning,

    Z. Xu, S. Guo, Y . Wang, R. Chen, H. Li, X. Li, and H. Jiang, “Code comment inconsistency detection based on confidence learning,” IEEE Transactions on Software Engineering , 2024

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.