Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs backed by issue retrieval and verification cover 88% of human-written supplementary comments.

desk verdict Useful taxonomy and dataset, but the headline coverage numbers are built into the dataset construction and don't survive a verbatim-issue-sentence baseline. read the letter →

arxiv 2506.14649 v1 pith:442Y6A4E submitted 2025-06-17 cs.SE

Yanzhen Zou , Xianlin Zhao , Xinglu Pan , Bing Xie This is my paper · ORCID
classification cs.SE
keywords supplementarycodecommentsissuereportsretrieval-augmentedgenerationLLM-basedcommenthallucinationreductionverifiabletextMESIASentenceBERTverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes IsComment, a pipeline that generates supplementary code comments — design rationale, usage directives, performance implications, concepts, and functionality — by first retrieving the relevant sentences from a method's linked issue report, prompting an LLM to write candidate comment sentences, and then filtering out candidates that are not relevant to the code or not verifiable against the issue. The central claim is that this issue-grounded retrieval and verification turns LLMs from weak generators of supplementary comments (covering roughly a third of human-written supplementary sentences) into strong ones (covering 86–88% of them). The authors care because supplementary comments are exactly the comments developers want for understanding and reusing code, yet they cannot be derived from source code alone, which is why ordinary code summarization fails on them. If the approach works, developers could automatically obtain reliable, evidence-backed supplementary comments for methods that have associated issue discussions, including methods that currently have no manual comments at all.

What carries the argument

The load-bearing machinery is the IsComment pipeline, a three-phase loop in which a prompted LLM first acts as a retrieval agent that classifies sentences of the method's issue report into five supplementary-information types (Functionality, Rationale, Implication, Concept, Directive), then generates up to five typed candidate comment sentences from those retrieved sentences, and finally filters the candidates through two code-relevance checks (does the sentence mention a code element, or does it have a positive SIDE code-comment alignment score?) and one issue-verifiability check (SentenceBERT cosine similarity above 0.6 to at least one issue sentence). The five-type taxonomy makes retrieval selective rather than dumping the whole noisy issue into the prompt, and the verification thresholds are what the paper credits for reducing hallucinations.

What would settle it

Build a test set of supplementary comments that are genuinely informative but are not paraphrases of their linked issue (for example, comments whose rationale comes from design discussions absent from the issue), then run IsComment on it; if coverage falls back to the plain-prompt level, the headline gains are an artifact of the overlap filter rather than of retrieval and verification themselves.

Watch

Extended reading notes

Core claim

The paper's central claim is that supplementary code comments, which describe things like why a method exists, when it must be called, or what domain concept it relies on, can be generated reliably from issue reports through a retrieve–generate–verify pipeline. From a new dataset of 443 manually validated method–comment–issue triples mined from ten large open-source projects, the authors report that plain LLM prompting covers only 33.6–35.8% of human-written supplementary comment sentences, that feeding the whole issue report raises coverage to roughly two-thirds, and that their full IsComment pipeline raises it further to 72.2% for ChatGPT, 88.4% for GPT-4o, and 86.2% for DeepSeek-V3. The verification stage keeps only candidate sentences that mention a code element or score positively on the SIDE alignment metric and that have a SentenceBERT similarity above 0.6 to some issue sentence; this filtering removes hallucinated content while leaving coverage at 66.5–83.9%. The paper also reports that the generated comments carry higher MESIA supplementarity than baselines and that the method produced supplementary comments for 82.5% of issue-linked methods lacking manual comments in one evaluated project.

Load-bearing premise

The reported gains depend on the dataset rule that keeps only manual comment sentences with over 70% word overlap with a linked issue sentence; since IsComment is prompted from those very issue sentences and coverage is measured against the retained comments, relaxing that rule would likely shrink the headline improvements.

Editorial extensions

If this is right

  • For methods with a linked issue report, LLMs can jump from covering roughly a third to 72–88% of human-written supplementary comment sentences.
  • Filtering by code relevance and issue verifiability removes large amounts of hallucinated content while mostly preserving the coverage gains.
  • Typed outputs (Functionality, Rationale, Implication, Concept, Directive) give developers selectable comment kinds, each traceable to evidence in the issue report.
  • The approach transfers to methods with no manual comments: in one dataset project it produced supplementary comments for 82.5% of issue-linked, uncommented methods.
  • Because the pipeline is model-agnostic, the same issue retrieval and verification can be layered onto newer LLMs, with results reported on ChatGPT, GPT-4o, and DeepSeek-V3.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the pipeline treats the issue report as evidence, the same retrieve-by-type-and-verify design could be applied to pull requests, mailing lists, and requirement documents; the paper names this as future work but does not test it.
  • Editorial inference: the five-type taxonomy suggests a natural user-facing design — letting developers request only Rationale, only Directive, or other types — which could be evaluated in a controlled experiment on developer comprehension.
  • Editorial inference: the verification thresholds (0.6 SentenceBERT similarity, code-element mention, positive SIDE) could be tuned per project or per information type; a calibration study across projects would show whether one global threshold is best.
  • Editorial inference: since the approach works with three different LLMs, a plausible next test is whether it also reduces hallucination in higher-stakes settings such as API documentation for security-sensitive libraries, where unverifiable claims are costlier.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes IsComment, a retrieval-and-verification pipeline for generating 'supplementary' code comments (design rationale, usage directives, implications, concepts, functionality) from issue reports. It introduces a new dataset, Issuecom, built by linking method-comment pairs to issue reports via commit messages, filtering comments by the MESIA metric and by a 70% word-overlap rule against issue sentences, and then manually validating the remaining comments. The pipeline retrieves issue sentences with an LLM prompt, generates candidate comments from those sentences, and filters them using code-element mention, SIDE, and SentenceBERT similarity to issue sentences. Experiments compare IsComment with non-LLM and LLM baselines across ChatGPT, GPT-4o, and DeepSeek-V3, reporting large gains in coverage (e.g., 88.4% for GPT-4o) and reduced hallucinations, plus a small applicability study on Apache Derby.

Significance. The high-level idea of using issue reports as a source of supplementary information for code comments and applying verification to reduce hallucinations is timely and would be valuable if the evidence were sound. The five-category taxonomy of supplementary information (Functionality, Rationale, Implication, Concept, Directive) is useful, and the authors make code and data available for reproducibility. However, the main quantitative claims rest on an evaluation design that is circular: reference comments are selected to be near-paraphrases of issue sentences, generation is prompted with those same sentences, and the coverage metric measures similarity to those near-paraphrases. As it stands, the paper does not provide valid evidence for its headline improvements.

major comments (3)
  1. [Sec. 2.1.4, Table 3] The dataset construction rule that retains comment sentences with more than 70% word overlap with an issue sentence makes the reference comments near-duplicates of the issue text. Since the generation pipeline retrieves issue sentences from the linked issue and produces comments from them, the reported coverage increases (e.g., GPT-4o 88.4% in Table 3) are largely a consequence of this inclusion rule rather than of the method's ability to generate supplementary comments. An evaluation on comments that are not near-paraphrases of issue sentences, or at least an analysis of how coverage changes as the overlap threshold is relaxed, is needed.
  2. [Secs. 3.1-3.3 and 4.4.1] The same SentenceBERT similarity instrument and the same 0.6 threshold are used both to verify/retain generated comments against issue sentences (Eq. (2)) and to score coverage against manual comments. This double use of the same metric confounds the filtering step with the evaluation. A baseline that simply emits the retrieved issue sentence, or a lightly paraphrased version, as the generated comment is missing; without it, the reader cannot tell whether the 88.4% figure reflects genuine comment generation or the fact that the reference sentence is already close to the prompt content.
  3. [Sec. 4.4.2, Fig. 5] The RQ2 hallucination-reduction claim is circular. A generated sentence is labeled 'verifiable' if it has SentenceBERT similarity above 0.6 with some issue sentence, and the pipeline filters on exactly this criterion. Therefore, the high verifiability of IsComment's outputs is by construction and does not establish that the comments are factually correct or meaningful. The manual validation of 100 pairs in Sec. 3.3 is small and is not used to validate the coverage metric itself.
minor comments (5)
  1. [Table 3] Table 3 has duplicated column headers and appears to misalign rows: the GPT-4o Code Prompt row lists after-filter values (0.6 avg sentences, 19.5 sent len, 93 full-cover, 33 partial-cover, 28.4% coverage) identical to the ChatGPT Code Prompt row, which is confusing.
  2. [Sec. 5.1] There are typos in the running text: 'Funtionality' and 'Dirictive' should be 'Functionality' and 'Directive'.
  3. [Sec. 3.3] The metric name is written inconsistently as 'SentenceBert' in Sec. 3.3 and 'Sentence-BERT' elsewhere; please standardize.
  4. [Sec. 2.1.4] The 70% word-overlap rule does not specify whether overlap is computed on exact word tokens, stems, or with stopword removal; this should be stated precisely for reproducibility.
  5. [Sec. 3.3] The sentence 'The threshold 0.6 is a trade-off obtained through our manual attempt' is vague; a sensitivity analysis of the threshold, or at least a description of the manual procedure, would strengthen the work.

Circularity Check

3 steps flagged · score 8.0 of 10

Coverage gain is baked into the dataset: manual-comment targets are >70%-word-overlap paraphrases of the issue sentences given to IsComment, and the same 0.6 SentenceBERT threshold is used both to filter generated comments and to score coverage.

  1. self definitional [Section 2.1.4 (Issuecom construction) and Section 4.4.1 (Coverage Evaluation)]
    "we compare each comment sentence with the issue sentences, and keep those comment sentences having over 70% of words overlapping with one issue sentence. Through the overlapping analysis, we mine a considerable amount of candidate code comments whose code supplementary information is likely available and verifiable by the issue report. ... We regard a generated comment sentence successfully covers a manual comment sentence if their SentenceBert Similarity exceeds 0.6."

    The evaluation target (the manual comment sentence) is admitted into the dataset only when it is a near-paraphrase of an issue sentence, defined as having more than 70% word overlap. IsComment then receives that same issue sentence as its generation input (Sections 3.1-3.2), and coverage is scored by semantic similarity between the generated sentence and the manual target. High coverage is therefore a consequence of the dataset's inclusion rule: the model is prompted with the very sentence the reference is defined to paraphrase. No baseline that simply outputs the retrieved issue sentence verbatim is reported, so the 88.4% figure does not isolate the contribution of comment generation beyond repeating the issue content.

  2. fitted input called prediction [Section 3.3, Eq. (2) (Comment Sentence Verification) and Section 4.4.1 (Coverage Evaluation)]
    "For the generated code comments, we retain only those sentences that have at least one issue sentence with a similarity score above 0.6. The threshold 0.6 is a trade-off obtained through our manual attempt ... We regard a generated comment sentence successfully covers a manual comment sentence if their SentenceBert Similarity exceeds 0.6."

    The same SentenceBERT cosine similarity (Eq. 2) and the same threshold of 0.6 are used twice: first as IsComment's issue-verifiability filter, and then as the coverage evaluation metric against the manual comment. Because the manual comment was selected to share over 70% of its words with an issue sentence, any generated sentence that already passes the 0.6 filter against that issue sentence will very likely also pass 0.6 against the manual target. The reported post-filter coverage (83.9% for GPT-4o, 81.0% for DeepSeek-V3) is therefore partly an artifact of reusing the same similarity measurement in both the pipeline and the evaluation, not an independent confirmation that the generated comments match human annotations.

1 more flagged steps
  1. other [Section 4.4.2 (Verifiability Evaluation)]
    "To answer RQ2, we evaluate how well the generated comments are both code-relevant and issue-verifiable in our comment sentence verification phase mentioned earlier. ... To see how effective our evaluation criteria are, we apply them to the manual supplementary comments. 87.1% of the manual comment sentences are both code-relevant and issue-verifiable."

    RQ2's hallucination-reduction claim is evaluated using the same code-relevancy and issue-verifiability criteria that constitute IsComment's own filtering step. The issue-verifiability test compares the generated sentence against the same issue report that was placed in the LLM prompt, so 'verifiable' is defined relative to the prompt's own content. Applying these same criteria to the manual comments is not independent validation, because the manual comments were already filtered in Section 2.1.4 to have strong overlap with issue sentences. Thus the 87.1% pass rate and the reported reduction in 'not verifiable' comments restate the filter's inclusion rule rather than demonstrating an external reduction in hallucination.

full rationale

The central quantitative claim of the paper, that IsComment raises coverage from 35.8% to 88.4% for GPT-4o, is largely determined by construction. Section 2.1.4 keeps a manual comment sentence only when more than 70% of its words overlap with an issue sentence, so the evaluation reference is by definition a paraphrase of the source text that IsComment is given. Generation (Sections 3.1-3.2) starts from those issue sentences, and the coverage metric (Section 4.4.1) measures semantic similarity to the retained manual sentences. The issue-verifiability filter (Section 3.3) then applies a SentenceBERT threshold of 0.6 against the same issue sentences, and the evaluation uses the same 0.6 threshold. This creates a closed loop: the source and target are near-duplicates, the pipeline is rewarded for paraphrasing the source, and the missing verbatim-issue-sentence baseline would likely achieve high coverage without any retrieval or generation contribution. The paper does include manual validation and external baselines, and the approach may be practically useful, but those elements do not break the construction loop because manual validation occurs after the overlap filter and the baselines are not given the same near-duplicate source-target relation. The supplementary-nature evaluation via MESIA also relies on the authors' own prior work, but that is secondary; the dominant circularity is the overlap-filter-plus-same-threshold evaluation chain. For these reasons, the headline coverage improvement should be treated as substantially built into the dataset, not as an independent measure of generation quality.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the dataset construction thresholds (70% word overlap, MESIA >3, manual validation) and on the 0.6 SentenceBERT threshold used both for filtering and for measuring coverage. The approach assumes issue reports contain useful supplementary information and that sentence-embedding similarity is a valid proxy for comment quality. No new entities are introduced; the five comment categories come from an existing taxonomy.

free parameters (3)
  • SentenceBERT similarity threshold (0.6) for verifiability and coverage = 0.6
    Manually tuned 'through our manual attempt' (Sec 3.3); validated on only 100 sentence pairs with 86% accuracy. Used both to filter generated comments and to measure coverage, so its choice directly determines reported coverage rates.
  • Word-overlap threshold (70%) for dataset inclusion = 70%
    Comment sentences are kept in Issuecom only if more than 70% of their words overlap with an issue sentence (Sec 2.1.4). This criterion is not derived or justified and pre-embeds the answer in the evaluation.
  • MESIA threshold (>3) for supplementary comment filtering = >3
    Mined comments are filtered by the MESIA metric from the authors' prior work (Sec 2.1.2, ref [1]); the threshold follows that paper's recommendation and is reused both to build the dataset and to evaluate supplementarity in RQ3.
assumptions (6)
  • domain assumption The ten Apache projects are representative of real-world Java projects and their issue reports.
    Dataset is built from 10 projects (Sec 2.1.1); generalization to other ecosystems is assumed.
  • domain assumption Issue reports contain code supplementary information that is useful and verifiable for code comments.
    Central premise of the method (Sec 1, Sec 2). If issues do not contain such information, the pipeline has no signal.
  • domain assumption Manual validation by three developers is a reliable ground truth for supplementary comments.
    Sec 2.1.5; no inter-rater agreement is reported.
  • ad hoc to paper Sentence-BERT cosine similarity above 0.6 is a valid measure of comment coverage and verifiability.
    Threshold is manually chosen (Sec 3.3) and used as evaluation oracle (Sec 4.4.1); only 100 pairs validated.
  • ad hoc to paper The 70% word-overlap rule identifies comments whose information is verifiable by the issue report.
    Sec 2.1.4; this rule defines the dataset and is not independently validated.
  • standard math Standard sentence embedding formulas (Eq. 1-2) compute meaningful similarities.
    Sentence-BERT mean pooling and cosine similarity are standard (Sec 3.3, ref [42]).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation." pith.science (2026). https://pith.science/paper/442Y6A4E

@misc{pith2026250614649,
  author       = {Pith},
  title        = {Pith review of: Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/442Y6A4E}},
  note         = {Machine review of arXiv:2506.14649}
}
read the original abstract

Issue reports have been recognized to contain rich information for retrieval-augmented code comment generation. However, how to minimize hallucinations in the generated comments remains significant challenges. In this paper, we propose IsComment, an issue-based LLM retrieval and verification approach for generating method's design rationale, usage directives, and so on as supplementary code comments. We first identify five main types of code supplementary information that issue reports can provide through code-comment-issue analysis. Next, we retrieve issue sentences containing these types of supplementary information and generate candidate code comments. To reduce hallucinations, we filter out those candidate comments that are irrelevant to the code or unverifiable by the issue report, making the code comment generation results more reliable. Our experiments indicate that compared with LLMs, IsComment increases the coverage of manual supplementary comments from 33.6% to 72.2% for ChatGPT, from 35.8% to 88.4% for GPT-4o, and from 35.0% to 86.2% for DeepSeek-V3. Compared with existing work, IsComment can generate richer and more useful supplementary code comments for programming understanding, which is quantitatively evaluated through the MESIA metric on both methods with and without manual code comments.

Figures

Figures reproduced from arXiv: 2506.14649 by the authors.

Figure 1
Figure 1. The comments generated by the large language model ChatGPT using code and related issue report. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Proportion of different supplementary code com [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Framework of our approach. that it can achieve an accuracy of 86%, which is highly effective in evaluating the verifiability of the generated comments. 4 Experimental Setup This section describes research questions, dataset, comparing ap￾proaches, and evaluation setup. 4.1 Research Questions RQ1. How well can our approach improve LLMs in generating the supplementary code comments? There is a lot of work on automatic… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An example of code comments generated by differ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Evaluation results of the relevancy and verifiability of the comments generated by different approaches. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: MESIA value of the code comments generated by different approaches. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: An example of code comments generated by our approach. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Supplementary comment generation results for [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 49 canonical work pages

  1. [1]

    MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation

    Xinglu Pan, Chenxiao Liu, Yanzhen Zou, Tao Xie, and Bing Xie. MESIA: Under- standing and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation. arXiv e-prints, page arXiv:2403.17357, March 2024

    work page Pith review arXiv 2024

  2. [2]

    Practitioners’ expectations on automated code comment generation

    Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zimmer- mann. Practitioners’ expectations on automated code comment generation. In Proceedings of the 44th International Conference on Software Engineering , ICSE ’22, page 1693–1705, New York, NY, USA, 2022. Association for Computing Ma- chinery

    work page 2022

  3. [3]

    Comments should describe things that aren’t obvious from the code

    John Ousterhout. Comments should describe things that aren’t obvious from the code. In A Philosophy of Software Design , chapter 13, pages 95–116. 2018

    work page 2018

  4. [4]

    de Souza, Nicolas Anquetil, and Káthia M

    Sergio Cozzetti B. de Souza, Nicolas Anquetil, and Káthia M. de Oliveira. A study of the documentation essential to software maintenance. In Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting & Designing for Pervasive Information , SIGDOC ’05, page 68–75, New York, NY, USA, 2005. Association for Computing Machinery

    work page 2005

  5. [5]

    A doc2vec-based assessment of comments and its application to change- prone method analysis

    Hirohisa Aman, Sousuke Amasaki, Tomoyuki Yokogawa, and Minoru Kawa- hara. A doc2vec-based assessment of comments and its application to change- prone method analysis. In 2018 25th Asia-Pacific Software Engineering Conference (APSEC), pages 643–647, 2018

    work page 2018

  6. [6]

    https://camel.apache.org/

    work page

  7. [7]

    Sum- marizing source code using a neural attention model

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Sum- marizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073–2083, 2016

    work page 2016

  8. [8]

    Deep code comment generation

    Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC) , pages 200–20010. IEEE, 2018

    work page 2018

Show all 64 references
  1. [9]

    A transformer-based approach for source code summarization

    Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A transformer-based approach for source code summarization. arXiv preprint arXiv:2005.00653, 2020

    2005 arXiv

  2. [10]

    Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. Improving Automatic Source Code Summarization via Deep Reinforce- ment Learning, page 397–407. Association for Computing Machinery, New York, NY, USA, 2018

    2018

  3. [11]

    Retrieval- based neural source code summarization

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. Retrieval- based neural source code summarization. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) , pages 1385–1397, 2020

    2020

  4. [12]

    Association for Computing Machinery, New York, NY, USA, 2020

    Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin.Retrieve and Refine: Exemplar- Based Neural Comment Generation , page 349–360. Association for Computing Machinery, New York, NY, USA, 2020

    2020

  5. [13]

    Ast- transformer: Encoding abstract syntax trees efficiently for code summarization

    Ze Tang, Chuanyi Li, Jidong Ge, Xiaoyu Shen, Zheling Zhu, and Bin Luo. Ast- transformer: Encoding abstract syntax trees efficiently for code summarization. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1193–1195, 2021

    2021

  6. [14]

    Editsum: A retrieve- and-edit framework for source code summarization

    Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. Editsum: A retrieve- and-edit framework for source code summarization. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages 155– 166, 2021

    2021

  7. [15]

    A neural model for gener- ating natural language summaries of program subroutines

    Alexander LeClair, Siyuan Jiang, and Collin McMillan. A neural model for gener- ating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 795–806. IEEE, 2019

    2019

  8. [16]

    Improving retrieval-augmented code comment generation by retrieving for generation

    Hanzhen Lu and Zhongxin Liu. Improving retrieval-augmented code comment generation by retrieving for generation. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, Flagstaff, AZ, USA, October 6-11, 2024, pages 350–362. IEEE, 2024

    2024

  9. [17]

    Automatic code documentation generation using gpt-3

    Junaed Younus Khan and Gias Uddin. Automatic code documentation generation using gpt-3. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023. Association for Computing Machinery

    2023

  10. [18]

    Automatic Code Summarization via ChatGPT: How Far Are We? arXiv e-prints, page arXiv:2305.12865, May 2023

    Weisong Sun, Chunrong Fang, Yudu You, Yun Miao, Yi Liu, Yuekang Li, Gelei Deng, Shenghan Huang, Yuchen Chen, Quanjun Zhang, Hanwei Qian, Yang Liu, and Zhenyu Chen. Automatic Code Summarization via ChatGPT: How Far Are We? arXiv e-prints, page arXiv:2305.12865, May 2023

    2023 arXiv

  11. [19]

    Few-shot training llms for project- specific code-summarization

    Toufique Ahmed and Premkumar Devanbu. Few-shot training llms for project- specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , ASE ’22, New York, NY, USA,

  12. [20]

    Large language models are few-shot summa- rizers: Multi-intent comment generation via in-context learning

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. Large language models are few-shot summa- rizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on S...

    2024

  13. [21]

    Auto- matic semantic augmentation of language model prompts (for code summariza- tion)

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. Auto- matic semantic augmentation of language model prompts (for code summariza- tion). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY, USA, 2024. ...

    2024

  14. [22]

    Rethinking- based code summarization with chain of comments

    Liuwen Cao, Hongkui He, Hailin Huang, Jiexin Wang, and Yi Cai. Rethinking- based code summarization with chain of comments. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st Internation...

    2025

  15. [23]

    Nikolenko

    Vadim Lomshakov, Andrey Podivilov, Sergey Savin, Oleg Baryshnikov, Alena Lisevych, and Sergey I. Nikolenko. Proconsul: Project context for code summa- rization with llms. In Franck Dernoncourt, Daniel Preotiuc-Pietro, and Anastasia Shimorina, editors, Proceedings of the 2024 C...

    2024

  16. [24]

    Summarization is (Almost) Dead

    Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (Almost) Dead. arXiv e-prints, page arXiv:2309.09558, September 2023

    2023 arXiv

  17. [25]

    Beyond literal meaning: Uncover and explain implicit knowledge in code through wikipedia- based concept linking.IEEE Transactions on Software Engineering, 49(5):3226–3240, 2023

    Chong Wang, Xin Peng, Zhenchang Xing, and Xiujie Meng. Beyond literal meaning: Uncover and explain implicit knowledge in code through wikipedia- based concept linking.IEEE Transactions on Software Engineering, 49(5):3226–3240, 2023

    2023

  18. [26]

    Arya, Wenting Wang, Jin L

    Deeksha M. Arya, Wenting Wang, Jin L. C. Guo, and Jinghui Cheng. Analysis and detection of information types of open source software issue discussions. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , pages 454–464, 2019

    2019

  19. [27]

    Understanding code fragments with issue reports

    Zexuan Li and Hao Zhong. Understanding code fragments with issue reports. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1312–1316, 2021

    2021

  20. [28]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv e-prints, page arXiv:2312.10997, December 2023

    2023 arXiv

  21. [29]

    M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das

    S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. arXiv e-prints, page arXiv:2401.01313, January 2024

    2024 arXiv

  22. [31]

    A large-scale empirical study on code-comment inconsistencies

    Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. A large-scale empirical study on code-comment inconsistencies. In 2019 IEEE/ACM 27th Inter- national Conference on Program Comprehension (ICPC) , pages 53–64, 2019

    2019

  23. [32]

    Internetware 2025, June 20-22, 2025, Trondheim, Norway Yanzhen Zou, Xianlin Zhao, Xinglu Pan, and Bing Xie

    https://eclipse.dev/jgit/. Internetware 2025, June 20-22, 2025, Trondheim, Norway Yanzhen Zou, Xianlin Zhao, Xinglu Pan, and Bing Xie

    2025

  24. [33]

    Robillard

    Walid Maalej and Martin P. Robillard. Patterns of knowledge in api reference documentation. IEEE Transactions on Software Engineering, 39(9):1264–1282, 2013

    2013

  25. [34]

    Why my code summarization model does not work: Code comment improvement with category prediction

    Qiuyuan Chen, Xin Xia, Han Hu, David Lo, and Shanping Li. Why my code summarization model does not work: Code comment improvement with category prediction. ACM Trans. Softw. Eng. Methodol., 30(2), feb 2021

    2021

  26. [35]

    Developer- Intent Driven Code Comment Generation

    Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. Developer- Intent Driven Code Comment Generation. arXiv e-prints, page arXiv:2302.07055, February 2023

    2023 arXiv

  27. [36]

    Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning. arXiv e-prints, page arXiv:2304.11384, April 2023

    2023 arXiv

  28. [37]

    https://en.wikipedia.org/wiki/Tf%E2%80%93idf

  29. [38]

    https://huggingface.co/facebook/dpr-ctx_encoder-single-nq-base

  30. [39]

    https://huggingface.co/flax-sentence-embeddings/st-codesearch-distilroberta- base

  31. [40]

    https://github.com/Iscomment/IsComment/blob/main/Appendix.pdf

  32. [41]

    Evaluating code summarization techniques: A new metric and an empiri- cal characterization

    Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. Evaluating code summarization techniques: A new metric and an empiri- cal characterization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  33. [42]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv e-prints, page arXiv:1908.10084, August 2019

    1908 arXiv

  34. [43]

    Semantic similarity metrics for evaluating source code summarization

    Sakib Haque, Zachary Eberhart, Aakash Bansal, and Collin McMillan. Semantic similarity metrics for evaluating source code summarization. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension , ICPC ’22, page 36–47, New York, NY, USA, 2022. Assoc...

    2022

  35. [44]

    https://github.com/xing-hu/TL-CodeSum

  36. [45]

    CodeBERT: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 ,...

    2020

  37. [46]

    https://github.com/github/CodeSearchNet

  38. [47]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 8696–8708, On...

    2021

  39. [48]

    https://platform.openai.com/docs/models/gpt-3-5-turbo

  40. [49]

    https://platform.openai.com/docs/models/gpt-4o

  41. [50]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024

    2024

  42. [51]

    On the evaluation of neural code summarization

    Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen, Shi Han, Hongyu Zhang, Dong- mei Zhang, and Hongbin Sun. On the evaluation of neural code summarization. In Proceedings of the 44th International Conference on Software Engineering , pages 1597–1608, 2022

    2022

  43. [52]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318, 2002

    2002

  44. [53]

    Can large language models serve as evaluators for code summarization? arXiv preprint arXiv:2412.01333, 2024

    Yang Wu, Yao Wan, Zhaoyang Chu, Wenting Zhao, Ye Liu, Hongyu Zhang, Xuanhua Shi, and Philip S Yu. Can large language models serve as evaluators for code summarization? arXiv preprint arXiv:2412.01333, 2024

    2024 arXiv

  45. [54]

    Human-like Summarization Evaluation with ChatGPT

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like Summarization Evaluation with ChatGPT. arXiv e-prints, page arXiv:2304.02554, April 2023

    2023 arXiv

  46. [55]

    Source Code Summarization in the Era of Large Language Models

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. Source Code Summarization in the Era of Large Language Models. arXiv e-prints, page arXiv:2407.07959, July 2024

    2024 arXiv

  47. [56]

    Murphy, and Gabriel Murray

    Sarah Rastkar, Gail C. Murphy, and Gabriel Murray. Automatic summarization of bug reports. IEEE Transactions on Software Engineering , 40(4):366–380, 2014

    2014

  48. [57]

    Saskia Gilmer, Avinash Bhat, Shuvam Shah, Kevin Cherry, Jinghui Cheng, and Jin L. C. Guo. SUMMIT: Scaffolding OSS Issue Discussion Through Summarization. arXiv e-prints, page arXiv:2308.02780, August 2023

    2023 arXiv

  49. [58]

    Kumar, P

    A. Kumar, P. Das, and P. Pratim Chakrabarti. Summarize me: The future of issue thread interpretation. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pages 341–345, Los Alamitos, CA, USA, oct

    2023

  50. [59]

    IEEE Computer Society

  51. [60]

    Mining source code descriptions from developer commu- nications

    Sebastiano Panichella, Jairo Aponte, Massimiliano Di Penta, Andrian Marcus, and Gerardo Canfora. Mining source code descriptions from developer commu- nications. In 2012 20th IEEE International Conference on Program Comprehension (ICPC), pages 63–72, 2012

    2012

  52. [61]

    LLatrieval: LLM-verified retrieval for verifiable generation

    Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. LLatrieval: LLM-verified retrieval for verifiable generation. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Ass...

    2024

  53. [62]

    Towards verifiable text generation with evolving memory and self-reflection

    Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, and Dawei Yin. Towards verifiable text generation with evolving memory and self-reflection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on ...

    2024

  54. [63]

    Towards Verifiable Text Generation with Symbolic References

    Lucas Torroba Hennigen, Shannon Shen, Aniruddha Nrusimha, Bernhard Gapp, David Sontag, and Yoon Kim. Towards Verifiable Text Generation with Symbolic References. arXiv e-prints, page arXiv:2311.09188, November 2023

    2023 arXiv

  55. [64]

    Verifiable generation with subsentence-level fine- grained citations

    Shuyang Cao and Lu Wang. Verifiable generation with subsentence-level fine- grained citations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2...

    2024

  56. [2023]

    Association for Computing Machinery

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.