Pith. sign in

REVIEW 4 major objections 6 minor 79 references

LLMCup: Ranking-Enhanced Comment Updating with LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper tries to establish that an LLM-based update-then-rank pipeline—multiple prompt strategies plus a learned ranker—outperforms prior comment-updating systems, with exact-match Accuracy of 0.383 versus 0.257 (HebCup) and 0.177 (CUP).

desk verdict Solid applied LLM pipeline for comment updating with a novel update-then-rank framing, but the ranker's added value over random selection is small and not statistically established. read the letter →

arxiv 2507.08671 v1 pith:VM5NRR7A submitted 2025-07-11 cs.SE

Hua Ge , Juan Zhai , Minxue Pan , Fusen He , Ziyue Tan This is my paper
classification cs.SE
keywords automaticcommentupdatinglargelanguagemodelsrankingpromptstrategiescode-commentco-evolutionlearningtorankdataaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that large language models can update outdated code comments better than existing neural and heuristic systems, provided the best of several prompt-generated candidates is chosen by a learned ranker. It introduces LLMCup, an update-then-rank pipeline: one LLM writes several candidate comments under different k-shot prompts, and a ranking model named CupRank picks the final comment. On a Java comment-update dataset the pipeline reaches 0.383 exact-match Accuracy, up from 0.257 for HebCup and 0.177 for CUP, and it also improves BLEU-4, METEOR, F1, and semantic similarity. A user study adds that when LLMCup and human-written updates differ, raters preferred the LLM output on consistency, naturalness, and helpfulness. The claim matters because keeping comments in sync with code is a persistent maintenance cost, and current tools rarely achieve exact agreement.

What carries the argument

The load-bearing object is CupRank, a dual-encoder neural ranker. Each training sample is flattened into two token-level edit sequences—one for the code change and one for the comment change—where every token carries the CodeBERT token embedding, a one-hot operation type (equal, insert, delete), and an origin flag (old or new). Two cross-attention encoders exchange information between the code and comment streams, and a two-layer transformer, max pooling, and linear projection produce vectors whose cosine similarity is the ranking score. Training uses a temperature-scaled listwise softmax loss over one positive and several negatives per group. This design lets the model score how well a candidate comment change matches the actual code change, which is information a prompt-only system never sees.

What would settle it

Independently hold out one LLM family, train CupRank on the other six, and run LLMCup on a fresh test set with the held-out model as generator; if top-1 Accuracy is statistically indistinguishable from the 0.360 random-selection baseline on the same candidates, the claimed generalization of CupRank is not supported. The paper's own RQ2 setup already does this for GPT-4o, so the check is a replication with another unseen model.

Watch

Extended reading notes

Core claim

The central discovery is that the main bottleneck in LLM-based comment updating is not generation but selection. With the same four prompt strategies, the best single GPT-4o prompt achieves 0.370 Accuracy, random choice among the four candidates achieves 0.360, and CupRank lifts this to 0.383; the ranker also beats LLM self-ranking and a RankNet adaptation. CupRank is trained on an augmented dataset in which six LLMs generate candidate updates for training samples, any output differing from ground truth is labeled negative, and the model learns a listwise softmax ranking over positive/negative groups. Although GPT-4o was excluded from augmentation, CupRank still improves over GPT-4o's candidates, which the paper advances as evidence that the learned ranking transfers to a novel LLM. The paper further shows the accuracy gap is largest on code-indicative single-token updates and smallest on non-code-indicative multi-token updates, where all systems remain weak.

Load-bearing premise

CupRank's training signal is the set of LLM-generated candidate comments that differ from ground truth, and the paper assumes that learning to reject those six models' mistakes transfers to a seventh model's candidates at test time; if that transfer is weak, the ranker adds little over random choice.

Editorial extensions

If this is right

  • With a fixed base LLM, running four prompt strategies and selecting by CupRank beats every single strategy on Accuracy, Aed, and Red in the reported experiments.
  • The learned ranker transfers to a model excluded from training augmentation: CupRank ranked GPT-4o candidates better than random selection even though GPT-4o produced none of the training negatives.
  • Exact-match accuracy more than doubles relative to CUP (0.383 versus 0.177) and improves 49 percent over HebCup (0.257), while BLEU-4, METEOR, F1, and semantic similarity also rise.
  • Human raters preferred LLMCup comments over ground-truth comments on consistency, naturalness, and helpfulness, which the paper offers as evidence that automatic metrics understate the method's quality.
  • The largest remaining weakness is non-code-indicative multi-token updates, where LLMCup reaches only 0.095 accuracy, so the method's gains are concentrated in code-aligned updates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the accuracy gain of CupRank over random selection is modest (0.383 versus 0.360), the paper leaves implicit that most of the advantage comes from the LLM generator itself; a cheaper system could skip ranking when compute is tight.
  • The same update-then-rank recipe could apply to other generation conditions, such as temperature sampling, multiple base LLMs, or different prompt templates, since the ranker's input is just code/comment edit sequences, but the paper only tests k-shot diversity.
  • The user-study preference suggests comment-updating tools should be evaluated with human judgment; if that finding replicates, exact-match Accuracy as a headline metric may understate progress on semantically valid rewrites.
  • The framework is evaluated only on Java method-level comments, so transfer to other languages, non-Javadoc comment styles, or whole-file comment updates is an untested extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes LLMCup, an update-then-rank framework for automatic comment updating. Given old/new method code and an old comment, an LLM generates candidate updated comments under several k-shot prompt strategies; a learned ranker CupRank, trained on an augmented dataset of LLM-generated candidates labeled by exact match to ground truth, selects the top candidate. The authors evaluate on a refined CUP dataset with seven LLMs, comparing against CUP and HebCup and several ranking baselines. They report large accuracy gains over CUP/HebCup (up to 0.383 vs 0.177/0.257) and smaller gains from ranking over Random (0.383 vs 0.360), plus a user study.

Significance. Strengths: release code and augmented dataset; broad evaluation across 7 LLMs and 4 prompt strategies; clear separation of training and inference for CupRank; the update-then-rank idea is novel for this task. If the ranking gain were statistically established, the framework would be a solid step beyond pre-LLM comment updating baselines. The exact-match accuracy gains over CUP/HebCup are large and plausible. However, the central ranking-specific contribution is supported by a small, untested difference, and the paper's own edit-distance metrics show a regression; both need addressing before the claims can be accepted.

major comments (4)
  1. [§VI-C, Table III] The central "ranking-enhanced" claim rests on a 0.023 accuracy gap between CupRank (0.383) and Random (0.360), with no confidence intervals, significance tests, or oracle upper bound. This gap is comparable to or smaller than the ablation deltas in Table IV (2.1%–3.7%) and only slightly larger than the difference between GPT-4o at 5-shot (0.370) and Random. Random selection already outperforms Self-Rank, suggesting that candidate quality, not ranking, drives most of the gain. Please add paired significance tests over multiple runs, confidence intervals, and an oracle upper bound (e.g., best candidate per instance) to show that CupRank's selection is better than chance.
  2. [§IV-C and §V-E] The Data Augmentor generates CupRank's training negatives by retrieving k similar examples from the training corpus and prompting LLMs; the inference pipeline uses the same training corpus as the retrieval corpus when generating GPT-4o candidates. Consequently, the test-time candidate distribution may be very close to the augmentation distribution on which CupRank was fitted, and the small CupRank-vs-Random gain (Table III) could reflect retrieval/rank-corpus overlap rather than generalization to unseen LLMs. Please report (i) ranker performance stratified by retrieval similarity between test and retrieved demonstrations, (ii) results when test-time demonstrations are retrieved from a held-out corpus, and (iii) an analysis of whether CupRank's selection correlates with prompt-strategy identity or with code-change features.
  3. [§VI-A, Table I] LLMCup's Red (0.991) and Aed (3.084) are worse than both CUP (0.938, 2.997) and HebCup (0.858, 2.951). A Red near 1.0 means LLMCup's updated comment is approximately as distant from the ground truth as the original outdated comment, which conflicts with the prompt instruction "The fewer changes, the better" and with the claim that the framework reduces editing effort. The text calls these "slightly higher" and attributes them to verbosity, but the magnitude, especially on Red, deserves direct analysis and a caveat in the abstract/conclusion.
  4. [§VI-F] The human evaluation uses 7 raters and 100 cases where LLMCup differs from ground truth, with no inter-rater reliability measure and no significance test for the reported differences (Consistency 4.14 vs 3.81, Helpfulness 4.42 vs 4.37). The abstract's claim that LLMCup comments "sometimes surpass human-written updates" is therefore not statistically supported. Please report per-rater variance, agreement statistics, and confidence intervals or a paired test, and either soften the claim or restrict it to the sampled cases.
minor comments (6)
  1. [Abstract and §VIII] The abstract and §VIII report an Accuracy improvement of 116.9%, while Table I reports 116.4% for CUP; please reconcile these numbers.
  2. [§II-B] The sentence "Bo et al. [6] designed heuristic rules from code comment update cases and implemented HebCup" attributes reference [6] to the wrong authors; reference [6] is Lin et al., and the attribution should be corrected.
  3. [Throughout] There are numerous typos and inconsistent capitalizations (e.g., "HebCUP" vs "HebCup", "Cup" vs "CUP", "Ranknet" vs "RankNet", "prooposed", "uesed", "exacting", "nstance-specific" in Fig. 4); a copyedit pass is needed.
  4. [§VI-A, Finding 1] Finding 1 reports an average improvement of 82.95%, but the two displayed values in Table I imply 82.7% (or 82.95% with unrounded values); please clarify how the average is computed.
  5. [§V-B] The Self-Rank prompt template is presented as a single paragraph with literal '{' and '}' symbols; consider formatting it as a figure or listing so the placeholders and line breaks are unambiguous.
  6. [§IV-D and Fig. 6] Equation (1) defines an edit token as a triplet (t_i, a_i, b_i), but Fig. 6 does not visually distinguish operation type from origin flag; a short annotated example would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: CupRank is trained with externally supplied ground-truth labels and GPT-4o is excluded from augmentation, so the ranking gain is a genuine held-out result.

full rationale

The paper's load-bearing claim is that CupRank, trained on augmented samples labeled by exact match to ground-truth comments, selects better candidates at test time than Random selection (Table III: 0.383 vs 0.360 for GPT-4o). This is not circular by construction. Section IV-C defines the training signal externally: 'A sample is labeled positive if its updated comment matches the ground truth; otherwise, negative,' where the ground truth is the human-written new comment in the original dataset, not an output of the framework. Section V-E states that 'GPT-4o is excluded from the Data Augmentation stage,' so the headline generalization to the strongest test-time generator is an unseen-LLM evaluation. The dataset is split into training/validation/testing with duplicate removal (Section V-C), and the ranker is selected on validation loss, so the Table III comparison is an out-of-sample measurement rather than a fitted value renamed as a prediction. No load-bearing argument depends on a self-citation: the cited prompt-variability observation [11] and RankNet baseline [56] are external prior work, and no uniqueness theorem or ansatz is imported from the authors' own previous papers. The acknowledged limitations (Section VII-A) concern generalizability to other LLMs and prompt sets, which is a validity threat, not a definitional circularity. Even for the open-source LLMs that serve as both augmentors and test generators, the test-time candidates come from a held-out split and the positive supervision is independent of the framework's own outputs; distributional similarity between training negatives and test candidates is a standard training-condition match, not an equivalence of input and prediction. Accordingly, no step reduces to its own input, and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on supervised machine learning assumptions about ground-truth labels and representative negatives, not on physical postulates. Free parameters are limited to temperature and loss temperature, both chosen by hand; no new physical or metaphysical entities are introduced. The main burden is that the ranking model's training distribution is constructed by the same LLM families it later ranks, with GPT-4o as the only held-out generator.

free parameters (2)
  • LLM decoding temperature = 0.2
    Chosen after RQ5 evaluated accuracy on 1,000 randomly sampled test cases across LLMs, then fixed for all main experiments. This is test-set-based tuning and affects all reported results.
  • CupRank loss temperature lambda = 0.07
    Set by hand to control similarity distribution smoothness; the ablation without lambda changes accuracy by 2.3 percent, so the central ranking result depends on this choice.
assumptions (5)
  • domain assumption The CUP dataset's ground-truth comments are the correct target for every code change.
    Accuracy and ranking labels treat the ground-truth comment as the unique positive; if multiple updates are acceptable, exact-match ranking becomes a proxy for quality rather than truth. Invoked in Section V-F metrics and Section IV-C labeling.
  • domain assumption LLM-generated comments that differ from ground truth are valid negatives for training the ranker.
    Data augmentation discards samples where all generated comments equal ground truth and keeps all diverging outputs as negatives, even semantically equivalent ones. This shapes CupRank's training distribution in Section IV-C.
  • domain assumption CupRank's edit-token representation and CodeBERT embeddings are sufficient to compare code changes with comment changes.
    The dual-encoder ranker relies on CodeBERT tokenization and token-level diffs; if the embeddings miss semantic links, ranking scores are arbitrary. Introduced in Section IV-D.
  • domain assumption CupRank trained without GPT-4o data generalizes to GPT-4o candidates.
    Finding 2 asserts generalization to unseen LLMs, but the paper provides no mechanistic justification beyond the empirical result. Stated in Section VI-B.
  • standard math Cosine similarity and listwise softmax loss produce a valid ranking objective.
    Used in Equations 6 through 8 without proof; this is standard in the learning-to-rank literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMCup: Ranking-Enhanced Comment Updating with LLMs." pith.science (2026). https://pith.science/paper/VM5NRR7A

@misc{pith2026250708671,
  author       = {Pith},
  title        = {Pith review of: LLMCup: Ranking-Enhanced Comment Updating with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VM5NRR7A}},
  note         = {Machine review of arXiv:2507.08671}
}
read the original abstract

While comments are essential for enhancing code readability and maintainability in modern software projects, developers are often motivated to update code but not comments, leading to outdated or inconsistent documentation that hinders future understanding and maintenance. Recent approaches such as CUP and HebCup have attempted automatic comment updating using neural sequence-to-sequence models and heuristic rules, respectively. However, these methods can miss or misinterpret crucial information during comment updating, resulting in inaccurate comments, and they often struggle with complex update scenarios. Given these challenges, a promising direction lies in leveraging large language models (LLMs), which have shown impressive performance in software engineering tasks such as comment generation, code synthesis, and program repair. This suggests their strong potential to capture the logic behind code modifications - an ability that is crucial for the task of comment updating. Nevertheless, selecting an appropriate prompt strategy for an LLM on each update case remains challenging. To address this, we propose a novel comment updating framework, LLMCup, which first uses multiple prompt strategies to provide diverse candidate updated comments via an LLM, and then employs a ranking model, CupRank, to select the best candidate as final updated comment. Experimental results demonstrate the effectiveness of LLMCup, with improvements over state-of-the-art baselines (CUP and HebCup) by 49.0%-116.9% in Accuracy, 10.8%-20% in BLEU-4, 4.6% in METEOR, 0.9%-1.9% in F1, and 2.1%-3.4% in SentenceBert similarity. Furthermore, a user study shows that comments updated by LLMCup sometimes surpass human-written updates, highlighting the importance of incorporating human evaluation in comment quality assessment.

Figures

Figures reproduced from arXiv: 2507.08671 by the authors.

Figure 1
Figure 1. Motivating Example this, we explore their potential for comment updating by prompting LLMs with a carefully designed template ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of Framework Old Code New Code Old Comment (Augmented) Updated Comment Code Change Sequence Comment Change Sequence Model Trained Model Data Fla�ening Model Training Scoring Code Change & Comment Change Ranking Score Augmented Samples Augmented Dataset [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training of CupRank “Version”), and inserts special tokens like “<s>”, “</s>”, and “G”, meaning start, end and prefixed space. ˙ Second, to manage multiple components, we concatenate tokens of old/new code and comments respectively, resulting in two flattened sequences for code/comment change per sample. Third, to represent implicit modifications, a token-level diff [65] identifies edit operations and token origins.… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Structure and Example of the Prompt Template for Comment Updating [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Data Augmentation [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Convert Code/Comment Change to Change Sequence in the stage of Data Flattening [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Architecture of CupRank Model updating. Inputs are token embeddings (from CodeBERT [64]) of old/new code, old comment, and a updated comment, con￾catenated and fed into a feedforward network with two fully￾connected layers, max-pooling, and a sigmoid for ranking. C. Da…
Figure 8
Figure 8. Figure 8: Number of Correct Updates Across Update Types [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Impact of Temperature Parameter on Comment Update [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Average user ratings for LLMCup and ground truth [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 62 canonical work pages

  1. [1]

    Software documentation: The practi- tioners’ perspective,

    E. Aghajani, C. Nagy, M. Linares-V ´asquez, L. Moreno, G. Bavota, M. Lanza, and D. C. Shepherd, “Software documentation: The practi- tioners’ perspective,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) , 2020, pp. 590–601

    work page 2020

  2. [2]

    A human study of comprehension and code sum- marization,

    S. Stapleton, Y . Gambhir, A. LeClair, Z. Eberhart, W. Weimer, K. Leach, and Y . Huang, “A human study of comprehension and code sum- marization,” in ICPC ’20: 28th International Conference on Program Comprehension, Seoul, Republic of Korea, July 13-15, 2020 . ACM, 2020, pp. 2–13

    work page 2020

  3. [3]

    Understanding source code comments at large-scale,

    H. He, “Understanding source code comments at large-scale,” Pro- ceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019

    work page 2019

  4. [4]

    Using traceability links to recommend adaptive changes for documentation evolution,

    B. Dagenais and M. P. Robillard, “Using traceability links to recommend adaptive changes for documentation evolution,” IEEE Transactions on Software Engineering, vol. 40, no. 11, pp. 1126–1146, 2014

    work page 2014

  5. [5]

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,

    M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 . ACM, 2024, pp. 39:1– 39:13

    work page 2024

  6. [6]

    Automated comment update: How far are we?

    B. Lin, S. Wang, K. Liu, X. Mao, and T. F. Bissyand ´e, “Automated comment update: How far are we?” in 29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021, Madrid, Spain, May 20-21, 2021 . IEEE, 2021, pp. 36–46

    work page 2021

  7. [7]

    Automating just-in-time comment updating,

    Z. Liu, X. Xia, M. Yan, and S. Li, “Automating just-in-time comment updating,” in 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, September 21- 25, 2020. IEEE, 2020, pp. 585–597

    work page 2020

  8. [8]

    How good are gpt models at machine translation? a comprehensive evaluation,

    A. Hendy, M. G. Abdelrehim, A. Sharaf, V . Raunak, M. Gabr, H. Mat- sushita, Y . J. Kim, M. Afify, and H. H. Awadalla, “How good are gpt models at machine translation? a comprehensive evaluation,” ArXiv, vol. abs/2302.09210, 2023

    arXiv 2023

Show all 79 references
  1. [9]

    Lamner: code comment generation using character language model and named entity recognition,

    R. Sharma, F. Chen, and F. Fard, “Lamner: code comment generation using character language model and named entity recognition,” in Pro- ceedings of the 30th IEEE/ACM International Conference on Program Comprehension, ser. ICPC ’22. New York, NY , USA: Association for Computing...

    2022

  2. [10]

    Few-shot training llms for project-specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022

    2022

  3. [11]

    Source code summarization in the era of large language models,

    W. Sun, Y . Miao, Y . Li, H. Zhang, C. Fang, Y . Liu, G. Deng, Y . Liu, and Z. Chen, “Source code summarization in the era of large language models,” 2024

    2024

  4. [12]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y . Tay, N. M. Shazeer, V . Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, ...

    2022

  5. [13]

    Emergent abilities of large language models,

    J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yo- gatama, M. Bosma, D. Zhou, D. Metzler, E. H. hsin Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” ArXiv, vol. abs/2206.07682, 2022

    2022 arXiv

  6. [14]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” ArXiv, vol. abs/2310.06825, 2023

    arXiv

  7. [15]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” ArXiv, vol. abs/2305.01210, 2023

    2023 arXiv

  8. [16]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. ...

    2021 arXiv

  9. [17]

    Recent advances in text-to-sql: A sur- vey of what we have and what we expect,

    N. Deng, Y . Chen, and Y . Zhang, “Recent advances in text-to-sql: A sur- vey of what we have and what we expect,” ArXiv, vol. abs/2208.10099, 2022

    2022 arXiv

  10. [18]

    An extensive study on pre-trained models for program understanding and generation,

    Z. Zeng, H. Tan, H. Zhang, J. Li, Y . Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2022. New York, NY , USA:...

    2022

  11. [19]

    Clarifygpt: Empowering llm-based code generation with intention clarification,

    F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang, “Clarifygpt: Empowering llm-based code generation with intention clarification,” ArXiv, vol. abs/2310.10996, 2023

    2023 arXiv

  12. [20]

    Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,

    Y . Deng, C. Xia, C. Yang, S. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,” ArXiv, vol. abs/2304.02014, 2023

    2023 arXiv

  13. [21]

    Fuzz4all: Universal fuzzing with large language models,

    C. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) , pp. 1547– 1559, 2023

    2024

  14. [22]

    Learning loop invariants for program verification,

    X. Si, H. Dai, M. Raghothaman, M. Naik, and L. Song, “Learning loop invariants for program verification,” in Neural Information Processing Systems, 2018

    2018

  15. [23]

    Can chatgpt support software verification?

    C. Janssen, C. Richter, and H. Wehrheim, “Can chatgpt support software verification?” in Fundamental Approaches to Software Engineering , 2023

    2023

  16. [24]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” 2024, last accessed 9 September 2024. [Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  17. [25]

    The official meta llama 3 github site,

    Meta, “The official meta llama 3 github site,” last accessed 11 September

  18. [26]

    Code llama: Open foundation models for code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. P. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D’efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, a...

    2023 arXiv

  19. [27]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Y . Li, H. Gao, S. Ma et al., “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,” arXiv preprint arXiv:2406.11931 , 2024

    2024 arXiv

  20. [28]

    Gemma: Open weights llm from google deepmind,

    Google, “Gemma: Open weights llm from google deepmind,” last accessed 9 September 2024. [Online]. Available: https://github.com/ google-deepmind/gemma

    2024

  21. [29]

    Examining the evolution of code comments in postgresql,

    Z. M. Jiang and A. E. Hassan, “Examining the evolution of code comments in postgresql,” in Proceedings of the 2006 International Workshop on Mining Software Repositories , ser. MSR ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 179–180

    2006

  22. [30]

    Hatcup: Hybrid analysis and attention based just-in-time comment updating,

    H. Zhu, X. He, and L. Xu, “Hatcup: Hybrid analysis and attention based just-in-time comment updating,” 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC) , pp. 619–630, 2022

    2022

  23. [31]

    Deep just-in-time consistent com- ment update via source code changes,

    S. Guo, X. Xu, H. Li, and R. Chen, “Deep just-in-time consistent com- ment update via source code changes,” 2022 IEEE 13th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP), pp. 1–6, 2022

    2022

  24. [32]

    On the use of automated text summarization techniques for summarizing source code,

    S. Haiduc, J. Aponte, L. Moreno, and A. Marcus, “On the use of automated text summarization techniques for summarizing source code,” 2010 17th Working Conference on Reverse Engineering , pp. 35–44, 2010

    2010

  25. [33]

    Evaluating source code summarization techniques: Replication and expansion,

    B. P. Eddy, J. A. Robinson, N. A. Kraft, and J. C. Carver, “Evaluating source code summarization techniques: Replication and expansion,” 2013 21st International Conference on Program Comprehension (ICPC), pp. 13–22, 2013

    2013

  26. [34]

    Supporting program com- prehension with source code summarization,

    S. Haiduc, J. Aponte, and A. Marcus, “Supporting program com- prehension with source code summarization,” 2010 ACM/IEEE 32nd International Conference on Software Engineering, vol. 2, pp. 223–226, 2010

    2010

  27. [35]

    Towards automatically generating summary comments for java methods,

    G. Sridhara, E. Hill, D. Muppaneni, L. L. Pollock, and K. Vijay- Shanker, “Towards automatically generating summary comments for java methods,” Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering , 2010

    2010

  28. [36]

    Clocom: Mining existing source code for automatic comment generation,

    E. Wong, T. Liu, and L. Tan, “Clocom: Mining existing source code for automatic comment generation,” 2015 IEEE 22nd International Confer- ence on Software Analysis, Evolution, and Reengineering (SANER) , pp. 380–389, 2015

    2015

  29. [37]

    Extracting code segments and their descriptions from research articles,

    P. Chatterjee, B. Gause, H. Hedinger, and L. L. Pollock, “Extracting code segments and their descriptions from research articles,” 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) , pp. 91–101, 2017

    2017

  30. [38]

    Automatically generating commit messages from diffs using neural machine translation,

    S. Jiang, A. Armaly, and C. McMillan, “Automatically generating commit messages from diffs using neural machine translation,” 2017 32nd IEEE/ACM International Conference on Automated Software En- gineering (ASE), pp. 135–146, 2017

    2017

  31. [39]

    Autocomment: Mining question and answer sites for automatic comment generation,

    E. Wong, J. Yang, and L. Tan, “Autocomment: Mining question and answer sites for automatic comment generation,” 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 562–567, 2013

    2013

  32. [40]

    Codeattention: translating source code to comments by exploiting the code constructs,

    W. Zheng, H.-Y . Zhou, M. Li, and J. Wu, “Codeattention: translating source code to comments by exploiting the code constructs,” Frontiers of Computer Science , vol. 13, pp. 565 – 578, 2018

    2018

  33. [41]

    Summarizing source code using a neural attention model,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016

    2016

  34. [42]

    Automatically gen- erating natural language descriptions for object-related statement se- quences,

    X. Wang, L. L. Pollock, and K. Vijay-Shanker, “Automatically gen- erating natural language descriptions for object-related statement se- quences,” 2017 IEEE 24th International Conference on Software Anal- ysis, Evolution and Reengineering (SANER) , pp. 205–216, 2017

    2017

  35. [43]

    Cider: Consensus-based image description evaluation,

    R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 4566–4575, 2014

    2015

  36. [44]

    Im- proving automatic source code summarization via deep reinforcement learning,

    Y . Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu, “Im- proving automatic source code summarization via deep reinforcement learning,” 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) , pp. 397–407, 2018

    2018

  37. [45]

    On the properties of neural machine translation: Encoder–decoder approaches,

    K. Cho, B. van Merrienboer, D. Bahdanau, and Y . Bengio, “On the properties of neural machine translation: Encoder–decoder approaches,” in SSST@EMNLP, 2014

    2014

  38. [46]

    How do professional developers comprehend software?

    T. Roehm, R. Tiarks, R. Koschke, and W. Maalej, “How do professional developers comprehend software?” 2012 34th International Conference on Software Engineering (ICSE) , pp. 255–265, 2012

    2012

  39. [47]

    A convolutional attention network for extreme summarization of source code,

    M. Allamanis, H. Peng, and C. Sutton, “A convolutional attention network for extreme summarization of source code,” ArXiv, vol. abs/1602.03001, 2016

    2016 arXiv

  40. [48]

    /*icomment: bugs or bad comments?*/,

    L. Tan, D. Yuan, G. Krishna, and Y . Zhou, “/*icomment: bugs or bad comments?*/,” in Symposium on Operating Systems Principles , 2007

    2007

  41. [49]

    Automatic quality assessment of source code comments: The javadocminer,

    N. Khamis, R. Witte, and J. Rilling, “Automatic quality assessment of source code comments: The javadocminer,” in International Conference on Applications of Natural Language to Data Bases , 2010

    2010

  42. [50]

    @tcomment: Testing javadoc comments to detect comment-code inconsistencies,

    S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens, “@tcomment: Testing javadoc comments to detect comment-code inconsistencies,” 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 260–269, 2012

    2012

  43. [51]

    Detecting fragile comments,

    I. K. Ratol and M. P. Robillard, “Detecting fragile comments,” 2017 32nd IEEE/ACM International Conference on Automated Software En- gineering (ASE), pp. 112–122, 2017

    2017

  44. [52]

    Android studio,

    JetBrains, “Android studio,” last accessed 9 September 2024. [Online]. Available: https://github.com/JetBrains/android

    2024

  45. [53]

    Xinyu: An efficient llm-based system for commentary generation,

    Y . Wu, B. Tang, C. Xi, Y . Yu, P. Wang, Y . Liu, K. Kuang, H. Deng, Z. Li, F. Xiong et al., “Xinyu: An efficient llm-based system for commentary generation,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6003–6014

    2024

  46. [54]

    Detecting code comment inconsistencies using llm and program analysis,

    Y . Zhang, “Detecting code comment inconsistencies using llm and program analysis,” in Companion Proceedings of the 32nd ACM Inter- national Conference on the Foundations of Software Engineering , 2024, pp. 683–685

    2024

  47. [55]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

    2024

  48. [56]

    Learning to rank using gradient descent,

    C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to rank using gradient descent,” in Pro- ceedings of the 22nd International Conference on Machine Learning , ser. ICML ’05. ACM, 2005, pp. 89–96

    2005

  49. [57]

    Contrastive learning with hard negative samples,

    J. Robinson, C.-Y . Chuang, S. Sra, and S. Jegelka, “Contrastive learning with hard negative samples,” 2021

    2021

  50. [58]

    Approximate nearest neighbor negative contrastive learning for dense text retrieval,

    L. Xiong, C. Xiong, Y . Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” 2020

    2020

  51. [59]

    Sentence-bert: Sentence embeddings using siamese bert- networks,

    N. Reimers, “Sentence-bert: Sentence embeddings using siamese bert- networks,” arXiv preprint arXiv:1908.10084 , 2019

    1908 arXiv

  52. [60]

    Rethinking the role of demonstrations: What makes in-context learning work?

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” in EMNLP 2022, 2022, pp. 11 048–11 064

    2022

  53. [61]

    Retrieval-based prompt se- lection for code-related few-shot learning,

    N. Nashid, M. Sintaha, and A. Mesbah, “Retrieval-based prompt se- lection for code-related few-shot learning,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , 2023, pp. 2450–2462

    2023

  54. [62]

    Learning to retrieve prompts for in-context learning,

    O. Rubin, J. Herzig, and J. Berant, “Learning to retrieve prompts for in-context learning,” in Proc. of the 2022 Conf. of the North American Chapter of the Association for Computational Linguistics , 2022, pp. 2655–2671

    2022

  55. [63]

    Retrieval-based prompt se- lection for code-related few-shot learning,

    N. Nashid, M. Sintaha, and A. Mesbah, “Retrieval-based prompt se- lection for code-related few-shot learning,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2450–2462

    2023

  56. [64]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020 . Online: Association for Comput...

    2020

  57. [65]

    Available: https://docs.python.org/3/library/difflib.html

    Python Software Foundation, difflib — Helpers for computing deltas , 2023, [Online]. Available: https://docs.python.org/3/library/difflib.html

    2023

  58. [66]

    Instruction distillation makes large language models efficient zero-shot rankers,

    W. Sun, Z. Chen, X. Ma, L. Yan, S. Wang, P. Ren, Z. Chen, D. Yin, and Z. Ren, “Instruction distillation makes large language models efficient zero-shot rankers,” ArXiv, vol. abs/2311.01555, 2023

    2023 arXiv

  59. [67]

    Is chatgpt good at search? investigating large language models as re-ranking agent,

    W. Sun, L. Yan, X. Ma, P. Ren, D. Yin, and Z. Ren, “Is chatgpt good at search? investigating large language models as re-ranking agent,” ArXiv, vol. abs/2304.09542, 2023

    2023 arXiv

  60. [68]

    Adapting deep ranknet for personalized search,

    Y . Song, H. Wang, and X. He, “Adapting deep ranknet for personalized search,” in Proc. 7th ACM Int. Conf. on Web Search and Data Mining , 2014, pp. 83–92

    2014

  61. [69]

    Ensemble network for ranking im- ages based on visual appeal,

    S. Singh, V . Sanchez, and T. Guha, “Ensemble network for ranking im- ages based on visual appeal,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 4507–4511

    2020

  62. [70]

    A large-scale empirical study on code-comment inconsistencies,

    F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A large-scale empirical study on code-comment inconsistencies,” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 2019, pp. 53–64

    2019

  63. [71]

    Qwen2.5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang et al., “Qwen2.5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  64. [72]

    On the use of automated text summarization techniques for summarizing source code,

    S. Haiduc, J. Aponte, L. Moreno, and A. Marcus, “On the use of automated text summarization techniques for summarizing source code,” in 2010 17th Working conference on reverse engineering . IEEE, 2010, pp. 35–44

    2010

  65. [73]

    Deep code comment generation,

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in Proceedings of the 26th Conference on Program Compre- hension, ICPC 2018, Gothenburg, Sweden, May 27-28, 2018, F. Khomh, C. K. Roy, and J. Siegmund, Eds. ACM, 2018, pp. 200–210

    2018

  66. [74]

    Summarizing source code using a neural attention model,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers . The A...

    2016

  67. [75]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W. jing Zhu, “Bleu: a method for automatic evaluation of machine translation,” 2002, pp. 311–318

    2002

  68. [76]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization , Ann Arbor, MI, Jun. 2005, pp. 65–72

    2005

  69. [77]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  70. [78]

    A technique for the measurement of attitudes

    R. Likert, “A technique for the measurement of attitudes.” archieves of psychology, vol. 22 140, pp. 1–55, 1932

    1932

  71. [2024]

    Available: https://github.com/meta-llama/llama3

    [Online]. Available: https://github.com/meta-llama/llama3

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.