Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Evaluating Generated Commit Messages with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4 can grade commit messages almost as well as human developers

desk verdict First real test of LLM judges for commit messages, with a usable benchmark and honest limitations—needs a holdout and a direct self-preference check, but worth peer review. read the letter →

arxiv 2507.10906 v1 pith:2ERRNPV6 submitted 2025-07-15 cs.SE

Qunhong Zeng , Yuxia Zhang , Zexiong Ma , Bo Jiang , Ningyuan Sun , Klaas-Jan Stol , Xingyu Mou , Hui Liu This is my paper · ORCID
classification cs.SE
keywords commitmessagegenerationqualityLLMevaluationchain-of-thoughtpromptingfew-shotlearningautomaticmetricshumanSpearmancorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that LLM-based evaluators can substitute for human raters when judging whether an automatically generated commit message is any good. Today's standard metrics, BLEU, ROUGE-L, and METEOR, compare a generated message against a single developer-written reference, but the same code change can be described in many equally valid ways, so those scores mislead. The authors show that prompting GPT-4 to reason step-by-step, with examples, and to score the What and Why dimensions together yields correlations with human ratings of 0.65 and 0.78, falling inside the range of pairwise human-rater agreement and far above the best traditional metric. If this holds, commit-message evaluation can shift from days of human annotation to an automated, cheap, LLM-based pipeline. The paper is explicitly a preliminary study and flags that LLM scores vary from run to run.

What carries the argument

The mechanism that carries the argument is the prompt configuration used with the LLM, not a new model. In a single prompt, the model acts as an experienced software engineer, receives definitions of the What and Why dimensions with a 5-point rubric, is asked to reason step-by-step (Chain-of-Thought), is shown exemplary high- and low-quality commit messages (few-shot demonstrations), and gives scores for both dimensions at once (unified evaluation); three repeated runs are averaged to obtain the final score. The comparison machinery is Spearman and Kendall correlation of the LLM's averaged scores against the averaged human ratings.

What would settle it

Run the same GPT-4 CoT plus few-shot prompt on a fresh sample of commits, have each message rated by a panel of ten or more independent developers, and compare the LLM's ranks with the panel's consensus. If the Spearman correlation on this stronger ground truth falls to METEOR's level, around 0.4 on What and 0.2 on Why, or if the six-volunteer ratings disagree substantially with the larger panel, the near-human-level claim would not transfer.

Watch

Extended reading notes

Core claim

The paper's central claim is that when a Large Language Model is prompted with chain-of-thought evaluation steps, a few high- and low-quality examples, and a single prompt that scores both content dimensions at once, it can grade commit messages nearly as well as experienced developers. In the authors' data, this configuration, realized with GPT-4, reached Spearman correlations of 0.65 for What and 0.78 for Why, versus at most 0.42 and 0.20 for the best reference-based metric, METEOR. Because the pairwise correlations among the six human raters ranged from about 0.61 to 0.78, the authors interpret the LLM's scores as human-level. They also report that the advantage holds across four models tested and that the LLM evaluator shows acceptable reproducibility, robustness, and fairness, though scores vary and should be treated as approximations rather than precise measurements.

Load-bearing premise

The load-bearing premise is that the averaged 5-point ratings of six experienced volunteers, who agree pairwise at 0.61 to 0.78, are a trustworthy measure of commit message quality; the paper validates those raters only against each other, not against any external standard, and only on Python and Java commits.

Editorial extensions

If this is right

  • If the central claim is right, commit-message evaluation can replace or complement days of human annotation with an LLM pipeline that runs in under an hour and costs roughly $0.08 per example.
  • The LLM evaluator's Why-dimension correlation of 0.78 is especially consequential, because no reference-based or semantic-similarity metric comes close to human judgment on the Why dimension.
  • The one-to-many mapping between a code change and valid commit messages is identified as the fundamental reason reference metrics fail, which explains why direct content judgment outperforms similarity to a single reference.
  • Because LLM scores are slightly unstable, the authors state that these evaluators should be used to approximate human judgment rather than to make precise comparisons between two models that score within about a point of each other.
  • The same CoT plus few-shot integrated prompt template can be reused across GPT-4, Llama-3.3, Qwen2.5, and QwQ, with all tested models showing moderate to strong correlation with human ratings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the evaluator could be reused as a reward signal for training or fine-tuning commit-message generators, not just as a post-hoc metric.
  • A natural testable extension is to check whether the same prompt ordering transfers to other languages, since the study samples only Python and Java commits; JavaScript, TypeScript, or C++ datasets would settle that question.
  • The paper's stability data suggest a practical guardrail: report averaged scores from multiple runs and only treat differences larger than about one point on the 5-point scale as meaningful.
  • Because the failure mode of reference metrics is high expression diversity, LLM-based evaluation may also benefit other code-related summarization tasks, such as release-note or API-documentation generation, where one input admits many valid outputs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether large language models (LLMs) can serve as automatic evaluators of generated commit message quality. The authors construct a benchmark of 400 commits (200 Python, 200 Java) with 800 commit messages: 400 human-written and 400 generated by GPT-4, Llama-3.3-70B, and Qwen-2.5-14B. Six experienced developers rate all messages on What and Why dimensions using a 5-point Likert scale. In RQ1, the authors test four LLMs with various prompting strategies (zero-shot, chain-of-thought, few-shot, unified vs. separate scoring) on the 400 human-written messages and report that GPT-4 with CoT+few-shot and unified evaluation achieves Spearman correlations of 0.65 (What) and 0.78 (Why) with averaged human ratings, which they claim closely match human inter-rater correlations (0.61-0.78). In RQ2, this configuration is applied to the 400 generated messages and compared with traditional metrics (BLEU, ROUGE-L, METEOR, CIDEr) and semantic metrics (BERTScore, SBERT); GPT-4 substantially outperforms all of them (e.g., METEOR's best Spearman is 0.42 for What and 0.20 for Why). The authors also analyze reproducibility across temperatures, robustness to paraphrased variants, and fairness between human- and LLM-generated messages. They conclude that LLM-based evaluators can approach human-level proficiency while being far cheaper and faster than human evaluation.

Significance. If the central claim holds, this is a useful and timely contribution to a practical problem: commit message evaluation is currently dominated by expensive human annotation or unreliable reference-based metrics. The paper is among the first to systematically investigate LLM-based evaluation specifically for commit messages, and it includes a credible human annotation effort (six experienced developers, pilot calibration, shuffled evaluation) and a broad comparison across models and prompt strategies. Strong points include the reproducibility analysis (temperature sweep), the robustness check with manually verified paraphrases, and the thematic analysis explaining why reference-based metrics fail. The replication package and the use of multiple open-source as well as proprietary models are also positive. However, the headline 'near human-level' and 'significantly outperforms' claims rest on methodological choices that currently leave the results uncertain: in-sample prompt selection, a possible self-preference confound in RQ2, and a lack of statistical inference for the reported differences.

major comments (4)
  1. [Section 4 (RQ1), Table 2] The optimal prompt configuration is selected on the same 400 human-written messages that are then used to report the headline correlations. The authors test 4 models, multiple prompt strategies, and two scoring modes, then select GPT-4 with CoT+Few-shot and unified evaluation as the best. The reported Spearman values of 0.65 and 0.78 are therefore in-sample estimates, not out-of-sample predictions, and are likely optimistically biased by the selection procedure. The claim that this configuration achieves 'human-level proficiency' is not supported by an unbiased estimate. Please use a held-out validation set (or cross-validation) for prompt selection and report the test-set correlations separately.
  2. [Section 5.2 (Table 3) and Section 6.3] In RQ2, GPT-4 evaluates 400 generated messages, and according to Section 3.1.2, approximately one-third of those messages were generated by GPT-4 itself. The fairness analysis in Section 6.3 does not control for this self-evaluation confound: it only compares human-authored messages with GPT-4-paraphrased variants of those same human-authored messages (Section 6.3.1), not with the actual GPT-4-generated messages used in RQ2. A systematic self-preference or style-matching effect would inflate the GPT-4 correlations in Table 3 (0.65/0.78) and the gap over METEOR, and the current fairness test would not detect it. Please report the LLM-human correlations separately for messages generated by each source (GPT-4, Llama-3.3, Qwen-2.5), or construct a fairness test on matched human/LLM-generated pairs drawn from the actual RQ2 corpus.
  3. [Section 3.2 and Figure 1] The human ratings are treated as the ground truth for message quality, but the six pairwise inter-rater Spearman correlations range from 0.61 to 0.78, which by common conventions is moderate rather than 'high' agreement. The paper's conclusion that GPT-4 is 'near human-level' is drawn by comparing GPT-4's correlation with the averaged six-rater scores to these pairwise inter-rater correlations; this comparison is not apples-to-apples, because an average of six raters is more reliable than any single rater. Moreover, the human ratings are not validated against any external standard, so the LLM-human correlations could reflect agreement with this particular group's preferences rather than objective message quality. Please provide additional reliability evidence (e.g., intra-rater consistency, item-level analyses, or a small expert-panel validation) and either benchmark the LLM against individual raters or use a reliability-corrected criterion when claiming parity with human evaluators.
  4. [Section 5.2 (Table 3)] The claim that the LLM-based evaluator 'significantly outperforms' automatic metrics is not supported by any significance test or confidence interval. The correlations are computed on the same 400 messages, so a dependent-correlation test is needed (e.g., Steiger's test or bootstrap confidence intervals for the difference between GPT-4's and METEOR's Spearman correlations). Without such inference, the reader cannot assess whether the large point differences (0.65 vs. 0.42 for What; 0.78 vs. 0.20 for Why) are statistically robust or partly due to sampling noise, especially given the moderate inter-rater reliability of the ground truth.
minor comments (6)
  1. [Section 3.1.2] The text states that 'each model generating one-third of the commit messages,' but 400 is not divisible by 3. Please clarify how the 400 commits were allocated among the three LLMs.
  2. [Section 5.1] The sentence 'Our evaluation involved 400 generated commit messages (see Sec. 3.2)' should refer to Section 3.1.2, where the generation of messages is described, not Section 3.2, which describes human evaluation.
  3. [Figure 1 caption] The caption states that 'the upper triangle (blue) represents What, the lower triangle (green) represents Why,' but the layout of the printed matrix is ambiguous. Please verify that the triangle colors and orientation match the figure's actual presentation.
  4. [Table 2] For GPT-4, the rows 'CoT' and 'Zero-shot+CoT' show identical unified-evaluation correlations (What 0.65, Why 0.78). If these are two distinct strategies, explain why the values coincide; if they are redundant, consider removing one row to avoid confusion.
  5. [Abstract and Section 7.2] The abstract claims 'near human-level evaluation proficiency,' while Section 7.2 advises that LLM metrics 'should be viewed as approximations of human judgment' and 'are unsuitable for precise comparative analyses.' Please align these statements so the conclusion matches the reported limitations.
  6. [References] Reference [25] attributes ROUGE to Lin and Och (2004), but the canonical ROUGE reference is Lin (2004), 'ROUGE: A Package for Automatic Evaluation of Summaries.' Please check and correct this citation.

Circularity Check

0 steps flagged · score 2.0 of 10

No definitional circularity: the central claim is an empirical correlation against independently collected human ratings, not a derivation from fitted inputs; the self-citation used for the What/Why rubric is external prior work, and the GPT-4 self-preference issue is a confound rather than a circular reduction.

full rationale

The paper makes no first-principles or predictive derivation claim; its central evidence is Spearman/Kendall correlations between LLM ratings and ratings from six human volunteers on 400 commit messages. Those human ratings are an external empirical benchmark, so the headline correlations (0.65 What, 0.78 Why) are not equal to the LLM's inputs by construction. The What/Why rubric is adopted from Tian et al. [41], a prior study sharing three authors with this paper, but that prior work is an empirical qualitative study and the rubric is used as the evaluation instrument rather than as a result derived from it; the paper does not invoke a uniqueness theorem or an ansatz smuggled in via self-citation. The main circularity-adjacent concern is that GPT-4 generated roughly one-third of the messages evaluated in RQ2 and also served as evaluator, so self-preference could inflate correlations on the RQ2 corpus; the fairness test in Sec. 6.3 uses only 25-40 matched pairs of human-authored versus GPT-4-paraphrased messages and does not isolate the actual GPT-4-generated messages used in RQ2. This is a validity threat (partially acknowledged in Sec. 7.2's note on limited fairness samples), not a definitional equivalence or a fitted parameter renamed as a prediction. A separate methodological concern is that the optimal prompt configuration was selected on the same 400 human-written messages from which the 0.65/0.78 correlations are reported (Sec. 4.2: 'Based on these results, we selected GPT-4 with the CoT+Few-shot strategy using unified evaluation as our optimal configuration, achieving Spearman correlations of 0.65 and 0.78'), which may inflate the headline number through selection on the test set; this is overfitting/selection bias and a correctness risk, but it is not circular reasoning under the strict reduction standard. No equation is defined in terms of the outcome, and no fitted parameter is renamed as a prediction, so the paper does not exhibit circular reasoning in the sense required by the analysis.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

This empirical study introduces no theoretical entities or fitted constants in the physics sense. The entries above are analyst choices and domain assumptions that shape the benchmark and the headline correlations; if any were changed materially, the reported numbers could shift.

free parameters (5)
  • LLM temperature settings = 0.25 (GPT-4), 0 (open-source models)
    Chosen based on empirical stability evaluation in Sec 6.1; temperature affects score variability and, indirectly, the reported correlations in RQ1-RQ2.
  • Proportion of deliberately poor-quality generated messages = 20%
    Introduced in Sec 3.1.2 to create a wide quality range in the benchmark; the instruction is not verified against actual scores, so the intended quality distribution is assumed.
  • Thresholds for discrepant example selection in thematic analysis = top/bottom 30%
    Used in Sec 5.2 to define high human / low METEOR and converse cases; the authors acknowledge the threshold trades sample size against false positives.
  • Reference quality threshold for high-quality subset = What >= 2.5
    Post hoc filter in Sec 5.2 Category 1, used to show that METEOR correlation improves to 0.53; not pre-specified.
  • Number of repeated LLM evaluations averaged = 3
    Sec 3.3; final LLM scores are the average of three runs, which reduces but does not eliminate variability.
assumptions (5)
  • domain assumption Commit message quality can be decomposed into What and Why dimensions
    Adopted from Tian et al. [41] and Li et al. [24]; the human evaluation and all LLM prompts measure only these two content dimensions and exclude conciseness and expressiveness (Sec 2.3).
  • domain assumption Averaged human ratings from six volunteers are the ground truth for commit message quality
    All correlations in RQ1 and RQ2 are computed against these averages; inter-rater agreement above 0.6 is treated as sufficient reliability without external validation.
  • domain assumption Spearman and Kendall correlations are sufficient measures of evaluator quality
    A high rank correlation does not imply agreement in absolute scores; systematic over- or under-scoring would not be detected by these coefficients.
  • domain assumption The sampled 400 commits from CommitBench_long are representative
    Only Python and Java commits are included; the paper does not test other languages, project sizes, or commit types, so generalizability is assumed.
  • domain assumption Instructing generators to produce poor-quality messages for 20% of samples yields the intended score distribution
    Sec 3.1.2; the paper never verifies that these instructions actually produced low human scores, so the benchmark's quality spread is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Generated Commit Messages with Large Language Models." pith.science (2026). https://pith.science/paper/2ERRNPV6

@misc{pith2026250710906,
  author       = {Pith},
  title        = {Pith review of: Evaluating Generated Commit Messages with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ERRNPV6}},
  note         = {Machine review of arXiv:2507.10906}
}
read the original abstract

Commit messages are essential in software development as they serve to document and explain code changes. Yet, their quality often falls short in practice, with studies showing significant proportions of empty or inadequate messages. While automated commit message generation has advanced significantly, particularly with Large Language Models (LLMs), the evaluation of generated messages remains challenging. Traditional reference-based automatic metrics like BLEU, ROUGE-L, and METEOR have notable limitations in assessing commit message quality, as they assume a one-to-one mapping between code changes and commit messages, leading researchers to rely on resource-intensive human evaluation. This study investigates the potential of LLMs as automated evaluators for commit message quality. Through systematic experimentation with various prompt strategies and state-of-the-art LLMs, we demonstrate that LLMs combining Chain-of-Thought reasoning with few-shot demonstrations achieve near human-level evaluation proficiency. Our LLM-based evaluator significantly outperforms traditional metrics while maintaining acceptable reproducibility, robustness, and fairness levels despite some inherent variability. This work conducts a comprehensive preliminary study on using LLMs for commit message evaluation, offering a scalable alternative to human assessment while maintaining high-quality evaluation.

Figures

Figures reproduced from arXiv: 2507.10906 by the authors.

Figure 1
Figure 1. Spearman Correlation Coefficient of Volunteers, [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

    cs.SE 2026-08 conditional novelty 6.0 of 10

    LLM judges match human expert opinions on binary reverse engineering outputs better than text-similarity metrics, and a learned router improves both accuracy and cost.

Reference graph

Works this paper leans on

60 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Anonymous Author(s). 2025. Evaluating Generated Commit Messages with Large Language Models. https://anonymous.4open.science/r/4k8asu

    work page 2025

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computationa...

    work page 2005

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [5]

    Rpl Buse and W. R. Weimer. 2010. Automatically documenting program changes. In ASE 2010, 25th IEEE/ACM International Conference on Automated Software Engineering. ACM, 33–42

    work page 2010

  6. [6]

    Raymond PL Buse and Westley R Weimer. 2010. Automatically documenting program changes. In Proceedings of the 25th IEEE/ACM international conference on automated software engineering . 33–42

    work page 2010

  7. [7]

    Cheng-Han Chiang and Hung yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Annual Meeting of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:258461287

    work page 2023

  8. [8]

    Luis Fernando Cortés-Coy, Mario Linares-Vásquez, Jairo Aponte, and Denys Poshyvanyk. 2014. On automatically generating commit messages via summa- rization of source code changes. In IEEE 14th International Working Conference on Source Code Analysis and Manipulation . 275–284

    work page 2014

Show all 60 references
  1. [9]

    Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 International Symposium on Empirical Software Engineering and Measurement. IEEE Computer Society, 275–284. https://doi.org/ 10.1109/ESEM.2011.36

    2011 doi

  2. [10]

    Jinhao Dong, Yiling Lou, Qihao Zhu, Zeyu Sun, Zhilin Li, Wenjie Zhang, and Dan Hao. 2022. FIRA: fine-grained graph-based code change representation for automated commit message generation. In Proceedings of the 44th International Conference on Software Engineering . 970–981

    2022

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  4. [12]

    Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N Nguyen. 2013. Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 422– 431

    2013

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  6. [14]

    Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall. 2020. CC2Vec: Dis- tributed representations of code changes. In ACM/IEEE 42nd International Con- ference on Software Engineering . 518–529

    2020

  7. [15]

    Xing Hu, Qiuyuan Chen, Haoye Wang, Xin Xia, David Lo, and Thomas Zimmer- mann. 2022. Correlating automated and human evaluation of code documentation generation quality. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 4 (2022), 1–28

    2022

  8. [16]

    huggingface. 2025. Evaluate: A library for easily evaluating machine learning models and datasets. https://github.com/huggingface/evaluate

    2025

  9. [17]

    Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically generat- ing commit messages from diffs using neural machine translation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering . IEEE, 135–146

    2017

  10. [18]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 1–30

    2024

  11. [19]

    Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1-2 (1938), 81–93

    1938

  12. [20]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2024. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associ...

    2024

  13. [21]

    Udo Kuckartz, Stefan Rädiker, Thomas Ebert, and Julia Schehl. 2013. Statistik: eine verständliche Einführung. Springer-Verlag

    2013

  14. [22]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  15. [23]

    Jiawei Li and Iftekhar Ahmed. 2023. Commit message matters: Investigating impact and evolution of commit message quality. In 2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE) . IEEE, 806–817

    2023

  16. [24]

    Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed. 2024. Only diff Is Not Enough: Generating Commit Messages Leveraging Reasoning and Action of Large Language Model. Proc. ACM Softw. Eng. 1, FSE, Article 34 (July 2024), 22 pages. https://doi.org/10.1145/3643760

    2024 doi

  17. [25]

    Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04). 605–612

    2004

  18. [26]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  19. [27]

    Qin Liu, Zihe Liu, Hongming Zhu, Hongfei Fan, Bowen Du, and Yu Qian. 2019. Generating commit messages from diffs using pointer-generator network. In2019 IEEE/ACM 16th International Conference on Mining Software Repositories . IEEE, 299–309

    2019

  20. [28]

    Shangqing Liu, Cuiyun Gao, Sen Chen, Lun Yiu Nie, and Yang Liu. 2019. ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking. CoRR abs/1912.02972 (2019). arXiv:1912.02972 http://arxiv.org/abs/1912.02972

    2019 arXiv

  21. [29]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika B...

    2023 doi

  22. [30]

    Hassan, David Lo, Zhenchang Xing, and Xinyu Wang

    Zhongxin Liu, Xin Xia, Ahmed E. Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. 2018. Neural-machine-translation-based commit message generation: how far are we?. In 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, Sept...

    2018

  23. [31]

    Pablo Loyola, Edison Marrese-Taylor, Jorge Balazs, Yutaka Matsuo, and Fumiko Satoh. 2018. Content aware source code change description generation. In 11th International Conference on Natural Language Generation . 119–128

    2018

  24. [32]

    Pablo Loyola, Edison Marrese-Taylor, and Yutaka Matsuo. 2017. A Neural Ar- chitecture for Generating Natural Language Descriptions from Source Code Changes. In 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Comput...

    2017

  25. [33]

    Lun Yiu Nie, Cuiyun Gao, Zhicong Zhong, Wai Lam, Yang Liu, and Zenglin Xu

  26. [34]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  27. [35]

    N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  28. [36]

    Maxmilian Schall, Tamara Czinczoll, and Gerard De Melo. 2024. CommitBench: A benchmark for commit message generation. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 728–739

    2024

  29. [37]

    Jinfeng Shen, Xiaobing Sun, Bin Li, Hui Yang, and Jiajun Hu. 2016. On automatic summarization of what and why information in source code changes. In 2016 IEEE 40th Annual Computer Software and Applications Conference , Vol. 1. IEEE, 103–112

    2016

  30. [38]

    Ensheng Shi, Yanlin Wang, Wei Tao, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2022. RACE: Retrieval-augmented Commit Message Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozar...

    2022 doi

  31. [39]

    Charles Spearman. 1961. The proof and measurement of association between two things. (1961)

    1961

  32. [40]

    Qwen Team. 2025. QwQ-32B: Embracing the Power of Reinforcement Learning. https://qwenlm.github.io/blog/qwq-32b/

    2025

  33. [41]

    Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What makes a good commit message?. InProceedings of the 44th International Conference on Software Engineering. 2389–2401

    2022

  34. [42]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  35. [43]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  36. [44]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition . 4566–4575

    2015

  37. [45]

    Guoqing Wang, Zeyu Sun, Jinhao Dong, Yuxia Zhang, Mingxuan Zhu, Qingyuan Liang, and Dan Hao. [n. d.]. Is It Hard to Generate Holistic Commit Message? ACM Transactions on Software Engineering and Methodology ([n. d.])

  38. [46]

    Guoqing Wang, Zeyu Sun, Jinhao Dong, Yuxia Zhang, Mingxuan Zhu, Qingyuan Liang, and Dan Hao. 2024. Is It Hard to Generate Holistic Commit Message? ACM Trans. Softw. Eng. Methodol. (Sept. 2024). https://doi.org/10.1145/3695996 Just Accepted

    2024 doi

  39. [47]

    Haoye Wang, Xin Xia, David Lo, Qiang He, Xinyu Wang, and John Grundy

  40. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  41. [49]

    ACM Transactions on Software Engineering and Methodology 30, 4, Article 56 (2021), 30 pages

    Context-Aware Retrieval-Based Deep Commit Message Generation. ACM Transactions on Software Engineering and Methodology 30, 4, Article 56 (2021), 30 pages

    2021

  42. [50]

    Pengyu Xue, Linhao Wu, Zhongxing Yu, Zhi Jin, Zhen Yang, Xinyi Li, Zhenyu Yang, and Yue Tan. 2024. Automated Commit Message Generation With Large Language Models: An Empirical Study and Beyond.IEEE Transactions on Software Engineering 50, 12 (2024), 3208–3224. https://doi.org/...

    2024

  43. [51]

    Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu

  44. [52]

    Linghao Zhang, Jingshu Zhao, Chong Wang, and Peng Liang. 2024. Using Large Language Models for Commit Message Generation: A Preliminary Study. arXiv preprint arXiv:2401.05926 (2024)

    2024 arXiv

  45. [53]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  46. [54]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 Technical Report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  47. [55]

    Yuxia Zhang, Zhiqing Qiu, Klaas-Jan Stol, Wenhui Zhu, Jiaxin Zhu, Yingchen Tian, and Hui Liu. 2024. Automatic commit message generation: A critical review and directions for future work. IEEE Transactions on Software Engineering (2024)

    2024

  48. [56]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  49. [57]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

  50. [2019]

    In Twenty-Eighth International Joint Conference on Artificial Intelligence

    Commit Message Generation for Source Code Changes. In Twenty-Eighth International Joint Conference on Artificial Intelligence . 3975–3981

  51. [2020]

    InInternational Confer- ence on Learning Representations

    BERTScore: Evaluating Text Generation with BERT. InInternational Confer- ence on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr

  52. [2021]

    Neurocomputing 459 (2021), 97–107

    Coregen: Contextualized code representation learning for commit message generation. Neurocomputing 459 (2021), 97–107

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.