Pith. sign in

REVIEW 6 major objections 4 minor 95 references

Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message

T0 review · 6 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Improving human commit messages beats generating new ones

desk verdict Novel optimization-over-generation idea with real developer evidence on two of four metrics, but the headline percentages are the researchers' rankings and the automated metric is both objective and outcome—a solid but overstated paper. read the letter →

arxiv 2501.09861 v1 pith:3Q42JMYO submitted 2025-01-16 cs.SE

Jiawei Li , David Faragó , Christian Petrov , Iftekhar Ahmed This is my paper · ORCID
classification cs.SE
keywords commitmessageoptimizationgenerationlargelanguagemodelssearch-basedsoftwarecontexthuman-writtenautomatedevaluationopen-sourceJavaprojects
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the most effective way to produce a high-quality commit message is to start from the developer's own draft and optimize it, rather than to generate a message from the code change or complete a human-typed prefix. The proposed method, CMO, iteratively revises the human-written message one software context at a time, using an LLM to incorporate contexts that automated tools can retrieve and automated evaluators to score each candidate. Blind human evaluations by two researchers and 22 external developers both ranked CMO-optimized messages first on Rationality, Comprehensiveness, and Expressiveness more often than the original human messages, the OMG generation baseline, and the CMC completion baseline; the researcher rankings put CMO first on 88.2%-95.4% of commits for those three metrics. Conciseness is the exception: original human messages remained more concise. The paper concludes that selective, search-guided optimization of human input is a better paradigm for commit-message tooling than context-heavy generation from scratch.

What carries the argument

The mechanism that carries the argument is the one-context-per-step priority-queue search over candidate commit messages. Starting from the human-written seed, CMO's UPDATE step asks the LLM to improve the current best candidate using exactly one new software context from the seven retrievable ones (file information, commit type, pull-request or issue titles, class and method summaries, enclosing statement blocks, invoked methods, and variable data types), and EVALUATE scores each candidate with the combined objective: the mean of a fine-tuned LLM metric classifier's Likert-style scores and a retrieval-based semantic-similarity score against high-quality human messages for nearby diffs. The priority queue keeps the highest-scoring candidate at the front; stopping is governed by a decaying improvement threshold and a step limit. This design lets the search include only contexts relevant to the specific commit and avoids flooding the LLM with irrelevant context.

What would settle it

Blind human raters who have no access to the evaluator scores should rank CMO-optimized messages above the original human messages on Rationality, Comprehensiveness, and Expressiveness on a fresh sample of commits; if they do not, the claimed improvement is an artifact of optimizing the proxy.

Watch

Extended reading notes

Core claim

CMO is a search-based optimizer rather than a generator. It takes the human-written commit message as the initial candidate, evaluates it with an objective function that averages a fine-tuned LLM scoring model and a retrieval-based similarity score against high-quality human messages for semantically similar git diffs, and then runs a priority-queue search. At each step it dequeues the best-scoring candidate, has the LLM revise it by incorporating exactly one additional software context, scores the new candidate, and enqueues it; a decaying improvement threshold and a step limit terminate the loop. The paper claims this design outperforms both state-of-the-art CMG/CMC techniques and the original human messages on three of four evaluation dimensions in both researcher and external-developer blind evaluations, and that removing the search component (feeding all contexts at once) significantly degrades quality. It also claims the automated objective and the selective context injection are what make the difference, while traditional similarity metrics (BLEU, METEOR, ROUGE-L) do not capture the improvement.

Load-bearing premise

The load-bearing premise is that the automated objective function—whose Rationality and Comprehensiveness classifier agrees with humans only about 72% of the time, and whose retrieval-based similarity score shows no significant correlation with human Conciseness—still rewards the right qualities closely enough that optimizing it genuinely improves human message quality.

Editorial extensions

If this is right

  • If the evaluation results hold, commit-message tools should treat the developer's draft as an asset to refine, since CMO improved human messages on Rationality, Comprehensiveness, and Expressiveness in both researcher and external-developer rankings.
  • Conciseness is a remaining gap: the paper's results show original human messages ranked first on Conciseness more often than optimized ones, so optimization as currently rewarded trades brevity for completeness.
  • The ablation result implies that giving an LLM all available software contexts at once is worse than feeding one context per optimization step, which extends to prompt construction for code-intelligence tasks.
  • Standard text-similarity metrics (BLEU, METEOR, ROUGE-L) systematically favor prefix-completion over optimization and should not be used to judge commit-message quality.
  • The approach depends on the automated quality evaluators, so improvements in the evaluators would directly translate into better optimization targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same optimize-the-human-draft-with-one-context-per-step loop should transfer to other human-written software artifacts, such as code review comments, issue reports, and changelog entries, where the author's implicit context is hard to retrieve automatically; the paper does not test this.
  • Because the evaluator's Rationality and Comprehensiveness accuracy is about 0.719, a testable next step is to replace the objective with a stronger evaluator and see whether CMO's measured gains increase.
  • Adding a conciseness penalty or an explicit completeness-brevity trade-off to the objective function could plausibly close the Conciseness gap the paper documents; this is a direct extension of its own results.
  • The paper's decision not to build tools for the harder contexts (adhoc maintenance goals, project requirements, personal mistakes) suggests CMO's ceiling is bounded by what the initial human draft already contains; if a developer's draft is poor, the optimizer may have little to build on.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper proposes Commit Message Optimization (CMO), a search-based approach that starts from a human-written commit message and iteratively refines it with GPT-4, using automatically retrieved software contexts and two automated evaluators (a fine-tuned GPT-3.5 quality scorer and a retrieval-based similarity scorer) as the objective function. The authors conduct a qualitative study (RQ1) that identifies eight context themes human developers consider but that the OMG generation technique misses, then evaluate CMO against OMG, CMC, and human-written messages using both human rankings (two researchers on 76 commits; 22 Apache developers on 10 commits) and automated scores (Table 5). The abstract claims that CMO is 'significantly more Rational, Comprehensive, and Expressive' and outperforms baselines and human messages 88.2%-95.4% of the time.

Significance. The core idea—refining an existing human-written message via search-based LLM optimization rather than generating from scratch—is a timely and useful contribution to commit message research, and the RQ1 qualitative analysis of missed context themes is a concrete asset. The inclusion of an independent developer survey and an ablation study is also commendable. However, the strength of the evidence as presented does not support the headline claims: the automatic evaluation is circular (the outcome measure is the same combined score that is optimized), hyperparameters are selected without a clear holdout, the human evaluations lack significance tests and inter-rater reliability, and the independent developer results are substantially weaker than the researcher-derived numbers. If these issues are addressed, the paper could make a solid contribution; in its current form the claims outrun the evidence.

major comments (6)
  1. [Section 5.3, Eq. (2), Table 5] The Combined Evaluation Score (Eq. 2) is used both as the objective function in Algorithm 1 (the EVALUATE function) and as the outcome measure in Table 5, and its weights (SimCoeff and LLMCoeff) are computed from Pearson correlations on the 76-commit testing split that is later used for the automated comparison. Consequently, the Welch's t-test results in Table 5 compare methods on the very function that CMO is designed to maximize, and the evaluation metric itself is fitted to the test set. This circularity means the automatic gains are not independent evidence of higher message quality. Please provide a held-out evaluation with pre-fixed weights, or make the human evaluation the primary outcome with proper statistical analysis.
  2. [Section 5.5, Section 6] Hyperparameters (p=5, temperature=1, step_limit=50, Equation 2) are selected via grid search on 10 randomly sampled commits with manual review by four authors, and the text does not state whether these 10 commits are drawn from the 76-commit testing split used for Table 5 and the human evaluations. If they are, the test set is used for model selection and the reported results are in-sample. Even if the 10 commits are from the training split, the Pearson-based weighting in Eq. (2) is estimated on the testing split (Section 5.3), so the evaluation measure itself is test-derived. Please clarify the provenance of the 10 commits and, if necessary, re-evaluate on a genuinely held-out set with a pre-specified metric.
  3. [Section 6, Tables 3-4, Abstract] The abstract's claim of outperforming baselines and human messages 88.2%-95.4% of the time is based solely on the two researchers' rankings (Table 3), not on the 22 independent Apache developers (Table 4), where CMO's first-rank rates are 65.0% (Rationality), 63.2% (Comprehensiveness), 38.6% (Expressiveness), and 6.8% (Conciseness). No significance tests (e.g., Wilcoxon signed-rank, exact binomial, Friedman) or inter-rater reliability (e.g., Cohen's kappa) are reported for either human evaluation. The independent developer results actually contradict the implication of superiority on Conciseness, where human messages were ranked first 45.5% of the time versus 6.8% for CMO. The claims must be restricted to the specific metric/researcher subset and supported by appropriate statistical tests.
  4. [Section 4.3, Tables 3-4 vs Table 6] Section 4.3 states that researchers ranked eight commit messages (CMO, baselines, and CMO variants), but Tables 3 and 4 report only four message types with column sums of 152 and 220, respectively. It is unclear whether the reported ranks are conditional on the four main messages or were recomputed from the full eight-message ranking after discarding variants. This ambiguity affects the interpretation of every headline percentage; please clarify the ranking procedure and report the actual distribution if full eight-message rankings were used.
  5. [Section 5.3, Table 5] The Conciseness component of the automated evaluation is degenerate: because the retrieval-based Sim Score has no significant correlation with human Conciseness judgments, Eq. (2) uses only the LLM Score for Conciseness, and Table 5 reports 4.0 for every method. The automated evaluation therefore provides no discrimination on the very dimension where the human evaluators found CMO to be worst, and the 'Total' scores in Table 5 are inflated by a constant. The paper should either develop a valid automated Conciseness measure or exclude Conciseness from the automated comparison.
  6. [Section 5.4] The construction of the retrieval corpus is underspecified: the paper does not state whether the target commit's own diff or human-written message is excluded from the top-10 retrieved diffs, nor does it report the corpus size or its overlap with the 381-commit evaluation dataset. If a target commit appears in its own retrieval set, the Sim Score would reward staying close to the original human message, which would bias the optimization; if it is excluded, that should be stated explicitly. This is important because the retrieval-based evaluator is part of the objective function.
minor comments (4)
  1. [Section 4, Section 8] There are several typos and grammatical errors, including 'best best-performing hyper-parameters' in Section 4 and 'could not exhaustively tune the of these hyper-parameters' in Section 8; these should be corrected.
  2. [Section 6, Table 4] The sentence 'This reinforces the finding that CMO outperforms OMG/CMC and enhances human-written messages' overstates the developer results: for Expressiveness the first-rank rate is 38.6%, and for Conciseness CMO is last. Suggest tempering the language to 'directionally consistent for Rationality and Comprehensiveness'.
  3. [Table 5] The constant Conciseness value of 4.0 for all methods is not commented on in the text; please add a sentence explaining that no statistical test was run for Conciseness because the automated score has zero variance.
  4. [General] No link to code or data is provided, which would be helpful for reproducing the search-based optimization and the evaluator training; consider adding an artifact or repository link.

Circularity Check

2 steps flagged · score 6.0 of 10

Automatic evaluation is circular: CMO is optimized against the same Combined Evaluation Score later reported as success, and that evaluator's weights and top-k are tuned on the tested split.

  1. self definitional [Section 5.3 'Objective Function' / Section 6 'RQ2: The Effectiveness of CMO', Table 5]
    "We also evaluated the performance using automated evaluators by calculating the average scores for the Combined Evaluation Score with Equation 2 (Section Section 5.3), averaged across all commits in the testing split of the LLM-based Quality Evaluator."

    Section 5.3 defines the Combined Evaluation Score as 'the objective function for our optimization process' (Equations 1-2), and Algorithm 1 selects candidates by this score via EVALUATE. Section 6 then reports the same Combined Evaluation Score as the automatic outcome in Table 5. CMO's output is therefore selected to maximize the very metric that is later presented as evidence of its superiority; any gain over the human-written starting message is built into the selection rule, not independently demonstrated. The Welch's t-test results in Section 6 are significance tests on the optimized objective itself, so they cannot validate the automatic comparison as an external check.

  2. fitted input called prediction [Section 5.3 'Objective Function', retrieval-based evaluator parameter selection and Equation 2; Table 5]
    "we performed a Pearson correlation analysis on the scores from the LLM-based Quality Evaluator in the testing split. The correlation coefficients were then used to weight 'Sim Score' and 'LLM Score' in the combined score (Equation 2). For the Retrieval-based Quality Evaluator, we set it to retrieve the top 10 most similar git diffs, as this configuration produced the highest correlation coefficient."

    The combined evaluator's weights and retrieval top-k were chosen by maximizing correlation with human labels on the testing split. The same testing split is later used for the automatic evaluation in Table 5. Thus the automatic metric is fitted to the data it evaluates, making the reported automatic scores a fitted input rather than an independent prediction. The paper also notes that 'Sim Score' had no significant correlation with human Conciseness scores, yet this non-validated, partially fitted combined score remains the core automatic outcome for the paper's superiority claims.

full rationale

The core circularity is in the automatic evaluation: CMO is optimized against the Combined Evaluation Score (Equations 1-2), and Table 5 reports that same score as the evidence of 'statistically significantly' better Rationality, Comprehensiveness, and Expressiveness. That makes the automatic comparison self-referential rather than an independent assessment of message quality. Additionally, the evaluator's Pearson weights and top-10 retrieval setting were selected on the testing split, which is exactly the split used for the final automatic evaluation, so the reported automatic metric is partly fitted to the data it claims to predict. The study does retain some independent evidence: the Apache developer survey shows CMO ranked first more often than OMG/CMC on Rationality and Comprehensiveness, though far less often than the researcher rankings and not on Conciseness, so the central contribution is not entirely circular. Self-citations to the authors' OMG work are used as a baseline, dataset source, and quality-metric source, but that is a normal baseline comparison and not itself a load-bearing circularity argument under the rules. The main deduction is therefore the automatic-evaluation circularity and test-set fitting, giving a partial-circularity score of 6 rather than a fully forced derivation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on free parameters that were fitted to sampled data or to the test split, and on several domain assumptions about evaluation validity. No new physical or conceptual entities are postulated; CMO, the evaluators, and the context tools are proposed methods, not invented entities.

free parameters (6)
  • improvement threshold percentage p = 5
    Selected by grid search over 10 sampled commits and four authors' manual evaluation (Section 5.5).
  • increased GPT-4 temperature = 1
    Grid-searched over 0.5 and 1; value 1 chosen for final CMO settings (Section 5.5).
  • step_limit N = 50
    Selected by grid search over 10, 30, 50, and 60 (Section 5.5).
  • combined evaluation score equation = Equation 2
    Grid-searched between Equation 1 and Equation 2; Equation 2 chosen (Section 5.5).
  • retrieval evaluator top-k = 10
    Set to 10 because this configuration produced the highest correlation with human-labeled scores on the testing split (Section 5.3), a parameter fitted to the evaluation data.
  • correlation weights SimCoeff and LLMCoeff = estimated from testing split
    Pearson correlation coefficients between the retrieval score, the LLM score, and human labels on the testing split are used as weights in Equation 2 (Section 5.3).
assumptions (5)
  • domain assumption The four metrics Rationality, Comprehensiveness, Conciseness, and Expressiveness adequately capture commit message quality.
    Adopted from prior work by Li et al. [48] and used without independent validation of the metric set (Section 4.2.1).
  • ad hoc to paper The fine-tuned GPT-3.5 Turbo evaluator predicts human quality scores well enough to serve as an optimization objective.
    Accuracy is 0.719 for Rationality and Comprehensiveness and higher for the other two metrics, yet the model is used as part of the objective function (Section 5.3).
  • domain assumption Cosine similarity to human-written messages for semantically similar diffs is a valid proxy for commit message quality.
    The retrieval-based evaluator assumes that closeness to good human messages for similar diffs indicates quality (Section 5.3).
  • domain assumption Human-written commit messages encode useful developer contexts that automated tools cannot retrieve reliably.
    This is the motivation for starting from human messages rather than generating from scratch (Section 5.2).
  • domain assumption The 381-commit Apache corpus is representative enough for training evaluators and for evaluating CMO.
    Acknowledged as a threat to internal validity in Section 8; the dataset is the only one manually labeled with the four metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message." pith.science (2026). https://pith.science/paper/3Q42JMYO

@misc{pith2026250109861,
  author       = {Pith},
  title        = {Pith review of: Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3Q42JMYO}},
  note         = {Machine review of arXiv:2501.09861}
}
read the original abstract

Commit messages are crucial in software development, supporting maintenance tasks and communication among developers. While Large Language Models (LLMs) have advanced Commit Message Generation (CMG) using various software contexts, some contexts developers consider are often missed by CMG techniques and can't be easily retrieved or even retrieved at all by automated tools. To address this, we propose Commit Message Optimization (CMO), which enhances human-written messages by leveraging LLMs and search-based optimization. CMO starts with human-written messages and iteratively improves them by integrating key contexts and feedback from external evaluators. Our extensive evaluation shows CMO generates commit messages that are significantly more Rational, Comprehensive, and Expressive while outperforming state-of-the-art CMG methods and human messages 88.2%-95.4% of the time.

Figures

Figures reproduced from arXiv: 2501.09861 by the authors.

Figure 1
Figure 1. Example Commits 3. Project Requirements (11.3%, 43/381): Developers often implicitly rely on project requirements or recommended practices to explain code changes. Phrases like “should,” “do not,” or “allow” suggest that a project requirement or convention influenced the decision, though developers did not reference specific requirements directly. Instead, they use certain words to imply the necessity of the change.… view at source ↗
Figure 2
Figure 2. Overview of Retrieval-based Quality Evaluator [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Example of Code Refactoring large); Comprehensiveness: p-val<1.773e-13, Cohen’s D (1.346, large); Expressiveness: p-val<7.965e-13, Cohen’s D (1.302, large)), indicating CMO produces higher-quality messages than state-of-the￾art generation/completion techniques. Additionally, CMO effectively optimizes human-written messages (Rationality: p-val<1.698e-9, Cohen’s D(1.065, large); Comprehensiveness: p-val<4.478e-6, Cohe… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

95 extracted references · 57 canonical work pages

  1. [1]

    Commit Example 1

    2004. Commit Example 1. github.com/apache/cocoon/commit/50507178cbe40b27b9052b487cfb1da475223da1

    work page arXiv 2004

  2. [2]

    Commit Example 2

    2006. Commit Example 2. github.com/apache/archiva/commit/ce29219ee6409b111ce8f8191b1792c2b65a04a8

    2006

  3. [3]

    Commit Example 4

    2006. Commit Example 4. github.com/apache/jmeter/commit/ffebf3c0589a4793969b0e5b3d21bacfb03c2e49

    2006

  4. [4]

    Commit Example 3

    2007. Commit Example 3. github.com/apache/tomee/commit/f1e8677a6cf85b899ba4727728b10cccdc79b39f

    2007

  5. [5]

    Commit Example 10

    2011. Commit Example 10. github.com/apache/wicket/commit/cc195638e57df89bcdb8f72b344107c673143d96

    2011

  6. [6]

    Commit Example 12

    2011. Commit Example 12. github.com/apache/logging-log4j2/commit/27c0a1556a47a2c594aece3c355e891ff4f19836

    2011

  7. [7]

    Commit Example 7

    2012. Commit Example 7. github.com/apache/hadoop/commit/6f2aae046d9ebeeae1cd82a687ed6fb165405988

    2012

  8. [8]

    Commit Example 9

    2012. Commit Example 9. github.com/apache/jclouds/commit/559a37ceff1f7d5dc83cd32e377610e410769eeb

    2012

Show all 95 references
  1. [9]

    Commit Example 6

    2016. Commit Example 6. github.com/apache/cassandra/commit/eaced9a541d09d55973b6f88d720e16ac948a559

    2016

  2. [10]

    Commit Example 8

    2016. Commit Example 8. github.com/apache/logging-log4j2/commit/856607faf268136fc6f2e27cf3716a5e16a0f48d

    2016

  3. [11]

    Successful Case Commit

    2020. Successful Case Commit. github.com/apache/jena/commit/ecfe9eda7adf6f9fcc0d88e8538e436103e823b0

    2020

  4. [12]

    ChatGPT: Optimizing Language Models for Dialogue

    2022. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt

    2022

  5. [13]

    GPT-3.5 Turbo

    2022. GPT-3.5 Turbo. https://platform.openai.com/docs/models/gpt-3-5-turbo

    2022

  6. [14]

    IBM Global AI Adoption Index 2022

    2022. IBM Global AI Adoption Index 2022. https://www.ibm.com/watson/resources/ai-adoption

    2022

  7. [15]

    awesome-chatgpt-prompts

    2023. awesome-chatgpt-prompts. https://github.com/f/awesome-chatgpt-prompts

    2023

  8. [16]

    2023. GPT-4. https://openai.com/research/gpt-4

    2023

  9. [17]

    imbalanced-learn documentation

    2024. imbalanced-learn documentation. https://imbalanced-learn.org/stable/

    2024

  10. [18]

    OpenAI API Pricing

    2024. OpenAI API Pricing. https://openai.com/api/pricing/

    2024

  11. [19]

    Sentence Transformers

    2024. Sentence Transformers. https://sbert.net/docs/sentence_transformer/pretrained_models.html

    2024

  12. [20]

    sentence-transformers/all-mpnet-base-v2

    2024. sentence-transformers/all-mpnet-base-v2. https://huggingface.co/sentence-transformers/all-mpnet-base-v2

    2024

  13. [21]

    sklearn.metrics

    2024. sklearn.metrics. https://scikit-learn.org/stable/api/sklearn.metrics.html

    2024

  14. [22]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  15. [23]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  16. [24]

    Marcel Böhme, Charaka Geethal, and Van-Thuan Pham. 2020. Human-in-the-loop automatic program repair. In 2020 IEEE 13th international conference on software testing, validation and verification (ICST) . IEEE, 274–285

    2020

  17. [25]

    Kuljit Kaur Chahal and Munish Saini. 2018. Developer dynamics and syntactic quality of commit messages in oss projects. In Open Source Systems: Enterprise Software and Solutions: 14th IFIP WG 2.13 International Conference, OSS 2018, Athens, Greece, June 8-10, 2018, Proceedings...

    2018

  18. [26]

    Dan Chen and Sally E Goldin. 2020. A project-level investigation of software commit comments and code quality. In 2020 3rd International Conference on Information and Communications Technology (ICOIACT) . IEEE, 240–245

    2020

  19. [27]

    Lei Chen, Dandan Wang, Junjie Wang, and Qing Wang. 2019. Enhancing unsupervised requirements traceability with sequential semantics. In 2019 26th Asia-Pacific Software Engineering Conference (APSEC) . IEEE, 23–30

    2019

  20. [28]

    Marc J Diener. 2010. Cohen’s d. The Corsini encyclopedia of psychology (2010), 1–1

    2010

  21. [29]

    Jinhao Dong, Yiling Lou, Dan Hao, and Lin Tan. 2023. Revisiting learning-based commit message generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 794–805

    2023

  22. [30]

    Jinhao Dong, Yiling Lou, Qihao Zhu, Zeyu Sun, Zhilin Li, Wenjie Zhang, and Dan Hao. 2022. FIRA: fine-grained graph- based code change representation for automated commit message generation. In Proceedings of the 44th International Conference on Software Engineering . 970–981

    2022

  23. [31]

    Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N Nguyen. 2013. Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 422–431

    2013

  24. [32]

    Aleksandra Eliseeva, Yaroslav Sokolov, Egor Bogomolov, Yaroslav Golubev, Danny Dig, and Timofey Bryksin. 2023. From commit message generation to history-aware commit message completion. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IE...

    2023

  25. [33]

    Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383 (2024)

    2024 arXiv

  26. [34]

    Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. 2023. What makes good in-context demonstrations for code intelligence tasks with llms?. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 761–773

    2023

  27. [35]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao

  28. [36]

    Yichen He, Liran Wang, Kaiyi Wang, Yupeng Zhang, Hang Zhang, and Zhoujun Li. 2023. COME: Commit Message Generation with Modification Embedding. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 792–803

    2023

  29. [37]

    Tobias Hey, Fei Chen, Sebastian Weigelt, and Walter F Tichy. 2021. Improving traceability link recovery using fine- grained requirements-to-code relations. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 12–22

    2021

  30. [38]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou

  31. [39]

    Yuan Huang, Nan Jia, Hao-Jie Zhou, Xiang-Ping Chen, Zi-Bin Zheng, and Ming-Dong Tang. 2020. Learning human- written commit messages to document code changes.Journal of Computer Science and Technology 35 (2020), 1258–1277

    2020

  32. [40]

    Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically generating commit messages from diffs using neural machine translation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 135–146

    2017

  33. [41]

    Xue Jiang, Yihong Dong, Lecheng Wang, Fang Zheng, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2023. Self-planning Code Generation with Large Language Models. ACM Transactions on Software Engineering and Methodology (2023)

    2023

  34. [42]

    Álvaro Barbero Jiménez, Jorge López Lázaro, and José R Dorronsoro. 2008. Finding optimal model parameters by discrete grid search. In Innovations in hybrid intelligent systems . Springer, 120–127

    2008

  35. [43]

    Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems 35 (2022), 11785–11799

    2022

  36. [44]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Repr...

    2023

  37. [45]

    Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas, et al. 2006. Handling imbalanced datasets: A review. GESTS international transactions on computer science and engineering 30, 1 (2006), 25–36

    2006

  38. [46]

    Stanislav Levin and Amiram Yehudai. 2017. Boosting automatic commit classification into maintenance activities by utilizing source code changes. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. 97–106

    2017

  39. [47]

    Jiawei Li and Iftekhar Ahmed. 2023. Commit message matters: Investigating impact and evolution of commit message quality. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 806–817

    2023

  40. [48]

    Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed. 2024. Only diff Is Not Enough: Generating Commit Messages Leveraging Reasoning and Action of Large Language Model. Proceedings of the ACM on Software Engineering 1, FSE (2024), 745–766

    2024

  41. [49]

    Jia Li, Yongmin Li, Ge Li, Zhi Jin, Yiyang Hao, and Xing Hu. 2023. Skcoder: A sketch-based approach for automatic code generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2124–2135

    2023

  42. [50]

    Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. 2023. Cct5: A code-change-oriented pre-trained model. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1509–1521

    2023

  43. [51]

    Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04). 605–612

    2004

  44. [52]

    Mario Linares-Vásquez, Luis Fernando Cortés-Coy, Jairo Aponte, and Denys Poshyvanyk. 2015. Changescribe: A tool for automatically generating commit messages. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 709–712

    2015

  45. [53]

    Shengcai Liu, Caishun Chen, Xinghua Qu, Ke Tang, and Yew-Soon Ong. 2023. Large language models as evolutionary optimizers. arXiv preprint arXiv:2310.19046 (2023)

    2023 arXiv

  46. [54]

    Shangqing Liu, Cuiyun Gao, Sen Chen, Lun Yiu Nie, and Yang Liu. 2020. Atom: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Transactions on Software Engineering 48, 5 (2020), 1800–1817

    2020

  47. [55]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634 (2023). , Vol. 1, No. 1, Article . Publication date: January 2025. Optimization is Better than Ge...

    2023 arXiv

  48. [56]

    Zhongxin Liu, Xin Xia, Ahmed E Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. 2018. Neural-machine- translation-based commit message generation: how far are we?. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering . 373–384

    2018

  49. [57]

    Umme Ayda Mannan, Iftekhar Ahmed, Carlos Jensen, and Anita Sarma. 2020. On the relationship between design discussions and design quality: a case study of Apache projects. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on...

    2020

  50. [58]

    Kevin Moran, David N Palacio, Carlos Bernal-Cárdenas, Daniel McCrystal, Denys Poshyvanyk, Chris Shenefiel, and Jeff Johnson. 2020. Improving the effectiveness of traceability link recovery using hierarchical bayesian networks. In Proceedings of the ACM/IEEE 42nd International ...

    2020

  51. [59]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354

    2024

  52. [60]

    Nicholas Smith, Danny van Bruggen, and Federico Tomassetti. 2023. Tools for your Java code. https://javaparser.org/

    2023

  53. [61]

    Lun Yiu Nie, Cuiyun Gao, Zhicong Zhong, Wai Lam, Yang Liu, and Zenglin Xu. 2021. Coregen: Contextualized code representation learning for commit message generation. Neurocomputing 459 (2021), 97–107

    2021

  54. [62]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  55. [63]

    Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. arXiv preprint arXiv:2303.13780 (2023)

    2023 arXiv

  56. [64]

    Graeme D Ruxton. 2006. The unequal variance t-test is an underused alternative to Student’s t-test and the Mann– Whitney U test. Behavioral Ecology 17, 4 (2006), 688–690

    2006

  57. [65]

    Jinfeng Shen, Xiaobing Sun, Bin Li, Hui Yang, and Jiajun Hu. 2016. On automatic summarization of what and why information in source code changes. In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 103–112

    2016

  58. [66]

    Ensheng Shi, Yanlin Wang, Wei Tao, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2022. RACE: Retrieval-augmented Commit Message Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . 5520–5530

    2022

  59. [67]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou

  60. [68]

    Edward Smith, Robert Loftin, Emerson Murphy-Hill, Christian Bird, and Thomas Zimmermann. 2013. Improving developer participation rates in surveys. In2013 6th International workshop on cooperative and human aspects of software engineering (CHASE). IEEE, 89–92

    2013

  61. [69]

    Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397 (2023)

    2023 arXiv

  62. [70]

    In International Conference on Machine Learning

    Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227

  63. [71]

    Haoye Tian, Xunzhu Tang, Andrew Habib, Shangwen Wang, Kui Liu, Xin Xia, Jacques Klein, and Tegawendé F Bissyandé. 2022. Is this change the answer to that problem? correlating descriptions of bug and code changes for evaluating patch correctness. In Proceedings of the 37th IEEE...

    2022

  64. [72]

    Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What makes a good commit message?. In Proceedings of the 44th International Conference on Software Engineering . 2389–2401

    2022

  65. [73]

    Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the evaluation of commit message generation models: An experimental study. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 126–136

    2021

  66. [74]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  67. [75]

    Yuvraj Virk, Premkumar Devanbu, and Toufique Ahmed. 2024. Enhancing Trust in LLM-Generated Code Summaries with Calibrated Confidence Scores. arXiv preprint arXiv:2404.19318 (2024)

    2024 arXiv

  68. [76]

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems 36 (2023), 75993–76005

    2023

  69. [77]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou

  70. [78]

    Liran Wang, Xunzhu Tang, Yichen He, Changyu Ren, Shuhua Shi, Chaoran Yan, and Zhoujun Li. 2023. Delving into commit-issue correlation to enhance commit message generation models. In 2023 38th IEEE/ACM International , Vol. 1, No. 1, Article . Publication date: January 2025. 22 ...

    2023

  71. [79]

    Bei Wang, Meng Yan, Zhongxin Liu, Ling Xu, Xin Xia, Xiaohong Zhang, and Dan Yang. 2021. Quality assurance for automated commit message generation. In 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 260–271

    2021

  72. [80]

    Yifan Wu, Ying Li, and Siyu Yu. 2024. Commit Message Generation via ChatGPT: How Far Are We?. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering . 124–129

    2024

  73. [81]

    arXiv preprint arXiv:2303.04048 (2023)

    Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048 (2023)

    2023 arXiv

  74. [82]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. InThe Twelfth International Conference on Learning Representations. https://openreview. net/forum?id=Bb4VGOWELI

    2024

  75. [83]

    Ying Wang, Bihuan Chen, Kaifeng Huang, Bowen Shi, Congying Xu, Xin Peng, Yijian Wu, and Yang Liu. 2020. An empirical study of usages, updates and risks of third-party libraries in java projects. In2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)...

    2020

  76. [84]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    2022 arXiv

  77. [85]

    Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu. 2019. Commit message generation for source code changes. In IJCAI

    2019

  78. [86]

    Chenyuan Zhang, Yanlin Wang, Zhao Wei, Yong Xu, Juhong Wang, Hui Li, and Rongrong Ji. 2023. EALink: An efficient and accurate pre-trained framework for issue-commit link recovery. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 217–229

    2023

  79. [87]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  80. [88]

    Yuxia Zhang, Zhiqing Qiu, Klaas-Jan Stol, Wenhui Zhu, Jiaxin Zhu, Yingchen Tian, and Hui Liu. 2024. Automatic commit message generation: A critical review and directions for future work.IEEE Transactions on Software Engineering (2024)

    2024

  81. [89]

    Lucas Zamprogno, Braxton Hall, Reid Holmes, and Joanne M Atlee. 2022. Dynamic human-in-the-loop assertion generation. IEEE Transactions on Software Engineering 49, 4 (2022), 2337–2351

    2022

  82. [90]

    Yaqin Zhou, Jing Kai Siow, Chenyu Wang, Shangqing Liu, and Yang Liu. 2021. Spi: Automated identification of security patches via commits. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–27

    2021

  83. [91]

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534 (2023)

    2023 arXiv

  84. [93]

    Teng Zhao, Qinghua Cao, and Qing Sun. 2017. An improved approach to traceability recovery based on word embeddings. In 2017 24th Asia-Pacific Software Engineering Conference (APSEC) . IEEE, 81–89

    2017

  85. [95]

    Thomas Zimmermann. 2016. Card-sorting: From text to themes. In Perspectives on data science for software engineering . Elsevier, 137–141. , Vol. 1, No. 1, Article . Publication date: January 2025

    2016

  86. [2023]

    arXiv preprint arXiv:2310.01798 (2023)

    Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798 (2023)

    2023 arXiv

  87. [2024]

    In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–13

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.