REVIEW 3 major objections 5 minor 18 references
Contextual Code Retrieval for Commit Message Generation: A Preliminary Study
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper argues that a commit-message generator cannot work well from a raw diff alone, and that adding repository code that calls or instantiates the changed functions yields more informative messages.
desk verdict A plausible retrieval-augmented CMG method with a useful new dataset, but the headline improvement rests on a two-author human eval with no significance testing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the augmented Code Structure Graph. C3Gen first parses each repository source file into a graph whose nodes are source files, classes, and functions, and records every definition in a DefinitionIndex. A second parse of the diff builds a ModifiedEntityList of changed functions and classes, and a second graph pass locates places where those entities are called or instantiated elsewhere. The enclosing function or class bodies, or 25 lines around global-scope uses, become the relevant code context prepended to the diff.
What would settle it
Have at least ten developers who did not write the paper rate the same 370 sampled generated messages on Completeness, Clarity, and Correctness while blind to which messages used retrieved context, and compare the Completeness difference; if the C3Gen advantage disappears or reverses, the central claim is refuted.
Extended reading notes
Core claim
The central claim is that diff-only input is insufficient for commit message generation. C3Gen retrieves the bodies of functions, or surrounding lines, where code in the repository invokes or instantiates the entities the diff modifies, and adds that context to the prompt. In human evaluation of 370 commits, this consistently improved Completeness by around 5% on a 1-5 scale across all four tested models, kept Correctness essentially the same, and produced a slight Clarity decrease. The same additions produced mixed, statistically insignificant changes on BLEU, ROUGE-L, METEOR, and CIDEr, leading the paper to argue that similarity-based metrics underestimate the practical benefit of repository context.
Load-bearing premise
The claimed Completeness improvement rests on the assumption that the two authors who rated the messages were unbiased judges of their own system's output, since no independent raters or inter-rater agreement check support the scores.
Editorial extensions
If this is right
- Across all four LLMs, adding retrieved repository context produced a consistent human-rated Completeness gain of roughly 5% on the 1-5 scale.
- Clarity dropped slightly for every model, so the added information carries a small cognitive-load cost.
- Correctness stayed about the same, indicating that the extra context did not cause more hallucinated or inaccurate statements about the change.
- Reference-based similarity metrics showed no consistent improvement, so evaluating such a system only with BLEU or similar metrics would miss the measured human-perceived benefit.
- Developer-written reference messages were rated lower than LLM-generated messages on all three subjective dimensions, casting doubt on the assumption that the human-written message is the best target.
Reading between the lines
- A direct extension would be to repeat the human evaluation with developers who did not author the system and who are blind to which messages used retrieved context; if the Completeness advantage reproduces, it would strongly confirm that similarity metrics undercount retrieval benefits.
- The program-structure retrieval idea could transfer to other code summarization tasks, such as release notes or code-review comments, where a diff alone also lacks the calling context.
- The paper's own proposed next step implies a testable trade-off: filtering retrieved snippets to those tied to the central modified function should recover the small Clarity loss while keeping the Completeness gain, a hypothesis that could be measured on the same dataset.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes C3Gen, a retrieval-augmented framework for commit message generation that supplements a raw code diff with repository-level code snippets. C3Gen constructs Code Structure Graphs (CSGs) via tree-sitter, identifies modified functions/classes, and extracts the bodies of functions/classes that invoke or instantiate those entities, plus a 25-line window around global-scope invocations. The authors also introduce ApacheCM, a dataset of over 230,000 commits from 50 Apache projects with metadata fields, and evaluate C3Gen against a diff-only baseline ("Naive") across GPT-4o, GPT-4.1, DeepSeek V3, and DeepSeek R1. Objective metrics (BLEU, ROUGE-L, METEOR, CIDEr) show mixed and statistically insignificant differences, while a human evaluation by two of the authors reports consistent Completeness gains of about 5% for C3Gen. The paper's central claim is that adding contextual code improves the informativeness and practical value of generated commit messages, and that similarity-based objective metrics are unreliable.
Significance. If the central claim is accepted, the paper makes a useful contribution to commit message generation: it challenges the diff-only formulation, provides a context-enrichment mechanism, and releases a dataset with rich metadata and a replication package. The paper is honest about the mixed objective results and provides a thought-provoking analysis of why similarity-based metrics may be misleading for CMG. The main weakness is that the only consistent evidence for the claimed improvement is a two-author human evaluation with no significance testing, no inter-rater reliability, and no blinding. Because the positive result and the paper's critique of objective metrics both rest on that evaluation, the evidence base is currently too thin to fully support the abstract and Section VII claims. The dataset and framework are nevertheless valuable resources for the community if the evaluation concerns are addressed.
major comments (3)
- [§VI-B, Table IV, Key Finding 2] The central claim that C3Gen improves informativeness rests on the Completeness gains in Table IV, which are averages from two of the paper's authors with no inter-rater reliability statistic, no significance test, and no blinding of the method being rated. Key Finding 2 states that C3Gen 'significantly enhance[s]' Completeness, but no statistical test accompanies this statement. The authors should report per-rater and per-item scores, an agreement measure such as Cohen's kappa, and a paired significance test (e.g., Wilcoxon signed-rank across instances), or justify why these are unnecessary at this sample size.
- [§VI-B, Table IV] In Table IV, developer-written reference messages scored far below all LLM outputs on every dimension (e.g., Completeness 2.69 vs. 3.51–4.50). This pattern is more consistent with a rating rubric or rater population that rewards verbose, conventional phrasing than with an objective quality gap, and it conflicts with the use of those same references as ground truth for the objective metrics in Table III. The authors' conclusion that developer-written messages are of lower quality in terms of human preference is not supported without validating the rubric (e.g., external developer raters, or a pilot on typical real-world messages). Because this result is also used to justify the unreliability of objective metrics, the validity of the Completeness construct is load-bearing for the paper's central claim.
- [§III-C, §VI-A] Section III-C leaves key retrieval parameters unspecified: the number of snippets retrieved (the analysis in Figure 2 refers to 'ten retrieved code snippets' but Stage III does not define a retrieval count), and the 25-line context window is a heuristic with no sensitivity analysis. The framework is also compared only against a diff-only baseline, so the improvement cannot be attributed to the CSG-based selection rather than to simply having more code text. I ask for the retrieval count to be stated, a sensitivity check or ablation for the window size, and ideally a control condition that retrieves non-contextual code of similar size.
minor comments (5)
- [Table II vs. §IV-B] Table II reports an ApacheCM train set of 249,830 commits, while §IV-B states the dataset comprises 234,799 commits; please reconcile this discrepancy.
- [§VI-A, Figure 3, Key Finding 2] The paper contains several typographical errors: 'sylictic' in §VI-A, 'relavantCodeResult' in Figure 3, and 'cular' in Key Finding 2.
- [Figure 2] Figure 2 appears to be a low-resolution rendering with garbled characters and unreadable glyphs; a clean vector version is needed for review.
- [§IV-B, Table I] The dataset filtering criterion that commit messages between 5 and 50 words correspond to 'high-quality' messages is asserted without evidence; please cite or soften this claim and state it as a limitation.
- [§VII] In §VII the project is described as using 'advanced retrieval techniques,' but Stage III is a deterministic static-analysis procedure; a more modest description would be appropriate.
Circularity Check
No circularity: C3Gen is an empirical retrieval pipeline whose outputs are not defined in terms of its inputs; the author self-rating issue is a validity threat, not a circular derivation.
full rationale
The paper contains no formal derivation chain in which an output is defined in terms of an input or in which a fitted parameter is renamed as a prediction. C3Gen is described as a static-analysis retrieval pipeline: it builds Code Structure Graphs, augments them with diff information, and extracts enclosing function/class bodies or 25-line windows around call sites (Section III). The 25-line window is an explicitly stated heuristic, not a parameter fitted to the evaluation data. The objective metrics (BLEU, ROUGE-L, METEOR, CIDEr) and subjective metrics (Clarity, Completeness, Correctness) are defined independently of C3Gen's mechanism; Completeness is defined as how thoroughly the message captures changes in the code diff, not in terms of whether C3Gen's retrieved snippets were used. The central empirical claim rests on Section VI-B's human evaluation, where two authors rated candidates. That is a methodological validity concern (lack of blinding, no inter-rater reliability, no significance testing), but it is not circularity: the scores are not constructed from the method's outputs or from the paper's own definitions in a way that forces the reported improvement. Self-citations [3] and [10] appear only in related-work positioning and do not supply the load-bearing argument for C3Gen's effectiveness. No uniqueness theorem, imported ansatz, or self-definitional equivalence is present. Accordingly, no specific circular step can be exhibited, and the correct finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Retrieval context window =
25 lines before and after invocation
- Number of retrieved code snippets =
10 (apparent, not explicitly stated in method)
- Human evaluation sample size =
370 instances
- Dataset filtering thresholds =
5-50 words, <=300 changed lines, exclude bots/reverts/merges
assumptions (4)
- domain assumption Tree-sitter parsing accurately identifies class/function definitions and invocation/instantiation relationships across the supported languages.
- ad hoc to paper Commit messages between 5 and 50 words and diffs with fewer than 300 changed lines correspond to high-quality, informative commits.
- domain assumption The repository's main branch commits since 2015 are representative of real-world commit messages.
- domain assumption BLEU, ROUGE-L, METEOR, and CIDEr are reasonable heuristic proxies for commit message quality.
invented entities (1)
-
Code Structure Graph (CSG)
independent evidence
Cite this review
Pith. "Pith review of Contextual Code Retrieval for Commit Message Generation: A Preliminary Study." pith.science (2026). https://pith.science/paper/KQQWNG5H
@misc{pith2026250717690,
author = {Pith},
title = {Pith review of: Contextual Code Retrieval for Commit Message Generation: A Preliminary Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/KQQWNG5H}},
note = {Machine review of arXiv:2507.17690}
}
read the original abstract
A commit message describes the main code changes in a commit and plays a crucial role in software maintenance. Existing commit message generation (CMG) approaches typically frame it as a direct mapping which inputs a code diff and produces a brief descriptive sentence as output. However, we argue that relying solely on the code diff is insufficient, as raw code diff fails to capture the full context needed for generating high-quality and informative commit messages. In this paper, we propose a contextual code retrieval-based method called C3Gen to enhance CMG by retrieving commit-relevant code snippets from the repository and incorporating them into the model input to provide richer contextual information at the repository scope. In the experiments, we evaluated the effectiveness of C3Gen across various models using four objective and three subjective metrics. Meanwhile, we design and conduct a human evaluation to investigate how C3Gen-generated commit messages are perceived by human developers. The results show that by incorporating contextual code into the input, C3Gen enables models to effectively leverage additional information to generate more comprehensive and informative commit messages with greater practical value in real-world development scenarios. Further analysis underscores concerns about the reliability of similaritybased metrics and provides empirical insights for CMG.
Figures
Reference graph
Works this paper leans on
-
[1]
What makes a good commit message?
Y . Tian, Y . Zhang, K.-J. Stol, L. Jiang, and H. Liu, “What makes a good commit message?” in Proceedings of the 44th International Conference on Software Engineering (ICSE) . ACM, 2022, pp. 2389–2401
work page 2022
-
[2]
Automatic commit message generation: A critical review and directions for future work,
Y . Zhang, Z. Qiu, K.-J. Stol, W. Zhu, J. Zhu, Y . Tian, and H. Liu, “Automatic commit message generation: A critical review and directions for future work,” IEEE Transactions on Software Engineering , vol. 50, no. 4, pp. 816–835, 2024
2024
-
[3]
Using large language models for commit message generation: A preliminary study,
L. Zhang, J. Zhao, C. Wang, and P. Liang, “Using large language models for commit message generation: A preliminary study,” in Proceedings of the 31st IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2024, pp. 126–130
work page 2024
-
[4]
B. Xiong, L. Zhang, C. Wang, and P. Liang, “Replication package of the paper: Contextual code retrieval for commit message generation: A preliminary study,” May 2025. [Online]. Available: https://doi.org/10.5281/zenodo.15502317
-
[5]
Automatically documenting program changes,
R. P. Buse and W. R. Weimer, “Automatically documenting program changes,” in Proceedings of the 25th IEEE/ACM International Confer- ence on Automated Software Engineering (ASE) . ACM, 2010, pp. 33–42
work page 2010
-
[6]
Changescribe: A tool for automatically generating commit messages,
M. Linares-V ´asquez, L. F. Cort ´es-Coy, J. Aponte, and D. Poshyvanyk, “Changescribe: A tool for automatically generating commit messages,” in Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE) . IEEE, 2015, pp. 709–712
work page 2015
-
[7]
On automatic summarization of what and why information in source code changes,
J. Shen, X. Sun, B. Li, H. Yang, and J. Hu, “On automatic summarization of what and why information in source code changes,” in Proceedings of the 40th IEEE Annual Computer Software and Applications Conference (COMPSAC). IEEE, 2016, pp. 103–112
work page 2016
-
[8]
Neural machine-translation-based commit message generation: how far are we?
Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural machine-translation-based commit message generation: how far are we?” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE) . ACM, 2018, pp. 373–384
work page 2018
Show all 18 references
-
[9]
Automatically generating commit messages from diffs using neural machine translation,
S. Jiang, A. Armaly, and C. McMillan, “Automatically generating commit messages from diffs using neural machine translation,” in Pro- ceedings of the 32nd ACM/IEEE International Conference on Automated Software Engineering (ASE) . IEEE, 2017, pp. 135–146
2017
-
[10]
Rag-enhanced commit message generation,
L. Zhang, H. Zhang, C. Wang, and P. Liang, “Rag-enhanced commit message generation,” arXiv preprint arXiv:2406.05514 , 2024
2024 arXiv
-
[11]
Commit message generation for source code changes,
S. Xu, Y . Yao, F. Xu, T. Gu, H. Tong, and J. Lu, “Commit message generation for source code changes,” in Proceedings of the 28th Inter- national Joint Conference on Artificial Intelligence (IJCAI) . IJCAI, 2019, pp. 3975–3981
2019
-
[12]
Context-aware retrieval-based deep commit message generation,
H. Wang, X. Xia, D. Lo, Q. He, X. Wang, and J. Grundy, “Context-aware retrieval-based deep commit message generation,” ACM Transactions on Software Engineering and Methodology , vol. 30, no. 4, pp. 1–30, 2021
2021
-
[13]
Fira: fine-grained graph-based code change representation for automated commit message generation,
J. Dong, Y . Lou, Q. Zhu, Z. Sun, Z. Li, W. Zhang, and D. Hao, “Fira: fine-grained graph-based code change representation for automated commit message generation,” in Proceedings of the 44th International Conference on Software Engineering (ICSE). IEEE, 2022, pp. 970–981
2022
-
[14]
Cc2vec: Distributed representations of code changes,
T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: Distributed representations of code changes,” in Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE). ACM, 2020, pp. 518–529
2020
-
[15]
Bleu: a method for automatic evaluation of machine translation,
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). ACM, 2002, pp. 311–318
2002
-
[16]
Rouge: A package for automatic evaluation of summaries,
C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text Summarization Branches Out , 2004, pp. 74–81
2004
-
[17]
Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,
S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization . ACM, 2005, pp. 65–72
2005
-
[18]
Cider: Consensus- based image description evaluation,
R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2015, pp. 4566–4575
2015
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.