Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Contextual Code Retrieval for Commit Message Generation: A Preliminary Study

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The paper argues that a commit-message generator cannot work well from a raw diff alone, and that adding repository code that calls or instantiates the changed functions yields more informative messages.

desk verdict A plausible retrieval-augmented CMG method with a useful new dataset, but the headline improvement rests on a two-author human eval with no significance testing. read the letter →

arxiv 2507.17690 v1 pith:KQQWNG5H submitted 2025-07-23 cs.SE

classification cs.SE
keywords commitmessagegenerationretrieval-augmentedcoderetrievalstructuregraphhumanevaluationlargelanguagemodelssimilaritymetricsApacheCMdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that commit message generation improves when the model sees not just the code diff but also repository code that uses the changed code. The authors build C3Gen, a retrieval-augmented pipeline that finds code invoking or instantiating the modified functions and classes, plus a new dataset of 234,799 commits from 50 open-source repositories. Across four large language models, automatic similarity metrics showed no consistent or statistically reliable gain, but a human evaluation by the paper's authors found that C3Gen raised Completeness by about 5% while leaving Correctness unchanged and slightly lowering Clarity. If true, this means retrieval of repository context is a practical way to make generated commit messages more informative, and that reference-similarity metrics mislead when used alone.

What carries the argument

The load-bearing object is the augmented Code Structure Graph. C3Gen first parses each repository source file into a graph whose nodes are source files, classes, and functions, and records every definition in a DefinitionIndex. A second parse of the diff builds a ModifiedEntityList of changed functions and classes, and a second graph pass locates places where those entities are called or instantiated elsewhere. The enclosing function or class bodies, or 25 lines around global-scope uses, become the relevant code context prepended to the diff.

What would settle it

Have at least ten developers who did not write the paper rate the same 370 sampled generated messages on Completeness, Clarity, and Correctness while blind to which messages used retrieved context, and compare the Completeness difference; if the C3Gen advantage disappears or reverses, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that diff-only input is insufficient for commit message generation. C3Gen retrieves the bodies of functions, or surrounding lines, where code in the repository invokes or instantiates the entities the diff modifies, and adds that context to the prompt. In human evaluation of 370 commits, this consistently improved Completeness by around 5% on a 1-5 scale across all four tested models, kept Correctness essentially the same, and produced a slight Clarity decrease. The same additions produced mixed, statistically insignificant changes on BLEU, ROUGE-L, METEOR, and CIDEr, leading the paper to argue that similarity-based metrics underestimate the practical benefit of repository context.

Load-bearing premise

The claimed Completeness improvement rests on the assumption that the two authors who rated the messages were unbiased judges of their own system's output, since no independent raters or inter-rater agreement check support the scores.

Editorial extensions

If this is right

  • Across all four LLMs, adding retrieved repository context produced a consistent human-rated Completeness gain of roughly 5% on the 1-5 scale.
  • Clarity dropped slightly for every model, so the added information carries a small cognitive-load cost.
  • Correctness stayed about the same, indicating that the extra context did not cause more hallucinated or inaccurate statements about the change.
  • Reference-based similarity metrics showed no consistent improvement, so evaluating such a system only with BLEU or similar metrics would miss the measured human-perceived benefit.
  • Developer-written reference messages were rated lower than LLM-generated messages on all three subjective dimensions, casting doubt on the assumption that the human-written message is the best target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would be to repeat the human evaluation with developers who did not author the system and who are blind to which messages used retrieved context; if the Completeness advantage reproduces, it would strongly confirm that similarity metrics undercount retrieval benefits.
  • The program-structure retrieval idea could transfer to other code summarization tasks, such as release notes or code-review comments, where a diff alone also lacks the calling context.
  • The paper's own proposed next step implies a testable trade-off: filtering retrieved snippets to those tied to the central modified function should recover the small Clarity loss while keeping the Completeness gain, a hypothesis that could be measured on the same dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes C3Gen, a retrieval-augmented framework for commit message generation that supplements a raw code diff with repository-level code snippets. C3Gen constructs Code Structure Graphs (CSGs) via tree-sitter, identifies modified functions/classes, and extracts the bodies of functions/classes that invoke or instantiate those entities, plus a 25-line window around global-scope invocations. The authors also introduce ApacheCM, a dataset of over 230,000 commits from 50 Apache projects with metadata fields, and evaluate C3Gen against a diff-only baseline ("Naive") across GPT-4o, GPT-4.1, DeepSeek V3, and DeepSeek R1. Objective metrics (BLEU, ROUGE-L, METEOR, CIDEr) show mixed and statistically insignificant differences, while a human evaluation by two of the authors reports consistent Completeness gains of about 5% for C3Gen. The paper's central claim is that adding contextual code improves the informativeness and practical value of generated commit messages, and that similarity-based objective metrics are unreliable.

Significance. If the central claim is accepted, the paper makes a useful contribution to commit message generation: it challenges the diff-only formulation, provides a context-enrichment mechanism, and releases a dataset with rich metadata and a replication package. The paper is honest about the mixed objective results and provides a thought-provoking analysis of why similarity-based metrics may be misleading for CMG. The main weakness is that the only consistent evidence for the claimed improvement is a two-author human evaluation with no significance testing, no inter-rater reliability, and no blinding. Because the positive result and the paper's critique of objective metrics both rest on that evaluation, the evidence base is currently too thin to fully support the abstract and Section VII claims. The dataset and framework are nevertheless valuable resources for the community if the evaluation concerns are addressed.

major comments (3)
  1. [§VI-B, Table IV, Key Finding 2] The central claim that C3Gen improves informativeness rests on the Completeness gains in Table IV, which are averages from two of the paper's authors with no inter-rater reliability statistic, no significance test, and no blinding of the method being rated. Key Finding 2 states that C3Gen 'significantly enhance[s]' Completeness, but no statistical test accompanies this statement. The authors should report per-rater and per-item scores, an agreement measure such as Cohen's kappa, and a paired significance test (e.g., Wilcoxon signed-rank across instances), or justify why these are unnecessary at this sample size.
  2. [§VI-B, Table IV] In Table IV, developer-written reference messages scored far below all LLM outputs on every dimension (e.g., Completeness 2.69 vs. 3.51–4.50). This pattern is more consistent with a rating rubric or rater population that rewards verbose, conventional phrasing than with an objective quality gap, and it conflicts with the use of those same references as ground truth for the objective metrics in Table III. The authors' conclusion that developer-written messages are of lower quality in terms of human preference is not supported without validating the rubric (e.g., external developer raters, or a pilot on typical real-world messages). Because this result is also used to justify the unreliability of objective metrics, the validity of the Completeness construct is load-bearing for the paper's central claim.
  3. [§III-C, §VI-A] Section III-C leaves key retrieval parameters unspecified: the number of snippets retrieved (the analysis in Figure 2 refers to 'ten retrieved code snippets' but Stage III does not define a retrieval count), and the 25-line context window is a heuristic with no sensitivity analysis. The framework is also compared only against a diff-only baseline, so the improvement cannot be attributed to the CSG-based selection rather than to simply having more code text. I ask for the retrieval count to be stated, a sensitivity check or ablation for the window size, and ideally a control condition that retrieves non-contextual code of similar size.
minor comments (5)
  1. [Table II vs. §IV-B] Table II reports an ApacheCM train set of 249,830 commits, while §IV-B states the dataset comprises 234,799 commits; please reconcile this discrepancy.
  2. [§VI-A, Figure 3, Key Finding 2] The paper contains several typographical errors: 'sylictic' in §VI-A, 'relavantCodeResult' in Figure 3, and 'cular' in Key Finding 2.
  3. [Figure 2] Figure 2 appears to be a low-resolution rendering with garbled characters and unreadable glyphs; a clean vector version is needed for review.
  4. [§IV-B, Table I] The dataset filtering criterion that commit messages between 5 and 50 words correspond to 'high-quality' messages is asserted without evidence; please cite or soften this claim and state it as a limitation.
  5. [§VII] In §VII the project is described as using 'advanced retrieval techniques,' but Stage III is a deterministic static-analysis procedure; a more modest description would be appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: C3Gen is an empirical retrieval pipeline whose outputs are not defined in terms of its inputs; the author self-rating issue is a validity threat, not a circular derivation.

full rationale

The paper contains no formal derivation chain in which an output is defined in terms of an input or in which a fitted parameter is renamed as a prediction. C3Gen is described as a static-analysis retrieval pipeline: it builds Code Structure Graphs, augments them with diff information, and extracts enclosing function/class bodies or 25-line windows around call sites (Section III). The 25-line window is an explicitly stated heuristic, not a parameter fitted to the evaluation data. The objective metrics (BLEU, ROUGE-L, METEOR, CIDEr) and subjective metrics (Clarity, Completeness, Correctness) are defined independently of C3Gen's mechanism; Completeness is defined as how thoroughly the message captures changes in the code diff, not in terms of whether C3Gen's retrieved snippets were used. The central empirical claim rests on Section VI-B's human evaluation, where two authors rated candidates. That is a methodological validity concern (lack of blinding, no inter-rater reliability, no significance testing), but it is not circularity: the scores are not constructed from the method's outputs or from the paper's own definitions in a way that forces the reported improvement. Self-citations [3] and [10] appear only in related-work positioning and do not supply the load-bearing argument for C3Gen's effectiveness. No uniqueness theorem, imported ansatz, or self-definitional equivalence is present. Accordingly, no specific circular step can be exhibited, and the correct finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The paper's central claim rests on several hand-chosen experimental settings rather than fitted parameters: the 25-line retrieval window, the number of retrieved snippets, and the dataset filtering thresholds. The most important assumptions are that static parsing captures meaningful use relationships and that the author-run human evaluation yields unbiased completeness judgments.

free parameters (4)
  • Retrieval context window = 25 lines before and after invocation
    Stage III (Section III-C) selects 25 lines around invocations in global scope as a heuristic for contextual relevance; no ablation justifies this width.
  • Number of retrieved code snippets = 10 (apparent, not explicitly stated in method)
    Figure 2 describes 'among the ten retrieved code snippets', but Stage III does not state how many snippets are selected or capped; the size of the context is a design choice that affects LLM input.
  • Human evaluation sample size = 370 instances
    Section VI-B samples 370 commits at a 95% confidence level and 0.05 margin of error, a statistical choice that determines the precision of the subjective results.
  • Dataset filtering thresholds = 5-50 words, <=300 changed lines, exclude bots/reverts/merges
    Table I sets commit message length and diff size bounds; these choices shape ApacheCM and may affect the conclusions.
assumptions (4)
  • domain assumption Tree-sitter parsing accurately identifies class/function definitions and invocation/instantiation relationships across the supported languages.
    Stage I and II (Sections III-A, III-B) rely on static parsing of Python, Java, JavaScript, and other languages without reporting parsing accuracy or failure rates.
  • ad hoc to paper Commit messages between 5 and 50 words and diffs with fewer than 300 changed lines correspond to high-quality, informative commits.
    Table I's filtering criteria are introduced for ApacheCM and are not derived from prior work or validated separately.
  • domain assumption The repository's main branch commits since 2015 are representative of real-world commit messages.
    Section IV-A selects top-50 Apache repos by stars and main-branch commits; this sample may not generalize to other ecosystems or branches.
  • domain assumption BLEU, ROUGE-L, METEOR, and CIDEr are reasonable heuristic proxies for commit message quality.
    Section V-C adopts these metrics from prior CMG work, and the paper later argues they are unreliable, which partially undermines the objective part of the evaluation.
invented entities (1)
  • Code Structure Graph (CSG) independent evidence
    purpose: A file/class/function graph augmented with diff information, used to locate callers and instantiations of modified entities.
    CSG is a new construction introduced in Section III-A; its behavior is observable through the released replication package, but no formal specification or independent validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contextual Code Retrieval for Commit Message Generation: A Preliminary Study." pith.science (2026). https://pith.science/paper/KQQWNG5H

@misc{pith2026250717690,
  author       = {Pith},
  title        = {Pith review of: Contextual Code Retrieval for Commit Message Generation: A Preliminary Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQQWNG5H}},
  note         = {Machine review of arXiv:2507.17690}
}
read the original abstract

A commit message describes the main code changes in a commit and plays a crucial role in software maintenance. Existing commit message generation (CMG) approaches typically frame it as a direct mapping which inputs a code diff and produces a brief descriptive sentence as output. However, we argue that relying solely on the code diff is insufficient, as raw code diff fails to capture the full context needed for generating high-quality and informative commit messages. In this paper, we propose a contextual code retrieval-based method called C3Gen to enhance CMG by retrieving commit-relevant code snippets from the repository and incorporating them into the model input to provide richer contextual information at the repository scope. In the experiments, we evaluated the effectiveness of C3Gen across various models using four objective and three subjective metrics. Meanwhile, we design and conduct a human evaluation to investigate how C3Gen-generated commit messages are perceived by human developers. The results show that by incorporating contextual code into the input, C3Gen enables models to effectively leverage additional information to generate more comprehensive and informative commit messages with greater practical value in real-world development scenarios. Further analysis underscores concerns about the reliability of similaritybased metrics and provides empirical insights for CMG.

Figures

Figures reproduced from arXiv: 2507.17690 by the authors.

Figure 1
Figure 1. Overview of the C3Gen Framework II. RELATED WORK Researchers have proposed various approaches for CMG, most of which either solely rely on raw code diffs as input or employ different generation mechanisms to produce commit messages. These methods can be broadly categorized into the following three groups: (1) Rule-based approaches [5]– [7] were used in CMG, where predefined rules or templates were utilized to constr… view at source ↗
Figure 2
Figure 2. Examples with Higher and Lower Scores from the key modifications in the diff tends to improve the quality of the generated message significantly. Another reason could be the stylistic divergence between ground truths and generated messages. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Two Examples of Stylistic Divergence [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

  1. [1]

    What makes a good commit message?

    Y . Tian, Y . Zhang, K.-J. Stol, L. Jiang, and H. Liu, “What makes a good commit message?” in Proceedings of the 44th International Conference on Software Engineering (ICSE) . ACM, 2022, pp. 2389–2401

  2. [2]

    Automatic commit message generation: A critical review and directions for future work,

    Y . Zhang, Z. Qiu, K.-J. Stol, W. Zhu, J. Zhu, Y . Tian, and H. Liu, “Automatic commit message generation: A critical review and directions for future work,” IEEE Transactions on Software Engineering , vol. 50, no. 4, pp. 816–835, 2024

  3. [3]

    Using large language models for commit message generation: A preliminary study,

    L. Zhang, J. Zhao, C. Wang, and P. Liang, “Using large language models for commit message generation: A preliminary study,” in Proceedings of the 31st IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2024, pp. 126–130

  4. [4]

    Replication package of the paper: Contextual code retrieval for commit message generation: A preliminary study,

    B. Xiong, L. Zhang, C. Wang, and P. Liang, “Replication package of the paper: Contextual code retrieval for commit message generation: A preliminary study,” May 2025. [Online]. Available: https://doi.org/10.5281/zenodo.15502317

  5. [5]

    Automatically documenting program changes,

    R. P. Buse and W. R. Weimer, “Automatically documenting program changes,” in Proceedings of the 25th IEEE/ACM International Confer- ence on Automated Software Engineering (ASE) . ACM, 2010, pp. 33–42

  6. [6]

    Changescribe: A tool for automatically generating commit messages,

    M. Linares-V ´asquez, L. F. Cort ´es-Coy, J. Aponte, and D. Poshyvanyk, “Changescribe: A tool for automatically generating commit messages,” in Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE) . IEEE, 2015, pp. 709–712

  7. [7]

    On automatic summarization of what and why information in source code changes,

    J. Shen, X. Sun, B. Li, H. Yang, and J. Hu, “On automatic summarization of what and why information in source code changes,” in Proceedings of the 40th IEEE Annual Computer Software and Applications Conference (COMPSAC). IEEE, 2016, pp. 103–112

  8. [8]

    Neural machine-translation-based commit message generation: how far are we?

    Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural machine-translation-based commit message generation: how far are we?” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE) . ACM, 2018, pp. 373–384

Show all 18 references
  1. [9]

    Automatically generating commit messages from diffs using neural machine translation,

    S. Jiang, A. Armaly, and C. McMillan, “Automatically generating commit messages from diffs using neural machine translation,” in Pro- ceedings of the 32nd ACM/IEEE International Conference on Automated Software Engineering (ASE) . IEEE, 2017, pp. 135–146

  2. [10]

    Rag-enhanced commit message generation,

    L. Zhang, H. Zhang, C. Wang, and P. Liang, “Rag-enhanced commit message generation,” arXiv preprint arXiv:2406.05514 , 2024

  3. [11]

    Commit message generation for source code changes,

    S. Xu, Y . Yao, F. Xu, T. Gu, H. Tong, and J. Lu, “Commit message generation for source code changes,” in Proceedings of the 28th Inter- national Joint Conference on Artificial Intelligence (IJCAI) . IJCAI, 2019, pp. 3975–3981

  4. [12]

    Context-aware retrieval-based deep commit message generation,

    H. Wang, X. Xia, D. Lo, Q. He, X. Wang, and J. Grundy, “Context-aware retrieval-based deep commit message generation,” ACM Transactions on Software Engineering and Methodology , vol. 30, no. 4, pp. 1–30, 2021

  5. [13]

    Fira: fine-grained graph-based code change representation for automated commit message generation,

    J. Dong, Y . Lou, Q. Zhu, Z. Sun, Z. Li, W. Zhang, and D. Hao, “Fira: fine-grained graph-based code change representation for automated commit message generation,” in Proceedings of the 44th International Conference on Software Engineering (ICSE). IEEE, 2022, pp. 970–981

  6. [14]

    Cc2vec: Distributed representations of code changes,

    T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: Distributed representations of code changes,” in Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE). ACM, 2020, pp. 518–529

  7. [15]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). ACM, 2002, pp. 311–318

  8. [16]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text Summarization Branches Out , 2004, pp. 74–81

  9. [17]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization . ACM, 2005, pp. 65–72

  10. [18]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2015, pp. 4566–4575

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.