REVIEW 3 major objections 4 minor 22 references
AI-Powered Commit Explorer (APCE)
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read APCE is a web tool that generates commit messages with a two-agent LLM pipeline and collects automated and human evaluations in one place.
desk verdict Useful tool scaffolding for LLM-based commit message research, but the default prompts contradict the 'what and why' claim and the paper never shows the tool actually works. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the two-agent commit generation pipeline. The commit generation agent answers a configurable prompt that includes placeholders for the diff, pull-request title, issue report, commit type, and original message; the refinement agent receives the candidate and a hard-constraints prompt, replying either with the identical message or a corrected version and nothing else. APCE then applies a fixed decision rule: if one candidate is invalid, keep the other; if both are valid, reject any over 72 characters; if both are under 72 characters, keep the longer one. The evaluation module is the second load-bearing mechanism: it stores prompts and approaches in a research view, computes standard summarization metrics, and logs per-submission fields including file-level change data, ratings, and rationale.
What would settle it
Compare APCE's default two-agent output against single-agent output on a sample of commits, using human raters blind to condition: if messages produced with the refinement agent and the 72-character/longer-message heuristics are not rated higher for accuracy, integrity, readability, applicability, and completeness than messages produced without refinement, then the paper's implied benefit of the refinement pipeline is unsupported. The tool already stores whether refinement was used, so this experiment can be run inside APCE itself.
Extended reading notes
Core claim
The central claim is that a single web-based tool can carry a researcher from a GitHub repository to a finished evaluation of LLM-generated commit messages. On the generation side, APCE configures any number of LLM-based approaches, each with its own prompt, and runs a two-agent process: a generation agent produces a message and an optional refinement agent, governed by a strict prompt that demands only the exact message or a corrected one, rewrites it when it violates criteria such as the 72-character limit or imperative mood. Deterministic heuristics then pick between the two candidates. On the evaluation side, APCE computes BLEU, ROUGE-L, and METEOR against the original message and collects human ratings on five criteria (accuracy, integrity, readability, applicability, completeness) with a rationale, storing all prompts, submissions, and metadata for later analysis. The paper argues that this integrated design removes much of the infrastructure burden from empirical studies of commit message generation.
Load-bearing premise
The load-bearing premise is that the two-agent generation pipeline with its selection heuristics actually produces commit messages good enough for APCE's stated purposes; the paper describes this design but gives no experimental evidence that the refinement step improves messages or that the length-based tie-breaking is beneficial.
Editorial extensions
If this is right
- A researcher can test a new LLM-based commit generation approach by adding a prompt in APCE and immediately collecting BLEU, ROUGE-L, METEOR, and human ratings for the same commits.
- Because prompts, refinement settings, and submissions are stored together, studies can compare approaches side by side and can separate the effect of the generation prompt from the effect of the refinement agent.
- Developers who skip the evaluation module can still use APCE as a commit message assistant that pulls real commit context from GitHub and returns a formatted, imperative message.
- The tool's integration with OpenRouter means the underlying LLM can be swapped without changing the evaluation pipeline, so results are not tied to a single model.
- Human ratings are collected with a rationale field and file-level metadata, which supports qualitative analysis of why certain messages are preferred, beyond aggregate metric scores.
Reading between the lines
- Since the paper does not evaluate the refinement step, an obvious ablation is to compare APCE output with refinement on versus off; the stored 'refinement used' flag makes this a within-tool experiment.
- The 72-character rule and 'prefer the longer' tie-break are untested editorial choices; they could be benchmarked directly by varying them and measuring human preference, which the tool's configuration supports.
- The same two-agent pattern could generalize to other code-related text generation tasks, such as pull-request descriptions or code review comments, by swapping the criteria prompt.
- A bulk dataset import, listed as future work, would let APCE serve as a batch benchmark for LLM commit generation rather than only an interactive tool, potentially aligning it with existing large-scale commit message datasets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents APCE, a web-based tool for generating and evaluating LLM-produced commit messages. The tool is organized into a commit generation module, which uses a two-agent design (a generation agent and a refinement agent) with configurable prompts, and an evaluation module that computes BLEU, METEOR, and ROUGE-L while also collecting human ratings. The authors claim that APCE enables seamless integration with GitHub, produces high-quality commit messages that describe both what changed and why, and streamlines empirical comparison of LLM-based commit message generation approaches. The paper includes architecture descriptions, prompt figures, GUI screenshots, a demo video link, and a public GitHub repository, but it reports no quantitative or qualitative evaluation of the tool or of the generated commit messages.
Significance. If the tool works as described and the claims were supported, APCE would be a useful infrastructure artifact for researchers studying LLM-based commit message generation: it centralizes prompt configuration, automated metric computation, and human feedback collection in one interface. The manuscript has concrete strengths: the source code and replication package are made available, the architecture is clearly described, and the evaluation module addresses an acknowledged need for standardized infrastructure in this area. However, the paper's central assertions about message quality and the benefits of the refinement agent are currently unsupported, and the default prompt configuration appears to conflict with the stated goal of generating messages that explain why changes were made. The novelty is incremental, but the tool could be a valid contribution if the claims are trimmed to what the tool demonstrably does and if at least a small empirical demonstration is added.
major comments (3)
- [Section I and Section III.A (Figures 3 and 4)] The abstract and Section I claim that APCE enables 'high-quality commit messages that describe both the what and the why of code changes.' This claim is contradicted by the default prompts shown in Figures 3 and 4. The generation prompt requires 'Must clearly describe what changed and why' but also 'Must not include any punctuation' and 'no explanations, no extra words'; the refinement prompt in Figure 3 explicitly lists 'Must not include explanations or reasoning' and enforces a 72-character limit. Since the 'why' of a change is normally expressed through explanation or reasoning, the default configuration is set up to remove exactly the rationale the paper claims to provide. Please either revise the default prompts so that the 'why' can actually be expressed (e.g., by allowing brief rationale within the length limit) or temper the claims to describe the tool as generating concise imperative summaries without a why component.
- [Section III.A and Abstract] The abstract states that APCE 'provides an additional evaluation prompt that can further enhance the commit message provided by LLMs,' and Section III.A describes the refinement agent as assessing and correcting the generation agent's output. However, no experiment or demonstration is reported that measures whether the refinement prompt actually improves commit messages. The selection heuristics are also unvalidated: if both messages are valid and at most 72 characters, APCE chooses the longer one; if one exceeds 72 characters, it chooses the other. These rules are presented without any justification beyond readability, and no data show that they improve quality. Please add at least a small pilot study comparing messages generated with and without the refinement agent, and provide evidence for the length-based tie-breaking choices, or rephrase the enhancement claim as a hypothesis rather than a demonstrated feature.
- [Section V and overall evaluation] The paper contains no empirical evaluation of the end-to-end tool. The only demonstration artifact is a demo video, and Section V admits that 'performance drops as the commit diff gets larger, which worsens the commit message quality.' Yet the paper's central contribution is presented as the generation of high-quality commit messages and the facilitation of empirical studies. Without any actual generated-message examples, metric scores, user ratings, or a small case study, the manuscript does not support its qualitative claims about message quality. I recommend adding a minimal demonstration on a few real commits (including generated messages, automated metric values, and optionally a small number of human raters) so that readers can see what the tool produces, or alternatively narrowing the claims to describe only the tool's functionality and architecture.
minor comments (4)
- [Section I] In the sentence 'providing them information regarding what? and why? a set of code changes where made,' the word 'where' should be 'were.'
- [Section III.A] The paragraph describing the refinement process says 'the refinement agent will asses the potential commit message'; this should be 'assess'.
- [Section III.A] The phrase 'seamless integration into GitHub repositories' (Section I) is stronger than what the described implementation supports: the user must supply a GitHub token, select a repository, and choose a commit manually. Consider softening this wording.
- [Section III.C] In the evaluation module, the human-rating criterion 'integrity (Does it explain what changed and why?)' is in tension with the default generation/refinement prompts that forbid explanations and reasoning. This mismatch should be acknowledged, as it will affect how raters interpret the criterion when the default prompt is used.
Circularity Check
No circularity: APCE is a tool paper with no derived quantity, fitted prediction, or load-bearing self-citation chain.
full rationale
The paper introduces a software tool and does not derive or predict any scientific quantity, so there is no derivation chain that could reduce to its own inputs. The abstract's claim that the refinement prompt 'can further enhance' commit messages is an unsupported assertion, not a result obtained by fitting a parameter to data and then re-predicting that data; the claim is simply untested, which is a correctness/evidence concern rather than circularity. The default prompt's prohibition on explanations or reasoning alongside the claimed goal of describing 'why' a change was made is an internal inconsistency, not a circular step. The only self-citations are to prior LLM-related work by one co-author (references [11] and [14]) in a list of related work; they are contextual and not load-bearing. Section V's admitted limitation that larger diffs worsen commit message quality is an empirical limitation and does not reveal that any conclusion was assumed into existence. There is no fitted input renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled in via citation. The evaluation module computes standard metrics and collects human ratings against human-written messages, which is a conventional external benchmark, not a circular validation. Score 0 reflects the absence of any circular step.
Assumptions & free parameters
free parameters (2)
- commit message length threshold (72 characters) =
72
- fallback length limit (200 characters) =
200
assumptions (3)
- domain assumption LLM APIs (OpenRouter, DeepSeek) are available and return responses that can be parsed as commit messages.
- domain assumption BLEU, ROUGE-L, and METEOR are appropriate metrics for assessing commit message quality.
- ad hoc to paper A second agent with a refinement prompt can improve the initial LLM-generated commit message.
Cite this review
Pith. "Pith review of AI-Powered Commit Explorer (APCE)." pith.science (2026). https://pith.science/paper/YCV5XR3Y
@misc{pith2026250716063,
author = {Pith},
title = {Pith review of: AI-Powered Commit Explorer (APCE)},
year = {2026},
howpublished = {\url{https://pith.science/paper/YCV5XR3Y}},
note = {Machine review of arXiv:2507.16063}
}
read the original abstract
Commit messages in a version control system provide valuable information for developers regarding code changes in software systems. Commit messages can be the only source of information left for future developers describing what was changed and why. However, writing high-quality commit messages is often neglected in practice. Large Language Model (LLM) generated commit messages have emerged as a way to mitigate this issue. We introduce the AI-Powered Commit Explorer (APCE), a tool to support developers and researchers in the use and study of LLM-generated commit messages. APCE gives researchers the option to store different prompts for LLMs and provides an additional evaluation prompt that can further enhance the commit message provided by LLMs. APCE also provides researchers with a straightforward mechanism for automated and human evaluation of LLM-generated messages. Demo link https://youtu.be/zYrJ9s6sZvo
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
How to write a git commit message,
C. Beams, “How to write a git commit message,” 2014, accessed: 2025- 04-11. [Online]. Available: https://chris.beams.io/posts/git-commit/
work page 2014
-
[2]
On automatically generating commit messages via summarization of source code changes,
L. F. Cort ´es-Coy, M. Linares-V ´asquez, J. Aponte, and D. Poshyvanyk, “On automatically generating commit messages via summarization of source code changes,” in Proceedings of the 14th IEEE Interna- tional Working Conference on Source Code Analysis and Manipula- tion(SCAM’14), 2014, pp. 275–284
work page 2014
-
[3]
Fira: fine-grained graph-based code change representation for automated commit message generation,
J. Dong, Y . Lou, Q. Zhu, Z. Sun, Z. Li, W. Zhang, and D. Hao, “Fira: fine-grained graph-based code change representation for automated commit message generation,” in Proceedings of the 44th IEEE/ACM International Conference on Software Engineering (ICSE’22) , 2022, pp. 970–981
work page 2022
-
[4]
Boa: A language and infrastructure for analyzing ultra-large-scale software repositories,
R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen, “Boa: A language and infrastructure for analyzing ultra-large-scale software repositories,” in Proccedings of the 35th IEEE International Conference on Software Engineering (ICSE’13). IEEE, 2013, pp. 422–431
work page 2013
-
[5]
J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 5, May 2025
work page 2025
-
[6]
Commit message matters: Investigating impact and evolution of commit message quality,
J. Li and I. Ahmed, “Commit message matters: Investigating impact and evolution of commit message quality,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE’23). IEEE, 2023, pp. 806–817
work page 2023
-
[7]
Generating commit messages from diffs using pointer-generator network,
Q. Liu, Z. Liu, H. Zhu, H. Fan, B. Du, and Y . Qian, “Generating commit messages from diffs using pointer-generator network,” in Proceedings of the 16th IEEE/ACM International Conference on Mining Software Repositories (MSR’19). IEEE, 2019, pp. 299–309
work page 2019
-
[8]
Atom: Commit message generation based on abstract syntax tree and hybrid ranking,
S. Liu, C. Gao, S. Chen, L. Y . Nie, and Y . Liu, “Atom: Commit message generation based on abstract syntax tree and hybrid ranking,” IEEE Transactions on Software Engineering , vol. 48, no. 5, pp. 1800–1817, 2020
work page 2020
Show all 22 references
-
[9]
Neural machine translation-based commit message generation: How far are we?
Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural machine translation-based commit message generation: How far are we?” in Proceedings of the 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE’18). ACM, 2018, pp. 373–384
2018
-
[10]
CCRep: Learning code change representations via pre-trained code model and query back,
Z. Liu, Z. Tang, X. Xia, and X. Yang, “CCRep: Learning code change representations via pre-trained code model and query back,” in Pro- ceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE’23). IEEE, 2023, pp. 17–29
2023
-
[11]
Chatting with ai: Deciphering developer conversations with chatgpt,
S. Mohamed, A. Parvin, and E. Parra, “Chatting with ai: Deciphering developer conversations with chatgpt,” in Proceedings of the 21st IEEE/ACM International Conference on Mining Software Repositories (MSR’24). Lisbon, Portugal: ACM, 2024, p. 187–191
2024
-
[12]
Reassessing automatic evaluation metrics for code summarization tasks,
D. Roy, S. Fakhoury, and V . Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC- FSE’21), 2...
2021
-
[13]
Breaking the silence: the threats of using llms in software engineering,
J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using llms in software engineering,” in Proceedings of the 44th ACM/IEEE International Conference on Software Engineering: New Ideas and Emerging Results (ICSE’24) , 2024, pp. 102–106
2024
-
[14]
Creating uml class diagrams with general-purpose llms,
M. Shehata, B. Lepore, H. Cummings, and E. Parra, “Creating uml class diagrams with general-purpose llms,” in Proceedings of the 20th IEEE Working Conference on Software Visualization (VISSOFT’24) , Flgstaff, Arizona, USA, 2024, pp. 157–158
2024
-
[15]
A large-scale empirical study of commit message genera- tion: models, datasets and evaluation,
W. Tao, Y . Wang, E. Shi, L. Du, S. Han, H. Zhang, D. Zhang, and W. Zhang, “A large-scale empirical study of commit message genera- tion: models, datasets and evaluation,” Empirical Software Engineering, vol. 27, no. 7, p. 198, 2022
2022
-
[16]
What makes a good commit message?
Y . Tian, Y . Zhang, K.-J. Stol, L. Jiang, and H. Liu, “What makes a good commit message?” in Proceedings of the IEEE/ACM 44th International Conference on Software Engineering (ICSE’22). New York, NY , USA: Association for Computing Machinery, 2022, p. 2389–2401
2022
-
[17]
Context-aware retrieval-based deep commit message generation,
H. Wang, X. Xia, D. Lo, Q. He, X. Wang, and J. Grundy, “Context-aware retrieval-based deep commit message generation,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 4, pp. 1– 30, 2021
2021
-
[18]
A prompt pattern catalog to enhance prompt engineering with chatgpt,
J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382, 2023
2023 arXiv
-
[19]
Commit message generation for source code changes,
S. Xu, Y . Yao, F. Xu, T. Gu, H. Tong, and J. Lu, “Commit message generation for source code changes,” in Proceedings of the 18th Inter- national Joint Conferences on Artificial Intelligence (IJCAI’19) , 2019, pp. 3975–3981
2019
-
[20]
Automated commit message generation with large language models: An empirical study and beyond,
P. Xue, L. Wu, Z. Yu, Z. Jin, Z. Yang, X. Li, Z. Yang, and Y . Tan, “Automated commit message generation with large language models: An empirical study and beyond,” IEEE Transactions on Software Engi- neering, vol. 50, no. 12, pp. 3208–3224, 2024
2024
-
[21]
Using large language models for commit message generation: A preliminary study,
L. Zhang, J. Zhao, C. Wang, and P. Liang, “Using large language models for commit message generation: A preliminary study,” in Proceedings of the 31st IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER’24) , 2024, pp. 126–130
2024
-
[22]
Automatic commit message generation: A critical review and directions for future work,
Y . Zhang, Z. Qiu, K.-J. Stol, W. Zhu, J. Zhu, Y . Tian, and H. Liu, “Automatic commit message generation: A critical review and directions for future work,” IEEE Transactions on Software Engineering , vol. 50, no. 4, pp. 816–835, 2024
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.