{"id":"7811326d-6479-46b9-806c-cf30eb528a10","arxiv_id":"2507.16063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"APCE is a web tool that generates commit messages with configurable LLM prompts and collects automated and human evaluations of them.","lead":"A new web tool, APCE, generates commit messages from code changes using two AI agents and lets researchers compare those messages with automated metrics and human ratings. It gives software engineering researchers a ready-made platform for studying how well different LLM prompts write commit messages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"APCE's default prompts forbid explanations/reasoning, so the claimed 'what and why' commit messages cannot be produced; no evaluation supports the 'high-quality' claim.","rationale":"The reader's weakest assumption focuses on the two-agent design and selection heuristics lacking evaluation. My concern is more specific and more load-bearing: the default refinement prompt (Figure 3) explicitly forbids 'explanations or reasoning,' which is precisely the vehicle for conveying 'why.' The generation prompt (Figure 4) simultaneously demands 'what changed and why' while forbidding explanations and punctuation. This internal inconsistency means the tool as configured cannot satisfy its own headline claim, independent of whether the refinement step improves fluency or length. The paper also provides no empirical evidence for the 'high-quality' assertion, and Section V acknowledges quality degradation on larger diffs. The evaluation-module side of the tool is described clearly and is checkable from the promised code, so the paper is not beyond repair; adjusting the default prompts or narrowing the claim would suffice. Therefore the reader's CONDITIONAL verdict remains appropriate, though the condition should include demonstrating that the default configuration can produce messages containing the rationale for changes.","tokens_in":7089,"tokens_out":3540,"duration_ms":38676,"concrete_test":"Run APCE's default pipeline (generation agent + refinement agent, with DeepSeek via OpenRouter) on 50 commits sampled from a public repository with varied commit types. Have two independent raters judge, for each final message, whether it conveys the rationale/why of the change (binary or 5-point scale), and also rate overall quality. Compare messages generated with refinement enabled vs disabled. If refinement does not increase, or decreases, the proportion of messages judged to explain 'why,' the central 'what and why' claim is not met. Also inspect whether any final message contains explanation-like language, given the prompts forbid it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section I claim APCE enables 'high-quality commit messages that describe both the what and the why of code changes.' The default configuration contradicts this. The generation prompt (Figure 4) requires 'Must clearly describe what changed and why' but also 'Must not include any punctuation' and 'no explanations, no extra words'; the refinement prompt (Figure 3) explicitly lists 'Must not include explanations or reasoning' among its criteria and enforces a 72-character limit (Section III.A). Since 'why' is normally expressed through explanation or reasoning, the refinement step is configured to remove exactly the rationale the tool claims to provide. The selection heuristics in Section III.A (reject messages over 72 characters, prefer the longer of two valid messages) are not validated and may further favor terse, content-free outputs. No empirical evaluation is reported: the only artifact is a demo video, and Section V admits 'performance drops as the commit diff gets larger, which worsens the commit message quality.' Thus the central claim rests on an internally inconsistent prompt design and unsupported qualitative assertions, not on evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents APCE, a web-based tool for generating and evaluating LLM-produced commit messages. The tool is organized into a commit generation module, which uses a two-agent design (a generation agent and a refinement agent) with configurable prompts, and an evaluation module that computes BLEU, METEOR, and ROUGE-L while also collecting human ratings. The authors claim that APCE enables seamless integration with GitHub, produces high-quality commit messages that describe both what changed and why, and streamlines empirical comparison of LLM-based commit message generation approaches. The paper includes architecture descriptions, prompt figures, GUI screenshots, a demo video link, and a public GitHub repository, but it reports no quantitative or qualitative evaluation of the tool or of the generated commit messages.","tokens_in":7403,"tokens_out":3732,"duration_ms":41951,"significance":"If the tool works as described and the claims were supported, APCE would be a useful infrastructure artifact for researchers studying LLM-based commit message generation: it centralizes prompt configuration, automated metric computation, and human feedback collection in one interface. The manuscript has concrete strengths: the source code and replication package are made available, the architecture is clearly described, and the evaluation module addresses an acknowledged need for standardized infrastructure in this area. However, the paper's central assertions about message quality and the benefits of the refinement agent are currently unsupported, and the default prompt configuration appears to conflict with the stated goal of generating messages that explain why changes were made. The novelty is incremental, but the tool could be a valid contribution if the claims are trimmed to what the tool demonstrably does and if at least a small empirical demonstration is added.","major_comments":[{"comment":"The abstract and Section I claim that APCE enables 'high-quality commit messages that describe both the what and the why of code changes.' This claim is contradicted by the default prompts shown in Figures 3 and 4. The generation prompt requires 'Must clearly describe what changed and why' but also 'Must not include any punctuation' and 'no explanations, no extra words'; the refinement prompt in Figure 3 explicitly lists 'Must not include explanations or reasoning' and enforces a 72-character limit. Since the 'why' of a change is normally expressed through explanation or reasoning, the default configuration is set up to remove exactly the rationale the paper claims to provide. Please either revise the default prompts so that the 'why' can actually be expressed (e.g., by allowing brief rationale within the length limit) or temper the claims to describe the tool as generating concise imperative summaries without a why component.","section":"Section I and Section III.A (Figures 3 and 4)"},{"comment":"The abstract states that APCE 'provides an additional evaluation prompt that can further enhance the commit message provided by LLMs,' and Section III.A describes the refinement agent as assessing and correcting the generation agent's output. However, no experiment or demonstration is reported that measures whether the refinement prompt actually improves commit messages. The selection heuristics are also unvalidated: if both messages are valid and at most 72 characters, APCE chooses the longer one; if one exceeds 72 characters, it chooses the other. These rules are presented without any justification beyond readability, and no data show that they improve quality. Please add at least a small pilot study comparing messages generated with and without the refinement agent, and provide evidence for the length-based tie-breaking choices, or rephrase the enhancement claim as a hypothesis rather than a demonstrated feature.","section":"Section III.A and Abstract"},{"comment":"The paper contains no empirical evaluation of the end-to-end tool. The only demonstration artifact is a demo video, and Section V admits that 'performance drops as the commit diff gets larger, which worsens the commit message quality.' Yet the paper's central contribution is presented as the generation of high-quality commit messages and the facilitation of empirical studies. Without any actual generated-message examples, metric scores, user ratings, or a small case study, the manuscript does not support its qualitative claims about message quality. I recommend adding a minimal demonstration on a few real commits (including generated messages, automated metric values, and optionally a small number of human raters) so that readers can see what the tool produces, or alternatively narrowing the claims to describe only the tool's functionality and architecture.","section":"Section V and overall evaluation"}],"minor_comments":[{"comment":"In the sentence 'providing them information regarding what? and why? a set of code changes where made,' the word 'where' should be 'were.'","section":"Section I"},{"comment":"The paragraph describing the refinement process says 'the refinement agent will asses the potential commit message'; this should be 'assess'.","section":"Section III.A"},{"comment":"The phrase 'seamless integration into GitHub repositories' (Section I) is stronger than what the described implementation supports: the user must supply a GitHub token, select a repository, and choose a commit manually. Consider softening this wording.","section":"Section III.A"},{"comment":"In the evaluation module, the human-rating criterion 'integrity (Does it explain what changed and why?)' is in tension with the default generation/refinement prompts that forbid explanations and reasoning. This mismatch should be acknowledged, as it will affect how raters interpret the criterion when the default prompt is used.","section":"Section III.C"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as a tool demonstration rather than a research contribution with empirical findings. The main issue is that the stated claims exceed the evidence: no evaluation is included, and the default prompts appear to undermine the 'what and why' claim. These are fixable with a revised default prompt, relaxed claims, and a small case study. I see no citation-pattern concerns or novelty-disclosure issues; the arXiv posting predates any journal submission. The fit with a software engineering venue is reasonable if the tool is positioned as infrastructure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Martin—quick take on 2507.16063. It's a tool paper with real scaffolding: a web platform that ties together multi-agent LLM commit generation, configurable prompts, BLEU/ROUGE-L/METEOR, and human ratings in one place, with source and replication package provided. That is genuinely useful for the LLM-commit-message subfield. The architecture is clearly described and the code is checkable. Credit where due: the integration itself is the contribution, and it's new.\n\nThe soft spots are in the claims, not the architecture. The abstract and intro promise 'high-quality commit messages that describe both the what and why.' The default prompts make that impossible: the generation prompt says 'no explanations, no punctuation, no extra words' while asking for 'what changed and why'; the refinement prompt explicitly forbids 'explanations or reasoning.' You cannot express 'why' without explanation. The 72-character limit and the arbitrary tie-breaking heuristics (prefer longer if both valid) are unvalidated. The paper also asserts without evidence that DeepSeek is 'most accurate' and that the refinement prompt 'can further enhance' messages—both unsupported. And there's no end-to-end demonstration: no user study, no pilot, no sample outputs, just a demo video and an admitted quality drop on larger diffs.\n\nThese are fixable. A serious referee should ask for either a small pilot with real commits showing what the tool actually produces, or a rewrite of the claims to match the prompts. The tool itself deserves a look; the overclaims don't sink it. I'd send it to review, conditionally. If you work in this area, it's a reasonable cite for the infrastructure.\n\nRecommendation: accept with major revision or conditional accept, but definitely not desk reject.","headline":"Useful tool scaffolding for LLM-based commit message research, but the default prompts contradict the 'what and why' claim and the paper never shows the tool actually works.","tokens_in":7815,"tokens_out":2957,"would_cite":false,"duration_ms":29318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"APCE is a web tool that generates commit messages with a two-agent LLM pipeline and collects automated and human evaluations in one place.","keywords":["Large Language Models","GitHub","Automated Commit Messages","AI4SE","Code Summarization","multi-agent LLM systems","evaluation metrics"],"falsifier":"Compare APCE's default two-agent output against single-agent output on a sample of commits, using human raters blind to condition: if messages produced with the refinement agent and the 72-character/longer-message heuristics are not rated higher for accuracy, integrity, readability, applicability, and completeness than messages produced without refinement, then the paper's implied benefit of the refinement pipeline is unsupported. The tool already stores whether refinement was used, so this experiment can be run inside APCE itself.","tokens_in":6889,"feed_emoji":"🛠️","tokens_out":5050,"duration_ms":49475,"temperature":0.7,"pith_summary":"This paper presents APCE, a web-based tool for researchers and developers working with LLM-generated commit messages. APCE connects to GitHub, lets users pick any commit, and produces a commit message through a two-agent pipeline: a generation agent writes a message from a configurable prompt, and an optional refinement agent critiques and corrects it. The tool also bundles an evaluation module that computes BLEU, ROUGE-L, and METEOR against the original message and collects human ratings on five quality criteria. A sympathetic reading is that APCE lowers the infrastructure cost of studying LLM-based commit message generation: prompts, models, and evaluation are all managed in one place.","feed_headline":"Tool writes LLM commit messages and scores them in one place","feed_subtitle":"APCE pairs a two-agent generation pipeline with BLEU, ROUGE-L, METEOR, and human Likert ratings for commit-message research.","key_machinery":"The central mechanism is the two-agent commit generation pipeline. The commit generation agent answers a configurable prompt that includes placeholders for the diff, pull-request title, issue report, commit type, and original message; the refinement agent receives the candidate and a hard-constraints prompt, replying either with the identical message or a corrected version and nothing else. APCE then applies a fixed decision rule: if one candidate is invalid, keep the other; if both are valid, reject any over 72 characters; if both are under 72 characters, keep the longer one. The evaluation module is the second load-bearing mechanism: it stores prompts and approaches in a research view, computes standard summarization metrics, and logs per-submission fields including file-level change data, ratings, and rationale.","core_discovery":"The central claim is that a single web-based tool can carry a researcher from a GitHub repository to a finished evaluation of LLM-generated commit messages. On the generation side, APCE configures any number of LLM-based approaches, each with its own prompt, and runs a two-agent process: a generation agent produces a message and an optional refinement agent, governed by a strict prompt that demands only the exact message or a corrected one, rewrites it when it violates criteria such as the 72-character limit or imperative mood. Deterministic heuristics then pick between the two candidates. On the evaluation side, APCE computes BLEU, ROUGE-L, and METEOR against the original message and collects human ratings on five criteria (accuracy, integrity, readability, applicability, completeness) with a rationale, storing all prompts, submissions, and metadata for later analysis. The paper argues that this integrated design removes much of the infrastructure burden from empirical studies of commit message generation.","pith_inferences":["Since the paper does not evaluate the refinement step, an obvious ablation is to compare APCE output with refinement on versus off; the stored 'refinement used' flag makes this a within-tool experiment.","The 72-character rule and 'prefer the longer' tie-break are untested editorial choices; they could be benchmarked directly by varying them and measuring human preference, which the tool's configuration supports.","The same two-agent pattern could generalize to other code-related text generation tasks, such as pull-request descriptions or code review comments, by swapping the criteria prompt.","A bulk dataset import, listed as future work, would let APCE serve as a batch benchmark for LLM commit generation rather than only an interactive tool, potentially aligning it with existing large-scale commit message datasets."],"forward_implications":["A researcher can test a new LLM-based commit generation approach by adding a prompt in APCE and immediately collecting BLEU, ROUGE-L, METEOR, and human ratings for the same commits.","Because prompts, refinement settings, and submissions are stored together, studies can compare approaches side by side and can separate the effect of the generation prompt from the effect of the refinement agent.","Developers who skip the evaluation module can still use APCE as a commit message assistant that pulls real commit context from GitHub and returns a formatted, imperative message.","The tool's integration with OpenRouter means the underlying LLM can be swapped without changing the evaluation pipeline, so results are not tied to a single model.","Human ratings are collected with a rationale field and file-level metadata, which supports qualitative analysis of why certain messages are preferred, beyond aggregate metric scores."],"supporting_citations":[{"why":"Supplies the 72-character readability rule that APCE's selection heuristic is based on.","marker":"[1]"},{"why":"Provides the original task formulation and the source for the five human quality criteria used in the evaluation module.","marker":"[2]"},{"why":"Motivates the multi-agent LLM design that APCE's two-agent generation pipeline follows.","marker":"[5]"},{"why":"Establishes why commit messages are critical for long-term maintenance, motivating the tool's purpose.","marker":"[6]"},{"why":"Supports the need for human evaluation alongside automated summarization metrics in code summarization tasks.","marker":"[12]"},{"why":"Supplies the standard evaluation metrics (BLEU, ROUGE-L, METEOR) that APCE computes.","marker":"[15]"},{"why":"One of the preconfigured LLM-based commit generation approaches bundled with APCE, and a source for evaluation criteria.","marker":"[20]"},{"why":"The other preconfigured LLM-based approach, providing a concrete empirical setup that APCE streamlines.","marker":"[21]"},{"why":"Cites the limitations of automated metrics and the need for better evaluation infrastructure in commit message generation.","marker":"[22]"}],"fun_headline_variants":["Two-agent LLM commit writer with built-in scores","APCE: generate, refine, and score commit messages","One web tool for LLM commit generation and evaluation","Commit messages: LLM generation plus automated and human scoring","Two-pass LLM commit writer with BLEU, ROUGE, and human ratings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two-agent generation pipeline with its selection heuristics actually produces commit messages good enough for APCE's stated purposes; the paper describes this design but gives no experimental evidence that the refinement step improves messages or that the length-based tie-breaking is beneficial.","fun_headline_variants_meta":{"raw":{"variants":["Two-agent LLM commit writer with built-in scores","APCE: generate, refine, and score commit messages","One web tool for LLM commit generation and evaluation","Commit messages: LLM generation plus automated and human scoring","Two-pass LLM commit writer with BLEU, ROUGE, and human ratings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001293,"raw_usage":{"total_tokens":5247,"prompt_tokens":885,"completion_tokens":4362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":4276}},"tokens_in":501,"tokens_out":4362,"duration_ms":30289,"temperature":1.0,"reasoning_tokens":4276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:17:54.130238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare APCE's default two-agent output against single-agent output on a sample of commits, using human raters blind to condition: if messages produced with the refinement agent and the 72-character/longer-message heuristics are not rated higher for accuracy, integrity, readability, applicability, and completeness than messages produced without refinement, then the paper's implied benefit of the refinement pipeline is unsupported. The tool already stores whether refinement was used, so this experiment can be run inside APCE itself.","supporting_citations":[{"cited_title":"Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-agent LLM design that APCE's two-agent generation pipeline follows."},{"cited_title":"How to write a git commit message,","cited_arxiv_id":null,"evidence_quote":"Supplies the 72-character readability rule that APCE's selection heuristic is based on."},{"cited_title":"On automatically generating commit messages via summarization of source code changes,","cited_arxiv_id":null,"evidence_quote":"Provides the original task formulation and the source for the five human quality criteria used in the evaluation module."},{"cited_title":"Commit message matters: Investigating impact and evolution of commit message quality,","cited_arxiv_id":null,"evidence_quote":"Establishes why commit messages are critical for long-term maintenance, motivating the tool's purpose."},{"cited_title":"Reassessing automatic evaluation metrics for code summarization tasks,","cited_arxiv_id":null,"evidence_quote":"Supports the need for human evaluation alongside automated summarization metrics in code summarization tasks."},{"cited_title":"A large-scale empirical study of commit message genera- tion: models, datasets and evaluation,","cited_arxiv_id":null,"evidence_quote":"Supplies the standard evaluation metrics (BLEU, ROUGE-L, METEOR) that APCE computes."},{"cited_title":"Automated commit message generation with large language models: An empirical study and beyond,","cited_arxiv_id":null,"evidence_quote":"One of the preconfigured LLM-based commit generation approaches bundled with APCE, and a source for evaluation criteria."},{"cited_title":"Using large language models for commit message generation: A preliminary study,","cited_arxiv_id":null,"evidence_quote":"The other preconfigured LLM-based approach, providing a concrete empirical setup that APCE streamlines."},{"cited_title":"Automatic commit message generation: A critical review and directions for future work,","cited_arxiv_id":null,"evidence_quote":"Cites the limitations of automated metrics and the need for better evaluation infrastructure in commit message generation."}],"review_version":1}