{"id":"0b343394-4386-4aaa-a509-fc63485ab162","arxiv_id":"2507.17690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Retrieving repository code that calls the changed functions and adding it to the model input makes generated commit messages more complete in human ratings, while automatic similarity scores show no consistent gain.","lead":"A new method, C3Gen, retrieves repository code snippets that use the functions changed in a commit and feeds them to large language models to write better commit messages. It also argues that standard similarity metrics are unreliable for judging generated text, which matters beyond commit messages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Only positive support for C3Gen is Table IV's ~5% Completeness gain from two author-raters, with no blinding, no inter-rater reliability, and no significance test; until an external blind evaluation replicates this, the improvement claim lacks direct support.","rationale":"Reader's weakest_assumption matches my reading: the positive Completeness result is the load-bearing evidence, and it rests on two author-raters without controls. My stress-test adds the specific missing pieces: non-blind condition identification, absent inter-rater reliability, and the internally inconsistent use of 'statistically insignificant' versus 'significantly enhance' without any reported test. This does not change the verdict: the paper is a plausible preliminary study, but the condition should require external, blind, reliability-checked human evaluation before the improvement claim is accepted. A secondary risk I weighed—whether the CSG is built from the parent-commit snapshot or from a later repository state—is also worth checking in the replication package, but the text-evident human-evaluation issue is the more immediate load-bearing point.","tokens_in":14232,"tokens_out":10168,"duration_ms":111642,"concrete_test":"Run a blind external human evaluation on the same 370 sampled commits: recruit at least three developers who are not authors, present Naive, C3Gen, and Reference messages in randomized order with method labels removed, and have each rater score Clarity/Completeness/Correctness under the paper's rubric. Compute quadratic-weighted Cohen's kappa for inter-rater reliability and a paired bootstrap or Wilcoxon signed-rank test on the C3Gen-vs-Naive Completeness difference. If the difference is not significant at α=0.05 or kappa is below 0.4, the central improvement claim should be downgraded to an unverified hypothesis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VI-B and Table IV provide the sole consistent evidence for the central claim (abstract; Section VII). The reported Completeness advantages of C3Gen over Naive (+0.18, +0.07, +0.24, +0.19 on a 1–5 scale) are averages of two participants, both authors of the paper. Candidate order was randomized, but the evaluation was not blind: the raters know the method and can reasonably identify C3Gen outputs, which are distinguished by retrieved context; the scoring may reflect expectation rather than message quality. No inter-rater reliability statistic is reported; no per-rater scores, per-item distribution, or paired test statistic is given. The paper asserts in Section VI-A that objective differences are 'statistically insignificant' and in Key Finding 2 that C3Gen 'significantly enhance[s]' Completeness, but no test accompanies either assertion. Additionally, Table IV shows developer-written reference messages scored far below all LLM outputs (e.g., Completeness 2.69 vs. 3.51–4.50), which is consistent with a rubric or participant population that favors verbose conventional-style text; this makes the 'Completeness' construct as rated by the two authors an unvalidated proxy. If the Completeness effect is an artifact of rater expectation or wording style, the abstract's 'more comprehensive and informative' claim and Section VII's 'improves the performance of the CMG task' lose their only direct empirical support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes C3Gen, a retrieval-augmented framework for commit message generation that supplements a raw code diff with repository-level code snippets. C3Gen constructs Code Structure Graphs (CSGs) via tree-sitter, identifies modified functions/classes, and extracts the bodies of functions/classes that invoke or instantiate those entities, plus a 25-line window around global-scope invocations. The authors also introduce ApacheCM, a dataset of over 230,000 commits from 50 Apache projects with metadata fields, and evaluate C3Gen against a diff-only baseline (\"Naive\") across GPT-4o, GPT-4.1, DeepSeek V3, and DeepSeek R1. Objective metrics (BLEU, ROUGE-L, METEOR, CIDEr) show mixed and statistically insignificant differences, while a human evaluation by two of the authors reports consistent Completeness gains of about 5% for C3Gen. The paper's central claim is that adding contextual code improves the informativeness and practical value of generated commit messages, and that similarity-based objective metrics are unreliable.","tokens_in":14540,"tokens_out":3894,"duration_ms":40399,"significance":"If the central claim is accepted, the paper makes a useful contribution to commit message generation: it challenges the diff-only formulation, provides a context-enrichment mechanism, and releases a dataset with rich metadata and a replication package. The paper is honest about the mixed objective results and provides a thought-provoking analysis of why similarity-based metrics may be misleading for CMG. The main weakness is that the only consistent evidence for the claimed improvement is a two-author human evaluation with no significance testing, no inter-rater reliability, and no blinding. Because the positive result and the paper's critique of objective metrics both rest on that evaluation, the evidence base is currently too thin to fully support the abstract and Section VII claims. The dataset and framework are nevertheless valuable resources for the community if the evaluation concerns are addressed.","major_comments":[{"comment":"The central claim that C3Gen improves informativeness rests on the Completeness gains in Table IV, which are averages from two of the paper's authors with no inter-rater reliability statistic, no significance test, and no blinding of the method being rated. Key Finding 2 states that C3Gen 'significantly enhance[s]' Completeness, but no statistical test accompanies this statement. The authors should report per-rater and per-item scores, an agreement measure such as Cohen's kappa, and a paired significance test (e.g., Wilcoxon signed-rank across instances), or justify why these are unnecessary at this sample size.","section":"§VI-B, Table IV, Key Finding 2"},{"comment":"In Table IV, developer-written reference messages scored far below all LLM outputs on every dimension (e.g., Completeness 2.69 vs. 3.51–4.50). This pattern is more consistent with a rating rubric or rater population that rewards verbose, conventional phrasing than with an objective quality gap, and it conflicts with the use of those same references as ground truth for the objective metrics in Table III. The authors' conclusion that developer-written messages are of lower quality in terms of human preference is not supported without validating the rubric (e.g., external developer raters, or a pilot on typical real-world messages). Because this result is also used to justify the unreliability of objective metrics, the validity of the Completeness construct is load-bearing for the paper's central claim.","section":"§VI-B, Table IV"},{"comment":"Section III-C leaves key retrieval parameters unspecified: the number of snippets retrieved (the analysis in Figure 2 refers to 'ten retrieved code snippets' but Stage III does not define a retrieval count), and the 25-line context window is a heuristic with no sensitivity analysis. The framework is also compared only against a diff-only baseline, so the improvement cannot be attributed to the CSG-based selection rather than to simply having more code text. I ask for the retrieval count to be stated, a sensitivity check or ablation for the window size, and ideally a control condition that retrieves non-contextual code of similar size.","section":"§III-C, §VI-A"}],"minor_comments":[{"comment":"Table II reports an ApacheCM train set of 249,830 commits, while §IV-B states the dataset comprises 234,799 commits; please reconcile this discrepancy.","section":"Table II vs. §IV-B"},{"comment":"The paper contains several typographical errors: 'sylictic' in §VI-A, 'relavantCodeResult' in Figure 3, and 'cular' in Key Finding 2.","section":"§VI-A, Figure 3, Key Finding 2"},{"comment":"Figure 2 appears to be a low-resolution rendering with garbled characters and unreadable glyphs; a clean vector version is needed for review.","section":"Figure 2"},{"comment":"The dataset filtering criterion that commit messages between 5 and 50 words correspond to 'high-quality' messages is asserted without evidence; please cite or soften this claim and state it as a limitation.","section":"§IV-B, Table I"},{"comment":"In §VII the project is described as using 'advanced retrieval techniques,' but Stage III is a deterministic static-analysis procedure; a more modest description would be appropriate.","section":"§VII"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a software engineering venue, but the evidence for the central claim is thin. The two-author human evaluation is the main risk; because the authors self-evaluate their own method, the 'Completeness' gains in Table IV could reflect rater expectation rather than message quality. I recommend requesting an external, blinded evaluation or at least a much more detailed statistical treatment before publication. The mixed objective results are honestly reported, but they do not, by themselves, support the abstract's claim of 'more comprehensive and informative' messages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent preliminary study that extends retrieval-augmented commit message generation by retrieving code snippets through Code Structure Graphs instead of pulling similar historical commits, and it ships a new dataset, ApacheCM. The main claim, though, rests on a human evaluation by the two authors with no inter-rater reliability and no significance tests, and the paper overstates what that evidence supports.\n\nThe genuinely new piece is the retrieval mechanism. C3Gen parses the codebase with tree-sitter, builds CSGs, identifies functions/classes modified by the diff, then extracts enclosing function/class bodies or 25-line windows around call sites. That is a concrete, plausible way to give an LLM repository context that REACT's similar-commit retrieval does not. ApacheCM also looks useful: 234k commits from 50 Apache projects with SHA, timestamps, repo names, and filtering criteria, plus a replication package. I credit that.\n\nWhat the paper does well: it is honest about the objective metrics. Key Finding 1 says outright that similarity-based metrics do not consistently favor C3Gen, and the analysis of stylistic divergence in Figure 3 is sensible. The automatic metric results are mixed, and the authors do not try to hide it.\n\nThe soft spot is the human evaluation. Section VI-B reports averages from two participants, both authors, who rated 370 instances. Candidate order was randomized, but the evaluation is not blind: the raters know the method and can identify C3Gen outputs because those messages were generated with extra retrieved context. No per-rater scores, no inter-rater reliability, no paired significance test. Calling the Completeness gain 'significant' in Key Finding 2 is unsupported. It might well be real, but the evidence as presented is not strong enough for the abstract's claim that C3Gen generates 'more comprehensive and informative' messages.\n\nThere's a second, subtler issue. Table IV has the developer-written reference messages scoring well below all LLM outputs across all three dimensions (e.g., Completeness 2.69 versus 3.51-4.50). That pattern suggests the rubric or the raters favor verbose conventional-style text, which means the Completeness construct may not be measuring what a typical developer values. That would bias the comparison toward any method that produces longer, more detailed messages, which is exactly what C3Gen does.\n\nMissing too is a RAG baseline like REACT; the comparison is only diff-only versus C3Gen, so we cannot tell whether the improvement comes from retrieval generally or from this specific CSG method. Minor point: the methodology mentions ten retrieved snippets and a 25-line window without sensitivity analysis, but those are tuning choices, not fatal.\n\nWho this is for: people working on CMG or RAG for code. It deserves a serious referee. I would send it out, but I would expect the authors to add an external blind evaluation with significance tests and a REACT comparison before accepting the performance claim.","headline":"A plausible retrieval-augmented CMG method with a useful new dataset, but the headline improvement rests on a two-author human eval with no significance testing.","tokens_in":15054,"tokens_out":2147,"would_cite":true,"duration_ms":21327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a commit-message generator cannot work well from a raw diff alone, and that adding repository code that calls or instantiates the changed functions yields more informative messages.","keywords":["commit message generation","retrieval-augmented generation","code retrieval","code structure graph","human evaluation","large language models","similarity metrics","ApacheCM dataset"],"falsifier":"Have at least ten developers who did not write the paper rate the same 370 sampled generated messages on Completeness, Clarity, and Correctness while blind to which messages used retrieved context, and compare the Completeness difference; if the C3Gen advantage disappears or reverses, the central claim is refuted.","tokens_in":14006,"feed_emoji":"📝","tokens_out":12045,"duration_ms":114251,"temperature":0.7,"pith_summary":"This paper tries to establish that commit message generation improves when the model sees not just the code diff but also repository code that uses the changed code. The authors build C3Gen, a retrieval-augmented pipeline that finds code invoking or instantiating the modified functions and classes, plus a new dataset of 234,799 commits from 50 open-source repositories. Across four large language models, automatic similarity metrics showed no consistent or statistically reliable gain, but a human evaluation by the paper's authors found that C3Gen raised Completeness by about 5% while leaving Correctness unchanged and slightly lowering Clarity. If true, this means retrieval of repository context is a practical way to make generated commit messages more informative, and that reference-similarity metrics mislead when used alone.","feed_headline":"Repository context lifts commit-message completeness by ~5%","feed_subtitle":"Human raters scored LLM outputs higher on completeness when prompts also include code that uses the changed functions.","key_machinery":"The load-bearing object is the augmented Code Structure Graph. C3Gen first parses each repository source file into a graph whose nodes are source files, classes, and functions, and records every definition in a DefinitionIndex. A second parse of the diff builds a ModifiedEntityList of changed functions and classes, and a second graph pass locates places where those entities are called or instantiated elsewhere. The enclosing function or class bodies, or 25 lines around global-scope uses, become the relevant code context prepended to the diff.","core_discovery":"The central claim is that diff-only input is insufficient for commit message generation. C3Gen retrieves the bodies of functions, or surrounding lines, where code in the repository invokes or instantiates the entities the diff modifies, and adds that context to the prompt. In human evaluation of 370 commits, this consistently improved Completeness by around 5% on a 1-5 scale across all four tested models, kept Correctness essentially the same, and produced a slight Clarity decrease. The same additions produced mixed, statistically insignificant changes on BLEU, ROUGE-L, METEOR, and CIDEr, leading the paper to argue that similarity-based metrics underestimate the practical benefit of repository context.","pith_inferences":["A direct extension would be to repeat the human evaluation with developers who did not author the system and who are blind to which messages used retrieved context; if the Completeness advantage reproduces, it would strongly confirm that similarity metrics undercount retrieval benefits.","The program-structure retrieval idea could transfer to other code summarization tasks, such as release notes or code-review comments, where a diff alone also lacks the calling context.","The paper's own proposed next step implies a testable trade-off: filtering retrieved snippets to those tied to the central modified function should recover the small Clarity loss while keeping the Completeness gain, a hypothesis that could be measured on the same dataset."],"forward_implications":["Across all four LLMs, adding retrieved repository context produced a consistent human-rated Completeness gain of roughly 5% on the 1-5 scale.","Clarity dropped slightly for every model, so the added information carries a small cognitive-load cost.","Correctness stayed about the same, indicating that the extra context did not cause more hallucinated or inaccurate statements about the change.","Reference-based similarity metrics showed no consistent improvement, so evaluating such a system only with BLEU or similar metrics would miss the measured human-perceived benefit.","Developer-written reference messages were rated lower than LLM-generated messages on all three subjective dimensions, casting doubt on the assumption that the human-written message is the best target."],"supporting_citations":[{"why":"Reports that about 44% of commit messages fail quality expectations, motivating the need for generated messages and human judgment beyond similarity.","marker":"[1]"},{"why":"Shows how LLMs have been applied to commit message generation and documents dataset-quality flaws that the new dataset is built to avoid.","marker":"[3]"},{"why":"Describes the closest prior retrieval-augmented commit message generation, which retrieves similar historical commits; C3Gen distinguishes itself by retrieving repository code snippets instead.","marker":"[10]"},{"why":"Provides the prior CMG evaluation setup whose objective metrics the paper adopts and then questions.","marker":"[11]"},{"why":"Defines BLEU, one of the four similarity metrics whose inability to capture semantic gains C3Gen's human evaluation is meant to expose.","marker":"[15]"},{"why":"Defines ROUGE-L, the longest-common-subsequence metric used as an objective similarity measure.","marker":"[16]"},{"why":"Defines METEOR, the metric that sometimes rises with added context even when BLEU falls.","marker":"[17]"},{"why":"Defines CIDEr, the fourth objective metric used in the comparison tables.","marker":"[18]"}],"fun_headline_variants":["Diff-only commit messages miss context; C3Gen fills it","C3Gen retrieves repo context to boost commit message quality","Commit messages improve 5% with code context retrieval","Repository context beats diff-only for commit messages","C3Gen: Adding repo context lifts commit message completeness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed Completeness improvement rests on the assumption that the two authors who rated the messages were unbiased judges of their own system's output, since no independent raters or inter-rater agreement check support the scores.","fun_headline_variants_meta":{"raw":{"variants":["Diff-only commit messages miss context; C3Gen fills it","C3Gen retrieves repo context to boost commit message quality","Commit messages improve 5% with code context retrieval","Repository context beats diff-only for commit messages","C3Gen: Adding repo context lifts commit message completeness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1478,"prompt_tokens":889,"completion_tokens":589,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":505,"tokens_out":589,"duration_ms":5847,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:18:07.137192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have at least ten developers who did not write the paper rate the same 370 sampled generated messages on Completeness, Clarity, and Correctness while blind to which messages used retrieved context, and compare the Completeness difference; if the C3Gen advantage disappears or reverses, the central claim is refuted.","supporting_citations":[{"cited_title":"What makes a good commit message?","cited_arxiv_id":null,"evidence_quote":"Reports that about 44% of commit messages fail quality expectations, motivating the need for generated messages and human judgment beyond similarity."},{"cited_title":"Using large language models for commit message generation: A preliminary study,","cited_arxiv_id":null,"evidence_quote":"Shows how LLMs have been applied to commit message generation and documents dataset-quality flaws that the new dataset is built to avoid."},{"cited_title":"Commit message generation for source code changes,","cited_arxiv_id":null,"evidence_quote":"Provides the prior CMG evaluation setup whose objective metrics the paper adopts and then questions."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation,","cited_arxiv_id":null,"evidence_quote":"Defines BLEU, one of the four similarity metrics whose inability to capture semantic gains C3Gen's human evaluation is meant to expose."},{"cited_title":"Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,","cited_arxiv_id":null,"evidence_quote":"Defines METEOR, the metric that sometimes rises with added context even when BLEU falls."},{"cited_title":"Cider: Consensus- based image description evaluation,","cited_arxiv_id":null,"evidence_quote":"Defines CIDEr, the fourth objective metric used in the comparison tables."}],"review_version":1}