{"id":"96f0ecd4-220c-448c-a5f2-3863c0d189fe","arxiv_id":"2507.10906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM evaluator using Chain-of-Thought and few-shot prompting correlates with human quality ratings at 0.65 to 0.78, beating BLEU, ROUGE, METEOR and semantic similarity metrics.","lead":"This paper tests whether large language models can grade the quality of commit messages, the short notes developers write when saving code changes. The best setup, GPT-4 with step-by-step reasoning and a few examples, matched human reviewers more closely than traditional text-similarity scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RQ3 fairness analysis does not test the confound that matters: GPT-4 evaluates its own generated commit messages, yet human raters and GPT-4 both see the same GPT-4-generated texts in RQ2, so the 0.65/0.78 correlations may partly reflect self-preference rather than general evaluation skill.","rationale":"The reader's weakest_assumption focuses on the reliability of the human ground truth (six volunteers, no external validation), which is a legitimate external-validity concern. My stress-test identifies a different internal-validity risk: the RQ1/RQ2 evaluation corpus contains GPT-4-generated messages, and GPT-4 is both the message generator and the evaluator in the headline configuration. Section 6.3's fairness analysis is misaligned with this risk because it uses paraphrases of human-authored messages rather than the actual generated messages. This is a concrete, testable confound that does not require the human annotations to be invalid. I agree with the reader's CONDITIONAL verdict but for a partially different reason, and I do not think the concern is fatal: the paper is a first study, the stability analyses (Sec. 6.1-6.2) are genuine evidence, and the correlation gap between GPT-4 (0.65 What / 0.78 Why) and METEOR (0.42 / 0.20) is large enough that even a substantially discounted GPT-4 advantage would likely persist. However, the exact magnitude of the near-human-level claim is vulnerable, and a simple per-source correlation breakdown would settle it. I recommend CONDITIONAL with the per-source analysis as a required condition for the headline claim.","tokens_in":18752,"tokens_out":1572,"duration_ms":15631,"concrete_test":"Recompute the Table 3 correlations separately for the three generation sources (GPT-4, Llama-3.3-70B, Qwen-2.5-14B), using the per-message human average and GPT-4 evaluator scores. If the Spearman correlation is materially higher on the GPT-4-generated subset than on the Llama and Qwen subsets, the self-preference confound lands and the headline comparison to METEOR must be re-argued on the non-GPT-4 subsets. If the correlations are statistically indistinguishable across sources (e.g., within a bootstrap confidence interval), the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that GPT-4's near-human correlations measure evaluation proficiency, not agreement with GPT-4's own stylistic or semantic preferences. In RQ2, the 400 evaluated messages are generated by GPT-4, Llama-3.3-70B, and Qwen-2.5-14B (Sec. 3.1.2), with GPT-4 generating roughly one-third. The high correlations of 0.65/0.78 are computed on exactly this mix. Section 6.3's fairness test only compares human-authored messages against GPT-4-paraphrased variants of those same human messages (Sec. 6.3.1), not against the actual GPT-4-generated messages used in RQ2, so a self-preference or style-matching effect on the RQ2 corpus is never isolated. If GPT-4 systematically favors messages that resemble its own output, the correlation would be inflated on the GPT-4-generated subset, and the METEOR comparison would inherit that inflation. The authors do flag limited fairness samples (Sec. 7.2), but the load-bearing issue is not sample size; it is that the experiment cannot separate self-preference from genuine proficiency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether large language models (LLMs) can serve as automatic evaluators of generated commit message quality. The authors construct a benchmark of 400 commits (200 Python, 200 Java) with 800 commit messages: 400 human-written and 400 generated by GPT-4, Llama-3.3-70B, and Qwen-2.5-14B. Six experienced developers rate all messages on What and Why dimensions using a 5-point Likert scale. In RQ1, the authors test four LLMs with various prompting strategies (zero-shot, chain-of-thought, few-shot, unified vs. separate scoring) on the 400 human-written messages and report that GPT-4 with CoT+few-shot and unified evaluation achieves Spearman correlations of 0.65 (What) and 0.78 (Why) with averaged human ratings, which they claim closely match human inter-rater correlations (0.61-0.78). In RQ2, this configuration is applied to the 400 generated messages and compared with traditional metrics (BLEU, ROUGE-L, METEOR, CIDEr) and semantic metrics (BERTScore, SBERT); GPT-4 substantially outperforms all of them (e.g., METEOR's best Spearman is 0.42 for What and 0.20 for Why). The authors also analyze reproducibility across temperatures, robustness to paraphrased variants, and fairness between human- and LLM-generated messages. They conclude that LLM-based evaluators can approach human-level proficiency while being far cheaper and faster than human evaluation.","tokens_in":19041,"tokens_out":6404,"duration_ms":67991,"significance":"If the central claim holds, this is a useful and timely contribution to a practical problem: commit message evaluation is currently dominated by expensive human annotation or unreliable reference-based metrics. The paper is among the first to systematically investigate LLM-based evaluation specifically for commit messages, and it includes a credible human annotation effort (six experienced developers, pilot calibration, shuffled evaluation) and a broad comparison across models and prompt strategies. Strong points include the reproducibility analysis (temperature sweep), the robustness check with manually verified paraphrases, and the thematic analysis explaining why reference-based metrics fail. The replication package and the use of multiple open-source as well as proprietary models are also positive. However, the headline 'near human-level' and 'significantly outperforms' claims rest on methodological choices that currently leave the results uncertain: in-sample prompt selection, a possible self-preference confound in RQ2, and a lack of statistical inference for the reported differences.","major_comments":[{"comment":"The optimal prompt configuration is selected on the same 400 human-written messages that are then used to report the headline correlations. The authors test 4 models, multiple prompt strategies, and two scoring modes, then select GPT-4 with CoT+Few-shot and unified evaluation as the best. The reported Spearman values of 0.65 and 0.78 are therefore in-sample estimates, not out-of-sample predictions, and are likely optimistically biased by the selection procedure. The claim that this configuration achieves 'human-level proficiency' is not supported by an unbiased estimate. Please use a held-out validation set (or cross-validation) for prompt selection and report the test-set correlations separately.","section":"Section 4 (RQ1), Table 2"},{"comment":"In RQ2, GPT-4 evaluates 400 generated messages, and according to Section 3.1.2, approximately one-third of those messages were generated by GPT-4 itself. The fairness analysis in Section 6.3 does not control for this self-evaluation confound: it only compares human-authored messages with GPT-4-paraphrased variants of those same human-authored messages (Section 6.3.1), not with the actual GPT-4-generated messages used in RQ2. A systematic self-preference or style-matching effect would inflate the GPT-4 correlations in Table 3 (0.65/0.78) and the gap over METEOR, and the current fairness test would not detect it. Please report the LLM-human correlations separately for messages generated by each source (GPT-4, Llama-3.3, Qwen-2.5), or construct a fairness test on matched human/LLM-generated pairs drawn from the actual RQ2 corpus.","section":"Section 5.2 (Table 3) and Section 6.3"},{"comment":"The human ratings are treated as the ground truth for message quality, but the six pairwise inter-rater Spearman correlations range from 0.61 to 0.78, which by common conventions is moderate rather than 'high' agreement. The paper's conclusion that GPT-4 is 'near human-level' is drawn by comparing GPT-4's correlation with the averaged six-rater scores to these pairwise inter-rater correlations; this comparison is not apples-to-apples, because an average of six raters is more reliable than any single rater. Moreover, the human ratings are not validated against any external standard, so the LLM-human correlations could reflect agreement with this particular group's preferences rather than objective message quality. Please provide additional reliability evidence (e.g., intra-rater consistency, item-level analyses, or a small expert-panel validation) and either benchmark the LLM against individual raters or use a reliability-corrected criterion when claiming parity with human evaluators.","section":"Section 3.2 and Figure 1"},{"comment":"The claim that the LLM-based evaluator 'significantly outperforms' automatic metrics is not supported by any significance test or confidence interval. The correlations are computed on the same 400 messages, so a dependent-correlation test is needed (e.g., Steiger's test or bootstrap confidence intervals for the difference between GPT-4's and METEOR's Spearman correlations). Without such inference, the reader cannot assess whether the large point differences (0.65 vs. 0.42 for What; 0.78 vs. 0.20 for Why) are statistically robust or partly due to sampling noise, especially given the moderate inter-rater reliability of the ground truth.","section":"Section 5.2 (Table 3)"}],"minor_comments":[{"comment":"The text states that 'each model generating one-third of the commit messages,' but 400 is not divisible by 3. Please clarify how the 400 commits were allocated among the three LLMs.","section":"Section 3.1.2"},{"comment":"The sentence 'Our evaluation involved 400 generated commit messages (see Sec. 3.2)' should refer to Section 3.1.2, where the generation of messages is described, not Section 3.2, which describes human evaluation.","section":"Section 5.1"},{"comment":"The caption states that 'the upper triangle (blue) represents What, the lower triangle (green) represents Why,' but the layout of the printed matrix is ambiguous. Please verify that the triangle colors and orientation match the figure's actual presentation.","section":"Figure 1 caption"},{"comment":"For GPT-4, the rows 'CoT' and 'Zero-shot+CoT' show identical unified-evaluation correlations (What 0.65, Why 0.78). If these are two distinct strategies, explain why the values coincide; if they are redundant, consider removing one row to avoid confusion.","section":"Table 2"},{"comment":"The abstract claims 'near human-level evaluation proficiency,' while Section 7.2 advises that LLM metrics 'should be viewed as approximations of human judgment' and 'are unsuitable for precise comparative analyses.' Please align these statements so the conclusion matches the reported limitations.","section":"Abstract and Section 7.2"},{"comment":"Reference [25] attributes ROUGE to Lin and Och (2004), but the canonical ROUGE reference is Lin (2004), 'ROUGE: A Package for Automatic Evaluation of Summaries.' Please check and correct this citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of ICSE and addresses a practical evaluation problem. The main concern I want to flag for the editor is the self-preference confound in RQ2: the fairness analysis does not cover the actual RQ2 corpus, so the headline correlation against METEOR could be inflated. I believe this is fixable with additional per-source correlations and a redesigned fairness test, which is why I recommend major revision rather than rejection. The in-sample prompt selection and the lack of significance tests are also fixable through re-analysis. The human-annotation effort is a genuine strength, but the 'near human-level' claim needs to be tempered or more carefully benchmarked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent first pass at using LLM judges for commit message quality, with a reusable human-rated benchmark and a sensible prompt-strategy comparison. The headline result—GPT-4 with CoT+few-shot reaches Spearman 0.65 on What and 0.78 on Why, versus a best traditional metric at 0.42/0.20—is credible for the corpus they built, but the paper oversells the 'human-level' claim and leaves one real confound unaddressed.\n\nWhat's genuinely new: LLM-as-a-judge has been applied in NLP, but nobody had done it for commit messages. The 800-message benchmark with six experienced raters, the systematic comparison of zero-shot, CoT, few-shot, and unified vs separate scoring, and the stability analysis across temperatures and paraphrases are useful contributions. The thematic analysis of why METEOR fails is solid and gives the paper practical value beyond the headline numbers. Replication package is promised.\n\nSoft spots, in order of real weight. First, the best prompt configuration was chosen on the same 400 messages used to report RQ1's headline correlations. That's selection on the evaluation data; without a holdout or cross-validation the reported numbers are upper-bound-ish. Second, the fairness analysis doesn't test the confound that matters most for RQ2: a third of the evaluated messages were generated by GPT-4 itself. The 6.3 analysis uses GPT-4-paraphrased human messages, not the actual GPT-4-generated messages, and the 25-pair subset where they do compare human vs LLM-generated actually shows a 0.32-point gap favoring LLM-generated—which they acknowledge but dismiss due to sample size. That's not fatal, but it means the RQ2 comparison against METEOR could be slightly inflated by self-preference. Third, comparing the LLM's correlation with the averaged human scores to the pairwise inter-rater correlations (0.61–0.78) is apples-to-oranges: the average of six raters is more reliable than any individual, so the LLM should be expected to correlate higher. The 'near-human-level' wording overstates it. Minor point: no confidence intervals on any of the correlations.\n\nThe human benchmark itself is reasonable: six experienced volunteers, pairwise correlations above 0.6, calibration session. It's small and Python/Java only, but that's fine for a first benchmark. The paper is honest about many of these limitations in Sec 7.2—they flag limited fairness samples—but they don't flag the prompt-selection-on-eval-data issue or the apples-to-oranges comparison.\n\nWho this is for: anyone working on commit message generation or LLM-based evaluation of code-related text. It deserves a serious peer review; the flaws are addressable in revision, and the benchmark plus the prompt-strategy findings are worth having in the literature. I'd recommend conditional accept with requests for a holdout or cross-validation, a direct self-preference test on the RQ2 corpus, and a rewrite of the human-level claim.","headline":"First real test of LLM judges for commit messages, with a usable benchmark and honest limitations—needs a holdout and a direct self-preference check, but worth peer review.","tokens_in":19593,"tokens_out":3321,"would_cite":true,"duration_ms":40122,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4 can grade commit messages almost as well as human developers","keywords":["commit message generation","commit message quality","LLM evaluation","chain-of-thought prompting","few-shot learning","automatic metrics","human evaluation","Spearman correlation"],"falsifier":"Run the same GPT-4 CoT plus few-shot prompt on a fresh sample of commits, have each message rated by a panel of ten or more independent developers, and compare the LLM's ranks with the panel's consensus. If the Spearman correlation on this stronger ground truth falls to METEOR's level, around 0.4 on What and 0.2 on Why, or if the six-volunteer ratings disagree substantially with the larger panel, the near-human-level claim would not transfer.","tokens_in":18564,"feed_emoji":"🤖","tokens_out":5664,"duration_ms":62745,"temperature":0.7,"pith_summary":"This paper tries to establish that LLM-based evaluators can substitute for human raters when judging whether an automatically generated commit message is any good. Today's standard metrics, BLEU, ROUGE-L, and METEOR, compare a generated message against a single developer-written reference, but the same code change can be described in many equally valid ways, so those scores mislead. The authors show that prompting GPT-4 to reason step-by-step, with examples, and to score the What and Why dimensions together yields correlations with human ratings of 0.65 and 0.78, falling inside the range of pairwise human-rater agreement and far above the best traditional metric. If this holds, commit-message evaluation can shift from days of human annotation to an automated, cheap, LLM-based pipeline. The paper is explicitly a preliminary study and flags that LLM scores vary from run to run.","feed_headline":"GPT-4 rates commit messages nearly as well as human devs","feed_subtitle":"With chain-of-thought prompting, it beats METEOR and other reference-based metrics on both What and Why quality.","key_machinery":"The mechanism that carries the argument is the prompt configuration used with the LLM, not a new model. In a single prompt, the model acts as an experienced software engineer, receives definitions of the What and Why dimensions with a 5-point rubric, is asked to reason step-by-step (Chain-of-Thought), is shown exemplary high- and low-quality commit messages (few-shot demonstrations), and gives scores for both dimensions at once (unified evaluation); three repeated runs are averaged to obtain the final score. The comparison machinery is Spearman and Kendall correlation of the LLM's averaged scores against the averaged human ratings.","core_discovery":"The paper's central claim is that when a Large Language Model is prompted with chain-of-thought evaluation steps, a few high- and low-quality examples, and a single prompt that scores both content dimensions at once, it can grade commit messages nearly as well as experienced developers. In the authors' data, this configuration, realized with GPT-4, reached Spearman correlations of 0.65 for What and 0.78 for Why, versus at most 0.42 and 0.20 for the best reference-based metric, METEOR. Because the pairwise correlations among the six human raters ranged from about 0.61 to 0.78, the authors interpret the LLM's scores as human-level. They also report that the advantage holds across four models tested and that the LLM evaluator shows acceptable reproducibility, robustness, and fairness, though scores vary and should be treated as approximations rather than precise measurements.","pith_inferences":["An implication the authors leave implicit is that the evaluator could be reused as a reward signal for training or fine-tuning commit-message generators, not just as a post-hoc metric.","A natural testable extension is to check whether the same prompt ordering transfers to other languages, since the study samples only Python and Java commits; JavaScript, TypeScript, or C++ datasets would settle that question.","The paper's stability data suggest a practical guardrail: report averaged scores from multiple runs and only treat differences larger than about one point on the 5-point scale as meaningful.","Because the failure mode of reference metrics is high expression diversity, LLM-based evaluation may also benefit other code-related summarization tasks, such as release-note or API-documentation generation, where one input admits many valid outputs."],"forward_implications":["If the central claim is right, commit-message evaluation can replace or complement days of human annotation with an LLM pipeline that runs in under an hour and costs roughly $0.08 per example.","The LLM evaluator's Why-dimension correlation of 0.78 is especially consequential, because no reference-based or semantic-similarity metric comes close to human judgment on the Why dimension.","The one-to-many mapping between a code change and valid commit messages is identified as the fundamental reason reference metrics fail, which explains why direct content judgment outperforms similarity to a single reference.","Because LLM scores are slightly unstable, the authors state that these evaluators should be used to approximate human judgment rather than to make precise comparisons between two models that score within about a point of each other.","The same CoT plus few-shot integrated prompt template can be reused across GPT-4, Llama-3.3, Qwen2.5, and QwQ, with all tested models showing moderate to strong correlation with human ratings."],"supporting_citations":[{"why":"Supplies the CommitBench dataset from which the 400 Python and Java commits are sampled.","marker":"[36]"},{"why":"GPT-4 is the proprietary model whose CoT plus few-shot configuration produces the headline correlations.","marker":"[1]"},{"why":"METEOR is the best-performing traditional metric that the LLM evaluator must beat.","marker":"[3]"},{"why":"Defines the What and Why dimensions that structure the whole evaluation.","marker":"[41]"},{"why":"Provides the prior four-dimension human evaluation framework that the authors consolidate into What and Why.","marker":"[24]"},{"why":"G-Eval is the precedent showing GPT-4 can align with human judgment in NLG evaluation.","marker":"[29]"},{"why":"Chain-of-thought prompting is the technique that most improves Why-dimension correlation.","marker":"[48]"},{"why":"Few-shot in-context learning supplies the example-based demonstration technique used in the winning prompt.","marker":"[4]"},{"why":"BERTScore is one of the semantic-similarity baselines compared against the LLM evaluator.","marker":"[53]"}],"fun_headline_variants":["LLM with CoT scores commit messages near human level","GPT-4 matches human devs in commit message evaluation","CoT-prompted LLM beats METEOR as commit message judge","AI rates commit messages as well as human developers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the averaged 5-point ratings of six experienced volunteers, who agree pairwise at 0.61 to 0.78, are a trustworthy measure of commit message quality; the paper validates those raters only against each other, not against any external standard, and only on Python and Java commits.","fun_headline_variants_meta":{"raw":{"variants":["LLM with CoT scores commit messages near human level","GPT-4 matches human devs in commit message evaluation","CoT-prompted LLM beats METEOR as commit message judge","AI rates commit messages as well as human developers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1293,"prompt_tokens":924,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":540,"tokens_out":369,"duration_ms":4747,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:22:07.592208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same GPT-4 CoT plus few-shot prompt on a fresh sample of commits, have each message rated by a panel of ten or more independent developers, and compare the LLM's ranks with the panel's consensus. If the Spearman correlation on this stronger ground truth falls to METEOR's level, around 0.4 on What and 0.2 on Why, or if the six-volunteer ratings disagree substantially with the larger panel, the near-human-level claim would not transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CommitBench dataset from which the 400 Python and Java commits are sampled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"METEOR is the best-performing traditional metric that the LLM evaluator must beat."}],"review_version":1}