{"id":"4d0aedec-c26d-4ce5-8739-01672ba0e75f","arxiv_id":"2412.18291","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An empirical study that rebuilds code review comment evaluation around a nine-criterion rubric and claims a training-free LLM baseline beats existing generators.","lead":"This paper argues that text similarity metrics like BLEU and ROUGE are the wrong way to judge generated code review comments, and offers a rubric-based framework called DeepCRCEval. Using that framework, a few-shot GPT-4 prompt outperforms existing comment generators, though the comparison is confounded by giving the prompt the rubric.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3 comparison is confounded: LLM-Reviewer's prompt contains the exact nine criteria used to score all models, so its top ranking may reflect rubric leakage rather than superior comment quality.","rationale":"The reader's weakest_assumption identifies the same core issue: using the same nine criteria to prompt LLM-Reviewer and to score every model makes the comparison unfair. My stress-test sharpens this into the specific mechanism of rubric leakage and argues that it is the most load-bearing threat to the central claim, because without a fair comparison the headline 'LLM-Reviewer surpasses previous models' is not established. The concrete ablation test directly isolates this mechanism. The rest of the paper—the RQ1 benchmark quality analysis and DeepCRCEval framework—may still be valuable, so the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed, but the stated condition (rubric-blind baseline and representative sample) should be made explicit as a requirement for acceptance.","tokens_in":17253,"tokens_out":5053,"duration_ms":46350,"concrete_test":"Run a rubric-blind ablation: remove the nine criteria from the LLM-Reviewer prompt, replacing them with a generic instruction ('provide a concise, useful review comment'), keep the same 1,000 test cases, and re-score all six systems with the DeepCRCEval human evaluators (and, secondarily, LLM evaluators) on the nine criteria, with evaluators blinded to which system produced which comment. If LLM-Reviewer's margin over the best baseline collapses or its rank falls below 1, the original RQ3 conclusion is an artifact of prompt–evaluation overlap. If the margin persists, the conclusion is robust. A supplementary check is to score all comments with an independent rubric-free judgment, e.g., professional developers marking whether each comment identifies a real defect, which tests whether the nine criteria themselves are the right target.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RQ3 comparison is confounded by prompt–evaluation rubric overlap. LLM-Reviewer's generation prompt (Table 10, Eq. 3) instructs the model to adhere to the exact nine criteria of Section 4.1, and the DeepCRCEval evaluator prompt (Table 9, Eq. 1) scores and ranks comments on those same nine criteria. Section 5.3 explicitly says 'we use a similar prompt template to the one used for prompt evaluation.' Thus LLM-Reviewer is the only system told the grading rubric in advance; the five baselines were trained or designed for text similarity (BLEU/ROUGE) and never saw these criteria. The observed margin (LLM-Reviewer scoring 8.89–10.00 across criteria vs. baselines often below 4, and ranking first by a large margin in Table 7) is therefore at least partly an artifact of 'teaching to the test': LLM-Reviewer is not shown to be a stronger generator in an absolute sense, only a generator that has been handed the evaluation rubric. This does not refute the claim that text similarity is an imperfect proxy, but it does undermine the headline that a training-free baseline outperforms all SOTA CRCGs. The paper's internal human/LLM agreement (Table 5) does not resolve this, because both evaluator types used the same biased prompt and criteria.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that text-similarity metrics (BLEU, ROUGE) are inadequate for evaluating code review comment generation (CRCG), and proposes DeepCRCEval, a nine-criteria evaluation framework with human and LLM evaluators, plus LLM-Reviewer, a training-free few-shot GPT-4 baseline. RQ1 analyzes 100 sampled comments from the Tufano and CodeReviewer datasets and finds that only 3% and 8% of reference comments, respectively, meet the authors' quality/category/tone/context criteria. RQ2 reports that DeepCRCEval discriminates better than text similarity and that LLM evaluators reduce time and cost by large margins while maintaining moderate-to-good agreement with humans. RQ3 uses DeepCRCEval to score and rank six CRCGs, including LLM-Reviewer, and reports that LLM-Reviewer ranks first by a wide margin on nearly every criterion.","tokens_in":17524,"tokens_out":2449,"duration_ms":22949,"significance":"If the RQ3 comparison were valid, the paper would make a strong contribution: it would show that reference-based text similarity metrics mislead model development, that a prompt-based LLM baseline can outperform tuned CRCGs on task-oriented criteria, and that LLM evaluation is a viable low-cost proxy for human panel evaluation. The RQ1 analysis is a useful, concretely documented empirical study with multiple annotators and public materials. However, the central RQ3 result is compromised by a prompt-evaluation rubric overlap and by the use of a hand-selected, simplified test set, so the headline claim that LLM-Reviewer surpasses all SOTA CRCGs is not established by the evidence as presented.","major_comments":[{"comment":"The RQ3 comparison is substantially confounded by rubric leakage: LLM-Reviewer's generation prompt (Table 10, Eq. 3) instructs the model to follow exactly the nine criteria in Section 4.1, while the DeepCRCEval evaluator prompt (Table 9, Eq. 1) scores and ranks comments on those same nine criteria. Section 5.3 explicitly states that 'we use a similar prompt template to the one used for prompt evaluation.' Thus LLM-Reviewer is the only system that has been told the grading rubric in advance; the five baseline CRCGs were trained with text-similarity objectives and never received the criteria. The large margins in Tables 6 and 7 are therefore at least partly an artifact of teaching to the test, not evidence that LLM-Reviewer is an absolutely better generator. The paper's internal human/LLM agreement (Table 5) does not resolve this because both evaluator types used the same criteria-based prompt. To support the RQ3 claim, the authors should either give the baseline CRCGs an equivalent criteria-guided prompt (e.g., through post-hoc prompt-based revision or a fair few-shot setup), or validate the comparison with an external criterion such as whether the comments lead to defect remediation or developer acceptance.","section":"Section 5.3 vs. Section 5.1, Tables 9 and 10"},{"comment":"The test set used for RQ3 is not the standard benchmark test split: the authors state that they 'selected a set of 1,000 code cases with typical issues' and 'processed by humans to enhance simplicity.' This hand-selection and simplification means the comparison does not measure performance on the distribution used to train and evaluate the baseline CRCGs, and it favors a model that can be prompted to look for 'typical issues' in simplified snippets. The claim that existing SOTA CRCGs are outperformed by LLM-Reviewer therefore conflates a model's quality with the choice of evaluation distribution. The authors should re-run the comparison on the original test sets of Tufano and CodeReviewer (or provide a principled justification for why the new distribution is the correct one for all models).","section":"Section 5.4 and Section 6.2"},{"comment":"The human evaluation is small-sample and uses non-developer proxies. Section 7.3 acknowledges the small sample size and that graduate students served as proxies for developers, and the user feedback study in Section 7.1 involved only 66 cases across 5 developers. Combined with the fact that the human evaluators scored the same criteria that were embedded in LLM-Reviewer's prompt, the human results do not provide independent validation of the RQ3 ordering. The paper should report how many distinct cases each human evaluator scored for RQ3, report inter-rater reliability for the ranking task, and either enlarge the human sample or frame the human results as illustrative rather than as the primary support for the superiority claim.","section":"Section 7.3 and Tables 5-6"},{"comment":"The nine criteria are used both to construct the evaluation framework and to instruct LLM-Reviewer, but their validity is supported only by internal agreement statistics and by the authors' qualitative process; there is no external validation against an outcome such as whether comments detect real defects or are accepted by developers. The paper should provide external validation evidence, for example by correlating DeepCRCEval scores with whether developers act on the comments, or by showing that comments judged high by C1-C9 are more likely to be followed by a code change. Without such validation, the circularity between the criteria, the evaluator, and the proposed baseline weakens the central conclusion.","section":"Section 4.1 and Section 5.1"}],"minor_comments":[{"comment":"Table 2 uses the label 'Tuano' for the Tufano et al. dataset; this appears to be a typo and should be corrected to 'Tufano'.","section":"Section 4.3"},{"comment":"The sentence 'To thoroughly explore previous code review comment datasets, we defining aspects with a qualitative and quantitative process' contains a grammatical error ('we defining'); it should read 'we define aspects...'.","section":"Section 4.1"},{"comment":"The phrase 'using a human scoring system created using QT and and a Delphi Method variant' has a duplicated 'and' and should be revised.","section":"Section 4.2"},{"comment":"The notation 'CRer' vs. 'Crer' is inconsistent; the dataset is abbreviated both ways in Tables 2 and 3 and in the text, which should be unified.","section":"Section 4.3"},{"comment":"The appendix title 'Detailed Analysis Methodology for Dateset Comment Quality' contains a typo: 'Dateset' should be 'Dataset'.","section":"Appendix C title"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful empirical contribution in RQ1, and the framework is clearly described with public artifacts. However, the RQ3 headline result is not currently supported because the baseline comparison is confounded by the prompt-rubric overlap and by the hand-selected test set. The authors should be pushed to either re-run the comparison fairly or substantially soften the claim. The external validation of the nine criteria is also important for the paper's central premise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the benchmark analysis in RQ1. It carefully shows, with multiple annotators and a defensible method, that only 3–8% of human comments in two standard datasets meet quality criteria that actually matter for automation. That is a concrete, reproducible finding, and the authors deserve credit for shipping their materials on Zenodo. The nine-criteria set, built from prior work plus developer interviews, is a reasonable starting point even if its validity is only internally supported.\n\nThe soft spot is exactly where the stress-test lands. For RQ3, LLM-Reviewer's generation prompt contains the same nine criteria used to score every model, and the paper even acknowledges the prompt template is similar. The baselines were trained or designed for text similarity and never saw the rubric. So the finding that LLM-Reviewer outscores them by a wide margin is substantially forced by construction. That does not refute the broader claim that text similarity is a poor proxy, but it does mean the paper's headline — that a training-free baseline beats all SOTA CRCGs — is not supported by this experiment. The human evaluation is also small-sample, and the test set is hand-selected to contain typical issues, which further limits how representative the comparison is.\n\nThere are two distinct claims in the paper: the benchmark-quality finding and the model-ranking finding. The first is solid and useful on its own. The second is confounded and needs to be redone with a rubric-blind baseline and a representative test set. The criteria themselves could also use external validation, for instance checking whether high-scoring comments actually correlate with defect remediation or developer acceptance.\n\nWho is this for? Researchers working on code review automation and LLM evaluation will get value from the criteria set and the benchmark-quality analysis. The comparative results should be treated as pilot evidence, not as proof. The paper deserves a serious referee — it addresses a real problem and ships reproducible artifacts — but the RQ3 comparison needs major revision before it can be accepted. I would recommend sending it out, with the expectation that reviewers push for a clean baseline comparison.","headline":"The benchmark-quality analysis is genuinely useful, but the headline comparison is compromised because LLM-Reviewer is handed the same rubric used to score everyone.","tokens_in":18089,"tokens_out":906,"would_cite":true,"duration_ms":9867,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that text-similarity metrics (BLEU, ROUGE) mislead code-review-comment evaluation, that fewer than 10% of benchmark comments clear a nine-criteria quality bar, and that a training-free LLM prompt beats five tuned…","keywords":["code review automation","code review comment generation","LLM evaluation","text similarity metrics","benchmark quality","LLM-Reviewer","DeepCRCEval","defect detection"],"falsifier":"A decisive test would blind working developers to which system produced each comment and ask them whether they would act on it; if comments scored high by DeepCRCEval are not more likely to be acted on than low-scoring ones, or if the five baseline models are chosen as often as LLM-Reviewer, the framework's central claim fails.","tokens_in":17008,"feed_emoji":"📝","tokens_out":7889,"duration_ms":65663,"temperature":0.7,"pith_summary":"This paper argues that the standard evaluation of code review comment generation, which measures word overlap between generated comments and human-written benchmark comments, gives a distorted picture of progress. It analyzes 200 benchmark comments from the two dominant datasets and finds that only 3% (Tufano) and 8% (CodeReviewer) meet its nine quality criteria, so most reference comments are not reliable ground truth. To fix this, the authors build DeepCRCEval, a scoring-and-ranking framework that uses human and LLM evaluators against those criteria, and introduce LLM-Reviewer, a training-free, prompt-based generator that is guided by the same criteria. Under DeepCRCEval, LLM-Reviewer is ranked first by both human and LLM evaluators, ahead of five tuned generators, suggesting that the field's text-similarity leaderboards have substantially overstated the quality of current models. The practical stake is that evaluation of automated review comments should target the actual goals of code review, defect detection and code improvement, rather than lexical similarity.","feed_headline":"Prompt-only reviewer beats five tuned code-review bots","feed_subtitle":"A criteria-based evaluation finds text-similarity scores overstate progress in automated code review comments.","key_machinery":"The central mechanism is a shared nine-criterion definition of comment quality: readability, relevance, explanation clarity, problem identification, actionability, completeness, specificity, contextual adequacy, and brevity, derived from prior work, developer interviews, and card sorting. DeepCRCEval turns those criteria into structured prompts that ask an evaluator to score each comment on all nine dimensions and then rank the models, with a chain-of-thought step linking scoring to ranking; the framework can be run by human raters or by GPT-4 as a low-cost evaluator. LLM-Reviewer is the mirror image of the evaluator: the same nine criteria are embedded in a few-shot prompt that instructs GPT-4 to generate a comment for a code snippet. The paper's argument is carried by this symmetry, because the generator is optimized for the same explicit targets the evaluator measures, which is precisely the alignment that text-similarity-trained models lack.","core_discovery":"The central claim, stated on the paper's own terms, is that code review comment generators trained and selected with BLEU and ROUGE are not actually producing the kind of comments that serve code review. The paper's empirical analysis of benchmark comments shows that a large share of human-written comments are interrogative, context-dependent, or otherwise unsuitable for one-shot automated review, leaving only 3% of the Tufano dataset and 8% of the CodeReviewer dataset as high-quality references for automation. DeepCRCEval, which scores comments from 1 to 10 on nine criteria and then ranks models, distinguishes high- from low-quality comments far more sharply than BLEU or ROUGE, and its LLM evaluator agrees with human raters on the most task-critical criteria while reducing time by 88.78% and cost by 90.32%. When all six generators are compared under DeepCRCEval, the training-free LLM-Reviewer is ranked first by both human and LLM evaluators across nearly every criterion and in overall ranking, ahead of Tufano et al., CommentFinder, CodeReviewer, AUGER, and CCT5. The paper concludes from this that existing SOTA CRCGs are outperformed by a prompt-only baseline, and that the relevant bottleneck is a training objective keyed to text similarity rather than to the real targets of code review.","pith_inferences":["A testable extension the paper does not run is to filter the training sets by the nine criteria and retrain an existing generator; improved scores would show that reference-set quality, not model capacity, is the main constraint.","The same criteria-based evaluation could be carried over to other code-review artifacts, such as review summaries or documentation comments, where word-overlap metrics may be equally misleading.","An implicit consequence of the results is that generation could be improved without fine-tuning by constraining decoding with the criteria themselves, essentially turning the evaluator's rubric into a reward signal."],"forward_implications":["BLEU- and ROUGE-based leaderboards for code review comments would no longer be interpretable as quality rankings, because the reference comments they compare against largely fail the nine quality criteria.","Any new code review comment generator would need to beat a training-free, prompt-based baseline under criteria-based evaluation before claiming an improvement over prior work.","Benchmark datasets such as Tufano and CodeReviewer would require substantial filtering or reannotation before they can serve as ground truth for training or evaluation.","Evaluation practice would shift toward multi-criteria scoring plus comparative ranking, with LLM evaluators used to scale the analysis and a small human panel to anchor it."],"supporting_citations":[{"why":"Supplies the prior evidence on how developers define code review quality that the nine criteria extend.","marker":"[12]"},{"why":"Provides the Tufano benchmark dataset whose comments are analyzed and found only 3% suitable.","marker":"[31]"},{"why":"Provides the CodeReviewer dataset and a baseline model; its comments are analyzed and found only 8% suitable.","marker":"[16]"},{"why":"Basis for using LLMs as evaluators, adapted into DeepCRCEval's chain-of-thought scoring-and-ranking prompts.","marker":"[35]"},{"why":"BLEU, one of the text-similarity metrics the paper argues is inadequate for review comment evaluation.","marker":"[22]"},{"why":"ROUGE, one of the text-similarity metrics the paper argues is inadequate for review comment evaluation.","marker":"[18]"},{"why":"Prior evidence that only a minority of real review comments are useful, which the paper's quality analysis corroborates.","marker":"[3]"},{"why":"Supplies the category taxonomy used to classify benchmark comments as defect detection, code improvement, or other types.","marker":"[1]"}],"fun_headline_variants":["Prompt-only LLM beats five tuned code-review bots","Text-similarity metrics misjudge code-review comments","Only 3-8% of code-review comments are automation-ready","LLM evaluator cuts code-review eval cost by 90%","DeepCRCEval: BLEU and ROUGE fail to judge code-review comments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on the nine criteria being a valid and complete definition of effective review comments; if those criteria miss what makes comments useful to developers, both the benchmark-quality analysis and the resulting model ordering lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-only LLM beats five tuned code-review bots","Text-similarity metrics misjudge code-review comments","Only 3-8% of code-review comments are automation-ready","LLM evaluator cuts code-review eval cost by 90%","DeepCRCEval: BLEU and ROUGE fail to judge code-review comments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3732,"prompt_tokens":1060,"completion_tokens":2672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":2581}},"tokens_in":676,"tokens_out":2672,"duration_ms":17668,"temperature":1.0,"reasoning_tokens":2581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:49:29.929375+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would blind working developers to which system produced each comment and ask them whether they would act on it; if comments scored high by DeepCRCEval are not more likely to be acted on than low-scoring ones, or if the five baseline models are chosen as often as LLM-Reviewer, the framework's central claim fails.","supporting_citations":[{"cited_title":"In: Proceedings of the 44th International Conference on Software Engineering","cited_arxiv_id":null,"evidence_quote":"Provides the Tufano benchmark dataset whose comments are analyzed and found only 3% suitable."},{"cited_title":"In: Proceedings of the 30th ACM Joint European Software Engineer- ing Conference and Symposium on the Foundations of Software Engineering","cited_arxiv_id":null,"evidence_quote":"Provides the CodeReviewer dataset and a baseline model; its comments are analyzed and found only 8% suitable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Basis for using LLMs as evaluators, adapted into DeepCRCEval's chain-of-thought scoring-and-ranking prompts."}],"review_version":1}