REVIEW 3 major objections 6 minor 14 references
Rethinking Training Data for Generating Code Review Comments
T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read The paper argues that nearly half of sampled code-review training pairs misalign with the task of generating actionable feedback from a diff, and that taxonomy-guided prompts barely improve LLM-based filtering.
desk verdict Useful taxonomy and an honest negative result on LLM filtering, but the headline 48% misalignment rate rides on a contestable definition of 'actionability' that the paper never defends. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key objects are the taxonomy of misaligned training pairs—semantic ambiguity, lack of actionability, and context dependence—and the prompt-engineering experiment built on it. The taxonomy operationalizes what 'misaligned' means by classifying why a diff-comment pair fails to supply a reliable learning signal; the experiment (three system prompts, two LLMs, two input representations) tests whether that taxonomy transfers to automatic filtering, thereby exposing the limits of zero-shot prompting for data cleaning.
What would settle it
Re-annotate the 383 sampled pairs with a broader validity criterion that accepts clarifying and conversational comments as valid training signal; if the misalignment share drops below, say, 25%, the paper's central estimate overstates structural misalignment. Alternatively, if a taxonomy-guided filter with few-shot examples reaches F1 above 0.85 on the same sample, the conclusion that taxonomy cues do not help filtering would fail.
Extended reading notes
Core claim
The central claim is that a large fraction of diff-comment training pairs are not merely noisy but structurally misaligned with the intended learning objective. In a manual review of 383 randomly sampled pairs, 184 (48%) were judged misaligned: either the comment's intent is ambiguous (25), the comment does not request or justify any change (98), or the comment depends on context outside the diff (61). Incorporating this taxonomy into LLM filtering prompts as definitions, criteria, and categories did not substantially improve filtering over simpler prompts: F1 gains were marginal and inconsistent across models and input representations, and the strongest configuration remained around 0.71 fo
Load-bearing premise
A training pair is counted as misaligned only if an actionable comment cannot be inferred from the diff alone; if the intended task includes clarification, coordination, or full-context reasoning, the 48% estimate would shrink.
Editorial extensions
If this is right
- Models trained on these datasets are rewarded for producing fluent, review-like text even when an instance does not support actionable feedback grounded in the diff, explaining generic or weakly grounded outputs.
- Cleaning datasets by removing noisy instances is necessary but not sufficient; the 48% misalignment estimate implies that significant portions of the data encode non-actionable or context-dependent review behavior.
- To support actionable feedback, input representations should include review-thread history, references to non-local code, project conventions, or CI signals, rather than relying on diffs alone.
- Evaluation of generated review comments should prioritize actionability and review intent over surface-level textual similarity.
- LLM-based data filtering with zero-shot taxonomy prompts leaves large room for improvement; better validity criteria and richer prompting or model configurations are needed.
Reading between the lines
- If the 48% estimate generalizes, published performance numbers on this dataset likely overstate quality, since many test-time comments also exhibit the same misalignment; evaluating on an aligned-only subset could change rankings.
- The taxonomy could be turned into a diagnostic benchmark for agentic review systems: each category suggests a specific capability (e.g., abstain on insufficient context, retrieve previous comments), making the categories directly testable.
- The null filtering result may reflect prompt fidelity rather than taxonomy uselessness; few-shot examples per category or fine-tuned classifiers are a natural next test the paper did not run.
- A direct implication the paper leaves implicit: training on the aligned subset alone, or adding an 'abstain' option during inference, could avoid forcing models to hallucinate grounded feedback.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates training data for code review comment generation. It manually samples 383 diff-comment pairs from the CodeReviewer dataset and labels each pair as aligned or misaligned, where misalignment means the comment is not actionable review feedback that could be generated from the diff alone. Two annotators achieve κ=0.758; consensus yields 184 misaligned pairs (48%). A card-sorting analysis produces a three-category taxonomy: semantic ambiguity, lack of actionability, and context dependence. The paper then compares three LLM-filtering prompts (a baseline from Liu et al., a criteria-expanded version, and a taxonomy-augmented version) across two LLMs and two input representations, using both the authors' 383-pair sample and Liu et al.'s 270-pair dataset. The taxonomy-augmented prompt does not substantially improve filtering. The authors conclude that dataset cleaning alone is insufficient and call for explicit validity criteria, richer contextual inputs, and evaluation aligned with review intent.
Significance. If the empirical characterization holds, the paper provides a useful reframing: problematic training pairs are not homogeneous noise but reflect systematic misalignment between the task and current diff-comment representations. The taxonomy (semantic ambiguity, lack of actionability, context dependence) is a practical instrument for dataset design. The negative result for taxonomy-guided LLM filtering is an important caution, especially because it is evaluated on an external dataset. The paper is methodologically transparent: it releases a replication package, uses two independent annotators, reports inter-rater reliability, and covers 12 configurations. However, the central 48% estimate is sensitive to a contestable coding of clarification requests as non-actionable, and the filtering comparison lacks statistical inference. These issues need addressing before the magnitude claims can be fully accepted.
major comments (3)
- [§III-A3 / Table I] The coding of 'Inquiry' comments (56 of 383) as non-actionable under 'Lack of Actionability' is a normative choice. The paper's own screening definition asks whether a model 'would be expected to generate' a comment from a diff; clarifying questions are legitimate review feedback that often lead to changes. The paper even notes in §V that these are 'legitimate artifacts of real-world code review, including clarification.' If the 56 Inquiry instances are reclassified as aligned, the misalignment rate drops from 184/383 (48%) to 128/383 (33%), and the most frequent source becomes Context Dependence (61) rather than Lack of Actionability (42 remaining). The paper should justify the actionability criterion with respect to the intended task and report a sensitivity analysis under alternative definitions. As written, the headline 48% is not robust to a reasonable alternative interpretation.
- [§IV-B / Table II] The claim that taxonomy-augmented prompting does not substantially improve filtering is made on F1 differences without significance tests or confidence intervals. Example: on the Liu et al. dataset with GPT-3.5 and UPNL, SPDC yields Valid F1 0.71 and Noisy F1 0.63, while SPDCT yields 0.69 and 0.60; on the authors' sample, the corresponding comparison is 0.65/0.68 vs. 0.63/0.62. These differences are within sampling variability. Report bootstrapped confidence intervals or a paired test (e.g., McNemar) to support the negative conclusion, or present the results as purely descriptive without the 'did not substantially improve' framing.
- [§IV-A5 / Table II] The random baseline is a single random labeling rather than an expected-random classifier. F1 from one random draw is highly variable; e.g., the reported random baseline on the Liu valid class (0.53) is not the expected value under random labeling. Average over multiple permutations, or provide the theoretical expected F1 given class priors, so the 'beyond chance' comparison is properly calibrated.
minor comments (6)
- [§III-A4] The reference [14] for the card-sorting methodology appears to be about data quality for unit test generation; please check the citation or supply the appropriate methodology reference.
- [Throughout] The label 'our dataset' for the 383-pair annotated sample is misleading; call it 'our annotated sample' since it is not a dataset in the corpus sense.
- [Fig. 1] The yellow highlighting used to indicate modifications may not be visible in monochrome printing; add a textual or boxed indicator.
- [§III-A3] The screening instruction phrase 'a model would be expected to generate' is ambiguous: expected by whom, and for what kind of model? Clarify the intended normative criterion.
- [§IV-B] The statement 'GPT-3.5-turbo consistently outperformed GPT-4o-mini' is too strong: while this holds for Valid-class F1, GPT-4o-mini often achieves higher Noisy-class F1 (e.g., SPDCT 4o-mini on the authors' sample, Noisy F1 0.71 vs. 0.62 for GPT-3.5). Qualify the claim by class or metric.
- [Table II] The green-cell notation 'above the random baseline' should be defined in the caption, especially given the single random draw issue.
Circularity Check
No significant circularity: the empirical claims are annotation summaries and external-benchmark evaluations, not results identical to their inputs by construction.
full rationale
The paper's central empirical claim, that 184/383 (48%) sampled pairs are misaligned (§III-B), is a prevalence count produced by independent human screening under an explicit, normative definition of actionable review feedback (§III-A3), with substantial inter-rater reliability (κ = 0.758). The taxonomy (§III-A4) was constructed after that screening via open card sorting over the already-labeled misaligned pairs; it is a descriptive explanation of why those pairs failed the screening definition, not a device used to derive the 48% figure. The second main claim, that adding taxonomy-derived categories to prompts did not substantially improve filtering (§IV-B), is evaluated on two datasets — including the external Liu et al. dataset — and compared against a random baseline and F1 metrics for both classes. The taxonomy prompt is an independent experimental condition; its categories are not the ground-truth labels, which come from human annotations. There are no fitted parameters renamed as predictions, no load-bearing self-citations (references [6] and [8] are external prior work), and no imported uniqueness theorems. The normative definition of 'actionable' — which places clarifying inquiries under lack of actionability — is a construct-validity or sensitivity concern, not circularity, because the paper does not define the result in terms of itself or fit anything to the target claim.
Assumptions & free parameters
assumptions (5)
- standard math A 95%-confidence, 5%-margin random sample of 383 pairs estimates the dataset-level misalignment proportion.
- domain assumption The CodeReviewer dataset [3] is representative of current practice in review-comment training data.
- domain assumption A training pair is valid only if it enables generating actionable review feedback from localized (diff-only) input.
- domain assumption The two postgraduate annotators' consensus labels are reliable ground truth (κ=0.758).
- ad hoc to paper Prompt SP_DCT (Definition+Criteria+Taxonomy) is a fair operationalization of incorporating the taxonomy into LLM filtering.
Cite this review
Pith. "Pith review of Rethinking Training Data for Generating Code Review Comments." pith.science (2026). https://pith.science/paper/TJGK46VH
@misc{pith2026260725851,
author = {Pith},
title = {Pith review of: Rethinking Training Data for Generating Code Review Comments},
year = {2026},
howpublished = {\url{https://pith.science/paper/TJGK46VH}},
note = {Machine review of arXiv:2607.25851}
}
read the original abstract
Generating code review comments has become a prominent research direction in automated code review, commonly formulated as a text generation task over diff-comment pairs. Despite advances in learning-based approaches, generated review comments are often generic, weakly grounded, or non-actionable. Recent studies have also shown that review comment datasets contain noisy or unsuitable training instances, motivating LLM-based dataset cleaning approaches. In this paper, we argue that problematic training instances are not homogeneous and that some limitations stem from deeper issues in the task formulation itself. Through an empirical inspection of a widely used review comment dataset, we identify misaligned training pairs: instances where the relationship between the code change and the review comment does not provide a reliable learning signal for generating actionable review feedback from localized inputs. We derive a taxonomy of misalignment capturing three recurring sources: semantic ambiguity, lack of actionability, and context dependence. We further explore whether incorporating this taxonomy into LLM-based filtering improves the identification of problematic training instances, observing that detecting misaligned training pairs remains challenging. Based on these observations, we argue that improving review comment generation requires more than dataset cleaning alone, motivating explicit validity criteria, richer contextual inputs, and evaluation practices aligned with review intent and actionability.
Figures
Reference graph
Works this paper leans on
-
[1]
Towards automating code review activities,
R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” inProceedings of the 43rd International Conference on Software Engineering, ser. ICSE ’21. IEEE Press, 2021, p. 163–174. [Online]. Available: https://doi.org/10.1109/ICSE43902.2021.00027
arXiv 2021
-
[2]
Autotransform: automated code transformation to support modern code review process,
P. Thongtanunam, C. Pornprasit, and C. Tantithamthavorn, “Autotransform: automated code transformation to support modern code review process,” inProceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 237–248. [Online]. Available: https://doi.org/10.1145/351...
arXiv 2022
-
[4]
Commentfinder: a simpler, faster, more accurate code review comments recommendation,
Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Commentfinder: a simpler, faster, more accurate code review comments recommendation,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY , USA: Association for Computing Machine...
arXiv 2022
-
[5]
Auger: automatically generating review comments with pre-training models,
L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY , USA: Association for Computing Machiner...
arXiv 2022
-
[6]
Code review automation: Strengths and weaknesses of the state of the art,
R. Tufano, O. Dabi ´c, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: Strengths and weaknesses of the state of the art,”IEEE Transactions on Software Engineering, vol. 50, no. 2, pp. 338–353, 2024
2024
-
[7]
Improving automated code reviews: Learning from experience,
H. Y . Lin, P. Thongtanunam, C. Treude, and W. Charoenwet, “Improving automated code reviews: Learning from experience,” in Proceedings of the 21st International Conference on Mining Software Repositories, ser. MSR ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 278–283. [Online]. Available: https://doi.org/10.1145/3643991.3644910
arXiv 2024
-
[8]
Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation ,
C. Liu, H. Y . Lin, and P. Thongtanunam, “ Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation ,” in2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). Los Alamitos, CA, USA: IEEE Computer Society, Apr. 2025, pp. 236–248. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/MSR66628.2025.00043
arXiv 2025
-
[9]
A systematic literature review and taxonomy of modern code review,
N. Davila and I. Nunes, “A systematic literature review and taxonomy of modern code review,”Journal of Systems and Software, vol. 177, p. 110951, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121221000480
2021
Show all 14 references
-
[10]
Automating code review: A systematic literature review,
R. Tufano and G. Bavota, “Automating code review: A systematic literature review,” 2025. [Online]. Available: https://arxiv.org/abs/2503.09510
2025 arXiv
-
[11]
Fine-tuning large language models to improve accuracy and comprehensibility of automated code review,
Y . Yu, G. Rong, H. Shen, H. Zhang, D. Shao, M. Wang, Z. Wei, Y . Xu, and J. Wang, “Fine-tuning large language models to improve accuracy and comprehensibility of automated code review,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 1, Dec. 2024. [Online]. Available: https://d...
2024 doi
-
[12]
Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning,
J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning,” 2023. [Online]. Available: https://arxiv.org/abs/2308.11148
2023 arXiv
-
[13]
Exploring the potential of chatgpt in automated code refinement: An empirical study,
Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Exploring the potential of chatgpt in automated code refinement: An empirical study,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , USA: Associati...
2024
-
[14]
Less is more: On the importance of data quality for unit test generation,
J. Zhang, X. Hu, S. Gao, X. Xia, D. Lo, and S. Li, “Less is more: On the importance of data quality for unit test generation,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715778
2025 doi
-
[15]
Replication package
“Replication package.” [Online]. Available: https://doi.org/10.6084/m9.figshare.32296611
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.