{"id":"78699cc8-5a8e-4c78-8f99-91159c380a3f","arxiv_id":"2607.25851","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Nearly half of the training pairs in a widely used code review comment dataset fail to support learning actionable, diff-grounded review feedback, and taxonomy-guided LLM filtering does not reliably fix this.","lead":"This paper manually inspects 383 code-review training pairs and finds 48% are 'misaligned' — comments that cannot support learning actionable feedback from a diff alone. It proposes a three-part taxonomy (semantic ambiguity, lack of actionability, context dependence) and shows that adding it to LLM filtering prompts yields little improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 48% misalignment estimate depends on coding clarifying inquiries as non-actionable; reclassifying them drops the rate to ~33% and shifts the top category.","rationale":"I chose the actionability definition because it directly controls the headline number. The paper is careful with inter-rater reliability and sampling, but reliability does not address validity of the coding scheme. The taxonomy's \"Inquiry\" category is particularly contestable: a question like \"Why is this value hard-coded?\" is arguably actionable feedback even though it doesn't contain an imperative. The paper's own §V position—that these are legitimate review interactions—undercuts treating them as misaligned for all training purposes; whether they are misaligned depends on whether the model is intended to emulate conversation or to provide code-change feedback. The reader's weakest assumption flagged the general task-definition issue; I sharpen it to a concrete reclassification test. The prompt-operationalization concern (SP_DCT may not faithfully encode the taxonomy) is real but secondary: the paper explicitly frames the filtering result as exploratory, and the central claim is the 48% estimate. The verdict remains CONDITIONAL because the concern is addressable through sensitivity analysis.","tokens_in":8388,"tokens_out":3593,"duration_ms":33203,"concrete_test":"Re-run the manual annotation (or a targeted re-labeling) using a revised screening guideline that counts information-seeking clarifying questions as aligned, or at minimum reclassify the 56 \"Inquiry\" instances as aligned and recompute the headline statistics in §III-B. Check whether the misalignment rate stays above 40% and whether \"Lack of Actionability\" remains the most frequent category. If both change (rate ~33%, Context Dependence top), the paper's central empirical claim requires moderation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical claim is the 48% misalignment rate (§III-B). This rate is produced by the instance-screening definition (§III-A3): reviewers label a pair misaligned when \"a model would be expected to generate\" the comment \"having only received the diff as input,\" and the taxonomy then treats \"Inquiry\" comments—\"Asks for clarification, but does not suggest or justify any modification\" (Table I, 56 instances)—as misaligned under \"Lack of Actionability.\" This coding is a normative choice, not an empirical fact: in real code review, clarifying questions are often actionable because they request information that can lead to code changes, and they fall within review intent. The paper even acknowledges in §V that categories correspond to \"legitimate artifacts of real-world code review, including clarification.\" If \"Inquiry\" instances were considered aligned under a broader actionability definition, the misalignment rate would fall from 184/383 (48%) to 128/383 (33%), and the most frequent source would become Context Dependence (61) rather than Lack of Actionability (42 remaining). The paper's central argument that \"current review comment datasets conflate actionable feedback with broader review discourse\" would still hold, but the magnitude and the category emphasis would shift. Because the 48% figure is quoted as the main quantitative result and motivates \"more than dataset cleaning,\" it needs to be robust to alternative reasonable definitions of actionability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates training data for code review comment generation. It manually samples 383 diff-comment pairs from the CodeReviewer dataset and labels each pair as aligned or misaligned, where misalignment means the comment is not actionable review feedback that could be generated from the diff alone. Two annotators achieve κ=0.758; consensus yields 184 misaligned pairs (48%). A card-sorting analysis produces a three-category taxonomy: semantic ambiguity, lack of actionability, and context dependence. The paper then compares three LLM-filtering prompts (a baseline from Liu et al., a criteria-expanded version, and a taxonomy-augmented version) across two LLMs and two input representations, using both the authors' 383-pair sample and Liu et al.'s 270-pair dataset. The taxonomy-augmented prompt does not substantially improve filtering. The authors conclude that dataset cleaning alone is insufficient and call for explicit validity criteria, richer contextual inputs, and evaluation aligned with review intent.","tokens_in":8754,"tokens_out":6496,"duration_ms":56928,"significance":"If the empirical characterization holds, the paper provides a useful reframing: problematic training pairs are not homogeneous noise but reflect systematic misalignment between the task and current diff-comment representations. The taxonomy (semantic ambiguity, lack of actionability, context dependence) is a practical instrument for dataset design. The negative result for taxonomy-guided LLM filtering is an important caution, especially because it is evaluated on an external dataset. The paper is methodologically transparent: it releases a replication package, uses two independent annotators, reports inter-rater reliability, and covers 12 configurations. However, the central 48% estimate is sensitive to a contestable coding of clarification requests as non-actionable, and the filtering comparison lacks statistical inference. These issues need addressing before the magnitude claims can be fully accepted.","major_comments":[{"comment":"The coding of 'Inquiry' comments (56 of 383) as non-actionable under 'Lack of Actionability' is a normative choice. The paper's own screening definition asks whether a model 'would be expected to generate' a comment from a diff; clarifying questions are legitimate review feedback that often lead to changes. The paper even notes in §V that these are 'legitimate artifacts of real-world code review, including clarification.' If the 56 Inquiry instances are reclassified as aligned, the misalignment rate drops from 184/383 (48%) to 128/383 (33%), and the most frequent source becomes Context Dependence (61) rather than Lack of Actionability (42 remaining). The paper should justify the actionability criterion with respect to the intended task and report a sensitivity analysis under alternative definitions. As written, the headline 48% is not robust to a reasonable alternative interpretation.","section":"§III-A3 / Table I"},{"comment":"The claim that taxonomy-augmented prompting does not substantially improve filtering is made on F1 differences without significance tests or confidence intervals. Example: on the Liu et al. dataset with GPT-3.5 and UPNL, SPDC yields Valid F1 0.71 and Noisy F1 0.63, while SPDCT yields 0.69 and 0.60; on the authors' sample, the corresponding comparison is 0.65/0.68 vs. 0.63/0.62. These differences are within sampling variability. Report bootstrapped confidence intervals or a paired test (e.g., McNemar) to support the negative conclusion, or present the results as purely descriptive without the 'did not substantially improve' framing.","section":"§IV-B / Table II"},{"comment":"The random baseline is a single random labeling rather than an expected-random classifier. F1 from one random draw is highly variable; e.g., the reported random baseline on the Liu valid class (0.53) is not the expected value under random labeling. Average over multiple permutations, or provide the theoretical expected F1 given class priors, so the 'beyond chance' comparison is properly calibrated.","section":"§IV-A5 / Table II"}],"minor_comments":[{"comment":"The reference [14] for the card-sorting methodology appears to be about data quality for unit test generation; please check the citation or supply the appropriate methodology reference.","section":"§III-A4"},{"comment":"The label 'our dataset' for the 383-pair annotated sample is misleading; call it 'our annotated sample' since it is not a dataset in the corpus sense.","section":"Throughout"},{"comment":"The yellow highlighting used to indicate modifications may not be visible in monochrome printing; add a textual or boxed indicator.","section":"Fig. 1"},{"comment":"The screening instruction phrase 'a model would be expected to generate' is ambiguous: expected by whom, and for what kind of model? Clarify the intended normative criterion.","section":"§III-A3"},{"comment":"The statement 'GPT-3.5-turbo consistently outperformed GPT-4o-mini' is too strong: while this holds for Valid-class F1, GPT-4o-mini often achieves higher Noisy-class F1 (e.g., SPDCT 4o-mini on the authors' sample, Noisy F1 0.71 vs. 0.62 for GPT-3.5). Qualify the claim by class or metric.","section":"§IV-B"},{"comment":"The green-cell notation 'above the random baseline' should be defined in the caption, especially given the single random draw issue.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The main robustness concern is the actionability definition; I'd suggest a sensitivity analysis in revision. The paper is otherwise solid and within scope for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's real contribution is not the headline number; it's the taxonomy of misaligned diff-comment pairs and the honest negative result that taxonomy-guided prompts don't fix LLM filtering. Second, the 48% figure is real but fragile: it counts clarifying questions as \"non-actionable,\" and the paper doesn't defend that choice. Reclassifying inquiries as valid drops the misalignment rate to about 33% and moves context dependence to the top category. That still supports their main argument — datasets conflate feedback with broader review discourse — but the emphasis changes.\n\nWhat's actually good: careful annotation protocol (two reviewers, κ=0.758, consensus, sample size formula), a clear taxonomy with three top-level categories that map to review behavior, and a thorough comparison of 12 LLM-filtering configurations. The negative filtering result is plausible: adding taxonomy categories to the prompt produces at best marginal F1 changes, and some GPT-4o-mini configurations essentially collapse to labeling everything noisy. The paper doesn't oversell the LLM results. The replication package is promised. Credit where due: this is a real empirical study, not a position paper.\n\nWhere the soft spots are. The load-bearing definitional choice: \"actionable\" is operationalized as \"can be inferred from the diff alone,\" and the screening instruction asks whether a model would be expected to generate the comment given only the diff. That definition enters before the taxonomy, so a good chunk of what is called \"misalignment\" (Inquiry, 56 instances) is a property of the chosen task objective, not of the data. The paper itself calls these \"legitimate artifacts\" in §V. The 48% estimate therefore needs a robustness check against alternative actionability definitions. Second, the LLM filtering comparison reports no significance tests or confidence intervals; several F1 differences are small and inconsistent across datasets, so the \"did not substantially improve\" conclusion is weaker than the wording implies. Third, the paper never trains models on aligned vs. misaligned pairs, so the claim that misaligned pairs degrade learning is asserted, not shown. Those are addressable in revision, but they matter.\n\nWho this is for: anyone building or cleaning datasets for review-comment generation, and anyone evaluating whether LLM-based filtering works. A serious referee would get value from this; the taxonomy is a useful vocabulary and the negative result is a useful checkpoint. I'd recommend sending to peer review with a request to (a) show sensitivity of the 48% rate to the actionability definition, (b) add error bars or significance tests on the filtering comparisons, and (c) either temper or test the learning-degradation claim.","headline":"Useful taxonomy and an honest negative result on LLM filtering, but the headline 48% misalignment rate rides on a contestable definition of 'actionability' that the paper never defends.","tokens_in":9157,"tokens_out":3227,"would_cite":true,"duration_ms":29705,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that nearly half of sampled code-review training pairs misalign with the task of generating actionable feedback from a diff, and that taxonomy-guided prompts barely improve LLM-based filtering.","keywords":["code review","review comment generation","training data quality","misaligned training pairs","dataset cleaning","LLM-based filtering","taxonomy","actionability"],"falsifier":"Re-annotate the 383 sampled pairs with a broader validity criterion that accepts clarifying and conversational comments as valid training signal; if the misalignment share drops below, say, 25%, the paper's central estimate overstates structural misalignment. Alternatively, if a taxonomy-guided filter with few-shot examples reaches F1 above 0.85 on the same sample, the conclusion that taxonomy cues do not help filtering would fail.","tokens_in":8329,"feed_emoji":"🔍","tokens_out":4440,"duration_ms":39556,"temperature":0.7,"pith_summary":"This paper tries to show that the standard way of building training data for automated code-review comment generation—pairing a localized code diff with a human review comment—produces a substantial share of instances that do not teach a model to give actionable feedback. In a random sample of 383 pairs from a widely used dataset, human reviewers judged 184 (48%) as misaligned, and the paper groups them into three categories: semantic ambiguity, lack of actionability, and missing context. It then tests whether making these categories explicit in prompts helps a large language model filter out bad pairs, finding only marginal and inconsistent gains. The upshot is that cleaning a noisy dataset is not enough: the task formulation itself needs richer inputs, explicit validity criteria, and evaluation that rewards actionable comments.","feed_headline":"48% of code-review training pairs give AI the wrong signal","feed_subtitle":"Adding taxonomy hints to LLM prompts barely improves filtering; fixing the task needs richer inputs and explicit validity rules.","key_machinery":"The key objects are the taxonomy of misaligned training pairs—semantic ambiguity, lack of actionability, and context dependence—and the prompt-engineering experiment built on it. The taxonomy operationalizes what 'misaligned' means by classifying why a diff-comment pair fails to supply a reliable learning signal; the experiment (three system prompts, two LLMs, two input representations) tests whether that taxonomy transfers to automatic filtering, thereby exposing the limits of zero-shot prompting for data cleaning.","core_discovery":"The central claim is that a large fraction of diff-comment training pairs are not merely noisy but structurally misaligned with the intended learning objective. In a manual review of 383 randomly sampled pairs, 184 (48%) were judged misaligned: either the comment's intent is ambiguous (25), the comment does not request or justify any change (98), or the comment depends on context outside the diff (61). Incorporating this taxonomy into LLM filtering prompts as definitions, criteria, and categories did not substantially improve filtering over simpler prompts: F1 gains were marginal and inconsistent across models and input representations, and the strongest configuration remained around 0.71 fo","pith_inferences":["If the 48% estimate generalizes, published performance numbers on this dataset likely overstate quality, since many test-time comments also exhibit the same misalignment; evaluating on an aligned-only subset could change rankings.","The taxonomy could be turned into a diagnostic benchmark for agentic review systems: each category suggests a specific capability (e.g., abstain on insufficient context, retrieve previous comments), making the categories directly testable.","The null filtering result may reflect prompt fidelity rather than taxonomy uselessness; few-shot examples per category or fine-tuned classifiers are a natural next test the paper did not run.","A direct implication the paper leaves implicit: training on the aligned subset alone, or adding an 'abstain' option during inference, could avoid forcing models to hallucinate grounded feedback."],"forward_implications":["Models trained on these datasets are rewarded for producing fluent, review-like text even when an instance does not support actionable feedback grounded in the diff, explaining generic or weakly grounded outputs.","Cleaning datasets by removing noisy instances is necessary but not sufficient; the 48% misalignment estimate implies that significant portions of the data encode non-actionable or context-dependent review behavior.","To support actionable feedback, input representations should include review-thread history, references to non-local code, project conventions, or CI signals, rather than relying on diffs alone.","Evaluation of generated review comments should prioritize actionability and review intent over surface-level textual similarity.","LLM-based data filtering with zero-shot taxonomy prompts leaves large room for improvement; better validity criteria and richer prompting or model configurations are needed."],"fun_headline_variants":["Half of code-review training pairs mislead AI","48% of code-review pairs give AI the wrong signal","Taxonomy-aware filtering barely improves code-review training","Code-review data misalignment: filtering alone won't fix","Rethinking code-review data: 48% pairs misaligned"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"A training pair is counted as misaligned only if an actionable comment cannot be inferred from the diff alone; if the intended task includes clarification, coordination, or full-context reasoning, the 48% estimate would shrink.","fun_headline_variants_meta":{"raw":{"variants":["Half of code-review training pairs mislead AI","48% of code-review pairs give AI the wrong signal","Taxonomy-aware filtering barely improves code-review training","Code-review data misalignment: filtering alone won't fix","Rethinking code-review data: 48% pairs misaligned"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1218,"prompt_tokens":731,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":475,"tokens_out":487,"duration_ms":4739,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:16:54.858165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 383 sampled pairs with a broader validity criterion that accepts clarifying and conversational comments as valid training signal; if the misalignment share drops below, say, 25%, the paper's central estimate overstates structural misalignment. Alternatively, if a taxonomy-guided filter with few-shot examples reaches F1 above 0.85 on the same sample, the conclusion that taxonomy cues do not help filtering would fail.","supporting_citations":[],"review_version":1}