{"id":"bdd382b6-2f0d-4d0b-b840-4b72d4ac0f08","arxiv_id":"2607.09524","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Selective LLM reformulation guided by high-quality exemplars yields cleaner, more diverse code-review datasets that improve downstream comment generation and code refinement.","lead":"Two LLM pipelines clean noisy code-review comments into higher-quality, more diverse datasets (CuREV and CuREV+). The cleaned data measurably improves automated comment generation and code-refinement models while keeping more natural human style.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Downstream gains may partly reflect easier n-gram targets rather than genuinely more useful comments, because the same LLM family both rewrites and judges quality.","rationale":"The reader correctly flags the dual use of LLMs as rewriter and judge plus the modest human validation as the weakest assumption. That assumption is load-bearing for both the quality scores and the interpretation of the downstream BLEU/CodeBLEU lifts. The paper already ships data, code, and a sensitivity table, so the concern is not fatal; it simply keeps the verdict at CONDITIONAL until an independent human usefulness study is performed. No stronger internal inconsistency was found.","tokens_in":27566,"tokens_out":517,"duration_ms":7548,"concrete_test":"Have 3–5 professional developers, blinded to condition, rate a stratified sample of 200 original / CuREV / CuREV+ triples on usefulness for implementing the change (1–5 Likert) and produce the corresponding code edit; compute inter-rater agreement and correlation of human usefulness with the paper’s LLM scores and with CodeBLEU/EM. If human usefulness does not rise (or rises far less than the automated metrics), the usefulness half of the claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CuREV/CuREV+ comments are higher-quality and more useful, evidenced by (a) LLM-as-judge score lifts (clarity ~6.89\to8.95, conciseness 7.71\to8.53, 100 % civil) and (b) higher BLEU / CodeBLEU / EM on comment generation and code refinement. Both (a) and (b) rest on the same family of models (Llama-3.1-70B for scoring and reformulation; DeepSeek-Coder for the downstream tasks). The human sanity check covers only 100 samples and reports only kappa, not absolute score calibration; the multi-LLM correlation for conciseness is only ρ≈0.34. Consequently the reported quality gains and the BLEU gains (especially CuREV’s formulaic “Consider …” openings) could be partly circular: the rewriter produces text that the judge and the n-gram metrics prefer, without an independent demonstration that developers find the comments more actionable. The selective CuREV+ design mitigates homogenization but does not break the circularity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces two LLM-based curation pipelines for large-scale code-review comment datasets (Li et al. 2022). CuREV filters low-relevance comments and reformulates all remaining ones with Llama-3.1-70B for clarity, conciseness and civility while preserving intent. CuREV+ first partitions into high- and low-quality subsets via the same multi-dimensional evaluation framework (Type/Nature/Civility + Relevance/Clarity/Conciseness scores), retains high-quality human comments unchanged, and reformulates only the low-quality ones using randomly sampled high-quality comments as in-context exemplars. Both curated datasets (size 170 718) are re-scored with the framework, analysed for lexical/stylistic diversity (TTR, n-gram diversity, Self-BLEU, entropy, prefix n-grams), and used to fine-tune or prompt DeepSeek-Coder-6.7B on comment generation (BLEU) and code refinement (CodeBLEU + Exact Match). Results claim substantial quality lifts (clarity ~6.89\to8.95, conciseness 7.71\to8.53, 100 % civil), higher diversity for CuREV+, and consistent downstream gains (BLEU 7.71\to11.05–11.26; CodeBLEU 0.36\to0.44–0.49; EM 408\to445–463).","tokens_in":27938,"tokens_out":1324,"duration_ms":29979,"significance":"If the claims hold, the work supplies two publicly released, higher-quality training resources (CuREV / CuREV+) that measurably improve two core automated code-review tasks and offers a practical selective-curation recipe that trades off standardisation against natural stylistic diversity. Strengths that raise confidence include the open HuggingFace/GitHub/Zenodo artefacts, the multi-LLM cross-validation (Llama + MiniMax + GPT-5-nano Spearman correlations), the human sanity check (Cohen’s κ 0.64–1.0, quadratic weighted near-perfect), the threshold sensitivity table, BERTScore semantic-preservation numbers, and the end-to-end task improvements obtained with a different model family (DeepSeek). These elements make the contribution immediately usable by the SE community and provide a concrete template for future dataset-curation studies.","major_comments":[{"comment":"Sections 3.3–3.5 and 4.1: the identical LLM-as-a-Judge framework (and largely the same model family) is used both to decide which comments are “poor” and therefore rewritten and later to quantify the quality gains of the rewritten set. Although a 100-sample human check and two secondary LLMs are supplied, the absolute score calibration remains weak (especially conciseness Spearman ρ≈0.34) and the human sample is not stratified by language or type. This circularity is load-bearing for the central quality claims; an independent larger human rating or an alternative non-LLM quality metric on a held-out stratified sample is needed to confirm that the reported lifts (clarity +30 %, conciseness +10.6 %) are not artefacts of the judge’s own stylistic preferences.","section":"§3.3–3.5, §4.1, Tables 5–6"},{"comment":"Section 6.1 and Tables 15–16: CuREV’s BLEU gain (7.71\to11.26) coincides with extreme lexical homogenisation (“consider” appears as the opening token of ~126 k comments). The paper itself documents this formulaic pattern; consequently a non-negligible fraction of the n-gram improvement may be surface-form rather than semantic. Reporting BERTScore (or another embedding-based metric) between generated and reference comments, or an ablation that normalises for the dominant prefix, is required before the claim that “curated comments enable more accurate review-comment generation” can be fully accepted.","section":"§6.1, Tables 15–16"},{"comment":"Section 6.2 / RQ5 and §7.1: usefulness is operationalised solely by CodeBLEU and Exact Match of the refined code. While these are reasonable automatic proxies, the manuscript acknowledges the absence of any developer perception or actionability study. Because the title and abstract emphasise “usefulness”, a modest human evaluation (even on a few hundred comments) or a clearer discussion of the proxy’s limitations is necessary for the usefulness claim to stand.","section":"§6.2, §7.1"}],"minor_comments":[{"comment":"Figure 3 reports Refactoring 69.7 % while the accompanying text (p. 9) states 80.07 %. Because Type is multi-label the percentages need not sum to 100, yet the two numbers should be reconciled or the multi-label nature made explicit in the figure caption.","section":"Fig. 3 / §3.6"},{"comment":"Table 10 and the surrounding prose mix absolute scores with relative percentages; a single consistent reporting style (e.g., absolute + Δ) would improve readability.","section":"Table 10"},{"comment":"The prompt excerpts in Tables 4, 8 and 9 are helpful but truncated; the full prompts should be placed in an appendix or the replication package with a clear pointer.","section":"Tables 4, 8, 9"},{"comment":"A few minor typographical inconsistencies appear (e.g., “prescreptive”, “mmmm welcome…”). A final proof-reading pass is recommended.","section":"throughout"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is real but partially mitigated; I would not reject on that ground alone. The public datasets and the selective-exemplar idea are genuine contributions. If the authors can add a modest independent human rating and a semantic metric for the generation task, the paper becomes a clear accept for an empirical SE venue. Scope fit is excellent."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The new piece is CuREV+: keep the high-quality human comments, rewrite only the poor ones with rotating high-quality exemplars. That is a clean, practical fix for the homogenization problem they created in their own earlier CuREV work, and they ship both datasets plus code.\n\nWhat they do well is the full pipeline. They define a multi-label evaluation framework (type/nature/civility + 1–10 relevance/clarity/conciseness), run a 100-sample human sanity check with solid kappas (0.64–1.0, quadratic weighted near-perfect on the ordinal scores), add two independent secondary judges, supply threshold sensitivity tables, and report BERTScore to show semantic preservation. Downstream, DeepSeek-Coder fine-tuned or prompted on the curated data improves BLEU on comment generation (7.71 → 11.05/11.26) and CodeBLEU/Exact Match on code refinement (0.36/408 → 0.49/463). Diversity metrics (n-gram, entropy, self-BLEU, prefix analysis) correctly show that CuREV collapses into “Consider …” while CuREV+ keeps more natural openings. The artifacts are public and the experimental design is careful about paired samples and identical hyperparameters.\n\nSoft spots are real but proportional. The same LLM family both rewrites and judges, so some of the clarity/conciseness lift and the BLEU lift (especially CuREV’s formulaic openings) could be self-reinforcing; multi-LLM Spearman on conciseness is only ~0.34 and the human check is only 100 samples with no absolute score calibration. Downstream metrics are still n-gram based, so they do not fully prove developer actionability. Thresholds (relevance ≥4, quality ≥7) are free parameters, though the sensitivity table shows the chosen point is not pathological. None of this collapses the central claim; it just keeps the paper in the “useful engineering result with residual circularity” category rather than a definitive quality oracle.\n\nThis is for people building review bots or cleaning SE datasets. It deserves a serious referee. I would cite the datasets and the selective-exemplar idea; I would not treat the absolute quality numbers as ground truth without further human validation.","headline":"Solid, usable dataset-curation paper for code-review automation; CuREV+ is a real methodological step, with residual LLM-judge circularity that is acknowledged and only partly mitigated.","tokens_in":28440,"tokens_out":563,"would_cite":true,"duration_ms":7068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Selective LLM reformulation of code-review comments yields comments that are both clearer and more human-like, improving automated generation and code refinement.","keywords":["code review","dataset curation","large language models","comment generation","code refinement","diversity","LLM-as-a-judge","software maintenance"],"falsifier":"Train identical models on the original Li et al. 2022 dataset versus CuREV+ and measure whether the claimed gains in BLEU (comment generation) and CodeBLEU/Exact-Match (code refinement) disappear, or run a human study showing that CuREV+ comments are judged less natural or less useful than the original comments.","tokens_in":28510,"feed_emoji":"💬","tokens_out":931,"duration_ms":10014,"temperature":0.7,"pith_summary":"Existing public code-review datasets are noisy: comments are often vague, verbose, uncivil, or formulaic, so models trained on them produce weak feedback. This paper shows that two LLM-based curation pipelines can fix that. The first pipeline (CuREV) rewrites every retained comment for clarity, conciseness and civility; the second (CuREV+) keeps already-good human comments untouched and rewrites only the weak ones, using the good ones as in-context style exemplars. Both raise average clarity from ~6.9 to ~9 and eliminate uncivil language, but CuREV+ also restores lexical and stylistic diversity that pure rewriting destroys. Models trained or prompted with the curated comments produce better review text (BLEU rises from 7.7 to ~11) and more accurate code changes (CodeBLEU 0.36\to0.49). The central claim is therefore practical: dataset quality, not just model size, is the bottleneck for automated code review, and a selective, exemplar-guided rewrite is enough to remove that bottleneck while keeping comments natural.","feed_headline":"Selective LLM rewrites make review comments clearer and more human","feed_subtitle":"CuREV+ keeps good human comments, rewrites only the weak ones, and lifts both generation and code-fix accuracy","key_machinery":"The dual-pipeline curation method: first an LLM-as-judge scores every comment on type/nature/civility/relevance/clarity/conciseness; then CuREV+ keeps comments above threshold unchanged and uses them as rotating in-context exemplars while the same LLM rewrites only the remaining low-quality comments.","core_discovery":"A selective, exemplar-guided curation pipeline (CuREV+) that preserves high-quality human comments and rewrites only low-quality ones produces review comments that are simultaneously clearer, more concise, fully civil, and stylistically diverse; models trained or prompted on this data measurably outperform those trained on the raw original dataset for both comment generation and code refinement.","pith_inferences":["If the same pipeline were run with multiple independent LLMs as a jury rather than a single judge, residual self-preference bias in reformulation style might shrink further.","The diversity metrics (lower self-BLEU, richer prefixes) suggest CuREV+ could reduce mode collapse when used for reinforcement-learning-from-human-feedback of review agents.","Because the method already separates relevance from form, it could be extended to multi-language review datasets without requiring new quality rubrics for each language."],"forward_implications":["Future code-review models can be trained on CuREV+ (or similarly curated data) and expect higher generation quality without larger architectures.","The same selective-rewrite pattern can be applied to other noisy software-engineering text corpora (issue reports, commit messages, forum answers).","Preserving a core of authentic high-quality human comments while rewriting the rest is a practical recipe for balancing standardization and naturalness in any instruction-tuning dataset.","Automated quality scoring of review comments becomes a reusable pre-processing step rather than an after-the-fact evaluation."],"fun_headline_variants":["Exemplar-guided LLM curation keeps strong reviews, rewrites weak ones","Selective rewrites yield clearer, more diverse code-review comments","CuREV+ preserves human quality while fixing noisy review data","High-quality exemplars lift LLM review comment generation and fixes","Curate only low-quality reviews to train better code-review models"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a single large language model (with two secondary models for cross-check) can reliably score the six quality dimensions and that the chosen numeric cut-offs cleanly separate useful from useless comments.","fun_headline_variants_meta":{"raw":{"variants":["Exemplar-guided LLM curation keeps strong reviews, rewrites weak ones","Selective rewrites yield clearer, more diverse code-review comments","CuREV+ preserves human quality while fixing noisy review data","High-quality exemplars lift LLM review comment generation and fixes","Curate only low-quality reviews to train better code-review models"]},"model":"grok-4.5","effort":"low","cost_usd":0.004964,"raw_usage":{"total_tokens":1476,"prompt_tokens":877,"num_sources_used":0,"completion_tokens":91,"cost_in_usd_ticks":49640000,"prompt_tokens_details":{"text_tokens":877,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":508,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":877,"tokens_out":91,"duration_ms":6304,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T02:23:08.433902+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train identical models on the original Li et al. 2022 dataset versus CuREV+ and measure whether the claimed gains in BLEU (comment generation) and CodeBLEU/Exact-Match (code refinement) disappear, or run a human study showing that CuREV+ comments are judged less natural or less useful than the original comments.","supporting_citations":[],"review_version":1}