{"id":"874a3582-b759-4b7f-a0e8-2368e1a58f53","arxiv_id":"2504.19444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-generated comments are claimed to be more consistent with code than human references, and retraining CodeT5 on an LLM-rebuilt CodeSearchNet improves several code intelligence tasks.","lead":"This paper replaces human-written comments in code datasets with comments generated by GPT-3.5, then retrains a code model on the rebuilt data. The authors report improvements in code summarization, generation, and translation, but some comparisons use AI comments as the scoring reference, which can favor the AI-trained model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3's large downstream gains are confounded: Tables IV-V change fine-tuning data and test references/inputs along with pre-training, so the headline improvement is not attributable to the rebuilt pre-training corpus.","rationale":"The reader's weakest_assumption targets bias in the reference-free metrics (Section III-B1/B2) used for RQ2. I see that as a real but secondary concern: the human evaluation in Section III-E independently rates LLM comments higher on consistency, naturalness, and usefulness, so RQ2 would likely survive even if IncRate and MRR are style-biased. The load-bearing problem for the paper's headline claim is instead in the RQ3 evaluation. The paper's strongest evidence for 'LLM-enhanced data improves NL-code generation' is Table V, but the winning condition changes both pre-training and fine-tuning data and possibly the test inputs; the abstract and RQ3 answer do not qualify this. The same circularity appears in Table IV's rebuilt summarization test set with GPT-3.5 references. The only unconfounded comparisons (pre-training changed, fine-tuning and test fixed) show small, unreplicated gains. A conditional acceptance is appropriate, but it should require the original-test-set reruns described above, release of cgpt-CSN and checkpoints, and variance reporting across seeds. If the reruns confirm the large effects, the paper is stronger; if not, the headline should be narrowed to 'pre-training on LLM-rebuilt comments gives small gains on some tasks' and the reference-free metric claims should be defended against the style-bias objection.","tokens_in":20268,"tokens_out":9129,"duration_ms":90800,"concrete_test":"Rerun Table V's four training conditions with the original CodeXGLUE Concode test set (original human NL inputs and target code) for all cells, using three seeds and reporting CodeBLEU/BLEU/EM with standard deviations. If the cgpt-CSN + cgpt-Concode cell's advantage over CSN + Concode shrinks from +11.04 CodeBLEU and +8.00 EM to roughly the pre-training-only effect (+1.11 CodeBLEU, +0.10 EM), the headline code-generation gain is an artifact of evaluating on rebuilt inputs. Separately, rerun Table IV rows 3-4 on the original CSN test set with human-written references for USE and MRR; if the large advantage vanishes or reverses, the summarization claim reduces to the modest row 1 versus row 2 pre-training effect.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central RQ3 claim (Section IV-B) is not established by the controlled comparisons. In Table V, the large code-generation gains (CodeBLEU 39.45 to 50.49, EM 22.00 to 30.00) come from the cell that changes both pre-training and fine-tuning inputs to GPT-3.5-Turbo text, and the paper does not state whether the Concode test set was also rebuilt. If it was, the evaluation changes the task itself: models are tested on GPT-paraphrased NL queries rather than the original human NL queries, so the improvement can reflect distribution matching to cgpt-CSN and cgpt-Concode rather than better code intelligence. The pre-training-only contrast in the same table (CSN/Concode vs. cgpt-CSN/Concode) is only +1.11 CodeBLEU and +0.10 EM. Table IV has the same structure: the two large summarization improvements are in rows where the fine-tuning data and the USE references are GPT-3.5-Turbo comments on a rebuilt test set, so the evaluation rewards imitation of the LLM comment distribution. No variance or significance testing is reported for any of these comparisons, so the remaining valid pre-training-only effects are too small to support 'significantly improves' as stated in the abstract and RQ3 answer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes two reference-free metrics—code-comment inconsistency detection (IncRate) and semantic code search (MRR)—to compare LLM-generated comments with human-written reference comments. The authors report that LLM-generated comments are more semantically consistent with code than human references, supported by automatic evaluation and a 357-sample human study, and then act on this finding by rebuilding CodeSearchNet with GPT-3.5-Turbo-generated comments (cgpt-CSN). They pre-train CodeT5 on this rebuilt corpus and evaluate it on code summarization, NL-code generation, code translation, code refinement, and clone detection, concluding that LLM-rebuilt pre-training data significantly improves NL-related code intelligence tasks and also enhances code translation.","tokens_in":20548,"tokens_out":5202,"duration_ms":49077,"significance":"If the central claim were established, the paper would make a useful contribution: it provides a large rebuilt pre-training corpus (about 2.07M PL-NL pairs), a re-implementation of CodeT5 pre-training, and a concrete proposal to replace human-written comments with LLM-generated ones in pre-training datasets. The human evaluation and the scale of the data-rebuilding effort are strengths. However, the downstream evaluation as currently reported does not isolate the effect of the rebuilt pre-training data: the largest gains come from also changing the fine-tuning data and the test references or inputs, and no variance or significance testing is reported. The reference-free metrics may also be format-biased in favor of one-sentence LLM comments. With additional controlled experiments and more cautious claims, this could become a solid empirical study; in its current form, the evidence is insufficient for the abstract's strong conclusion.","major_comments":[{"comment":"The USE references for the cgpt-CSN-Sum rows are GPT-3.5-Turbo-generated comments, as the paper itself states. The model fine-tuned on GPT-3.5-Turbo comments is then scored against GPT-3.5-Turbo references, so the large USE increases (e.g., average 0.5941 to 0.7777 for CSN pre-training) largely reflect distribution matching rather than improved summarization quality. The pre-training-only comparison in the same table (CSN/CSN vs. cgpt-CSN/CSN) shows a much smaller effect (average USE 0.5941 vs. 0.6002). Please report the pre-training-only contrast on the original human-reference test set, or on both reference sets, and include variance or significance tests.","section":"Section IV-B, Table IV"},{"comment":"The Concode NL inputs are replaced with GPT-3.5-Turbo-generated text to form cgpt-Concode, but the manuscript does not state whether the test split is also rebuilt. If the test inputs are rebuilt, the evaluation task changes: models are tested on GPT-paraphrased queries rather than original human NL queries, so the gain from 39.45 to 50.49 CodeBLEU cannot be attributed to the rebuilt pre-training corpus. The only pre-training-only contrast (CSN/Concode vs. cgpt-CSN/Concode) is +1.11 CodeBLEU and +0.10 EM, which is not shown to be significant. Please evaluate on the original Concode test set and explicitly state which splits were rebuilt, and report error bars or significance tests.","section":"Section IV-B, Table V"},{"comment":"The two reference-free metrics may be systematically biased toward the LLM comment style. The CCID classifier (Section III-B1) was trained on commit-derived Javadoc labels from Panthaplackel et al., whereas the LLM comments are one-sentence prose generated with the prompt in Section III-C2; the classifier may flag structured Javadoc differently from one-sentence summaries. The MRR metric uses CodeBERT fine-tuned on CodeSearchNet human comments, so retrieval may favor query styles close to that distribution. Because RQ2 is the basis for rebuilding cgpt-CSN, please validate the two metrics on LLM-format comments or provide calibration evidence across comment formats.","section":"Section III-B (RQ2)"},{"comment":"The code translation and negative-result claims are based on single runs without variance or significance testing. For instance, Java-to-Python EM changes from 2.57 to 2.60 (Table VI), and refinement results differ by less than 0.2 EM (Table VII), yet the text describes improvements as meaningful and the non-improvements as non-significant. Please add repeated-seed results or confidence intervals and either provide significance tests or substantially soften the significance language in the abstract and the RQ3 answer.","section":"Section IV-B, Tables VI-VIII"}],"minor_comments":[{"comment":"The metric name is inconsistent: the table header says \"InRate\" while Equation (1) and the surrounding text use \"IncRate.\" Please unify the notation.","section":"Table II and Section III-D"},{"comment":"There is a typo in the sentence \"six LLMs achieve lower IncRate and higher MMR scores than all three DL baselines\": MMR should be MRR.","section":"Section III-D3"},{"comment":"The sentence \"This none non-significant difference can be attributed...\" is grammatically garbled; it should be rephrased.","section":"Section IV-B, Clone Detection paragraph"},{"comment":"There are minor typos: \"the the National Key Research and Development Program\" in the footnote, and \"tempreture\" in the sampling-parameter description.","section":"Acknowledgments and Section III-C2"},{"comment":"In the reference-free metric analysis, \"CSN performs the lowest MRR score\" should presumably read \"NCS performs the lowest MRR score.\"","section":"Section III-D2"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is not outside the journal's scope and the reference-free evaluation direction is reasonable. My main concern is purely evidential: the downstream claims in Section IV are confounded by simultaneous changes to fine-tuning data and evaluation references/inputs, and the pre-training-only effects are small and unaccompanied by significance testing. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I also see no obvious novelty-disclosure problem; the main weaknesses are experimental design and presentation, not scope fit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the CCID and semantic-code-search framing is a genuinely useful contribution, and the TLC-stage comparison is internally consistent. But the downstream section overclaims. The big numbers in Tables IV and V come from changing the fine-tuning data and the test set/references to GPT-3.5-turbo text, so the headline result—that rebuilding pre-training data significantly helps—is not actually demonstrated. The pre-training-only differences are small and have no error bars or significance tests.\n\nWhat the paper does well: using a commit-trained inconsistency classifier and retrieval MRR as reference-free evaluations of comment quality is a sensible move for a setting where no gold reference exists. On TLC the pattern is coherent: all six LLMs beat human references on both proxies, and the human evaluation of 357 samples aligns with the automatic results. The authors also honestly report null effects on refinement and clone detection, and they acknowledge in the text that the fine-tuning phase contributes more than pre-training. The literature coverage is adequate.\n\nThe soft spots are real. In Table IV, the large USE jump appears in rows where the fine-tuning data and the USE reference are GPT-3.5-turbo comments on a rebuilt test set; that measures imitation of a target distribution, not better summarization. In Table V, the 39.45 to 50.49 CodeBLEU gain is in the cell that changes both pre-training and fine-tuning inputs, and the test set is presumably rebuilt too. The valid pre-training-only contrast in the same table is about +1 CodeBLEU and +0.1 EM. The translation results (Table VI) are the cleanest pre-training-only evidence, and they are moderate. Second, the two reference-free metrics may be biased toward one-sentence LLM comment style: the CCID classifier was trained on commit-pair Javadoc data, and CodeBERT retrieval may prefer short queries. The human eval partially offsets this for RQ2, but the paper should probe the bias. Third, no artifacts are released—no cgpt-CSN data, no pre-training code, no checkpoints—so the main practical contribution is not verifiable without a lot of expense. The paper also re-implements CodeT5 from scratch, and we do not know how that re-implementation compares to the original.\n\nWho is this for? People working on code comment evaluation, synthetic training data, or dataset quality for code models. The TLC-stage method is reusable; the downstream claims need rework. I would send it to serious peer review, but I would expect reviewers to require a major revision: re-run the downstream comparisons on unchanged test sets with human references, report variance, and release the dataset and code.","headline":"The reference-free evaluation idea is the real contribution; the downstream RQ3 claim overstates what the controlled comparisons actually show.","tokens_in":21103,"tokens_out":5564,"would_cite":true,"duration_ms":55159,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing human-written code comments with LLM-generated comments in pre-training data improves code models on natural-language tasks, and proposes two reference-free metrics to establish the comparison.","keywords":["code comment quality","reference-free evaluation","code-comment inconsistency detection","semantic code search","pre-training dataset rebuilding","LLM-generated comments","code summarization","code intelligence"],"falsifier":"Rewrite a sample of human-written Javadoc comments into the same one-sentence, API-free summary format that the LLM prompt produces, re-run the inconsistency detector and semantic search on the style-matched comments, and check whether their scores on the two reference-free measures become comparable to LLM-generated comments. If they do, the claimed LLM superiority is a style artifact of the metrics rather than a semantic-quality difference.","tokens_in":20065,"feed_emoji":"🤖","tokens_out":6803,"duration_ms":64085,"temperature":0.7,"pith_summary":"The paper argues that the human-written comments used to pre-train code models are not the gold standard they are assumed to be, and that comments generated by large language models can serve as a better bridge between code and natural language. It introduces two reference-free evaluation tasks—flagging code-comment mismatches and using comments as search queries—to compare comment quality directly, and finds LLM-generated comments look more consistent with code than human comments. On that basis, it replaces the human comments in a widely used code-comment corpus with LLM-generated ones, re-trains a standard pre-trained code model, and reports gains on code summarization, natural-language-to-code generation, and code translation, while code refinement and clone detection are unaffected. The upshot is a challenge to the routine reliance on human reference comments and a recipe for rebuilding pre-training data with LLMs.","feed_headline":"LLM-written comments beat human comments for training code models","feed_subtitle":"Rebuilt pre-training data with generated comments improves summarization, generation, and translation tasks.","key_machinery":"Two auxiliary reference-free evaluation tasks carry the argument. The first, an inconsistency rate, uses a trained code-comment inconsistency detector to measure what fraction of code-comment pairs are semantically mismatched, with lower values indicating better comments. The second, a retrieval rank score, uses a semantic code-search model with each comment as a query and measures where the correct code snippet ranks among distractors, with higher values indicating better comments. Both are used to compare comment quality without treating human comments as the gold standard. The same comparison then justifies rebuilding the pre-training corpus: the paper generates over two million comments with one LLM and re-pre-trains a widely used encoder-decoder code model on the resulting pairs.","core_discovery":"The paper's central discovery is that the semantic link between code and natural language in pre-training data can be improved by discarding human-written comments and replacing them with one-sentence comments generated by a large language model. On two reference-free measures—how often a trained classifier labels a code-comment pair as inconsistent, and how well the comment retrieves its own code in a semantic search—LLM-generated comments are far less inconsistent and far more retrievable than the original human comments, with inconsistency rates below four percent versus roughly fifteen percent and retrieval scores up to about 0.97 versus 0.82. Re-pre-training an encoder-decoder code model on the rebuilt corpus improves downstream code summarization, natural-language-to-code generation, and code translation, while tasks driven by code structure alone, such as refinement and clone detection, show no meaningful change. The paper reads this as evidence that large language models are qualified builders of code-intelligence benchmarks and that the field's reliance on human reference comments should be reconsidered.","pith_inferences":["Editorial inference: the same rebuilding logic likely transfers to other code-natural-language paired resources, such as API documentation, notebooks, and code search queries, since the mechanism is semantic consistency rather than the specific corpus.","Editorial inference: a testable consequence is that the gains should be largest where pre-training comments are stale or noisy; if a corpus is already clean, LLM replacement may be neutral or even harmful, and the paper's null results on structure-only tasks already suggest a ceiling.","Editorial inference: style-matching experiments, where human comments are rewritten in the same one-sentence, API-free format the LLM prompt produces, would isolate whether the advantage is semantic or stylistic; if style is the driver, the conclusion should be reinterpreted as \"LLM style suits the metrics\" rather than \"LLM comments are better.\"","Editorial inference: the inconsistency detector could be repurposed as a data-cleaning filter, flagging human comments before they ever enter a pre-training corpus."],"forward_implications":["Pre-training corpora can be improved by replacing human-written comments with LLM-generated ones, since semantic consistency between code and natural language is the active ingredient.","Natural-language-heavy downstream tasks inherit the gain, while structure-only tasks do not, which demarcates where comment quality actually matters.","Reference-free metrics are necessary for comparing comment quality at all, because reference-based metrics assume the human reference is already the gold standard.","Rebuilding the dataset should be treated as a component of the model-building pipeline, on par with architecture and pre-training objective choices."],"supporting_citations":[{"why":"Defines the encoder-decoder pre-trained model architecture used for the re-pre-training experiment.","marker":"[13]"},{"why":"Provides the large code-comment corpus whose human comments are replaced by LLM-generated ones.","marker":"[21]"},{"why":"Provides the commit-pair construction and labels used to train the inconsistency detector.","marker":"[52]"},{"why":"Supplies the classification model used to compute the inconsistency-rate metric.","marker":"[53]"},{"why":"Supplies the semantic code-search model and configuration used to compute the retrieval rank metric.","marker":"[11]"},{"why":"Provides the Java code-comment test set used for the direct quality comparison between human and generated comments.","marker":"[20]"},{"why":"Establishes that noisy comments degrade code-model performance, motivating the dataset-rebuilding approach.","marker":"[6]"},{"why":"Defines the downstream code intelligence tasks used to evaluate the re-pre-trained model.","marker":"[48]"},{"why":"Provides the fine-tuning dataset for natural-language-to-code generation.","marker":"[74]"},{"why":"Provides the parallel corpus used to evaluate code translation.","marker":"[76]"}],"fun_headline_variants":["LLM comments beat human ones for code pre-training","Rebuilt code data with LLM comments boosts model performance","Code models improve when comments are LLM-generated","LLM-generated comments outperform human in code training","Swap human comments for LLM to improve code AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the two reference-free measures judge semantic consistency fairly rather than simply preferring the short, API-agnostic one-sentence style that LLMs output over the detailed Javadoc style that humans write.","fun_headline_variants_meta":{"raw":{"variants":["LLM comments beat human ones for code pre-training","Rebuilt code data with LLM comments boosts model performance","Code models improve when comments are LLM-generated","LLM-generated comments outperform human in code training","Swap human comments for LLM to improve code AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1477,"prompt_tokens":922,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":538,"tokens_out":555,"duration_ms":5718,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:52:33.470148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rewrite a sample of human-written Javadoc comments into the same one-sentence, API-free summary format that the LLM prompt produces, re-run the inconsistency detector and semantic search on the style-matched comments, and check whether their scores on the two reference-free measures become comparable to LLM-generated comments. If they do, the claimed LLM superiority is a style artifact of the metrics rather than a semantic-quality difference.","supporting_citations":[{"cited_title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,","cited_arxiv_id":null,"evidence_quote":"Defines the encoder-decoder pre-trained model architecture used for the re-pre-training experiment."},{"cited_title":"Deep just- in-time inconsistency detection between comments and source code,","cited_arxiv_id":null,"evidence_quote":"Provides the commit-pair construction and labels used to train the inconsistency detector."},{"cited_title":"Codebert: A pre-trained model for programming and natural languages,","cited_arxiv_id":null,"evidence_quote":"Supplies the semantic code-search model and configuration used to compute the retrieval rank metric."},{"cited_title":"Summarizing source code with transferred api knowledge,","cited_arxiv_id":null,"evidence_quote":"Provides the Java code-comment test set used for the direct quality comparison between human and generated comments."},{"cited_title":"On the importance of building high-quality training datasets for neural code search,","cited_arxiv_id":null,"evidence_quote":"Establishes that noisy comments degrade code-model performance, motivating the dataset-rebuilding approach."},{"cited_title":"Mapping language to code in programmatic context,","cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuning dataset for natural-language-to-code generation."}],"review_version":1}