{"id":"c2532361-23b9-4aae-8225-e050b1e4f22c","arxiv_id":"2507.12558","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAGSum couples retriever and generator in one CodeT5 model with contrastive pre-training, joint fine-tuning, and ROUGE-L based self-refinement, outperforming CMR-Sum, JOINTCOM, and Llama-3.1-8B on three code-comment datasets.","lead":"This paper presents RAGSum, a model that combines code retrieval and comment generation in a single CodeT5 network, trained jointly with contrastive pre-training and a self-refinement step. The authors report improved BLEU and related scores over three baselines on Java, Python, and C code-comment benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significance testing or variance reporting is provided; the claimed 'substantial' gains over JOINTCOM on the C dataset (Table II: C-BLEU 27.95 vs 26.32, ROUGE-L 47.35 vs 46.15) are small point estimates that could be within run-to-run noise, so the central SOTA claim is not yet settled.","rationale":"The reader's weakest assumption focuses on baseline fairness, specifically that the frozen Llama-3.1-8B RAG baseline receives less training effort than RAGSum. That is a legitimate concern, but it is secondary because the paper transparently states the Llama configuration (Section IV-A: 'the LLM serves as the generator in the RAG framework, with one-shot and few-shot exemplars retrieved using CodeT5 embeddings'), so the comparison is at least explicit. The more load-bearing concern is the absence of any statistical evidence for the headline claim. Table II reports one run per model, and the smallest margins are on CCSD, where RAGSum beats JOINTCOM by 1.63 C-BLEU points and 1.20 ROUGE-L points. For neural fine-tuning, these gaps are small enough that a different random seed or a slightly different hyperparameter choice could reverse them. The paper itself labels the results 'preliminary,' which strengthens the need for error bars or significance tests before claiming a new state of the art. A paired bootstrap or multi-seed rerun directly settles whether the C-dataset advantage is real, and it also provides a template for evaluating the Java and Python results more rigorously. I therefore agree with the reader's CONDITIONAL verdict: the approach is plausible and the ablation is informative, but the central empirical claim is not yet fully supported. My recommendation is UNCHANGED because the conditional verdict already captures this limitation.","tokens_in":10280,"tokens_out":6309,"duration_ms":77724,"concrete_test":"On the CCSD benchmark, run RAGSum and JOINTCOM (and CMR-Sum if feasible) with at least 5 different random seeds, using identical data splits, decoding parameters, and evaluation scripts. Compute per-example BLEU and ROUGE-L scores and apply a paired bootstrap (or Wilcoxon signed-rank) test to the mean differences. If the 95% confidence interval for the C-BLEU or ROUGE-L difference includes zero, the claim that RAGSum consistently outperforms JOINTCOM on C is unsupported and the SOTA statement should be weakened to 'numerically higher in a single run'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RAGSum 'substantially outperforms' the baselines on all three benchmarks rests on single point estimates in Table II with no error bars, no repeated seeds, and no significance testing. The most load-bearing case is the C dataset (CCSD), where the margin over the strongest fine-tuned baseline JOINTCOM is small: C-BLEU 27.95 vs 26.32, Sentence-BLEU 21.36 vs 19.99, ROUGE-L 47.35 vs 46.15, METEOR 23.76 vs 22.82, and CIDEr 3.03 vs 2.91. These differences may be within normal seed-to-seed variation for a transformer fine-tuning run, especially given the paper calls its results 'preliminary' in the abstract and Section V-B asserts 'substantially outperforms' without any inferential statistics. The same issue affects the ablation study in Table III, where the self-refinement contribution is only 0.10-0.43 C-BLEU points. Notably, this concern is independent of the baseline-fairness issue raised by the reader: even if the Llama-3.1-8B configuration is transparently described as frozen and under-trained, the comparison against JOINTCOM on CCSD is the decisive test of the claimed universal superiority, and that comparison lacks statistical support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes RAGSum, a code-comment generation model that couples retrieval and generation in a single CodeT5 backbone. The approach has three stages: contrastive pretraining of the encoder for code-code and code-comment retrieval, joint fine-tuning of the retriever and generator with a weighted generation loss, and a self-refinement loop that augments training data with model-generated comments selected by ROUGE-L against ground truth. The authors evaluate on JCSD, PCSD, and CCSD, comparing against CMR-Sum, JOINTCOM, and Llama-3.1-8B in a RAG setting, and report improvements on BLEU, ROUGE-L, METEOR, and CIDEr, with ablations showing the contribution of each component. The paper is explicitly framed as preliminary but also claims to set a new state of the art in comment generation.","tokens_in":10615,"tokens_out":4440,"duration_ms":55411,"significance":"If the reported gains are reproducible, RAGSum would be a useful contribution to retrieval-augmented code summarization: unifying retriever and generator in one backbone is a natural design, the contrastive pretraining objective is reasonable, and the authors provide a replication package. The significance is moderated, however, by the absence of any uncertainty quantification, the small margins over the strongest fine-tuned baseline on CCSD, and an unbalanced comparison with the frozen Llama baseline. These issues mean the central state-of-the-art claim is not yet established, although the underlying architecture and evaluation framework are sound enough to warrant further work.","major_comments":[{"comment":"The central claim that RAGSum substantially outperforms the baselines rests on single point estimates with no error bars, no repeated seeds, and no significance testing. On CCSD, the margin over JOINTCOM is C-BLEU 27.95 vs. 26.32 and ROUGE-L 47.35 vs. 46.15, which could plausibly fall within run-to-run variation for transformer fine-tuning. The abstract and the answer to RQ2 use the word 'substantially' even though the paper elsewhere calls the results preliminary. Please report at least 3-5 independent runs and include paired significance tests (e.g., bootstrap or Wilcoxon) over the test set, at least for the decisive RAGSum vs. JOINTCOM comparison.","section":"Section V-B, Table II"},{"comment":"The Llama-3.1-8B baselines are not trained on the benchmark data: the model is used as a frozen generator with CodeT5 embeddings for retrieval, while RAGSum is fine-tuned on the full training set. This is an unequal comparison, and the resulting gap does not establish that RAGSum is superior to an equivalently trained LLM. The paper should either fine-tune or adapt the LLM baseline under the same training budget, or explicitly relabel these rows as zero-shot/few-shot upper-bound references and exclude them from the state-of-the-art comparison.","section":"Section IV-A, Table II"},{"comment":"The abstract states that the composite objective 'rewards accurate top-k retrieval,' but the joint loss in Eq. (3) is a generation cross-entropy loss weighted by retrieval similarity; there is no retrieval-ranking term in the joint phase. Retrieval-specific learning appears only in the contrastive pretraining of Eqs. (1)-(2). This discrepancy is load-bearing for the claimed tight coupling of retriever and generator, and the paper should either add an explicit retrieval loss to the joint objective or revise the claim.","section":"Section III-B, Eq. (3); also Abstract"},{"comment":"The self-refinement process is under-specified and its reported contribution is small: Table III shows that removing self-refinement changes C-BLEU by 0.10-0.43 points across datasets. The number of candidates K, the sampling/decoding strategy, and the temperature during candidate generation are not given, which limits reproducibility. Also, because the selection criterion is ROUGE-L against the ground-truth comment on the training set, the practice is not circular, but the authors should clarify that this is a self-training procedure and report the variance of the refinement gain, as the observed differences may not be significant.","section":"Section III-C and Table III"}],"minor_comments":[{"comment":"The abstract contains the typo 'ROUTE-L' and should read 'ROUGE-L'; also, 'substantially outperforms' is inconsistent with the paper's own characterization of the results as preliminary.","section":"Abstract"},{"comment":"The text says L_q2c 'aims to minimize the similarity between the input code q_i and its corresponding comment c_i,' but the equation maximizes this similarity; the wording should be corrected.","section":"Section III-A, Eq. (2)"},{"comment":"The implementation details do not state the number of retrieved exemplars k used in the main experiments; the k-sensitivity analysis in Figure 4 reports optimal values per dataset, so the paper should explicitly state which k was used to produce Table II.","section":"Section IV-D"},{"comment":"The ablation labels are hard to parse: 'RAGSum w/o pretrained + SR' and 'RAGSum w/o combined' should be explicitly defined in the text so that readers know exactly which components are removed in each row.","section":"Table III"},{"comment":"The narrative says the C-dataset performance 'remains competitive' while the numbers are described as gains; consider aligning the wording with the actual magnitude of the differences, especially given the lack of significance testing.","section":"Section V-B and V-C"}],"recommendation":"major_revision","confidential_remarks":"This is a preliminary empirical paper with a plausible architecture and a replication package, but the state-of-the-art claim is not yet justified: the decisive comparisons lack statistical support, the LLM baseline is not trained to a comparable degree, and the joint-loss description does not match the equations. These issues can be fixed within the manuscript's scope by adding repeated runs and significance tests, rebalancing the baseline setup or its interpretation, and clarifying the loss formulation. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a decent empirical paper that combines known pieces — contrastive retriever pre-training, joint fine-tuning of retriever and generator on one CodeT5 backbone, and a self-refinement pass — and reports consistent gains on three standard benchmarks (JCSD, PCSD, CCSD). The combination is new enough, and the ablation study cleanly shows that each component contributes, with joint fine-tuning carrying most of the weight. The replication package and use of the standard datasets are real pluses.\n\nWhere it gets soft: the central claim of a new SOTA is not actually supported by the numbers as reported. Table II gives single point estimates with no error bars, no repeated seeds, no significance tests. On CCSD, the decisive comparison against JOINTCOM is 27.95 vs 26.32 C-BLEU, 47.35 vs 46.15 ROUGE-L — margins that could easily be run-to-run noise. The abstract calls the results preliminary, and Section V-B then says 'substantially outperforms' and 'new state-of-the-art.' Those claims need to be scaled back or backed with variance information. The Llama-3.1-8B baseline is also not a fair opponent: it is a frozen generator in the RAG framework while RAGSum is fully fine-tuned, so the headline comparison against it mostly shows that fine-tuning helps. There is also a plain typo in Section III-A where the q2c contrastive loss is described as 'minimize the similarity' between code and its comment, when the formula does the opposite; that needs fixing, as does 'ROUTE-L' in the abstract.\n\nNone of this is fatal. The core idea is sensible and the engineering appears to be honest. I just would not let the 'new state-of-the-art' sentence survive. My recommendation for peer review: send it out. A serious referee can usefully push on the evaluation rigor, and the paper is short enough and the replication package concrete enough that a revision cycle could make it work. I would want to see either error bars or repeated-seed runs, a descoped claim, and an LLM baseline that gets the same training budget or is explicitly positioned as a lower bound.","headline":"A coherent incremental RAG system for code comment generation with a solid replication package, but the SOTA claim rests on point estimates without error bars and a stacked LLM baseline.","tokens_in":11136,"tokens_out":1990,"would_cite":false,"duration_ms":24318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fusing retrieval and generation into a single CodeT5 encoder-decoder, trained with contrastive retrieval pre-training, joint fine-tuning, and a self-refinement loop, outperforms three established baselines on Java…","keywords":["code comment generation","retrieval-augmented generation","joint retriever-generator training","contrastive learning","CodeT5","code summarization","self-refinement"],"falsifier":"Retrain the baselines with the same CodeT5 backbone, the same retrieval corpus, and the same number of fine-tuning epochs as RAGSum, without freezing the LLM generator, and compare on identical test splits; if the BLEU and ROUGE-L gap shrinks to noise or reverses, the claimed advantage is training budget rather than joint modeling.","tokens_in":10137,"feed_emoji":"💬","tokens_out":9444,"duration_ms":91152,"temperature":0.7,"pith_summary":"RAGSum is a code-comment generator built on one CodeT5 backbone, a pre-trained encoder-decoder for code, so that retrieval and generation are trained as a single system rather than as separate stages. The recipe has three parts: contrastive pre-training shapes the encoder's embeddings for nearest-neighbor search; joint fine-tuning weights each retrieved exemplar's generation loss by its retrieval similarity, so the retriever learns from how well the generator does; and a self-refinement loop re-trains the model on its own most ROUGE-L-faithful outputs. In preliminary results on Java, Python, and C benchmarks, the paper reports that RAGSum beats CMR-Sum, JOINTCOM, and a Llama-3.1-8B-based RAG baseline on every metric considered. The paper argues that this points to a cheaper route to better documentation automation: tighter coupling of retrieval and generation instead of larger models.","feed_headline":"One joint model beats three rivals at code comment generation","feed_subtitle":"Coupling retrieval and generation in one CodeT5 backbone lifts BLEU, ROUGE-L, and METEOR across Java, Python, and C.","key_machinery":"The central object is a single CodeT5 encoder-decoder whose encoder doubles as the retriever and whose decoder acts as the generator. The load-bearing mechanism is the joint fine-tuning loss: for each code query, the retriever returns the top-$k$ code-comment pairs, the decoder computes a cross-entropy loss for each augmented input, and the per-exemplar losses are averaged with weights $\\nu_j = sim(q_i, q_j^r)$, the cosine similarity between the query and the retrieved code. That weighting lets generation feedback shape retrieval. Before that, a contrastive pre-training phase aligns code and comment embeddings with in-batch negatives; after that, a self-refinement loop fine-tunes the model on its own highest-scoring candidates, selected by ROUGE-L against the reference comment.","core_discovery":"The central claim is that retrieval and generation for code comments should not be optimized in isolation: when the retriever is trained through the generator's loss, it stops fetching irrelevant neighbors, and the decoder conditions on exemplars that actually help. RAGSum realizes this by pre-training the CodeT5 encoder with code-to-code and code-to-comment contrastive losses, joint fine-tuning encoder and decoder with a similarity-weighted composite loss, and refining the model on self-generated candidates scored by ROUGE-L. The paper reports that this achieves the best results among the compared methods on all three datasets, with the clearest margin on Python, where corpus-BLEU rises to 33.0 against 28.89 for CMR-Sum, and with smaller but consistent gains on Java and C.","pith_inferences":["Editorial inference: the same three-phase recipe—contrastive retriever pre-training, similarity-weighted joint fine-tuning, and self-refinement—could transfer to other retrieval-plus-generation tasks such as commit message generation, API documentation, or code search, since none of the components is comment-specific.","Editorial inference: the self-refinement step selects candidates by ROUGE-L against the ground-truth comment during training; a deployment variant would need a learned reward model or reference-free scorer, and whether the gain survives that substitution is untested here.","Editorial inference: because the optimal $k$ varies across languages, an adaptive retrieval-width controller could squeeze out further gains; the paper does not propose one.","Editorial inference: comparing RAGSum against a fully fine-tuned instruction-tuned model of similar parameter count would clarify whether the reported edge comes from joint modeling or from the baselines' frozen weights."],"forward_implications":["On the JCSD, PCSD, and CCSD benchmarks, RAGSum outperforms CMR-Sum, JOINTCOM, and Llama-3.1-8B with RAG prompting across corpus-BLEU, sentence-BLEU, ROUGE-L, METEOR, and CIDEr.","Joint fine-tuning is the main driver of the gain: ablating it lowers corpus-BLEU by 13.07% on Java, 15.02% on Python, and 20.99% on C relative to the full model.","Contrastive encoder pre-training and self-refinement each add measurable improvements; on PCSD, removing self-refinement drops corpus-BLEU from 33.0 to 32.57.","At inference, RAGSum needs only the single highest-scoring retrieved exemplar, so the joint model adds accuracy without a heavy retrieval-time ensemble.","The best number of retrieved exemplars for training is dataset-dependent: $k=4$ for Java and $k=3$ for Python and C."],"supporting_citations":[{"why":"Supplies the CodeT5 encoder-decoder backbone that RAGSum fuses retrieval and generation into.","marker":"[15]"},{"why":"The JOINTCOM baseline and the preceding joint retriever-generator approach whose separate-model design RAGSum aims to improve.","marker":"[13]"},{"why":"The CMR-Sum baseline with independently tuned retriever and generator that RAGSum compares against.","marker":"[14]"},{"why":"Establishes the contrastive retrieval-augmented generation paradigm that RAGSum's pre-training phase builds on.","marker":"[11]"},{"why":"EditSum is the retrieve-and-refine predecessor that leaves retrieved noise unaddressed, motivating joint training.","marker":"[12]"},{"why":"Supplies the in-batch negative sampling technique and the CCSD C-language dataset used in evaluation.","marker":"[27]"},{"why":"Motivates the self-refinement loop by documenting exposure bias and hallucination in autoregressive generation.","marker":"[28]"}],"fun_headline_variants":["Single CodeT5 model fuses retrieval and generation for comments","Joint retriever-generator beats separate pipelines on comments","Tightly coupled retrieval and generation improve comment BLEU","RAGSum: one backbone for retrieval and comment generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes the reproduced baselines—CMR-Sum, JOINTCOM, and Llama-3.1-8B in a RAG setup—are configured at a fair, standard setting, so the reported gains come from joint fine-tuning rather than from unequal training effort or frozen baseline weights.","fun_headline_variants_meta":{"raw":{"variants":["Single CodeT5 model fuses retrieval and generation for comments","Joint retriever-generator beats separate pipelines on comments","Tightly coupled retrieval and generation improve comment BLEU","RAGSum: one backbone for retrieval and comment generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000879,"raw_usage":{"total_tokens":3802,"prompt_tokens":949,"completion_tokens":2853,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2784}},"tokens_in":565,"tokens_out":2853,"duration_ms":20998,"temperature":1.0,"reasoning_tokens":2784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:43:34.446558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the baselines with the same CodeT5 backbone, the same retrieval corpus, and the same number of fine-tuning epochs as RAGSum, without freezing the LLM generator, and compare on identical test splits; if the BLEU and ROUGE-L gap shrinks to noise or reverses, the claimed advantage is training budget rather than joint modeling.","supporting_citations":[{"cited_title":"Cross-modal retrieval-enhanced code summarization based on joint learning for retrieval and generation,","cited_arxiv_id":null,"evidence_quote":"The CMR-Sum baseline with independently tuned retriever and generator that RAGSum compares against."},{"cited_title":"Editsum: A retrieve-and-edit framework for source code summarization,","cited_arxiv_id":null,"evidence_quote":"EditSum is the retrieve-and-refine predecessor that leaves retrieved noise unaddressed, motivating joint training."},{"cited_title":"Better language models of code through self-improvement,","cited_arxiv_id":null,"evidence_quote":"Motivates the self-refinement loop by documenting exposure bias and hallucination in autoregressive generation."}],"review_version":1}