{"id":"2d3f43dc-c5db-4b6d-ae43-a823442545d7","arxiv_id":"2506.14649","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"IsComment generates supplementary code comments by retrieving issue-report sentences with an LLM and filtering them for code relevance and issue verifiability, reporting coverage of manual comments up to 88.4%.","lead":"IsComment is a system that retrieves sentences from software issue reports and uses large language models to turn them into supplementary code comments, then filters out comments that cannot be tied back to the issue. A specialist might read it because it claims to sharply reduce hallucinations in AI-generated code comments, though the evaluation design makes the reported gains look partly built into the test data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage gain is built into the dataset: issue-sentence source and manual-comment target are near-duplicates, and no verbatim issue-sentence baseline is reported.","rationale":"The paper's central assertion is quantitative, and the load-bearing condition is that coverage measures comment-generation quality. That condition is violated because the reference comments are filtered to be near-verbatim restatements of issue sentences, while the generator is fed those same issue sentences. The decisive missing experiment is a verbatim issue-sentence baseline; without it, the reported improvements over LLM prompting are compatible with a much weaker claim: that issue reports contain sentences closely matching the selected human comments. The reader's weakest assumption identifies exactly this circularity, so I agree. I credit the paper for releasing code and data, for a plausible retrieval-plus-verification architecture, and for an explicit threat-to-validity discussion, but these do not repair the evaluation design. The headline should be reframed as 'issue sentences can be retrieved to align with human supplementary comments,' not 'IsComment generates supplementary comments better than LLMs.' The paper's own RQ2 verification claim is self-referential for the same reason. Given the centrality of the coverage numbers to the paper's contribution, the reader's REJECT verdict is appropriate and I do not adjust it.","tokens_in":17621,"tokens_out":6085,"duration_ms":60212,"concrete_test":"Run a direct issue-sentence oracle on Issuecom: take the issue sentence(s) selected by IsComment's retrieval step (Sec 3.1) and emit them verbatim, with no LLM rewriting and no verification, then apply the Sec 4.4.1 coverage rule. If this oracle matches or approaches IsComment's 88.4% coverage for GPT-4o, the claimed gain is an artifact of the dataset construction and evaluation loop rather than of the generation pipeline. As a robustness check, rebuild the evaluation set from the same projects without the Sec 2.1.4 70% overlap filter (keeping only the manual validation step) and rerun the key rows of Table 3; if the IsComment-versus-baseline gap shrinks materially, the generalizability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (88.4% coverage for GPT-4o in Table 3) is not a fair test of IsComment because the evaluation target is constructed from the same issue sentences the method is given. Section 2.1.4 retains a manual comment sentence only if more than 70% of its words overlap with a sentence in the linked issue. The pipeline then retrieves issue sentences (Sec 3.1), rewrites them into comments (Sec 3.2), filters them by SentenceBERT similarity to issue sentences (Sec 3.3, Eq. 2), and is evaluated (Sec 4.4.1) by whether a generated sentence has SentenceBERT similarity >0.6 with the retained manual sentence. Thus the issue sentence that IsComment is prompted with is, by construction, a near-paraphrase of the reference comment, and the same similarity measure used for verification is used for scoring. The most direct baseline that would break this loop—simply outputting the retrieved issue sentence verbatim as the generated comment—is absent from Table 3. Without this baseline, the 88.4% figure may only reflect the overlap filter's success at mining comments that are already written in the issue, rather than an ability of IsComment to generate supplementary comments. The RQ2 hallucination-reduction claim is circular in the same way: 'verifiable' is scored against the issue sentence that was placed in the prompt. The approach may still be useful, but the paper's central quantitative evidence does not demonstrate it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IsComment, a retrieval-and-verification pipeline for generating 'supplementary' code comments (design rationale, usage directives, implications, concepts, functionality) from issue reports. It introduces a new dataset, Issuecom, built by linking method-comment pairs to issue reports via commit messages, filtering comments by the MESIA metric and by a 70% word-overlap rule against issue sentences, and then manually validating the remaining comments. The pipeline retrieves issue sentences with an LLM prompt, generates candidate comments from those sentences, and filters them using code-element mention, SIDE, and SentenceBERT similarity to issue sentences. Experiments compare IsComment with non-LLM and LLM baselines across ChatGPT, GPT-4o, and DeepSeek-V3, reporting large gains in coverage (e.g., 88.4% for GPT-4o) and reduced hallucinations, plus a small applicability study on Apache Derby.","tokens_in":17924,"tokens_out":5893,"duration_ms":56491,"significance":"The high-level idea of using issue reports as a source of supplementary information for code comments and applying verification to reduce hallucinations is timely and would be valuable if the evidence were sound. The five-category taxonomy of supplementary information (Functionality, Rationale, Implication, Concept, Directive) is useful, and the authors make code and data available for reproducibility. However, the main quantitative claims rest on an evaluation design that is circular: reference comments are selected to be near-paraphrases of issue sentences, generation is prompted with those same sentences, and the coverage metric measures similarity to those near-paraphrases. As it stands, the paper does not provide valid evidence for its headline improvements.","major_comments":[{"comment":"The dataset construction rule that retains comment sentences with more than 70% word overlap with an issue sentence makes the reference comments near-duplicates of the issue text. Since the generation pipeline retrieves issue sentences from the linked issue and produces comments from them, the reported coverage increases (e.g., GPT-4o 88.4% in Table 3) are largely a consequence of this inclusion rule rather than of the method's ability to generate supplementary comments. An evaluation on comments that are not near-paraphrases of issue sentences, or at least an analysis of how coverage changes as the overlap threshold is relaxed, is needed.","section":"Sec. 2.1.4, Table 3"},{"comment":"The same SentenceBERT similarity instrument and the same 0.6 threshold are used both to verify/retain generated comments against issue sentences (Eq. (2)) and to score coverage against manual comments. This double use of the same metric confounds the filtering step with the evaluation. A baseline that simply emits the retrieved issue sentence, or a lightly paraphrased version, as the generated comment is missing; without it, the reader cannot tell whether the 88.4% figure reflects genuine comment generation or the fact that the reference sentence is already close to the prompt content.","section":"Secs. 3.1-3.3 and 4.4.1"},{"comment":"The RQ2 hallucination-reduction claim is circular. A generated sentence is labeled 'verifiable' if it has SentenceBERT similarity above 0.6 with some issue sentence, and the pipeline filters on exactly this criterion. Therefore, the high verifiability of IsComment's outputs is by construction and does not establish that the comments are factually correct or meaningful. The manual validation of 100 pairs in Sec. 3.3 is small and is not used to validate the coverage metric itself.","section":"Sec. 4.4.2, Fig. 5"}],"minor_comments":[{"comment":"Table 3 has duplicated column headers and appears to misalign rows: the GPT-4o Code Prompt row lists after-filter values (0.6 avg sentences, 19.5 sent len, 93 full-cover, 33 partial-cover, 28.4% coverage) identical to the ChatGPT Code Prompt row, which is confusing.","section":"Table 3"},{"comment":"There are typos in the running text: 'Funtionality' and 'Dirictive' should be 'Functionality' and 'Directive'.","section":"Sec. 5.1"},{"comment":"The metric name is written inconsistently as 'SentenceBert' in Sec. 3.3 and 'Sentence-BERT' elsewhere; please standardize.","section":"Sec. 3.3"},{"comment":"The 70% word-overlap rule does not specify whether overlap is computed on exact word tokens, stems, or with stopword removal; this should be stated precisely for reproducibility.","section":"Sec. 2.1.4"},{"comment":"The sentence 'The threshold 0.6 is a trade-off obtained through our manual attempt' is vague; a sensitivity analysis of the threshold, or at least a description of the manual procedure, would strengthen the work.","section":"Sec. 3.3"}],"recommendation":"reject","confidential_remarks":"The circularity between dataset construction, generation, and evaluation is central and cannot be fixed by a local revision. Re-running the study with a non-overlapping reference set or a verbatim retrieved-sentence baseline would be needed to establish whether the claimed gains are real, which is more than a routine revision. I would also note that the paper is submitted to a conference venue and the appendix materials are only available via a GitHub link; this is not a substitute for a proper reproducibility package in the published record."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper tackles a real gap: generating supplementary comments (design rationale, usage directives) that can't be read off the code, using issue reports as the source. The five-type taxonomy from the manual analysis is reasonable, and the Issuecom dataset, while small, could be a useful resource if released cleanly. The RQ4 extension to methods without manual comments is a sensible direction. So the paper is not without merit.\n\nBut the main numbers don't hold up. Section 2.1.4 keeps a manual comment sentence only if more than 70% of its words overlap with a sentence in the linked issue. The pipeline then retrieves those issue sentences, generates comments from them, filters by Sentence-BERT similarity to those same issue sentences, and evaluates coverage by similarity to the retained manual comments. That means the reference comments are near-paraphrases of the issue sentences the method is given. The reported 88.4% coverage for GPT-4o is largely a consequence of this setup, not a demonstration of good comment generation. The missing baseline is simply outputting the retrieved issue sentence verbatim; it would likely score close to the method's numbers.\n\nThe verification/hallucination claim is circular in the same way: 'verifiable' is scored against the issue sentence that was put in the prompt. The 0.6 threshold was manually tuned, and the 70% overlap filter is arbitrary. Table 3 has what looks like a copy-paste error: the GPT-4o Code Prompt row repeats the ChatGPT numbers in the After Filtering columns. There are also no statistical tests.\n\nNone of this makes the approach worthless. The taxonomy, the retrieval idea, and the prototype are plausible, and fixing the evaluation is a matter of adding the right baseline and building the dataset without pre-selecting on overlap. But as written, the central quantitative claims are not supported. I'd send this to peer review rather than desk-reject, because the flaw is subtle and fixable, and a good reviewer could turn the paper into something useful. Just don't quote the coverage numbers in your next talk.\n\nBest,","headline":"Useful taxonomy and dataset, but the headline coverage numbers are built into the dataset construction and don't survive a verbatim-issue-sentence baseline.","tokens_in":18466,"tokens_out":3138,"would_cite":false,"duration_ms":30539,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs backed by issue retrieval and verification cover 88% of human-written supplementary comments.","keywords":["supplementary code comments","issue reports","retrieval-augmented generation","LLM-based comment generation","hallucination reduction","verifiable text generation","MESIA","SentenceBERT verification"],"falsifier":"Build a test set of supplementary comments that are genuinely informative but are not paraphrases of their linked issue (for example, comments whose rationale comes from design discussions absent from the issue), then run IsComment on it; if coverage falls back to the plain-prompt level, the headline gains are an artifact of the overlap filter rather than of retrieval and verification themselves.","tokens_in":17358,"feed_emoji":"💬","tokens_out":8126,"duration_ms":72855,"temperature":0.7,"pith_summary":"The paper proposes IsComment, a pipeline that generates supplementary code comments — design rationale, usage directives, performance implications, concepts, and functionality — by first retrieving the relevant sentences from a method's linked issue report, prompting an LLM to write candidate comment sentences, and then filtering out candidates that are not relevant to the code or not verifiable against the issue. The central claim is that this issue-grounded retrieval and verification turns LLMs from weak generators of supplementary comments (covering roughly a third of human-written supplementary sentences) into strong ones (covering 86–88% of them). The authors care because supplementary comments are exactly the comments developers want for understanding and reusing code, yet they cannot be derived from source code alone, which is why ordinary code summarization fails on them. If the approach works, developers could automatically obtain reliable, evidence-backed supplementary comments for methods that have associated issue discussions, including methods that currently have no manual comments at all.","feed_headline":"Issue-backed LLM comments cover 88% of human-written comments","feed_subtitle":"Retrieval by information type plus verification against the issue lifts coverage from about a third to near nine in ten.","key_machinery":"The load-bearing machinery is the IsComment pipeline, a three-phase loop in which a prompted LLM first acts as a retrieval agent that classifies sentences of the method's issue report into five supplementary-information types (Functionality, Rationale, Implication, Concept, Directive), then generates up to five typed candidate comment sentences from those retrieved sentences, and finally filters the candidates through two code-relevance checks (does the sentence mention a code element, or does it have a positive SIDE code-comment alignment score?) and one issue-verifiability check (SentenceBERT cosine similarity above 0.6 to at least one issue sentence). The five-type taxonomy makes retrieval selective rather than dumping the whole noisy issue into the prompt, and the verification thresholds are what the paper credits for reducing hallucinations.","core_discovery":"The paper's central claim is that supplementary code comments, which describe things like why a method exists, when it must be called, or what domain concept it relies on, can be generated reliably from issue reports through a retrieve–generate–verify pipeline. From a new dataset of 443 manually validated method–comment–issue triples mined from ten large open-source projects, the authors report that plain LLM prompting covers only 33.6–35.8% of human-written supplementary comment sentences, that feeding the whole issue report raises coverage to roughly two-thirds, and that their full IsComment pipeline raises it further to 72.2% for ChatGPT, 88.4% for GPT-4o, and 86.2% for DeepSeek-V3. The verification stage keeps only candidate sentences that mention a code element or score positively on the SIDE alignment metric and that have a SentenceBERT similarity above 0.6 to some issue sentence; this filtering removes hallucinated content while leaving coverage at 66.5–83.9%. The paper also reports that the generated comments carry higher MESIA supplementarity than baselines and that the method produced supplementary comments for 82.5% of issue-linked methods lacking manual comments in one evaluated project.","pith_inferences":["Editorial inference: because the pipeline treats the issue report as evidence, the same retrieve-by-type-and-verify design could be applied to pull requests, mailing lists, and requirement documents; the paper names this as future work but does not test it.","Editorial inference: the five-type taxonomy suggests a natural user-facing design — letting developers request only Rationale, only Directive, or other types — which could be evaluated in a controlled experiment on developer comprehension.","Editorial inference: the verification thresholds (0.6 SentenceBERT similarity, code-element mention, positive SIDE) could be tuned per project or per information type; a calibration study across projects would show whether one global threshold is best.","Editorial inference: since the approach works with three different LLMs, a plausible next test is whether it also reduces hallucination in higher-stakes settings such as API documentation for security-sensitive libraries, where unverifiable claims are costlier."],"forward_implications":["For methods with a linked issue report, LLMs can jump from covering roughly a third to 72–88% of human-written supplementary comment sentences.","Filtering by code relevance and issue verifiability removes large amounts of hallucinated content while mostly preserving the coverage gains.","Typed outputs (Functionality, Rationale, Implication, Concept, Directive) give developers selectable comment kinds, each traceable to evidence in the issue report.","The approach transfers to methods with no manual comments: in one dataset project it produced supplementary comments for 82.5% of issue-linked, uncommented methods.","Because the pipeline is model-agnostic, the same issue retrieval and verification can be layered onto newer LLMs, with results reported on ChatGPT, GPT-4o, and DeepSeek-V3."],"supporting_citations":[{"why":"Defines the MESIA supplementarity metric used both to filter the dataset and to evaluate generated comments.","marker":"[1]"},{"why":"Documents developers' expectations for supplementary comments, motivating the task.","marker":"[2]"},{"why":"Provides the plain 'Code Prompt' LLM baseline that IsComment is compared against.","marker":"[18]"},{"why":"Provides the in-context few-shot baseline augmented with project examples.","marker":"[19]"},{"why":"Supplies the retrieval-augmented generation formulation that issue retrieval builds on.","marker":"[28]"},{"why":"Provides the twelve-category comment taxonomy from which the five supplementary-information types are derived.","marker":"[33]"},{"why":"Defines the SIDE code-comment alignment score used in the code-relevance filter.","marker":"[41]"},{"why":"Provides the SentenceBERT similarity measure used for both coverage evaluation and issue-verifiability filtering.","marker":"[42]"},{"why":"Supports using semantic-similarity metrics such as SentenceBERT to evaluate generated code comments.","marker":"[43]"}],"fun_headline_variants":["Issue retrieval and verification lift LLM comment coverage to 88%","LLM code comments get 88% coverage via issue verification","From 33% to 88%: issue-driven code comment generation","Issue verification boosts LLM comment coverage from 33% to 88%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the dataset rule that keeps only manual comment sentences with over 70% word overlap with a linked issue sentence; since IsComment is prompted from those very issue sentences and coverage is measured against the retained comments, relaxing that rule would likely shrink the headline improvements.","fun_headline_variants_meta":{"raw":{"variants":["Issue retrieval and verification lift LLM comment coverage to 88%","LLM code comments get 88% coverage via issue verification","From 33% to 88%: issue-driven code comment generation","Issue verification boosts LLM comment coverage from 33% to 88%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3286,"prompt_tokens":1020,"completion_tokens":2266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2198}},"tokens_in":636,"tokens_out":2266,"duration_ms":16007,"temperature":1.0,"reasoning_tokens":2198,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:50:04.676820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of supplementary comments that are genuinely informative but are not paraphrases of their linked issue (for example, comments whose rationale comes from design discussions absent from the issue), then run IsComment on it; if coverage falls back to the plain-prompt level, the headline gains are an artifact of the overlap filter rather than of retrieval and verification themselves.","supporting_citations":[{"cited_title":"MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation","cited_arxiv_id":"2403.17357","evidence_quote":"Defines the MESIA supplementarity metric used both to filter the dataset and to evaluate generated comments."},{"cited_title":"Practitioners’ expectations on automated code comment generation","cited_arxiv_id":null,"evidence_quote":"Documents developers' expectations for supplementary comments, motivating the task."},{"cited_title":"Few-shot training llms for project- specific code-summarization","cited_arxiv_id":null,"evidence_quote":"Provides the in-context few-shot baseline augmented with project examples."},{"cited_title":"Robillard","cited_arxiv_id":null,"evidence_quote":"Provides the twelve-category comment taxonomy from which the five supplementary-information types are derived."},{"cited_title":"Evaluating code summarization techniques: A new metric and an empiri- cal characterization","cited_arxiv_id":null,"evidence_quote":"Defines the SIDE code-comment alignment score used in the code-relevance filter."},{"cited_title":"Semantic similarity metrics for evaluating source code summarization","cited_arxiv_id":null,"evidence_quote":"Supports using semantic-similarity metrics such as SentenceBERT to evaluate generated code comments."}],"review_version":2}