{"id":"efa4ad60-a8b9-4dd0-9e7a-289c483bae39","arxiv_id":"2504.19459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Including the bodies of helper methods in the prompt enables large language models to generate better comments for dependent Java methods than standard comment generators.","lead":"Automatic comment generators struggle with Java methods that call other methods, which make up about 69% of methods in popular projects. A proposed tool called HelpCOM feeds the called helper methods into GPT-4o's prompt and produces comments that human developers prefer about 75% of the time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Helper context effect is small and mostly non-significant against plain GPT-4o, so the central causal claim is not yet established.","rationale":"The reader's weakest assumption is call-graph extraction noise. That is a real threat but secondary here: imperfect matching by name and parameter count would likely add noise and dilute HelpCOM's measured effect, and it cannot explain why the same-model ablation is small and mostly non-significant. The most load-bearing point is internal validity of the causal claim. The central contribution is dependency-aware prompting, so the comparison to plain GPT-4o in Table 4 is the appropriate control; without a significant, non-trivial gain there, the survey and weak-baseline comparisons do not establish that helper methods are the cause. The paper is not fatally flawed: the replication package, the human survey, and the RQ2 descriptive differences are useful, and the missing analysis is a bounded fix. I would keep the conditional verdict, with the condition being a rigorous same-model comparison (significance, confidence interval, effect size) and/or a survey arm including plain GPT-4o.","tokens_in":20225,"tokens_out":8982,"duration_ms":87017,"concrete_test":"Run a paired bootstrap over the 380 dependent-method samples: for each bootstrap resample, compute the per-method OMSssl for HelpCOMN(GPT-4o) and for plain GPT-4o, then the mean difference; report the 95% percentile interval and a paired effect size. If the interval includes 0 or the gain is below the minimal detectable effect, the abstract's causal wording should be softened to 'beats older baselines in this sample,' and the headline 5.6-50.4% should be accompanied by the GPT-4o-only comparison. Optionally, add plain GPT-4o as a fourth comment option in the survey to test whether the 75% preference is driven by dependency context or by prompt/model strength.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: adding helper-method bodies to the prompt improves comments for dependent methods. The cleanest test is Table 4's comparison between HelpCOMN(GPT-4o) and the plain GPT-4o row, which differ only in the presence of helper context. The result is weak: OMSssl is 56.82 vs 55.70, a ~2% relative gain, and OMSss is 40.35 vs 39.33. Of the nine per-metric Wilcoxon tests, the asterisks mark only METEOR and CIDEr as significant; BLEU, ROUGE-L, SBERT, USEnc, SIDE, and both LLM-judge scores are not marked. No significance test is reported for the OMS values themselves, and no confidence intervals or effect sizes are given. The abstract's headline range '5.6% to 50.4%' is computed against CodeT5+, CodeBERT, and ASAP, which are much weaker/older models; the 5.6% lower bound is ASAP (OMSssl 53.83), and the 50.4% is CodeBERT. The relevant control for 'helper methods help' is the same LLM without helper bodies. As reported, that contrast is small and largely non-significant. The practitioner survey is informative but compares HelpCOM only with CodeT5+ and ASAP, not with plain GPT-4o, so it does not isolate the dependency-aware mechanism. This does not falsify the hypothesis, but it means the paper's central causal assertion currently rests on a ~1-point OMS shift on an arbitrary weighted composite (Eqs. 1-2), without evidence that the shift is reliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether method dependencies influence automatic comment generation. It builds a dataset of 647,769 methods from 10 popular Java GitHub projects, classifies methods as dependent or independent based on a Tree-sitter-based call-graph construction, and reports three results: (RQ1) dependent methods are more frequent (69.25%) and more change-prone; (RQ2) existing baselines (CodeT5+, CodeBERT, ASAP) produce worse comments for dependent than for independent methods; and (RQ3) a proposed prompt-augmentation technique, HelpCOM, which adds helper-method bodies to the prompt, improves comment quality for dependent methods. The headline claim is that HelpCOM outperforms baselines by 5.6% to 50.4% in an author-defined Overall Metric Score, and a survey of 156 practitioners reports that approximately 75% prefer HelpCOM-generated comments.","tokens_in":20512,"tokens_out":5968,"duration_ms":58432,"significance":"If the central causal claim is established, the paper identifies a practically important blind spot in code summarization (the prevalence and difficulty of dependent methods) and shows that a cheap, model-agnostic prompt change materially improves generated comments. The paper has clear strengths: it releases a replication package, constructs a large real-world dataset (647,769 methods from 10 active projects), uses git history for RQ1, employs multiple automatic metrics plus a 156-participant practitioner survey, and includes pilot cross-language experiments for Python and PHP. The main weakness is that the causal mechanism (helper-method context causes the improvement) is not yet supported by rigorous statistics against the proper control: the same LLM without helper context. The OMS weights and the SIDE-based filtering/ground-truth revision also need robustness analysis to make the quantitative claims credible.","major_comments":[{"comment":"The central claim that helper-method context is responsible for the improvement requires a comparison of HelpCOMN (GPT-4o) with plain GPT-4o, which differ only by the presence of helper-method bodies in the prompt. In Table 4, that contrast is small: OMSssl is 56.82 vs. 55.70 and OMSss is 40.35 vs. 39.33. The asterisks in Table 4 mark only METEOR and CIDEr as significant differences for this pair; BLEU, ROUGE-L, SBERT, USEnc, SIDE, and both LLM-based judges show no significant difference, and no significance test or confidence interval is reported for the OMS scores themselves. The abstract's 5.6%-50.4% range is computed against CodeT5+, CodeBERT, and ASAP, not against the same-LLM control, so it does not support the mechanism claim. The paper should report significance tests and effect sizes for the GPT-4o vs. HelpCOMN contrast, including for OMS, and should clearly distinguish the mechanism result from the overall superiority of a strong LLM baseline.","section":"3.3, Table 4"},{"comment":"The RQ3 evaluation is performed on a dataset filtered by SIDE >= 0.8 (a model-based quality threshold) and on ground-truth comments that were revised by a senior developer after the original comments were rated as 'unacceptable'. This means the quantitative comparisons in Table 4 are not on the original, unmodified dataset. The authors should justify the choice of the 0.8 threshold, report how many methods were excluded, and provide a sensitivity analysis (e.g., without the SIDE filter, with different thresholds, and before vs. after ground-truth revision) to establish that the observed improvements in Table 4 are not an artifact of this selection or revision procedure.","section":"3.2-3.3"},{"comment":"The dependency graph that defines dependent and independent methods is built by matching Tree-sitter method_invocation nodes to previously extracted methods using only the method name and parameter count. This matching cannot resolve overloading, inheritance, dynamic dispatch, or calls to methods defined outside the repository, so helper methods can be silently mis-assigned or missed. Since RQ2 and RQ3 both compare dependent vs. independent methods, this categorization is load-bearing. The authors should validate the call-graph construction on a random sample (e.g., manual inspection of, say, 100-200 methods) and quantify the misclassification rate, or at least discuss how prevalent such cases are in the selected projects.","section":"2.4.1"},{"comment":"The Overall Metric Score (Eqs. 1-2) uses author-defined weights (0.46/0.54 for OMSss and 0.30/0.35/0.35 for OMSssl) with no sensitivity analysis. Since the central headline range '5.6% to 50.4%' is an OMS improvement, the ranking of HelpCOMN (GPT-4o) over plain GPT-4o and over the baselines may depend on these arbitrary weights. The authors should report how the OMS ranking changes under reasonable weight variations and, more importantly, show which individual metrics drive the improvement, so that the reader can judge the robustness of the headline claim.","section":"2.7.4"}],"minor_comments":[{"comment":"Section 7 says 'we proposed a noble technique'; this should be 'novel technique'.","section":"7"},{"comment":"The inter-rater agreement is reported as 'Fleiss' Kappa score of 83.6% (above the 70% threshold)'. Fleiss' kappa is a coefficient (usually between 0 and 1) rather than a percentage; please clarify whether the reported value is percent agreement or kappa multiplied by 100, and describe how the 70% threshold was determined.","section":"3.3"},{"comment":"The practitioner survey compares HelpCOM only with CodeT5+ and ASAP, not with plain GPT-4o. As a result, the 75% preference for HelpCOM does not isolate the effect of helper-method context; including plain GPT-4o as an anonymous option in the survey would strengthen the mechanistic interpretation.","section":"4.3"},{"comment":"The dataset description reports 69,058 Java files and 647,769 methods, but there is no per-project breakdown. Reporting the number of files, methods, dependent methods, and helpers per repository would improve transparency and reproducibility.","section":"2.4.1"},{"comment":"The repository versions are not pinned: the paper says 'main' or 'master' branches were cloned, but does not report commit hashes or retrieval dates. This makes exact replication difficult.","section":"2.3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical setup and the replication package are clear strengths, and the paper addresses a real gap in the code-summarization literature. However, the central mechanistic claim—that helper-method context, rather than the underlying LLM, is what drives the improvement—is currently supported mainly by a small and mostly non-significant contrast in Table 4. I would like the editor to weigh the additional statistical rigor expected for a headline causal claim of this kind; the sensitivity analyses requested in the report are essential before the abstract's 5.6%-50.4% range can be taken at face value. If those analyses confirm the current point estimates, the paper would be a solid contribution for EASE."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper builds a genuinely useful dependency-labeled dataset (647K Java methods from 10 large projects) and shows, convincingly, that dependent methods are the majority and receive worse comments from CodeT5+, CodeBERT, and ASAP. Second, the headline causal claim—that adding helper-method bodies to the prompt is what improves comments—is not established by the current numbers. The clean comparison is HelpCOMN(GPT-4o) against plain GPT-4o in Table 4: 56.82 vs 55.70 on OMSssl, a ~2% relative gain, and only METEOR and CIDEr are individually significant. No test is reported on the OMS composite and no confidence intervals are given. The 5.6%–50.4% range in the abstract compares against CodeT5+, CodeBERT, and ASAP, which are older or much weaker models; it is not evidence about the dependency mechanism.\n\nThe dataset is the real contribution. The RQ1 prevalence and change-proneness analysis is straightforward and useful, and the replication package is shipped, which is creditable. The survey with 156 practitioners is informative for end-user preference, though it compares HelpCOM only against CodeT5+ and ASAP, not plain GPT-4o, so it cannot isolate the effect of helper context.\n\nThe evaluation stack has several compounding choices. The RQ3 sample is filtered by SIDE >= 0.8, the ground-truth comments were revised by a senior developer before scoring, and the OMS weights in Eqs. (1)-(2) are arbitrary. None of these is fatal alone, but together they make point estimates fragile. The call-graph extraction by method name and parameter count is another soft spot: overloads, inherited implementations, and dynamic dispatch will be silently mis-assigned. If the dependency graph is noisy, both the RQ2 and RQ3 results could shift. That is a threat, not a demonstrated error.\n\nThis paper deserves a serious referee, not a desk reject. The dataset and RQ1/RQ2 findings are worth publishing, and the RQ3 mechanism is plausible even if under-supported. I would send it out with a request for the authors to add the direct control comparison with confidence intervals, test the OMS deltas, and either validate or honestly bound the call-graph noise.","headline":"A useful dependency-labeled dataset and a believable RQ1/RQ2 story, but the causal claim that helper-method context improves comments is not yet supported by the numbers.","tokens_in":21073,"tokens_out":2542,"would_cite":false,"duration_ms":26361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dependency-aware prompt that feeds helper-method bodies to an LLM produces measurably better comments for the majority of Java methods that call other methods.","keywords":["method dependencies","code comment generation","code summarization","helper methods","large language models","in-context learning","code understanding","Java methods"],"falsifier":"Re-run the HelpCOM comparison on a sample where helper-method relationships are verified by a build tool or dynamic call tracing, and include a control prompt with unrelated method bodies; if the advantage over the same LLM without helpers drops below the reported 5.6% or disappears, the dependency-specific claim would be refuted.","tokens_in":19980,"feed_emoji":"💬","tokens_out":6799,"duration_ms":65361,"temperature":0.7,"pith_summary":"The paper tries to establish that method dependencies—one method calling another—are a hidden reason why automatic comment generators produce weak or generic comments. In ten large, active Java projects, dependent methods make up 69.25% of all methods, are changed more often, and draw more developer engagement than independent methods. Existing fine-tuned and LLM-based comment generators score noticeably lower on dependent methods, and the paper's proposed technique, HelpCOM, closes much of that gap simply by adding the bodies of the called helper methods to the prompt. If the claim holds, dependency-aware prompt construction is a cheap, broadly applicable fix for a problem that affects the majority of methods in real code.","feed_headline":"Adding helper methods to the prompt lifts comments by up to 50.4%","feed_subtitle":"A dependency-aware prompt beats single-method summarizers on the 69% of Java methods that call other methods.","key_machinery":"The load-bearing mechanism is prompt construction: HelpCOM retrieves the helper methods called by a dependent method, writes the dependent method's body followed by the helper bodies at each level, and instructs the LLM to output only the summary comment. Dependency classification is built by parsing Java files and matching method-invocation call sites to stored method definitions by method name and parameter count; HelpCOM1 includes only immediate helpers, while HelpCOMN includes the full transitive helper chain. This turns method dependency into extra in-context evidence, letting the LLM reason about what the helpers contribute before writing the comment.","core_discovery":"The paper's central claim is that method dependencies are the hidden variable in automatic comment generation. In ten large Java projects, 69.25% of methods call at least one other method, and these dependent methods account for 77.37% of commits and show higher author engagement than independent methods. Across fine-tuned code models and LLM-based summarizers, comments for dependent methods score 7.3% to 13.5% lower on the paper's overall metric than comments for independent methods. The proposed technique, HelpCOM, places the bodies of helper methods into the prompt alongside the dependent method and asks the LLM to write a Javadoc-style summary; the full-chain version outperformed every baseline by 5.6% to 50.4% on the overall metric combining syntactic, semantic, and LLM-based scores, and about 75% of 156 practitioners chose HelpCOM's comment in a blinded comparison.","pith_inferences":["Because helper matching relies on name and parameter count, the 69% figure and the reported performance gap are upper-bound estimates of dependency effects; a precise call graph from build artifacts or dynamic traces could change both.","A natural control experiment is to append unrelated method bodies to the same prompt; if comment quality rises equally, the gain is generic extra context rather than dependency information specifically.","The low end of the 5.6%–50.4% range may reflect methods with shallow dependencies, while the high end may apply where helper bodies carry most of the meaning; tool builders should expect context relevance to matter most for delegating methods.","An iterative extension would generate a first comment, let the LLM request specific helper bodies, then revise; the paper's static chain inclusion is one point on that design space."],"forward_implications":["Dependent methods are the common case, not the exception: roughly 69% of methods in the studied projects call other methods, so single-method comment generators are working on the majority case without the context it needs.","Existing fine-tuned and LLM-based summarizers score 7.3% to 13.5% higher on independent methods than on dependent ones, meaning benchmark results over mixed sets overstate what developers will experience on real code.","Adding helper-method bodies to the prompt improved the best overall metric by 5.6% to 50.4% over baselines, and the improvement held whether immediate helpers or the full helper chain were included.","The practitioner survey supports the same conclusion: about 75% preferred HelpCOM's comment, and participants rated dependent methods easier to understand once helper definitions were visible.","The same prompting approach transferred to Python and PHP samples with higher overall scores than on the Java sample, suggesting the mechanism is not Java-specific."],"supporting_citations":[{"why":"Supplies the LLM-based prompt-augmentation baseline that HelpCOM extends and must beat.","marker":"[4]"},{"why":"Provides the fine-tuned CodeBERT baseline used to compare dependent versus independent method comment quality.","marker":"[16]"},{"why":"Supplies the in-context-learning approach and prompt-design insights for LLM-based comment generation.","marker":"[20]"},{"why":"Provides the large code-comment corpus behind the fine-tuned baselines and motivates why a new dataset with helper-method relationships is needed.","marker":"[31]"},{"why":"Provides the SIDE metric used to filter weak ground-truth comments and align generated summaries with source code.","marker":"[50]"},{"why":"Supplies the SentenceBERT semantic similarity metric used in evaluation.","marker":"[57]"},{"why":"Provides the LLM-as-evaluator prompt used for the LLM-based evaluation scores.","marker":"[67]"},{"why":"Provides the fine-tuned CodeT5+ baseline used in the dependent-versus-independent comparison and the HelpCOM evaluation.","marker":"[76]"}],"fun_headline_variants":["Method calls matter: 69% of code depends on context for comments","Helper methods in prompt boost comment relevance by up to 50.4%","Dependency-aware prompts beat single-method summarizers in Java","Why code comments fail: dependent methods are the culprit","HelpCOM: adding helper code lifts comment scores 5-50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that matching called method names plus parameter counts to method definitions correctly identifies which methods are dependent and which bodies are the relevant helpers; if that graph is noisy, both the baseline gap and HelpCOM's measured advantage could be artifacts of the matching rather than of genuine method dependencies.","fun_headline_variants_meta":{"raw":{"variants":["Method calls matter: 69% of code depends on context for comments","Helper methods in prompt boost comment relevance by up to 50.4%","Dependency-aware prompts beat single-method summarizers in Java","Why code comments fail: dependent methods are the culprit","HelpCOM: adding helper code lifts comment scores 5-50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000829,"raw_usage":{"total_tokens":3651,"prompt_tokens":1006,"completion_tokens":2645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":2554}},"tokens_in":622,"tokens_out":2645,"duration_ms":21138,"temperature":1.0,"reasoning_tokens":2554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:51:39.656304+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the HelpCOM comparison on a sample where helper-method relationships are verified by a build tool or dynamic call tracing, and include a control prompt with unrelated method bodies; if the advantage over the same LLM without helpers drops below the reported 5.6% or disappears, the dependency-specific claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SIDE metric used to filter weak ground-truth comments and align generated summaries with source code."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LLM-as-evaluator prompt used for the LLM-based evaluation scores."}],"review_version":1}