{"id":"061036d2-12c7-418e-8780-0fff2a1cbae2","arxiv_id":"2412.09360","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using only Javadoc as input, fine-tuned LLMs generate test oracles that match ground truth as often as those using method code, and detect more Defects4J bugs than TOGA and nl2postcondition.","lead":"This paper tests whether Java documentation comments (Javadoc) can replace the actual code when training AI models to generate test assertions. It finds that documentation alone produces as many or more correct assertions and catches more real software bugs than using the code itself.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-generated Javadoc for 86 of 374 Defects4J samples is generated from the method implementation, so the 'Javadoc-only' bug-detection claim may leak MUT behavior and cannot support the stated SOTA result as presented.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: GPT-generated Javadoc derived from the MUT implementation can encode the very behavior the approach claims to avoid. This concern directly undermines the paper's strongest contribution, the Defects4J SOTA comparison, because 86 of the 374 samples in the headline count use such generated documentation. The paper's other issues—selection of the top three models after observing Defects4J results, omission of TOGLL as a baseline, and the non-significant P3–P4 t-test—are real but secondary: they affect the strength and framing of the comparison, whereas the generated-Javadoc leakage threatens the validity of the 'Javadoc-only' condition itself. The paper does have independent support for weaker claims: Table 1's P1→P2 improvement with developer-written Javadoc is consistent across ten models, RQ2's ablation identifying description and @return as most valuable is a reasonable empirical finding, and the RQ5 qualitative taxonomy is useful. Those contributions would survive even if the SOTA claim were weakened. Because the central SOTA claim is conditional on a check that the paper does not perform, the conditional verdict is appropriate; my analysis does not move the verdict, so I recommend UNCHANGED.","tokens_in":23408,"tokens_out":5063,"duration_ms":54176,"concrete_test":"Regenerate Javadoc for the 86 Defects4J samples that lacked developer-written comments using the fixed version of the method (or from a specification-only prompt that excludes all method code), then rerun the RQ4 P2 bug-detection study for all five models from Table 4. If the total unique bugs detected remains at approximately 68, the leakage concern is not material; if it drops materially, the reported SOTA gain depends on Javadoc that summarizes the buggy implementation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim—that Javadoc comments alone detect 19–94% more real-world bugs than prior methods—rests on RQ4's Table 5, which aggregates over all 374 Defects4J samples. Section 3.4.1 states that for 86 samples without Javadoc, comments were 'generated using the procedure described in RQ3,' and Section 3.3.1 describes that procedure as generating Javadoc 'directly from the method implementation using a GPT model.' On Defects4J, the natural input to that generator is the buggy version of the MUT. If so, the generated Javadoc can summarize the buggy behavior rather than the intended contract, and the 'Javadoc-only' condition is not actually independent of the implementation. The leaked code can supply the very behavioral hints that the paper attributes to documentation, inflating the RQ4 bug count for those 86 samples. The same confound affects RQ3: the observed ~10 pp improvement from GPT-generated Javadoc may reflect the model accessing implementation details, not documentation value. The paper never reports a check that generated Javadoc is behaviorally equivalent to developer-written Javadoc, nor does it compare Javadoc generated from buggy versus fixed versions. This is the load-bearing soft spot because the headline 'replace the MUT entirely' claim is only meaningful if the documentation source is decoupled from the implementation under test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Doc2OracLL, a study of how Javadoc documentation affects LLM-based test oracle generation. The authors fine-tune 10 code LLMs on subsets of the SF110 benchmark, using six prompts arranged in three pairs that differ only in whether Javadoc is present, while holding the other inputs (test prefix, method signature, or full method body) constant. RQ1 reports that adding developer-written Javadoc improves exact-match oracle accuracy, most dramatically from 57.95% with only the test prefix to 78.11% with prefix plus Javadoc. RQ2 ablates Javadoc components and concludes that the description and @return tag are the most valuable. RQ3 finds that GPT-3.5-generated Javadoc also improves accuracy by about 10 percentage points. RQ4 applies the fine-tuned models to Defects4J and reports that Javadoc-only prompts detect up to 68 bugs, which the paper claims is 19-94% more than TOGA and nl2postcondition. RQ5 provides a qualitative taxonomy of helpful and harmful Javadoc characteristics. The paper's central conclusion is that Javadoc alone can replace the method-under-test code in the prompt.","tokens_in":23614,"tokens_out":7453,"duration_ms":71642,"significance":"If the central claims hold, the paper is a useful empirical contribution: it systematically quantifies the marginal value of Javadoc across a broad family of small code LLMs, identifies the description and @return tag as the high-value components, and demonstrates on an external benchmark that implementation-free oracles can be competitive. The controlled prompt-pair design is a genuine strength, as is the breadth of models and the use of Defects4J for external validation. The qualitative taxonomy in RQ5 is also valuable for practitioners. However, the headline bug-detection result is currently supported only by a protocol that (a) selects models after observing their Defects4J bug counts, and (b) uses GPT-generated Javadoc that is produced directly from the method implementation, including for 86 of the 374 Defects4J samples. This means the central 'Javadoc can replace the MUT' conclusion is not yet established at the strength claimed in the abstract.","major_comments":[{"comment":"The Defects4J evaluation selects models using the evaluation data itself. The authors first evaluate five models on the 288-sample subset (Table 4), then 'selected the top three for further analysis' before reporting Table 5 on all 374 samples. The headline count of 68 bugs is the best P2 count among the three selected models (CodeGemma-7B). Selecting models after observing their bug-detection counts on the target benchmark and then reporting the best count as 'our method' overstates expected performance and invalidates the comparison against TOGA and nl2postcondition. The paper should either fix the model and prompt pair before evaluation, report all models with confidence intervals, or validate the selection on a separate set of bugs.","section":"Section 3.4.1, Table 5"},{"comment":"The GPT-generated Javadoc leaks implementation information. RQ3 states that Javadoc is 'generated directly from the method implementation using a GPT model,' and RQ4 uses 'the procedure described in RQ3' for the 86 Defects4J samples that lacked developer-written Javadoc. On a buggy Defects4J version, the natural input to that procedure is the buggy implementation, so the generated Javadoc can summarize faulty behavior rather than the intended contract. The 'Javadoc-only' condition is then not independent of the MUT, and the higher bug counts on those 86 samples may be due to leaked code rather than documentation value. The paper reports no check that GPT-generated Javadoc is behaviorally equivalent to developer-written Javadoc, and no comparison of Javadoc generated from buggy versus fixed versions. To support the 'replace the MUT entirely' claim, RQ4 must be re-run on the 288 developer-written samples alone, or with generated comments derived from the fixed versions, with a clear statement of which version was used.","section":"Sections 3.3.1 and 3.4.1"},{"comment":"The claimed 19-94% improvement over prior methods is not a controlled comparison. The 68-bug count is the best P2 result among three post-hoc selected models, while TOGA's 57 includes bugs detected via implicit oracles and nl2postcondition uses GPT-4, ten generated assertions per sample, and class-level comments. Furthermore, the 19-94% range is derived from a single selected model's count, with no confidence interval or significance test. A credible SOTA comparison needs a pre-specified model/prompt pair (or all models reported individually), a matched definition of what counts as a detected bug, and, ideally, a comparison of the same number of generated assertions per sample.","section":"Section 3.4.2, 'Comparison With Baselines'"},{"comment":"The conclusion that 'Javadoc comments alone can encode sufficient contextual information to replace the MUT code entirely' is stronger than Table 5 supports. The P2-versus-P5 comparisons are inconsistent across models (67 vs 62 for CodeGen-350M, 63 vs 68 for CodeLlama-7B, 68 vs 62 for CodeGemma-7B), no statistical test is reported for these differences, and the sample sizes are small. In addition, the P2 prompt still includes a test prefix containing calls and variable names, so 'alone' overstates the degree of isolation. The claim should be softened to 'comparable on this dataset' and supported with a paired analysis across models or additional oracle-strength evidence.","section":"Section 3.4.2, RQ4 Finding"}],"minor_comments":[{"comment":"The text says '56% of the SF110 dataset... lacks method-level comments,' but Section 2.1.1 reports 55,575 of 140,514 samples (about 40%) contain Javadoc, which means about 60% lack it; the 56% figure is inconsistent with the paper's own numbers.","section":"Section 3.3, first paragraph"},{"comment":"The sentence 'Figure 6 presents two Defects4J bugs uniquely detected due to Javadoc comments' appears to refer to the Defects4J examples in Table 6, while the existing Figure 6 in Section 3.2 shows @param and @see tag effects; the figure/table numbering should be corrected.","section":"Section 3.4.2, Figure reference"},{"comment":"The improvement ranges are stated inconsistently: the abstract says '19% and 94%,' the introduction says 'up to 44%,' and the RQ4 Finding says '19-45%' for methods that use the MUT and '94%' for methods that omit it; these should be harmonized with a precise definition of the denominator in each comparison.","section":"Abstract and RQ4 Finding"},{"comment":"The last row labels the statistic as 't-test (p-value)' but reports only p-values; it should state whether a paired t-test was used, whether multiple-comparison correction was applied across the three prompt pairs, and ideally report the test statistic or effect size.","section":"Table 1, t-test row"},{"comment":"The text says Defects4J 'includes a total of 374 input samples' but Defects4J contains 835 bugs; the paper should specify how the 374 samples were selected and filtered, and should state that this follows prior TOG work rather than using the full Defects4J suite.","section":"Section 3.4.1, dataset description"}],"recommendation":"major_revision","confidential_remarks":"The RQ1/RQ2 accuracy findings appear solid and likely publishable; the difficulty is concentrated in the RQ4 SOTA claim and the RQ3 generation design. If the authors can separate developer-written from GPT-generated Javadoc in the Defects4J analysis and replace the post-hoc model-selection protocol with a pre-specified model or an honest reporting of all models, the 'Javadoc can replace the MUT' conclusion would be credible. As it stands, the abstract's state-of-the-art claim is not supported by the reported protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the controlled ablation. Across ten LLMs, adding Javadoc to a minimal prompt gives about 20pp exact-match gain, and the RQ2 result that description and @return carry most of that value is solid and practically useful. The RQ5 qualitative taxonomy is also a genuine contribution. Filtering the SF110 dataset to samples that actually have Javadoc fixes a real weakness in the prior TOGLL setup. The soft spots are in the bug-detection headline. The claim that Javadoc-only prompts detect 19-94% more bugs than prior methods rests on Table 5, which aggregates over 374 Defects4J samples. For 86 of those, the Javadoc was generated with GPT-3.5 directly from the method implementation. That leaks implementation behavior into the documentation-only condition; if the generator saw the buggy version, the generated doc can encode the bug itself. The paper never reports results on the 288 human-doc samples separately for the final three models, nor does it compare generated Javadoc from buggy versus fixed versions. This alone prevents the SOTA claim from standing. Two further issues. First, the top three models for the full Defects4J run were selected after evaluating all five on a subset of that same dataset. That is model selection on the test set, and it will inflate reported performance. Second, TOGLL - the authors' own prior SOTA - is never included as a baseline, so new state-of-the-art is unsubstantiated even setting leakage aside. Also, the RQ1 text reports the P3-to-P4 comparison as an improvement, but the t-test p-value is 0.64, clearly not significant; that should be stated honestly. The RQ1-RQ3 findings are mostly supported by the tables, and the authors do acknowledge the exact-match metric is conservative. The P2 approximately equal to P6 result is interesting but should be read as documentation can substitute for code in these prompt designs, not Javadoc alone is sufficient, given the confounds. Who gets value from this: researchers working on LLM-based test generation, and anyone building oracle prompts. The component ablation is worth citing. It deserves a serious referee, but acceptance should require re-running the Defects4J study with Javadoc generated from fixed versions or a separate specification, reporting all models without post-hoc selection, and adding TOGLL as a baseline. With those changes, the central claims could hold.","headline":"Useful ablation of Javadoc components for oracle generation, but the bug-detection SOTA claim is undermined by generated-documentation leakage and test-set model selection.","tokens_in":731,"tokens_out":1206,"would_cite":true,"duration_ms":34083,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Javadoc comments alone can generate test oracles as accurately as the method code and catch more real bugs.","keywords":["test oracle generation","Javadoc","large language models","software testing","bug detection","Defects4J","documentation quality","prompt ablation"],"falsifier":"Replace every GPT-generated Javadoc in the Defects4J study with documentation written independently of the buggy code (for example, from the fixed version or by a developer who has not seen the bug), then re-run oracle generation; the claim that Javadoc alone detects 19-94% more bugs than prior methods stands only if the bug counts remain in that range.","tokens_in":23126,"feed_emoji":"🐞","tokens_out":7624,"duration_ms":66104,"temperature":0.7,"pith_summary":"This paper tries to show that a method's documentation, not its implementation, is enough for a large language model to generate strong test oracles. Across ten fine-tuned models and prompt pairs that differ only in whether Javadoc comments are present, adding the documentation raises exact-match oracle accuracy from about 58% to 78% on a large Java corpus. On the Defects4J benchmark, oracles generated from Javadoc alone detect between 19% and 94% more real-world bugs than two earlier oracle-generation methods. The reason that matters is that oracles can then be anchored to intended behavior instead of to code that may contain the very bug the test is supposed to reveal.","feed_headline":"Javadoc alone beats method code at finding bugs","feed_subtitle":"Docs-only prompts catch 19-94% more real bugs than earlier oracle-generation methods.","key_machinery":"The load-bearing object is the Javadoc comment, a structured natural-language header attached to Java methods, and within it the description sentence and the @return tag. The controlled prompt-pair design is the central mechanism: three pairs of prompts differ by exactly one input element, so any accuracy shift is attributable to the presence or absence of documentation. A second mechanism is the component ablation, which removes the description and each tag in turn to isolate which pieces carry the behavioral contract. On the bug-detection side, the machinery is running the generated oracles against both buggy and fixed versions of Defects4J programs so that only bugs the oracles actually expose are counted.","core_discovery":"The paper's central claim is that Javadoc comments encode enough of a method's contract to replace the method-under-test code in test-oracle generation. With three prompt pairs that add or remove Javadoc while holding all other context fixed, a minimal prompt containing only the test prefix plus Javadoc reaches about 78% exact match to ground-truth oracles, close to the roughly 81% achieved by the maximum-information prompt that also includes the full method body. Removing individual Javadoc elements shows that the description sentence and the @return tag carry most of the value; removing both drops accuracy by about 25 percentage points. In a real-bug study, oracles produced from Javadoc alone detect 68 unique bugs, compared with 47 for a prior LLM-based method that also had access to the buggy implementation. The authors conclude that documentation-based oracles are stronger because they capture intended behavior and avoid learning buggy implementation details.","pith_inferences":["Because the GPT-generated Javadoc used for 86 of the 374 Defects4J samples is produced from the method implementation itself, the documentation-only condition in those cases is not fully independent of the code; a stricter test would use documentation written without access to the buggy behavior.","The same prompt-pair and ablation design could be carried to other documentation formats, such as Python docstrings or API reference pages, to test whether the description-plus-return dominance generalizes.","If the result holds, documentation quality could be treated as a testability feature: linters and review checklists that reward explicit return-behavior statements would also be improving the automated oracle quality of the codebase.","A direct extension would be to report disagreements between Javadoc-based and implementation-based oracles to developers, converting the generation method into a tool for finding doc-code inconsistencies."],"forward_implications":["Test-oracle generation can work without the method implementation, so oracles are not biased toward reproducing whatever the current code happens to do.","When prompt length is limited, prioritizing a method's description and @return tag preserves most of the documentation's value for oracle quality.","Reliable oracle generation does not require waiting for human-written documentation: generated summaries of methods improve oracle accuracy by about 10 percentage points over no documentation.","Oracles generated from documentation alone can be checked against oracles generated from the implementation, highlighting methods whose code does not satisfy its documented contract."],"supporting_citations":[{"why":"Supplies the baseline neural oracle-generation method whose bug-detection counts and use of the method implementation the paper compares against.","marker":"[10]"},{"why":"Supplies the baseline LLM postcondition-generation method from Javadoc plus buggy code, giving the 47-bug and 35-bug comparison points.","marker":"[11]"},{"why":"Provides the prior fine-tuning approach and prompt design the study extends, and the earlier finding that 60% of its data lacked Javadoc motivates this work.","marker":"[17]"},{"why":"Defects4J provides the real-world bugs and the fixed or buggy program pairs used for the bug-detection evaluation.","marker":"[23]"},{"why":"SF110 supplies the 110-project, 140,514-sample corpus from which the 55,575 samples with Javadoc are drawn for fine-tuning and testing.","marker":"[13]"},{"why":"Documents that Javadoc quality varies in practice, motivating the generated-documentation variant and the qualitative taxonomy.","marker":"[42]"},{"why":"Defines the Javadoc format and its components, the elements the ablation study removes and ranks.","marker":"[30]"}],"fun_headline_variants":["Javadoc alone nearly matches full-context oracle accuracy","Description and @return tags drive Javadoc oracle generation","Javadoc docs beat method code for finding real bugs","Skip the code, use Javadoc contract for stronger oracles","Javadoc's intended behavior outweighs buggy implementation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Javadoc comments describe intended behavior rather than the quirks of the buggy implementation; since some comments in the study are generated from the implementation itself, that independence is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Javadoc alone nearly matches full-context oracle accuracy","Description and @return tags drive Javadoc oracle generation","Javadoc docs beat method code for finding real bugs","Skip the code, use Javadoc contract for stronger oracles","Javadoc's intended behavior outweighs buggy implementation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1550,"prompt_tokens":871,"completion_tokens":679,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":596}},"tokens_in":487,"tokens_out":679,"duration_ms":6191,"temperature":1.0,"reasoning_tokens":596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:34.588489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace every GPT-generated Javadoc in the Defects4J study with documentation written independently of the buggy code (for example, from the fixed version or by a developer who has not seen the bug), then re-run oracle generation; the claim that Javadoc alone detects 19-94% more bugs than prior methods stands only if the bug counts remain in that range.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the baseline LLM postcondition-generation method from Javadoc plus buggy code, giving the 47-bug and 35-bug comparison points."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that Javadoc quality varies in practice, motivating the generated-documentation variant and the qualitative taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Javadoc format and its components, the elements the ablation study removes and ranks."}],"review_version":1}