Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation

Matthew Dwyer; Raygan Taylor; Soneya Binta Hossain

REVIEW 4 major objections 5 minor 3 cited by

Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Javadoc comments alone can generate test oracles as accurately as the method code and catch more real bugs.

desk verdict Useful ablation of Javadoc components for oracle generation, but the bug-detection SOTA claim is undermined by generated-documentation leakage and test-set model selection. read the letter →

arxiv 2412.09360 v2 pith:YZ6Q7P33 submitted 2024-12-12 cs.SE

Soneya Binta Hossain , Raygan Taylor , Matthew Dwyer This is my paper · ORCID

classification cs.SE

keywords testoraclegenerationJavadoclargelanguagemodelssoftwaretestingbugdetectionDefects4Jdocumentationqualitypromptablation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a method's documentation, not its implementation, is enough for a large language model to generate strong test oracles. Across ten fine-tuned models and prompt pairs that differ only in whether Javadoc comments are present, adding the documentation raises exact-match oracle accuracy from about 58% to 78% on a large Java corpus. On the Defects4J benchmark, oracles generated from Javadoc alone detect between 19% and 94% more real-world bugs than two earlier oracle-generation methods. The reason that matters is that oracles can then be anchored to intended behavior instead of to code that may contain the very bug the test is supposed to reveal.

What carries the argument

The load-bearing object is the Javadoc comment, a structured natural-language header attached to Java methods, and within it the description sentence and the @return tag. The controlled prompt-pair design is the central mechanism: three pairs of prompts differ by exactly one input element, so any accuracy shift is attributable to the presence or absence of documentation. A second mechanism is the component ablation, which removes the description and each tag in turn to isolate which pieces carry the behavioral contract. On the bug-detection side, the machinery is running the generated oracles against both buggy and fixed versions of Defects4J programs so that only bugs the oracles actually expose are counted.

What would settle it

Replace every GPT-generated Javadoc in the Defects4J study with documentation written independently of the buggy code (for example, from the fixed version or by a developer who has not seen the bug), then re-run oracle generation; the claim that Javadoc alone detects 19-94% more bugs than prior methods stands only if the bug counts remain in that range.

Watch

Extended reading notes

Core claim

The paper's central claim is that Javadoc comments encode enough of a method's contract to replace the method-under-test code in test-oracle generation. With three prompt pairs that add or remove Javadoc while holding all other context fixed, a minimal prompt containing only the test prefix plus Javadoc reaches about 78% exact match to ground-truth oracles, close to the roughly 81% achieved by the maximum-information prompt that also includes the full method body. Removing individual Javadoc elements shows that the description sentence and the @return tag carry most of the value; removing both drops accuracy by about 25 percentage points. In a real-bug study, oracles produced from Javadoc alone detect 68 unique bugs, compared with 47 for a prior LLM-based method that also had access to the buggy implementation. The authors conclude that documentation-based oracles are stronger because they capture intended behavior and avoid learning buggy implementation details.

Load-bearing premise

The load-bearing premise is that the Javadoc comments describe intended behavior rather than the quirks of the buggy implementation; since some comments in the study are generated from the implementation itself, that independence is not guaranteed.

Editorial extensions

If this is right

Test-oracle generation can work without the method implementation, so oracles are not biased toward reproducing whatever the current code happens to do.
When prompt length is limited, prioritizing a method's description and @return tag preserves most of the documentation's value for oracle quality.
Reliable oracle generation does not require waiting for human-written documentation: generated summaries of methods improve oracle accuracy by about 10 percentage points over no documentation.
Oracles generated from documentation alone can be checked against oracles generated from the implementation, highlighting methods whose code does not satisfy its documented contract.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the GPT-generated Javadoc used for 86 of the 374 Defects4J samples is produced from the method implementation itself, the documentation-only condition in those cases is not fully independent of the code; a stricter test would use documentation written without access to the buggy behavior.
The same prompt-pair and ablation design could be carried to other documentation formats, such as Python docstrings or API reference pages, to test whether the description-plus-return dominance generalizes.
If the result holds, documentation quality could be treated as a testability feature: linters and review checklists that reward explicit return-behavior statements would also be improving the automated oracle quality of the codebase.
A direct extension would be to report disagreements between Javadoc-based and implementation-based oracles to developers, converting the generation method into a tool for finding doc-code inconsistencies.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful core here is the controlled ablation. Across ten LLMs, adding Javadoc to a minimal prompt gives about 20pp exact-match gain, and the RQ2 result that description and @return carry most of that value is solid and practically useful. The RQ5 qualitative taxonomy is also a genuine contribution. Filtering the SF110 dataset to samples that actually have Javadoc fixes a real weakness in the prior TOGLL setup. The soft spots are in the bug-detection headline. The claim that Javadoc-only prompts detect 19-94% more bugs than prior methods rests on Table 5, which aggregates over 374 Defects4J samples. For 86 of those, the Javadoc was generated with GPT-3.5 directly from the method implementation. That leaks implementation behavior into the documentation-only condition; if the generator saw the buggy version, the generated doc can encode the bug itself. The paper never reports results on the 288 human-doc samples separately for the final three models, nor does it compare generated Javadoc from buggy versus fixed versions. This alone prevents the SOTA claim from standing. Two further issues. First, the top three models for the full Defects4J run were selected after evaluating all five on a subset of that same dataset. That is model selection on the test set, and it will inflate reported performance. Second, TOGLL - the authors' own prior SOTA - is never included as a baseline, so new state-of-the-art is unsubstantiated even setting leakage aside. Also, the RQ1 text reports the P3-to-P4 comparison as an improvement, but the t-test p-value is 0.64, clearly not significant; that should be stated honestly. The RQ1-RQ3 findings are mostly supported by the tables, and the authors do acknowledge the exact-match metric is conservative. The P2 approximately equal to P6 result is interesting but should be read as documentation can substitute for code in these prompt designs, not Javadoc alone is sufficient, given the confounds. Who gets value from this: researchers working on LLM-based test generation, and anyone building oracle prompts. The component ablation is worth citing. It deserves a serious referee, but acceptance should require re-running the Defects4J study with Javadoc generated from fixed versions or a separate specification, reporting all models without post-hoc selection, and adding TOGLL as a baseline. With those changes, the central claims could hold.

Referee Report

4 major / 5 minor

Summary. The paper presents Doc2OracLL, a study of how Javadoc documentation affects LLM-based test oracle generation. The authors fine-tune 10 code LLMs on subsets of the SF110 benchmark, using six prompts arranged in three pairs that differ only in whether Javadoc is present, while holding the other inputs (test prefix, method signature, or full method body) constant. RQ1 reports that adding developer-written Javadoc improves exact-match oracle accuracy, most dramatically from 57.95% with only the test prefix to 78.11% with prefix plus Javadoc. RQ2 ablates Javadoc components and concludes that the description and @return tag are the most valuable. RQ3 finds that GPT-3.5-generated Javadoc also improves accuracy by about 10 percentage points. RQ4 applies the fine-tuned models to Defects4J and reports that Javadoc-only prompts detect up to 68 bugs, which the paper claims is 19-94% more than TOGA and nl2postcondition. RQ5 provides a qualitative taxonomy of helpful and harmful Javadoc characteristics. The paper's central conclusion is that Javadoc alone can replace the method-under-test code in the prompt.

Significance. If the central claims hold, the paper is a useful empirical contribution: it systematically quantifies the marginal value of Javadoc across a broad family of small code LLMs, identifies the description and @return tag as the high-value components, and demonstrates on an external benchmark that implementation-free oracles can be competitive. The controlled prompt-pair design is a genuine strength, as is the breadth of models and the use of Defects4J for external validation. The qualitative taxonomy in RQ5 is also valuable for practitioners. However, the headline bug-detection result is currently supported only by a protocol that (a) selects models after observing their Defects4J bug counts, and (b) uses GPT-generated Javadoc that is produced directly from the method implementation, including for 86 of the 374 Defects4J samples. This means the central 'Javadoc can replace the MUT' conclusion is not yet established at the strength claimed in the abstract.

major comments (4)

[Section 3.4.1, Table 5] The Defects4J evaluation selects models using the evaluation data itself. The authors first evaluate five models on the 288-sample subset (Table 4), then 'selected the top three for further analysis' before reporting Table 5 on all 374 samples. The headline count of 68 bugs is the best P2 count among the three selected models (CodeGemma-7B). Selecting models after observing their bug-detection counts on the target benchmark and then reporting the best count as 'our method' overstates expected performance and invalidates the comparison against TOGA and nl2postcondition. The paper should either fix the model and prompt pair before evaluation, report all models with confidence intervals, or validate the selection on a separate set of bugs.
[Sections 3.3.1 and 3.4.1] The GPT-generated Javadoc leaks implementation information. RQ3 states that Javadoc is 'generated directly from the method implementation using a GPT model,' and RQ4 uses 'the procedure described in RQ3' for the 86 Defects4J samples that lacked developer-written Javadoc. On a buggy Defects4J version, the natural input to that procedure is the buggy implementation, so the generated Javadoc can summarize faulty behavior rather than the intended contract. The 'Javadoc-only' condition is then not independent of the MUT, and the higher bug counts on those 86 samples may be due to leaked code rather than documentation value. The paper reports no check that GPT-generated Javadoc is behaviorally equivalent to developer-written Javadoc, and no comparison of Javadoc generated from buggy versus fixed versions. To support the 'replace the MUT entirely' claim, RQ4 must be re-run on the 288 developer-written samples alone, or with generated comments derived from the fixed versions, with a clear statement of which version was used.
[Section 3.4.2, 'Comparison With Baselines'] The claimed 19-94% improvement over prior methods is not a controlled comparison. The 68-bug count is the best P2 result among three post-hoc selected models, while TOGA's 57 includes bugs detected via implicit oracles and nl2postcondition uses GPT-4, ten generated assertions per sample, and class-level comments. Furthermore, the 19-94% range is derived from a single selected model's count, with no confidence interval or significance test. A credible SOTA comparison needs a pre-specified model/prompt pair (or all models reported individually), a matched definition of what counts as a detected bug, and, ideally, a comparison of the same number of generated assertions per sample.
[Section 3.4.2, RQ4 Finding] The conclusion that 'Javadoc comments alone can encode sufficient contextual information to replace the MUT code entirely' is stronger than Table 5 supports. The P2-versus-P5 comparisons are inconsistent across models (67 vs 62 for CodeGen-350M, 63 vs 68 for CodeLlama-7B, 68 vs 62 for CodeGemma-7B), no statistical test is reported for these differences, and the sample sizes are small. In addition, the P2 prompt still includes a test prefix containing calls and variable names, so 'alone' overstates the degree of isolation. The claim should be softened to 'comparable on this dataset' and supported with a paired analysis across models or additional oracle-strength evidence.

minor comments (5)

[Section 3.3, first paragraph] The text says '56% of the SF110 dataset... lacks method-level comments,' but Section 2.1.1 reports 55,575 of 140,514 samples (about 40%) contain Javadoc, which means about 60% lack it; the 56% figure is inconsistent with the paper's own numbers.
[Section 3.4.2, Figure reference] The sentence 'Figure 6 presents two Defects4J bugs uniquely detected due to Javadoc comments' appears to refer to the Defects4J examples in Table 6, while the existing Figure 6 in Section 3.2 shows @param and @see tag effects; the figure/table numbering should be corrected.
[Abstract and RQ4 Finding] The improvement ranges are stated inconsistently: the abstract says '19% and 94%,' the introduction says 'up to 44%,' and the RQ4 Finding says '19-45%' for methods that use the MUT and '94%' for methods that omit it; these should be harmonized with a precise definition of the denominator in each comparison.
[Table 1, t-test row] The last row labels the statistic as 't-test (p-value)' but reports only p-values; it should state whether a paired t-test was used, whether multiple-comparison correction was applied across the three prompt pairs, and ideally report the test statistic or effect size.
[Section 3.4.1, dataset description] The text says Defects4J 'includes a total of 374 input samples' but Defects4J contains 835 bugs; the paper should specify how the 374 samples were selected and filtered, and should state that this follows prior TOG work rather than using the full Defects4J suite.

Circularity Check

2 steps flagged · score 4.0 of 10

GPT-generated Javadoc for 86 of 374 Defects4J samples is derived from the MUT implementation, and top models are selected on a Defects4J subset, so the headline 'Javadoc-only' SOTA claim is partially self-fed; the SF110 accuracy results remain independent.

self definitional [Section 3.4.1 (Experimental Setup) and Section 3.3.1 (RQ3 Experimental Setup)]
"For the 86 samples where Javadoc comments were unavailable, we generated them using the procedure described in RQ3 (Section 3.3). ... we generate Javadoc comments directly from the method implementation using a GPT model."

The paper's central claim is that 'Javadoc comments alone can encode sufficient contextual information to replace the MUT code entirely' (Section 3.4.2). For 86 of the 374 Defects4J samples, the 'Javadoc-only' prompt P2 is constructed from Javadoc that was generated directly from the MUT implementation. The documentation input is therefore a function of the implementation it is claimed to replace; the comparison P2 (prefix+Javadoc) vs P5 (prefix+MUT) is, for those samples, a comparison between the MUT and a lossy summary of the MUT. If the generator summarizes buggy behavior, the 'Javadoc-only' condition leaks the very code it claims to exclude.
fitted input called prediction [Section 3.4.1 (RQ4 Experimental Setup, model selection paragraph)]
"After an initial study on all five models using P1 and P2 on the Defects4J dataset (Table 4), we selected the top three for further analysis. ... Table 5 reports the total number of unique bugs detected by different model-prompt combinations."

The headline '19% to 94% more real-world bugs' is computed from Table 5, which reports bug counts on all 374 Defects4J samples. The models used in Table 5 were selected because they performed best on Table 4, a 288-sample subset of the same 374-sample benchmark. Model choice is thus a selection hyperparameter fitted to a subset of the evaluation set, and the full-benchmark bug counts are not a held-out prediction: the reported SOTA numbers are biased upward by the selection step. This is the 'fitted parameter renamed as prediction' pattern, since the best-performing model-prompt pairs were chosen after seeing a large part of the test data.

full rationale

The SF110* training/evaluation (RQ1 and RQ2) uses a proper 90/5/5 split with held-out test data, and the developer-written Javadoc is independent of the oracle labels; those accuracy results are not circular. The Defects4J evaluation is an external real-world benchmark, so the core bug-detection result is not circular in the strongest sense. However, the central SOTA claim is partially compromised by two reductions visible in the paper's own text: (1) for 86 of 374 Defects4J samples, the 'Javadoc-only' input is generated directly from the MUT implementation, so the documentation is a function of the very code it is claimed to replace; and (2) the top models whose results populate the headline comparison were selected after an initial study on a 288-sample subset of the same Defects4J benchmark. The self-citations ([17], [18]) are not load-bearing: they provide context and prior results, but the present claims are empirically evaluated here. Score 4 reflects partial circularity and test-set leakage, not a full reduction of the derivation to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

This is an empirical ML/SE study with no mathematical derivation. The central claim depends on dataset-label correctness, the independence of Javadoc from implementation, and the validity of evaluation protocols. The main free choice is the model selection for the Defects4J bug-detection headline, which affects the reported SOTA numbers.

free parameters (1)

Model selection for Defects4J bug detection = Top 5 models (CodeParrot-110M, CodeGen-350M, Phi-1.3B, CodeLlama-7B, CodeGemma-7B) then top 3 (CodeGen-350M…
The models reported in the headline bug counts were chosen by hand after evaluating on a subset of the same Defects4J samples (Table 4), which inflates the reported maxima in Table 5.

assumptions (4)

domain assumption Ground-truth oracles in SF110* are correct and representative of intended method behavior.
Used as labels for fine-tuning and accuracy computation. If the labels are wrong, all exact-match accuracy numbers are unreliable. Invoked throughout Section 2.1 and 3.1.
domain assumption Developer-written Javadoc in SF110* describes intended behavior independent of the implementation, so Javadoc-only prompts avoid encoding buggy behavior.
Underpins the central claim that Javadoc can replace the MUT. The paper contrasts Figure 1 (Javadoc says throw, MUT returns null) but does not verify this independence for the full dataset. Introduced in the introduction and used in RQ1 and RQ4.
domain assumption The Defects4J bug detection protocol (test passes on fixed version, fails on buggy version) is a valid measure of oracle strength.
Standard benchmark assumption, stated in Section 2.2.1, inherited from prior work.
domain assumption Exact match with the ground truth oracle is a meaningful proxy for oracle correctness.
The authors acknowledge this may underestimate accuracy (Section 2.1.4), but still use it as the primary metric in RQ1-RQ3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation." pith.science (2026). https://pith.science/paper/YZ6Q7P33

@misc{pith2026241209360,
  author       = {Pith},
  title        = {Pith review of: Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZ6Q7P33}},
  note         = {Machine review of arXiv:2412.09360}
}

read the original abstract

Code documentation is a critical aspect of software development, serving as a bridge between human understanding and machine-readable code. Beyond assisting developers in understanding and maintaining code, documentation also plays a critical role in automating various software engineering tasks, such as test oracle generation (TOG). In Java, Javadoc comments provide structured, natural language documentation embedded directly in the source code, typically detailing functionality, usage, parameters, return values, and exceptions. While prior research has utilized Javadoc comments in test oracle generation (TOG), there has not been a thorough investigation into their impact when combined with other contextual information, nor into identifying the most relevant components for generating correct and strong test oracles, or understanding their role in detecting real bugs. In this study, we dive deep into investigating the impact of Javadoc comments on TOG.

Figures

Figures reproduced from arXiv: 2412.09360 by the authors.

**Figure 1.** Incorrect oracle generated from buggy MUT and correct oracle generated from Javadoc comments. witnessed enormous advances in automating support for the first of these [7, 13, 25, 31] and such techniques are now regularly used in practice. Providing effective automated support for the oracle problem has been less successful, though in recent years, researchers have made some progress using machine learning (ML) techn… view at source ↗

**Figure 2.** Overview of our approach Doc2OracLL. 2 APPROACH In [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Examples showing that MUT Sig is not enough to generate correct oracles. prompt (P2). Columns 4 and 5 show the accuracy for prompts P3 and P4, both of which incorporate the test prefix and the signature of the MUT. However, P4 also includes Javadoc comments. Finally, prompts P5 and P6 utilize the implementation of the MUT, with P6 additionally including Javadoc comments. Rows 12 and 13 show the average accuracy and … view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Impact of Javadoc comment’s description on test oracle generation // Example : 1 =================== "/** * Get the L Norm used . * @return the L- norm used */" ---------------------- public void test1 () { NaiveBayesMultinomialText nBText = new NaiveBayesMultinomialTe…

**Figure 5.** Figure 5: Examples where removing @return tag affects (right) and does not affect (left) the generated oracle [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Impact of removing @param tag (left) and @see tag (right) on test oracle generation. To further investigate this overlap, we removed both the description and the @return tags, affecting 10% of the samples (288). Removing the description alone caused a 10 pp accuracy dr…

**Figure 7.** Figure 7: Impact of GPT-generated Javadoc comments on ground truth oracle generation. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Examples showing positive reasons of Javadoc comments for effective TOG comments, identifying recurring patterns, and categorizing the reasons. We iteratively refined these categories to ensure clear and meaningful distinctions. 3.5.2 Results. To find the positive reas…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle
cs.SE 2026-07 conditional novelty 7.0 of 10

LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.
Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing
cs.SE 2025-06 conditional novelty 6.0 of 10

GUIRepair, a cross-modal LLM pipeline that converts issue screenshots into reproduction code and rendered patch screenshots into validation feedback, resolves 157/517 SWE-bench M instances with GPT-4o and 175 with o4-mini.
Large Language Models for Unit Testing: A Systematic Literature Review
cs.SE 2025-06 conditional novelty 5.0 of 10

The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

Reference graph

Works this paper leans on

46 extracted references · 20 canonical work pages · cited by 3 Pith papers

[1]

Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shep- herd. 2020. Software documentation: the practitioners’ perspective. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 590–601

work page 2020
[2]

Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

work page 2019
[3]

Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41, 5 (2014), 507–525

2014
[4]

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. InProceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623

2021
[5]

Ernst, Mauro Pezzè, and Sergio Del- gado Castellanos

Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Sergio Del- gado Castellanos. 2018. Translating Code Comments to Procedure Specifications. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (Amsterdam, Netherlands) (ISSTA 2018). Association for Computing ...

arXiv 2018
[6]

Arianna Blasi, Alessandra Gorla, Michael D Ernst, Mauro Pezzè, and Antonio Carzaniga. 2021. MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation. Journal of Systems and Software 181 (2021), 111041

work page 2021
[7]

Marcel Bohme, Van-Thuan Pham, and Abhik Roychoudhury. 2019. Coverage-Based Greybox Fuzzing as Markov Chain. IEEE Transactions on Software Engineering 45, 05 (2019), 489–506

work page 2019
[8]

CodeLlama. 2024. CodeLlama-7B-Instruct Model on Hugging Face. https://huggingface.co/codellama/CodeLlama-7b- Instruct-hf

work page 2024

Show all 46 references

[9]

CodeParrot. [n. d.]. https://github.com/huggingface/transformers/tree/main/examples/research_projects/codeparrot
[10]

Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: a neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machine...

2022
[11]

Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?Proceedings of the ACM on Software Engineering 1, FSE (2024), 1889–1912

2024
[12]

Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

2011
[13]

Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Trans. Softw. Eng. Methodol. 24, 2, Article 8 (dec 2014), 42 pages. https://doi.org/10.1145/2685612

2014 doi
[14]

Ernst, and Mauro Pezzè

Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Automatic Generation of Oracles for Exceptional Behaviors. InProceedings of the 25th International Symposium on Software Testing and Analysis(Saarbrücken, Germany) (ISSTA 2016). Association for Computing...

2016
[15]

Google. 2024. CodeGemma-7B Model on Hugging Face. https://huggingface.co/google/codegemma-7b

2024
[16]

Soneya Binta Hossain. 2024. Ensuring Critical Properties of Test Oracles for Effective Bug Detection. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (Lisbon, Portugal) (ICSE-Companion ’24). Association for Compu...

2024
[17]

Soneya Binta Hossain and Matthew Dwyer. 2024. TOGLL: Correct and Strong Test Oracle Generation with LLMs. arXiv preprint arXiv:2405.03786 (2024)

2024 arXiv
[18]

Dwyer, Sebastian Elbaum, and Willem Visser

Soneya Binta Hossain, Antonio Filieri, Matthew B. Dwyer, Sebastian Elbaum, and Willem Visser. 2023. Neural-Based Test Oracle Generation: A Large-Scale Evaluation and Lessons Learned. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on...

2023 doi
[19]

Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A Deep Dive into Large Language Models for Automated Bug Localization and Repair. Proc. ACM Softw. Eng. 1, FSE, Article 66 (jul 2024), 23 pages. https://do...

2024 doi
[20]

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang
[21]

Xing Hu, Xin Xia, David Lo, Zhiyuan Wan, Qiuyuan Chen, and Thomas Zimmermann. 2022. Practitioners’ expectations on automated code comment generation. In Proceedings of the 44th International Conference on Software Engineering . 1693–1705

2022
[22]

Hugging Face. 2024. Hugging Face: The AI community building the future. https://huggingface.co/. Accessed: 2024-05-1

2024
[23]

René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis . 437–440. https://doi.org/10.1145/261038...

2014
[24]

Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. A neural model for generating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 795–806

2019
[25]

Caroline Lemieux and Koushik Sen. 2018. Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage. In Proceedings of the 33rd ACM/IEEE international conference on automated software engineering . 475–485

2018
[26]

Zhongxin Liu, Kui Liu, Xin Xia, and Xiaohu Yang. 2023. Towards more realistic evaluation for neural test oracle generation. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 589–600

2023
[27]

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. arXiv preprint arXiv:2102.04664 (2021)

2021 arXiv
[28]

Paul W McBurney and Collin McMillan. 2015. Automatic source code summarization of context for java methods. IEEE Transactions on Software Engineering 42, 2 (2015), 103–119

2015
[29]

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. arXiv:2203.13474 [cs.LG]

2023 arXiv
[30]

Oracle. [n. d.]. https://www.oracle.com/technical-resources/articles/java/javadoc-tool.html#format
[31]

Carlos Pacheco and Michael D Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion . 815–816

2007
[32]

Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring method specifications from natural language API descriptions. In 2012 34th International Conference on Software Engineering (ICSE). 815–825. https://doi.org/10.1109/ICSE.2012.6227137

2012
[33]

Phi-1. [n. d.]. https://huggingface.co/microsoft/phi-1
[34]

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . ...

2020
[35]

Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing automatic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . ...

2021
[36]

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al . 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

2023 arXiv
[37]

Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering (2023)

2023
[38]

Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Car- valho Lopes. 2023. Exploring the effectiveness of large language models in generating unit tests. arXiv preprint arXiv:2305.00418 (2023). , Vol. 1, No. 1, Article . Pu...

2023 arXiv
[39]

Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. 260–269. https://doi.org/10.1109/ICST.2012.106

2012 doi
[40]

Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020)

2020 arXiv
[41]

Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, and Neel Sundaresan. 2022. Generating accurate assert statements for unit test cases using pretrained transformers. In Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test. 54–64. https://doi.o...

2022
[42]

Chao Wang, Hao He, Uma Pal, Darko Marinov, and Minghui Zhou. 2023. Suboptimal comments in java projects: From independent comment changes to commenting practices. ACM Transactions on Software Engineering and Methodology 32, 2 (2023), 1–33

2023
[43]

Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1398–1409. https://doi.org/10.1145/3377811.3380429

2020
[44]

Xu, Uri Alon, Graham Neubig, and Vincent J

Frank F. Xu, Uri Alon, Graham Neubig, and Vincent J. Hellendoorn. 2022. A Systematic Evaluation of Large Language Models of Code. arXiv:2202.13169 [cs.PL]

2022 arXiv
[45]

Michal Zalewski. 2015. American Fuzzy Lop (AFL). http://lcamtuf.coredump.cx/afl/ Accessed: 29 April, 2024. , Vol. 1, No. 1, Article . Publication date: March 2025

2015
[2023]

arXiv preprint arXiv:2308.10620 (2023)

Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023)

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shep- herd. 2020. Software documentation: the practitioners’ perspective. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 590–601

work page 2020

[2] [2]

Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

work page 2019

[3] [3]

Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41, 5 (2014), 507–525

2014

[4] [4]

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. InProceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623

2021

[5] [5]

Ernst, Mauro Pezzè, and Sergio Del- gado Castellanos

Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Sergio Del- gado Castellanos. 2018. Translating Code Comments to Procedure Specifications. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (Amsterdam, Netherlands) (ISSTA 2018). Association for Computing ...

arXiv 2018

[6] [6]

Arianna Blasi, Alessandra Gorla, Michael D Ernst, Mauro Pezzè, and Antonio Carzaniga. 2021. MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation. Journal of Systems and Software 181 (2021), 111041

work page 2021

[7] [7]

Marcel Bohme, Van-Thuan Pham, and Abhik Roychoudhury. 2019. Coverage-Based Greybox Fuzzing as Markov Chain. IEEE Transactions on Software Engineering 45, 05 (2019), 489–506

work page 2019

[8] [8]

CodeLlama. 2024. CodeLlama-7B-Instruct Model on Hugging Face. https://huggingface.co/codellama/CodeLlama-7b- Instruct-hf

work page 2024