Pith. sign in

REVIEW 3 major objections 5 minor 48 references

A Tool for Generating Exceptional Behavior Tests With Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read exLong fine-tunes the CodeLlama code model on prompts built from stack traces, guard expressions, and existing non-exceptional tests, and claims that this context-conditioned generation produces runnable Java exception tests 67.36% of the…

desk verdict Useful tool paper, but the machine-oriented Venn diagram contradicts the text as printed; needs an artifact check before the claim can be trusted. read the letter →

arxiv 2505.22818 v1 pith:AY26CS4B submitted 2025-05-28 cs.SE cs.AI

Linghan Zhong , Samuel Yuan , Jiyang Zhang , Yu Liu , Pengyu Nie , Junyi Jessy Li , Milos Gligoric This is my paper
classification cs.SEcs.AI
keywords exceptionalbehaviortestsautomatictestgenerationlargelanguagemodelsfine-tuningCodeLlamaJavastacktracesguardexpressions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

exLong is a command-line tool that automatically writes exceptional behavior tests for Java code: tests that execute a specific throw statement and assert the expected exception. The tool's claim is that a large language model fine-tuned on carefully selected context can do this better than general foundation models and existing test generators. Given a method and a target throw statement, exLong supplies the model with the stack trace that reaches the throw, the guard expression that controls it, and existing non-exceptional tests of similar traces, then asks the model to write the test. In an evaluation on 434 developer-oriented examples, the tool's generated tests were runnable 67.36% of the time and covered the target throw 59.45% of the time, beating GPT-3.5 by 9.9% and CAT-LM by 83.8% on runnable tests; in a repository-wide use case it also covered more target throw statements than Randoop or EvoSuite. The paper matters because exceptional paths are exactly the tests developers tend to skip, and the result suggests those tests can be generated automatically rather than left to manual effort.

What carries the argument

The load-bearing object is the constructed prompt, which packages three pieces of evidence about the target throw statement: a stack trace collected by instrumenting and executing existing non-exceptional tests that reach the method containing the throw; a guard expression, built by collecting AST nodes for conditionals and assignments along the trace and substituting symbolic variables, giving a conjunction of conditions that must hold for the throw to execute; and relevant non-exceptional tests from the same repository, which show the model how to set up the object under test and what coding conventions to follow. The fine-tuned LLM is trained to take this prompt and emit a full exceptional behavior test; at inference time exLong can sample multiple candidates and keep the one that compiles, runs, and covers the target throw.

What would settle it

Re-run the developer-oriented evaluation on the same 434 examples with the test-name field removed from the prompt; if Runnable% and ThrowCov% fall to near the baselines, the reported advantage depends on the name hint rather than on the trace-and-guard context. Conversely, run exLong end-to-end on repositories where stack-trace extraction fails for a substantial share of throw statements and compare ThrowCov% against the filtered benchmark; if coverage collapses, the reported rates overstate the tool's performance in ordinary use.

Watch

Extended reading notes

Core claim

The central discovery is that the context surrounding a throw statement—how execution reaches it, what condition guards it, and how existing happy-path tests set up similar calls—is enough for a fine-tuned LLM to write a usable exceptional behavior test. exLong builds each prompt from three extracted pieces: a stack trace from an existing non-exceptional test that reaches the method containing the target throw; a guard expression, the conjunction of conditional expressions and assignments along that trace that must hold for the throw to fire; and one or more relevant non-exceptional tests that exercise similar paths. A CodeLlama model is instruction-fine-tuned on prompts built this way. In the developer-oriented evaluation, exLong generated 82.10% compilable, 67.36% executable, and 59.45% target-covering tests, while GPT-3.5 reached 61.29% executable and 48.39% target-covering and CAT-LM reached 36.64% executable and 30.03% target-covering. In the machine-oriented case, across 30 projects, exLong covered more target throw statements than both Randoop and EvoSuite, with the three tools covering partly disjoint sets. A quantized variant cuts memory use by 75% with 13.1% fewer executable tests.

Load-bearing premise

The load-bearing assumption is that the stack traces and guard expressions needed for the prompt can be reliably extracted for real targets, and that the examples where extraction fails are not systematically harder; it also assumes that including the target test's name in the prompt, as the developer-oriented evaluation does, is a fair representation of how developers will use the tool.

Editorial extensions

If this is right

  • Developers can ask for a test of a specific throw statement and get a compilable candidate about 82% of the time, a runnable one about 67%, and one that actually triggers the target throw about 59%, in the paper's evaluation.
  • In the developer-oriented comparison, exception-specific fine-tuning with trace and guard context beats both a general foundation model and a specialized test-generation model on every reported metric.
  • In the repository-wide use case, exLong covers more distinct throw statements per project than random-based Randoop or search-based EvoSuite, and the three tools' coverage overlaps only partially.
  • The quantized model option makes the tool usable on a single local GPU, with about 13% fewer executable tests than the full-precision model.
  • Generated tests are specific to Java and Maven; the same context-conditioning recipe would need adaptation for other languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Table 1 includes the ground-truth test method name in the prompt; if that name is a strong hint about the expected exception, absolute rates in unconstrained developer use may be lower, though the relative ordering of models could still hold.
  • The developer-oriented benchmark only includes examples where stack-trace extraction succeeded; projects where extraction fails may be exactly the ones where the generated context is weakest, so end-to-end rates on unfiltered repositories could differ.
  • The same context trio of trace, guard, and analogous passing test could plausibly condition models to generate tests for other hard-to-reach branches, not only throws; that extension is not explored in this paper.
  • The quantization results suggest that memory-constrained settings trade a small drop in executable tests for a large drop in compute cost, which may make this approach practical for routine local development.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents exLong, a command-line tool built on a CodeLlama model fine-tuned to generate Java unit tests that exercise specified throw statements. The tool augments prompts with stack traces leading to the target throw, guard expressions, and existing non-exceptional tests; it supports a developer-oriented mode (user selects MUT, throw line, and destination test file) and a machine-oriented mode (whole-repository coverage). The evaluation reports that exLong outperforms GPT-3.5, CAT-LM, Randoop, and EvoSuite on metrics for executable and target-covering exceptional behavior tests, and the paper describes installation and usage via Maven, Docker, and Ollama. The manuscript is a companion tool paper that defers full training and evaluation details to the authors' prior paper [44].

Significance. If the reported results are accurate, exLong is a useful contribution: it targets an under-supported task, makes the workflow practical through a CLI and a quantization option, and combines structured program context (stack traces, guard expressions, and non-EBTs) with an LLM in a way that prior test-generation tools do not. The paper ships an open-source implementation, a demonstration video, and Docker-based setup instructions, which are concrete strengths for a tool paper. However, the evaluation as currently presented does not fully establish the headline 'outperforms' claims for the default tool configuration, because of the ground-truth test-name conditioning in Table 1 and the ambiguous, potentially contradictory Venn diagram in Figure 5. The paper is also explicit about filtering to examples with extractable traces, but it does not quantify how that filtering affects the external validity of the results.

major comments (3)
  1. [Section 5, Table 1 and Section 4.1] Table 1 is captioned 'Results on developer-oriented use case with ground-truth EBT's name in the prompt,' but the developer-oriented CLI described in Section 4.1 has --test_name with a default of none. The reported Runnable% and ThrowCov% therefore measure a partially label-conditioned generation task, not the tool's default invocation. The test name in this domain is often semantically informative (e.g., 'reject_scheduling_a_job_with_same_name_but_different_runnable' in Figure 1b), so providing it can leak the exceptional behavior to be tested. The abstract's claim that exLong automatically generates EBTs and the 83.8% and 9.9% improvements are not established for the unconstrained setting; the authors should report results without the test-name hint or explicitly scope the claims to the conditioned setting.
  2. [Section 5, Figure 5] The machine-oriented conclusion that 'exLong covers the most target throw statements' and 'outperforms both Randoop and EvoSuite' is not verifiable from Figure 5. The figure lists seven region counts (61, 38, 44, 17, 46, 18, 36) with labels Randoop, EvoSuite, and EXLóNG but provides no region legend in the caption. Under the conventional reading of those seven regions (Randoop-only, Randoop∩EvoSuite, EvoSuite-only, EvoSuite∩exLong, exLong-only, Randoop∩exLong, all three), the totals are Randoop=153, EvoSuite=135, and exLong=117, which contradicts the text; under any other mapping, the reader cannot check the claim. Please provide an explicit region-to-count mapping or a raw per-project coverage matrix and recompute the aggregate comparison.
  3. [Section 5] The developer-oriented benchmark is restricted to 'a subset of 434 examples from which we are able to extract stack traces,' and the machine-oriented benchmark is run on 649 examples after 'filtering out data for which our heuristic failed to locate the corresponding destination test file.' The paper does not report how many examples were attempted overall, how many were excluded, or whether the excluded examples are systematically harder (e.g., throw statements guarded by complex conditions). Since extraction or search failure is part of the tool's end-to-end behavior, the reported percentages may overstate performance on the full input distribution. Reporting the exclusion rates and an analysis of the failures, or rerunning on an unfiltered sample, is necessary to support the general claims in the abstract.
minor comments (5)
  1. [Section 5, Table 1] The percentages in Table 1 are presented without confidence intervals or per-example counts; because LLM sampling is stochastic, a small table of per-example outcomes or multiple runs would strengthen the comparison.
  2. [Section 4.2] The example command is introduced as 'An example command to invoke developer-oriented use case' but the command shown uses machine_view; this is a typo that should be corrected.
  3. [Section 5] The abstract states that quantization 'results in the generation of 13.1% fewer executable EBTs,' but Section 5 explicitly says the evaluation uses the full-precision LLM and no quantized results are reported in this manuscript; the number is verifiable only through the full paper [44].
  4. [Section 3 and Section 4] The command examples use the module name etestgen while the tool is called exLong throughout; a one-sentence note connecting the module name to the tool would avoid confusion.
  5. [Figure 5] The label 'EXLóNG' in Figure 5 is inconsistent with the tool name exLong used in the text and captions, and the figure would benefit from a colorblind-safe palette.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: exLong's effectiveness claims rest on the paper's own empirical comparisons against external baselines, and the self-citation to the full exLong paper is not load-bearing.

full rationale

This is an empirical tool paper rather than a derivation or first-principles result. exLong's central claims are supported by Table 1 (developer-oriented use case: Compilable%, Runnable%, ThrowCov% against GPT3.5 and CAT-LM) and Figure 5 (machine-oriented use case: Venn diagram of target throw statement coverage against Randoop and EvoSuite), with external baselines and CodeSearchNet-derived Java projects. The paper does defer training and full evaluation details to the authors' prior work [44], and it says 'For more details of our evaluation, refer to the full paper [44]'; however, the key quantitative results are reproduced in this paper itself, so the self-citation is not load-bearing in the sense of substituting for evidence. The ground-truth EBT name provided in the prompt in Table 1 is a benchmark leakage concern, not a circular reduction: it does not make the compiled, runnable, or throw-covering rates equal to the input by construction. The ambiguity about Figure 5's Venn-region mapping is an internal evidence-consistency issue, not a circularity issue. No step in the pipeline—stack trace extraction, guard expression construction, non-EBT selection, prompt assembly, LLM inference, or runtime evaluation—is defined in terms of the evaluation outcome, so no prediction reduces to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the method introduces no equations. The key premises are domain assumptions about test extraction and LLM capability, plus the implicit assumption that the filtered evaluation set is representative. No invented entities are introduced.

assumptions (3)
  • domain assumption Fine-tuned CodeLlama can generate valid EBTs when prompted with stack traces, guard expressions, and non-EBTs.
    The entire method relies on the LLM learning to map this context to a test; Section 2.
  • domain assumption Static and dynamic analysis can extract a useful guard expression and stack trace for any target throw statement.
    Section 2.1; the evaluation filters examples where extraction fails, so this assumption is not universally tested.
  • domain assumption Existing non-EBTs exercising similar traces are available in the project; the tool cannot generate EBTs for code without such tests.
    Section 2.1 preparation phase; relevant non-EBTs come from the repository's existing test suite.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Tool for Generating Exceptional Behavior Tests With Large Language Models." pith.science (2026). https://pith.science/paper/AY26CS4B

@misc{pith2026250522818,
  author       = {Pith},
  title        = {Pith review of: A Tool for Generating Exceptional Behavior Tests With Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AY26CS4B}},
  note         = {Machine review of arXiv:2505.22818}
}
read the original abstract

Exceptional behavior tests (EBTs) are crucial in software development for verifying that code correctly handles unwanted events and throws appropriate exceptions. However, prior research has shown that developers often prioritize testing "happy paths", e.g., paths without unwanted events over exceptional scenarios. We present exLong, a framework that automatically generates EBTs to address this gap. exLong leverages a large language model (LLM) fine-tuned from CodeLlama and incorporates reasoning about exception-throwing traces, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. Our demonstration video illustrates how exLong can effectively assist developers in creating comprehensive EBTs for their project (available at https://youtu.be/Jro8kMgplZk).

Figures

Figures reproduced from arXiv: 2505.22818 by the authors.

Figure 1
Figure 1. Developer-oriented use case example. importance of EBTs and developers’ desire to improve the testing of exceptional behaviors. However, in practice, developers tend to focus on “happy paths” and have limited time to test exceptional behaviors. This results in a lower number of EBTs compared to non-EBTs in most projects. Sadly, tool support for automatically generating EBTs is limited. Most existing analysis-based t… view at source ↗
Figure 2
Figure 2. Machine-oriented use case example. triggers the target throw statement. In figures 1 and 2, we show examples of EBTs generated by exLong. This paper extends exLong by introducing a new command-line interface that simplifies the process of extracting the necessary context for EBTs generation and querying the fine-tuned LLM. We describe two use cases supported by exLong: (1) developer-oriented use case: developers sel… view at source ↗
Figure 3
Figure 3. Overview of exLong. 1 schedule(Scheduler.java:186) 2 ... 3 Job job = prepareJob(name, runnable, when); 4 ... 5 prepareJob(Scheduler.java:340) 6 ... 7 throw new IllegalArgumentException("A job is already scheduled with the name:" + name); 8 ... (a) Stack trace from MUT to target throw statement. 1 findJob(nullableName == null ? runnable.toString() : nullableName).orElse( null) != null && findJob(nullableName == null … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Context for exLong. the test class covering the MUT or the class of the MUT. Finally, ex￾Long constructs the prompt with all the available context. exLong can optionally create multiple prompts from different non-EBTs, generating and evaluating multiple EBTs then selec…
Figure 5
Figure 5. Figure 5: Venn diagram of target throw statements coverage [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 39 canonical work pages

  1. [44]

    Jiyang Zhang, Yu Liu, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2025. exLong: Generating exceptional behavior tests with large language models. In International Conference on Software Engineering. 5

    work page 2025

  2. [1]

    Hussein Almulla and Gregory Gay. 2020. Learning how to search: Generat- ing exception-triggering tests through adaptive fitness function selection. In International Conference on Software Testing, Verification, and Validation. 63–73

    work page 2020

  3. [2]

    Rafael Di Bernardo, Ricardo Sales Jr., Fernando Castor, Roberta Coelho, Nelio Cacho, and Sergio Soares. 2011. Agile testing of exceptional behavior. InBrazilian Symposium on Software Engineering. 204–213

    work page 2011

  4. [3]

    Arianna Blasi, Alessandra Gorla, Michael D Ernst, and Mauro Pezzè. 2022. Call me maybe: Using NLP to automatically generate unit test cases respecting temporal constraints. InAutomated Software Engineering. 1–11

    work page 2022

  5. [4]

    Islem Bouzenia and Michael Pradel. 2023. When to say what: Learning to find condition-message inconsistencies. InInternational Conference on Software Engi- neering. 868–880

    work page 2023

  6. [5]

    Conda. 2024. Conda. https://docs.conda.io/projects/conda/en/stable

    work page 2024

  7. [6]

    2018.Improving readability in automatic unit test generation

    Ermira Daka. 2018.Improving readability in automatic unit test generation. Ph. D. Dissertation. University of Sheffield

    work page 2018

  8. [7]

    Ermira Daka, José Miguel Rojas, and Gordon Fraser. 2017. Generating unit tests with descriptive names or: Would you name your children thing1 and thing2?. InInternational Symposium on Software Testing and Analysis. 57–67

    work page 2017

Show all 48 references
  1. [8]

    Francisco Dalton, Márcio Ribeiro, Gustavo Pinto, Leo Fernandes, Rohit Gheyi, and Baldoino Fonseca. 2020. Is exceptional behavior testing an exception? An empirical assessment using Java automated tests. InInternational Conference on Evaluation and Assessment in Software Engine...

    2020

  2. [9]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2024. LLM.int8(): 8-bit matrix multiplication for transformers at scale. InInternational Conference on Neural Information Processing Systems

    2024

  3. [10]

    Khalid El Haji, Carolin Brandt, and Andy Zaidman. 2024. Using GitHub Copilot for test generation in Python: An empirical study.International Workshop on Automation of Software Test, 45–55

    2024

  4. [11]

    Michael D Ernst, Jeff H Perkins, Philip J Guo, Stephen McCamant, Carlos Pacheco, Matthew S Tschantz, and Chen Xiao. 2007. The Daikon system for dynamic detection of likely invariants.Science of computer programming69, 1-3, 35–45

    2007

  5. [12]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic test suite generation for object-oriented software. InInternational Symposium on the Foundations of Software Engineering. 416–419

    2011

  6. [13]

    Patrice Godefroid. 2012. Test generation using symbolic execution. InAnnual Conference on Foundations of Software Technology and Theoretical Computer Sci- ence

    2012

  7. [14]

    Ernst, and Mauro Pezzè

    Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Auto- matic generation of oracles for exceptional behaviors. InInternational Symposium on Software Testing and Analysis. 213–224

    2016

  8. [15]

    2000.The Java language specification

    James Gosling. 2000.The Java language specification. Addison-Wesley Profes- sional

    2000

  9. [16]

    Mark Harman and Phil McMinn. 2009. A theoretical and empirical study of search-based testing: Local, global, and hybrid search.Transactions on Software Engineering36, 2, 226–247

    2009

  10. [17]

    2003.C# language specifi- cation

    Anders Hejlsberg, Scott Wiltamuth, and Peter Golde. 2003.C# language specifi- cation. Addison-Wesley Longman Publishing Co., Inc

    2003

  11. [18]

    Josie Holmes, Iftekhar Ahmed, Caius Brindescu, Rahul Gopinath, He Zhang, and Alex Groce. 2020. Using relative lines of code to guide automated test generation for Python.Transactions on Software Engineering and Methodology29, 4, 1–38

    2020

  12. [19]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436

    2019 arXiv

  13. [20]

    Lahiri, and Siddhartha Sen

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen

  14. [21]

    Lima, Lincoln S

    Luan P. Lima, Lincoln S. Rocha, Carla I. M. Bezerra, and Matheus Paixao. 2021. Assessing exception handling testing practices in open-source libraries.Empirical Software Engineering26, 5

    2021

  15. [22]

    Yu Liu, Pengyu Nie, Anna Guo, Milos Gligoric, and Owolabi Legunsen. 2023. Extracting Inline Tests from Unit Tests. InInternational Symposium on Software Testing and Analysis. 1–13

    2023

  16. [23]

    Yu Liu, Aditya Thimmaiah, Owolabi Legunsen, and Milos Gligoric. 2024. ExLi: An Inline-Test Generation Tool for Java. InInternational Symposium on Software Testing and Analysis. 1–5

    2024

  17. [24]

    Diego Marcilio and Carlo A. Furia. 2021. How Java programmers test exceptional behavior. InInternational Working Conference on Mining Software Repositories. 207–218

    2021

  18. [25]

    2023.Machine learning for executable code in software testing and verification

    Pengyu Nie. 2023.Machine learning for executable code in software testing and verification. Ph. D. Dissertation. The University of Texas at Austin

    2023

  19. [26]

    Mooney, and Milos Gligoric

    Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. 2023. Learning deep semantics for test completion. InInternational Conference on Software Engineering. 2111–2123

    2023

  20. [27]

    OpenAI. 2024. GPT-3.5-turbo. https://platform.openai.com/docs/models/gpt-3- 5-turbo

    2024

  21. [28]

    Lahiri, Michael D

    Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed random test generation. InInternational Conference on Soft- ware Engineering. 75–84

    2007

  22. [29]

    Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J Hellendoorn. 2022. Test smells 20 years later: Detectability, validity, and reliability.Empirical Software Engineering27, 7, 170

    2022

  23. [30]

    Hellendoorn

    Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn

  24. [31]

    Brian Robinson, Michael D Ernst, Jeff H Perkins, Vinay Augustine, and Nuo Li

  25. [32]

    InAutomated Software Engineering

    CAT-LM: Training language models on aligned code and tests. InAutomated Software Engineering. 409–420

  26. [33]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Mu- rali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware prompting: A study of coverage guided test generation in regression setting using LLM. In International Symposium on the Foundations of Softwa...

    2024

  27. [34]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al

  28. [35]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code Llama: Open foundation models for code.arXiv preprint arXiv:2308.12950

    2023 arXiv

  29. [36]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617

    2020 arXiv

  30. [37]

    2010.The Python language reference

    Guido VanRossum and Fred L Drake. 2010.The Python language reference. Vol. 561. Python Software Foundation Amsterdam, The Netherlands

    2010

  31. [38]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision.Transactions on Software Engineering

    2024

  32. [39]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. Transactions on Software Engineering

    2023

  33. [40]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652

    2021 arXiv

  34. [41]

    Rødseth, tusharhero, Eli Bendersky, Lei Jitang, and Mraiser

    Michael Yang, Jeffrey Morgan, Daniel Hiltgen, Bruce MacDonald, Matt Williams, Patrick Devine, Blake Mizerany, Michael, Jesse Gross, Josh, royjhan, Jeremy, frob, Dane Madsen, Parth Sareen, Eva H, Mark Ward, James Braza, Arne Müller, Hernan Martinez, Ikko Eltociear Ashimine, Rap...

    2025

  35. [42]

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. InConference on Neural Information Processing Systems

    2022

  36. [43]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. InAnnual Meeting of the Association for Computational Linguistics. 13484–13508

    2023

  37. [47]

    Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No more manual tests? Evaluating and improving ChatGPT for unit test generation.arXiv preprint arXiv:2305.04207

    2023 arXiv

  38. [2011]

    InAutomated Software Engineering

    Scaling up automated test generation: Automatically generating main- tainable regression unit tests for programs. InAutomated Software Engineering. 23–32

  39. [2021]

    Multitask prompted training enables zero-shot task generalization.arXiv preprint arXiv:2110.08207

    arXiv

  40. [2023]

    InInternational Conference on Software Engineering

    CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models. InInternational Conference on Software Engineering. 919– 931

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.