Pith. sign in

REVIEW 4 major objections 6 minor 42 references

Quality Assessment of Python Tests Generated by Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most test suites written by LLMs contain at least one error or test smell, and the prompt style changes the mix.

desk verdict Useful descriptive data on LLM-generated Python tests, but the 'most test suites' claim and the RQ4 'correlation' are not supported by the detection-level counts as presented. read the letter →

arxiv 2506.14297 v1 pith:7L4ACJZA submitted 2025-06-17 cs.SE

Victor Alves , Carla Bezerra , Ivan Machado , Larissa Rocha , Tássio Virgínio , Publio Silva This is my paper
classification cs.SE
keywords largelanguagemodelstestgenerationsmellsPythonunittestingpromptengineeringHumanEvalcodequality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models, despite their promise, still produce Python unit test suites that frequently fail or violate structural quality rules. Generating tests for the same programming problems under two prompt styles—Text2Code, where the model sees only a natural-language description, and Code2Code, where it sees the production code—the authors find 151 execution errors and 512 test-smell detections across three LLMs. The most common failure is AssertionError (64.9% of errors), meaning many tests encode wrong expectations about the code; the most common structural flaw is Lack of Cohesion of Test Cases (41.2% of smells), meaning test cases in a suite are poorly related. The paper also finds that prompt context matters but not uniformly: GPT-4o produced the fewest errors in both contexts, while Amazon Q produced the most, especially in Text2Code. If true, this means LLM-generated tests cannot yet be trusted without review, and prompt choice should be tuned per model rather than assumed one-size-fits-all.

What carries the argument

The study's machinery is a two-prompt-context comparison over the HumanEval benchmark, paired with automated error classification and test-smell detection. The two prompt contexts—Text2Code (T2C, description only) and Code2Code (C2C, production code only)—define the independent variable; errors are collected by executing each generated suite and classifying every Python exception using the language's error taxonomy, and test smells are collected with two static detectors, Pynose and TEMPY, whose detections are deduplicated. The central object carrying the analysis is the co-occurrence count: an indicator function $\mathbb{1}(S \cap E)$ that marks each of the 512 smell detections as co-occurring or not with an execution error in the same suite, yielding 265 co-occurrences and the paper's claim that structural flaws and test failures travel together.

What would settle it

Re-run the identical prompt-generation procedure for every (model, prompt context, HumanEval problem) pair multiple times, for example five or ten generations with non-zero sampling temperature, and compute the distribution of errors and smells per pair; if the per-model percentages move by more than a few points across runs, or if any model's ranking flips, the paper's specific rates and comparative conclusions are not stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a quantitative failure profile of current LLM test generation. Across the test suites generated for the HumanEval problems, 151 errors and 512 test smells were recorded. AssertionError dominated the error profile at 64.9%, followed by IndentationError (11.3%) and SyntaxError (4.6%), and the smell profile was dominated by Lack of Cohesion of Test Cases (41.2%) and Assertion Roulette (24.2%). The authors further claim that these two quality dimensions are connected: 265 of the 512 smell detections appeared in suites that also raised errors, with low-cohesion suites strongly associated with AssertionError. They interpret the results as showing that structural problems in generated tests have a direct impact on the reliability of automated tests, and that the choice between code-based and text-based prompts changes quality outcomes differently for each model.

Load-bearing premise

Each generation condition was run only once, so a single produced test suite is treated as standing in for everything that LLM would produce for that problem and prompt; if the models are as variable as earlier work suggests, the exact error and smell counts and the model rankings could shift with a different sample.

Editorial extensions

If this is right

  • Teams that adopt LLM-generated unit tests should expect a substantial share of suites to need manual correction, because assertion errors alone make up 64.9% of all recorded failures.
  • Prompt style is not neutral: the same model can produce few errors but many smells from text prompts, or the reverse from code prompts, so prompt engineering must be tuned per model.
  • Static test-smell detection can act as a warning system, since 265 of 512 smell detections occurred in suites that also failed at runtime.
  • Among the three models, GPT-4o is the most stable low-error choice in both prompt contexts, while Amazon Q's text-based output is the least reliable, with indentation and name errors unique to it.
  • The link between structural smells and execution errors argues for quality gates that combine both checks before LLM-generated tests are accepted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that single-sample generation is too thin to characterize an LLM's output distribution; future comparisons should sample multiple generations per pair and report spread.
  • The strong pairing of AssertionError with Lack of Cohesion of Test Cases suggests that cheap static smell checks could flag suites worth human review, though the observed correlation does not by itself prove that low cohesion causes assertion failures.
  • Because the HumanEval problems may already sit in LLM training data, the measured error and smell rates are plausibly optimistic; repeating the procedure on unseen or rotated tasks would test whether the quality profile holds outside the training distribution.
  • The model-by-context divergence points toward a practical 'context router' that picks T2C or C2C per model—for example, text prompts for LLama and code prompts for Amazon Q—rather than a single prompt style for all models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports an empirical study of Python unit tests generated by three LLMs (GPT-4o, Amazon Q Developer, and Llama 3.3) from 164 HumanEval problems under two prompt contexts, Text2Code and Code2Code. The authors count execution errors reported by the Python interpreter (151 total) and test smells detected by Pynose and TEMPY (512 total), then report distributions by model and prompt context, and finally analyze co-occurrences between errors and smells. The paper's central claims are that most generated test suites contain at least one error or smell, that prompt context significantly influences test quality, and that there is a strong relationship between specific errors and test smells.

Significance. If fully supported, this would be a useful descriptive benchmark for LLM-generated Python test code, complementing prior work on production code and extending the authors' earlier Copilot study to three models and two prompt contexts. The study has concrete strengths: the artifact is publicly available, the tools and datasets are external references (HumanEval, Python interpreter errors, Pynose, TEMPY), and the procedure is described in enough detail to be replicated. However, the headline suite-level claim and the causal/inferential language ('significantly', 'correlation', 'strong relationship') go beyond what the detection-level descriptive counts can establish. With suite-level aggregation and proper statistical treatment, the contribution would be credible; in its current form, the central claims are overstated.

major comments (4)
  1. [Abstract; Sections 5.1 and 5.2; Sections 4.2.4 and 4.2.5] The abstract's claim that 'most test suites generated by the LLMs contained at least one error or test smell' is not supported by the reported data. All reported metrics are detection-level counts: 151 error occurrences and 512 smell detections, with multiple errors or smells per suite counted separately. The paper never reports the number of generated test suites, the number of suites with at least one error, the number with at least one smell, or the union of those two sets. Without a suite-level denominator and a suite-level aggregation, the 512 smell detections and 151 error detections could be concentrated in a relatively small number of poorly structured suites, so the 'most suites' conclusion does not follow.
  2. [Section 4.2.6; Section 5.4; Table 4; Table 3] RQ4 asks about 'correlation' but the analysis is only a co-occurrence count: 265 of 512 smell detections occurred in suites that also had an error. No statistical test, no expected co-occurrence under independence, and no confidence interval are provided, so the words 'correlation' and 'strong relationship' are not justified. In addition, the counts in Section 5.4 are internally inconsistent with Tables 3 and 4 unless they are intended to be co-occurrence subsamples, which is not stated. Section 5.4 reports 131 occurrences of Lack of Cohesion of Test Cases and 40 occurrences of Assertion Roulette, whereas Table 4 implies totals of about 211 and 124 for those smells; it reports 'AssertionError, which accounted for 138 occurrences,' whereas Table 3 implies about 98 total AssertionErrors. These numbers need to be reconciled and the co-occurrence analysis needs a proper statistical formulation.
  3. [Section 4.2.3; Section 7 (Internal Validity)] Each (LLM, prompt context, HumanEval problem) pair was sampled exactly once, and the paper treats a single generated test suite as representative of the model's output distribution. There is no repeated sampling with different seeds, no variance estimate, and no confidence interval around any of the reported percentages or rankings. The model rankings (e.g., GPT-4o fewest errors, Amazon Q highest error rate, Llama best in T2C for smells) are therefore point estimates from one draw each and could change substantially under stochastic resampling. The acknowledgment in Section 7 that LLM variability can influence reproducibility does not substitute for an analysis that quantifies this variability.
  4. [Section 5.3; Abstract (RQ3 claim)] The abstract says that 'prompt context significantly influenced test quality,' and RQ3 is answered with statements about context effects, but no significance test is performed anywhere in Section 5.3. The observations are purely descriptive comparisons of counts (e.g., GPT-4o T2C smell detections doubling, Amazon Q T2C errors rising to 43). Without a statistical test, such as a chi-square or Fisher's exact test on suite-level counts, the word 'significantly' is unsupported. If the claim is intended to be qualitative, the wording should be changed; if it is intended to be quantitative, the appropriate test should be added.
minor comments (6)
  1. [Table 2; Table 4; Section 5.2] Table 2 defines a 'Magic Number Test' smell, but no Magic Number Test column appears in Table 4 or in the results text; conversely, Table 4 and Section 5.2 report an 'Exception Handling' smell that is not defined in Table 2. The lists need to be aligned.
  2. [Table 2] The smell named 'Unknow Test' in Table 2 is referred to as 'Unknown Test' everywhere else; the table entry should be corrected.
  3. [Section 5.2; Table 2] The description of Programming Paradigms Blend in Table 2 ('Mixing paradigms in the same test file') is not the same as the explanation in Section 5.2 ('fields initialized outside the test class'); the intended definition should be clarified.
  4. [Section 4.2.6; Equations 5 and 6] The notation in Equations 5 and 6 is confusing: k is introduced as the total number of smell detections, but then used as the number of cases in the co-occurrence sum, and the indicator function 1(S∩E) is not formally defined in terms of the measured quantities. The equations should be rewritten to match the actual counting procedure.
  5. [Tables 3 and 4] The tables use decimal commas (e.g., 0,112) while the text uses decimal points; this should be made consistent with the venue's style guidelines.
  6. [Section 4.2.3] The model row in Table 3 and the text refer to 'GitHub Copilot, version GPT-4o' in Section 4.2.3, while the abstract and title list 'GPT-4o' as the model; the naming should be consistent so readers know whether the tool or the underlying model is being evaluated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the study is an external empirical benchmark whose counts come from generated outputs, not from its own assumptions.

full rationale

This paper contains no derivation chain that reduces to its inputs. The pipeline is empirical: HumanEval problems are used as the prompt source, three LLMs generate test suites, the Python interpreter reports execution errors, and Pynose/TEMPY report test smells. The reported quantities (151 error detections, 512 smell detections, distributions, prompt-context comparisons, co-occurrences) are measurements of artifacts produced by external systems, not values fitted from the research questions or reconstructed from the paper's own equations. Equations (1)-(6) merely sum detections and define co-occurrence; they do not encode the results. The only self-citation, Alves et al. [3], is used as motivation and as a contrast point ('this analysis was limited to one LLM and one prompt'), not as evidence for any current quantitative claim. The acknowledged threats to validity (single sample per condition, tool limitations, Python-only scope) are limitations on generalization, not signs that the conclusions are forced by definition. The abstract's suite-level statement that 'most test suites contained at least one error or test smell' is not directly supported by the detection-level tables, since no suite-level denominator or at-least-one aggregation is reported; however, that is an evidentiary gap in the strength of a claim, not a circularity. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions: the validity of the smell-detection tools, the representativeness of a single LLM sample, the representativeness of HumanEval, and the adequacy of Python interpreter errors for categorization. No free parameters were fitted, and no new entities were invented.

assumptions (4)
  • domain assumption Pynose and TEMPY correctly identify test smells in LLM-generated Python test code.
    The study uses these tools as ground truth for smell measurement (Section 4.2.5, Table 2). If the tools are incomplete or noisy, smell counts are biased.
  • domain assumption Each single generated test suite per (model, context, problem) is representative of that model's output distribution.
    No repeated sampling is reported; LLM outputs are stochastic. Acknowledged as a threat to internal validity in Section 7.
  • domain assumption HumanEval problems and their production codes are representative of Python test generation tasks.
    Used as the sole dataset; the authors acknowledge limited external validity in Section 7.
  • domain assumption The Python interpreter's error messages are sufficient to categorize test failures into the eight error types.
    Construction of Table 1 from official Python docs; authors note interpretation differences in Section 7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quality Assessment of Python Tests Generated by Large Language Models." pith.science (2026). https://pith.science/paper/7L4ACJZA

@misc{pith2026250614297,
  author       = {Pith},
  title        = {Pith review of: Quality Assessment of Python Tests Generated by Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7L4ACJZA}},
  note         = {Machine review of arXiv:2506.14297}
}
read the original abstract

The manual generation of test scripts is a time-intensive, costly, and error-prone process, indicating the value of automated solutions. Large Language Models (LLMs) have shown great promise in this domain, leveraging their extensive knowledge to produce test code more efficiently. This study investigates the quality of Python test code generated by three LLMs: GPT-4o, Amazon Q, and LLama 3.3. We evaluate the structural reliability of test suites generated under two distinct prompt contexts: Text2Code (T2C) and Code2Code (C2C). Our analysis includes the identification of errors and test smells, with a focus on correlating these issues to inadequate design patterns. Our findings reveal that most test suites generated by the LLMs contained at least one error or test smell. Assertion errors were the most common, comprising 64% of all identified errors, while the test smell Lack of Cohesion of Test Cases was the most frequently detected (41%). Prompt context significantly influenced test quality; textual prompts with detailed instructions often yielded tests with fewer errors but a higher incidence of test smells. Among the evaluated LLMs, GPT-4o produced the fewest errors in both contexts (10% in C2C and 6% in T2C), whereas Amazon Q had the highest error rates (19% in C2C and 28% in T2C). For test smells, Amazon Q had fewer detections in the C2C context (9%), while LLama 3.3 performed best in the T2C context (10%). Additionally, we observed a strong relationship between specific errors, such as assertion or indentation issues, and test case cohesion smells. These findings demonstrate opportunities for improving the quality of test generation by LLMs and highlight the need for future research to explore optimized generation scenarios and better prompt engineering strategies.

Figures

Figures reproduced from arXiv: 2506.14297 by the authors.

Figure 1
Figure 1. Methodological procedures. def separate_paren_groups ( paren_string : str ) -> List [ str ]: result = [] current_string = [] current_depth = 0 for c in paren_string : if c == '(': current_depth += 1 current_string . append ( c ) elif c == ') ': current_depth -= 1 current_string . append ( c ) if current_depth == 0: result . append (' '. join ( current_string ) ) current_string . clear () return result ✝ ✆ Listing 2:… view at source ↗
Figure 2
Figure 2. Quantity of errors and test smells in the C2C context [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Quantity of errors and test smells in the T2C context [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Cooccurrences of Errors and Test Smells by type [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 19 canonical work pages

  1. [1]

    Mayank Agarwal, Yikang Shen, Bailin Wang, Yoon Kim, and Jie Chen. 2024. Struc- tured Code Representations Enable Data-Efficient Adaptation of Code Language Models. arXiv:2401.10716 [cs.CL] https://arxiv.org/abs/2401.10716

    work page Pith review arXiv 2024

  2. [2]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang

  3. [3]

    Victor Alves, Cristiano Santos, Carla Bezerra, and Ivan Machado. 2024. Detecting Test Smells in Python Test Code Generated by LLM: An Empirical Study with GitHub Copilot. InAnais do XXXVIII Simpósio Brasileiro de Engenharia de Software (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 581–587. https://doi.org/10.5753/ sbes.2024.3561

    work page arXiv 2024

  4. [5]

    Alexandru Bodea. 2022. Pytest-Smell: a smell detection tool for Python unit tests. InProceedings of the 31st ACM SIGSOFT International Symposium on Soft- ware Testing and Analysis(Virtual, South Korea)(ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 793–796. https://doi.org/10.1145/ 3533767.3543290

    arXiv 2022

  5. [6]

    W. Cain. 2024. InPrompting Change: Exploring Prompt Engineering in Large Language Model AI and Its Potential to Transform Education, Vol. 68. TechTrends, 47–57. https://doi.org/10.1007/s11528-023-00896-0

    work page doi:10.1007/s11528-023-00896-0 2024

  6. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar...

    arXiv 2021

  7. [8]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jian- wei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. InCompanion Proceedings of the 32nd ACM International Conference on the Quality Assessment of Python Tests Generated by Large Language Models Foundations of Software Engineering(Porto de Galinhas, Brazil)(FSE 2024...

    arXiv 2024

  8. [10]

    Ermira Daka and Gordon Fraser. 2014. InA Survey on Unit Testing Practices and Problems. 2014 IEEE 25th International Symposium on Software Reliability Engineering, 201–211. https://doi.org/10.1109/ISSRE.2014.11

    doi:10.1109/issre.2014.11 2014

Show all 42 references
  1. [11]

    Humberto Damasceno, Carla Bezerra, Denivan Campos, Ivan Machado, and Emanuel Coutinho. 2023. Test smell refactoring revisited: What can inter- nal quality attributes and developers’ experience tell us?Journal of Soft- ware Engineering Research and Development11, 1 (Oct. 2023),...

    2023

  2. [12]

    Khalid El Haji, Carolin Brandt, and Andy Zaidman. 2024. InUsing GitHub Copilot for Test Generation in Python: An Empirical Study(Lisbon, Portugal)(AST ’24). ACM, New York, NY, USA, 11. https://doi.org/10.1145/3644032.3644443

    2024

  3. [13]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shu- vendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation.IEEE Transactions on Software Engineering50, 9 (2024), 2254–2268. https://doi.org/10.1109/T...

    2024

  4. [14]

    Daniel Fernandes, Ivan Machado, and Rita Maciel. 2022. TEMPY: Test Smell De- tector for Python. InProceedings of the XXXVI Brazilian Symposium on Software Engineering(Virtual Event, Brazil)(SBES ’22). Association for Computing Ma- chinery, New York, NY, USA, 214–219. https://d...

    2022

  5. [15]

    Santos, Andrew Popovich, Mehdi Mirakhorli, and Mei Nagappan

    Danielle Gonzalez, Joanna C.S. Santos, Andrew Popovich, Mehdi Mirakhorli, and Mei Nagappan. 2017. A Large-Scale Study on the Usage of Testing Patterns That Address Maintainability Attributes: Patterns for Ease of Modification, Diagnoses, and Comprehension. In2017 IEEE/ACM 14th...

    2017 doi

  6. [16]

    Graham, R

    D. Graham, R. Black, and E. van Veenendaal. 2021.Foundations of Software Testing ISTQB Certification, 4th edition. Cengage Learning. https://books.google.com. br/books?id=mOwxEAAAQBAJ

    2021

  7. [17]

    2023.Code Correctness and Quality in the Era of AI Code Generation: Examining ChatGPT and GitHub Copilot

    Emilia Hansson and Oliwer Ellréus. 2023.Code Correctness and Quality in the Era of AI Code Generation: Examining ChatGPT and GitHub Copilot. Ph. D. Dissertation. https://urn.kb.se/resolve?urn=urn:nbn:se:lnu:diva-121545

    2023

  8. [18]

    Khorikov

    V. Khorikov. 2020.Unit Testing Principles, Practices, and Patterns: Effective testing styles, patterns, and reliable automation for unit testing, mocking, and integration testing with examples in C#. Manning. https://books.google.com.br/books?id= CbvZyAEACAAJ

    2020

  9. [19]

    Dong Jae Kim. 2020. An Empirical Study on the Evolution of Test Smell. In2020 IEEE/ACM 42nd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 149–151

    2020

  10. [20]

    Chun Li. 2022. InMobile GUI test script generation from natural language descriptions using pre-trained model(Pittsburgh, Pennsylvania)(MOBILESoft ’22). Association for Computing Machinery, New York, NY, USA, 112–113. https://doi.org/10.1145/3524613.3527809

    2022

  11. [21]

    Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan. 2024. Guiding ChatGPT for Better Code Generation: An Empirical Study. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 102–113. https://doi.o...

    2024

  12. [22]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021 arXiv

  13. [23]

    Maragathavalli

    P. Maragathavalli. 2011. Search-based software test data generation using evolu- tionary computation.ArXivabs/1103.0125 (2011). https://api.semanticscholar. org/CorpusID:25209645

    2011 arXiv

  14. [24]

    Marvin, N

    G. Marvin, N. Hellen, D. Jjingo, and J. Nakatumba-Nabende. 2024. Engen- haria de Prompt em Grandes Modelos de Linguagem. InInteligência de Da- dos e Informática Cognitiva. ICDICI 2023 (Algoritmos para Sistemas Inteligentes), IJ Jacob, S. Piramuthu, and P. Falkowski-Gilski (Eds...

    2024 doi

  15. [26]

    Phil McMinn. 2004. Search-based software test data generation: a survey: Re- search Articles.Softw. Test. Verif. Reliab.14, 2 (jun 2004), 105–156

    2004

  16. [27]

    Jhonatan Oliveira, Luigi Mateus, Tássio Virgínio, and Larissa Rocha. 2024. SNUTS.js: Sniffing Nasty Unit Test Smells in Javascript. InAnais do XXXVIII Simpósio Brasileiro de Engenharia de Software(Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 720–726. https://doi.org/10.5753/sb...

    2024

  17. [28]

    Fabio Palomba, Andy Zaidman, and Andrea De Lucia. 2018. Automatic Test Smell Detection Using Information Retrieval Techniques. In2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). 311–322. https: //doi.org/10.1109/ICSME.2018.00040

    2018

  18. [29]

    Zedong Peng, Xuanyi Lin, Michelle Simon, and Nan Niu. 2021. Unit and regres- sion tests of scientific software: A study on SWMM.Journal of Computational Science53 (2021), 101347. https://doi.org/10.1016/j.jocs.2021.101347

    2021

  19. [30]

    Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen

    Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. InProceedings of the 2023 ...

    2023

  20. [31]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Mu- rali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompt- ing: A Study of Coverage-Guided Test Generation in Regression Setting us- ing LLM.Proc. ACM Softw. Eng.1, FSE, Article 43 (July 2024),...

    2024 doi

  21. [32]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv:2402.07927 [cs.AI] https: //arxiv.org/abs/2402.07927

    2024 arXiv

  22. [33]

    Railana Santana, Luana Martins, Larissa Rocha, Tássio Virgínio, Adriana Cruz, Heitor Costa, and Ivan Machado. 2020. RAIDE: a tool for Assertion Roulette and Duplicate Assert identification and refactoring. InProceedings of the XXXIV Brazilian Symposium on Software Engineering. 374–379

    2020

  23. [34]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. InAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation, Vol. 50. IEEE Transactions on Software Engineering, 85–105. https://doi.org/10. 1109/TSE.2023.3334955

    2024

  24. [35]

    Sina Shamshiri, José Miguel Rojas, Juan Pablo Galeotti, Neil Walkinshaw, and Gordon Fraser. 2018. How Do Automatically Generated Unit Tests Influence Software Maintenance?. In2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 250–2...

    2018

  25. [36]

    Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using Large Language Models to Generate JUnit Tests: An Empirical Study. InProceedings of the 28th International Conference on Evalua...

    2024

  26. [37]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit Test Case Generation with Transformers and Focal Context. arXiv:2009.05617 [cs.SE]

    2021 arXiv

  27. [38]

    Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An empirical investi- gation into the nature of test smells. InProceedings of the 31st IEEE/ACM Inter- national Conference on Automated Software E...

    2016

  28. [39]

    Arie van Deursen, Leon Moonen, Alex van den Bergh, and Gerard Kok. 2001. In Refactoring Test Code, M. Marchesi and G. Succi (Eds.). Proceedings 2nd Interna- tional Conference on Extreme Programming and Flexible Processes in Software Engineering (XP2001)

    2001

  29. [40]

    Tongjie Wang, Yaroslav Golubev, Oleg Smirnov, Jiawei Li, Timofey Bryksin, and Iftekhar Ahmed. 2022. InPyNose: a test smell detector for python(Melbourne, Australia)(ASE ’21). IEEE Press, 593–605. https://doi.org/10.1109/ASE51524. 2021.9678615

    2022

  30. [41]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machin...

    2024

  31. [42]

    Tao Xie and David Notkin. 2006. Tool-assisted unit test generation and selection based on operational abstractions.Automated Software Engineering Journal13, 3 (July 2006), 345–371

    2006

  32. [43]

    Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of GitHub copilot’s code generation. InAssessing the quality of GitHub copilot’s code generation(Singapore, Singapore)(PROMISE 2022). Association for Computing Machinery, New York, NY, USA, 62–71. https:...

    2022

  33. [44]

    Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023. InEvaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. https://doi.org/10.48550/ arXiv.2304.10778 Alves et al

    2023 doi arXiv

  34. [45]

    Shengcheng Yu, Chunrong Fang, Yucheng Ling, Chentian Wu, and Zhenyu Chen. 2023. LLM for Test Script Generation and Migration: Challenges, Ca- pabilities, and Opportunities. InLLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities. 2023 IEEE 2...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.