Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells: An Empirical Study

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims Gemini-1.5 Pro is the most accurate LLM for test smell detection and the only one that improves coverage when refactoring.

desk verdict Useful empirical comparison, but the 'accuracy' numbers measure agreement with PyNose/TsDetect rather than ground truth, and the abstract's coverage claim contradicts the paper's own results. read the letter →

arxiv 2506.07594 v1 pith:JHFJR7JE submitted 2025-06-09 cs.SE

E. G. Santana Jr , Jander Pereira Santos Junior , Erlon P. Almeida , Iftekhar Ahmed , Paulo Anselmo da Mota Silveira Neto , Eduardo Santana de Almeida This is my paper
classification cs.SE
keywords testsmellslargelanguagemodelsLLMrefactoringGPT-4GeminiLLaMAPyNoseTsDetect
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can do more than flag test smells—whether they can also refactor the offending test code—and tests three models (GPT-4-Turbo, LLaMA 3 70B, Gemini-1.5 Pro) on Python and Java test suites. The headline finding is that Gemini detects test smells most accurately, with 74.35% accuracy on Python and 80.32% on Java, while LLaMA trails; Gemini also tends to improve statement coverage during refactoring, whereas GPT-4 and LLaMA often reduce it. The study is notable for pairing traditional static detectors with LLM-driven transformation, and for measuring side effects such as newly introduced smells and coverage changes. If the results hold, they make a concrete case for where LLM-based test maintenance tools are ready to help and where they still need guardrails.

What carries the argument

The study's central machinery is a two-stage hybrid workflow: rule-based detectors (PyNose for Python, TsDetect for Java) supply both the labeled identification tasks and the ground-truth oracle, and prompt-engineered LLMs perform detection and refactoring. The argument is carried by the prompt templates—built from persona, few-shot, chain-of-thought, and zero-shot chain-of-thought techniques and iterated to saturation—and by the before/after comparison of test smells and coverage (using pytest-cov and JaCoCo), with statistical significance tested by the Wilcoxon signed-rank test. The 'collateral smell' concept, tracking smells unintentionally added or removed during refactoring, is what turns the evaluation from simple detection accuracy into an assessment of refactoring side effects.

What would settle it

Re-label a sample of the same test cases with human experts (or a second, independent detection tool) and recompute each LLM's accuracy against that gold standard; if agreement with human labels is much lower than the reported 74.35% and 80.32%, the claimed Gemini advantage is an artifact of oracle agreement rather than true detection skill.

Watch

Extended reading notes

Core claim

The paper claims that state-of-the-art LLMs can detect test smells with practically useful accuracy and can perform automated refactoring, but that effectiveness varies sharply by model, language, and smell type. Gemini-1.5 Pro is the strongest overall performer, with the highest detection accuracy (74.35% Python, 80.32% Java) and positive statement-coverage effects in Java, while GPT-4-Turbo is the most consistent for Python and LLaMA 3 70B is the weakest on both tasks. The authors further claim that refactoring frequently removes non-targeted smells as a side benefit, but also introduces new smells and, in many cases, reduces code coverage, so automated refactoring is not yet safe to run unattended.

Load-bearing premise

The study treats PyNose and TsDetect as ground-truth oracles: LLM detection accuracy is the match rate with these tools, and refactoring is deemed successful when the tools stop flagging the smell, so if either tool mislabels code every headline number shifts.

Editorial extensions

If this is right

  • Integrating an LLM like Gemini into an IDE or CI pipeline could catch and fix test smells automatically, but only if the model version is pinned and quality-assured, since the three models differ sharply.
  • Refactoring one smell often removes related smells (e.g., Ignored Test almost always disappears), so teams may get cleanup beyond the explicit request.
  • Coverage can drop after refactoring, especially for Python and for LLaMA; a refactoring step should therefore be followed by a coverage check before acceptance.
  • The prompt engineering results provide a recipe: chain-of-thought with few-shot examples performed best for identification, and the winning refactoring prompts were selected by balancing direct removals against collateral smells.
  • For smells like Magic Number Test and Sleepy Test, models only refactor when explicitly instructed, so generic cleanup prompts will not touch them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reported detection accuracy is agreement with PyNose and TsDetect rather than with a human gold standard, the true detection quality on real projects could be lower; a human-annotated benchmark would settle whether Gemini's lead is real or an artifact of tool bias.
  • The large between-model gaps and per-smell variation imply that 'LLMs for test refactoring' is not a single capability; a model-selection policy may need per-smell routing, since the best model for one smell can be the worst for another.
  • The observed coverage losses might reflect tests being deleted or assertions weakened, not just reordering; running a mutation-testing pass on refactored suites would test whether coverage loss corresponds to real fault-detection loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript reports an empirical study comparing GPT-4-Turbo, LLaMA 3 70B, and Gemini 1.5 Pro on detecting and refactoring test smells in Python and Java. The authors use PyNose and TsDetect to label candidate smells, engineer five prompt templates per language, and report per-smell detection percentages as well as refactoring outcomes such as smell removal, collateral introduction, and coverage changes. The abstract claims Gemini achieved the highest detection accuracy (74.35% Python, 80.32% Java) and that Gemini improved test coverage, while GPT-4 and LLaMA often reduced it. The body of the paper reports Observation 4, which states that coverage impacts were inconsistent and often negative across both languages, with Gemini and LLaMA frequently introducing more uncovered statements than GPT-4.

Significance. The problem is relevant and underexplored, and the study has strengths: it is large in scope (more than 1,400 detection tasks and 450 refactoring tasks over 15 smell types and two languages), it uses a fixed temperature, and it makes artifacts and scripts publicly available. As a comparative analysis of how LLMs agree with established detectors, the study could be a useful empirical data point. However, the headline 'accuracy' numbers are agreement rates with the same tools used for labeling and prompt selection, and the abstract's coverage claim is contradicted by the paper's own Observation 4. The contribution is therefore not yet established as stated; reframing and additional validation are needed.

major comments (4)
  1. [Sections III-A and III-C, Table IV] The reported detection 'accuracy' (abstract: 74.35% Python, 80.32% Java for Gemini) is an agreement rate with PyNose and TsDetect outputs, not accuracy against independent ground truth. The same tools label the sampled data (Section III-A), are the comparison target during prompt selection (Section III-C), and are the reference for the final reported numbers. One of these tools, PyNose [23], is co-authored by a study author, and no human-annotated subset or independent detector is used. Any systematic error in these oracles therefore propagates into every headline number, so the current evidence supports only an agreement ranking. Please either add a manually validated sample to estimate true detection accuracy, or consistently reframe all 'accuracy' claims as 'agreement with PyNose/TsDetect' in the abstract, results, and conclusions.
  2. [Section III-C] The prompt-selection and final-evaluation phases are not separated by a held-out split. The paper states that the LLM outputs were compared against the oracle tools during each iteration of prompt selection, and that after five iterations the best prompt was applied to the 'entire sampled test smell dataset' and compared again. If the final evaluation set overlaps with the set used to select prompts, the reported percentages are optimistically biased and do not measure generalization. Please specify the exact split between prompt-selection and evaluation data, or re-run the evaluation on a disjoint held-out sample.
  3. [Abstract vs. Section IV-B, Observation 4, Figures 4-7] The abstract's claim that 'Gemini also improved test coverage, unlike GPT-4 and LLaMA, which often reduced it' is contradicted by the paper's own results. Observation 4 states that refactoring led to 'inconsistent and often negative impacts on code coverage across both Java and Python projects, with Gemini and LLaMA frequently introducing more uncovered statements than GPT-4,' and Figures 6 and 7 show all models increasing uncovered statements, with Gemini the worst in Python. This internal contradiction changes the paper's headline message; the abstract, Section IV, and Section V must be aligned with Observation 4.
  4. [Section VI] The Threats to Validity section acknowledges reliance on PyNose but frames it mainly as a definitional issue rather than as a threat to the 'accuracy' construct. The manuscript does not quantify oracle misclassification risk, and the same oracle is used to measure refactoring success (Section III-D, Table V). Since PyNose's published precision and recall are for its own detection task, not for the LLM comparison, the construct validity of both RQ1 and RQ2 depends on an unvalidated assumption. Please add an explicit oracle-error analysis (e.g., manual inspection of a random sample) or weaken the conclusions to state that the results reflect agreement with existing tools.
minor comments (6)
  1. [Index Terms] The index terms are placeholder template text ('Article submission, IEEE, IEEEtran, LATEX, paper, template, typesetting'); replace them with actual keywords describing test smells, LLMs, and software testing.
  2. [Section I, footnote 4] The footnote about AI chatbots and history questions is irrelevant to the model-selection rationale; replace it with a software-engineering benchmark or remove it.
  3. [Section IV-A, Table IV] The per-smell percentages are labeled 'accuracy,' but the class balance of the samples is not reported; please clarify whether these are raw agreement rates, balanced accuracy, or another metric, and report sample sizes and confidence intervals.
  4. [Section III-B] The text says Gemini-1.5 Pro is a '130B model,' but the parameter count of Gemini 1.5 Pro is not publicly documented; either cite a reliable source or remove the specific number.
  5. [Table V] The table is said to contain only statistically significant Wilcoxon results, but no p-values, effect sizes, or multiple-comparison corrections are reported; please add them or explicitly describe the table as descriptive.
  6. [Section IV-B, Figure 3] Figure 3, the correlation matrix of test smells, is not explained in the text; please add a paragraph describing what the correlations mean and how they support the refactoring analysis.

Circularity Check

2 steps flagged · score 6.0 of 10

Detection 'accuracy' is agreement with PyNose/TsDetect, the same tools used to label data and select prompts; the Python oracle rests on a load-bearing self-citation.

  1. fitted input called prediction [Section III-C, Test Smell Identification]
    "For each iteration, we compared the LLMs’ outputs against the results produced by PyNose and TsDetect, which served as oracle tools during this prompt selection phase. Once the best-performing prompt for each language was identified, we applied it to the entire sampled test smell dataset, and again compared the outcomes against the oracle tools to assess final performance."

    The reported 'detection accuracy' is computed as agreement with PyNose and TsDetect, the same tools used to label the sampled dataset and to select the best prompt. The prompt was optimized over five iterations to maximize agreement with those tools, and the final accuracy is then measured against the same tools on the same sampled data. Thus the headline accuracy figures are not predictions against independent ground truth; they measure how closely the best prompt can make each LLM reproduce the oracle tools' decisions. A systematic bias in either tool propagates directly into the ranking, and prompt selection on the evaluation sample inflates the agreement. This matches the 'fitted input called prediction' pattern: the selection criterion and the reported metric coincide.

  2. self citation load bearing [Section II-B, Non-LLM Test Smell Detection Tools; reference [23]]
    "PyNose [23] is a specialized tool for Python, adapting 17 language-agnostic test smells and introducing a Python-specific smell, SuboptimalAssert. It has demonstrated high accuracy, with a precision of 94% and recall of 95.8%."

    The Python oracle is justified by citing the PyNose paper [23], whose author list includes Iftekhar Ahmed, also an author of the present paper. The study does not independently validate PyNose's labels on its sampled test files; it takes the tool's outputs as ground truth and measures LLM 'accuracy' against them. The Python half of the central ranking therefore rests on a self-citation chain: the oracle's correctness is asserted from the self-cited paper, and the same oracle defines the measured accuracy. If PyNose's labels are biased, every Python detection figure in Table IV changes. The tool's reported precision and recall come from its own paper rather than from a third-party replication within this study.

full rationale

The central detection claim is an agreement rate with the PyNose and TsDetect oracle tools rather than an accuracy measurement against human-validated ground truth. Section III-A states that these tools served as oracles, and Section III-C confirms that the same tools were used during prompt selection and for the final performance comparison. Because the best prompt was selected by maximizing agreement with the oracle over five iterations and the final accuracy is measured against the same oracle, the reported accuracy figures are not independent predictions; they reflect how well each LLM can be prompted to reproduce the tools' decisions. This is a partial circularity under the 'fitted input called prediction' pattern. Additionally, the Python oracle, PyNose, is justified by a self-cited paper whose authors overlap with the present study, and the paper does not independently validate PyNose's labels on the sampled data; the Python half of the ranking therefore relies on a load-bearing self-citation. The coverage findings are not circular, but they are internally inconsistent with Observation 4 and Figures 6-7, which show Gemini and LLaMA frequently increasing uncovered statements; this is a correctness and consistency issue rather than a circularity issue. Overall, the study is a legitimate comparison of LLM agreement with two available detectors, so the circularity is partial rather than total: the relative ranking among LLMs is empirically measured, not definitionally forced. Score 6 reflects that one central metric reduces to agreement with a tool used for both labeling and prompt selection, plus a load-bearing self-citation for the Python oracle.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The analysis rests on two tool-based assumptions (oracle correctness and tool-based refactoring measurement) plus a coverage proxy. No new entities are introduced; the only fitted quantities are the prompt template choices, which were selected on a validation sample and may leak into the reported results.

free parameters (2)
  • Prompt template version per language = v4 (Python, ZS-CoT + 1-shot) and v2 or v4 (Java) selected via validation
    The best-performing prompt template was selected based on accuracy/F1 against the oracle on a sample (Section III-C), then used for final evaluation. This is a hyperparameter fitted to the data.
  • Token budget limit = Not specified
    The paper limited prompt plus test class token count to avoid response truncation (Section III-B) but does not report the limit. This affects which tests could be refactored.
assumptions (4)
  • domain assumption PyNose and TsDetect detection outputs are correct ground truth for test smell presence or absence.
    Invoked in Sections III-A and III-C; the tools are used as oracles for sampling and for measuring LLM detection accuracy and refactoring success.
  • domain assumption The same detection tools reliably measure refactoring success, meaning a smell is 'removed' when the tool no longer flags it.
    Used throughout Section IV-B; refactoring percentages are computed from tool outputs on before and after code.
  • domain assumption Statement and line coverage is a valid proxy for test suite reliability when evaluating refactoring side effects.
    RQ2.3 uses coverage reports from JaCoCo and PyTest-cov as the outcome measure.
  • domain assumption LLM outputs at temperature 0 are deterministic and comparable across API versions.
    Stated in Section III-B; API model versions may update during the study, but the paper treats them as fixed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells: An Empirical Study." pith.science (2026). https://pith.science/paper/JHFJR7JE

@misc{pith2026250607594,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells: An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JHFJR7JE}},
  note         = {Machine review of arXiv:2506.07594}
}
read the original abstract

Test smells indicate poor development practices in test code, reducing maintainability and reliability. While developers often struggle to prevent or refactor these issues, existing tools focus primarily on detection rather than automated refactoring. Large Language Models (LLMs) have shown strong potential in code understanding and transformation, but their ability to both identify and refactor test smells remains underexplored. We evaluated GPT-4-Turbo, LLaMA 3 70B, and Gemini-1.5 Pro on Python and Java test suites, using PyNose and TsDetect for initial smell detection, followed by LLM-driven refactoring. Gemini achieved the highest detection accuracy (74.35\% Python, 80.32\% Java), while LLaMA was lowest. All models could refactor smells, but effectiveness varied, sometimes introducing new smells. Gemini also improved test coverage, unlike GPT-4 and LLaMA, which often reduced it. These results highlight LLMs' potential for automated test smell refactoring, with Gemini as the strongest performer, though challenges remain across languages and smell types.

Figures

Figures reproduced from arXiv: 2506.07594 by the authors.

Figure 1
Figure 1. Workflow for Test Smell Identification and Refactoring. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of Test Smell Detection Prompt for Java. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Correlation matrix based on refactoring task. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Statement Coverage Comparison in Python Projects [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Statements missing coverage in Java projects. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Statements missing coverage in Python projects. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 29 canonical work pages

  1. [23]

    Pynose: a test smell detector for python,

    T. Wang, Y . Golubev, O. Smirnov, J. Li, T. Bryksin, and I. Ahmed, “Pynose: a test smell detector for python,” in Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’21. IEEE Press, 2022, p. 593–605. [Online]. Available: https://doi.org/10.1109/ASE51524.2021.9678615

    arXiv 2022

  2. [1]

    On the relation of test smells to software code quality,

    D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, and A. Bacchelli, “On the relation of test smells to software code quality,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2018, pp. 1–12

    work page 2018

  3. [2]

    On the diffusion of test smells in automatically generated test code: An empirical study,

    F. Palomba, D. Di Nucci, A. Panichella, R. Oliveto, and A. De Lucia, “On the diffusion of test smells in automatically generated test code: An empirical study,” in Proceedings of the 9th International Workshop on Search-Based Software Testing , ser. SBST ’16. New York, NY , USA: Association for Computing Machinery, 2016, p. 5–14. [Online]. Available: http...

    work page arXiv 2016

  4. [3]

    When and why your code starts to smell bad,

    M. Tufano, F. Palomba, G. Bavota, R. Oliveto, M. Di Penta, A. De Lucia, and D. Poshyvanyk, “When and why your code starts to smell bad,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1, 2015, pp. 403–414

    work page 2015

  5. [4]

    An empirical analysis of the distribution of unit test smells and their impact on software maintenance,

    G. Bavota, A. Qusef, R. Oliveto, A. De Lucia, and D. Binkley, “An empirical analysis of the distribution of unit test smells and their impact on software maintenance,” in 2012 28th IEEE International Conference on Software Maintenance (ICSM) , 2012, pp. 56–65

    work page 2012

  6. [5]

    Just-in-time test smell detection and refactoring: The darts project,

    S. Lambiase, A. Cupito, F. Pecorelli, A. De Lucia, and F. Palomba, “Just-in-time test smell detection and refactoring: The darts project,” in Proceedings of the 28th International Conference on Program Comprehension, ser. ICPC ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 441–445. [Online]. Available: https://doi.org/10.1145/33879...

    arXiv 2020

  7. [6]

    Gpt-4 technical report,

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, and et al., “Gpt-4 technical report,” 2024. [Online]. Available: https: //arxiv.org/abs/2303.08774

    arXiv 2024

  8. [7]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, and et al., “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    arXiv 2024

Show all 53 references
  1. [8]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, and et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024. [Online]. Available: https://arxiv.org/abs/2403.05530

    2024 arXiv

  2. [9]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” 2020

    2020

  3. [10]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code...

    2021

  4. [11]

    Top programming languages - the state of the octoverse 2022,

    GitHub, “Top programming languages - the state of the octoverse 2022,” 2022, accessed: 2024-11-21. [Online]. Available: https:// octoverse.github.com/2022/top-programming-languages

    2022

  5. [12]

    Utilization of pre-trained language model for adapter-based knowledge transfer in software engineering,

    I. Saberi, F. Fard, and F. Chen, “Utilization of pre-trained language model for adapter-based knowledge transfer in software engineering,” 2023

    2023

  6. [13]

    To- wards efficient fine-tuning of pre-trained code models: An experimental study and beyond,

    E. Shi, Y . Wang, H. Zhang, L. Du, S. Han, D. Zhang, and H. Sun, “To- wards efficient fine-tuning of pre-trained code models: An experimental study and beyond,” 2023

    2023

  7. [14]

    An empirical comparison of pre-trained models of source code,

    C. Niu, C. Li, V . Ng, D. Chen, J. Ge, and B. Luo, “An empirical comparison of pre-trained models of source code,” in Proceedings of the 45th International Conference on Software Engineering , ser. ICSE ’23. IEEE Press, 2023, p. 2136–2148. [Online]. Available: https://doi.org/...

    2023

  8. [15]

    (2024) Testsmellsrefactoringbyllms

    LLMs-TestSmells. (2024) Testsmellsrefactoringbyllms. [On- line]. Available: https://github.com/ts-group-icse26/testsmells.llms. study-replication.package-ICSE26.git

    2024

  9. [16]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 8, Dec. 2024. [Online]. Available: https://doi.org/10.11...

    2024 doi

  10. [17]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936, 2024

    2024

  11. [18]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2024

    2024

  12. [19]

    Automated test case repair using language models,

    A. Saboor Yaraghi, D. Holden, N. Kahani, and L. Briand, “Automated test case repair using language models,” IEEE Transactions on Software Engineering, pp. 1–31, 2025

    2025

  13. [20]

    Chatunitest: a chatgpt- based automated unit test generation tool,

    Z. Xie, Y . Chen, C. Zhi, S. Deng, and J. Yin, “Chatunitest: a chatgpt- based automated unit test generation tool,” 2023

    2023

  14. [21]

    An empirical study of using large language models for unit test generation,

    M. L. Siddiq, J. C. S. Santos, R. H. Tanvir, N. Ulfat, F. A. Rifat, and V . C. Lopes, “An empirical study of using large language models for unit test generation,” 2024

    2024

  15. [22]

    Towards an understanding of large language models in software engineering tasks,

    S. Hou, Y . Liu, and J. Lee, “Towards an understanding of large language models in software engineering tasks,” in 42nd International Conference on Software Engineering (ICSE) , 2024, pp. 754–768

    2024

  16. [24]

    Tempy: Test smell detector for python,

    D. Fernandes, I. Machado, and R. Maciel, “Tempy: Test smell detector for python,” in Proceedings of the XXXVI Brazilian Symposium on Software Engineering , ser. SBES ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 214–219. [Online]. Available: https://do...

    2022

  17. [25]

    Handling test smells in python: Results from a mixed-method study,

    ——, “Handling test smells in python: Results from a mixed-method study,” in Proceedings of the XXXV Brazilian Symposium on Software Engineering, ser. SBES ’21. New York, NY , USA: Association for Computing Machinery, 2021, p. 84–89. [Online]. Available: https://doi.org/10.1145...

    2021

  18. [26]

    A trend analysis of test smells in python test code over commit history,

    Y . Fushihara, H. Aman, S. Amasaki, T. Yokogawa, and M. Kawahara, “A trend analysis of test smells in python test code over commit history,” in 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) , 2023, pp. 310–314

    2023

  19. [27]

    Pytest-smell: A smell detection tool for python unit tests,

    A. Bodea, “Pytest-smell: A smell detection tool for python unit tests,” ser. ISSTA 2022. New York, NY , USA: Association for Computing Machinery, 2022, p. 793–796. [Online]. Available: https://doi.org/10.1145/3533767.3543290

    2022

  20. [28]

    A trend analysis of test smells in python test code over commit history,

    Y . Fushihara, H. Aman, S. Amasaki, T. Yokogawa, and M. Kawahara, “A trend analysis of test smells in python test code over commit history,” in 49th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2023, Durres, Albania, September 6-8, 2023 . IEEE, ...

    2023

  21. [29]

    Detecting test smells in python test code generated by LLM: an empirical study with github copilot,

    V . A. Alves, C. Santos, C. I. M. Bezerra, and I. Machado, “Detecting test smells in python test code generated by LLM: an empirical study with github copilot,” in Proceedings of the 38th Brazilian Symposium on Software Engineering, SBES 2024, Curitiba, Brazil, September 30 - ...

    2024

  22. [30]

    Tsdetect: An open source test smells detection tool,

    A. Peruma, K. Almalki, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “Tsdetect: An open source test smells detection tool,” in Proceedings of the 28th ACM Joint Meeting on European JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12 Software Engineering Confer...

    2021

  23. [31]

    The secret life of test smells - an empirical study on test smell evolution and maintenance,

    D. J. Kim, T.-H. P. Chen, and J. Yang, “The secret life of test smells - an empirical study on test smell evolution and maintenance,” Empirical Software Engineering, vol. 26, no. 5, p. 100, 2021

    2021

  24. [32]

    An empirical investigation into the nature of test smells,

    M. Tufano, F. Palomba, G. Bavota, M. Di Penta, R. Oliveto, A. De Lucia, and D. Poshyvanyk, “An empirical investigation into the nature of test smells,” in 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2016, pp. 4–15

    2016

  25. [33]

    An empirical evaluation of raide: A semi-automated approach for test smells detection and refactoring,

    R. Santana, L. Martins, T. Virg ´ınio, L. Rocha, H. Costa, and I. Machado, “An empirical evaluation of raide: A semi-automated approach for test smells detection and refactoring,” Science of Computer Programming, vol. 231, p. 103013, 2024. [Online]. Available: https://www.scie...

    2024

  26. [34]

    Machine learning-based test smell detection,

    V . Pontillo, D. A. d’Aragona, F. Pecorelli, D. D. Nucci, F. Ferrucci, and F. Palomba, “Machine learning-based test smell detection,” Empirical Software Engineering, vol. 29, no. 2, p. 55, 2024

    2024

  27. [35]

    Ml test smell detection - online appendix,

    D. Amoroso, “Ml test smell detection - online appendix,” 2024, accessed: 2024-08-17. [Online]. Available: https://github.com/darioamorosodaragona-tuni/ ML-Test-Smell-Detection-Online-Appendix/tree/ 6c74556fe8332e5e0864b8019cd8ddee2979af84/dataset

    2024

  28. [36]

    The prompt report: A systematic survey of prompt engineering techniques,

    S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y . Li, A. Gupta, H. Han, S. Schulhoff, P. S. Dulepet, S. Vidyadhara, D. Ki, S. Agrawal, C. Pham, G. Kroiz, F. Li, H. Tao, A. Srivastava, H. D. Costa, S. Gupta, M. L. Rogers, I. Goncearenco, G. Sarli, I. Galynker, D...

    2025 arXiv

  29. [37]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382, 2023

    2023 arXiv

  30. [38]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” 2022. [Online]. Available: https://arxiv.org/abs/2109.01652

    2022 arXiv

  31. [39]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B....

  32. [40]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903

    2023 arXiv

  33. [41]

    Enhancing zero-shot chain-of-thought reasoning in large language models through logic,

    X. Zhao, M. Li, W. Lu, C. Weber, J. H. Lee, K. Chu, and S. Wermter, “Enhancing zero-shot chain-of-thought reasoning in large language models through logic,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluatio...

    2024

  34. [42]

    Wilcoxon, Individual Comparisons by Ranking Methods

    F. Wilcoxon, Individual Comparisons by Ranking Methods . New York, NY: Springer New York, 1992, pp. 196–202. [Online]. Available: https://doi.org/10.1007/978-1-4612-4380-9 16

    1992 doi

  35. [43]

    Copilot evaluation harness: Evaluating llm-guided software programming,

    A. Agarwal, A. Chan, S. Chandel, J. Jang, S. Miller, R. Z. Moghaddam, Y . Mohylevskyy, N. Sundaresan, and M. Tufano, “Copilot evaluation harness: Evaluating llm-guided software programming,” arXiv preprint arXiv:2402.14261, 2024

    2024 arXiv

  36. [44]

    Towards effective validation and integration of llm-generated code,

    N. Tang, “Towards effective validation and integration of llm-generated code,” in 2024 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC) . IEEE, 2024, pp. 369–370

    2024

  37. [45]

    Challenges and opportunities in integrating llms into con- tinuous integration/continuous deployment (ci/cd) pipelines,

    T. Chen, “Challenges and opportunities in integrating llms into con- tinuous integration/continuous deployment (ci/cd) pipelines,” in 2024 5th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT). IEEE, 2024, pp. 364–367

    2024

  38. [46]

    Next-generation refactoring: Combining llm insights and ide capabilities for extract method,

    D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bogomolov, T. Bryksin, and D. Dig, “Next-generation refactoring: Combining llm insights and ide capabilities for extract method,” in 2024 IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME) , 2024, pp...

    2024

  39. [47]

    Llm-based multi-agent systems for software engineering: Literature review, vision and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision and the road ahead,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  40. [48]

    Autorefactoring: A platform to build refactoring agents,

    B. F. dos Santos Neto, M. Ribeiro, V . T. Da Silva, C. Braga, C. J. P. De Lucena, and E. de Barros Costa, “Autorefactoring: A platform to build refactoring agents,” Expert systems with applications , vol. 42, no. 3, pp. 1652–1664, 2015

    2015

  41. [49]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  42. [50]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  43. [51]

    Runeson, M

    P. Runeson, M. Host, A. Rainer, and B. Regnell, Case study research in software engineering: Guidelines and examples . John Wiley & Sons, 2012

    2012

  44. [52]

    Qualitative methods in empirical studies of software engineering,

    C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on software engineering, vol. 25, no. 4, pp. 557–572, 1999

    1999

  45. [2020]

    Available: https://arxiv.org/abs/2005.14165

    [Online]. Available: https://arxiv.org/abs/2005.14165

    2005 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.