Pith. sign in

REVIEW 3 major objections 5 minor 78 references

Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompting LLMs with buggy code measurably steers generated unit tests toward validating the bug, while replacing the code with an LLM-generated specification docstring cuts 'misguided tests' and nearly doubles bug-finding tests.

desk verdict A solid empirical study with a genuinely better metric for the misguidance effect; the main results hold, but the statistics need tightening before acceptance. read the letter →

arxiv 2607.22883 v1 pith:CKMRKGSB submitted 2026-07-24 cs.SE cs.AIcs.LG

Junda Zhao , Shurui Zhou , Eldan Cohen This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords LLMunittestgenerationmisguidanceeffectspecification-basedtestingbuggycodepromptingmisguidedtestseffectivebugdetectioneffectiveness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that showing an LLM buggy code has a measurable and harmful 'misguidance effect' on the unit tests it writes: the model starts treating the bug as intended behavior. The authors quantify the effect with a new metric that runs every generated test against both the buggy code and its fixed counterpart; tests that pass on the buggy version but fail on the fixed one are 'misguided,' and tests that fail on the buggy version but pass on the fixed one are 'effective' at finding the bug. Across 11 large language models and hundreds of real Java methods, buggy input produced on average 137.69 misguided tests and only 104.15 effective tests, and model-internal probability scores confirm that the buggy code biases the model's preferences. The paper's proposed fix is to replace the code in the prompt with an LLM-generated specification docstring, which lowers misguided tests to 113.00 and raises effective tests to 186.77. If the effect is real, LLM-based test generation should be built around recovered specifications rather than the code under test.

What carries the argument

The mechanism is a dual-execution test classifier: every generated test is compiled and run against both the buggy version and its fixed counterpart, and the four outcome combinations define the labels (true negative, effective, misguided, false positive). This classifier is what makes the misguidance effect measurable rather than anecdotal. The mitigation machinery is a two-step specification pipeline: the model first writes a docstring that describes intended behavior while avoiding code quotation, optionally after an explicit audit for logical mistakes and robustness gaps, and then that docstring replaces the code entirely in the test-generation prompt. The paper's ablations show that both halves are necessary: removing the code without providing a spec loses too many effective tests, and adding the docstring without removing the code barely helps.

What would settle it

Sample the tests the metric labels 'misguided' and have independent human judges decide, without seeing the fixed code, whether each assertion contradicts the method's documented or intended behavior; if a substantial fraction are judged acceptable, the measured misguidance effect would shrink or disappear.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the same LLM produces a very different test suite depending on whether the prompt contains buggy code or its fixed counterpart: relative to fixed-code prompting, buggy-code prompting increases tests that assert the buggy behavior and suppresses tests that expose it, averaging 137.69 misguided versus 104.15 effective tests per model, compared with 16.46 and 304.08 for fixed-code input. This pattern is not a measurement artifact of simply counting tests that pass on buggy code, since over 90% of such tests pass on the fixed version too; the new 'misguided test' definition isolates the minority that actively assert the bug. The paper then shows that a two-step specification-based prompt, generate a behavioral docstring from the buggy code and then feed only that docstring to the test-writing model, reduces misguided tests to 113.00 and increases effective tests to 186.77, with a stronger analysis-driven docstring prompt improving both further. Sequence-score evidence shows that conditioned on buggy code, the model assigns higher likelihood to misguided tests, and conditioned on fixed code, to effective tests, so the behavioral change tracks the model's internal preference.

Load-bearing premise

The load-bearing premise is that the fixed version uniquely defines intended behavior, so a test that passes on the buggy version and fails on the fixed version is automatically 'misguided'; if some such tests assert acceptable alternative behaviors, the effect sizes and the reported improvements are overstated.

Editorial extensions

If this is right

  • Existing benchmarks that feed only bug-free code to LLMs likely overestimate real bug-detection ability, because the same models produce roughly three times fewer effective tests when the code is buggy.
  • Practical test-generation tools should treat the code under test as suspect and derive tests from a separately recovered specification, and the code must be removed entirely rather than supplemented with documentation.
  • Models with stronger code comprehension are also more susceptible to misguidance, so choosing a more capable model does not by itself mitigate the effect.
  • Multi-round feedback-driven test generation inherits and accumulates the misguidance effect, while starting from a specification docstring keeps misguided-test growth lower across refinement rounds.
  • The mitigation adds only one extra LLM call and, on bug-free code, keeps compilation-failure, false-alarm, and coverage metrics comparable to code-based prompting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same spec-replacement design could be applied to other bug-inducing feedback loops, such as LLM-based program repair and code generation with test feedback, where erroneous code repeatedly re-enters the prompt.
  • Beyond the paper: the paper's oracle assumption can be stress-tested by re-labelling a sample of 'misguided' tests with human judges; if many are acceptable alternative behaviours, the reported effect sizes would shrink.
  • Beyond the paper: because the advanced docstring prompt slightly raises hallucinated-behaviour tests, roughly from 16% to 18%, investing in specification quality, for example by cross-checking generated specs against execution traces, should be a high-leverage next step.
  • Beyond the paper: a direct comparison against human-written specifications would bracket the upper bound of the mitigation, since the paper shows only what LLM-recovered specifications can achieve.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a new metric for quantifying the 'misguidance effect' of buggy code on LLM-generated unit tests: a test is 'misguided' if it passes on the buggy version of a focal method and fails on its fixed version. Using 318 focal methods from Defects4J and 11 LLMs (13 configurations), the authors report that prompting with buggy code increases misguided tests (average 137.69 vs 16.46 for fixed code) and suppresses effective tests (104.15 vs 304.08). They then propose a specification-based approach that replaces the code under test with an LLM-generated docstring, reporting a reduction in misguided tests to 113.00 and an increase in effective tests to 186.77. The paper also analyzes sequence scores, multi-round feedback, and manual docstring annotations, and evaluates the approach on bug-free code.

Significance. If the results hold, this is a valuable contribution to LLM-based test generation: it sharpens the definition of misguidance relative to prior work (Huang et al.), provides a practical mitigation strategy, and ships a public replication package with manual annotations. The study's strengths include a large and diverse model set, a consistent experimental pipeline, cross-scoring sequence-score analysis, and good inter-annotator agreement. The main concerns are the unvalidated oracle assumption, the biased benchmark filter, and the lack of inferential statistics for the headline comparisons.

major comments (3)
  1. [Section 2.6, Table 2] The classification in Table 2 labels a test as 'misguided' if it passes on the buggy version and fails on the fixed version, assuming the fixed Defects4J version is the unique gold standard for intended behavior. Because Defects4J patches can include changes beyond the minimal bug fix, a test in this category may assert behavior that merely differs from the post-patch state rather than the buggy behavior itself, which would overstate the misguidance effect. This is load-bearing for the central claims of RQ1 (Tables 4 and 5) and RQ2 (Tables 6 and 8). Please manually or automatically validate a sample of 'misguided' tests against the patch diff to estimate the rate of non-bug-related behavioral changes, and report how the reported effect sizes change if those tests are excluded.
  2. [Section 2.2, criterion (3)] The benchmark filter restricts the focal methods to those that trigger at least one existing human-written test in Defects4J, which ensures the patch is a bug fix but also limits the study to bugs that are already detectable by the provided test suite. Such bugs may have more clearly identifiable buggy behavior, potentially inflating the measured misguidance effect relative to the general population of defects. The paper does not discuss this selection bias in Section 6. Please add either a sensitivity analysis on a sample of unfiltered methods or a discussion of how the filter affects the interpretation and generalizability of the effect sizes.
  3. [Sections 3.1, 4.1, and 4.2] The paper repeatedly uses 'significantly' and 'substantially' for the main comparisons (e.g., Section 4.1 states that the approach 'significantly improves test quality'), but no p-values, confidence intervals, or effect sizes are reported for these contrasts; the only inferential statistics are the Pearson correlations in Section 3.1. Since the results are based on 13 model configurations, paired tests (e.g., Wilcoxon signed-rank) and effect sizes are feasible. Please report these for the key comparisons in Tables 4, 6, and 8.
minor comments (5)
  1. [Section 6] The paper does not discuss flaky tests, which could affect the pass/fail labels in Table 2 and therefore the classification of tests; please state how flakiness was handled or mitigated.
  2. [Section 4.3] The multi-round setup is described as following ChatTester, but the exact number of rounds, the feedback format, and the stopping criterion are not specified; please provide these details for reproducibility.
  3. [Table 12] The paper should define the denominators for CFR and FAR (e.g., percentage of all generated tests versus percentage of compiled tests) explicitly in the text.
  4. [Section 4.4] The manual inspection selects the two models with the largest and smallest reductions in misguided test suites, which is a selection for extremes that may overstate the relationship between docstring quality and test outcomes; please clarify the implications or consider a random sample.
  5. [Equation (1)] The sequence score uses log probabilities but does not specify the base of the logarithm; please state the base for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central metric is an external-oracle definition and every headline comparison is empirical.

full rationale

The paper's central quantity, the 'misguided test' (passes on buggy, fails on fixed) and 'effective test' (fails on buggy, passes on fixed), is defined in Section 2.6 against the fixed Defects4J version, which is an external oracle chosen before the experiments. All headline comparisons (buggy vs. fixed input in Table 4; docstring vs. code in Table 6; advanced prompt in Table 8) vary only the prompt input and then measure the resulting tests by execution against that same external oracle. The observed increases in misguided tests are therefore empirical findings, not consequences of the definitions: the models could in principle ignore the buggy code or emit tests failing on both versions, and indeed Table 3 shows that over 90% of tests that pass on the buggy version are true negatives rather than misguided. The specification-docstring mitigation is also not circular because docstring generation never accesses the fixed version; it relies on the LLM's own inference, and the evaluation shows the expected trade-off (e.g., advanced prompts also raise false-positive ratios in Table 10). The only same-author references are the replication package [78] and a general paraphrase-for-contamination citation [62] used as supporting context in Section 6; neither supplies the metric, the oracle, or the measured effects. The fixed-version-as-gold-standard assumption is a contestable construct-validity choice, but it is a threat to the interpretation of the labels, not a circularity: the derivation chain never feeds the conclusion back into the definition.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the oracle definition and benchmark selection rather than on free parameters or invented constructs. No numbers are fitted; the main assumptions are domain-level choices about what counts as intended behavior.

assumptions (4)
  • domain assumption The fixed version of each Defects4J defect is the gold standard for intended behavior.
    All test labels (misguided/effective) are defined by comparing execution against the fixed version; Section 2.6 states the fixed version is treated as the gold standard.
  • domain assumption Focal methods that trigger at least one human-written test on the buggy version are representative of real-world buggy code.
    Section 2.2 criterion (3) restricts the benchmark to 318 focal methods with a known detecting test, which may overrepresent detectable bugs.
  • domain assumption Sequence scores computed by open-source evaluator models reflect the test-generating model's internal preference.
    Section 3.2 uses cross-scoring with three open-source models; it assumes their token probabilities proxy the generator's preference.
  • domain assumption The Defects4J benchmark and the 11 evaluated LLMs are representative of real-world buggy code and SOTA models.
    Generalizability claim in Section 6; the authors acknowledge Defects4J may not capture proprietary or stateful systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests." pith.science (2026). https://pith.science/paper/CKMRKGSB

@misc{pith2026260722883,
  author       = {Pith},
  title        = {Pith review of: Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKMRKGSB}},
  note         = {Machine review of arXiv:2607.22883}
}
read the original abstract

While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Our analysis reveals that prompting LLMs with buggy code has a severe, twofold impact: it significantly increases "misguided tests" that assert incorrect behavior while simultaneously suppressing the generation of effective, bug-finding tests. We further corroborate this effect from a model-internal perspective, showing that buggy code skews LLMs' preference toward tests that assert the same erroneous behavior. To counter this, we introduce and validate a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. Our results show that this paradigm effectively reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code. Overall, these results suggest that specification-based prompting is a promising strategy for mitigating misguidance from buggy code in LLM-generated unit tests.

Figures

Figures reproduced from arXiv: 2607.22883 by the authors.

Figure 1
Figure 1. Example of a “misguided test” generated from buggy code that asserts its buggy behavior, and how the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The prompt template used for generating unit tests. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Merged prompt templates for generating specification docstrings from code under test. Each row [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Example of a false-alarm test generated from a hallucinated docstring. The highlighted lines mark the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Workflow for evaluation of the misguidance effect from buggy code. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Qualitative example comparing the Base Docstring Prompt and the Advanced Docstring Prompt. tests by 32.00 on average (-0.98 pp) and increasing effective tests by 56.15 (+0.59 pp). The tailored multi-step prompt similarly improves base models, decreasing misguided tests…
Figure 7
Figure 7. Figure 7: Changes in the counts of compiled, effective, and misguided tests (top) and in the number of focal [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 16 canonical work pages

  1. [1]

    Azat Abdullin, Pouria Derakhshanfar, and Annibale Panichella. 2025. Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generation. arXiv:2501.10200 [cs.SE] doi:10.48550/arXiv.2501.10200

    arXiv doi:10.48550/arxiv.2501.10200 2025

  2. [2]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 2655–2668. doi:10.18653/v...

    doi:10.18653/v1/2021.naacl-main.211 2021

  3. [3]

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2024. A3Test: Assertion-Augmented Automated Test Case Generation.Information and Software Technology176 (2024), 107565. doi:10.1016/j.infsof.2024.107565

    arXiv 2024

  4. [5]

    Juan Altmayer Pizzorno and Emery D. Berger. 2025. CoverUp: Effective High Coverage Test Generation for Python. Proc. ACM Softw. Eng.2, FSE (June 2025), 2897–2919. doi:10.1145/3729398

    doi:10.1145/3729398 2025

  5. [6]

    Ammann, P.E

    P.E. Ammann, P.E. Black, and W. Majurski. 1998. Using model checking to generate tests from specifications. In Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241). 46–54. doi:10.1109/ ICFEM.1998.730569

    work page arXiv 1998

  6. [7]

    Anthropic. 2024. Claude 3.5 Sonnet. Accessed: 2025-02-19. https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  7. [8]

    Anthropic. 2024. Introducing the Next Generation of Claude. Accessed: 2025-03-13. https://www.anthropic.com/ news/claude-3-family

    work page 2024

  8. [9]

    Anthropic. 2025. Claude 4 Sonnet. Accessed: 2025-05-23. https://www.anthropic.com/claude/sonnet

    work page 2025

Show all 78 references
  1. [10]

    2002.Test-Driven Development: By Example

    Kent Beck. 2002.Test-Driven Development: By Example. Addison-Wesley Professional

    2002

  2. [11]

    Brown et al

    Tom B. Brown et al. 2020. Language models are few-shot learners. InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages. doi:10.48550/arXiv.2...

    2020 doi arXiv

  3. [12]

    Max Brunsfeld. 2018. Tree-sitter: An incremental parsing system for programming tools. Accessed: 2025-02-21. doi:10.5281/zenodo.4619183

    2018 doi

  4. [13]

    Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. InProceedings of the 8th USENIX Conference on Operating Systems Design and Implementation(San Diego, California)(OSDI’08). USE...

    2008

  5. [14]

    Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] doi:10.48550/ arXiv.2107.03374

    2021 doi arXiv

  6. [15]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering(Porto de Galinhas, Brazil)(...

    2024

  7. [16]

    Bonnie Chinh, Himanshu Zade, Abbas Ganji, and Cecilia Aragon. 2019. Ways of qualitative coding: A case study of four strategies for resolving disagreements. InExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6. doi:10.1145/3290607.3312879

    2019

  8. [17]

    Ermira Daka and Gordon Fraser. 2014. A Survey on Unit Testing Practices and Problems. In2014 IEEE 25th International Symposium on Software Reliability Engineering. 201–211. doi:10.1109/ISSRE.2014.11

    2014 doi

  9. [18]

    Desmarais

    Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre-trained Large Language Models and mutation testing.Information and Software Technology 171 (2024), 107468. doi:10.1016/j.infsof.2024.107468...

    2024

  10. [19]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] doi:10.48550/arXiv.2501.12948

    2025 doi arXiv

  11. [20]

    DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] doi:10.48550/arXiv.2412.19437

    2025 doi arXiv

  12. [21]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: a neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery,...

    2022

  13. [22]

    Enoiu, Adnan Čaušević, Thomas J

    Eduard P. Enoiu, Adnan Čaušević, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, and Paul Pettersson. 2016. Automated test generation using model checking: an industrial evaluation.Int. J. Softw. Tools Technol. Transf.18, 3 (June 2016), 335–353. doi:10.1007/s10009-014-0355-9

    2016 doi

  14. [23]

    Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Trans. Softw. Eng. Methodol.24, 2, Article 8 (Dec. 2014), 42 pages. doi:10.1145/2685612

    2014 doi

  15. [24]

    Jacob Tsao, and Ye Wu

    Jerry Gao, H.-S. Jacob Tsao, and Ye Wu. 2003.Testing and Quality Assurance for Component-Based Software. Artech House

    2003

  16. [25]

    Google. 2025. Gemini 2.5 Flash Model. https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash. Accessed: 2025-08-19

    2025

  17. [26]

    Google. 2025. Gemini 2.5 Pro Model. https://ai.google.dev/gemini-api/docs/models#gemini-2.5-pro. Accessed: 2025-08-19

    2025

  18. [27]

    Lehan He, Zeren Chen, Zhe Zhang, Jing Shao, Xiang Gao, and Lu Sheng. 2025. Use Property-Based Testing to Bridge LLM Code Generation and Validation. arXiv:2506.18315 [cs.SE] doi:10.48550/arXiv.2506.18315

    2025 doi arXiv

  19. [28]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. InInternational Conference on Learning Representations. doi:10.48550/arXiv.1904.09751

    2020 doi arXiv

  20. [29]

    Soneya Binta Hossain and Matthew B. Dwyer. 2025. TOGLL: Correct and Strong Test Oracle Generation with LLMs. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1475–1487. doi:10.1109/ICSE55347. 2025.00098

    2025

  21. [30]

    Soneya Binta Hossain, Raygan Taylor, and Matthew Dwyer. 2025. Doc2OracLL: Investigating the Impact of Docu- mentation on LLM-Based Test Oracle Generation.Proc. ACM Softw. Eng.2, FSE, Article FSE084 (June 2025), 22 pages. doi:10.1145/3729354

    2025 doi

  22. [31]

    Zhang, Mark Harman, Mingzhe Du, and Heming Cui

    Dong Huang, Jie M. Zhang, Mark Harman, Mingzhe Du, and Heming Cui. 2025. Measuring the Influence of Incorrect Code on Test Generation. arXiv:2409.09464 [cs.SE] doi:10.48550/arXiv.2409.09464

    2025 doi arXiv

  23. [32]

    Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering(Hyderabad, India)(ICSE 2014). Association for Computing Machinery, New York, NY, USA, 435–445...

    2014

  24. [33]

    Kush Jain and Claire Le Goues. 2025. TestForge: Feedback-Driven, Agentic Test Suite Generation. arXiv:2503.14713 [cs.SE] doi:10.48550/arXiv.2503.14713

    2025 doi arXiv

  25. [34]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. InProceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA)(ISSTA 2014). Associat...

    2014

  26. [35]

    Richard Landis and Gary G

    J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data.Biometrics33, 1 (1977), 159–174. doi:10.2307/2529310

    1977 doi

  27. [36]

    Lahiri, and Siddhartha Sen

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 919–931. doi:10.11...

    2023

  28. [37]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. InProceedings of the 58th...

    2020 doi

  29. [38]

    Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting. arXiv:2304.11686 [cs.SE] doi:10.48550/arXiv.2304.11686

    2023 doi arXiv

  30. [39]

    Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang

    Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M. Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang. 2025. LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs. InProceedings of the 63rd Annual Meeting of the Association for Computation...

    2025 doi

  31. [40]

    Andrea Lops, Fedelucio Narducci, Azzurra Ragone, and Michelantonio Trizio. 2024. AgoneTest: Automated creation and assessment of Unit tests leveraging Large Language Models. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacrament...

    2024

  32. [41]

    Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs. InProceedings of the 62nd Annual Meeting of the Associat...

    2024 doi

  33. [42]

    Stephan Lukasczyk and Gordon Fraser. 2022. Pynguin: automated unit test generation for Python. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings(Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, ...

    2022

  34. [43]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Design choices made by LLM-based test generators prevent them from finding bugs. arXiv:2412.14137 [cs.SE] doi:10.48550/arXiv.2412.14137

    2024 doi arXiv

  35. [44]

    Miles, A

    Matthew B. Miles, A. Michael Huberman, and Johnny Saldaña. 2014.Qualitative Data Analysis: A Methods Sourcebook (third ed.). SAGE Publications, Thousand Oaks, California

    2014

  36. [45]

    Manning, and Chelsea Finn

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: zero- shot machine-generated text detection using probability curvature. InProceedings of the 40th International Conference on Machine Learning(Honolulu, Hawaii, USA)(ICM...

    2023 doi arXiv

  37. [46]

    Cliodhna O’Connor and Helene Joffe. 2020. Intercoder reliability in qualitative research: Debates and practical guidelines.International Journal of Qualitative Methods19 (2020), 1609406919899220. doi:10.1177/1609406919899220

    2020 doi

  38. [47]

    OpenAI. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] doi:10.48550/arXiv.2410.21276

    2024 doi arXiv

  39. [48]

    OpenAI. 2025. GPT-4.1. https://openai.com/index/gpt-4-1/. Accessed: 2025-08-19

    2025

  40. [49]

    OpenAI. 2025. gpt-oss-120b and gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL] doi:10.48550/arXiv.2508.10925

    2025 doi arXiv

  41. [50]

    OpenAI. 2025. OpenAI Models - O4 Mini. https://platform.openai.com/docs/models/o4-mini. Accessed: 2025-08-19

    2025

  42. [51]

    Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. ASTER: Natural and Multi- language Unit Test Generation with LLMs. arXiv:2409.03093 [cs.SE] doi:10.48550/arXiv.2409.03093

    2025 doi arXiv

  43. [52]

    Păsăreanu, Peter C

    Corina S. Păsăreanu, Peter C. Mehlitz, David H. Bushnell, Karen Gundy-Burlet, Michael Lowry, Suzette Person, and Mark Pape. 2008. Combining unit-level symbolic execution and system-level concrete execution for testing NASA software. InProceedings of the 2008 International Symp...

    2008

  44. [53]

    Qwen Team. 2025. Qwen3-Coder: Agentic Coding in the World. https://qwenlm.github.io/blog/qwen3-coder/. Accessed: 2025-08-19

    2025

  45. [54]

    Qwen Team. 2025. Qwen3: Think Deeper, Act Faster. https://qwenlm.github.io/blog/qwen3/. Accessed: 2025-08-19

    2025

  46. [55]

    Hellendoorn

    Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2023. CAT-LM Training Language Models on Aligned Code and Tests. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering(Echternach, Luxembourg)(ASE ’23). IEEE ...

    2023

  47. [56]

    2025.The Coding Manual for Qualitative Researchers(fifth ed.)

    Johnny Saldaña. 2025.The Coding Manual for Qualitative Researchers(fifth ed.). SAGE Publications Ltd. doi:10.4135/ 9781036235611

    2025

  48. [57]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation.IEEE Transactions on Software Engineering50, 1 (2024), 85–105. doi:10. 1109/TSE.2023.3334955

    2024

  49. [58]

    Ye Shang, Quanjun Zhang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. 2025. A Large-Scale Empirical Study on Fine-Tuning Large Language Models for Unit Testing.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA074 (June 2025), 23 pages. doi:10.1145/3728951

    2025 doi

  50. [59]

    Jiho Shin, Sepehr Hashtroudi, Hadi Hemmati, and Song Wang. 2024. Domain Adaptation for Code Model-based Unit Test Case Generation. arXiv:2308.08033 [cs.SE] doi:10.48550/arXiv.2308.08033

    2024 doi arXiv

  51. [60]

    Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using Large Language Models to Generate JUnit Tests: An Empirical Study. In Proceedings of the 28th International Conference on Evalu...

    2024

  52. [61]

    2011.Software Engineering(ninth ed.)

    Ian Sommerville. 2011.Software Engineering(ninth ed.). Pearson Education, Boston, MA

    2011

  53. [62]

    Yuliang Song and Eldan Cohen. 2025. Do LLMs Understand Constraint Programming? Zero-Shot Constraint Pro- gramming Model Generation Using LLMs. InProceedings of the 19th Learning and Intelligent Optimization Conference (LION-25). 16–31. doi:10.1007/978-3-032-09156-7_2

    2025 doi

  54. [63]

    André Storhaug and Jingyue Li. 2024. Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study. arXiv:2411.02462 [cs.SE] doi:10.48550/arXiv.2411.02462 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA113. Publication date: Octo...

    2024 doi

  55. [64]

    Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2024. ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation.IEEE Transactions on Software Engineering50, 6 (2024), 1340–1359. doi:10.1109/TSE.2024.3382365

    2024

  56. [65]

    Hugo Touvron et al . 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] doi:10.48550/arXiv.2302.13971

    2023 doi arXiv

  57. [66]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit Test Case Generation with Transformers and Focal Context. arXiv:2009.05617 [cs.SE] doi:10.48550/arXiv.2009.05617

    2021 doi arXiv

  58. [67]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 [cs.CL] doi:10.48550/arXiv.1706.03762

    2017 doi arXiv

  59. [68]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision.IEEE Transactions on Software Engineering50, 4 (April 2024), 911–936. doi:10.1109/TSE.2024.3368208

    2024

  60. [69]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder- Decoder Models for Code Understanding and Generation. arXiv:2109.00859 [cs.CL] doi:10.48550/arXiv.2109.00859

    2021 doi arXiv

  61. [70]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machin...

    2024 doi

  62. [71]

    xAI. 2025. Grok-3 Model Documentation. https://docs.x.ai/docs/models/grok-3. Accessed: 2025-08-19

    2025

  63. [72]

    xAI. 2025. Grok-4 Model Documentation. https://docs.x.ai/docs/models/grok-4. Accessed: 2025-08-19

    2025

  64. [73]

    Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. 2024. On the Evaluation of Large Language Models in Unit Test Generation. In Proceedings of the 39th IEEE/ACM International Conference on ...

    2024

  65. [74]

    Gonzalez, and Ion Stoica

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. 2023. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. arXiv:2311.04850 [cs.CL] doi:10.48550/arXiv.2311.04850

    2023 doi arXiv

  66. [75]

    Zhen Yang, Fang Liu, Zhongxing Yu, Jacky Wai Keung, Jia Li, Shuo Liu, Yifan Hong, Xiaoxue Ma, Zhi Jin, and Ge Li

  67. [76]

    Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024. ThinkRepair: Self-Directed Automated Program Repair. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis(Vienna, Austria)(ISSTA 2024). Association for Comp...

    2024

  68. [77]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation.Proc. ACM Softw. Eng.1, FSE, Article 76 (July 2024), 24 pages. doi:10.1145/3660783

    2024 doi

  69. [78]

    Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

    Junda Zhao, Shurui Zhou, and Eldan Cohen. 2026. Replication Package for “Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests”. doi:10.5281/zenodo.21428156 Received 2026-01-30; accepted 2026-06-25 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, ...

    2026 doi

  70. [2024]

    ACM Softw

    Exploring and Unleashing the Power of Large Language Models in Automated Code Translation.Proc. ACM Softw. Eng.1, FSE, Article 71 (July 2024), 24 pages. doi:10.1145/3660778

    2024 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.