Pith. sign in

REVIEW 3 major objections 6 minor 39 references

Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that documented interfaces plus chain-of-thought prompting are the main levers for automated unit-test quality, and that Gemini 2.5 Pro is the best of six tested models on mutation score and branch coverage.

desk verdict Useful single-run benchmark with public artifacts; the ranking claims outrun the data because there are no error bars. read the letter →

arxiv 2507.14256 v1 pith:DBJXCIQ3 submitted 2025-07-18 cs.SE cs.AI

Jakub Walczak , Piotr Tomalak , Artur Laskowski This is my paper
classification cs.SEcs.AI
keywords automatedunittestgenerationlargelanguagemodelscodecontextpromptingstrategieschain-of-thoughtmutationtestingbranchcoveragesoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which parts of the prompt matter when a general-purpose LLM is asked to write unit tests, and whether asking the model to plan before implementing pays off. Using twelve custom, previously unpublished Python methods from a miniature shopping-cart system, the authors compare six LLMs under three code-context conditions—signatures only, signatures plus docstrings, and full implementations—and under two prompting strategies, a single-step simple instruction and a multi-step chain-of-thought instruction. Their central claim is that docstrings are the main context that improves test adequacy, that adding the full implementation yields only small further gains, and that chain-of-thought prompting produces the best measured results, up to 96.3% branch coverage and a 57% average mutation score with near-perfect compilation. They also report that chain-of-thought helps rather than degrades 'reasoning' models. If these results hold, the actionable recipe for practitioners is to feed the model a well-documented interface and let it reason out scenarios before writing test code.

What carries the argument

The controlling object is a deliberately scoped micro-benchmark: twelve custom Python methods of a shopping-cart class, written for this study so they could not have appeared in any model's training data, with typed signatures, detailed docstrings, and full implementations fixed in advance. The experimental design crosses three code-context levels (CF1 signatures only, CF2 signatures plus docstrings, CF3 the complete implementation) with two prompting strategies (S1 a single prompt asking for tests, S2 a chain-of-thought sequence that first asks for test scenarios and then asks for the implementation). Quality is measured by compilation success rate, branch coverage, method coverage, mutation score computed with the mutmut mutation-testing tool, the number of unique scenarios after manual deduplication, and generation time; a single exception branch unreachable through the public API caps achievable branch coverage at 98%.

What would settle it

Re-run the published 12-method benchmark across all model-strategy-context combinations several times, say 10 seeds each, and compute confidence intervals for compilation success rate, branch coverage, and mutation score. If the full-context versus docstring-only mutation-score gap of about 4 percentage points and Gemini 2.5 Pro's 86–87% mutation score overlap with the intervals of other configurations, the stated hierarchy and the claim that full implementation gives only small gains would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that, in automated unit-test generation, the documented interface carries most of the value: moving from raw signatures to signatures plus docstrings raises compilation success, branch coverage, and mutation scores substantially, while moving from that to the full implementation raises them only slightly (about 4 to 7 percentage points on mutation score and compilation success, depending on prompting strategy). Chain-of-thought prompting—where the model first enumerates test scenarios and then implements them—gives the best overall configuration, and the paper explicitly contests the common assertion that chain-of-thought degrades reasoning models. On the tested set, Gemini 2.5 Pro is the strongest model, reaching 86–87% mutation score with full context and simple prompting, while the best average configuration reaches 96.33% branch coverage, a 56.83% average mutation score, and 99.67% compilation success. The paper also documents a 98% coverage ceiling caused by one branch unreachable through the public interface, and notes that no model tried to force it.

Load-bearing premise

The conclusions assume that a single generated test suite for each model–prompt–context combination is a reliable sample of what the model does; the paper reports no repeated runs or confidence intervals, so the observed gaps could be within normal run-to-run variation.

Editorial extensions

If this is right

  • Adding detailed docstrings to method signatures should be the first improvement a team makes; the paper finds it raises compilation success, branch coverage, and mutation score far more than adding the full implementation afterward.
  • Chain-of-thought prompting can be used on reasoning-class LLMs without the degradation some vendors warn about; the best measured configuration relies on it and reaches 96.3% branch coverage.
  • LLM-generated test suites on this benchmark can exceed a human-written suite in fault detection: several configurations score above the practitioner baseline's 44% mutation score, with the best at 86–87%.
  • Human review of generated tests should be pointed at the systematic gaps the paper documents—performance scenarios and edge values such as None, infinity, and NaN—rather than at ordinary coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the single-run sampling issue proves real, the paper's model ranking is likely its least stable output, while the docstring-over-implementation ranking is more credible because the gaps are larger across configurations.
  • A benchmark whose exception branches are reachable through the public API would give coverage a wider dynamic range; the current 98% ceiling flattens the top models into a very narrow band.
  • The docstring effect suggests a testable extension: deliberately degrade or omit docstrings and measure how much test quality drops, which would separate documentation quality from mere context length.
  • The 187% generation-time cost of chain-of-thought invites an adaptive strategy—reserve chain-of-thought for methods with many branches or tricky edge cases and use simple prompting elsewhere—that the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper reports an empirical comparison of six general-purpose large language models (GPT-4.5, GPT-o3, GPT-o4-mini-high, Claude 3.7 Sonnet, Gemini 2.5 Pro, DeepSeek-V3) on a custom, previously unpublished Python shopping-cart module. For each of 12 methods, the authors construct unit-test generation prompts under two strategies (simple prompting and chain-of-thought) and three levels of code context (signatures only; signatures with docstrings; full implementation), and evaluate the generated suites by compilation success rate, branch and method coverage, mutation score, number of unique scenarios, and generation time. The central claims are that docstrings contribute substantially, that extending context to full implementation gives only small further gains, that chain-of-thought prompting yields the best results, and that Gemini 2.5 Pro is the strongest model.

Significance. The topic is timely and the design choices reduce two common threats to validity: the custom benchmark mitigates data leakage, and the use of standard coverage and mutation metrics facilitates comparison with prior work. The authors also publish prompts, code, and generated test suites, which is a concrete reproducibility strength. If the reported effect sizes were accompanied by repeated runs and uncertainty quantification, the study would provide useful practical guidance on prompt and context selection. As it stands, however, the paper's headline rankings and the strength of the CF2-versus-CF3 comparison exceed what a single sample per condition can support.

major comments (3)
  1. [Section III.F and Table IV] Each reported metric is based on a single generated test suite per model-strategy-context condition; the paper provides no repeated runs, confidence intervals, or significance tests. For stochastic LLMs, the observed CF2-to-CF3 differences are small in absolute terms (e.g., S1 branch coverage 93.50 vs. 91.50; S2 mutation score 52.83 vs. 56.83) and could easily be within sampling noise. Consequently, the abstract's claim that full implementation yields 'definitely smaller gains' and the conclusion that CF3 yields the best results are not established by the reported data. Additionally, Section IV.A's assertion that the differences are 'statistically negligible' is not supported by any statistical test.
  2. [Abstract and Table III] The abstract states that chain-of-thought prompting achieves a 'near-perfect compilation success rate,' but under M5 (Gemini 2.5 Pro) the CF2 condition has CSR of 64% with S1 and 69% with S2, while several other configurations reach 95-100%. The claim of M5's superiority is also conditional: its high mutation scores (86-87%) occur only at CF3, and at CF2 its CSR is the lowest among all models. The abstract should be qualified to the specific conditions under which these outcomes hold.
  3. [Section IV.A and Table IV] The text says that full context (CF3) 'consistently yields the highest performance across most evaluation metrics,' but under S1, CF3 has lower branch coverage (91.50%) than CF2 (93.50%). Since CF3 is not uniformly better than CF2, the narrative in the abstract and conclusions overstates the dominance of full implementation over the signature-plus-docstring context.
minor comments (6)
  1. [Section III.F] The phrase 'code context level (CF1-CF2)' should read 'CF1-CF3' to match the three defined levels.
  2. [Section IV.F] 'Software partitioner (SP)' appears to be a typo for 'software practitioner (SP)'; the same baseline is called 'practitioner' elsewhere in the section.
  3. [Generative AI Statement] 'In out studies' should be 'In our studies.'
  4. [Section IV.C] 'ad type validation' should be 'and type validation.'
  5. [Author affiliations] The affiliations list the city as 'd, Poland,' which appears to be a placeholder that should be replaced with the actual city name.
  6. [Section IV.C] The text refers to '99% statement coverage,' but statement coverage is not defined in Section III.B; the defined coverage measures are branch coverage and method coverage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all quality metrics come from external tools and independent comparison, with no fitted parameters or conclusion defined in terms of its own inputs.

full rationale

The paper's claims are empirical comparisons of LLM-generated test suites evaluated by external, independently meaningful metrics: compilation success rate is computed from actual test execution, branch and method coverage from coverage tooling, and mutation score from the mutmut framework against manually designed mutants. No parameter is fitted to the data and then renamed as a prediction; the code context levels (CF1-CF3) and prompting strategies (S1-S2) are defined operationally before measurement, and the reported outcomes (CSR, MS, BC, count, time) are not used in the definitions of those treatments. The custom unpublished benchmark code is a deliberate methodological choice to avoid data leakage, and the self-citation [27] is used only to motivate that precaution; it is not load-bearing for any numerical result or ranking. The human practitioner baseline (SP) provides an external reference, and the paper repeatedly notes limitations, including single-run settings and statistically negligible aggregate differences. Those are validity concerns about sampling noise, not circularity: no equation or claim reduces by construction to its own input. The derivation chain is therefore self-contained with respect to the measured outcomes.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted to data. The paper's central claims rest on assumptions about benchmark representativeness, single-run stability, mutation-test validity, data-leakage avoidance, and the branch-coverage ceiling; none of these are mathematical axioms, but all are domain assumptions the reader must accept.

assumptions (5)
  • domain assumption The 12 custom shopping-cart Python methods are representative of unit-testing workloads and can support generalization to larger systems.
    Section III.A acknowledges the small dataset but argues it demonstrates a single unit of code and aligns with unit-testing paradigms; the paper's practical conclusions depend on this representativeness.
  • domain assumption One generated test suite per model, prompt, and context condition at default or zero temperature is stable enough for comparisons.
    Section III.F reports a single run per condition with no repeated trials; Tables III and IV report no confidence intervals, so all observed differences carry unquantified sampling noise.
  • domain assumption Mutation score computed on this code is a valid measure of test-suite fault-detection ability.
    Section III-B.3 and Appendix A use mutation score as a quality measure; the paper is inconsistent about whether mutants were designed manually or generated by mutmut, which affects how the score should be interpreted.
  • domain assumption The custom code was absent from LLM training data, so evaluations are not inflated by memorization.
    Section III.A claims the previously unpublished code avoids data leakage; this is asserted rather than independently verified.
  • domain assumption Branch coverage is capped at 98% because one branch is unreachable through the public interface.
    Section IV.A states the theoretical maximum is 98%, so several top models sit at the ceiling and branch coverage cannot distinguish their performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models." pith.science (2026). https://pith.science/paper/DBJXCIQ3

@misc{pith2026250714256,
  author       = {Pith},
  title        = {Pith review of: Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBJXCIQ3}},
  note         = {Machine review of arXiv:2507.14256}
}
read the original abstract

Generative AI is gaining increasing attention in software engineering, where testing remains an indispensable reliability mechanism. According to the widely adopted testing pyramid, unit tests constitute the majority of test cases and are often schematic, requiring minimal domain expertise. Automatically generating such tests under the supervision of software engineers can significantly enhance productivity during the development phase of the software lifecycle. This paper investigates the impact of code context and prompting strategies on the quality and adequacy of unit tests generated by various large language models (LLMs) across several families. The results show that including docstrings notably improves code adequacy, while further extending context to the full implementation yields definitely smaller gains. Notably, the chain-of-thought prompting strategy -- applied even to 'reasoning' models -- achieves the best results, with up to 96.3\% branch coverage, a 57\% average mutation score, and near-perfect compilation success rate. Among the evaluated models, M5 (Gemini 2.5 Pro) demonstrated superior performance in both mutation score and branch coverage being still in top in terms of compilation success rate. All the code and resulting test suites are publicly available at https://github.com/peetery/LLM-analysis.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 29 canonical work pages

  1. [1]

    Harnessing the power of llms in practice: A survey on chatgpt and beyond,

    J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, S. Zhong, B. Yin, and X. Hu, “Harnessing the power of llms in practice: A survey on chatgpt and beyond,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 6, pp. 1–32, 2024

    2024

  2. [2]

    Bias and unfairness in information retrieval systems: New challenges in the llm era,

    S. Dai, C. Xu, S. Xu, L. Pang, Z. Dong, and J. Xu, “Bias and unfairness in information retrieval systems: New challenges in the llm era,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6437–6447

    work page 2024

  3. [3]

    Art or artifice? large language models and the false promise of creativity,

    T. Chakrabarty, P. Laban, D. Agarwal, S. Muresan, and C.-S. Wu, “Art or artifice? large language models and the false promise of creativity,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–34

    work page 2024

  4. [4]

    Art and the science of generative ai,

    Z. Epstein, A. Hertzmann, I. of Human Creativity, M. Akten, H. Farid, J. Fjeld, M. R. Frank, M. Groh, L. Herman, N. Leach et al., “Art and the science of generative ai,” Science, vol. 380, no. 6650, pp. 1110–1111, 2023. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 9

    work page 2023

  5. [5]

    Comparing methods for large- scale agile software development: A systematic literature review,

    H. Edison, X. Wang, and K. Conboy, “Comparing methods for large- scale agile software development: A systematic literature review,” IEEE Transactions on Software Engineering , vol. 48, no. 8, pp. 2709–2731, 2021

    work page 2021

  6. [6]

    Hybrid intelligence,

    D. Dellermann, P. Ebel, M. S ¨ollner, and J. M. Leimeister, “Hybrid intelligence,” Business & Information Systems Engineering , vol. 61, no. 5, pp. 637–643, 2019

    work page 2019

  7. [7]

    Artificial intelligence, human intelligence and hybrid intelligence based on mutual augmentation,

    M. H. Jarrahi, C. Lutz, and G. Newlands, “Artificial intelligence, human intelligence and hybrid intelligence based on mutual augmentation,” Big Data & Society , vol. 9, no. 2, p. 20539517221142824, 2022

    work page 2022

  8. [8]

    Experimental evidence on the productivity effects of generative artificial intelligence,

    S. Noy and W. Zhang, “Experimental evidence on the productivity effects of generative artificial intelligence,” Science, vol. 381, no. 6654, pp. 187–192, 2023

    work page 2023

Show all 39 references
  1. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  2. [10]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proc. ACL, 2016, pp. 1715–1725

    2016

  3. [11]

    Googles neural machine translation system: Bridging the gap between human and machine translation,

    Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi, W. Macherey, M. Krikun, Y . Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, . Kaiser, S. Gouws, Y . Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rud...

    2016 arXiv

  4. [12]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  5. [13]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin et al. , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Transactions on Information Systems , vol. 43, no. 2, pp. 1–55, 2025

    2025

  6. [14]

    Study of the software development life cycle and the function of testing,

    S. Gupta and N. Gayathri, “Study of the software development life cycle and the function of testing,” in 2022 international interdisciplinary humanitarian conference for sustainability (iihc) . IEEE, 2022, pp. 1270–1275

    2022

  7. [15]

    Cohn, Succeeding with agile: software development using Scrum

    M. Cohn, Succeeding with agile: software development using Scrum . Pearson Education, 2010

    2010

  8. [16]

    Test automation pyramid from theory to practice,

    A. Contan, C. Dehelean, and L. Miclea, “Test automation pyramid from theory to practice,” in 2018 IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR) . IEEE, 2018, pp. 1–5

    2018

  9. [17]

    The testing mechanism for software and services based on mike cohn’s testing pyramid modification,

    V . Mukhin, Y . Kornaga, Y . Bazaka, I. Krylov, A. Barabash, A. Yakovl- eva, and O. Mukhin, “The testing mechanism for software and services based on mike cohn’s testing pyramid modification,” in 2021 11th IEEE International Conference on Intelligent Data Acquisition and Advan...

    2021

  10. [18]

    Toward successful devops: a decision-making frame- work,

    M. A. Akbar, S. Rafi, A. A. Alsanad, S. F. Qadri, A. Alsanad, and A. Alothaim, “Toward successful devops: a decision-making frame- work,” IEEE Access, vol. 10, pp. 51 343–51 362, 2022

    2022

  11. [19]

    Approach to automation of the initial stages of software design,

    Y . A. Orlova, “Approach to automation of the initial stages of software design,” in 2013 BRICS Congress on Computational Intelligence and 11th Brazilian Congress on Computational Intelligence . IEEE, 2013, pp. 375–380

    2013

  12. [20]

    H ¨uttermann, DevOps for developers

    M. H ¨uttermann, DevOps for developers . Apress, 2012

    2012

  13. [21]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2023

    2023

  14. [22]

    Evaluating instruction-tuned large language models on code comprehension and generation,

    Z. Yuan, J. Liu, Q. Zi, M. Liu, X. Peng, and Y . Lou, “Evaluating instruction-tuned large language models on code comprehension and generation,” arXiv preprint arXiv:2308.01240 , 2023

    2023 arXiv

  15. [23]

    Ontology driven software development for automated documentation

    M. P. S. Bhatia, A. Kumar, and R. Beniwal, “Ontology driven software development for automated documentation.” Webology, vol. 15, no. 2, 2018

    2018

  16. [24]

    Adopting devops in the real world: A theory, a model, and a case study,

    W. P. Luz, G. Pinto, and R. Bonif ´acio, “Adopting devops in the real world: A theory, a model, and a case study,” Journal of Systems and Software, vol. 157, p. 110384, 2019

    2019

  17. [25]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  18. [26]

    Testeval: Benchmarking large language models for test case generation,

    W. Wang, C. Yang, Z. Wang, Y . Huang, Z. Chu, D. Song, L. Zhang, A. R. Chen, and L. Ma, “Testeval: Benchmarking large language models for test case generation,” in Findings of the Association for Computational Linguistics: NAACL 2025 , 2025

    2025

  19. [27]

    Evidence-based methodological framework for machine learning studies,

    J. Walczak, “Evidence-based methodological framework for machine learning studies,” Authorea Preprints, 2025

    2025

  20. [28]

    Reforms: Consensus-based recommendations for machine-learning- based science,

    S. Kapoor, E. M. Cantrell, K. Peng, T. H. Pham, C. A. Bail, O. E. Gundersen, J. M. Hofman, J. Hullman, M. A. Lones, M. M. Malik et al., “Reforms: Consensus-based recommendations for machine-learning- based science,” Science Advances, vol. 10, no. 18, p. eadk3452, 2024

    2024

  21. [29]

    Unit testing in practice,

    M. Ellims, J. Bridges, and D. C. Ince, “Unit testing in practice,” in 15th International Symposium on Software Reliability Engineering . IEEE, 2004, pp. 3–13

    2004

  22. [30]

    No more manual tests? evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y . Lou, M. Liu, S. Ding, K. Wang, Y . Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” arXiv preprint arXiv:2305.04207 , 2023

    2023 arXiv

  23. [31]

    An empirical study of unit test generation with large language models,

    L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang et al. , “An empirical study of unit test generation with large language models,” arXiv preprint arXiv:2406.18181 , 2024

    2024 arXiv

  24. [32]

    A system for automated unit test generation using large language models and assessment of generated test suites,

    A. Lops, F. Narducci, A. Ragone, M. Trizio, and C. Bartolini, “A system for automated unit test generation using large language models and assessment of generated test suites,” in 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (IC...

    2025

  25. [33]

    Bidirectional symbolic analysis for effective branch testing,

    M. Baluda, G. Denaro, and M. Pezz `e, “Bidirectional symbolic analysis for effective branch testing,” IEEE Transactions on Software Engineer- ing, vol. 42, no. 5, pp. 403–426, 2015

    2015

  26. [34]

    Mutation-driven generation of unit tests and oracles,

    G. Fraser and A. Zeller, “Mutation-driven generation of unit tests and oracles,” in Proceedings of the 19th international symposium on Software testing and analysis , 2010, pp. 147–158

    2010

  27. [35]

    Performance regression unit testing: a case study,

    V . Hork `y, F. Haas, J. Kotr ˇc, M. Lacina, and P. T ˚uma, “Performance regression unit testing: a case study,” in Computer Performance Engi- neering: 10th European Workshop, EPEW 2013, Venice, Italy, September 16-17, 2013. Proceedings 10 . Springer, 2013, pp. 149–163

    2013

  28. [36]

    Utilizing performance unit tests to increase performance awareness,

    V . Hork `y, P. Libi ˇc, L. Marek, A. Steinhauser, and P. T ˚uma, “Utilizing performance unit tests to increase performance awareness,” in Proceed- ings of the 6th ACM/SPEC International Conference on Performance Engineering, 2015, pp. 289–300

    2015

  29. [37]

    Microsoft announces new copilot copyright commitment for customers,

    B. Smith and H. Nowbar, “Microsoft announces new copilot copyright commitment for customers,” Microsoft On the Issues Blog , 2023

    2023

  30. [38]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  31. [39]

    Mutmut: Python mutation testing tool,

    A. Hovmller, “Mutmut: Python mutation testing tool,” https://mutmut. readthedocs.io/, 2024, accessed: 2024-07-01. BIOGRAPHY SECTION Jakub Walczak defended his PhD in 2022, he is an AI researcher with both academic and industrial experience across a wide range of domains, inclu...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.