Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Tracking the Moving Target: A Framework for Continuous Evaluation of LLM Test Generation in Industry

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Commercial LLM test generators improved from a 32.44% weighted quality score in March 2024 to over 90% by December 2024, and a continuous evaluation framework is needed to track such fast-moving tools.

desk verdict The framework is a useful, honest industrial measurement contribution; the 'rapid evolution' headline is confounded with prompt refinement and model switching. read the letter →

arxiv 2504.18985 v1 pith:YL4KFMLD submitted 2025-04-26 cs.SE

Maider Azanza , Beatriz Pérez Lamancha , Eneko Pizarro This is my paper · ORCID
classification cs.SE
keywords LargeLanguageModelsTestGenerationContinuousEvaluationIndustrialCaseStudyDevSecOpsCodeCoveragePromptEngineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the rapid evolution of commercial LLM test generators makes point-in-time evaluations obsolete for industry, and that a continuous measurement framework can keep adoption decisions current. It builds such a framework around metrics that combine automated measurements (compilation success, static analysis, coverage, test isolation) with expert judgment (equivalence partitioning, boundary values, parameterization, and coverage of expert-written scenarios), then validates it in a nine-month longitudinal study at a mid-sized software consultancy. The weighted quality score rose from 32.44% for the first GPT-4 evaluation to over 90% for the best models by December 2024, while compilation errors per test suite dropped from 31 to near zero. The argument matters because DevSecOps teams must decide when to adopt these tools, which model to pick, and how much prompt investment is worthwhile, and any of those decisions based on a single snapshot will go stale within months.

What carries the argument

The load-bearing mechanism is a weighted scoring framework that converts heterogeneous quality signals into a single comparable number over time. Objective metrics—compilation errors (−20%), static analysis issues (−5%), setup/teardown usage (+10%), and line, branch/decision coverage with test isolation (+40% combined)—are weighted 50% against subjective black-box metrics: equivalence partitioning, boundary value analysis, test parameterization, and expert-generated test coverage (the other 50%). Around this sits an iterative prompt-engineering loop that refines the prompt after each evaluation cycle, so the pipeline measures the model-plus-prompt system rather than the model alone; reproducibility comes from a documented toolchain and a fixed set of seven test subjects with expert ground truth.

What would settle it

Re-run the full seven-function evaluation with the March 2024 prompt frozen, applying it to the December 2024 models. If the weighted scores stay near the March levels or rise only modestly, the improvement is mostly prompt engineering; if they still reach roughly the 90% range, the capability-evolution claim survives.

Watch

Extended reading notes

Core claim

The central claim is that LLM test generation capabilities improved dramatically during 2024, and that this improvement can be tracked and managed only with a measurement process designed for repeated evaluation. Using seven carefully selected functions from projects without public test suites to avoid data leakage, expert QA engineers built ground-truth tests, and the framework scored LLM-generated tests across code quality, white-box, and black-box metric categories. The longitudinal results show GitHub Copilot's GPT-4 producing an average of 31 compilation errors per suite in March 2024, falling to 3 by May 2024 after iterative prompt refinement, while newer models reached near-perfect compilation, line and branch coverage above 95%, test parameterization at 88–92%, and total weighted assessment scores above 90%. The authors interpret these results as evidence that the technology is maturing toward industrial viability, while still requiring expert oversight.

Load-bearing premise

The load-bearing premise is that the measured improvement over time reflects real evolution in LLM capabilities rather than the authors' own iterative prompt refinement; the paper itself flags this as a confound, and if it gives way, the headline 32% to 90% trend becomes a prompt-engineering artifact.

Editorial extensions

If this is right

  • Adoption decisions based on a single evaluation will routinely be wrong: a tool that looks unusable in March can clear a 90% weighted bar by December, so companies should schedule re-evaluations on the same timescale as model releases.
  • Prompt engineering is part of the measured system; GPT-4's score rose from 32.44% to 67.96% between March and May 2024 with the same underlying model, so organizations should track prompt versions as carefully as model versions.
  • The highest-scoring models reached expert-generated test coverage above 90%, suggesting generated tests can now capture most, though not all, of what expert testers write.
  • Cost and latency remain load-bearing: newer top-scoring models carry higher computational cost, and per-seat licensing matters for larger teams, so adoption is a portfolio decision rather than a pure quality ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would caution that the headline 32% to 90% rise should be read as the improvement of the combined model-plus-prompt system, not of the models alone; a clean separation would require a frozen-prompt longitudinal run.
  • The framework's equal weighting of objective and subjective metrics is an organizational choice, not a universal law; the authors make this explicit, and a company that cares more about maintainability than coverage would likely reorder the ranking by changing weights.
  • A natural extension is to run the same seven test subjects and prompts against locally hosted or in-house models, which would directly test the privacy-motivated deployment path the paper names as future work.
  • The seven-function selection means the 90% figure is a tracking instrument for one consultancy's priorities, not a general benchmark; other organizations should expect different absolute numbers but likely the same steep slope.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for continuously evaluating commercial LLM-based test generators in industrial settings, combining objective metrics (code quality, white-box coverage) with subjective expert-assessed black-box metrics into a weighted composite score. The framework is instantiated at LKS Next, where seven functions from internal projects are used as subjects, expert-written tests serve as ground truth, and prompts are iteratively refined. A longitudinal study from March 2024 to December 2024 reports a rise in the weighted score from 32.44% (ChatGPT-4, March) to over 90% (o1-Preview and Claude 3.5 Sonnet, December), which the paper interprets as evidence of rapid evolution of LLM test-generation capabilities. The paper also discusses industrial adoption factors such as cost, integration, data privacy, and expert oversight.

Significance. The framework addresses a real, under-served need: industrial practitioners lack systematic, repeatable methods for re-evaluating LLM test generators as models and tools change. The paper's strengths include a concrete metric taxonomy, integration with widely used tools (SonarQube, JaCoCo, JUnit, Maven), a replication package, and an unusually candid threats-to-validity section that acknowledges prompt-refinement confounding. If the methodological issues below are resolved, the framework could serve as a template for other organizations and as a baseline for future longitudinal studies. The paper also adds useful qualitative evidence on practical adoption hurdles, such as the need for prompt engineering, training, and expert oversight. However, as written, the central empirical claim of 'rapid evolution of LLM capabilities' is not cleanly supported by the data, because the observed changes conflate three variables: calendar time, model identity, and the authors' own iterative prompt refinement.

major comments (4)
  1. [§6, Table 5; §5.3–5.4; §8 Internal Validity] The paper's headline result—scores rising from 32.44% to over 90%—is presented as evidence of 'rapid evolution of LLM capabilities,' but the comparison conflates prompt refinement with model evolution. The March-to-May improvement for ChatGPT-4 (32.44% to 67.96%) occurred during the exact period in which §5.3 and §5.4 describe iterative prompt refinement, which added six new instruction areas, including an explicit mandate to 'need a 100% of line, condition and condition/decision coverage.' The Section 8 mitigation that 'maintain consistent base prompts' is difficult to reconcile with §5.4's detailed prompt evolution. Please re-analyze the data separating prompt version from model version, for example by running the March prompt on the May model and the May prompt on the March model, or by reporting the prompt version used for each column of Table 5. Without such a separation, the improvement cannot be attributed to LLM capability evolution.
  2. [§6, Table 5; §10] The December 'over 90%' rows come from different models (o1-Preview, Claude 3.5 Sonnet), not from the same model tracked over time. The abstract and conclusions state that 'scores rising from 32.44% ... to over 90%' demonstrate 'the rapid evolution of LLM capabilities,' but this trajectory mixes model identity, calendar time, and prompt version. The data are equally consistent with the interpretation that newer models simply perform better at this task, or that the authors' prompt engineering transferred to new models. If the claim is about the ecosystem evolving, the text should say so explicitly and not imply that a single model's capabilities improved. Please either restrict the longitudinal claim to same-model comparisons (e.g., ChatGPT-4 March vs. ChatGPT-4 May) or reframe the conclusion as an evaluation of the current state of multiple commercial models.
  3. [§4.4, Table 4] The scoring formulas for penalty metrics CE and SAI normalize by MAX(CE value for all LLMs) and MAX(SAI value for all LLMs). This makes each LLM's score depend on the cohort of models included in the evaluation, not on an absolute standard. When a new model with worse compilation errors is added to the comparison, the MAX increases and every other model's penalty decreases, so the composite scores from different evaluation waves are not directly comparable. Since Table 5 compares March, May, and December evaluations with different model cohorts, the reported total scores are not on a common scale. Please either fix the normalization to an absolute reference (e.g., a fixed maximum or the expert baseline) or demonstrate that the conclusions are robust to the normalization choice. This is load-bearing for the longitudinal claim.
  4. [§8 Internal Validity; §6] The paper reports a single value per metric per model per time point, with no repeated runs, no variance measures, and no statistical tests. Section 8 states that 'validate significant changes in performance through multiple test runs' is a mitigation, but the results section reports no such replications, and the methodology section does not specify the number of runs or how sampling was handled. Given the known nondeterminism of LLM generation, a single run per configuration is insufficient to support the precise percentage differences (e.g., 98.00% vs. 95.71% line coverage, differences of 2–6 percentage points in several black-box metrics). Please add replication details and error bars, or explicitly state the run counts and any observed variability; if only one run was performed, temper the precision of the claims accordingly.
minor comments (5)
  1. [§3] The word 'reseaarch' is a typo; it should read 'research.'
  2. [§4.4] The abbreviation 'BV A' appears with an extra space; it should be 'BVA.' The same issue recurs in the list of black-box metrics.
  3. [§1, §6] The paper variously refers to 'GitHub Copilot' and 'ChatGPT-4' as the subject of the early evaluations. Clarify the relationship: if GitHub Copilot used GPT-4 in March 2024, state explicitly how the 'ChatGPT-4' rows of Table 5 were obtained, and whether the same interface (Copilot vs. direct API/chat) was used across all evaluations.
  4. [§10 vs. §6] The conclusions state the study ran 'From April 2024 to January 2025,' but the results section and Table 5 describe evaluations from March 2024 to December 2024. Please reconcile the reported time frame.
  5. [Appendix/References] The ACM reference template lines 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' are left in the manuscript and are irrelevant to the study; they should be removed or replaced with actual submission dates.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline 32.44%-to-90% trajectory is partly self-referential: the prompt was iteratively tuned against the same metrics and the same seven functions used in the final evaluation, so the reported 'evolution' conflates prompt fitting with model improvement.

  1. fitted input called prediction [Sections 5.3-5.4 (Prompt Engineering and Iterative Evaluation) and Section 6 (Results, Table 5)]
    "We mandated the use of the white box method, focusing on condition/decision coverage... 'I need a 100% of line, condition and condition/decision coverage'... 'This cycle continued until the generated tests met our quality standards'... 'scores rising from 32.44% in our initial evaluation to over 90% for the best performing models.'"

    The final prompt was obtained by iteratively refining it until the generated tests satisfied the exact metrics (line/condition/decision coverage, SonarQube static analysis, parameterization, etc.) on the same seven functions used for the final evaluation. The March-to-May ChatGPT-4 improvement (32.44% to 67.96%) therefore largely measures the success of prompt fitting on the evaluation set, not an independent capability evolution; the December 'over 90%' values additionally come from different models rather than a single tracked model.

full rationale

The paper proposes a measurement framework, and as a framework it is not circular: metrics are defined through standard tools (JaCoCo, SonarQube), expert-written ground-truth tests are external to the LLM, and the industrial context is real. No load-bearing self-citations or imported uniqueness theorems appear. The circularity is in the interpretation of Table 5 as evidence of LLM capability evolution. Because the same seven functions served both as the prompt-tuning target and as the evaluation benchmark, the improvement from 32.44% to about 68% for ChatGPT-4 is partly constructed by the methodology; the final prompt explicitly demands 100% line/condition/decision coverage and SonarQube cleanliness, which are precisely the measured outcomes. The December comparison is cross-sectional (o1-Preview, Claude 3.5 Sonnet, etc.) and not a continuation of the same model, so the 'over 90%' rows cannot be read as longitudinal evolution either. The authors themselves flag the confound in Section 8: 'our iterative refinement of prompting strategies might confound the results.' The framework's practical value—continuous evaluation infrastructure, tool integration, expert assessment—remains independent; only the evolutionary conclusion is partially circular. Score 6 reflects that one or more headline 'predictions' reduce, at least in part, to the prompt-optimization loop.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The ledger shows the central claims rely on hand-picked weights, expert-defined ground truth, tool-based quality proxies, and a separability assumption between prompt engineering and model evolution. No new physical or theoretical entities are introduced.

free parameters (2)
  • Weighting scheme for composite score = CE -20%, SAI -5%, STU +10%, white-box metrics 40%, black-box metrics 50%
    Table 4 defines the weights; the paper states they reflect LKS Next's specific priorities, not derived from data or theory.
  • MAX normalization for penalty metrics = Maximum CE and SAI values across the evaluated LLMs (e.g., 31 and 45 in the reported set)
    The CE and SAI penalty formulas divide by MAX(CE value for all LLMs) and MAX(SAI value for all LLMs), making the penalties dependent on the specific set of models evaluated.
assumptions (5)
  • domain assumption The seven selected functions are representative of industrial code patterns and testing challenges.
    Section 5.2 states the functions were 'deliberately selected' to cover common enterprise patterns, but no external validation or sampling frame is provided.
  • domain assumption Expert-written ground truth tests from LKS Next QA engineers are a valid, unbiased benchmark.
    Section 5.2 describes expert-created tests validated by a QA team leader, but the experts come from the same organization that prioritizes the evaluation criteria.
  • domain assumption SonarQube Sonarway profile and JaCoCo metrics accurately reflect test quality and thoroughness.
    Sections 4.1 and 4.2 use these tools as objective measures of maintainability, security, and coverage; the paper does not justify their validity for generated test code beyond citing standard practice.
  • domain assumption LLM outputs are stable enough that single-run measurements represent model performance.
    Table 5 reports single point estimates for each model and time point, with no repeated runs, variance, or confidence intervals, although LLM generation is stochastic.
  • domain assumption Score changes over time are primarily attributable to LLM capability evolution rather than prompt changes.
    Section 8 (Internal Validity) admits that 'iterative refinement of prompting strategies might confound the results'; the paper assumes its mitigation measures suffice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tracking the Moving Target: A Framework for Continuous Evaluation of LLM Test Generation in Industry." pith.science (2026). https://pith.science/paper/YL4KFMLD

@misc{pith2026250418985,
  author       = {Pith},
  title        = {Pith review of: Tracking the Moving Target: A Framework for Continuous Evaluation of LLM Test Generation in Industry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YL4KFMLD}},
  note         = {Machine review of arXiv:2504.18985}
}
read the original abstract

Large Language Models (LLMs) have shown great potential in automating software testing tasks, including test generation. However, their rapid evolution poses a critical challenge for companies implementing DevSecOps - evaluations of their effectiveness quickly become outdated, making it difficult to assess their reliability for production use. While academic research has extensively studied LLM-based test generation, evaluations typically provide point-in-time analyses using academic benchmarks. Such evaluations do not address the practical needs of companies who must continuously assess tool reliability and integration with existing development practices. This work presents a measurement framework for the continuous evaluation of commercial LLM test generators in industrial environments. We demonstrate its effectiveness through a longitudinal study at LKS Next. The framework integrates with industry-standard tools like SonarQube and provides metrics that evaluate both technical adequacy (e.g., test coverage) and practical considerations (e.g., maintainability or expert assessment). Our methodology incorporates strategies for test case selection, prompt engineering, and measurement infrastructure, addressing challenges such as data leakage and reproducibility. Results highlight both the rapid evolution of LLM capabilities and critical factors for successful industrial adoption, offering practical guidance for companies seeking to integrate these technologies into their development pipelines.

Figures

Figures reproduced from arXiv: 2504.18985 by the authors.

Figure 1
Figure 1. BPMN diagram showcasing the iterative measurement process. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Barr, E., Harman, M., McMinn, P., Shahbaz, M., and Yoo, S. (2015). The oracle problem in software testing: A survey. IEEE transactions on software engineering , 41(5):507--525

    work page 2015

  2. [2]

    Beller, M., Gousios, G., and Zaidman, A. (2019). Developer testing in the ide: Patterns, beliefs, and behavior. In IEEE Transactions on Software Engineering , volume 45, pages 261--284

    work page 2019

  3. [3]

    and Papapetrou, P

    Campbell, A. and Papapetrou, P. (2013). SonarQube in action . Manning Publications Co

    work page 2013

  4. [4]

    Chen, Y., Hu, Z., Zhi, C., Han, J., Deng, S., and Yin, J. (2024). Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE) . ACM

    work page 2024

  5. [5]

    Fan, Z. (2019). A systematic evaluation of problematic tests generated by evosuite. In 2019 IEEE/ACM International Conference on Software Engineering Companion Proceedings , pages 165--166

    work page 2019

  6. [6]

    Freeman, S., Mackinnon, T., Pryce, N., and Walnes, J. (2004). j M ock: supporting responsibility-based design with mock objects. In Companion to the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA . ACM

    work page 2004

  7. [7]

    Gao, C., Hu, X., Gao, S., Xia, X., and Jin, Z. (2025). The current challenges of software engineering in the era of large language models. ACM Trans. Softw. Eng. Methodol. Just Accepted

    work page 2025

  8. [8]

    Garousi, V., Felderer, M., and Hacalo g lu, T. (2017). Software test maturity assessment and test process improvement: A multivocal literature review. Information and Software Technology , 85:16--42

    work page 2017

Show all 33 references
  1. [9]

    and Zhi, J

    Garousi, V. and Zhi, J. (2016). Challenges, benefits, and best practices of software testing in practice: Interviews with practitioners. Journal of Software: Evolution and Process , 28(11):1018--1026

    2016

  2. [10]

    Washizaki, e

    H. Washizaki, e. (2024). Guide to the Software Engineering Body of Knowledge (SWEBOK Guide) . IEEE Computer Society

    2024

  3. [11]

    Hilton, M., Tunnell, T., Huang, K., Marinov, D., and Dig, D. (2016). Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering , ASE '16, page 426–437, New York, ...

    2016

  4. [12]

    Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., and Wang, H. (2024). Large language models for software engineering: A systematic literature review. ACM Trans. Softw. Eng. Methodol. , 33(8)

    2024

  5. [13]

    IEEE/ISO/IEC international standard - software and systems engineering--software testing--part 4: Test techniques

    ISO (2021). IEEE/ISO/IEC international standard - software and systems engineering--software testing--part 4: Test techniques. ISO/IEC/IEEE 29119-4:2021(E)

    2021

  6. [14]

    Systems and software engineering -- systems and software quality requirements and evaluation (square) -- system and software quality models

    ISO/IEC (2011). Systems and software engineering -- systems and software quality requirements and evaluation (square) -- system and software quality models

    2011

  7. [15]

    Jiang, W., Gao, X., Zhai, J., Ma, S., Zhang, X., and Shen, C. (2024). From effectiveness to efficiency: Comparative evaluation of code generated by lcgms for bilingual programming questions

    2024

  8. [16]

    J., Tsantalis, N., Chen, T.-H., and Yang, J

    Kim, D. J., Tsantalis, N., Chen, T.-H., and Yang, J. (2021). Studying test annotation maintenance in the wild. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , pages 62--73

    2021

  9. [17]

    Kim, G., Humble, J., Debois, P., and Willis, J. (2016). The DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations . IT Revolution Press

    2016

  10. [18]

    S., Petrovic, J

    Kracht, J. S., Petrovic, J. Z., and Walcott-Justice, K. R. (2014). Empirically evaluating the quality of automatically generated and manually written test suites. 14th International Conference on Quality Software , pages 256--265

    2014

  11. [19]

    Liu, H., Liu, L., Yue, C., Wang, Y., and Deng, B. (2024). Autotestgpt: A system for the automated generation of software test cases based on chatgpt. Journal of Software , 19(4)

    2024

  12. [20]

    López, J. A. H., Chen, B., Saad, M., Sharma, T., and Varró, D. (2025). On inter-dataset code duplication and data leakage in large language models. IEEE Transactions on Software Engineering , 51(1)

    2025

  13. [21]

    and Rothermel, G

    Orso, A. and Rothermel, G. (2014). Software testing: a research travelogue (2000--2014). Future of Software Engineering Proceedings , pages 117--132

    2014

  14. [22]

    and Pereira, R

    Prates, L. and Pereira, R. (2025). Devsecops practices and tools. International Journal of Information Security , 24(1):1--25

    2025

  15. [23]

    Schäfer, M., Nadi, S., Eghbali, A., and Tip, F. (2024). An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering , 50(1):85--105

    2024

  16. [24]

    Sergeyuk, A., Golubev, Y., Bryksin, T., and Ahmed, I. (2025). Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward. Information and Software Technology , 178

    2025

  17. [25]

    Siddiq, M., Da Silva, J., Tanvir, R., Ulfat, N., Al Rifat, F., and Carvalho, V. (2024). Using large language models to generate junit tests: An empirical study. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE) . ACM

    2024

  18. [26]

    F., Bruntink, M., and Bacchelli, A

    Spadini, D., Aniche, M. F., Bruntink, M., and Bacchelli, A. (2019). Mock objects for testing java systems - why and how developers use them, and how they evolve. Empir. Softw. Eng. , 24(3):1461--1498

    2019

  19. [27]

    Tang, Y., Liu, Z., Zhou, Z., and Luo, X. (2024). Chatgpt vs sbst: A comparative assessment of unit test suite generation. IEEE Transactions on Software Engineering , 50(6):1340--1359

    2024

  20. [28]

    and Schulte, W

    Tillmann, N. and Schulte, W. (2005). Parameterized unit tests. In Proceedings of the 10th European Software Engineering Conference . ACM

    2005

  21. [29]

    Wagner, S., Goeb, A., Heinemann, L., Kläs, M., Lampasona, C., Lochmann, K., Mayr, A., Plösch, R., Seidl, A., Streit, J., et al. (2012). Software quality models in practice: A systematic literature review. Information and Software Technology , 54(6):622--634

    2012

  22. [30]

    Wang, C., Li, Z., Gao, C., Wang, W., Peng, T., Huang, H., Deng, Y., Wang, S., and Lyu, M. (2024a). Exploring multi-lingual bias of large code models in code generation

    2024

  23. [31]

    Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., and Wang, Q. (2024b). Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering , 50(4):911--936

    2024

  24. [32]

    Wu, T., Terry, M., and Cai, C. (2022). Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . ACM

    2022

  25. [33]

    M., and Liu, Y

    Wu, Y., Li, Z., Zhang, J. M., and Liu, Y. (2024). Condefects: A complementary dataset to address the data leakage concern for llm-based fault localization and program repair. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engin...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.