Pith. sign in

REVIEW 4 major objections 6 minor 61 references

Combining TSL and LLM to Automate REST API Testing: A Comparative Study

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a pipeline combining Test Specification Language (TSL) with decomposed few-shot prompting lets general-purpose LLMs generate executable REST API integration tests from OpenAPI specifications, and that Claude 3.5…

desk verdict The paper is a useful, reproducible comparison of eight LLMs on REST API test generation, but it never tests whether the TSL intermediate step is actually the thing that helps. read the letter →

arxiv 2509.05540 v1 pith:TTNGLBTC submitted 2025-09-05 cs.SE cs.AI

Thiago Barradas , Aline Paes , Vânia de Oliveira Neves This is my paper · ORCID
classification cs.SEcs.AI
keywords RESTAPItestingintegrationtestgenerationLargeLanguageModelsSpecification(TSL)Openpromptengineeringmutation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces RestTSLLM, a prompt-driven pipeline that turns REST API documentation into executable integration tests without fine-tuning any model. Its key move is to split the work into two steps with Test Specification Language (TSL) in between: first the LLM converts the OpenAPI specification into declarative test scenarios, then it converts those scenarios into xUnit (.NET) test code. The authors run this pipeline on six open-source .NET REST APIs with eight general-purpose LLMs and score the results by success rate, branch coverage, and mutation score. They report that every model produced mostly executable tests, with Claude 3.5 Sonnet ranking first on all metrics and being the only model with zero failed tests. The point of the study is to show that scenario design and input generation—historically the hard parts of integration testing—can be automated cheaply through prompting alone.

What carries the argument

The central machinery is a two-stage decomposed prompting chain with Test Specification Language (TSL) as the bridge. TSL is a high-level, declarative YAML-style format for test cases: each case declares an id, group, name, endpoint, method, preconditions, request body, and expected response. The pipeline uses four prompts: a behavior prompt that sets the model as an experienced tester; an example showing OpenAPI converted to TSL; an example showing TSL converted to xUnit code; and action prompts that run the same two conversions on the target specification. Because TSL expresses scenarios without code, the first conversion isolates business-rule understanding while the second isolates syntax and framework concerns—the decomposition is what lets general-purpose models produce coherent output. When models truncate long responses, a loop segments generation by OpenAPI tags carried into the TSL, preserving completeness of the test suite.

What would settle it

Repeat the exact RestTSLLM pipeline on a held-out set of large industrial REST APIs—with hundreds of endpoints, asynchronous flows, and strict authentication—using the same eight models and prompts. If Claude 3.5 Sonnet no longer ranks first, or if the best model's calculated score drops far below the roughly 71% seen here because generated tests frequently fail or fail to compile, then the paper's central generalization is falsified.

Watch

Extended reading notes

Core claim

RestTSLLM's central claim is that a general-purpose LLM, given a well-structured prompt plus a few worked examples, can perform the whole REST API integration-testing chain: reading OpenAPI, deciding what scenarios matter (happy paths, validation failures, edge cases), generating valid input data, and emitting compilable xUnit tests. The intermediate TSL representation is not decorative: it forces the model to settle business logic before syntax, and the examples teach the model to preserve TSL-defined inputs while making small adjustments, such as generating a unique email dynamically to keep tests repeatable. In the evaluation on six small open-source .NET projects, all eight models achieved success rates above 95%, and the calculated score—an equal-weight TOPSIS combination of success rate, branch coverage, and mutation score—placed Claude 3.5 Sonnet first at 70.9%, followed by Deepseek R1, Qwen 2.5 32b, and Sabiá 3, with all models within 8.7 percentage points. The paper also reports that only 2.38% of the 1,635 generated tests failed, with most failures traced to property-length boundary errors and misreadings of the specification.

Load-bearing premise

The results rest on the assumption that six small, simple, open-source .NET example projects fairly represent the REST APIs that teams actually test in industry, so the same success rates and model ranking would hold for large enterprise systems.

Editorial extensions

If this is right

  • If the central claim holds, teams can generate a first-pass integration test suite from an OpenAPI file with no model fine-tuning and no per-API training data.
  • The TSL-bridge result implies test scenario design can be decoupled from test code generation, so teams could swap xUnit for another framework by changing only the second prompt.
  • Because costs per project stayed below $0.80 even for the most expensive model, per-regression automation with LLMs is economically plausible.
  • The six recurring failure categories give a concrete repair agenda: boundary-value property lengths, authentication handling, required-field semantics, and response deserialization, which together account for most of the 39 failed tests.
  • The comparison suggests all eight evaluated models are viable for this task, with Claude 3.5 Sonnet the current best option according to the paper's calculated score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's failure taxonomy suggests a direct ablation: add explicit few-shot examples for property-length boundaries, required versus optional fields, and authentication errors, and the 39 failed tests should shrink; this is an extension because the paper does not run that ablation.
  • Because results were captured on a single date with temperature set to 1, the leaderboard is version-sensitive; a fair inference is that the durable contribution is the pipeline itself, not the specific model version that won.
  • The gap between near-universal success rates and much lower mutation scores hints that specification-driven tests mostly check the documented contract rather than implementation edges, so adding implementation-aware oracles may improve fault detection more than generating additional test cases.
  • The prompts were primarily written in Portuguese while the OpenAPI specifications stayed in English, so an obvious testable extension is whether the TSL-bridge benefit persists for other languages and for specifications with heavier natural-language business rules.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces RestTSLLM, an approach that uses Test Specification Language (TSL) as an intermediate representation together with decomposed few-shot prompting to generate executable xUnit integration tests for REST APIs from OpenAPI specifications. The study evaluates eight LLMs on six small open-source .NET projects, measuring success rate, branch coverage, mutation score, and a "calculated score" that the authors label as TOPSIS. The authors report that Claude 3.5 Sonnet performs best across all metrics, followed by Deepseek R1, Qwen 2.5 32b, and Sabiá 3, and conclude that general-purpose LLMs combined with TSL can generate contextually coherent REST API integration tests.

Significance. The paper provides a reproducible empirical comparison of eight LLMs for REST API integration test generation, with public artifacts, an automated multi-model execution script, and a clear failure categorization. If the TSL-specific claim were supported, the decomposed-prompting design would be a useful methodological contribution to LLM-based test generation. The study also offers useful evidence on the feasibility of black-box test generation from OpenAPI specifications. However, the central attribution of the results to TSL is not empirically tested, and several methodological choices (single runs, no statistical significance testing, no external baselines, and a mislabeled aggregation method) substantially reduce the confidence in the model ranking and in the generalizability claims.

major comments (4)
  1. [Section 3; Section 4; Table 3] The paper never empirically isolates the TSL component. Every experimental condition runs the full OpenAPI-to-TSL-to-xUnit pipeline, so the reported success rate, coverage, and mutation score reflect the joint effect of the intermediate language, the decomposed/few-shot prompting, and the LLM's code-generation ability. A direct OpenAPI-to-xUnit condition, or a condition with an alternative intermediate representation, is required to support the claim that TSL 'improves the LLM's effectiveness' (Section 3) and the title's emphasis on combining TSL with LLMs. As presented, the role of TSL is a design hypothesis rather than an empirical finding.
  2. [Section 4.6; Section 5.2; Table 3] Each model/project combination appears to have been executed only once, with temperature set to 1, as acknowledged in Section 6 under 'Result Randomness', and no significance tests or confidence intervals are reported. The differences among models in Table 3 are small (overall scores from 62.2% to 70.9%, with several adjacent models within 1-2 points), so the ranking of models, including the claim that Claude 3.5 Sonnet 'outperformed all other models across every metric', may be within run-to-run noise. Repeated runs, variance reporting, or statistical testing are needed before RQ2 can be answered as stated.
  3. [Section 4.3, Eq. (S)] The formula labeled TOPSIS is, by the authors' own description, a weighted arithmetic mean of the three metrics with equal weights w=33.33%. TOPSIS requires computing distances to the positive-ideal and negative-ideal solutions after normalization; the presented formula is not TOPSIS. This mislabel matters because the 'calculated score' is the basis for the final model ranking in Table 3. The authors should either implement actual TOPSIS (including normalization and ideal/anti-ideal distance computation) or rename the aggregation as a weighted average.
  4. [Section 2; Section 5] The evaluation includes no baseline comparison against established REST API testing tools such as RESTler, RESTTestGen, bBOXRT, or RESTest, which are cited in the related work. Without such a baseline, the significance of the reported success, coverage, and mutation values cannot be assessed, and the suggestion that the approach compares favorably with 'traditional black-box tools' (Section 2) is not substantiated by the experiments. The paper should either add such a comparison or explicitly restrict its claims to a within-LLM comparative study.
minor comments (6)
  1. [Section 4.3] The word 'denotated' should be 'denoted'.
  2. [Abstract and Table 1] The spelling of the model name is inconsistent between 'Sabiá' and the unaccented 'Sabia'; please standardize.
  3. [Table 3] The table uses decimal commas (e.g., '70,9%') while the text uses decimal points; standardize the decimal format throughout.
  4. [Table 3] The column 'T' reports average numbers of tests with fractional values (e.g., 38.3). Clarify that this is an average across projects and, if possible, also report integer total counts per model.
  5. [Section 4.6] The statement that tests were 'manually copied' into duplicated projects should be elaborated, since this manual step is a potential source of experimental error and could affect reproducibility.
  6. [Section 3, Listings 1 and 2] The listings are not explicitly referenced in the running text; add references such as 'as shown in Listing 1'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation metrics are external to the method and the derivation chain does not reduce to its inputs.

full rationale

The paper makes no fitted-parameter claim and contains no equation in which a predicted quantity is defined from the data it claims to predict. RestTSLLM prompts an LLM to convert OpenAPI specifications into TSL scenarios and then into xUnit tests; the outcomes (success rate, branch coverage, mutation score) are obtained by compiling and executing the generated tests against six external .NET projects, so the measured values are not constructed from the prompt inputs. The 'Calculated Score' is an explicit weighted average of those three metrics, which is an aggregation rule rather than a hidden fit. The central comparison across eight LLMs is empirically evaluated on external projects and artifacts, with results reported in Table 3. The only self-citations, such as Mendoza et al. [31] in related work, are contextual and not load-bearing; no uniqueness theorem or methodological premise is imported from the authors' prior work. The paper explicitly acknowledges prompt dependency, project-context limitations, and subjectivity of the qualitative RQ1 review in Section 6, which further indicates that those aspects are treated as threats rather than as derived predictions. The absence of an OpenAPI-to-xUnit control condition means the specific contribution of the TSL intermediate representation is not isolated experimentally, but that is a validity or ablation gap, not circularity: the reported metrics would remain external evidence even if the TSL step were ineffective. No circular step satisfying the quoted-reduction standard was found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The approach introduces no free parameters and no speculative entities. Its load-bearing assumptions are domain-level choices about the sufficiency of OpenAPI, the validity of the chosen metrics, the representativeness of the selected projects, the stability of single-run LLM sampling, and the reliability of manual review.

assumptions (5)
  • domain assumption OpenAPI specifications provide sufficient information to generate meaningful integration test scenarios and input data.
    The entire method takes OpenAPI as the sole source of behavioral knowledge. Entered in Section 1 and Section 4.2 where projects are required to have an OpenAPI specification.
  • domain assumption Branch coverage and mutation score are valid and sufficient proxies for integration test effectiveness.
    Used in Section 4.3 as the evaluation criteria; the paper cites literature but does not justify that these metrics capture business-logic alignment for integration tests.
  • domain assumption The six selected .NET open-source projects are representative of typical REST APIs for the purpose of ranking LLMs.
    Selection criteria in Section 4.2 limit to small, simple, English-documented projects, which the authors acknowledge in Section 6 as a limitation.
  • domain assumption LLM outputs at temperature 1 with a fixed seed are stable enough to support a single-run comparative ranking.
    Section 6 states temperature 1 introduces stochasticity and that a fixed seed was used where supported; the final metrics come from one execution.
  • domain assumption The manual qualitative review by the first author is a reliable measure of business-logic alignment.
    Section 4.3 describes the review as subjective; Section 6 acknowledges it may vary if performed by other reviewers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combining TSL and LLM to Automate REST API Testing: A Comparative Study." pith.science (2026). https://pith.science/paper/TTNGLBTC

@misc{pith2026250905540,
  author       = {Pith},
  title        = {Pith review of: Combining TSL and LLM to Automate REST API Testing: A Comparative Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTNGLBTC}},
  note         = {Machine review of arXiv:2509.05540}
}
read the original abstract

The effective execution of tests for REST APIs remains a considerable challenge for development teams, driven by the inherent complexity of distributed systems, the multitude of possible scenarios, and the limited time available for test design. Exhaustive testing of all input combinations is impractical, often resulting in undetected failures, high manual effort, and limited test coverage. To address these issues, we introduce RestTSLLM, an approach that uses Test Specification Language (TSL) in conjunction with Large Language Models (LLMs) to automate the generation of test cases for REST APIs. The approach targets two core challenges: the creation of test scenarios and the definition of appropriate input data. The proposed solution integrates prompt engineering techniques with an automated pipeline to evaluate various LLMs on their ability to generate tests from OpenAPI specifications. The evaluation focused on metrics such as success rate, test coverage, and mutation score, enabling a systematic comparison of model performance. The results indicate that the best-performing LLMs - Claude 3.5 Sonnet (Anthropic), Deepseek R1 (Deepseek), Qwen 2.5 32b (Alibaba), and Sabia 3 (Maritaca) - consistently produced robust and contextually coherent REST API tests. Among them, Claude 3.5 Sonnet outperformed all other models across every metric, emerging in this study as the most suitable model for this task. These findings highlight the potential of LLMs to automate the generation of tests based on API specifications.

Figures

Figures reproduced from arXiv: 2509.05540 by the authors.

Figure 1
Figure 1. Overview of RestTSLLM approach The proposed prompt engineering is organized into three main stages: (1) Behavior – defining the behavior of the LLM, (2) Exam￾ples – presenting examples, and finally, (3) Action – requesting the execution of the task. The diagram in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Experimental methodology flow specification, were relevant on GitHub, had at least one integration with a database or external service, and developed using the target technology of the experiment (Subsection 4.2). The third step, Definition of Evaluation Criteria, involved defining the set of metrics to assess the effectiveness of the integra￾tion tests generated by the LLM tools. For this purpose, we select the suc… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 41 canonical work pages

  1. [1]

    Hugo Abonizio, Thales Sales Almeida, Thiago Laitz, Roseval Malaquias Junior, Giovana Kerche Bonás, Rodrigo Nogueira, and Ramon Pires. 2024. Sabiá-3 Technical Report. https://arxiv.org/abs/2410.12049

    arXiv 2024

  2. [2]

    Nadia Alshahwan, Mark Harman, and Alexandru Marginean. 2023. Software Testing Research Challenges: An Industrial Perspective. In2023 IEEE Confer- ence on Software Testing, Verification and Validation. https://doi.org/10.1109/ ICST57152.2023.00008

    arXiv 2023

  3. [3]

    Abhineet Anand and Azeem Uddin. 2019. Importance of software testing in the process of software development.International Journal for Scientfic Research and Development (IJSRD)(2019). https://www.researchgate.net/publication/ 331223692

    work page 2019

  4. [4]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. REST-ler: Stateful REST API Fuzzing. InProceedings of the 41st International Conference on Software Engineering (ICSE). IEEE Press. https://doi.org/10.1109/ICSE.2019.00083

    arXiv 2019

  5. [5]

    Omer Aydin, Enis Karaarslan, Fatih Safa Erenay, and Nebojsa Bacanin. 2025. Generative AI in Academic Writing: A Comparison of DeepSeek, Qwen, ChatGPT, Gemini, Llama, Mistral, and Gemma. https://arxiv.org/abs/2503.04765 Combining TSL and LLM to Automate REST API Testing: A Comparative Study SBES ’25, September 22–26, 2025, Recife, PE

    arXiv 2025

  6. [6]

    Thiago Barradas. 2025. Integration Test Generation - LLM Efficiency - Repository. https://github.com/uffsoftwaretesting/RestTSLLM Accessed on: June 26, 2025

    work page 2025

  7. [7]

    Lenz Belzner, Thomas Gabor, and Martin Wirsing. 2023. Large Language Model Assisted Software Engineering: Prospects, Challenges, and a Case Study. In AISoLA. https://doi.org/10.1007/978-3-031-46002-9_23

    work page doi:10.1007/978-3-031-46002-9_23 2023

  8. [8]

    BotPress. 2025. Discover models by Popularity. https://botpress.com/llm-ranking Accessed on: January 25, 2025

    work page 2025

Show all 61 references
  1. [9]

    Mohamed Boukhlif, Nassim Kharmoum, and Mohamed Hanine. 2024. LLMS for intelligent software testing: a comparative study. InProceedings of the 7th International Conference on Networking, Intelligent Systems and Security. https: //dl.acm.org/doi/10.1145/3659677.3659749

    2024

  2. [10]

    Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Empirical Comparison of Black-box Test Case Generation Tools for RESTful APIs. https://arxiv.org/abs/2108.08196

    2021 arXiv

  3. [11]

    Corradini, A

    D. Corradini, A. Zampieri, M. Pasqua, and M. Ceccato. 2021. Restats: A Test Coverage Tool for RESTful APIs. In37th IEEE International Conference on Software Maintenance and Evolution (ICSME). https://arxiv.org/abs/2108.08209

    2021 arXiv

  4. [12]

    Hasan Erdal. 2025. Shortener API - Repository. https://github.com/Filiphasan/ dotnet-minify-url Accessed on: March 24, 2025

    2025

  5. [13]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large Language Models for Software En- gineering: Survey and Open Problems. In2023 IEEE/ACM International Con- ference on Software Engineering: Future of Software Engine...

    2023 arXiv

  6. [14]

    2023.Dot Net Development Service Market Size, Share, Growth

    Fortune Business Insights. 2023.Dot Net Development Service Market Size, Share, Growth. https://www.fortunebusinessinsights.com/dot-net-development- service-market-111361 Accessed: January 12, 2025

    2023

  7. [15]

    David Fowler. 2025. Todo API - Repository. https://github.com/davidfowl/ TodoApp Accessed on: March 24, 2025

    2025

  8. [16]

    Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2022. Testing RESTful APIs: A Survey. https://arxiv.org/abs/2212.14604

    2022 arXiv

  9. [17]

    Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2023. Testing RESTful APIs: A Survey.ACM Trans. Softw. Eng. Methodol.(2023). https://doi.org/10.1145/ 3617175

    2023

  10. [18]

    Evandro Gomes. 2025. Supermarket API - Repository. https://github.com/ evgomes/supermarket-api Accessed on: March 24, 2025

    2025

  11. [19]

    Desta Haileselassie Hagos, Rick Battle, and Danda B. Rawat. 2024. Recent Ad- vances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives. https://arxiv.org/abs/2407.14962

    2024 arXiv

  12. [20]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  13. [21]

    This is Definition. 2025. What’s the most popular LLM? https://www. thisisdefinition.com/insights/most-popular-llm Accessed on: January 20, 2025

    2025

  14. [22]

    Kush Jain, Goutamkumar Tulajappa Kalburgi, Claire Le Goues, and Alex Groce

  15. [23]

    Tanu Jindal. 2016. Importance of Testing in SDLC.International Journal of Engi- neering and Applied Computer Science (IJEACS)(2016). https://www.researchgate. net/publication/312041152

    2016

  16. [24]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Pe- ter Clark, and Ashish Sabharwal. 2023. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. https://arxiv.org/abs/2210.02406

    2023 arXiv

  17. [25]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. LlamaRestTest: Effective REST API Testing with Small Language Models. https://arxiv.org/abs/ 2501.08598

    2025 arXiv

  18. [26]

    Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, and Alessandro Orso. 2024. Leveraging Large Language Models to Improve REST API Testing. (2024). https://doi.org/10.1145/3639476.3639769

    2024

  19. [27]

    Jakub Kozera. 2025. Restaurants API - Repository. https://github.com/ jakubkozera/Restaurants Accessed on: March 24, 2025

    2025

  20. [28]

    Alexander Lercher. 2024. Managing API Evolution in Microservice Architecture. InProceedings of the 46th International Conference on Software Engineering (ICSE). ACM. https://doi.org/10.1145/3639478.3639800

    2024

  21. [29]

    LiveBench. 2024. LiveBench - Leaderboard. https://livebench.ai/#/ Accessed on: January 25, 2025

    2024

  22. [30]

    Mitra Madanchian and Hamed Taherdoost. 2023. A comprehensive guide to the TOPSIS method for multi-criteria decision making.Madanchian M, Taherdoost H. A comprehensive guide to the TOPSIS method for multi-criteria decision making. Sustainable Social Development(2023)

    2023

  23. [31]

    Isela Mendoza, Fernando Silva Filho, Gustavo Medeiros, Aline Paes, and Vânia Neves. 2024. Comparative Analysis of Large Language Model Tools for Automated Test Data Generation from BDD. InAnais do XXXVIII Simpósio Brasileiro de Engenharia de Software(Curitiba/PR). SBC, Porto A...

    2024

  24. [32]

    2025.Integration tests in ASP.NET Core

    Microsoft. 2025.Integration tests in ASP.NET Core. https://learn.microsoft.com/en- us/aspnet/core/test/integration-tests Accessed: April 06, 2025

    2025

  25. [33]

    Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner. 2025. A Closer Look at System Prompt Robustness. https://arxiv.org/abs/2502.12197

    2025 arXiv

  26. [34]

    2024.LLM System Prompt vs

    Nebuly. 2024.LLM System Prompt vs. User Prompt. https://www.nebuly.com/ blog/llm-system-prompt-vs-user-prompt Accessed: April 07, 2025

    2024

  27. [35]

    2015.Building Microservices: Designing Fine-Grained Systems

    Sam Newman. 2015.Building Microservices: Designing Fine-Grained Systems. O’Reilly Media

    2015

  28. [36]

    Mitchell Olsthoorn. 2022. More Effective Test Case Generation with Multiple Tribes of AI. InProceedings of the 44th International Conference on Software Engineering (ICSE) - Doctoral Symposium. ACM. https://doi.org/10.1145/3510454. 3517066

    2022 doi

  29. [37]

    OpenRouter. 2025. LLM Rankings - Programming. https://openrouter.ai/ rankings/programming?view=month Accessed on: January 25, 2025

    2025

  30. [38]

    Alessandro Orso and Gregg Rothermel. 2014. Software testing: a research travel- ogue (2000-2014). InFuture of Software Engineering Proceedings. Georgia Institute of Technology. https://dl.acm.org/doi/10.1145/2593882.2593885

    2014

  31. [39]

    Ostrand and Marc J

    Thomas J. Ostrand and Marc J. Balcer. 1988. The category-partition method for specifying and generating fuctional tests.Commun. ACM(1988). https: //dl.acm.org/doi/10.1145/62959.62964

    1988

  32. [40]

    Ouédraogo, Kader Kaboré, Haoye Tian, Yewei Song, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F

    Wendkûuni C. Ouédraogo, Kader Kaboré, Haoye Tian, Yewei Song, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F. Bissyandé. 2024. Large-scale, Inde- pendent and Comprehensive study of the power of LLMs for test case generation. https://arxiv.org/abs/2407.00225

    2024

  33. [41]

    Mauro Pezzè, Matteo Ciniselli, Luca Grazia, Niccolò Puccinelli, and Ketai Qiu

  34. [42]

    Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. 2024. How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark. https://arxiv.org/abs/2406.06647

    2024 arXiv

  35. [43]

    Shakudo. 2025. Top 9 Large Language Models as of January 2025. https://www. shakudo.io/blog/top-9-large-language-models Accessed on: January 26, 2025

    2025

  36. [44]

    2010.Software Engineering(9 ed.)

    Ian Sommerville. 2010.Software Engineering(9 ed.). Addison-Wesley, Harlow, England

    2010

  37. [45]

    Stelios Sotiriadis, Andrus Lehmets, Euripides G. M. Petrakis, and Nik Bessis

  38. [46]

    Poorna Soysa. 2025. Books API - Repository. https://github.com/poorna-soysa/ books-api-docker-compose-postgresql-redis Accessed on: March 24, 2025

    2025

  39. [47]

    2025.Stryker .NET - Configuration

    Stryker. 2025.Stryker .NET - Configuration. https://stryker-mutator.io/docs/ stryker-net/configuration/ Accessed: April 09, 2025

    2025

  40. [48]

    Aichernig, and Bettina Könighofer

    Martin Tappler, Andrea Pferscher, Bernhard K. Aichernig, and Bettina Könighofer

  41. [49]

    Maneela Tuteja, Gaurav Dubey, et al. 2012. A research study on importance of testing and quality assurance in software development life cycle (SDLC) models. International Journal of Soft Computing and Engineering (IJSCE)(2012). https: //www.ijsce.org/portfolio-item/c0761062312/

    2012

  42. [50]

    Unite.AI. 2025. Best Large Language Models (LLMs) in 2025. https://www.unite. ai/best-large-language-models-llms/ Accessed on: January 20, 2025

    2025

  43. [51]

    Vellum. 2025. LLM Leaderboard - Model Comparison. https://www.vellum.ai/llm- leaderboard Accessed on: January 26, 2025

    2025

  44. [52]

    InProceedings of the 46th International Conference on Software Engineering (ICSE)

    Learning and Repair of Deep Reinforcement Learning Policies from Fuzz- Testing Data. InProceedings of the 46th International Conference on Software Engineering (ICSE). ACM. https://doi.org/10.1145/3597503.3623311

  45. [53]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing with Large Language Models: Survey, Landscape, and Vision. https://arxiv.org/abs/2307.07221

    2024 arXiv

  46. [54]

    Trevoir Williams. 2025. Hotels API - Repository. https://github.com/ trevoirwilliams/HotelListing.API.NET Accessed on: March 24, 2025

    2025

  47. [55]

    Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. 2024. Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Bench- marks via LLM. https://arxiv.org/abs/2403.19114

    2024 arXiv

  48. [56]

    Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTEST- GEN: Automated Black-Box Testing of RESTful APIs. In2020 IEEE 13th Inter- national Conference on Software Testing, Validation and Verification (ICST). IEEE. https://doi.org/10.1109/ICST46399.2020.00024

    2020

  49. [57]

    Peng Zhang, Yang Wang, Xutong Liu, Yibiao Yang, Yanhui Li, Lin Chen, Ziyuan Wang, Chang ai Sun, and Yuming Zhou. 2022. Test suite effectiveness metric evaluation: what do we know and what should we do? https://arxiv.org/abs/ 2204.09165

    2022 arXiv

  50. [60]

    Lechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. SPRIG: Improving Large Language Model Performance by System Prompt Optimization. https://arxiv.org/abs/2410.14826

    2024 arXiv

  51. [2017]

    In2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA)

    Unit and Integration Testing of Modular Cloud Services. In2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA). https://doi.org/10.1109/AINA.2017.57

    2017 doi

  52. [2023]

    https://arxiv.org/abs/2309.02395

    Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts. https://arxiv.org/abs/2309.02395

    arXiv

  53. [2024]

    https: //www.inf.usi.ch/faculty/pezze/media/SE2030SENreport.pdf [Online; accessed 1-Apr-2025]

    The Trailer of the ACM 2030 Roadmap for Software Engineering. https: //www.inf.usi.ch/faculty/pezze/media/SE2030SENreport.pdf [Online; accessed 1-Apr-2025]

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.