Pith. sign in

REVIEW 3 major objections 5 minor 106 references

Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most tests for LLM-based agents are narrow unit tests, a 2,572-test study finds

desk verdict The most substantial empirical look at how agent apps are actually tested, with a real dataset behind it, but the headline percentages rest on a 12-project sample with one project carrying a third of the tests. read the letter →

arxiv 2608.08413 v1 pith:OUNPRBWC submitted 2026-08-09 cs.SE

Rangeet Pan , Tyler Stennett , Divya Sankar , Bridget McGinn , Alessandro Orso , Raju Pavuluri , Saurabh Sinha , Maja Vukovic This is my paper · ORCID
classification cs.SE
keywords agenttestingLLM-basedagentsempiricalsoftwareengineeringtaxonomynon-functionaltestpatternsunit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM-based agent applications are being tested far more narrowly than their complexity demands. By manually labeling 2,572 test methods from 240 modules in 12 open-source projects and interviewing 10 industry practitioners, the paper finds that 61.8% of tests exercise a single tool or agent in isolation, 34.6% target interactions, and only 7.5% address non-functional requirements like security or performance. Tests typically rely on simple example inputs, heavy mocking, and shallow assertions that rarely check reasoning or side effects. The paper argues this pattern holds across frameworks, whose abstraction level shapes how much integration testing developers do, and it extracts a 23-pattern taxonomy plus a research roadmap to close the gaps.

What carries the argument

The load-bearing machinery is a four-dimensional classification scheme applied to 2,572 manually labeled test methods. Each test is classified along testing level (unit, module, integration, API), test type (functional vs non-functional), and 23 patterns grouped into test fixture, test data, test objective, and test assertion categories. The taxonomy turns qualitative impressions about shallow testing into countable fractions: 61.8% isolated, 34.6% interaction, 7.5% non-functional, 39.9% dummy data, 34.4% mocked agent or tool, and 8.7% reasoning assertions. The framework-abstraction comparison then explains part of the distribution, and the structural-complexity analysis (median 14 non-comment lines of code, median cyclomatic complexity 1) reinforces the shallowness claim.

What would settle it

Re-run the manual labeling on a random sample of 1,000 test methods drawn from agent repositories across languages and repository sizes; if more than half are interaction-level or non-functional tests, the claimed unit-test dominance would be overturned.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that current testing practice for LLM-based agent applications is dominated by narrowly scoped unit tests with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. The evidence is a manually labeled dataset of 2,572 test methods from 240 modules across 12 open-source Python projects, plus structured interviews with 10 senior practitioners. Testing levels vary systematically with framework abstraction: applications built on higher-level tool protocols lean toward unit tests, middle-level orchestration frameworks balance unit and integration tests, and lower-level composition frameworks lean toward integration tests. Only 7.5% of tests target non-functional requirements, 39.9% use simple dummy data, 34.4% mock an agent or tool, and 8.7% assert on reasoning-related behavior. From these observations the paper derives a taxonomy of 23 testing patterns and a roadmap calling for formal test objectives, adequacy criteria for agentic abstractions, fault-based testing, and better testability foundations for agent frameworks.

Load-bearing premise

The 12 open-source projects and 10 interviewed practitioners are representative of how LLM-based agent applications are tested in general; if this sample is atypical, the reported percentages describe a narrow subset rather than the field.

Editorial extensions

If this is right

  • If correct, agent testing practice is systematically shallow relative to the complexity of agentic systems, so interaction, environment, and non-functional failures are likely under-detected in current test suites.
  • The framework correlation implies testability is partly a design property of agent frameworks: choosing a higher-level protocol shifts the testing burden toward end users who compose components, while lower-level frameworks force more integration testing on developers.
  • The near-absence of reasoning assertions indicates that reasoning quality is currently validated by manual observation or benchmarks rather than by automated tests, leaving a large gap in regression protection.
  • The lack of formal adequacy criteria means agent test suites cannot be objectively evaluated; coverage of tool selection, delegation patterns, and multi-step execution paths is left unmeasured.
  • Interview responses suggest industry emphasizes non-functional testing more than open-source projects do, but both share the same foundational gaps in adequacy metrics and formal test objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: use the paper's 23-pattern taxonomy as a checklist to measure whether new agent frameworks or testing tools shift the distribution toward interaction-level and non-functional tests.
  • The findings imply that improving benchmark scores on agent task suites is largely orthogonal to improving developer testing practice, so progress in one does not automatically translate into better-tested agent applications.
  • If the framework-abstraction correlation holds as a causal claim, then designing agent APIs with explicit seams for isolating, observing, and controlling agent behavior should increase module-level testing; this prediction could be tested by comparing test distributions across framework versions over time.
  • The paper's sample is Python-only open-source projects with at least 20 test modules; applying the same taxonomy to a broader sample (other languages, smaller projects, closed-source systems) would reveal whether the unit-test dominance is universal or an artifact of the selection criteria.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an empirical study of testing practices for LLM-based agent applications. The authors mine GitHub for Python projects using agent frameworks (LangChain, LangGraph, MCP, etc.), statically detect agents and tools, filter and manually label a final dataset of 2,572 test methods from 240 test modules in 12 projects, and complement this with structured interviews of 10 IBM practitioners. They derive a taxonomy of 23 testing patterns, analyze testing levels and types, structural complexity, and practitioner perceptions, and propose a research roadmap. The central claim is that current agent testing is dominated by narrowly scoped unit tests with limited interaction coverage, unrealistic test data, heavy mocking, shallow assertions, low structural complexity, and little non-functional testing.

Significance. If the findings hold, this is a timely and useful contribution: it provides some of the first quantitative evidence about how LLM-based agent applications are actually tested, and the 23-pattern taxonomy gives the community a shared vocabulary. The paper's strengths include a transparent data-collection pipeline, a released artifact, explicit raw counts in Table 2, and a high inter-rater agreement (Cohen's kappa 0.92). The main weakness is that the headline percentages rest on only 12 projects with strong concentration (Upsonic alone contributes about 30% of the labeled tests), and the practitioner interviews come from a single company. These limitations are disclosed in Section 8, but the abstract and several findings state population-level conclusions that outrun the sample.

major comments (3)
  1. [§3.2, Finding 1; Table 1] The headline percentages (61.8% unit-level, 34.6% interaction, 7.5% NFR) are reported only at the aggregate level. With 12 projects, and with Upsonic contributing 770 of 2,572 tests (about 30%) and 66.5% of all NFR tests, a small number of projects can move the aggregate by double digits. Please add a per-project breakdown of testing levels and NFR focus, and report a sensitivity analysis (e.g., excluding Upsonic or using project-level weights). Without this, the abstract's population-level claim that 'testing of LLM-based agent applications is dominated by...' is not adequately supported.
  2. [§6 and Abstract] The practitioner evidence consists of ten interviews, all with employees of a single organization (IBM). The abstract and §6 describe industry practice as placing 'greater emphasis on non-functional testing than open-source projects,' but a single-company sample cannot independently corroborate the open-source measurements. Please reframe §6 as a single-organization qualitative perspective and adjust the abstract to avoid implying independent confirmation of the repository findings.
  3. [§2, Phase 4] The filtering pipeline excludes repositories with fewer than 20 test modules, as well as example and framework repositories, but the manuscript does not quantify how many candidate projects were removed at each exclusion step or how the retained 12 projects compare with the excluded ones (164 repositories with relevant test modules before filtering). Reporting these numbers and the characteristics of excluded projects would help readers assess selection bias. This is particularly relevant because the same filters may systematically select mature, well-tested projects and thereby shape the observed patterns.
minor comments (5)
  1. [Figure 2] Figures 2(b) and 2(c) show framework and version breakdowns, but the text does not report the number of tests per framework or version; adding sample sizes would let readers judge the reliability of the apparent associations.
  2. [Table 2] The note explains that tests can belong to multiple patterns, but the subcategory percentages (e.g., 'Mock some component' 21.66% and 'Mock entire agent' 22.28%) are not mutually exclusive and do not sum to the parent category. Please clarify whether percentages are computed over all tests or over the parent-category tests.
  3. [Table 3] The questionnaire jumps from Q21 to Q23; if Q22 was intentionally omitted, please say so, otherwise renumber the questions.
  4. [§5] The comparison with the authors' prior Hamster study [69] is mentioned without any details about that study's method or dataset; since the reference is to an unpublished manuscript, please add a brief description or soften the comparison.
  5. [Abstract and §3.2] The wording 'testing of LLM-based agent applications is dominated by...' appears in the abstract and Finding 1; consider qualifying it as 'in the studied sample' to match the evidence presented.

Circularity Check

0 steps flagged · score 1.0 of 10

The paper's findings are descriptive summaries of manually labeled data; the only self-citation (Hamster) is a non-load-bearing comparative reference, so there is no circular derivation.

full rationale

This paper is an observational empirical study rather than a derivation, so the main circularity patterns do not apply. The headline percentages (61.8% unit, 34.6% interaction, 7.5% NFR) are computed directly from 2,572 manually labeled test methods with a reported Cohen's kappa of 0.92; they are summaries of the labels, not predictions derived from the labels. The 23-pattern taxonomy was induced from the same corpus but is used descriptively to organize the data, and the findings do not assume the taxonomy's truth as an input. The LLM-assisted pre-filtering in Phase 3 is followed by Phase 4 human verification of every retained test, so the final dataset is not an LLM output being used as evidence for the same LLM's behavior. The only self-citation is reference [69] (Hamster), used in Section 5 as a comparative reference for Java test complexity; this comparison is not load-bearing for the central claim about agent testing, and the Hamster finding itself concerns a different corpus and is not a fitted parameter in this paper. The practitioner interviews are independent qualitative evidence, and the paper appropriately discloses in Section 8 that all interviewees are from IBM, which is a generality limitation rather than a circularity. No equation, definition, or fitted parameter makes any reported finding equivalent to its input.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The ledger contains no invented entities and no fitted mathematical parameters. The hand-chosen sampling thresholds and representativeness assumptions carry the evidentiary burden, not a fitted model or derivation.

free parameters (2)
  • Minimum GitHub star count = 50
    Phase 1 inclusion threshold for GitHub repositories; chosen by hand as a quality proxy; shapes the sampled population and all downstream results.
  • Minimum test-module count per project = 20
    Phase 4 filter that reduced candidate projects to the 12 projects that were manually labeled; all test-level proportions in RQ1 to RQ3 are computed only over this hand-picked subset.
assumptions (5)
  • domain assumption The final 12-project subset is representative of the broader population of LLM-based agent application testing.
    Load-bearing for generalizing RQ1 to RQ3 percentages; selection used Python GitHub repos with at least 50 stars and at least 20 test modules, excluding example and framework projects. Acknowledged as an external validity threat in Section 8.
  • domain assumption Manual labels by two authors (Cohen's kappa 0.92) are accurate ground truth for test characteristics.
    All quantitative findings derive from these labels; agreement was measured, but only two human raters labeled 2,572 methods after one labeler pre-filtered the candidate modules.
  • domain assumption Gemini labeling agents identify candidate test modules with acceptable recall.
    Phase 3 used permissive LLM labeling to reduce over 17,000 test modules to 3,050; if the labeling agents systematically missed agent-related modules, the final sample is incomplete.
  • domain assumption The 10 IBM interviewees represent industry practitioners beyond a single company.
    Qualitative findings in RQ4 and Section 6 rest entirely on interviews from one company; the paper acknowledges this limitation but still uses the interviews to support field-wide conclusions.
  • domain assumption Tests committed to GitHub are a valid proxy for how developers test agent applications.
    External validity: committed tests may be AI-generated or may not reflect internal testing practice; the authors note this in Section 8 but the central claim depends on this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications." pith.science (2026). https://pith.science/paper/OUNPRBWC

@misc{pith2026260808413,
  author       = {Pith},
  title        = {Pith review of: Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUNPRBWC}},
  note         = {Machine review of arXiv:2608.08413}
}
read the original abstract

Agents built on large language models (LLMs) are increasingly used to build applications that perform complex, multi-step tasks involving reasoning, tool use, and interaction with external environments. Despite rapid progress in benchmarking LLM-based agents, very few studies have attempted to understand how such systems are tested in practice. In particular, testing levels, objectives, data patterns, test complexity, and validation strategies for agent applications remain underexplored. In this paper, we present an empirical study of testing practices in LLM-based agent applications using a large corpus of mined open-source projects. We construct a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules. From this analysis, we derive a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level (unit, module, integration). We complement this with structured interviews of 10 senior industry practitioners building agentic systems. Our results show that testing of LLM-based agent applications is dominated by narrowly scoped unit tests, with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. Tests frequently rely on simplistic inputs, heavy mocking, and shallow validation, and agent-related tests exhibit low structural complexity. While industry practice places greater emphasis on non-functional testing than open-source projects, both reveal common gaps, including the lack of formal testing foundations, unclear test objectives, and challenges in generating high-quality test data. Based on these findings, we outline research directions toward more systematic and rigorous testing of agent applications, including foundations for agent testability, formalized test objectives, and fault-based testing techniques.

Figures

Figures reproduced from arXiv: 2608.08413 by the authors.

Figure 1
Figure 1. The Tangent dataset collection and test labeling steps. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Distribution of NFR testing types. Lack of Non-Functional Testing. Only 7.5% of the manually labeled tests target non-functional requirements (NFRs), indicating that such testing is uncommon in agent applications. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Distribution of test types: (a) overall, (b) across different frameworks, (c) different versions of frameworks. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The testing patterns identified from our dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Research roadmap for addressing common chal [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

106 extracted references · 46 canonical work pages

  1. [1]

    Tangent Artifact

    2026. Tangent Artifact. https://github.com/aster-test-generation/tangent-ase- 2026

    2026

  2. [2]

    Mozilla AI. [n. d.]. any-agent. https://github.com/mozilla-ai/any-agent. Ac- cessed: 2026-03-27

    2026

  3. [3]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992 (2024)

    arXiv 2024

  4. [4]

    Paul Ammann and Jeff Offutt. 2016. Introduction to Software Testing (2 ed.). Cambridge University Press

    2016

  5. [5]

    Anthropic. 2024. Model Context Protocol. https://modelcontextprotocol.io. Accessed: 2026-03-19

    2024

  6. [6]

    Arize AI. 2026. Arize AI. https://arize.com/docs/ax/evaluate/evaluators/trace- and-session-evals/trace-level-evaluations/agent-trajectory-evaluations

    2026

  7. [7]

    Jean Arlat, Martine Aguera, Louis Amat, Yves Crouzet, J-C Fabre, J-C Laprie, Eliane Martins, and David Powell. 1990. Fault injection for dependability vali- dation: A methodology and some applications. IEEE Transactions on software engineering 16, 2 (1990), 166–182

    1990

  8. [8]

    Marina Bagić Babac and Dragan Jevtić. 2014. AgentTest: A specification language for agent-based system testing. Neurocomputing 146 (2014), 230–248

    2014

Show all 106 references
  1. [9]

    Ali Basiri, Niosha Behnam, Ruud De Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE software 33, 3 (2016), 35–41

    2016

  2. [10]

    Robert V Binder. 1994. Design for testability in object-oriented systems. Commun. ACM 37, 9 (1994), 87–101

    1994

  3. [11]

    Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage- based greybox fuzzing as markov chain. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 1032–1043

    2016

  4. [12]

    Hudson Borges and Marco Tulio Valente. 2018. What’s in a github star? under- standing repository starring practices in a social coding platform. Journal of Systems and Software 146 (2018), 112–129

    2018

  5. [13]

    Islem Bouzenia and Michael Pradel. 2025. Understanding software engi- neering agents: A study of thought-action-result trajectories. arXiv preprint arXiv:2506.18824 (2025)

    2025

  6. [14]

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, et al. 2026. Why do multi-agent llm systems fail? Advances in Neural Information Processing Systems 38 (2026)

    2026

  7. [15]

    Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR) 51, 1 (2018), 1–27

    2018

  8. [16]

    Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016. Coverage-directed differential testing of JVM implementations. In proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation. 85–99

    2016

  9. [17]

    Tsun S. Chow. 1978. Testing software design modeled by finite-state machines. IEEE transactions on software engineering 3 (1978), 178–187

    1978

  10. [18]

    Andrew G Clark, Neil Walkinshaw, and Robert M Hierons. 2021. Test case generation for agent-based models: A systematic literature review. Information and Software Technology 135 (2021), 106567

    2021

  11. [19]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46

    1960

  12. [20]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al . 2025. Swe- bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941 (2025)

    2025 arXiv

  13. [21]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems 36 (2023), 28091–28114

    2023

  14. [22]

    Felix Dobslaw, Robert Feldt, Juyeon Yoon, and Shin Yoo. 2025. Challenges in testing large language model based software: A faceted taxonomy. arXiv preprint arXiv:2503.00481 (2025)

    2025

  15. [23]

    Saikat Dutta, Owolabi Legunsen, Zixin Huang, and Sasa Misailovic. 2018. Testing probabilistic programming systems. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 574–586

    2018

  16. [24]

    Saikat Dutta, August Shi, Rutvik Choudhary, Zhekun Zhang, Aryaman Jain, and Sasa Misailovic. 2020. Detecting flaky tests in probabilistic and machine learning applications. In Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 211–224

    2020

  17. [25]

    Virtual Patient Engine. [n. d.]. AIAgents4Pharma. https://github.com/ VirtualPatientEngine/AIAgents4Pharma. Accessed: 2026-03-27

    2026

  18. [26]

    Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. 2025. Interactive debugging and steering of multi-agent ai systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–15

    2025

  19. [27]

    Ali Eslami and Jiangbo Yu. 2026. A Control-Theoretic Foundation for Agentic Systems. arXiv preprint arXiv:2603.10779 (2026)

    2026

  20. [28]

    Vojtěch Forejt, Marta Kwiatkowska, Gethin Norman, and David Parker. 2011. Automated verification techniques for probabilistic systems. In International school on formal methods for the design of computer, communication and software systems. Springer, 53–113

    2011

  21. [29]

    Roy S Freedman. 1991. Testability of software components. IEEE transactions on Software Engineering 17, 6 (1991), 553–564

    1991

  22. [30]

    Vahid Garousi, Michael Felderer, and Feyza Nur Kılıçaslan. 2019. A survey on software testability. Information and Software Technology 108 (2019), 35–64

    2019

  23. [31]

    Dwyer, and Willem Visser

    Jaco Geldenhuys, Matthew B. Dwyer, and Willem Visser. 2012. Probabilistic symbolic execution. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (Minneapolis, MN, USA) (ISSTA 2012). Associ- ation for Computing Machinery, New York, NY, USA, 166–...

    2012

  24. [32]

    Google. 2025. Gemini Models. https://ai.google.dev/gemini-api/docs/models. Accessed: 2026-03-16

    2025

  25. [33]

    Guardian. 2026. Meta AI agent’s instruction causes large sensitive data leak to employees. https://tinyurl.com/ycxtuzs9

    2026

  26. [34]

    Aayush Gupta. 2026. ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions. arXiv preprint arXiv:2601.06112 (2026)

    2026

  27. [35]

    Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajba- hadur, Bram Adams, and Ahmed E Hassan. 2025. An empirical study of testing practices in open source AI agent frameworks and agentic applications. arXiv preprint arXiv:2509.19185 (2025)

    2025 arXiv

  28. [36]

    Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2026. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions. arXiv preprint arXiv:2602.14878 (2026)

    2026 arXiv

  29. [37]

    John Hughes. 2007. QuickCheck testing for fun and profit. In International Symposium on Practical Aspects of Declarative Languages. Springer, 1–32

    2007

  30. [38]

    IBM. 2025. Agent Lifecycle Toolkit. https://github.com/AgentToolkit/agent- lifecycle-toolkit. Toolkit for building, testing, and managing AI agents. Accessed: 2026-03-07

    2025

  31. [39]

    ISO/IEC. 2023. Software Engineering — Systems and Software Quality Re- quirements and Evaluation (SQuaRE) — Quality Model for AI Systems. https: //www.iso.org/standard/80655.html

    2023

  32. [40]

    ISO/IEC. 2023. Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—Product Quality Model

    2023

  33. [41]

    ISO/IEC/IEEE. 2017. Systems and Software Engineering—Vocabulary

    2017

  34. [42]

    ISO/IEC/IEEE. 2022. Software and Systems Engineering—Software Testing— Part 1: General Concepts

    2022

  35. [43]

    ISTQB. [n. d.]. Standard Glossary of Terms Used in Software Testing. https: //glossary.istqb.org

  36. [44]

    ISTQB. 2021. Certified Tester AI Testing (CT-AI) Syllabus. Technical Report. International Software Testing Qualifications Board. https://astqb.org/assets/ documents/ISTQB_CT-AI_Syllabus_v1.0.pdf

    2021

  37. [45]

    Nils Jansen, Christian Dehnert, Benjamin Lucien Kaminski, Joost-Pieter Katoen, and Lukas Westhofen. 2016. Bounded model checking for probabilistic programs. In International Symposium on Automated Technology for Verification and Analysis. Springer, 68–85. Tangent: An Empirical...

    2016

  38. [46]

    Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi- Agent Systems. arXiv preprint arXiv:2602.19843 (2026)

    2026

  39. [47]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, and Percy Liang

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, and Percy Liang. 2023. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  40. [48]

    Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, and Chany- oung Park. 2025. Beyond the Final Answer: Evaluating the Reasoning Trajecto- ries of Tool-Augmented Agents. arXiv preprint arXiv:2510.02837 (2025)

    2025 arXiv

  41. [49]

    Jens Kohl, Otto Kruse, Youssef Mostafa, Andre Luckow, Karsten Schroer, Thomas Riedl, Ryan French, David Katz, Manuel P Luitz, Tanrajbir Takher, et al. 2025. Automated structural testing of LLM-based agents: methods, framework, and case studies. In2025 IEEE International Confer...

    2025

  42. [50]

    Solace Labs. [n. d.]. solace-agent-mesh. https://github.com/SolaceLabs/solace- agent-mesh. Accessed: 2026-03-27

    2026

  43. [51]

    LangChain. 2023. LangChain. https://github.com/langchain-ai/langchain. Framework for building agents and LLM-powered applications. Accessed: 2026- 03-07

    2023

  44. [52]

    LangChain. 2024. LangGraph. https://github.com/langchain-ai/langgraph. Or- chestration framework for building, managing, and deploying stateful agents. Accessed: 2026-03-07

    2024

  45. [53]

    LangChain. 2026. Trajectory match evaluator. https://docs.langchain.com/ langsmith/trajectory-evals#trajectory-match-evaluator

    2026

  46. [54]

    Li Li, Jiawei Wang, and Haowei Quan. 2022. Scalpel: The Python Static Analysis Framework. arXiv preprint arXiv:2202.11840 (2022)

    2022 arXiv

  47. [55]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977 (2024)

    2024 arXiv

  48. [56]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. AgentBench: Evaluating LLMs as agents. arXiv preprint arXiv:2308.03688 (2023)

    2023 arXiv

  49. [57]

    Păsăreanu, Matthew B

    Kasper Luckow, Corina S. Păsăreanu, Matthew B. Dwyer, Antonio Filieri, and Willem Visser. 2014. Exact and approximate probabilistic symbolic execution for nondeterministic programs. InProceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (...

    2014

  50. [58]

    Wei Ma, Yixiao Yang, Qiang Hu, Shi Ying, Zhi Jin, Bo Du, Zhenchang Xing, Tianlin Li, Junjie Shi, Yang Liu, et al . 2025. Rethinking Testing for LLM Ap- plications: Characteristics, Challenges, and a Lightweight Interaction Protocol. arXiv preprint arXiv:2508.20737 (2025)

    2025 arXiv

  51. [59]

    Muhammad Maaz, Liam DeVoe, Zac Hatfield-Dodds, and Nicholas Carlini. 2025. Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem. arXiv preprint arXiv:2510.09907 (2025)

    2025

  52. [60]

    Tim Mackinnon, Steve Freeman, and Philip Craig. 2000. Endo-testing: unit testing with mock objects. Extreme programming examined (2000), 287–301

    2000

  53. [61]

    Cleyton Magalhaes, Italo Santos, Brody Stuart-Verner, and Ronnie De Souza Santos. 2025. Testing the Untestable? An Empirical Study on the Testing Process of LLM-Powered Software Systems. In 2025 IEEE International Conference on Source Code Analysis & Manipulation (SCAM). IEEE, 78–88

    2025

  54. [62]

    manykarim. [n. d.]. rf-mcp. https://github.com/manykarim/rf-mcp. Accessed: 2026-03-27

    2026

  55. [63]

    Maxim AI. 2026. Maxim AI. https://www.getmaxim.ai/docs/library/evaluators/ pre-built-evaluators/ai-evaluators/agent-trajectory

    2026

  56. [64]

    Bjorn Melin. [n. d.]. docmind-ai-llm. https://github.com/BjornMelin/docmind- ai-llm. Accessed: 2026-03-27

    2026

  57. [65]

    Gerard Meszaros. 2007. xUnit Test Patterns: Refactoring Test Code. Addison- Wesley

    2007

  58. [66]

    Erik Miehling, Karthikeyan Natesan Ramamurthy, Kush R Varshney, Matthew Riemer, Djallel Bouneffouf, John T Richards, Amit Dhurandhar, Elizabeth M Daly, Michael Hind, Prasanna Sattigeri, et al. 2025. Agentic AI needs a systems theory. arXiv preprint arXiv:2503.00237 (2025)

    2025 arXiv

  59. [67]

    Ivan Milev, Mislav Balunović, Maximilian Baader, and Martin Vechev. 2025. ToolFuzz–Automated Agent Tool Testing. arXiv preprint arXiv:2503.04479 (2025)

    2025 arXiv

  60. [68]

    Intelligent Node. [n. d.]. Intelli. https://github.com/intelligentnode/Intelli. Ac- cessed: 2026-03-27

    2026

  61. [69]

    Rangeet Pan, Tyler Stennett, Raju Pavuluri, Nate Levin, Alessandro Orso, and Saurabh Sinha. 2026. Hamster: A Large-Scale Study and Characterization of Developer-Written Tests. (2026)

    2026

  62. [70]

    phil65. [n. d.]. agentpool. https://github.com/phil65/agentpool. Accessed: 2026-03-27

    2026

  63. [71]

    PRArena. [n. d.]. PRArena. https://prarena.ai. Accessed: 2026-03-27

    2026

  64. [72]

    Prefect. 2024. FastMCP: A Fast Implementation of the Model Context Protocol. https://github.com/PrefectHQ/fastmcp. Accessed: 2026-03-19

    2024

  65. [73]

    Erasmo Purificato, Ludovico Boratto, and Ernesto William De Luca. 2024. User modeling and user profiling: A comprehensive survey. arXiv preprint arXiv:2402.09660 (2024)

    2024 arXiv

  66. [74]

    Yaoyao Qian, Yuanli Wang, Jinda Zhang, Yun Zong, Meixu Chen, Hanhan Zhou, Jindan Huang, Yifan Zeng, Xinyu Hu, Chan Hee Song, et al. 2025. WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation. arXiv preprint arXiv:2510.19205 (2025)

    2025

  67. [75]

    Qredence. [n. d.]. agentic-fleet. https://github.com/Qredence/agentic-fleet. Accessed: 2026-03-27

    2026

  68. [76]

    Shanto Rahman, Saikat Dutta, and August Shi. 2025. Understanding and im- proving flaky test classification. Proceedings of the ACM on Programming Languages 9, OOPSLA2 (2025), 1345–1371

    2025

  69. [77]

    Clemens Rawert, Marc Klingen, and Maximilian Deichmann. 2023. Langfuse — Open-Source AI Engineering Platform. https://langfuse.com/ Software available from https://langfuse.com

    2023

  70. [78]

    rgoerwit. [n. d.]. ai-investment-agent. https://github.com/rgoerwit/ai- investment-agent. Accessed: 2026-03-27

    2026

  71. [79]

    Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, and Jarek Szlichta

  72. [80]

    Zary Segall, D Vrsalovic, D Siewiorek, D Ysskin, J Kownacki, J Barton, R Dancey, A Robinson, and T Lin. 1995. Fiat-fault injection based automated test- ing environment. In Twenty-Fifth International Symposium on Fault-Tolerant Computing, 1995, ’Highlightsfrom Twenty-Five Year...

    1995

  73. [81]

    SMAT Lab. 2022. Scalpel: The Python Static Analysis Framework. https://github. com/SMAT-Lab/Scalpel. Accessed: 2026-03-16

    2022

  74. [82]

    Stanford NLP. 2024. DSPy. https://github.com/stanfordnlp/dspy. Programming framework for language model pipelines. Accessed: 2026-03-07

    2024

  75. [83]

    Upsonic. [n. d.]. Upsonic. https://github.com/Upsonic/Upsonic. Accessed: 2026-03-27

    2026

  76. [84]

    Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Software testing, verification and reliability 22, 5 (2012), 297–312

    2012

  77. [85]

    Voas and Keith W Miller

    Jeffrey M. Voas and Keith W Miller. 2002. Software testability: The new verifica- tion. IEEE software 12, 3 (2002), 17–28

    2002

  78. [86]

    Wall Street Journal. 2026. We Let AI Run Our Office Vending Machine. It Lost Hundreds of Dollars. https://tinyurl.com/4uuus8vk

    2026

  79. [87]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345

    2024

  80. [88]

    Yanlin Wang, Xinyi Xu, Jiachi Chen, Tingting Bi, Wenchao Gu, and Zibin Zheng

  81. [89]

    Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, and Dawn Song. 2025. Agentvigil: Generic black-box red-teaming for indirect prompt injection against llm agents. arXiv preprint arXiv:2505.05849 (2025)

    2025 arXiv

  82. [90]

    arXiv preprint arXiv:2512.01939 (2025)

    An empirical study of agent developer practices in ai agent frameworks. arXiv preprint arXiv:2512.01939 (2025)

    2025

  83. [91]

    E. J. Weyuker. 1988. The evaluation of program-based software test data ade- quacy criteria. Commun. ACM 31, 6 (June 1988), 668–675. doi:10.1145/62959. 62963

    1988 doi

  84. [92]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  85. [93]

    Jiangrong Wu, Zitong Yao, Yuhong Nan, and Zibin Zheng. 2026. ChainFuzzer: Greybox Fuzzing for Workflow-Level Multi-Tool Vulnerabilities in LLM Agents. arXiv preprint arXiv:2603.12614 (2026)

    2026

  86. [94]

    wshobson. [n. d.]. maverick-mcp. https://github.com/wshobson/maverick-mcp. Accessed: 2026-03-27

    2026

  87. [95]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al

  88. [96]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2025. The rise and potential of large language model based agents: A survey. Science China Information Sciences 68, 2 (2025), 121101

    2025

  89. [97]

    John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2026. Swe-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems 38 (2026). ASE ’26...

    2026

  90. [98]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. 𝜏- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045 (2024)

    2024 arXiv

  91. [99]

    Ziluo Xue, Yanjie Zhao, Shenao Wang, Kai Chen, and Haoyu Wang. 2025. A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 3369–3380

    2025

  92. [100]

    zhongyu09. [n. d.]. openchatbi. https://github.com/zhongyu09/openchatbi. Accessed: 2026-03-27

    2026

  93. [101]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv preprint arXiv:2307.13854 (2023). https://webarena.dev

    2023 arXiv

  94. [102]

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. 2025. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. arXiv preprint arXiv:2505.00212 (2025)

    2025 arXiv

  95. [103]

    Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, and Jiaxuan You. 2025. MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents. In Proceedings of the 63rd Annual Meeting of the ...

    2025

  96. [105]

    Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy. Comput. Surveys 29, 4 (1997), 366–427. doi:10.1145/ 267580.267590

    1997

  97. [2024]

    Advances in Neural Information Processing Systems 37 (2024), 52040–52094

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37 (2024), 52040–52094

    2024

  98. [2025]

    LADYBUG: an LLM Agent DeBUGger for data-driven applications.. In EDBT. 1082–1085

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.