{"id":"23a99d02-7e57-4722-ba0e-416b7931e14c","arxiv_id":"2608.08413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An analysis of 2,572 tests and 10 practitioner interviews shows that agent testing is mostly unit-level, mock-heavy, and built on simple example data.","lead":"This paper measures how developers actually test LLM-based agent applications, using 2,572 manually labeled tests plus interviews with 10 practitioners. It finds testing is dominated by narrow unit tests, heavy mocking, and simple example inputs, with little focus on realistic complex behavior or non-functional checks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline percentages rest on 12 projects with one project (Upsonic) providing 30% of tests; project-level clustering and selection filters could dominate the 61.8% unit-test claim, but this is disclosed and internally consistent, so the concern is about generality rather than soundness.","rationale":"The reader's weakest assumption was that the final 12 projects and 10 IBM practitioners are representative of LLM-based agent testing practice; my analysis agrees and identifies the same load-bearing concern. I verified the paper's internal consistency: the percentages in Figure 2(a), Table 2, and the text are mutually consistent, and the threats-to-validity section explicitly discloses the selection filters and single-company interviews. What elevates this from a routine limitation to a load-bearing concern is that the central claim is a distributional statement about practice, and the sampling unit is the repository while the headline proportion is computed over tests. With one project contributing 770 of 2,572 tests and another 381, the test-level proportion is not protected by project diversity. The paper does not report project-level distributions for the headline finding, so the reader cannot assess how much of the 61.8% is a within-project phenomenon, e.g., Upsonic's test style or MCP-tool-heavy structure. This does not undermine the study as an exploratory empirical characterization; the taxonomy, dataset, and qualitative interview findings stand. However, because the abstract states the dominance of unit testing as a general fact about the field, the lack of project-level analysis is the appropriate target of a condition: either add this analysis and tighten the generalizability statement, or keep the claim but present it as true of the studied subset. Conditional accept is thus the right verdict; the paper is sound but needs a generalization-level revision before its central claim can be taken at face value.","tokens_in":23603,"tokens_out":1718,"duration_ms":16658,"concrete_test":"Recompute the headline percentages (unit vs. integration vs. NFR) at the project level: for each of the 12 projects, compute the proportion of unit-labeled tests, then report the median, min, max, and a bootstrap or clustered confidence interval treating project as the sampling unit. If the 61.8% point estimate falls outside a 95% CI computed with project-level clustering, or if the median project-level unit proportion differs by more than 10 percentage points from 61.8%, the abstract's generalization should be softened to 'in the studied projects' or the sample should be reweighted by project.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative claim, Finding 1, is that 61.8% of tests are unit-level and 34.6% interaction-level, and the abstract generalizes this to 'testing of LLM-based agent applications' broadly. The load-bearing assumption is that the 2,572 labeled tests are a representative sample of agent-testing practice. Two features in the pipeline make this insecure. First, Phase 4 excludes repositories with fewer than 20 test modules and excludes framework and example repositories, so the final 12 projects are heavily filtered. Second, the test distribution is highly concentrated: Upsonic contributes 770 of 2,572 tests (about 30%) and 66.5% of all NFR tests. No project-level breakdown of the 61.8% unit figure is given, so a reader cannot tell whether one or two high-volume projects dominate it. With n=12 projects, a handful of projects can move the headline proportion by double digits. The practitioner interview evidence (10 IBM employees, all from one company) does not independently corroborate the open-source percentages; it is qualitative. This is a recognized limitation (Section 8, External Validity), and the study is methodologically transparent, but the abstract's broad claim outruns the sample. The concern is not that the numbers are wrong for the studied sample; it is that the studied sample's selection and clustering are not sufficient to support the population-level generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of testing practices for LLM-based agent applications. The authors mine GitHub for Python projects using agent frameworks (LangChain, LangGraph, MCP, etc.), statically detect agents and tools, filter and manually label a final dataset of 2,572 test methods from 240 test modules in 12 projects, and complement this with structured interviews of 10 IBM practitioners. They derive a taxonomy of 23 testing patterns, analyze testing levels and types, structural complexity, and practitioner perceptions, and propose a research roadmap. The central claim is that current agent testing is dominated by narrowly scoped unit tests with limited interaction coverage, unrealistic test data, heavy mocking, shallow assertions, low structural complexity, and little non-functional testing.","tokens_in":23833,"tokens_out":3178,"duration_ms":33316,"significance":"If the findings hold, this is a timely and useful contribution: it provides some of the first quantitative evidence about how LLM-based agent applications are actually tested, and the 23-pattern taxonomy gives the community a shared vocabulary. The paper's strengths include a transparent data-collection pipeline, a released artifact, explicit raw counts in Table 2, and a high inter-rater agreement (Cohen's kappa 0.92). The main weakness is that the headline percentages rest on only 12 projects with strong concentration (Upsonic alone contributes about 30% of the labeled tests), and the practitioner interviews come from a single company. These limitations are disclosed in Section 8, but the abstract and several findings state population-level conclusions that outrun the sample.","major_comments":[{"comment":"The headline percentages (61.8% unit-level, 34.6% interaction, 7.5% NFR) are reported only at the aggregate level. With 12 projects, and with Upsonic contributing 770 of 2,572 tests (about 30%) and 66.5% of all NFR tests, a small number of projects can move the aggregate by double digits. Please add a per-project breakdown of testing levels and NFR focus, and report a sensitivity analysis (e.g., excluding Upsonic or using project-level weights). Without this, the abstract's population-level claim that 'testing of LLM-based agent applications is dominated by...' is not adequately supported.","section":"§3.2, Finding 1; Table 1"},{"comment":"The practitioner evidence consists of ten interviews, all with employees of a single organization (IBM). The abstract and §6 describe industry practice as placing 'greater emphasis on non-functional testing than open-source projects,' but a single-company sample cannot independently corroborate the open-source measurements. Please reframe §6 as a single-organization qualitative perspective and adjust the abstract to avoid implying independent confirmation of the repository findings.","section":"§6 and Abstract"},{"comment":"The filtering pipeline excludes repositories with fewer than 20 test modules, as well as example and framework repositories, but the manuscript does not quantify how many candidate projects were removed at each exclusion step or how the retained 12 projects compare with the excluded ones (164 repositories with relevant test modules before filtering). Reporting these numbers and the characteristics of excluded projects would help readers assess selection bias. This is particularly relevant because the same filters may systematically select mature, well-tested projects and thereby shape the observed patterns.","section":"§2, Phase 4"}],"minor_comments":[{"comment":"Figures 2(b) and 2(c) show framework and version breakdowns, but the text does not report the number of tests per framework or version; adding sample sizes would let readers judge the reliability of the apparent associations.","section":"Figure 2"},{"comment":"The note explains that tests can belong to multiple patterns, but the subcategory percentages (e.g., 'Mock some component' 21.66% and 'Mock entire agent' 22.28%) are not mutually exclusive and do not sum to the parent category. Please clarify whether percentages are computed over all tests or over the parent-category tests.","section":"Table 2"},{"comment":"The questionnaire jumps from Q21 to Q23; if Q22 was intentionally omitted, please say so, otherwise renumber the questions.","section":"Table 3"},{"comment":"The comparison with the authors' prior Hamster study [69] is mentioned without any details about that study's method or dataset; since the reference is to an unpublished manuscript, please add a brief description or soften the comparison.","section":"§5"},{"comment":"The wording 'testing of LLM-based agent applications is dominated by...' appears in the abstract and Finding 1; consider qualifying it as 'in the studied sample' to match the evidence presented.","section":"Abstract and §3.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the most useful empirical characterization of agent-application testing practices I have seen, and it deserves a serious referee. But the abstract's broad claims outrun the sample that supports them, and the fix is straightforward.\n\nWhat is actually new: the paper manually labels 2,572 test methods from 240 modules, produces a 23-pattern taxonomy across fixtures, data, objectives, and assertions, and reports testing-level distributions and structural-complexity statistics. That is a real step beyond Hasan et al., which analyzed 759 unit test functions with 10 patterns. The released artifact and the kappa of 0.92 give the labeling credibility. The interviews, though limited to one company, add qualitative depth and mostly align with the repository findings.\n\nThe soft spots are real but not disqualifying. The final labeled sample is 12 projects after heavy filtering in Phase 4: Python-only, high-star, at least 20 test modules, examples and frameworks excluded. Upsonic alone contributes about 30% of the tests and 66.5% of the NFR tests. The paper does not give a project-level breakdown of the headline 61.8% unit-test figure, so you cannot tell whether one or two high-volume projects dominate it. With n=12, a handful of projects can move that proportion by double digits. The practitioner evidence is 10 people from IBM, which is a single-company convenience sample. These limitations are disclosed in Section 8, and the study is transparent about its pipeline, but the abstract's phrasing—\"testing of LLM-based agent applications is dominated by narrowly scoped unit tests\"—generalizes further than the evidence supports. No confidence intervals around the proportions would at least signal uncertainty.\n\nThe framework-effect finding (Finding 2) is intriguing but built on small per-project counts; treat it as suggestive. The comparison to the authors' own Hamster study for Java test complexity is contextual and does not affect the central claim. The related-work section fairly positions this against Hasan et al. and Kohl et al.\n\nWho gets value from this: anyone building agent-testing tools, benchmark designers, and SE researchers studying test practices for LLM-based systems. It is a reading-group paper, and I would cite it if I worked in this area.\n\nMy recommendation: accept for peer review. The right outcome is likely a major revision that either restricts the headline claims to the studied sample or adds project-level analyses showing the unit-test and NFR percentages are not artifacts of Upsonic's size. The dataset should be inspected during review. The paper's core contribution—a large, labeled, reusable test corpus and taxonomy—is solid and worth publishing.","headline":"The most substantial empirical look at how agent apps are actually tested, with a real dataset behind it, but the headline percentages rest on a 12-project sample with one project carrying a third of the tests.","tokens_in":24431,"tokens_out":1583,"would_cite":true,"duration_ms":17962,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most tests for LLM-based agents are narrow unit tests, a 2,572-test study finds","keywords":["agent testing","LLM-based agents","empirical software engineering","testing taxonomy","non-functional testing","test patterns","unit testing"],"falsifier":"Re-run the manual labeling on a random sample of 1,000 test methods drawn from agent repositories across languages and repository sizes; if more than half are interaction-level or non-functional tests, the claimed unit-test dominance would be overturned.","tokens_in":23358,"feed_emoji":"🧪","tokens_out":7021,"duration_ms":66899,"temperature":0.7,"pith_summary":"LLM-based agent applications are being tested far more narrowly than their complexity demands. By manually labeling 2,572 test methods from 240 modules in 12 open-source projects and interviewing 10 industry practitioners, the paper finds that 61.8% of tests exercise a single tool or agent in isolation, 34.6% target interactions, and only 7.5% address non-functional requirements like security or performance. Tests typically rely on simple example inputs, heavy mocking, and shallow assertions that rarely check reasoning or side effects. The paper argues this pattern holds across frameworks, whose abstraction level shapes how much integration testing developers do, and it extracts a 23-pattern taxonomy plus a research roadmap to close the gaps.","feed_headline":"Most LLM-agent tests are narrow unit tests, 2,572-test study finds","feed_subtitle":"Only 7.5% cover non-functional requirements; interviews and open-source data show the gaps.","key_machinery":"The load-bearing machinery is a four-dimensional classification scheme applied to 2,572 manually labeled test methods. Each test is classified along testing level (unit, module, integration, API), test type (functional vs non-functional), and 23 patterns grouped into test fixture, test data, test objective, and test assertion categories. The taxonomy turns qualitative impressions about shallow testing into countable fractions: 61.8% isolated, 34.6% interaction, 7.5% non-functional, 39.9% dummy data, 34.4% mocked agent or tool, and 8.7% reasoning assertions. The framework-abstraction comparison then explains part of the distribution, and the structural-complexity analysis (median 14 non-comment lines of code, median cyclomatic complexity 1) reinforces the shallowness claim.","core_discovery":"On its own terms, the paper establishes that current testing practice for LLM-based agent applications is dominated by narrowly scoped unit tests with limited coverage of complex interactions, realistic scenarios, and non-functional requirements. The evidence is a manually labeled dataset of 2,572 test methods from 240 modules across 12 open-source Python projects, plus structured interviews with 10 senior practitioners. Testing levels vary systematically with framework abstraction: applications built on higher-level tool protocols lean toward unit tests, middle-level orchestration frameworks balance unit and integration tests, and lower-level composition frameworks lean toward integration tests. Only 7.5% of tests target non-functional requirements, 39.9% use simple dummy data, 34.4% mock an agent or tool, and 8.7% assert on reasoning-related behavior. From these observations the paper derives a taxonomy of 23 testing patterns and a roadmap calling for formal test objectives, adequacy criteria for agentic abstractions, fault-based testing, and better testability foundations for agent frameworks.","pith_inferences":["A testable extension: use the paper's 23-pattern taxonomy as a checklist to measure whether new agent frameworks or testing tools shift the distribution toward interaction-level and non-functional tests.","The findings imply that improving benchmark scores on agent task suites is largely orthogonal to improving developer testing practice, so progress in one does not automatically translate into better-tested agent applications.","If the framework-abstraction correlation holds as a causal claim, then designing agent APIs with explicit seams for isolating, observing, and controlling agent behavior should increase module-level testing; this prediction could be tested by comparing test distributions across framework versions over time.","The paper's sample is Python-only open-source projects with at least 20 test modules; applying the same taxonomy to a broader sample (other languages, smaller projects, closed-source systems) would reveal whether the unit-test dominance is universal or an artifact of the selection criteria."],"forward_implications":["If correct, agent testing practice is systematically shallow relative to the complexity of agentic systems, so interaction, environment, and non-functional failures are likely under-detected in current test suites.","The framework correlation implies testability is partly a design property of agent frameworks: choosing a higher-level protocol shifts the testing burden toward end users who compose components, while lower-level frameworks force more integration testing on developers.","The near-absence of reasoning assertions indicates that reasoning quality is currently validated by manual observation or benchmarks rather than by automated tests, leaving a large gap in regression protection.","The lack of formal adequacy criteria means agent test suites cannot be objectively evaluated; coverage of tool selection, delegation patterns, and multi-step execution paths is left unmeasured.","Interview responses suggest industry emphasizes non-functional testing more than open-source projects do, but both share the same foundational gaps in adequacy metrics and formal test objectives."],"supporting_citations":[{"why":"the inter-rater agreement coefficient used to validate the reliability of the manual labeling of all 2,572 test methods","marker":"[19]"},{"why":"the LLM service powering the automated labeling agents that filter agent-relevant test modules from the broader corpus","marker":"[32]"},{"why":"the tool-interconnection protocol whose isolation boundaries define the unit-test-friendly framework end of the testing-level spectrum","marker":"[5]"},{"why":"a widely used agent-building framework whose characteristic usage patterns drive the repository-mining detection rules","marker":"[51]"},{"why":"the orchestration framework whose explicit coordination definitions push applications toward integration testing","marker":"[52]"},{"why":"the static analysis tool used to extract symbol tables and populate the code model for agent and tool detection","marker":"[54]"},{"why":"the prior empirical study of agent testing patterns that this work extends and contrasts with at finer granularity","marker":"[35]"},{"why":"the standard reference on xUnit test patterns that provides the conceptual basis for the fixture, data, objective, and assertion categories","marker":"[65]"},{"why":"the foundational description of mock objects, invoked to define and interpret the heavy mocking observed in agent tests","marker":"[60]"}],"fun_headline_variants":["92.5% of LLM-agent tests skip non-functional checks","LLM-agent tests lean on unit tests and heavy mocking","Study: agent tests rarely cover real interactions or failures","2,572 tests reveal: agent testing is shallow and narrow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 12 open-source projects and 10 interviewed practitioners are representative of how LLM-based agent applications are tested in general; if this sample is atypical, the reported percentages describe a narrow subset rather than the field.","fun_headline_variants_meta":{"raw":{"variants":["92.5% of LLM-agent tests skip non-functional checks","LLM-agent tests lean on unit tests and heavy mocking","Study: agent tests rarely cover real interactions or failures","2,572 tests reveal: agent testing is shallow and narrow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1855,"prompt_tokens":1055,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":731}},"tokens_in":671,"tokens_out":800,"duration_ms":7873,"temperature":1.0,"reasoning_tokens":731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:35:59.184737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the manual labeling on a random sample of 1,000 test methods drawn from agent repositories across languages and repository sizes; if more than half are interaction-level or non-functional tests, the claimed unit-test dominance would be overturned.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"a widely used agent-building framework whose characteristic usage patterns drive the repository-mining detection rules"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the orchestration framework whose explicit coordination definitions push applications toward integration testing"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the standard reference on xUnit test patterns that provides the conceptual basis for the fixture, data, objective, and assertion categories"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the foundational description of mock objects, invoked to define and interpret the heavy mocking observed in agent tests"}],"review_version":1}