Pith. sign in

REVIEW 4 major objections 59 references

A learned supervisor can replace human oversight of coding agents and raise test coverage by about 30%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 01:13 UTC pith:VWFDPOZO

load-bearing objection Solid empirical SE paper: a three-arm LinUCB supervisor that actually improves agentic test generation, though the abstract oversells the single-shot baseline and the residual multi-iteration edge is modest. the 4 major comments →

arxiv 2607.08983 v1 pith:VWFDPOZO submitted 2026-07-09 cs.SE cs.AI

SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation

Sijia Gu , Noor Nashid , Ali Mesbah This is my paper
classification cs.SE cs.AI
keywords test generationcoding agentscontextual banditLinUCBcode coverageclass testabilityprogram analysislazy generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Coding agents that write unit tests still stop early and avoid hard logic, so people have to keep watching them and telling them what to try next. That human bottleneck cancels most of the automation gain. This paper claims you can replace that human with an automated supervisor that watches only a few numbers—class size, complexity, coupling, current coverage, and remaining uncovered complexity—and at each step chooses among three moves: keep generating normally, invoke deeper path analysis, or stop. The supervisor is trained online as a contextual bandit so it learns which move pays off for the current class state while charging a cost for tokens spent. On a suite of complex Java classes the same agent, when supervised this way, reaches substantially higher line and branch coverage (and higher mutation score) than either the unsupervised agent or leading non-agent pipelines, and the policy automatically shortens or lengthens its effort depending on how capable and expensive the underlying agent is.

Core claim

Framing test-generation supervision as a three-action contextual bandit over static class-testability metrics and live coverage lets an automated supervisor replace human-in-the-loop steering, raising line and branch coverage of coding agents by roughly thirty percent relative to the unsupervised agent while keeping token cost under control and outperforming non-agentic baselines.

What carries the argument

SCATE’s LinUCB contextual bandit: a linear upper-confidence-bound policy that maps a seven-dimensional context (log-scaled LOC, WMC, RFC, missed complexity, plus line and branch coverage) onto the three macro-actions Default, Analysis, and Stop, updated online from a reward that trades relative coverage gain for API cost.

Load-bearing premise

That a handful of size, complexity, coupling and coverage numbers is enough for a linear bandit to learn a general routing rule that works on new classes without seeing their source code.

What would settle it

Train the same LinUCB policy on the 40-class warm-up set, then measure coverage and early-stop decisions on a fresh set of complex classes drawn from projects outside Defects4J; if the supervised agent no longer beats the unsupervised multi-iteration baseline by a statistically significant margin, the claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Human monitoring loops for agentic test generation can be removed once a short online warm-up has calibrated the policy.
  • The same supervisor automatically spends fewer iterations on stronger, more expensive agents and more analysis on weaker, cheaper ones.
  • Pre-trained policies can be shipped and reused for new agents or models without re-deriving project-specific rules.
  • Macro-level routing (continue / analyse / stop) is sufficient; micro-managing individual methods or prompts is unnecessary for large coverage gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same bandit-over-testability pattern could supervise other agentic SE tasks—bug localization, repair, or documentation—where agents also quit early on hard instances.
  • If the feature set is incomplete, simply adding a few more cheap static metrics (e.g., nesting depth or number of exceptional paths) may restore transfer without changing the rest of the architecture.
  • Cost-aware early stopping may become a standard plug-in layer once commercial coding agents expose token-usage logs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. SCATE formulates automated supervision of coding agents for unit-test generation as a contextual bandit (LinUCB) over a seven-dimensional state of static class testability metrics (LOC, WMC, RFC) and dynamic coverage signals, with a three-action macro space {Default, Analysis, Stop}. The Analysis arm is backed by a custom MCP program-analysis tool that batches high-priority uncovered paths and external calls. On an 80-class Defects4J evaluation set of complex classes, SCATE with GEMINI-CLI reports 82.8% line / 74.7% branch coverage and 64.7% mutation score, statistically outperforming single-shot and 8-iteration unsupervised agent loops as well as the non-agentic PANTA and SYMPROMPT baselines under a shared model; with CLAUDE CODE the policy adapts toward early stopping and higher final coverage. The paper claims that this adaptive, cost-aware routing replaces human-in-the-loop supervision and mitigates lazy generation.

Significance. If the residual gains over equal-budget unsupervised loops and the agent-adaptive behavior hold under broader conditions, SCATE is a useful systems contribution: it is the first explicit adaptive supervisor for coding agents in test generation, is agent-agnostic, and couples a standard contextual-bandit formulation with a practical MCP analysis tool and an online train/eval protocol. Strengths include a clear experimental design (shuffled 40/80 split with comparable complexity, warm-up then α-decay, Wilcoxon tests, Pitest mutation scores, two agents, two non-agentic baselines on the same model) and explicit reporting of cost and trajectory length. The work is empirical rather than circular: rewards are measured coverage deltas and logged token cost, and the supervisor never trains on source text. These elements make the paper a credible step toward cost-effective agentic testing if the magnitude and source of the gains are stated carefully.

major comments (4)
  1. Abstract and §I lead with “32.3% higher line coverage and 30.9% higher branch coverage than the agent-only baseline.” Table II shows that baseline is single-shot Agent (Default) at 50.5%/43.8%. Against the equal-budget 8-iteration unsupervised loops that match SCATE’s mean trajectory length (~7.9), absolute gains shrink to +5.5%/+4.5% (Default-8) and +8.1%/+7.2% (Analysis-8). The residual differences are statistically significant, but the central marketing claim overstates the practical value of adaptive supervision relative to simply iterating the agent. The abstract, contribution list, and conclusion should lead with the multi-iteration comparisons and report effect sizes against those controls.
  2. Table II and RQ2 establish that dynamic interleaving of Default/Analysis/Stop beats homogeneous 8-iteration loops, but there is no ablation against a simple non-learned supervisor that uses the same action space and MCP tool under fixed rules (e.g., Default until coverage plateau, then Analysis, then Stop on zero gain). Without that control, it remains unclear how much of the residual edge is due to LinUCB learning versus the action space, the MCP tool, the rollback guardrail, or the multi-iteration budget itself. A rule-based or random-routing ablation on the same 80 classes is needed to support the claim that the contextual-bandit policy is the load-bearing ingredient.
  3. Section II-A3 and III-A2 introduce free parameters (κ=0.5, ω=1, α schedule 1.5→0.2, K_max=10, class filters WMC≥35 or RFC≥55, MCP batch size 10) with limited or no sensitivity analysis. The reward (Eq. 3) and relative-gain formulation (Eq. 2) are central to when Stop is preferred and how late-stage coverage is valued. If residual gains over equal-budget baselines are sensitive to these choices, the generality of the learned routing policy is overstated. At minimum, report sensitivity of final coverage/cost to κ, ω, and K_max on the evaluation set, and justify the class-selection thresholds relative to the claimed project-independent transfer.
  4. Section II-A1 / Eq. (1): the seven-dimensional linear context (normalized LOC, WMC, RFC, Cov_L, Cov_B, missed_complexity plus intercept) is assumed rich enough for project-independent transfer after a 40-class online phase. RQ1 shows sensible correlations (Analysis and longer trajectories on higher-complexity classes), but there is no diagnostic of whether LinUCB’s linear assumption or the feature set misses important difficulty factors (e.g., exception density, mocking depth, dependency graph structure). A short feature-ablation or residual-error analysis on misrouted classes would strengthen the claim that this context supports a general routing rule.

Circularity Check

0 steps flagged

No circularity: SCATE is an empirical contextual-bandit system whose rewards and coverage outcomes are measured after actions, not defined by the policy parameters.

full rationale

The paper's load-bearing chain is empirical, not definitional. Contextual features (LOC, WMC, RFC, Cov_L, Cov_B, missed_complexity; Eq. 1) are computed from static CFGs and JaCoCo reports; the action space {Default, Analysis, Stop} is a fixed routing choice; the reward (Eq. 3) is an explicit function of measured relative coverage gain, covered-complexity increase, and logged token cost after the agent runs. LinUCB then updates A_a and b_a from those observed (x_t, a_t, r_t) triples. Reported line/branch coverage and mutation scores on the held-out 80-class set are independent measurements of the generated suites, not algebraic rearrangements of the fitted θ̂_a. Training and evaluation classes are disjoint after shuffle; the supervisor never trains on source text. Self-citations (e.g., PANTA [3] by the same authors) appear only as comparison baselines, not as uniqueness theorems or load-bearing premises that force the result. Hyperparameters (ω=1, κ=0.5, α-decay, K_max=10) are design choices that affect policy behavior but do not make the coverage numbers true by construction. Concerns about single-shot vs. multi-iteration baselines are experimental-fairness issues, not circularity. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on a small set of hand-chosen hyperparameters, the standard LinUCB linear-reward assumption, and the domain premise that a few classical testability metrics plus coverage suffice as context. No new physical entities are postulated; the main invented artifacts are the supervisor policy, the three-arm action design, the reward formula, and the MCP analysis tool.

free parameters (6)
  • failure penalty κ = 0.5
    Fixed at 0.5 after empirical tuning to discourage zero-gain actions while preserving exploration (Section II-A3).
  • cost weight ω = 1
    Defaulted to 1 to balance complexity-scaled coverage gain against USD token cost (Section II-A3).
  • LinUCB exploration α schedule = 1.5 → 0.2 (eval fixed 0.2)
    Warm-up uses random actions; α decays from 1.5 to 0.2 on training classes, then fixed at 0.2 for evaluation (Section III-A2).
  • K_max trajectory length = 10
    Hard cap of 10 iterations per class chosen to bound cost; average realized length is 7.9 for Gemini (Section II-C, III-B).
  • class selection thresholds WMC≥35 or RFC≥55 = WMC≥35 or RFC≥55
    Used to retain only complex classes for evaluation, following external thresholds; shapes the entire reported dataset (Section III-A1).
  • MCP path batch size = 10
    At most 10 under-covered methods with one critical path each per tool call, chosen from empirical observation (Section II-B2).
axioms (4)
  • standard math Expected reward of each action is linear in the 7-dimensional context vector (LinUCB assumption).
    Stated in Section II-A4 following Li et al. 2010; enables closed-form ridge updates.
  • domain assumption LOC, WMC, and RFC (plus coverage and missed_complexity) are adequate proxies for the difficulty coding agents face when generating tests.
    Section II-A1 cites Bruntink et al. and classical OO metrics; the supervisor never sees source text.
  • ad hoc to paper A three-action macro space {Default, Analysis, Stop} is sufficient; finer tool or prompt micro-management is unnecessary for cost-effective coverage.
    Explicit design choice in Section II-A2 to keep the bandit tractable.
  • ad hoc to paper Relative coverage gain (Eq. 2) plus logarithmic complexity multiplier correctly values late-stage hard coverage over early easy gains.
    Reward design in Section II-A3; not derived from an external optimality theorem beyond optimal-stopping intuition for Stop reward = 0.
invented entities (3)
  • SCATE supervisor (LinUCB policy over testability+coverage context) no independent evidence
    purpose: Replace human-in-the-loop routing of coding agents during unit test generation.
    Core contribution; existence and value are justified only by the paper’s own experiments.
  • SCATE MCP get_class_analysis tool no independent evidence
    purpose: Supply batched uncovered paths and external calls to the agent under the Analysis action.
    Custom program-analysis server described in Section II-B2; not an independently validated product.
  • Action-dependent reward r_t (Eq. 3) no independent evidence
    purpose: Trade off relative coverage gain, covered complexity, token cost, and failure penalty for bandit updates.
    Hand-designed objective; reported gains depend on this definition of success.

pith-pipeline@v1.1.0-grok45 · 23016 in / 3542 out tokens · 41545 ms · 2026-07-13T01:13:55.670538+00:00 · methodology

0 comments
read the original abstract

While autonomous coding agents have significantly advanced automated test generation, they remain fundamentally limited by lazy generation, a phenomenon where agents prematurely terminate tasks and systematically avoid complex programmatic logic, resulting in inadequate code coverage. Currently, mitigating this premature termination requires continuous human-in-the-loop supervision. This heavy reliance on human intuition creates a bottleneck that negates the efficiency gains of automated generation. We propose SCATE, a framework for adaptive, automated supervision of coding agents that replaces human intervention during test generation. By formulating supervision as a contextual bandit problem, SCATE learns to select the most promising testing actions based on the current coverage and class testability metrics, maximizing coverage gains while minimizing wasted generation effort. Our empirical evaluation demonstrates that SCATE integrates seamlessly with different coding agents. When applied to GEMINI-CLI, it achieves 32.3% higher line coverage and 30.9% higher branch coverage than the agent-only baseline. A comparison with CLAUDE CODE confirms the framework dynamically adapts its policy to optimize each agent's unique strengths. SCATE also consistently outperforms state-of-the-art non-agentic approaches across all metrics.

Figures

Figures reproduced from arXiv: 2607.08983 by Ali Mesbah, Noor Nashid, Sijia Gu.

Figure 1
Figure 1. Figure 1: An example of test generation by Gemini CLI [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SCATE. 1) Contextual Features: Generating comprehensive unit tests requires LLM-based coding agents to understand source code, including complex control flows and extensive external dependencies, within a limited context window [3], [2]. The inherent complexity of a target class directly influences both achievable code coverage and the effort required to improve it. More complex classes are typ… view at source ↗
Figure 3
Figure 3. Figure 3: Prompt template provided to a coding agent (condensed). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Class metrics across the dataset partitions. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Execution Dynamics of SCATE Generation Trajectories reward. The cumulative number of early-terminated classes steadily increases, ultimately reaching 32 of the 80 evaluation classes (40%) by Iteration 10. This aligns with the reward formulation described in Section II-A3, which balances cov￾erage gains against computational cost. Because the API cost of gemini-3.1-flash-lite remains low (below $0.10 USD pe… view at source ↗
Figure 8
Figure 8. Figure 8: Comparing SCATE with PANTA and SYMPROMPT lead to longer trajectories (7.9 iterations on average) and more frequent Analysis usage (33.8%). Ultimately, SCATE paired with CLAUDE CODE achieves higher final coverage (90.3% line and 84.9% branch) alongside a 75.0% mutation score at a total cost of $1.143, while with GEMINI-CLI it reaches competitive coverage (82.8% line and 74.7% branch) and a 64.7% mutation sc… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 7 linked inside Pith

  1. [1]

    HITS: High-coverage llm-based unit test generation via method slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: High-coverage llm-based unit test generation via method slicing,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineer- ing, 2024, pp. 1258–1268

    2024

  2. [2]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 951–971, 2024

    2024

  3. [3]

    LLM test generation via iterative hybrid program analysis,

    S. Gu, N. Nashid, and A. Mesbah, “LLM test generation via iterative hybrid program analysis,” inIEEE/ACM International Conference on Software Engineering (ICSE), IEEE. ACM, 2026, p. 12 pages

    2026

  4. [4]

    Coverup: Effective high cov- erage test generation for python,

    J. Altmayer Pizzorno and E. D. Berger, “Coverup: Effective high cov- erage test generation for python,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 2897–2919, 2025

    2025

  5. [5]

    Leveraging large language models for enhancing the understandability of generated unit tests,

    A. Deljouyi, R. Koohestani, M. Izadi, and A. Zaidman, “Leveraging large language models for enhancing the understandability of generated unit tests,” in2025 IEEE/ACM 47th International Conference on Soft- ware Engineering (ICSE). IEEE Computer Society, 2024, pp. 392–404

    2024

  6. [6]

    Aster: Natural and multi-language unit test generation with llms,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “Aster: Natural and multi-language unit test generation with llms,” in2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2025, pp. 413– 424

    2025

  7. [7]

    Codex CLI,

    OpenAI, “Codex CLI,” https://developers.openai.com/codex/cli, 2026, accessed: 2026-01-21

    2026

  8. [8]

    Claude Code,

    Claude, “Claude Code,” https://www.claude.com/product/claude-code, 2026, accessed: 2026-01-21

    2026

  9. [9]

    Gemini CLI: Code assist command-line interface,

    Google Cloud, “Gemini CLI: Code assist command-line interface,” https: //cloud.google.com/gemini/docs/codeassist/gemini-cli, 2026, accessed: 2026-05-21

    2026

  10. [10]

    Qwen Code: Large language models for code,

    Alibaba Cloud, “Qwen Code: Large language models for code,” https: //qwenlm.github.io/qwen-code-docs/en/, 2026, accessed: 2026-01-21

    2026

  11. [11]

    Can coding agents be general agents?

    M. Ivanov, A. Rana, and G. Prabhakaran, “Can coding agents be general agents?”arXiv preprint arXiv:2604.13107, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” inPro- ceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

    2014

  13. [13]

    A contextual-bandit approach to personalized news article recommendation,

    L. Li, W. Chu, J. Langford, and R. E. Schapire, “A contextual-bandit approach to personalized news article recommendation,” inProceedings of the 19th international conference on World wide web, 2010, pp. 661– 670

    2010

  14. [14]

    R. S. Sutton, A. G. Bartoet al.,Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1

    1998

  15. [15]

    Reinforcement learning in practice: Opportunities and chal- lenges,

    Y . Li, “Reinforcement learning in practice: Opportunities and chal- lenges,”arXiv preprint arXiv:2202.11296, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    A metrics suite for object oriented design,

    S. R. Chidamber and C. F. Kemerer, “A metrics suite for object oriented design,”IEEE Transactions on software engineering, vol. 20, no. 6, pp. 476–493, 1994

    1994

  17. [17]

    An empirical study into class testability,

    M. Bruntink and A. van Deursen, “An empirical study into class testability,”Journal of systems and software, vol. 79, no. 9, pp. 1219– 1232, 2006

    2006

  18. [18]

    A survey on software testability,

    V . Garousi, M. Felderer, and F. N. Kılıçaslan, “A survey on software testability,”Information and Software Technology, vol. 108, pp. 35–64, 2019

    2019

  19. [19]

    Increasing the effectiveness of automatically generated tests by improving class observability,

    G. Galindo-Gutierrez, J. P. S. Alcocer, N. Jimenez-Fuentes, A. Bergel, and G. Fraser, “Increasing the effectiveness of automatically generated tests by improving class observability,” in2025 IEEE/ACM 47th Inter- national Conference on Software Engineering (ICSE). IEEE Computer Society, 2025, pp. 693–693

    2025

  20. [20]

    Predicting different levels of the unit testing effort of classes using source code metrics: a multiple case study on open-source software,

    F. Toure, M. Badri, and L. Lamontagne, “Predicting different levels of the unit testing effort of classes using source code metrics: a multiple case study on open-source software,”Innovations in Systems and Software Engineering, vol. 14, no. 1, pp. 15–46, 2018

    2018

  21. [21]

    A complexity measure,

    T. McCabe, “A complexity measure,”IEEE Transactions on Software Engineering, vol. SE-2, no. 4, pp. 308–320, 1976

    1976

  22. [22]

    Jacoco java code coverage library,

    EclEmma, “Jacoco java code coverage library,” https://www.jacoco.org/ jacoco/, 2026, accessed: May 4, 2026

    2026

  23. [23]

    Numerical data: Normalization,

    Google for Developers, “Numerical data: Normalization,” https://developers.google.com/machine-learning/crash-course/ numerical-data/normalization, 2026, accessed: May 4, 2026

    2026

  24. [24]

    Model context protocol specification,

    Anthropic, “Model context protocol specification,” https: //modelcontextprotocol.io, 2026, accessed: 2026-05-15

    2026

  25. [25]

    Optimal stopping and applications,

    T. S. Ferguson, “Optimal stopping and applications,” 2006, https://www. math.ucla.edu/~tom/Stopping/Contents.html

    2006

  26. [26]

    Dart code metrics,

    DCM, “Dart code metrics,” https://dcm.dev/docs/, 2026, accessed: 2026- 05-15

    2026

  27. [27]

    Contextual bandits analysis of linucb hybrid algorithm with movielens dataset,

    K. Foo, “Contextual bandits analysis of linucb hybrid algorithm with movielens dataset,” https://kfoofw.github.io/ contextual-bandits-linear-ucb-hybrid/, 2026, accessed: 2026-05-15

    2026

  28. [28]

    Linucb: A comprehensive guide for 2025,

    ShadeCoder, “Linucb: A comprehensive guide for 2025,” https: //www.shadecoder.com/topics/linucb-a-comprehensive-guide-for-2025, 2026, accessed: 2026-05-15

    2025

  29. [29]

    Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms,

    Q. H. Nguyen, T. Dao, D. C. Hoang, J. Decugis, S. Manchanda, N. V . Chawla, and K. D. Doan, “Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms,”arXiv preprint arXiv:2407.10834, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Gemini 3.1 flash-lite is now generally available on gem- ini enterprise agent platform,

    Google Cloud, “Gemini 3.1 flash-lite is now generally available on gem- ini enterprise agent platform,” https://cloud.google.com/blog/products/ ai-machine-learning/gemini-3-1-flash-lite-is-now-generally-available, 2026, accessed: 2026-06-04

    2026

  31. [31]

    Build resilient llm applications on vertex ai and reduce 429 errors,

    ——, “Build resilient llm applications on vertex ai and reduce 429 errors,” https://cloud.google.com/blog/products/ai-machine-learning/ reduce-429-errors-on-vertex-ai, 2026, accessed: 2026-06-04

    2026

  32. [32]

    D. L. Poole and A. K. Mackworth,Artificial Intelligence: foundations of computational agents. Cambridge University Press, 2010, https: //artint.info/3e/html/ArtInt3e.Ch13.S6.html

    2010

  33. [33]

    LLM test generation via iterative hybrid program analysis,

    Panta, “LLM test generation via iterative hybrid program analysis,” https://github.com/PANTA-TestAutomation/Panta, 2025, accessed: Au- gust 20, 2025

    2025

  34. [34]

    Evosuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: Automatic test suite generation for object-oriented software,” in19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE ’11. ACM, 2011, p. 416–419

    2011

  35. [35]

    Evosuite: On the challenges of test case generation in the real world,

    ——, “Evosuite: On the challenges of test case generation in the real world,” in2013 IEEE sixth international conference on software testing, verification and validation. IEEE, 2013, pp. 362–369

    2013

  36. [36]

    Pynguin: Automated unit test generation for python,

    S. Lukasczyk and G. Fraser, “Pynguin: Automated unit test generation for python,” in2022 IEEE/ACM 44th International Conference on Soft- ware Engineering: Companion Proceedings (ICSE-Companion), 2022, pp. 168–172

    2022

  37. [37]

    Automated unit test generation for python,

    S. Lukasczyk, F. Kroiß, and G. Fraser, “Automated unit test generation for python,” inSearch-Based Software Engineering: 12th International Symposium, SSBSE 2020, Bari, Italy, October 7–8, 2020, Proceedings

    2020

  38. [38]

    Springer, 2020, pp. 9–24

    2020

  39. [39]

    On learning meaningful assert statements for unit test cases,

    C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert statements for unit test cases,” inProceedings of the ACM/IEEE 42nd International Conference on Software Engineer- ing, ser. ICSE ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 1398–1409

    2020

  40. [40]

    Unit test case generation with transformers and focal context,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,”arXiv preprint arXiv:2009.05617, 2020

    Pith/arXiv arXiv 2009

  41. [41]

    Generating accurate assert statements for unit test cases using pretrained trans- formers,

    M. Tufano, D. Drain, A. Svyatkovskiy, and N. Sundaresan, “Generating accurate assert statements for unit test cases using pretrained trans- formers,” in3rd ACM/IEEE International Conference on Automation of Software Test. ACM, 2022, p. 54–64

    2022

  42. [42]

    Automated unit test improvement using large language models at meta,

    N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” inCompanion Proceedings of the 32nd ACM International Conference on the Founda- tions of Software Engineering, 2024, pp. 185–196

    2024

  43. [43]

    Large language models are few- shot testers: Exploring llm-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few- shot testers: Exploring llm-based general bug reproduction,” in2023 11 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2312–2323

    2023

  44. [44]

    Retrieval-Based Prompt Selec- tion for Code-Related Few-Shot Learning,

    N. Nashid, M. Sintaha, and A. Mesbah, “Retrieval-Based Prompt Selec- tion for Code-Related Few-Shot Learning,” inIEEE/ACM International Conference on Software Engineering (ICSE), IEEE. IEEE Press, 2023, pp. 2450–2462

    2023

  45. [45]

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 919–931

    2023

  46. [46]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

    2024

  47. [47]

    Fusing llms and genetic algorithm for high-quality unit test generation,

    Y . Fu, X. Gao, B. Qi, Y . Yuan, and H. Sun, “Fusing llms and genetic algorithm for high-quality unit test generation,”ACM Transactions on Software Engineering and Methodology, 2026

    2026

  48. [48]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2023

    2023

  49. [49]

    On the evaluation of large language models in unit test generation,

    L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wanget al., “On the evaluation of large language models in unit test generation,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 1607–1619

    2024

  50. [50]

    Using large language models to generate junit tests: An empirical study,

    M. L. Siddiq, J. C. Da Silva Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V . Carvalho Lopes, “Using large language models to generate junit tests: An empirical study,” inProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 2024, pp. 313–322

    2024

  51. [51]

    Testeval: Benchmarking large language models for test case generation,

    W. Wang, C. Yang, Z. Wang, Y . Huang, Z. Chu, D. Song, L. Zhang, A. R. Chen, and L. Ma, “Testeval: Benchmarking large language models for test case generation,”arXiv preprint arXiv:2406.04531, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    No more manual tests? evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y . Lou, M. Liu, S. Ding, K. Wang, Y . Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,”arXiv preprint arXiv:2305.04207, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Empirical study on test generation using github copilot,

    K. El Haji, “Empirical study on test generation using github copilot,” Master’s thesis, Delft University of Technology, 2023

    2023

  54. [54]

    A large-scale empirical study on fine-tuning large language models for unit testing,

    Y . Shang, Q. Zhang, C. Fang, S. Gu, J. Zhou, and Z. Chen, “A large-scale empirical study on fine-tuning large language models for unit testing,” Proc. ACM Softw. Eng., vol. 2, no. ISSTA, Jun. 2025

    2025

  55. [55]

    Navigating the labyrinth: Path-sensitive unit test generation with large language mod- els,

    D. Liao, X. Yin, S. Pan, C. Ni, Z. Xing, and X. Sun, “Navigating the labyrinth: Path-sensitive unit test generation with large language mod- els,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Press, 2025, p. 687–699

    2025

  56. [56]

    Mutation-guided unit test generation with a large language model,

    G. Wang, Q. Xu, L. Briand, and K. Liu, “Mutation-guided unit test generation with a large language model,”IEEE Transactions on Software Engineering, 2026

    2026

  57. [57]

    Llm bandit: Cost-efficient llm generation via preference- conditioned dynamic routing,

    Y . Li, “Llm bandit: Cost-efficient llm generation via preference- conditioned dynamic routing,”arXiv preprint arXiv:2502.02743, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Mixllm: Dynamic routing in mixed large language models,

    X. Wang, Y . Liu, W. Cheng, X. Zhao, Z. Chen, W. Yu, Y . Fu, and H. Chen, “Mixllm: Dynamic routing in mixed large language models,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 10 912– 10 922

    2025

  59. [59]

    Stop wasting your tokens: Towards efficient runtime multi-agent systems,

    F. Lin, S. Chen, R. Fang, H. Wang, and T. Lin, “Stop wasting your tokens: Towards efficient runtime multi-agent systems,”arXiv preprint arXiv:2510.26585, 2025. 12

    arXiv 2025