Test Coverage Analysis of Agentic Pull Requests

Atish Kumar Dipongkor; Kevin Moran; Talank Baral; Wing Lam

REVIEW 3 major objections 6 minor 32 references

Test Coverage Analysis of Agentic Pull Requests

T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes that code changes made by AI coding agents and merged through pull requests are poorly exercised by tests, with existing tests covering only 61.5% of added executable lines in Java and 27.0% in Python, and agent-writt

desk verdict A genuinely new measurement of test coverage in agentic PRs, held back by a real but addressable validity gap in the Python zero-coverage statistic. read the letter →

arxiv 2607.18057 v1 pith:OTCNS3UG submitted 2026-07-20 cs.SE

Atish Kumar Dipongkor , Talank Baral , Wing Lam , Kevin Moran This is my paper · ORCID

classification cs.SE

keywords AIcodingagentstestcoveragepullrequestsdiffempiricalsoftwareengineeringautomatedtestingmeasurement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that code changes introduced by autonomous AI coding agents are, at merge time, far less exercised by tests than a green CI run implies. Analyzing 4,882 pull requests produced by five coding agents across Java and Python, it measures 'diff coverage': the fraction of added executable lines that any test executes. The central findings are that existing tests cover only 61.5% of changed lines in Java and 27.0% in Python, that 64.8% of Python PRs have no changed line executed by any existing test, and that agent-written tests add coverage of the agent's own changes in only a minority of PRs (35.9% Java, 22.5% Python). The paper argues this means a passing test suite is a weak safety signal for agentic code, and that error-handling constructs (try-catch and throw) are the most consistently untested category in both languages.

What carries the argument

The argument rests on a diff-coverage pipeline. For each PR it reconstructs the patch from base to head commit, separates test-file changes from code-under-test changes, runs the repository's full test suite with and without the agent's test changes (by reverse-applying the test-only patch), and intersects the inventory of added executable lines — each labeled by syntactic construct via source-code parsing — with line-level coverage reports produced by standard Java and Python coverage tools. The carrying metric is PR-level diff coverage: the ratio of covered added lines to total added lines, aggregated by summing covered and missed lines across changed files.

What would settle it

A replication that independently runs the same PRs' test suites (or compares a sample against an external coverage service) and finds markedly higher diff-coverage numbers, or discovers that most zero-coverage Python PRs have no runnable tests at all, would undercut the central claim.

Watch

Extended reading notes

Core claim

The central discovery is a measurement: for 213 Java and 1,664 Python merged agentic PRs, existing tests execute only 61.5% (Java) and 27.0% (Python) of the executable lines agents add, and in 64.8% of Python PRs none of the added lines are executed. When agents do include tests, those tests raise coverage on average (+15.6 percentage points Java, +9.6 Python), but the gain comes from a minority: only 35.9% of Java and 22.5% of Python Code + Tests PRs show any coverage gain on the agent's own changes. Error-handling lines are the weakest spot: newly added throw statements are missed in 67.5% of Java and 82.3% of Python cases, and try-catch block lines are missed in 86.0% of Java and 81.0% of

Load-bearing premise

The coverage figures depend on the assumption that running each repository's test suite with the assumed runner and coverage tool at the PR head yields a valid executable-line coverage report, and that reverse-applying the test-only patch cleanly isolates the existing tests' coverage.

Editorial extensions

If this is right

A green CI run on an agentic PR cannot be interpreted as evidence that the change is tested; teams should treat diff coverage as a separate, load-bearing quality signal.
Coding agents need a coverage-aware feedback loop that checks whether their own added lines are exercised by their own tests before submitting a PR.
Error-handling constructs (try-catch, throw/raise) are the highest-yield target for agent test generation in both Java and Python.
Agent benchmarks that reward passing existing tests without measuring diff coverage will continue to reward under-tested changes.
Repositories adopting agentic workflows could use minimum diff-coverage thresholds as a practical quality gate on merged PRs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the coverage pattern holds beyond this dataset, the 64.8% zero-coverage Python figure likely mixes 'repository has no runnable tests' with 'tests do not reach the change'; separating those two would sharpen the prescription for agent designers.
A direct extension would measure diff coverage on human-written PRs from the same repositories; the gap between human and agent coverage would estimate how much of the shortfall is attributable to the agent rather than to project testing culture.
The construct-level miss rates suggest a targeted experiment: prompt agents with explicit error-handling test requirements and measure whether try-catch miss rates drop while other categories stay flat, isolating whether the weakness is a prompting problem or a model capability limit.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is a first quantified look at whether agent-written code in merged PRs is actually exercised by tests. Nobody else has measured diff coverage in agentic PRs at this scale, and the authors separate coverage by existing tests from coverage gained by agent-written tests. That distinction matters and the results are plausible as a lower bound: agents frequently omit tests, and when they add them, the tests often don't touch the changed lines. The pipeline description is careful—git diff reconstruction, tree-sitter filtering, reverse-applying test-only patches—and the replication package is a plus.

The soft spot is exactly where the stress-test note lands. The Python zero-coverage headline (64.8% of PRs with no changed line covered) is only meaningful if the 34 analyzed Python repos actually have runnable, discoverable test suites. The paper says these repos "could be built and instrumented," but that does not guarantee pytest collects any tests. If a repo has no tests or uses non-standard test discovery, every PR in that repo is assigned 0% coverage, inflating the zero-coverage rate and dragging down the 27.0% aggregate. The paper does not report per-repo test counts, pytest collect-only validation, or a comparison with the repo's own CI invocation. This is not an academic nit; it could substantially change a headline number. If the authors add a simple sanity check—how many of the 34 Python repos actually have at least one collected test, and how many PRs come from test-less repos—the central qualitative finding (existing tests are a weak safety net in Python) would likely survive, but the magnitude might not.

Other issues are smaller. The Java subset is disclosed as skewing small, yet the 61.5% Java figure still gets used prominently. Several per-construct miss rates rest on very few lines (Java Try-Catch: 43 lines). And there is no non-agentic baseline, so we cannot know how much of this is agent-specific versus typical of human PRs. Those are limitations, not fatal flaws.

This paper deserves a serious referee. The question is real, the measurement is new, and the weaknesses are fixable with additional reporting rather than being conceptual dead ends. I'd want the authors to address the pytest-collection question before publication, and I'd also want CIs on the per-construct samples. But the work is honest and the analysis is transparent; it is worth engaging with.

Referee Report

3 major / 6 minor

Summary. The paper analyzes 4882 agent-generated pull requests (532 Java, 4350 Python) from the AIDev v3 dataset, asking (RQ1) how often agents include test changes and (RQ2) how well existing and agent-written tests cover the changed executable lines. The pipeline reconstructs per-PR git diffs, separates test-file changes, runs JaCoCo/pytest-cov at PR heads, reverse-applies test-only patches for paired comparisons, and computes per-file and per-PR diff coverage (Eqs. 1-2). Main results: 49.6% of code-modifying PRs include test changes; existing tests cover 61.5% of changed lines in Java vs. 27.0% in Python, with 64.8% of Python PRs at zero diff coverage; agent-written tests improve coverage in only 35.9% of Java and 22.5% of Python Code+Tests PRs; error-handling constructs are the most consistently missed (86.0% Java, 81.0% Python). The paper also releases a replication artifact.

Significance. If the measurements are valid, this is a valuable first cross-language evidence base on testing in autonomous agentic PRs, with concrete implications for CI gating, review prioritization, and agent design. The study is observational and parameter-free: coverage numbers are produced by standard external tools (JaCoCo, pytest-cov) run on public repositories, the pipeline is described in detail, and the artifact is publicly available. The paper also candidly discloses the Java representativeness limitation (Section VI). The main risk is measurement validity in the Python pipeline: the headline 64.8% zero-coverage and 27.0% average figures depend on every analyzed Python repository having a non-empty, pytest-discoverable test suite, which is not currently demonstrated. With that validation, the contribution would be a solid empirical result for a software engineering venue.

major comments (3)

[Section III-A, IV-B] The central RQ2a claim (61.5% Java, 27.0% Python; 64.8% Python PRs with zero coverage) requires that every analyzed Python repository has a non-empty test suite that pytest can discover. 'Could be built and instrumented' does not guarantee this: pytest's default discovery only picks up test_*.py/*_test.py files, and projects may use custom testpaths, unittest, tox, or a non-pytest entrypoint. A repo where pytest collects zero tests will assign 0% diff coverage to every PR, inflating the zero-coverage fraction and depressing the aggregate. The paper reports no per-repo test counts, no pytest --collect-only validation, and no comparison with each repository's own CI test invocation. Please report per-repo collected-test counts or rerun the measurement using each repo's CI test command, and re-state RQ2a if any analyzed repos contribute no tests.
[Section III-A, RQ2b] The reverse-apply isolation of agent-written tests rests on the test-file identification heuristics (directory names and filename patterns). Files such as conftest.py, shared fixture modules, or tests in non-standard layouts may be misclassified as code-under-test or omitted from the test-only patch. If agent-added tests install shared fixtures or configuration that the existing tests depend on, `git apply -R` can remove those files and make the existing-suite run fail, so DiffCov_without is measured under a broken suite and the apparent agent gain (Table I) is inflated. Conversely, agent tests in non-standard locations would be counted as added code-under-test lines. Please quantify the number of PRs where the reverse-apply succeeded cleanly and the suite passed, and validate the test-file heuristics against each repository's CI configuration.
[Section VI] The Limitations section addresses sampling bias (smaller/less-starred Java repositories) but does not mention the zero-test/measurement-validity threat to the Python result. Since the paper's headline is the Python coverage gap, this omission is material. The limitation discussion should be updated once the pytest validation from the previous comments is performed, reporting how many analyzed Python repositories actually contributed a non-empty test signal.

minor comments (6)

[Eq. (1)] Define precisely what counts as an 'executable line' and how comments, docstrings, and non-executable added lines are excluded. The denominator of Eq. (1) is clear in principle, but the operationalization in Section III-A should be stated explicitly.
[Section IV-B] The first paragraph says 'for all merged PRs' but the coverage analysis is restricted to merged PRs in repositories with at least 10 agentic PRs that could be built and instrumented. Present the N after each filter clearly so the reader can trace 213 Java and 1664 Python PRs back to the dataset description.
[Figure 1] The Venn diagram is hard to read in the text; add a legend and ensure the region labels are legible. Also clarify whether the counts in each region are exclusive or non-exclusive.
[Table II] Java has no row for Import while Python does; add a dash or explanation. Also consider adding a row with 'total executable lines' to help contextualize the miss percentages.
[Affiliations] Minor text issues: 'Fairfax, V A' should be 'Fairfax, VA'; the phrase in Section V 'with existing tests covering no changed lines in 64.8% of Python PRs' duplicates the sentence immediately before it.
[References] References [21] and [22] point to the same artifact with two URLs; consider merging or clarifying which is the archival DOI and which is the GitHub repository.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the coverage results are direct measurements from independent tools (JaCoCo/pytest-cov), not derivations from fitted or self-cited inputs.

full rationale

The paper's central claims are observational measurements. Eq. 1 defines file-level diff coverage as a ratio of measured covered and missed lines; Eq. 2 is a difference of two measured values. Both are definitions, not predictions, and neither parameter is fitted to produce the reported percentages. The pipeline (Section III-A) runs JaCoCo and pytest-cov, external independent tooling, on public repositories, and the 'without tests' condition is created by reverse-applying the test-only patch—an isolation procedure, not a circular reduction. The only self-references are the replication package citations [21],[22], which are artifacts for reproducibility and are not load-bearing evidence. The limitations paragraph concedes sampling bias but this is a validity threat, not circularity. No uniqueness theorem, ansatz, or renamed known result is imported from the authors' prior work. Therefore no circular step can be exhibited under the quoted-evidence standard.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No free parameters are fitted and no entities are invented: this is an observational measurement with an external dataset. The central claim rests entirely on domain assumptions about the AIDev agent labels, the correctness of scaled-up coverage instrumentation (unvalidated per repo), the clean removability of test-only patches, and line coverage as a testedness proxy. The coverage-subset filter (repos with at least 10 agentic PRs that could be built and instrumented; Section II) is a hand-chosen sampling threshold, not a fitted parameter.

assumptions (6)

domain assumption AIDev dataset v3 correctly labels the 4,882 PRs as agent-generated and attributes each to the right agent
Section II takes all PRs and agent labels from the cited AIDev dataset [1]. Every rate in the paper inherits the accuracy of this external labeling.
domain assumption Executing each repository's suite with the standard runner and JaCoCo/pytest-cov yields valid executable-line coverage at the PR head commit
Section III-A. The RQ2 numbers depend on the runner matching each repo's actual configuration and on a non-empty suite existing; neither is validated per repository.
domain assumption Reverse-applying the test-only patch (git apply -R) cleanly isolates coverage contributed by the repository's existing tests
Section III-A (PR-level coverage without tests). Assumes test-only patches are removable without breaking other tests and that coverage line numbers remain valid after reversal.
domain assumption Test-file identification by directory and filename conventions captures all test files
Section III-A. Misclassified test files would mislabel code-under-test vs test code and shift coverage denominators and the RQ1 categories.
domain assumption srcML-based classification following Zhu et al. assigns each added line to the correct syntactic construct
Section III-A; Table II miss rates are computed from this classification, so classification errors propagate into the per-construct findings.
domain assumption Line coverage is a meaningful proxy for how tested a change is
Diff coverage (Eq. 1) treats any line executed by at least one test as covered regardless of assertions, so 'covered' can overstate behavioral verification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Test Coverage Analysis of Agentic Pull Requests." pith.science (2026). https://pith.science/paper/OTCNS3UG

@misc{pith2026260718057,
  author       = {Pith},
  title        = {Pith review of: Test Coverage Analysis of Agentic Pull Requests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTCNS3UG}},
  note         = {Machine review of arXiv:2607.18057}
}

read the original abstract

AI coding agents increasingly submit complete pull requests (PRs) with minimal human intervention, shifting software development from AI-assisted to autonomous workflows. As these agents become more prevalent, ensuring the code they generate is adequately tested, by existing tests or by tests the agents write, is critical to preventing regressions, yet little is known about testing in agentic PRs. To address this gap, we analyze 4882 agent-generated PRs from the AIDev dataset (532 Java and 4350 Python PRs) produced by five coding agents. We study (i) how often agents include test changes and (ii) how well covered are code changes by existing and agent-written tests. Agents include test changes in only 49.6% of PRs that change code under test files. Existing tests provide an incomplete safety net: they cover 61.5% of agents' changed executable lines in Java and only 27.0% in Python, where 64.8% of PRs have no changed line executed by any existing test. Agent-written tests improve coverage over existing tests, but only in a minority of PRs: 35.9% of Java and 22.5% of Python Code + Tests PRs show a coverage gain. Across both languages, error-handling constructs (e.g., try and catch blocks) are the most consistently under-tested, with miss rates reaching 86.0% in Java and 81.0% in Python. These findings motivate coverage-aware development practices, coverage feedback loops for coding agents, and evaluation benchmarks that measure test quality to better help agents reliably test their own code.

Figures

Figures reproduced from arXiv: 2607.18057 by the authors.

**Figure 2.** Existing-test diff coverage per merged PR. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 3 canonical work pages

[1]

The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,

H. Li, H. Zhang, and A. E. Hassan, “The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,”arXiv preprint arXiv:2507.15003,

arXiv
[2]

EvoSuite: Automatic test suite generation for object-oriented software,

G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” inEuropean Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2011, pp. 416–419. [Online]. Available: https://doi.org/10.1145/2025113.2025179

arXiv 2011
[3]

Ammann and J

P. Ammann and J. Offutt,Introduction to Software Testing. Cambridge University Press, 2016

2016
[4]

Pezz `e and M

M. Pezz `e and M. Young,Software testing and analysis: process, principles, and techniques. John Wiley & Sons, 2008

2008
[5]

The guide to the software engineering body of knowledge,

P. Bourque, R. Dupuis, A. Abran, J. Moore, and L. Tripp, “The guide to the software engineering body of knowledge,”IEEE Software, vol. 16, pp. 35–44, 1999. [Online]. Available: https://doi.org/10.1109/52.805471

doi:10.1109/52.805471 1999
[6]

An empirical study of regression test selection techniques,

T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel, “An empirical study of regression test selection techniques,”ACM Transactions on Software Engineering Methodology, vol. 10, pp. 184– 208, 2001. [Online]. Available: https://doi.org/10.1145/367008.367020

arXiv 2001
[7]

SWE-bench: Can language models resolve real-world github issues?

C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=VTF8yNQM66

2024
[8]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chanez, E. Barnes, A. Herbert-V o...

arXiv doi:10.48550/arxiv.2107.03374 2021

Show all 32 references

[9]

On the robustness of code generation techniques: An empirical study on GitHub Copilot,

A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota, “On the robustness of code generation techniques: An empirical study on GitHub Copilot,” inInternational Conference on Software Engineering, 2023, pp. 2149–2160. [Online]. Avai...

2023
[10]

Using GitHub Copilot for test generation in Python: An empirical study,

K. El Haji, C. Brandt, and A. Zaidman, “Using GitHub Copilot for test generation in Python: An empirical study,” inInternational Conference on Automation of Software Test, 2024, pp. 45–55. [Online]. Available: https://doi.org/10.1145/3644032.3644443

2024
[11]

CoverUp: Effective high coverage test generation for Python,

J. Altmayer Pizzorno and E. D. Berger, “CoverUp: Effective high coverage test generation for Python,”Proceedings of the ACM on Software Engineering, vol. 2, pp. 2897–2919, 2025. [Online]. Available: https://doi.org/10.1145/3729398

2025 doi
[12]

Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM,

G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM,”Proceedings of the ACM on Software Engineering, vol. 1, pp. 951–971, 2024. [Online]. Available: https://...

2024 doi
[13]

Automated unit test improvement using large language models at Meta,

N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at Meta,” inSymposium on the Foundations of Software Engineering, industry track, 2024, pp. 185–196. [...

2024
[14]

SWT-bench: Testing and validating real-world bug-fixes with code agents,

N. M ¨undler, M. N. M ¨uller, J. He, and M. Vechev, “SWT-bench: Testing and validating real-world bug-fixes with code agents,” inAdvances in Neural Information Processing Systems, 2024, pp. 81 857–81 887. [Online]. Available: https://doi.org/10.52202/079017-2601

2024 doi
[15]

On the use of agentic coding: An empirical study of pull requests on GitHub,

M. Watanabe, H. Li, Y . Kashiwa, B. Reid, H. Iida, and A. E. Hassan, “On the use of agentic coding: An empirical study of pull requests on GitHub,”ACM Transactions on Software Engineering Methodology,
[16]

Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward,

A. Sergeyuk, Y . Golubev, T. Bryksin, and I. Ahmed, “Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward,”Information and Software Technology, vol. 178, p. 107610,
[17]

The WyCash portfolio management system,

W. Cunningham, “The WyCash portfolio management system,”ACM SIGPLAN OOPS Messenger, vol. 4, pp. 29–30, 1992. [Online]. Available: https://dl.acm.org/doi/10.1145/157709.157715

1992
[18]

Hidden technical debt in machine learning systems,

D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V . Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” inAdvances in Neural Information Processing Systems, 2015

2015
[19]

Available: https://doi.org/10.1016/j.infsof.2024.107610

[Online]. Available: https://doi.org/10.1016/j.infsof.2024.107610

2024
[20]

The relation of test-related factors to software quality: A case study on Apache systems,

F. Pecorelli, F. Palomba, A. De Lucia, and A. Bacchelli, “The relation of test-related factors to software quality: A case study on Apache systems,”Empirical Software Engineering, vol. 26, pp. 1–42, 2021. [Online]. Available: https://doi.org/10.1007/s10664-020-09891-y

2021 doi
[21]

Replication package,

SageSELab, “Replication package,” inIEEE International Conference on Software Maintenance and Evolution (ICSME), 2026, replication package: https://github.com/SageSELab/ Agentic-Pull-Request-Test-Coverage/

2026
[22]

An empirical study of self-admitted technical debt in machine learning software,

A. Bhatia, F. Khomh, B. Adams, and A. Hassan, “An empirical study of self-admitted technical debt in machine learning software,”ACM Transactions on Software Engineering Methodology, vol. 35, pp. 1–44,
[23]

Available: https://doi.org/10.1145/3785001

[Online]. Available: https://doi.org/10.1145/3785001

doi
[24]

JaCoCo: Java code coverage library,

EclEmma Team, “JaCoCo: Java code coverage library,” https://www. jacoco.org/jacoco/, 2026

2026
[25]

pytest-cov: Coverage plugin for pytest,

pytest-dev contributors, “pytest-cov: Coverage plugin for pytest,” https: //pypi.org/project/pytest-cov/, 2026

2026
[26]

SageSELab/Agentic-Pull-Request-Test-Coverage,

A. K. Dipongkor, “SageSELab/Agentic-Pull-Request-Test-Coverage,” Jul. 2026. [Online]. Available: https://doi.org/10.5281/zenodo.21419686

2026 doi
[27]

Tree-sitter: An incremental parsing system for program- ming tools,

M. Brunsfeld, “Tree-sitter: An incremental parsing system for program- ming tools,” https://tree-sitter.github.io/tree-sitter/, accessed: 2026

2026
[28]

Individual comparisons by ranking methods,

F. Wilcoxon, “Individual comparisons by ranking methods,” inBreak- throughs in statistics: Methodology and distribution, 1992, pp. 196–202. [Online]. Available: https://doi.org/10.1007/978-1-4612-4380-9 16

1992 doi
[30]

srcML: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration,

M. L. Collard, M. J. Decker, and J. I. Maletic, “srcML: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration,” inInternational Conference on Software Maintenance, 2013, pp. 516–519. [Online]. Available: https://doi.org/10.1109/...

2013 doi
[31]

An analysis of programming language statement frequency in C, C++, and Java source code,

X. Zhu, E. J. Whitehead Jr, C. Sadowski, and Q. Song, “An analysis of programming language statement frequency in C, C++, and Java source code,”Software: Practice and Experience, vol. 45, no. 11, 2015. [Online]. Available: https://doi.org/10.1002/spe.2298

2015 doi
[2025]

Available: https://doi.org/10.48550/arXiv.2507.15003

[Online]. Available: https://doi.org/10.48550/arXiv.2507.15003

doi arXiv
[2026]

Available: https://doi.org/10.1145/3798166

[Online]. Available: https://doi.org/10.1145/3798166

doi

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,

H. Li, H. Zhang, and A. E. Hassan, “The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,”arXiv preprint arXiv:2507.15003,

arXiv

[2] [2]

EvoSuite: Automatic test suite generation for object-oriented software,

G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” inEuropean Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2011, pp. 416–419. [Online]. Available: https://doi.org/10.1145/2025113.2025179

arXiv 2011

[3] [3]

Ammann and J

P. Ammann and J. Offutt,Introduction to Software Testing. Cambridge University Press, 2016

2016

[4] [4]

Pezz `e and M

M. Pezz `e and M. Young,Software testing and analysis: process, principles, and techniques. John Wiley & Sons, 2008

2008

[5] [5]

The guide to the software engineering body of knowledge,

P. Bourque, R. Dupuis, A. Abran, J. Moore, and L. Tripp, “The guide to the software engineering body of knowledge,”IEEE Software, vol. 16, pp. 35–44, 1999. [Online]. Available: https://doi.org/10.1109/52.805471

doi:10.1109/52.805471 1999

[6] [6]

An empirical study of regression test selection techniques,

T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel, “An empirical study of regression test selection techniques,”ACM Transactions on Software Engineering Methodology, vol. 10, pp. 184– 208, 2001. [Online]. Available: https://doi.org/10.1145/367008.367020

arXiv 2001

[7] [7]

SWE-bench: Can language models resolve real-world github issues?

C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=VTF8yNQM66

2024

[8] [8]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chanez, E. Barnes, A. Herbert-V o...

arXiv doi:10.48550/arxiv.2107.03374 2021