{"id":"b164fc64-b0b2-4c7f-bb7c-a1291e0b3e07","arxiv_id":"2607.18057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Agent-generated code is poorly tested: only 49.6% of agentic PRs include test changes, existing tests cover 61.5% of changed Java lines but just 27.0% in Python, and error-handling code is the most often missed.","lead":"AI coding agents rarely test their own changes: across 4,882 agentic pull requests, only 49.6% of code-changing PRs include any test changes, and existing tests leave most new Python code unexecuted. The paper is the first quantitative portrait of testing in autonomous agent workflows, giving practitioners and benchmark builders concrete numbers to design coverage gates and agent feedback loops.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Python 'zero coverage' may be driven by repos where pytest collects 0 tests, not by existing tests missing agent changes.","rationale":"The reader's weakest_assumption already identifies the RQ2 pipeline validity as the load-bearing premise, and specifically mentions that the Python zero-coverage headline could conflate 'no runnable tests exist' with 'tests do not reach the change.' My concern sharpens this to a concrete measurement-validity gap: the paper never reports how many Python PRs come from repos where pytest collected zero tests. This is directly checkable using the provided replication package. If the zero-coverage rate persists after excluding those repos, the paper's central claim is strengthened; if it drops, the headline must be reframed. I credit the paper for publishing the replication package, disclosing the Java subset's small-project bias, and providing a clear methodology, but the missing per-repo test count is a genuine soft spot. The verdict remains CONDITIONAL: the paper is transparent and its qualitative conclusions are plausible, but the magnitude of the Python coverage findings depends on a validation that has not been reported. No change to the reader's verdict is warranted; this test would either resolve the concern or force a reconditioned claim.","tokens_in":10952,"tokens_out":7649,"duration_ms":101850,"concrete_test":"In the public replication package, run `pytest --collect-only -q` for each of the 34 analyzed Python repos (or replicate the exact coverage commands from Section III-A and record the number of tests collected per repo). (1) Report how many repos yield 0 collected tests. (2) Recompute the 64.8% zero-diff-coverage PR percentage and the 27.0% aggregate line coverage excluding all PRs from repos with 0 collected tests. (3) For repos with test files but 0 collected tests, compare against the repo's CI workflow (e.g., .github/workflows/*.yml) to see whether a non-pytest runner is used. If the zero-coverage rate remains ~64.8% after excluding no-test repos, the concern is resolved; if it drops materially, the Python headline must be reconditioned to distinguish 'repos without test infrastructure' from 'existing tests fail to reach agent changes.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the RQ2a coverage measurements (Section IV-B) measure what they claim. The pipeline in Section III-A runs pytest-cov for every Python repo and records any PR as 0% diff coverage if the coverage report lists no executed lines. The paper filters to the 34 repos that 'could be built and instrumented,' but building a repo does not guarantee the test suite contains tests that pytest can discover. If pytest collects zero tests because the repo genuinely has no tests, or because the test layout doesn't match pytest's default discovery patterns (e.g., custom testpaths, files not named test_*.py, or an entrypoint that isn't pytest), every PR from that repo is assigned 0% coverage. The resulting 64.8% zero-coverage PR statistic and the aggregate 27.0% coverage figure are then contaminated by repos that contribute no test signal at all. The paper reports no per-repo test counts, no pytest collect-only validation, and no comparison against the repo's own CI test invocation (e.g., tox, unittest, or GitHub Actions steps). The limitation section addresses project representativeness but not this measurement-validity gap, leaving the central quantitative claim one check away from being either confirmed or substantially revised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes 4882 agent-generated pull requests (532 Java, 4350 Python) from the AIDev v3 dataset, asking (RQ1) how often agents include test changes and (RQ2) how well existing and agent-written tests cover the changed executable lines. The pipeline reconstructs per-PR git diffs, separates test-file changes, runs JaCoCo/pytest-cov at PR heads, reverse-applies test-only patches for paired comparisons, and computes per-file and per-PR diff coverage (Eqs. 1-2). Main results: 49.6% of code-modifying PRs include test changes; existing tests cover 61.5% of changed lines in Java vs. 27.0% in Python, with 64.8% of Python PRs at zero diff coverage; agent-written tests improve coverage in only 35.9% of Java and 22.5% of Python Code+Tests PRs; error-handling constructs are the most consistently missed (86.0% Java, 81.0% Python). The paper also releases a replication artifact.","tokens_in":11114,"tokens_out":3738,"duration_ms":39505,"significance":"If the measurements are valid, this is a valuable first cross-language evidence base on testing in autonomous agentic PRs, with concrete implications for CI gating, review prioritization, and agent design. The study is observational and parameter-free: coverage numbers are produced by standard external tools (JaCoCo, pytest-cov) run on public repositories, the pipeline is described in detail, and the artifact is publicly available. The paper also candidly discloses the Java representativeness limitation (Section VI). The main risk is measurement validity in the Python pipeline: the headline 64.8% zero-coverage and 27.0% average figures depend on every analyzed Python repository having a non-empty, pytest-discoverable test suite, which is not currently demonstrated. With that validation, the contribution would be a solid empirical result for a software engineering venue.","major_comments":[{"comment":"The central RQ2a claim (61.5% Java, 27.0% Python; 64.8% Python PRs with zero coverage) requires that every analyzed Python repository has a non-empty test suite that pytest can discover. 'Could be built and instrumented' does not guarantee this: pytest's default discovery only picks up test_*.py/*_test.py files, and projects may use custom testpaths, unittest, tox, or a non-pytest entrypoint. A repo where pytest collects zero tests will assign 0% diff coverage to every PR, inflating the zero-coverage fraction and depressing the aggregate. The paper reports no per-repo test counts, no pytest --collect-only validation, and no comparison with each repository's own CI test invocation. Please report per-repo collected-test counts or rerun the measurement using each repo's CI test command, and re-state RQ2a if any analyzed repos contribute no tests.","section":"Section III-A, IV-B"},{"comment":"The reverse-apply isolation of agent-written tests rests on the test-file identification heuristics (directory names and filename patterns). Files such as conftest.py, shared fixture modules, or tests in non-standard layouts may be misclassified as code-under-test or omitted from the test-only patch. If agent-added tests install shared fixtures or configuration that the existing tests depend on, `git apply -R` can remove those files and make the existing-suite run fail, so DiffCov_without is measured under a broken suite and the apparent agent gain (Table I) is inflated. Conversely, agent tests in non-standard locations would be counted as added code-under-test lines. Please quantify the number of PRs where the reverse-apply succeeded cleanly and the suite passed, and validate the test-file heuristics against each repository's CI configuration.","section":"Section III-A, RQ2b"},{"comment":"The Limitations section addresses sampling bias (smaller/less-starred Java repositories) but does not mention the zero-test/measurement-validity threat to the Python result. Since the paper's headline is the Python coverage gap, this omission is material. The limitation discussion should be updated once the pytest validation from the previous comments is performed, reporting how many analyzed Python repositories actually contributed a non-empty test signal.","section":"Section VI"}],"minor_comments":[{"comment":"Define precisely what counts as an 'executable line' and how comments, docstrings, and non-executable added lines are excluded. The denominator of Eq. (1) is clear in principle, but the operationalization in Section III-A should be stated explicitly.","section":"Eq. (1)"},{"comment":"The first paragraph says 'for all merged PRs' but the coverage analysis is restricted to merged PRs in repositories with at least 10 agentic PRs that could be built and instrumented. Present the N after each filter clearly so the reader can trace 213 Java and 1664 Python PRs back to the dataset description.","section":"Section IV-B"},{"comment":"The Venn diagram is hard to read in the text; add a legend and ensure the region labels are legible. Also clarify whether the counts in each region are exclusive or non-exclusive.","section":"Figure 1"},{"comment":"Java has no row for Import while Python does; add a dash or explanation. Also consider adding a row with 'total executable lines' to help contextualize the miss percentages.","section":"Table II"},{"comment":"Minor text issues: 'Fairfax, V A' should be 'Fairfax, VA'; the phrase in Section V 'with existing tests covering no changed lines in 64.8% of Python PRs' duplicates the sentence immediately before it.","section":"Affiliations"},{"comment":"References [21] and [22] point to the same artifact with two URLs; consider merging or clarifying which is the archival DOI and which is the GitHub repository.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's topic is timely and the artifact is valuable, but I am not yet convinced the Python headline figures are robust because the manuscript does not demonstrate that every analyzed Python repository has a non-empty, pytest-discoverable test suite. The issue is fixable within the manuscript's scope by adding per-repo validation or using CI-discovered test commands, so I recommend major revision rather than rejection. The Java representativeness caveat is honestly disclosed and acceptable once the Python validity question is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is a first quantified look at whether agent-written code in merged PRs is actually exercised by tests. Nobody else has measured diff coverage in agentic PRs at this scale, and the authors separate coverage by existing tests from coverage gained by agent-written tests. That distinction matters and the results are plausible as a lower bound: agents frequently omit tests, and when they add them, the tests often don't touch the changed lines. The pipeline description is careful—git diff reconstruction, tree-sitter filtering, reverse-applying test-only patches—and the replication package is a plus.\n\nThe soft spot is exactly where the stress-test note lands. The Python zero-coverage headline (64.8% of PRs with no changed line covered) is only meaningful if the 34 analyzed Python repos actually have runnable, discoverable test suites. The paper says these repos \"could be built and instrumented,\" but that does not guarantee pytest collects any tests. If a repo has no tests or uses non-standard test discovery, every PR in that repo is assigned 0% coverage, inflating the zero-coverage rate and dragging down the 27.0% aggregate. The paper does not report per-repo test counts, pytest collect-only validation, or a comparison with the repo's own CI invocation. This is not an academic nit; it could substantially change a headline number. If the authors add a simple sanity check—how many of the 34 Python repos actually have at least one collected test, and how many PRs come from test-less repos—the central qualitative finding (existing tests are a weak safety net in Python) would likely survive, but the magnitude might not.\n\nOther issues are smaller. The Java subset is disclosed as skewing small, yet the 61.5% Java figure still gets used prominently. Several per-construct miss rates rest on very few lines (Java Try-Catch: 43 lines). And there is no non-agentic baseline, so we cannot know how much of this is agent-specific versus typical of human PRs. Those are limitations, not fatal flaws.\n\nThis paper deserves a serious referee. The question is real, the measurement is new, and the weaknesses are fixable with additional reporting rather than being conceptual dead ends. I'd want the authors to address the pytest-collection question before publication, and I'd also want CIs on the per-construct samples. But the work is honest and the analysis is transparent; it is worth engaging with.","headline":"A genuinely new measurement of test coverage in agentic PRs, held back by a real but addressable validity gap in the Python zero-coverage statistic.","tokens_in":11716,"tokens_out":1264,"would_cite":true,"duration_ms":16417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that code changes made by AI coding agents and merged through pull requests are poorly exercised by tests, with existing tests covering only 61.5% of added executable lines in Java and 27.0% in Python, and agent-writt","keywords":["AI coding agents","test coverage","pull requests","diff coverage","empirical software engineering","automated testing","coverage measurement"],"falsifier":"A replication that independently runs the same PRs' test suites (or compares a sample against an external coverage service) and finds markedly higher diff-coverage numbers, or discovers that most zero-coverage Python PRs have no runnable tests at all, would undercut the central claim.","tokens_in":10743,"feed_emoji":"🧪","tokens_out":3192,"duration_ms":33533,"temperature":0.7,"pith_summary":"The paper tries to establish that code changes introduced by autonomous AI coding agents are, at merge time, far less exercised by tests than a green CI run implies. Analyzing 4,882 pull requests produced by five coding agents across Java and Python, it measures 'diff coverage': the fraction of added executable lines that any test executes. The central findings are that existing tests cover only 61.5% of changed lines in Java and 27.0% in Python, that 64.8% of Python PRs have no changed line executed by any existing test, and that agent-written tests add coverage of the agent's own changes in only a minority of PRs (35.9% Java, 22.5% Python). The paper argues this means a passing test suite is a weak safety signal for agentic code, and that error-handling constructs (try-catch and throw) are the most consistently untested category in both languages.","feed_headline":"Existing tests cover just 27% of agent-written Python code","feed_subtitle":"Java fares better at 61.5%; agent-authored tests add coverage in only a minority of pull requests.","key_machinery":"The argument rests on a diff-coverage pipeline. For each PR it reconstructs the patch from base to head commit, separates test-file changes from code-under-test changes, runs the repository's full test suite with and without the agent's test changes (by reverse-applying the test-only patch), and intersects the inventory of added executable lines — each labeled by syntactic construct via source-code parsing — with line-level coverage reports produced by standard Java and Python coverage tools. The carrying metric is PR-level diff coverage: the ratio of covered added lines to total added lines, aggregated by summing covered and missed lines across changed files.","core_discovery":"The central discovery is a measurement: for 213 Java and 1,664 Python merged agentic PRs, existing tests execute only 61.5% (Java) and 27.0% (Python) of the executable lines agents add, and in 64.8% of Python PRs none of the added lines are executed. When agents do include tests, those tests raise coverage on average (+15.6 percentage points Java, +9.6 Python), but the gain comes from a minority: only 35.9% of Java and 22.5% of Python Code + Tests PRs show any coverage gain on the agent's own changes. Error-handling lines are the weakest spot: newly added throw statements are missed in 67.5% of Java and 82.3% of Python cases, and try-catch block lines are missed in 86.0% of Java and 81.0% of","pith_inferences":["If the coverage pattern holds beyond this dataset, the 64.8% zero-coverage Python figure likely mixes 'repository has no runnable tests' with 'tests do not reach the change'; separating those two would sharpen the prescription for agent designers.","A direct extension would measure diff coverage on human-written PRs from the same repositories; the gap between human and agent coverage would estimate how much of the shortfall is attributable to the agent rather than to project testing culture.","The construct-level miss rates suggest a targeted experiment: prompt agents with explicit error-handling test requirements and measure whether try-catch miss rates drop while other categories stay flat, isolating whether the weakness is a prompting problem or a model capability limit."],"forward_implications":["A green CI run on an agentic PR cannot be interpreted as evidence that the change is tested; teams should treat diff coverage as a separate, load-bearing quality signal.","Coding agents need a coverage-aware feedback loop that checks whether their own added lines are exercised by their own tests before submitting a PR.","Error-handling constructs (try-catch, throw/raise) are the highest-yield target for agent test generation in both Java and Python.","Agent benchmarks that reward passing existing tests without measuring diff coverage will continue to reward under-tested changes.","Repositories adopting agentic workflows could use minimum diff-coverage thresholds as a practical quality gate on merged PRs."],"fun_headline_variants":["Python agent PRs: 64.8% have no lines covered by tests","Agent PRs skip test changes half the time","Try-catch lines: agent code's biggest coverage blind spot","Only 22.5% of Python Code+Tests PRs gain coverage"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The coverage figures depend on the assumption that running each repository's test suite with the assumed runner and coverage tool at the PR head yields a valid executable-line coverage report, and that reverse-applying the test-only patch cleanly isolates the existing tests' coverage.","fun_headline_variants_meta":{"raw":{"variants":["Python agent PRs: 64.8% have no lines covered by tests","Agent PRs skip test changes half the time","Try-catch lines: agent code's biggest coverage blind spot","Only 22.5% of Python Code+Tests PRs gain coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001295,"raw_usage":{"total_tokens":5187,"prompt_tokens":875,"completion_tokens":4312,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":4237}},"tokens_in":619,"tokens_out":4312,"duration_ms":32243,"temperature":1.0,"reasoning_tokens":4237,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:15:29.140686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication that independently runs the same PRs' test suites (or compares a sample against an external coverage service) and finds markedly higher diff-coverage numbers, or discovers that most zero-coverage Python PRs have no runnable tests at all, would undercut the central claim.","supporting_citations":[],"review_version":1}