REVIEW 6 cited by
Otter: Generating Tests from Issues to Validate SWE Patches
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch that resolves the issue. This paper focuses on the scenario where that code patch does not yet exist. Doing so supports two major use-cases. First, it supports TDD (test-driven development), the discipline of "test first, write code later" that has well-documented benefits for human software engineers. Second, it also validates SWE (software engineering) agents, which generate code patches for resolving issues. This paper introduces TDD-Bench-Verified, a benchmark for generating tests from issues, and Otter, an LLM-based solution for this task. Otter augments LLMs with rule-based analysis to check and repair their outputs, and introduces a novel self-reflective action planner. Experiments show Otter outperforming state-of-the-art systems for generating tests from issues, in addition to enhancing systems that generate patches from issues. We hope that Otter helps make developers more productive at resolving issues and leads to more robust, well-tested code.
Forward citations
Cited by 6 Pith papers
-
Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?
By replaying each validation command on buggy, candidate, and gold-fix code, this study finds 46% of positive checks do not discriminate the bug, and a reminder alone explains part of the feedback benefit.
-
Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches
Behavioral signals in bug reports propagate only partially into tests and fixes; alignment is measurable but representation-dependent, and LLM judges are systematically optimistic versus human ratings.
-
FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents
A staged LLM agent that reasons about data flow and branch conditions generated valid proof-of-vulnerability tests for 16 of 100 Java, C, and C++ projects, versus 9 for the CodeAct 2.1 baseline.
-
ExplainBench: Evaluating Code Explanations from Agents
Agent explanation quality can be scored automatically with LLM questionnaires, and this score ranks agents differently from SWE-bench Verified.
-
DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences
Change-directed LLM-based testing with call-graph access information and union-coverage feedback exposes behavioral differences in 78.2% of 463 Python pull requests.
-
Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
An empirical study of five LLM-based program repair agents shows they excel on simple fixes but overfit tests and struggle with test generation, regression selection, and advanced tooling.
Discussion (0). Continue with ORCID to comment.