TEBench is a new project-level benchmark for test evolution showing coding agents achieve only 45-49% F1 on identifying tests needing changes, with stale tests hardest due to reliance on execution failures.
TDD-Bench verified: Can LLMs generate tests for issues before they get resolved?arXiv preprint arXiv:2412.02883
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
citation-role summary
background 1
method 1
citation-polarity summary
fields
cs.SE 10representative citing papers
A purpose-built, staged LLM agent correctly sets up and executes software analysis tools on 33 of 35 benchmark tasks, outperforming general-purpose agent baselines by at least 17 percentage points.
The first empirical study of test overfitting shows that auto-generated tests from issues can lead to code that passes observed tests but misses important cases or breaks functionality in SWE-bench issue resolution.
Introduces the first benchmark for Java reproduction test generation from repository issues and adapts a prior Python tool to produce high performance on it.
iCoRe improves bug reproduction test generation by combining differentiated code/test retrieval, function-call-structure similarity, and iterative generation-to-retrieval feedback, achieving state-of-the-art results on SWT-bench Lite and TDD-bench Verified.
Agent-CoEvo is a multi-agent LLM framework that coevolves code patches and test patches to resolve repository-level issues, outperforming fixed-test baselines on SWE-bench Lite and SWT-bench Lite.
TestPrune minimizes regression test suites to improve bug reproduction and patch validation in LLM-based agentic repair pipelines, delivering 6-13% relative gains on SWE-Bench benchmarks at low API cost.
icat-agent improves resolution rates on SWE-bench Verified and Pro by 3.6-18.5% over baselines via event-based multi-agent scaffolding and rubric-driven workflow pivoting while using the same models.
citing papers explorer
-
Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution
TEBench is a new project-level benchmark for test evolution showing coding agents achieve only 45-49% F1 on identifying tests needing changes, with stale tests hardest due to reliance on execution failures.
-
Evaluating LLM Agents on Automated Software Analysis Tasks
A purpose-built, staged LLM agent correctly sets up and executes software analysis tools on 33 of 35 benchmark tasks, outperforming general-purpose agent baselines by at least 17 percentage points.
-
Investigating Test Overfitting on SWE-bench
The first empirical study of test overfitting shows that auto-generated tests from issues can lead to code that passes observed tests but misses important cases or breaks functionality in SWE-bench issue resolution.
-
Reproduction Test Generation for Java SWE Issues
Introduces the first benchmark for Java reproduction test generation from repository issues and adapts a prior Python tool to produce high performance on it.
-
iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation
iCoRe improves bug reproduction test generation by combining differentiated code/test retrieval, function-call-structure similarity, and iterative generation-to-retrieval feedback, achieving state-of-the-art results on SWT-bench Lite and TDD-bench Verified.
-
Beyond Fixed Tests: Repository-Level Issue Resolution as Coevolution of Code and Behavioral Constraints
Agent-CoEvo is a multi-agent LLM framework that coevolves code patches and test patches to resolve repository-level issues, outperforming fixed-test baselines on SWE-bench Lite and SWT-bench Lite.
-
Can Old Tests Do New Tricks for Resolving SWE Issues?
TestPrune minimizes regression test suites to improve bug reproduction and patch validation in LLM-based agentic repair pipelines, delivering 6-13% relative gains on SWE-Bench benchmarks at low API cost.
-
Unlocking Model Potentials Through Adaptive Multi-Agent Scaffolding for Efficient Issue Resolution
icat-agent improves resolution rates on SWE-bench Verified and Pro by 3.6-18.5% over baselines via event-based multi-agent scaffolding and rubric-driven workflow pivoting while using the same models.
- Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions
- From Plan to Action: How Well Do Agents Follow the Plan?