SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.
Meet devin, the first ai software engineer
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.SE 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.