SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

Avi Arora; Jinu Jang; Roshanak Zilouchian Moghaddam

REVIEW 4 major objections 4 minor 5 cited by

SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SetupBench isolates the environment-bootstrap skill and finds current coding agents succeed on only 34–62% of 93 bare-sandbox setup tasks.

desk verdict A genuinely useful benchmark for environment-bootstrap evaluation, but the validation protocol is less uniform than advertised; fix the success-command audit before trusting the numbers. read the letter →

arxiv 2507.09063 v1 pith:FGC62KDM submitted 2025-07-11 cs.SE cs.AIcs.LG

Avi Arora , Jinu Jang , Roshanak Zilouchian Moghaddam This is my paper

classification cs.SEcs.AIcs.LG

keywords environmentbootstrapLLMagentsbenchmarkdependencyresolutiondatabasesetupDevOpssoftwareengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that existing LLM-agent benchmarks test agents in pre-baked environments where every dependency is already installed, so they miss a core developer skill: getting code to run from scratch. SetupBench closes that gap with 93 curated tasks that start in a bare Linux sandbox and require installing packages, resolving dependency conflicts, initializing databases, and configuring background services. Each task comes with a deterministic one-line success command, and evaluation of the OpenHands agent across five model variants shows low success rates, especially in repository setup (38.9–57.4%) and local database configuration (20.0–53.3%). If correct, SetupBench provides a reproducible yardstick for a previously unevaluated capability and exposes concrete failure modes that agent designers can target.

What carries the argument

The load-bearing mechanism is the deterministic one-line validation command (success_command) paired with a fresh minimal Linux sandbox. Each task defines a single command that echoes 'Setup successful' only when the intended environment changes have taken effect, allowing automated, reproducible pass/fail evaluation without LLM-as-a-judge or flaky test suites. This design lets the benchmark isolate the bootstrap skill and attribute success or failure to the agent's setup behavior rather than to pre-installed dependencies.

What would settle it

Take the 93 success commands, run each in a fresh, correctly configured sandbox ten times (with small delays before execution), and count how often 'Setup successful' appears when every documented setup step was performed; any false-negative rate above a negligible threshold, or any case where a command prints success despite an obvious missing component (e.g., a missing test runner), would undermine the reported success rates and cross-model comparisons.

Watch

Extended reading notes

Core claim

SetupBench is a 93-instance benchmark that isolates the environment-bootstrap skill by presenting agents with a natural-language problem statement, a workspace snapshot, and a deterministic success command that prints 'Setup successful' or 'Setup failed'. It spans seven language ecosystems, five database engines, and multi-service orchestration scenarios, and it evaluates agents in minimal containers rather than pre-configured Docker images. The paper's central finding is that current state-of-the-art coding agents resolve fewer than two-thirds of these tasks (best model 62.4%), with the largest gaps in repository setup and local database configuration. The paper also identifies three recurring failure modes—incomplete development-tooling installation, hallucinated task constraints, and non-persistent environment modifications—and quantifies agent inefficiency: 38–69% of agent steps are wasted compared with an optimal human baseline across the ten instances analyzed.

Load-bearing premise

The benchmark's validity rides on the assumption that each single-line success command deterministically and correctly captures whether the environment was truly set up, even though the full set of commands is not shown in the paper and commands that probe localhost endpoints or file contents can be sensitive to service timing and port availability.

Editorial extensions

If this is right

If SetupBench is a valid measure, environment bootstrap is a distinct, currently under-evaluated capability that leaderboard rankings built on pre-baked environments do not capture.
Agent architectures need explicit persistence protocols—writing environment changes to configuration files and sourcing them—so that setup survives agent-human handoffs.
Context-aware exploration mechanisms, such as repository-tree injection or semantic file ranking, could reduce the 38–69% wasted steps and improve setup success.
Hybrid model-selection strategies are implied: lightweight models might handle routine setup while stronger models are reserved for dependency resolution and multi-service orchestration.
The benchmark's categories provide a graded testbed for future agents, with database setup and repository setup acting as the hardest currently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper leaves implicit is chaining SetupBench with downstream coding tasks (bug fixing, feature implementation) so the evaluation measures whether a correctly bootstrapped environment actually enables subsequent work; such an extension would test continuity across the full agent workflow.
The validity of every comparison in the paper rests on the success commands being both sufficient and timing-robust; a reader should check whether commands that curl localhost endpoints or check file contents can produce false negatives due to service startup latency rather than setup failure.
The efficiency results suggest a concrete testable design: agents given a pre-computed repository structure (e.g., tree of files) should show a measurable drop in wasted exploration steps; this could be validated within SetupBench's existing 10-instance subset.
The paper's failure-mode taxonomy could be turned into a diagnostic tool: automatically labeling failing trajectories by the three modes (missing tooling, hallucinated constraints, non-persistence) would let the community track whether agent improvements actually address these bottlenecks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

SetupBench is worth a close look. It isolates a real capability that SWE-bench-style benchmarks ignore: bootstrapping a dev environment from a bare sandbox. Existing work like EnvBench covers a subset; SetupBench adds databases, background services, OS-level packages, and multi-language repo setup, and it ships mining and validation scripts. That is a genuine contribution, and the evaluation across five OpenHands models gives useful signal (Claude 4 at 62% vs GPT-4o at 34%). Section 5 is refreshingly honest about limitations, and the release of prompts and scripts makes the work reproducible in principle.

The main soft spot is the validation protocol. The paper promises every instance has a deterministic one-line success_command that prints "Setup successful" or "Setup failed", but the appendix's dependency-resolution example is just `bundle install --jobs=1 --retry=2 --without development test` — no echo at all. The validation script in Appendix C actually scores dependency tasks by regex-matching an error message, which is a different mechanism. Meanwhile the repo-setup example commands in the body are malformed: `&& echo 'Setup successful ' echo 'Setup failed'` prints the literal word "echo" and, on failure, prints nothing. This may be a typo, but it raises real doubt about whether the success labels in Table 2 are all produced by the same deterministic protocol. The authors should release the full command set and run a false-positive/false-negative audit; until that's done, the cross-category comparisons are not fully trustworthy.

The efficiency analysis is the next weak spot. The "optimal human" baseline is hand-constructed on 10 instances with an arbitrary conversion factor (a folder exploration costs 2 LLM steps), so the 38-69% wasted-step range is more interpretive than measured. Fine as an exploratory finding; it should not be a headline claim. The failure-mode counts are single-annotator, no inter-annotator agreement, which is a minor but real limitation. And the abstract's 38-89% vs the body's 38-69% discrepancy needs a fix.

None of this is fatal. The benchmark itself is plausible and useful, and the central gap it fills — agents failing to bootstrap environments — is real. But the paper currently overstates the determinism of its evaluation, and the numbers need that audit. I'd send it to serious peer review with major revisions requested.

Referee Report

4 major / 4 minor

Summary. SetupBench is a 93-instance benchmark that tests whether LLM-based software engineering agents can bootstrap a development environment from a bare Linux sandbox: installing packages, resolving dependency conflicts, initializing databases, and starting background services. Each instance provides a natural-language problem statement, a workspace snapshot, and a supposedly deterministic one-line validation command that prints 'Setup successful' or 'Setup failed'. The authors evaluate five OpenHands model variants and report overall resolve rates from 34.4% to 62.4%, with particularly low performance on repository setup (38.9–57.4%) and local database setup (20.0–53.3%). They also analyze failure modes (missing test tooling, hallucinated task constraints, non-persistent environment changes) and quantify agent inefficiency relative to a hand-constructed 'optimal human' baseline, reporting 38–69% wasted steps. The paper argues that SetupBench fills a gap left by pre-baked-environment benchmarks such as SWE-bench and DevBench.

Significance. If the benchmark's validation commands are reliable, SetupBench addresses a genuinely underevaluated capability: getting code to run, not just editing it. The benchmark's breadth across seven language ecosystems, five database engines, and multi-service orchestration is valuable, and the release of prompts, scripts, and validation harnesses is a concrete reproducibility strength. The central empirical claim that current OpenHands variants succeed on less than two-thirds of these tasks is falsifiable and directly informs agent development. However, the paper's validity hinges on the assumption that each success_command is a deterministic oracle for true setup success, and that assumption is not fully established in the manuscript. The efficiency analysis is also based on an arbitrary baseline and should be treated as exploratory until strengthened.

major comments (4)

[§2.3, Appendix A, Appendix D/E] The claimed deterministic success/failure string protocol is not consistently implemented. Section 2.3 states that every validation command prints exactly 'Setup successful' or 'Setup failed', but the dependency-resolution entry in Appendix A has success_command = 'bundle install --jobs=1 --retry=2 --without development test', which emits no such string at all; its success can only be determined by exit status. Moreover, the commands in Appendices D and E are written as '... && echo "Setup successful" echo "Setup failed"', which on success prints both strings and on failure prints nothing. If the harness parses output for the literal strings, these commands cannot work as claimed; if it falls back to exit codes for some tasks, then Table 2 mixes incomparable success criteria across categories. This undermines the benchmark's core deterministic-evaluation claim and needs to be fixed by publishing and validating all success_command strings.
[§3.4, Tables 3 and 4] The efficiency analysis depends on a hand-constructed 'optimal human' baseline whose conversion factors and exclusion rules are arbitrary. In particular, the claim that one human folder exploration equals two LLM steps is not justified, and the exclusions (think calls, finish invocations, first three steps, polling actions) are defined ad hoc with no sensitivity analysis. Since the headline result of 38–69% wasted steps is computed against this baseline, it is not robust. The paper should either validate the baseline through human study data or reframe the efficiency numbers as a rough exploratory comparison rather than a quantitative finding.
[§3.3] The failure-mode attribution counts are based on manual inspection of evaluation logs, but no coding rubric, inter-annotator agreement measure, or example classifications are provided. For instance, the statement that 'neglecting test-tool installation' accounts for roughly 17–26% of failures is presented as a precise result, yet there is no evidence that the categorization is reproducible. The authors should release the annotated trajectories and report inter-annotator agreement, or at minimum state that these are qualitative observations rather than measured percentages.
[§3.1] The evaluation protocol runs the validation command once in a fresh terminal subprocess immediately after the agent's final action. For repo-setup and background-service tasks, validation commands such as 'curl -s http://localhost:9090/metrics | grep -q prometheus_build_info' are sensitive to service startup timing, and a one-shot check can produce false negatives for services that need a few more seconds to bind. The paper reports no characterization of validation-command flakiness (e.g., repeated runs in fresh sandboxes, startup latency measurements), which is important because even a small number of mislabeled instances can materially shift the reported per-category success rates (for example, Local-DB setup has only 15 instances).

minor comments (4)

[Abstract] The abstract states 38–89% unnecessary actions, while the body and Table 4 report 38–69%; this inconsistency should be corrected.
[§3.5] The subsection 'Environment persistence across agent-human transitions' appears twice with nearly identical wording; one copy should be removed.
[Table 3] The language column contains formatting artifacts such as 'JA V A' and 'RUST'; these should be cleaned up.
[§3.4] The text refers to 'fresh Ubuntu 22.0', while Appendix B says 'Ubuntu 22.04'; the version should be consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SetupBench is an empirical benchmark whose success criteria are externally validated, not derived from the results.

full rationale

The paper makes no mathematical derivation; its claims are benchmark construction and empirical measurement. Success is defined by task-specific success_commands that were validated end-to-end in fresh sandboxes (Section 2.1) and executed by an independent harness in a fresh terminal (Section 3.1). Agent performance is measured by command output, not by model self-report or by a fitted parameter, so there is no fitted-input-called-prediction step. The efficiency analysis compares agent steps to a manually constructed human baseline, which is a comparison, not a circular derivation. The paper cites prior work (EnvBench, Installamatic) as external corroboration, not as load-bearing self-citation, and there are no uniqueness theorems or ansatz-smuggling citations. One non-circular validity concern is flagged: Section 2.3 promises every validator prints literal 'Setup successful'/'Setup failed' strings, but Appendix A's dependency-resolution success_command is only `bundle install --jobs=1 --retry=2 --without development test`, which emits no such string; this affects the reliability of the measurement but is a correctness/benchmark-fidelity issue, not a circularity of the derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the benchmark environment and validation commands, plus the representativeness of the agent and the stability of the manual analysis. No new physical or mathematical entities are introduced.

free parameters (2)

Human-to-agent step conversion factor = 2 LLM steps per human folder exploration
Used in Section 3.4 to translate human navigation into an 'optimal' step count; chosen by hand, no empirical basis.
Excluded action categories = think, finish, first three steps, polling
Section 3.4 filters these actions out of the wasted-step computation; the exclusions are ad hoc and materially affect the reported percentages.

assumptions (4)

domain assumption The base environment is Ubuntu 22.04 with no preinstalled packages
Appendix B.1 and Section 3.1; all tasks assume this minimal sandbox, and results may not transfer to cloud sandboxes with preinstalled toolchains.
domain assumption Each success command deterministically and correctly reflects successful setup
Section 2.3 claims determinism; commands were validated in fresh sandboxes but the full set is not shown, and network/service-dependent commands (e.g., curl localhost) can be flaky.
domain assumption Manual failure-mode classification is reliable
Section 3.3 assigns failures to three modes by manual inspection of logs without inter-annotator agreement or a codebook.
domain assumption OpenHands is representative of state-of-the-art coding agents
Only OpenHands with five base models is evaluated; other agent architectures (Codex, Devin, Copilot) are referenced but not tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments." pith.science (2026). https://pith.science/paper/FGC62KDM

@misc{pith2026250709063,
  author       = {Pith},
  title        = {Pith review of: SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGC62KDM}},
  note         = {Machine review of arXiv:2507.09063}
}

read the original abstract

Modern Large Language Model (LLM) agents promise end to end assistance with real-world software tasks, yet existing benchmarks evaluate LLM agents almost exclusively in pre-baked environments where every dependency is pre-installed. To fill this gap, we introduce SetupBench, a 93 instance benchmark that isolates the environment-bootstrap skill: starting from a bare Linux sandbox, an agent must install packages, resolve dependency conflicts, initialize databases, and configure background services. Our tasks span seven language ecosystems, five database engines, and multi-service orchestration scenarios, each accompanies by a natural language problem statement and a deterministic success command. Through evaluation of OpenHands, a state-of-the-art coding agent, we find low success rates across task categories, with particular challenges in repository setup (38.9-57.4%) and local database configuration (20.0-53.3%). Our analysis reveals systematic failure modes including incomplete development tooling installation, hallucinated task constraints, and non-persistent environment modifications that break agent-human collaboration workflows. We identify substantial inefficiencies in agent exploration strategies, with 38-89% of actions being unnecessary compared to optimal human behavior. These findings highlight gaps in current agents' practical environment-bootstrap capabilities. By targeting this critical yet under-evaluated capability, SetupBench provides a rigorous yard-stick for the next generation of software developer agents aiming to solve end to end real-wold tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
cs.SE 2026-06 unverdicted novelty 7.0 of 10

TUA-Bench provides 120 manually designed terminal tasks across five families with execution-based scoring; the top agent reaches 65.8% success.
Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer
cs.CR 2026-06 unverdicted novelty 7.0 of 10

CLAWAUDIT applies a STRIDE-derived taxonomy and 47 Semgrep plus 30 CodeQL rules to local LLM agent code, lifting recall on held-out OpenClaw advisories from 21.7% and 13.8% baselines to 66.8% and 75.1%.
DeployBench: Benchmarking LLM Agents for Research Artifact Deployment
cs.SE 2026-06 unverdicted novelty 7.0 of 10

DeployBench is a new benchmark of 51 research-artifact deployment tasks where four LLMs with OpenHands achieve 7.8-51% pass rates, with failures mostly from agents stopping after weaker self-checks than the paper requires.
BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledge
cs.SE 2026-05 unverdicted novelty 7.0 of 10

BootstrapAgent distills repository bootstrapping heuristics into a persistent .bootstrap contract via multi-agent evidence extraction, Docker verification, and trace-driven repair, reporting 92.9% success and efficien...
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
cs.AI 2026-07 conditional novelty 6.0 of 10

The paper introduces a 41-mode taxonomy that assigns each agent failure to an interaction edge and a fault side, and shows LLM judges can reproduce the labels with Cohen's κ=0.76.

Reference graph

Works this paper leans on

25 extracted references · 13 canonical work pages · cited by 5 Pith papers

[1]

Codemirage: Hallucinations in code generated by large language models

Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. Codemirage: Hallucinations in code generated by large language models. In Proc.\ AutoMates Workshop @ IJCAI, 2024. URL https://arxiv.org/abs/2408.08333

arXiv 2024
[2]

Aider code editing

Aider. Aider code editing. https://aider.chat/docs/benchmarks.html#the-benchmark

work page
[3]

Introducing claude 4

Anthropic. Introducing claude 4. https://www.anthropic.com/news/claude-4, 2025

work page 2025
[4]

Meet devin, the first ai software engineer

CognitionAI. Meet devin, the first ai software engineer. https://www.cognition-labs.com/blog/devin, 2024

work page 2024
[5]

Envbench: A benchmark for automated environment setup

Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. Envbench: A benchmark for automated environment setup. In ICLR 2025 Third Workshop on Deep Learning for Code, 2025. URL https://openreview.net/forum?id=izy1oaAOeX

work page 2025
[6]

Code completions with github copilot

GitHub. Code completions with github copilot. https://code.visualstudio.com/docs/copilot/ai-powered-suggestions, 2022

work page 2022
[7]

Meet the new github copilot coding agent

GitHub. Meet the new github copilot coding agent. https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/, 2025. Blog post

work page 2025
[8]

S table T ool B ench: Towards stable large-scale benchmarking on tool learning of large language models

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. S table T ool B ench: Towards stable large-scale benchmarking on tool learning of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 11143--...

doi:10.18653/v1/2024.findings-acl.664 2024

Show all 25 references

[9]

Collu- B ench: A benchmark for predicting language model hallucinations in code

Nan Jiang, Qi Li, Lin Tan, and Tianyi Zhang. Collu- B ench: A benchmark for predicting language model hallucinations in code. In Proc.\ ACM/IEEE Int’l Symp.\ on Software Testing and Analysis (ISSTA), 2024. URL https://arxiv.org/abs/2410.09997

2024 arXiv
[10]

SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/f...

2024
[11]

Butt, and Ali Anwar

Ahmad Faraz Khan, Azal Ahmad Khan, Anas Mohamed, Haider Ali, Suchithra Moolinti, Sabaat Haroon, Usman Tahir, Mattia Fazzini, Ali R. Butt, and Ali Anwar. LADs : Leveraging LLMs for AI\!-Driven DevOps . arXiv preprint arXiv:2502.20825, 2025

2025 arXiv
[12]

Devbench: A comprehensive benchmark for software development

Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, et al. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604, 2024 a

2024 arXiv
[13]

ALR ^ 2 : A retrieve-then-reason framework for long-context question answering

Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, and Yixuan Su. ALR ^ 2 : A retrieve-then-reason framework for long-context question answering. arXiv preprint arXiv:2410.03227, 2024 b . URL https://arxiv.org/abs/2410.03227

arXiv
[14]

Long-context llms struggle with long in-context learning

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024 c . URL https://arxiv.org/abs/2404.02060

2024 arXiv
[15]

Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. NL2Bash : A corpus and semantic parser for natural language interface to the linux operating system. arXiv preprint arXiv:1802.08979, 2018

2018 arXiv
[16]

Gitgoodbench: A novel benchmark for evaluating agentic performance on git

Tobias Lindenbauer, Egor Bogomolov, and Yaroslav Zharov. Gitgoodbench: A novel benchmark for evaluating agentic performance on git. arXiv preprint arXiv:2505.22583, 2025

2025 arXiv
[17]

Agent B ench: Evaluating LLM s as agents

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agent B enc...

2024
[18]

Opseval: A comprehensive benchmark suite for evaluating large language models’ capability in the IT operations domain

Yuhe Liu, Changhua Pei, Longlong Xu, Bohan Chen, Mingze Sun, Zhirui Zhang, Yongqian Sun, Shenglin Zhang, Kun Wang, et al. Opseval: A comprehensive benchmark suite for evaluating large language models’ capability in the IT operations domain. arXiv preprint arXiv:2310.07637, 2025

arXiv
[19]

Beyond pip install : Evaluating llm agents for the automated installation of python projects

Louis Milliken, Sungmin Kang, and Shin Yoo. Beyond pip install : Evaluating llm agents for the automated installation of python projects. In Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER '25, 2025

2025
[20]

Designing deep learning frameworks for llms: Challenges, expectations, and opportunities

Yanzhou Mu, Rong Wang, Juan Zhai, Chunrong Fang, Xiang Chen, Jiacong Wu, An Guo, Jiawei Shen, Bingzhuo Li, and Zhenyu Chen. Designing deep learning frameworks for llms: Challenges, expectations, and opportunities. arXiv preprint arXiv:2506.13114, 2025

2025 arXiv
[21]

Mitigating configuration differences between development and production environments: A catalog of strategies

Marcos Nazário, Rodrigo Bonifacio, and Gustavo Pinto. Mitigating configuration differences between development and production environments: A catalog of strategies. arXiv preprint arXiv:2505.09392, 2025

2025 arXiv
[22]

Identifying factors contributing to ``bad days'' for software developers: A mixed-methods study

Ike Obi, Jenna Butler, Sankeerti Haniyur, Brian Hassan, Margaret-Anne Storey, and Brendan Murphy. Identifying factors contributing to ``bad days'' for software developers: A mixed-methods study. arXiv preprint arXiv:2410.18379, 2024

arXiv
[23]

Introducing codex

OpenAI . Introducing codex. https://openai.com/index/introducing-codex, 2025. Blog post

2025
[24]

Tool LLM : Facilitating large language models to master 16000+ real-world API s

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. Tool LLM : Facilitating large language models to...

2024
[25]

Retrieval models aren't tool-savvy: Benchmarking tool retrieval for large language models

Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. Retrieval models aren't tool-savvy: Benchmarking tool retrieval for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lingui...

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Codemirage: Hallucinations in code generated by large language models

Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. Codemirage: Hallucinations in code generated by large language models. In Proc.\ AutoMates Workshop @ IJCAI, 2024. URL https://arxiv.org/abs/2408.08333

arXiv 2024

[2] [2]

Aider code editing

Aider. Aider code editing. https://aider.chat/docs/benchmarks.html#the-benchmark

work page

[3] [3]

Introducing claude 4

Anthropic. Introducing claude 4. https://www.anthropic.com/news/claude-4, 2025

work page 2025

[4] [4]

Meet devin, the first ai software engineer

CognitionAI. Meet devin, the first ai software engineer. https://www.cognition-labs.com/blog/devin, 2024

work page 2024

[5] [5]

Envbench: A benchmark for automated environment setup

Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. Envbench: A benchmark for automated environment setup. In ICLR 2025 Third Workshop on Deep Learning for Code, 2025. URL https://openreview.net/forum?id=izy1oaAOeX

work page 2025

[6] [6]

Code completions with github copilot

GitHub. Code completions with github copilot. https://code.visualstudio.com/docs/copilot/ai-powered-suggestions, 2022

work page 2022

[7] [7]

Meet the new github copilot coding agent

GitHub. Meet the new github copilot coding agent. https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/, 2025. Blog post

work page 2025

[8] [8]

S table T ool B ench: Towards stable large-scale benchmarking on tool learning of large language models

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. S table T ool B ench: Towards stable large-scale benchmarking on tool learning of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 11143--...

doi:10.18653/v1/2024.findings-acl.664 2024