{"id":"e1366bb3-f3a6-40e5-b328-cce75f9bf493","arxiv_id":"2507.09063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.","lead":"The authors present SetupBench, a 93-task benchmark that asks AI coding agents to install packages, resolve dependency conflicts, set up databases, and run background services from a bare Linux sandbox. Across five OpenHands model variants, success rates ranged from 34.4% to 62.4%, showing that environment bootstrap remains a major weakness for LLM agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity rests on one-line success commands, but their fidelity is unverified: Appendix A's dependency-resolution 'success_command' prints no 'Setup successful' string, so the claimed deterministic string protocol may not be applied consistently.","rationale":"The reader's weakest assumption was that the single-line success commands deterministically and correctly capture whether the environment is truly set up. My independent reading lands on exactly that point, and I found an internal inconsistency that strengthens it: Appendix A's dependency-resolution success_command is the bare `bundle install ...` command, with no 'Setup successful' echo, contradicting Section 2.3's universal string protocol. This suggests either the paper's examples are incomplete or the harness actually mixes exit-code and output-string criteria. Neither case is fatal by itself: the benchmark could still be valid after releasing commands and auditing them. But the central empirical claim (34.4-62.4% success, low rates in DB and repo categories) cannot be taken at face value until the oracle is verified, because every downstream result—failure-mode attribution, efficiency comparison, and design implications—is conditioned on those labels. The efficiency analysis has its own arbitrary mapping from human UI actions to LLM steps, but the validation-oracle issue is more load-bearing because it determines all success-rate numbers. I therefore keep the reader's CONDITIONAL verdict: accept only after publishing the validators and showing they are robust to timing and partial-setup states. The concrete sabotage audit proposed above would settle whether false positives and false negatives are material enough to shift the reported rates.","tokens_in":12933,"tokens_out":4318,"duration_ms":55043,"concrete_test":"Publish all 93 success_command strings and run a sabotage audit. For each instance, start from the fully set-up reference environment and (a) individually omit each required dependency or service, (b) execute the validator at 0s, 2s, and 10s after the relevant service is started, and (c) execute the validator in the reference environment from a fresh shell. Count false positives (validator prints 'Setup successful' despite a missing required component) and false negatives (validator fails on a complete setup). Separately, verify every command contains a literal echo of 'Setup successful' or specify exactly how the harness uses exit codes; if commands like Appendix A's `bundle install ...` rely on exit status, confirm that the same criterion is applied across all four task categories.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SetupBench isolates and measures environment-bootstrap skill depends entirely on each success_command being a deterministic oracle: true positive iff the environment is fully set up, and insensitive to transient conditions. That premise is not established. Section 2.3 says every validator is a single-line command printing 'Setup successful' or 'Setup failed', and Section 2.1 says commands were validated end-to-end in fresh sandboxes, but the full command set is not published and no false-positive/false-negative audit is reported. The one concrete dependency-resolution entry in Appendix A is just `bundle install --jobs=1 --retry=2 --without development test`, which emits no 'Setup successful' string at all. If the harness actually scores some tasks by exit code and others by output-string matching, Table 2 mixes incomparable success criteria across categories. Timing is a second concrete risk: repo-setup validators such as `curl -s http://localhost:9090/metrics | grep -q 'prometheus_build_info'` run once in a fresh terminal immediately after the agent's final action; a service that takes two more seconds to bind produces a false failure, while a leftover process from an earlier run can produce a false pass. With 93 instances and one run per model, even a few mislabeled commands materially shift rates: Local-DB setup has 15 instances, so one misclassification moves a cell by about 6.7 points, and Repo Setup cells have 54 instances, where a handful of bad labels can move rates outside the reported 38.9-57.4% band. Until the actual command strings are released and audited, the empirical success rates and all downstream failure-mode and efficiency analyses conditioned on those labels are not fully trustworthy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SetupBench is a 93-instance benchmark that tests whether LLM-based software engineering agents can bootstrap a development environment from a bare Linux sandbox: installing packages, resolving dependency conflicts, initializing databases, and starting background services. Each instance provides a natural-language problem statement, a workspace snapshot, and a supposedly deterministic one-line validation command that prints 'Setup successful' or 'Setup failed'. The authors evaluate five OpenHands model variants and report overall resolve rates from 34.4% to 62.4%, with particularly low performance on repository setup (38.9–57.4%) and local database setup (20.0–53.3%). They also analyze failure modes (missing test tooling, hallucinated task constraints, non-persistent environment changes) and quantify agent inefficiency relative to a hand-constructed 'optimal human' baseline, reporting 38–69% wasted steps. The paper argues that SetupBench fills a gap left by pre-baked-environment benchmarks such as SWE-bench and DevBench.","tokens_in":13227,"tokens_out":3880,"duration_ms":45163,"significance":"If the benchmark's validation commands are reliable, SetupBench addresses a genuinely underevaluated capability: getting code to run, not just editing it. The benchmark's breadth across seven language ecosystems, five database engines, and multi-service orchestration is valuable, and the release of prompts, scripts, and validation harnesses is a concrete reproducibility strength. The central empirical claim that current OpenHands variants succeed on less than two-thirds of these tasks is falsifiable and directly informs agent development. However, the paper's validity hinges on the assumption that each success_command is a deterministic oracle for true setup success, and that assumption is not fully established in the manuscript. The efficiency analysis is also based on an arbitrary baseline and should be treated as exploratory until strengthened.","major_comments":[{"comment":"The claimed deterministic success/failure string protocol is not consistently implemented. Section 2.3 states that every validation command prints exactly 'Setup successful' or 'Setup failed', but the dependency-resolution entry in Appendix A has success_command = 'bundle install --jobs=1 --retry=2 --without development test', which emits no such string at all; its success can only be determined by exit status. Moreover, the commands in Appendices D and E are written as '... && echo \"Setup successful\" echo \"Setup failed\"', which on success prints both strings and on failure prints nothing. If the harness parses output for the literal strings, these commands cannot work as claimed; if it falls back to exit codes for some tasks, then Table 2 mixes incomparable success criteria across categories. This undermines the benchmark's core deterministic-evaluation claim and needs to be fixed by publishing and validating all success_command strings.","section":"§2.3, Appendix A, Appendix D/E"},{"comment":"The efficiency analysis depends on a hand-constructed 'optimal human' baseline whose conversion factors and exclusion rules are arbitrary. In particular, the claim that one human folder exploration equals two LLM steps is not justified, and the exclusions (think calls, finish invocations, first three steps, polling actions) are defined ad hoc with no sensitivity analysis. Since the headline result of 38–69% wasted steps is computed against this baseline, it is not robust. The paper should either validate the baseline through human study data or reframe the efficiency numbers as a rough exploratory comparison rather than a quantitative finding.","section":"§3.4, Tables 3 and 4"},{"comment":"The failure-mode attribution counts are based on manual inspection of evaluation logs, but no coding rubric, inter-annotator agreement measure, or example classifications are provided. For instance, the statement that 'neglecting test-tool installation' accounts for roughly 17–26% of failures is presented as a precise result, yet there is no evidence that the categorization is reproducible. The authors should release the annotated trajectories and report inter-annotator agreement, or at minimum state that these are qualitative observations rather than measured percentages.","section":"§3.3"},{"comment":"The evaluation protocol runs the validation command once in a fresh terminal subprocess immediately after the agent's final action. For repo-setup and background-service tasks, validation commands such as 'curl -s http://localhost:9090/metrics | grep -q prometheus_build_info' are sensitive to service startup timing, and a one-shot check can produce false negatives for services that need a few more seconds to bind. The paper reports no characterization of validation-command flakiness (e.g., repeated runs in fresh sandboxes, startup latency measurements), which is important because even a small number of mislabeled instances can materially shift the reported per-category success rates (for example, Local-DB setup has only 15 instances).","section":"§3.1"}],"minor_comments":[{"comment":"The abstract states 38–89% unnecessary actions, while the body and Table 4 report 38–69%; this inconsistency should be corrected.","section":"Abstract"},{"comment":"The subsection 'Environment persistence across agent-human transitions' appears twice with nearly identical wording; one copy should be removed.","section":"§3.5"},{"comment":"The language column contains formatting artifacts such as 'JA V A' and 'RUST'; these should be cleaned up.","section":"Table 3"},{"comment":"The text refers to 'fresh Ubuntu 22.0', while Appendix B says 'Ubuntu 22.04'; the version should be consistent.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction is plausible and the repository release is a useful contribution, but the success-command inconsistency in Appendix A is a load-bearing issue that the authors must fix before the paper can be accepted. The efficiency baseline needs to be substantially strengthened or descoped. I would not reject outright, because the underlying idea is sound and the issues appear addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SetupBench is worth a close look. It isolates a real capability that SWE-bench-style benchmarks ignore: bootstrapping a dev environment from a bare sandbox. Existing work like EnvBench covers a subset; SetupBench adds databases, background services, OS-level packages, and multi-language repo setup, and it ships mining and validation scripts. That is a genuine contribution, and the evaluation across five OpenHands models gives useful signal (Claude 4 at 62% vs GPT-4o at 34%). Section 5 is refreshingly honest about limitations, and the release of prompts and scripts makes the work reproducible in principle.\n\nThe main soft spot is the validation protocol. The paper promises every instance has a deterministic one-line success_command that prints \"Setup successful\" or \"Setup failed\", but the appendix's dependency-resolution example is just `bundle install --jobs=1 --retry=2 --without development test` — no echo at all. The validation script in Appendix C actually scores dependency tasks by regex-matching an error message, which is a different mechanism. Meanwhile the repo-setup example commands in the body are malformed: `&& echo 'Setup successful ' echo 'Setup failed'` prints the literal word \"echo\" and, on failure, prints nothing. This may be a typo, but it raises real doubt about whether the success labels in Table 2 are all produced by the same deterministic protocol. The authors should release the full command set and run a false-positive/false-negative audit; until that's done, the cross-category comparisons are not fully trustworthy.\n\nThe efficiency analysis is the next weak spot. The \"optimal human\" baseline is hand-constructed on 10 instances with an arbitrary conversion factor (a folder exploration costs 2 LLM steps), so the 38-69% wasted-step range is more interpretive than measured. Fine as an exploratory finding; it should not be a headline claim. The failure-mode counts are single-annotator, no inter-annotator agreement, which is a minor but real limitation. And the abstract's 38-89% vs the body's 38-69% discrepancy needs a fix.\n\nNone of this is fatal. The benchmark itself is plausible and useful, and the central gap it fills — agents failing to bootstrap environments — is real. But the paper currently overstates the determinism of its evaluation, and the numbers need that audit. I'd send it to serious peer review with major revisions requested.","headline":"A genuinely useful benchmark for environment-bootstrap evaluation, but the validation protocol is less uniform than advertised; fix the success-command audit before trusting the numbers.","tokens_in":13776,"tokens_out":3706,"would_cite":true,"duration_ms":44158,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SetupBench isolates the environment-bootstrap skill and finds current coding agents succeed on only 34–62% of 93 bare-sandbox setup tasks.","keywords":["environment bootstrap","LLM agents","benchmark","dependency resolution","database setup","DevOps","software engineering"],"falsifier":"Take the 93 success commands, run each in a fresh, correctly configured sandbox ten times (with small delays before execution), and count how often 'Setup successful' appears when every documented setup step was performed; any false-negative rate above a negligible threshold, or any case where a command prints success despite an obvious missing component (e.g., a missing test runner), would undermine the reported success rates and cross-model comparisons.","tokens_in":12756,"feed_emoji":"🐧","tokens_out":2518,"duration_ms":29481,"temperature":0.7,"pith_summary":"This paper claims that existing LLM-agent benchmarks test agents in pre-baked environments where every dependency is already installed, so they miss a core developer skill: getting code to run from scratch. SetupBench closes that gap with 93 curated tasks that start in a bare Linux sandbox and require installing packages, resolving dependency conflicts, initializing databases, and configuring background services. Each task comes with a deterministic one-line success command, and evaluation of the OpenHands agent across five model variants shows low success rates, especially in repository setup (38.9–57.4%) and local database configuration (20.0–53.3%). If correct, SetupBench provides a reproducible yardstick for a previously unevaluated capability and exposes concrete failure modes that agent designers can target.","feed_headline":"AI agents fail 38% of environment-setup tasks in new benchmark","feed_subtitle":"Starting from bare Linux sandboxes, the best model still finishes only 62% of 93 bootstrap tasks, exposing a hidden gap.","key_machinery":"The load-bearing mechanism is the deterministic one-line validation command (success_command) paired with a fresh minimal Linux sandbox. Each task defines a single command that echoes 'Setup successful' only when the intended environment changes have taken effect, allowing automated, reproducible pass/fail evaluation without LLM-as-a-judge or flaky test suites. This design lets the benchmark isolate the bootstrap skill and attribute success or failure to the agent's setup behavior rather than to pre-installed dependencies.","core_discovery":"SetupBench is a 93-instance benchmark that isolates the environment-bootstrap skill by presenting agents with a natural-language problem statement, a workspace snapshot, and a deterministic success command that prints 'Setup successful' or 'Setup failed'. It spans seven language ecosystems, five database engines, and multi-service orchestration scenarios, and it evaluates agents in minimal containers rather than pre-configured Docker images. The paper's central finding is that current state-of-the-art coding agents resolve fewer than two-thirds of these tasks (best model 62.4%), with the largest gaps in repository setup and local database configuration. The paper also identifies three recurring failure modes—incomplete development-tooling installation, hallucinated task constraints, and non-persistent environment modifications—and quantifies agent inefficiency: 38–69% of agent steps are wasted compared with an optimal human baseline across the ten instances analyzed.","pith_inferences":["A natural extension the paper leaves implicit is chaining SetupBench with downstream coding tasks (bug fixing, feature implementation) so the evaluation measures whether a correctly bootstrapped environment actually enables subsequent work; such an extension would test continuity across the full agent workflow.","The validity of every comparison in the paper rests on the success commands being both sufficient and timing-robust; a reader should check whether commands that curl localhost endpoints or check file contents can produce false negatives due to service startup latency rather than setup failure.","The efficiency results suggest a concrete testable design: agents given a pre-computed repository structure (e.g., tree of files) should show a measurable drop in wasted exploration steps; this could be validated within SetupBench's existing 10-instance subset.","The paper's failure-mode taxonomy could be turned into a diagnostic tool: automatically labeling failing trajectories by the three modes (missing tooling, hallucinated constraints, non-persistence) would let the community track whether agent improvements actually address these bottlenecks."],"forward_implications":["If SetupBench is a valid measure, environment bootstrap is a distinct, currently under-evaluated capability that leaderboard rankings built on pre-baked environments do not capture.","Agent architectures need explicit persistence protocols—writing environment changes to configuration files and sourcing them—so that setup survives agent-human handoffs.","Context-aware exploration mechanisms, such as repository-tree injection or semantic file ranking, could reduce the 38–69% wasted steps and improve setup success.","Hybrid model-selection strategies are implied: lightweight models might handle routine setup while stronger models are reserved for dependency resolution and multi-service orchestration.","The benchmark's categories provide a graded testbed for future agents, with database setup and repository setup acting as the hardest currently."],"supporting_citations":[{"why":"SWE-Bench is the primary contrast: it ships tasks in pre-baked Docker images with all dependencies installed, which SetupBench argues leaves environment bootstrap untested.","marker":"[Jimenez et al., 2024]"},{"why":"DevBench similarly distributes ready-made containers and broadens scope to design/coding/testing but still assumes a working environment, reinforcing the gap SetupBench fills.","marker":"[Li et al., 2024a]"},{"why":"AgentBench evaluates multi-step agents across generic domains but covers few software tasks and no system configuration, supporting the claim that setup is missing from agent benchmarks.","marker":"[Liu et al., 2024]"},{"why":"EnvBench is the closest antecedent targeting automated environment setup; SetupBench extends it by adding OS-level packages, databases, and daemon orchestration, and its non-persistence failure alignment is cited.","marker":"[Eliseeva et al., 2025]"},{"why":"Installamatic's finding that 45% of runs break because executables installed with --user are not visible in later sessions supports the paper's non-persistent environment failure mode.","marker":"[Milliken et al., 2025]"}],"fun_headline_variants":["New benchmark: AI agents flunk 38% of dev-env setups","SetupBench: agents struggle to bootstrap dev environments","62% success: AI agents trip on environment setup","Bare Linux sandbox exposes AI agents' setup failures","Benchmark isolates why AI agents botch environment setup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rides on the assumption that each single-line success command deterministically and correctly captures whether the environment was truly set up, even though the full set of commands is not shown in the paper and commands that probe localhost endpoints or file contents can be sensitive to service timing and port availability.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark: AI agents flunk 38% of dev-env setups","SetupBench: agents struggle to bootstrap dev environments","62% success: AI agents trip on environment setup","Bare Linux sandbox exposes AI agents' setup failures","Benchmark isolates why AI agents botch environment setup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3281,"prompt_tokens":982,"completion_tokens":2299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2218}},"tokens_in":598,"tokens_out":2299,"duration_ms":17313,"temperature":1.0,"reasoning_tokens":2218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:05:02.700476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 93 success commands, run each in a fresh, correctly configured sandbox ten times (with small delays before execution), and count how often 'Setup successful' appears when every documented setup step was performed; any false-negative rate above a negligible threshold, or any case where a command prints success despite an obvious missing component (e.g., a missing test runner), would undermine the reported success rates and cross-model comparisons.","supporting_citations":[{"cited_title":"Envbench: A benchmark for automated environment setup","cited_arxiv_id":null,"evidence_quote":"EnvBench is the closest antecedent targeting automated environment setup; SetupBench extends it by adding OS-level packages, databases, and daemon orchestration, and its non-persistence failure alignment is cited."},{"cited_title":"Beyond pip install : Evaluating llm agents for the automated installation of python projects","cited_arxiv_id":null,"evidence_quote":"Installamatic's finding that 45% of runs break because executables installed with --user are not visible in later sessions supports the paper's non-persistent environment failure mode."}],"review_version":1}