{"id":"fdda4489-8fa3-48ef-9156-8e96a54f088e","arxiv_id":"2505.23419","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SWE-bench-Live provides a live, automatically curated, Docker-backed benchmark of 1,319 fresh GitHub issue-fixing tasks, on which leading agents score around 19%, well below their SWE-bench Verified results.","lead":"SWE-bench-Live is a new benchmark that collects 1,319 recent real GitHub issues across 93 Python repositories and packages each task in its own Docker image. It introduces an automated pipeline to build and validate these tasks, and reports that leading AI coding agents solve far fewer of them than on the older SWE-bench test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automated REPOLAUNCH environments are un-audited and explicitly tolerate baseline test failures; if base environments are wrong, FAIL_TO_PASS labels, resolved rates, and the Live-vs-Verified overfitting gap all inherit the error.","rationale":"The reader's weakest assumption identifies exactly the load-bearing vulnerability I find: the automated environment setup and validation have no reported success rate, no human audit, and an unquantified tolerance for baseline test failures. This matters because the paper's two central outputs — a valid 1,319-instance executable benchmark and the overfitting conclusion drawn from the Live-versus-Verified performance gap — both presuppose that the Docker environments are faithful to the repository state at each base_commit. My proposed check is a direct audit of that presupposition: rebuilding a sample of environments with a deterministic procedure and comparing test transitions to the released labels. If the labels survive the audit, the benchmark's validity claim is substantially strengthened. If they do not, the reported resolved rates are not interpretable. The reader's CONDITIONAL verdict is the right one; my analysis does not move it, because the condition it names is precisely the missing evidence the paper must supply. I also note that Table 5's column header appears mislabeled (the first numeric column reads more naturally as average files than as instance counts), but this is a presentation issue secondary to the environment-fidelity question, so it does not change my verdict.","tokens_in":20517,"tokens_out":4689,"duration_ms":46356,"concrete_test":"Independently rebuild a random sample of 50 released instances: for each, recreate the repository at its base_commit using a deterministic, non-agentic procedure (e.g., dependency versions pinned to the base_commit timestamp, following the repository's own CI workflow), run the released test_cmds before and after applying the gold patch, and compare the resulting PASS/FAIL transitions against the released FAIL_TO_PASS and PASS_TO_PASS lists. Also record how many base environments still have any failing test at base_commit. If the environment setup success rate is materially below 100% or any sampled instance's F2P/P2P labels change, the benchmark's validity claim and the 19.25%-versus-43.20% comparison are not yet supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SWE-bench-Live is a valid, executable, contamination-resistant benchmark, and the headline comparison of 19.25% on SWE-bench-Live versus 43.20% on SWE-bench Verified, both rest on REPOLAUNCH faithfully recreating each repository at its base_commit. Section 3.2 defines a valid environment as one whose test suite passes with 'zero or tolerable failures', and the Verify Agent prompt in Appendix G instructs the verifier to 'tolerate a few test cases failures—as long as most tests pass'. The tolerance threshold is never quantified, the setup is performed by an LLM agent and then self-validated by another LLM agent rather than against a ground-truth procedure, and neither the environment setup success rate nor any human audit of the 1,319 Docker images is reported. Concretely, if a test fails at base_commit because of dependency drift or an incomplete setup, and that failure is accepted as 'tolerable', the test can be recorded as FAIL_TO_PASS merely because the gold patch happens to touch the same code path, producing a spurious task label. Conversely, if the setup agent stops once 'most tests pass' on the wrong dependency versions, every resolved-rate evaluation is measuring behavior in an environment that does not match the issue's historical snapshot. Because the Live-versus-Verified gap in Section 4.2 is the sole quantitative evidence for the overfitting conclusion, a systematic bias in which repositories and instances REPOLAUNCH can set up — for example, favoring small, dependency-light projects — would change both the composition of the 1,319 instances and the reported performance gap. The paper's Limitations section (Appendix F) mentions only LLM randomness and Python-only coverage, omitting this setup-fidelity risk entirely.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWE-bench-Live, a continuously updatable benchmark of 1,319 issue-resolution instances drawn from 93 Python repositories, with issues created between January 2024 and April 2025. The benchmark is constructed by REPOLAUNCH, an automated pipeline that crawls issue-PR pairs, builds per-instance Docker images, and validates FAIL_TO_PASS and PASS_TO_PASS transitions by executing tests. On a 300-instance Lite subset and the full benchmark, the authors evaluate OpenHands, SWE-Agent, and Agentless with four LLMs and report resolved rates of 7.00-17.67% on Lite and a top full-set rate of 19.25%. A controlled rerun of OpenHands with Claude 3.7 Sonnet on SWE-bench Verified reaches 43.20% under the same setup, which the authors interpret as evidence that agents overfit to static benchmarks. The paper also analyzes resolved rates by repository origin, issue recency, and patch and repository difficulty.","tokens_in":20827,"tokens_out":6919,"duration_ms":67905,"significance":"If the benchmark's environment construction and task labels are trustworthy, SWE-bench-Live would be a valuable community resource: it is among the first live, automatically curated repository-level issue-resolution benchmarks, it ships Docker images for reproducibility, and its time-machine mechanism for dependency pinning is a sensible design choice. The same-agent, same-setup rerun on SWE-bench Verified is a useful methodological step and gives a concrete, falsifiable observation (19.25% vs. 43.20%). However, the central comparison rests on automated environment setup that is not audited and whose tolerance for test failures is not quantified, and the repository-origin analysis contains an internal inconsistency. Because these issues directly affect the benchmark-validity and overfitting claims, the paper needs substantial revision before the headline conclusions can be accepted.","major_comments":[{"comment":"The definition of a valid execution environment as one whose test suite passes with 'zero or tolerable failures' is never quantified, and the Verify Agent prompt explicitly instructs the verifier to 'tolerate a few test cases failures—as long as most tests pass.' Since both environment setup and verification are performed by LLM agents, with no reported setup success rate and no human audit of the 1,319 Docker images or FAIL_TO_PASS labels, a baseline test failure that is accepted as tolerable can turn a test into a spurious FAIL_TO_PASS whenever the gold patch touches the same code path. Please report the REPOLAUNCH setup success rate, quantify the tolerance threshold, and provide at least a sampled human audit of environments and labels; without this, the benchmark-validity claim and the headline resolved rates are not yet supported.","section":"§3.2 and Appendix G"},{"comment":"The repository-origin analysis is internally inconsistent: the text states that '216 instances are derived from 8 repositories' originally in SWE-bench and that the remaining 1,103 instances are from other repositories, but Table 5 reports 744 and 383 instances for the two groups. These counts do not sum to the 1,319 total, and the table does not state which agent-model combination produced the 22.96% and 18.89% resolved rates or on which subset (Lite or full) they were computed. Because this table is the main evidence for the claim that agents are overfit to SWE-bench repositories, the counts, labels, and evaluation condition must be corrected and reconciled with Tables 3 and 4.","section":"§4.2 and Table 5"},{"comment":"The controlled comparison is not controlled for instance difficulty or curation quality. SWE-bench Verified is a manually curated subset of 500 instances selected for resolvability, whereas SWE-bench-Live is automatically constructed with no human validation and with an unquantified 'tolerable failures' rule. The 43.20% versus 19.25% gap could therefore reflect differences in task difficulty or label quality rather than overfitting to static benchmarks. The descriptive difficulty analysis in Section 4.4 does not provide a difficulty-matched comparison. Please add either a matched analysis (e.g., comparing instances from the same repositories with similar patch sizes and repository sizes) or explicitly temper the overfitting conclusion as a hypothesis requiring further evidence.","section":"§4.2"},{"comment":"The headline claim rests on single-run numbers with no error bars, confidence intervals, or repeated runs; the authors acknowledge in Section F and in the NeurIPS checklist that experiments were not repeated. Given that the resolved-rate gap is the central quantitative result, please provide at least bootstrap confidence intervals over instances and, if feasible, repeated runs of the top configuration, so that the reader can judge whether the 19.25% versus 43.20% gap is stable.","section":"§4.1 and Section F"}],"minor_comments":[{"comment":"There are several typos and infelicities: 'be be unpurposely' in the Introduction, 'task istances' in the Introduction, and 'interpreatation' in Appendix C should be corrected.","section":"Abstract, §1, Appendix C"},{"comment":"Figure 5 appears to contain three heatmap panels with the same caption; the layout and the meaning of each panel should be clarified in the caption and in the text.","section":"Figure 5"},{"comment":"The paper states that repositories were manually categorized, but the classification criteria and the number of annotators are not described; please provide this information.","section":"§3.4 and Appendix A"},{"comment":"Appendix A lists 'N/A' licenses for sphinx and sympy, which appears to contradict Section 3.1's statement that only repositories with a valid open-source license were retained; please clarify.","section":"Appendix A"},{"comment":"The mention of OmniGIRL in Section 4.2 lacks a citation or reference; please add it or remove the comparison.","section":"§4.2"},{"comment":"References [14] and [15] appear to be the same SWE-Gym paper, and references [20] and [21] appear to be the same SWE-Agent paper; please merge or correct the duplicates.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a software-engineering or ML benchmark venue, and the core idea is timely. The main risk is not novelty but verification: the automated environment pipeline is self-validated by LLM agents, the tolerance threshold is unquantified, and the repository-origin analysis contains a clear internal inconsistency. I would ask the authors to supply the audit and corrected analyses before publication; no concerns about the citation pattern beyond the internal inconsistencies noted above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2505.23419. The benchmark itself is the real contribution: 1,319 fresh, repository-level issue-resolving instances across 93 Python repos, each with a Docker image, collected by an automated pipeline. That is genuinely new and fills a real gap — LiveCodeBench only covers algorithmic puzzles, and every SWE-bench variant is static and at least partly hand-curated. The controlled rerun of OpenHands + Claude 3.7 on SWE-bench Verified under the same settings (43.20%) versus 19.25% on SWE-bench-Live is the right kind of evidence for the overfitting claim, and the paper earns that comparison.\n\nBut the load-bearing risk is exactly what the stress-test flags. Section 3.2 defines a valid environment as one whose tests pass with 'zero or tolerable failures', and the Verify Agent prompt in Appendix G tells the verifier to 'tolerate a few test cases failures—as long as most tests pass'. That threshold is never quantified. Environment setup is done by an LLM agent and then self-validated by another LLM agent, with no reported setup success rate and no human audit of the 1,319 Docker images. If a base-commit test fails due to dependency drift and gets waved through as tolerable, a spurious FAIL_TO_PASS label can be created. That would poison the instance labels and the resolved-rate numbers, including the headline gap. This is a real hole, and the Limitations appendix (F) omits it entirely — it only mentions LLM randomness and Python-only coverage.\n\nTable 5 also has an internal inconsistency the authors should fix: the text says 216 instances come from 8 SWE-bench repos, but the table lists 744 instances 'From SWE-bench Repos' and 383 from non-SWE-bench repos, which also does not sum to 1,319. And the difficulty comparison is muddy: non-SWE-bench repos are supposedly simpler (68k vs 223k LOC) yet resolved rates are lower there, which is consistent with overfitting but equally consistent with curation or selection effects. The recency analysis in Figure 4 is a good start, but it does not isolate recency from instance difficulty.\n\nMinor: no error bars or repeated runs, acknowledged; for a benchmark paper that is acceptable but worth noting. Citation pattern is fine — the self-cites (SWE-Fixer, DI-Bench, UFO) are for pipeline components, not for the central result.\n\nBottom line: this is a useful, citable benchmark that deserves a serious referee. The fixes are tractable: report REPOLAUNCH success rate, audit a sample of environments (even 50–100), quantify the failure tolerance, fix Table 5, and add the setup-fidelity limitation. I would engage with it in review rather than desk-reject.","headline":"Genuinely useful live benchmark, but the overfitting claim leans on an un-audited automated environment pipeline and a contradictory table.","tokens_in":21456,"tokens_out":2341,"would_cite":true,"duration_ms":20143,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces SWE-bench-Live, a continuously updated benchmark of 1,319 fresh GitHub issues, and shows the best agent-model pair resolves only 19.25% of them, under half its 43.20% score on SWE-bench Verified under identical…","keywords":["code LLM evaluation","issue resolution","live benchmark","data contamination","Docker environments","software engineering agents","GitHub issues","automated benchmark curation"],"falsifier":"Select, say, 100 instances at random from SWE-bench-Live, have a human with a fresh checkout rebuild each Docker environment from the recorded base commit following only the repository README, and run the gold patch. If the published FAIL_TO_PASS tests fail to reproduce in a substantial fraction (more than 10%) of those manual rebuilds, the benchmark's labels, and hence the resolved-rate gap versus SWE-bench, would not reflect genuine issue resolution.","tokens_in":20319,"feed_emoji":"🐛","tokens_out":5412,"duration_ms":51596,"temperature":0.7,"pith_summary":"The paper is trying to establish that a live, automatically curated benchmark of recent real-world GitHub issues can replace static ones like SWE-bench, and that current state-of-the-art coding agents perform far worse on it. It reports 1,319 task instances drawn from issues opened since January 2024 across 93 Python repositories, each packaged with its own Docker image. Its central evidence is a controlled comparison: the same agent-model combination that resolves 43.20% of SWE-bench Verified resolves only 19.25% of SWE-bench-Live. The authors read this gap as evidence that agents overfit to static benchmarks, and they show that difficulty rises sharply once a fix touches multiple files or large codebases.","feed_headline":"Fresh GitHub bugs drop top coding agents to 19%","feed_subtitle":"The same model that scores 43% on SWE-bench manages just 19% on new issues, a sign of benchmark overfitting.","key_machinery":"The machine that carries the argument is REPOLAUNCH, an agent-driven pipeline that converts a raw issue-pull-request pair into a validated task instance. Its steps are: locating relevant setup files, selecting a base Docker image, interactively installing dependencies and building the project inside a container, verifying that the test suite passes with an unquantified 'zero or tolerable failures' threshold, and committing the container as an instance-specific image. A time-machine proxy forces pip to resolve package versions no newer than the base commit's date, protecting the environment from dependency drift. Validation then requires at least one test that transitions from failing to passing after the gold patch, confirmed across repeated runs.","core_discovery":"The central claim is that SWE-bench-Live is a valid, contamination-resistant, executable benchmark whose fresh instances expose generalization gaps hidden by static benchmarks. Each instance is a real issue and its merged pull request, validated by reproducible FAIL_TO_PASS and PASS_TO_PASS test transitions inside an automatically built Docker environment. Running three agent frameworks plus four LLMs, the paper finds the best resolved rate on the full benchmark is 19.25%, versus 43.20% for the same setup on SWE-bench Verified, and that performance drops below ten percent once the gold patch touches three or more files or exceeds one hundred lines. The authors conclude that high scores on SWE-bench-like datasets may partly reflect memorization or implicit optimization for those repositories rather than general ability to fix unseen bugs.","pith_inferences":["A direct test of the overfitting hypothesis that the paper does not run: evaluate models on instances created after their training cutoff and on pre-cutoff instances matched for difficulty; if post-cutoff performance is systematically lower, contamination rather than task difficulty explains part of the gap.","One could use SWE-bench-Live's granular difficulty labels (files, hunks, lines, repo size) to build a cheap stratified evaluation set with calibrated confidence, instead of running the full 1,319 instances.","The benchmark's design suggests a natural adversarial use: agents could be trained on past SWE-bench-Live months and tested on the next month's release, turning the benchmark into a rolling generalization test rather than a one-time snapshot.","Because the dataset is Python-only and dominated by popular repositories, the contamination-resistant conclusion may not transfer to less popular codebases or other languages until the pipeline is extended."],"forward_implications":["If the benchmark's labels hold, leaderboard scores on static benchmarks cannot be taken as evidence of general issue-resolving ability until confirmed on fresh instances.","The same agent-model pair's 43.20%-versus-19.25% gap becomes a concrete upper bound on how much of SWE-bench Verified performance generalizes to unseen, recent issues.","REPOLAUNCH's automation, if reliable, removes the main bottleneck to benchmark construction, making monthly updates and much broader repository coverage feasible.","Difficulty analysis predicts that progress will need to come from multi-file coordination: single-file, few-line fixes are solved nearly half the time, while three-plus-file patches fall below ten percent.","Subsequent releases of the benchmark provide a standing contamination check: any jump in resolved rate on issues opened before a model's training cutoff would signal memorization."],"supporting_citations":[{"why":"Defines the issue-resolving task and the evaluation protocol that SWE-bench-Live adopts.","marker":"[10]"},{"why":"The SWE-bench Verified subset used as the controlled comparison baseline in the 43.20% versus 19.25% result.","marker":"[13]"},{"why":"SWE-Gym, cited as evidence that manual environment setup takes over 200 hours, motivating REPOLAUNCH.","marker":"[14]"},{"why":"LiveCodeBench, the live-benchmark precedent that this work extends from algorithmic problems to repository-level tasks.","marker":"[9]"},{"why":"OpenHands, one of the three agent frameworks evaluated and the one in the best-performing pair.","marker":"[16]"},{"why":"SWE-Agent, the second evaluated agent framework.","marker":"[20]"},{"why":"Agentless, the third evaluated agent framework.","marker":"[17]"},{"why":"ReAct, the thought-action-observation loop that REPOLAUNCH's setup agent follows.","marker":"[24]"},{"why":"SWE-Fixer, whose improved issue-PR pair heuristics are incorporated into the crawling stage.","marker":"[18]"}],"fun_headline_variants":["New live bug benchmark exposes coding AI's overfitting","Coding agents fall to 19% on fresh GitHub issues","Static SWE-bench scores don't hold up on live bugs","Fresh issues show coding LLMs overfit old benchmarks","SWE-bench-Live reveals 43% to 19% generalization gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on REPOLAUNCH faithfully recreating each repository at its base commit and on the verifier correctly identifying the test command and accepting only genuinely healthy environments; if environments are systematically wrong or biased toward easy projects, the FAIL_TO_PASS labels and the 19.25%-versus-43.20% gap become unreliable.","fun_headline_variants_meta":{"raw":{"variants":["New live bug benchmark exposes coding AI's overfitting","Coding agents fall to 19% on fresh GitHub issues","Static SWE-bench scores don't hold up on live bugs","Fresh issues show coding LLMs overfit old benchmarks","SWE-bench-Live reveals 43% to 19% generalization gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3051,"prompt_tokens":978,"completion_tokens":2073,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":1986}},"tokens_in":594,"tokens_out":2073,"duration_ms":13707,"temperature":1.0,"reasoning_tokens":1986,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:45:45.416073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select, say, 100 instances at random from SWE-bench-Live, have a human with a fresh checkout rebuild each Docker environment from the recorded base commit following only the repository README, and run the gold patch. If the published FAIL_TO_PASS tests fail to reproduce in a substantial fraction (more than 10%) of those manual rebuilds, the benchmark's labels, and hence the resolved-rate gap versus SWE-bench, would not reflect genuine issue resolution.","supporting_citations":[{"cited_title":"Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"Defines the issue-resolving task and the evaluation protocol that SWE-bench-Live adopts."},{"cited_title":"Introducing swe-bench verified, 2025","cited_arxiv_id":null,"evidence_quote":"The SWE-bench Verified subset used as the controlled comparison baseline in the 43.20% versus 19.25% result."}],"review_version":1}