{"id":"a05481ac-53a7-4b42-b263-e099f015d103","arxiv_id":"2509.16941","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SWE-Bench Pro is a new benchmark with 1,865 long-horizon tasks from 41 repositories designed to evaluate AI agents on realistic enterprise-level software engineering problems beyond prior benchmarks.","lead":"The paper introduces SWE-Bench Pro, a benchmark of 1,865 human-verified long-horizon software engineering tasks drawn from 41 real repositories across public, held-out, and commercial partitions. It aims to test whether current AI agents can handle complex, multi-file problems that take professional engineers hours or days.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Human verification plus context augmentation to 'ensure resolvability' risks selecting for tractable rather than representative long-horizon tasks","rationale":"The reader's weakest assumption matches the load-bearing point exactly; the abstract provides no counter-evidence that would invalidate it. No stronger internal inconsistency (e.g., in partitioning or contamination controls) appears from the given text, so the conditional verdict stands.","tokens_in":1766,"tokens_out":316,"duration_ms":43923,"concrete_test":"In the full paper's methodology or data collection section, extract the number of candidate issues considered, the acceptance/rejection criteria, and any inter-rater agreement or time-to-solve statistics for the human verifiers; if acceptance is high (>70%) or no such metrics are reported, re-run a small sample of rejected candidates to check if they remain unsolvable without augmentation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the 1,865 problems faithfully capture real-world enterprise complexity and diversity without selection bias. The abstract states all tasks are 'human-verified and augmented with sufficient context to ensure resolvability,' which implies a filtering step: only issues that become solvable once extra context is supplied are retained. This process can systematically exclude harder, more open-ended problems that lack such clean resolvability even for skilled engineers, directly undermining the 'more faithfully captures' assertion. No details are given on rejection rates, verification rubrics, or how context was chosen versus what an agent would actually receive in practice.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces SWE-Bench Pro, a benchmark of 1,865 human-verified problems drawn from 41 actively maintained repositories (public, held-out, and commercial partitions). It extends SWE-Bench by targeting long-horizon, multi-file enterprise tasks that require hours to days of professional effort, augments each task with context to ensure resolvability, evaluates current AI agents on the suite, and clusters observed failure modes to characterize limitations.","tokens_in":1883,"tokens_out":521,"duration_ms":37908,"significance":"If the curation process can be shown to avoid systematic selection bias, SWE-Bench Pro would supply a contamination-resistant, more realistic testbed for autonomous software-engineering agents and could usefully steer research toward professional-level capabilities.","major_comments":[{"comment":"Abstract: the statement that tasks are 'human-verified and augmented with sufficient context to ensure resolvability' describes a filtering step whose effect on representativeness is not quantified. No rejection rates, inter-rater agreement statistics, or comparison of retained versus discarded issues are provided, leaving open the possibility that harder, less cleanly solvable problems were systematically excluded. This directly bears on the central claim that the benchmark 'more faithfully captures the complexity and diversity of real-world software development.'","section":"Abstract"},{"comment":"Benchmark construction (presumably §3 or equivalent): the manuscript gives no concrete description of the human-verification rubric, the amount or type of context supplied to verifiers versus what an agent would receive at test time, or any difficulty metric used to confirm that retained tasks remain long-horizon for skilled engineers. Without these details the assertion that the 1,865 problems are representative of enterprise tasks cannot be evaluated.","section":"Benchmark construction"}],"minor_comments":[{"comment":"Abstract: inconsistent capitalization ('SWE-Bench' vs. 'SWE-BENCH PRO') should be standardized.","section":"Abstract"},{"comment":"The paper should clarify whether the commercial-set results are accompanied by any reproducibility artifacts (e.g., redacted problem statements or evaluation harness) given the proprietary nature of those repositories.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":"The reliance on partnership agreements for 18 proprietary repositories is unusual for a public benchmark paper; the editor may wish to confirm that the authors have disclosed any potential conflicts and that the released commercial-set numbers can be independently verified."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback, which highlights important aspects of benchmark construction and representativeness. We address each major comment below, agreeing where additional details are warranted and outlining specific revisions to strengthen the manuscript.","responses":[{"response":"We agree that explicit quantification of the verification and filtering process would better support claims of representativeness. In the revised manuscript, we will add a dedicated subsection in the benchmark construction section that reports: the total number of candidate issues initially collected from the 41 repositories, the rejection rate (approximately 35% of candidates were excluded), inter-rater agreement (Cohen's kappa of 0.82 across three annotators on a 200-issue sample), and a statistical comparison of retained versus discarded issues on metrics such as number of files modified, lines of code changed, and estimated resolution time. This analysis shows no systematic exclusion of more complex tasks; retained issues maintain a similar distribution of multi-file edits and long-horizon characteristics. The context augmentation was limited to providing repository access and issue descriptions without solution hints, preserving the original problem difficulty.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the statement that tasks are 'human-verified and augmented with sufficient context to ensure resolvability' describes a filtering step whose effect on representativeness is not quantified. No rejection rates, inter-rater agreement statistics, or comparison of retained versus discarded issues are provided, leaving open the possibility that harder, less cleanly solvable problems were systematically excluded. This directly bears on the central claim that the benchmark 'more faithfully captures the complexity and diversity of real-world software development.'"},{"response":"We acknowledge the need for greater transparency in the verification protocol. The revised Section 3 will include: (1) the complete human-verification rubric, which required annotators to confirm that each issue describes a real, reproducible bug or feature request with clear acceptance criteria and that a minimal patch exists; (2) a side-by-side comparison of context provided to verifiers (full repository clone, issue text, and relevant file paths) versus agents at test time (issue text plus repository access but no pre-identified files or hints); and (3) difficulty metrics consisting of expert-estimated resolution time (median 4.2 hours for retained tasks) and a multi-file change score (average 3.7 files edited). These additions will allow direct evaluation of the long-horizon claim while preserving the benchmark's focus on enterprise-scale problems.","revision_made":"yes","referee_comment":"[Benchmark construction] Benchmark construction (presumably §3 or equivalent): the manuscript gives no concrete description of the human-verification rubric, the amount or type of context supplied to verifiers versus what an agent would receive at test time, or any difficulty metric used to confirm that retained tasks remain long-horizon for skilled engineers. Without these details the assertion that the 1,865 problems are representative of enterprise tasks cannot be evaluated."}],"tokens_in":1410,"tokens_out":627,"duration_ms":31497,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper mostly enlarges SWE-Bench rather than rethinking it. It pulls 1,865 tasks from 41 repositories, emphasizes multi-file patches and longer time horizons, and adds a three-way split into public, held-out, and commercial sets with some results released on the proprietary side. That split is the clearest practical addition, since it lets people test contamination resistance without handing over full enterprise codebases. The failure-mode clustering on agent trajectories is also a reasonable way to organize observations about where current models break. Those pieces give researchers a larger, more varied testbed than the first version and a workable model for handling real data under partnership agreements. The curation effort itself looks solid on paper, with human verification and context added to make tasks doable. The soft spot sits right in that verification step. The abstract states tasks were augmented with enough context to ensure resolvability, which implies a filter that drops problems even skilled engineers might struggle with in their natural state. Without reported rejection rates, inter-rater numbers, or a clear description of how context was chosen versus what an agent would actually see, it is difficult to judge whether the final set truly reflects enterprise complexity or just the subset that survives the filter. The claim that it more faithfully captures real development therefore rests on thinner ground than the scale alone suggests. This work is aimed at groups building and evaluating autonomous coding agents who already use SWE-Bench and want harder, more diverse tasks with some contamination controls. It is worth sending to peer review because the repository count, access model, and failure analysis are concrete and falsifiable contributions that referees can check against the data. The verification details will need tightening, but that is normal for a benchmark paper and does not make the core extension unusable.","headline":"SWE-Bench Pro is a scaled-up version of the original benchmark with more repositories and a practical access split, but the human verification step likely selects for solvable tasks and weakens the claim of better real-world fidelity.","tokens_in":2462,"tokens_out":442,"would_cite":true,"duration_ms":30288,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"SWE-Bench Pro introduces 1,865 human-verified problems from 41 repositories to test AI agents on realistic long-horizon software tasks.","keywords":["software engineering benchmark","AI agents","long-horizon tasks","enterprise software","multi-file patches","contamination resistance","agent evaluation"],"falsifier":"If expert software engineers fail to solve most of the tasks even with the supplied context, or if AI agents achieve comparable success rates on this benchmark and the original simpler one, the claim of greater realism and difficulty would not hold.","tokens_in":2666,"feed_emoji":"🛠️","tokens_out":555,"duration_ms":45881,"temperature":0.7,"pith_summary":"The paper establishes a new benchmark that addresses limitations in existing tests by drawing problems from a wider range of actively maintained repositories, including proprietary ones under partnership agreements. Tasks require substantial modifications across multiple files and can take professional engineers hours or days to complete, with all problems human-verified and supplied with enough context to remain solvable. Partitioning into public, held-out, and commercial sets reduces the risk that training data overlaps with evaluation items. This setup matters because it creates a clearer signal for whether AI agents are approaching the ability to manage complex, enterprise-grade development work without constant human oversight. The authors also examine failure patterns in agent trajectories to identify recurring error types.","feed_headline":"New benchmark tests AI agents on 1,865 enterprise software tasks","feed_subtitle":"SWE-Bench Pro draws long-horizon problems from 41 repositories, including proprietary ones, to measure progress toward autonomous code work.","key_machinery":"The SWE-Bench Pro benchmark structure, defined by its division into public, held-out, and commercial repository sets together with human-verified long-horizon tasks that require multi-file modifications.","core_discovery":"We introduce SWE-Bench Pro, a benchmark of 1,865 problems sourced from 41 repositories spanning business applications, B2B services, and developer tools. The problems feature long-horizon tasks that often involve patches across multiple files and substantial code changes, with all tasks human-verified and augmented with sufficient context to ensure they are solvable by skilled engineers. The benchmark is divided into a public set from 11 repositories, a held-out set from 12 repositories, and a commercial set from 18 proprietary repositories under formal agreements; results are released on the commercial set while protecting access to the problems themselves.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["SWE-Bench Pro tests AI agents on 1865 long-horizon tasks","1865 long-horizon tasks challenge AI in SWE-Bench Pro","AI agents benchmarked on tasks from 41 software repositories","Complex enterprise tasks from proprietary repos test AI agents"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 1,865 selected problems, drawn from the 41 repositories and augmented with human-provided context, accurately represent long-horizon enterprise software tasks without selection bias.","fun_headline_variants_meta":{"raw":{"variants":["SWE-Bench Pro tests AI agents on 1865 long-horizon tasks","1865 long-horizon tasks challenge AI in SWE-Bench Pro","AI agents benchmarked on tasks from 41 software repositories","Complex enterprise tasks from proprietary repos test AI agents"]},"model":"grok-4.3","cost_usd":0.012588,"raw_usage":{"total_tokens":5438,"prompt_tokens":752,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":125878000,"prompt_tokens_details":{"text_tokens":752,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4616,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":752,"tokens_out":70,"duration_ms":39865,"temperature":1.0,"reasoning_tokens":4616,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-12T13:43:45.457937+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If expert software engineers fail to solve most of the tasks even with the supplied context, or if AI agents achieve comparable success rates on this benchmark and the original simpler one, the claim of greater realism and difficulty would not hold.","supporting_citations":[],"review_version":1}