{"id":"b45acfb8-d899-4fbc-a45a-d835c26f5fa6","arxiv_id":"2602.23866","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 32,079-task, 20-language corpus of executable GitHub issue-fix environments with Docker images and test oracles, plus 120k+ PR-derived tasks, targeting training of software-engineering agents.","lead":"SWE-rebench V2 is a pipeline and dataset of 32,079 executable, containerized software-repair tasks mined from 3,617 open-source repositories across 20 programming languages, plus 120,000+ additional tasks with generated problem statements. It gives AI coding-agent trainers a multilingual pool of real bug-fix environments, a resource that has so far been concentrated in Python.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported stage-4 keep rate (77.6%) is arithmetically incompatible with every calibrated issue-clarity filter (max ≈ 12–40%); the deployed filter is undocumented, so the 'validated filtering' claim does not cover the released data.","rationale":"Good-faith reading: the paper contributes a genuinely large multilingual executable dataset with pre-built images, a dual-pass F2P oracle protocol, setup-synthesis ablations, and a candid limitations section. The strongest claim is that automated quality filtering is validated against human-verified SWE-bench annotations, so no per-instance human verification is needed. The reader's weakest assumption targets exactly that claim, and my check confirms it is the most load-bearing issue. The arithmetic is internal, not a matter of external consensus: using precision/recall from Tables 3 and 5, the maximum keep rate under any calibrated configuration is below 40%, far from the 77.6% shown in Table 1. This matters because the abstract and §3.4 promise validated filtering, and RL training depends on test/spec alignment. The inconsistency does not disprove the existence or usefulness of the corpus—the setup ablations, dual-pass validation, and diagnostic study are real evidence—so a conditional verdict remains appropriate. What is needed is either a stage-4 count decomposition or a re-calibration of the production filter on the 1,699 human annotations. The authors' own acknowledgment that training ablations are missing does not undermine the current claim as directly as the filter-deployment gap does, and the unevaluated 120k PR corpus is a secondary concern. No ad hominem is implied; the issue is documentation and validation, not intent.","tokens_in":23103,"tokens_out":8976,"duration_ms":87551,"concrete_test":"Run the exact production filter on a random sample of the 41,349 pre-stage-4 tasks (or, if those are not released, on the public metadata for the 32,079 released tasks plus the 9,270 dropped tasks): apply the §4.2 VERIFIED-E prompt with the three judges (gpt-oss-120b, GLM-4.7, DeepSeek-V3.2) and the stated consensus rule. Record the accepted fraction. If it is ≈0.05–0.40 rather than 0.776, Table 1's stage 4 cannot be produced by the validated filter, confirming a deployment gap. If it is ≈0.776, the production prompt, aggregation, or threshold differs from every reported ablation, and that production configuration must be separately calibrated on the 1,699 SWE-bench Verified annotations before the 'validated' claim is used. Additionally, report the number of instances removed by sensitive-information exclusion separately so stage 4 is decomposable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central artifact claim is that the released 32,079 tasks were filtered for soundness by an ensemble of LLM judges validated on human-verified SWE-bench annotations (§3.4, Abstract). Table 1 reports that stage 4 ('Issue text based filtering') keeps 32,079/41,349 = 77.6% of tasks. This is impossible with any configuration calibrated in §4.2. For a classifier with precision P, recall R, and pre-filter well-specified base rate π, the keep fraction is K = πR/P (since FP = (1−P)/P · TP), assuming the classifier operates at a valid threshold. Using the reported operating points: MIXED(CONSENSUS) P=0.88,R=0.06 gives K = π·0.068; VERIFIED-E P=0.83,R=0.10 gives K = π·0.120; the most permissive reported point, MIXED-AVG P=0.73,R=0.31, gives K = π·0.425. Even with a very optimistic π=0.9, the keep rate is about 38%; at π=0.95 it is still only about 40%. The paper states 'we use the Verified-E configuration throughout the pipeline' (§4.2), and §3.4 says all three judges must rate the issue adequate; the consensus numbers in Table 5 would keep roughly 5–6% of tasks. To remove only 22% of tasks, the deployed filter would need recall far above any reported value (≤0.31). Stage 4 as defined also bundles 'excluding instances with potentially sensitive information' (§3.4), but no count is reported for that sub-step. Therefore either the production filter differs from every validated configuration, or the funnel mixes in a large undocumented exclusion. Both possibilities mean the 'validated against human annotations' claim does not describe the released pipeline, and downstream users cannot infer the effective quality of the 32,079 tasks without re-auditing them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SWE-rebench V2, an automated pipeline for harvesting executable, containerized software-engineering tasks from GitHub pull-request histories. The pipeline has five stages: mining and filtering PRs, synthesizing repository-level setup scripts via an interactive agent, dual-pass execution-based validation, LLM-judge filtering for issue clarity, and metadata enrichment. The authors release 32,079 issue-linked tasks across 20 languages and 3,617 repositories, plus more than 120,000 PR-derived tasks with generated problem statements. They also report ablations of setup synthesis and issue-clarity filtering, and a diagnostic study of 300 tasks across seven models, producing instance-level labels for confounders such as test-suite coupling and implicit naming.","tokens_in":1444,"tokens_out":1568,"duration_ms":64916,"significance":"If the pipeline works as described, this is a substantial and timely resource: a multilingual, executable training substrate for RL-based SWE agents, with reproducible Docker images, automated setup, and instance-level diagnostics. The setup-synthesis ablation with pass@k against manual reference setups, the dual-pass full-suite validation, and the public release of code and datasets are concrete strengths. However, the central claim that the released tasks were filtered by an LLM-judge ensemble validated on human-verified SWE-bench annotations is not supported by the reported numbers, so the quality assurance of the released dataset is not yet established.","major_comments":[{"comment":"The reported stage-4 keep rate is arithmetically incompatible with every calibrated filter configuration. Table 1 shows 32,079/41,349 = 77.6% of F2P tasks surviving 'Issue text based filtering'. For a classifier with precision P and recall R, the keep fraction is at most R/P (when all pre-filter tasks are well-specified). Using the reported operating points — Verified-E (P=0.83, R=0.10), MIXED(CONSENSUS) (P=0.88, R=0.06), and the most permissive MIXED-AVG (P=0.73, R=0.31) — the maximum keep rates are about 12%, 7%, and 42%, respectively. Even the most permissive point cannot yield 77.6%. Either the deployed filter differs from every reported configuration, or stage 4 bundles a substantial undocumented exclusion (e.g., sensitive-information removal). The paper must report the actual deployed filter, the number of tasks removed by each sub-step, and a calibration measurement on the deploye","section":"Table 1, §3.4, §4.2"},{"comment":"There is no description of the exact production configuration. §3.4 states that an instance is retained only when 'all three judges rate the specification as adequate' (a consensus rule). §4.2 states that 'we use the Verified-E configuration throughout the pipeline'. But Tables 3–5 do not include a row for Verified-E combined with a three-judge consensus (or any ensemble). Verified-E is a prompt variant, not an ensembling strategy, and the tables vary prompts, models, and ensembles but never the exact combination used in production. Consequently, the headline claim that the pipeline 'filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations' is not backed by a measured operating point for the actual filter.","section":"§3.4 vs §4.2"},{"comment":"The clarity filter is validated exclusively on the 1,699 SWE-bench Verified instances, which are Python-based. The released dataset spans 20 languages, including long-tail ecosystems with different issue-writing conventions and code representations. The paper provides no evidence that the judge scores transfer across languages, nor does it report per-language keep rates. At minimum, the authors should report the distribution of filter outcomes by language and, ideally, a manual or held-out precision sample for a few non-Python languages. Without this, the 'language-agnostic' filtering claim is unsubstantiated.","section":"§4.2, Dataset languages"}],"minor_comments":[{"comment":"The setup-synthesis prompt is titled 'PROMPT FOR SETUP SYNTHESIS' but begins 'You are a non-interactive build-and-test agent', which seems inconsistent with §3.2's description of an interactive agent. Please clarify whether this is the prompt for the interactive agent or a separate non-interactive baseline.","section":"Appendix A.3.2"},{"comment":"The test-file identification regex (?i)(test(?:ing|s)?|e2e) is quite broad. The paper does not report false-positive/negative rates for test-file identification, even though this is upstream of the F2P oracle extraction. A brief validation or discussion would help.","section":"§3.1"},{"comment":"The first column header is 'SETUP', which appears to be a typo for the ensemble configuration (e.g., 'ENSEMBLE'). Please correct.","section":"Table 5"},{"comment":"For the 120k+ PR-derived tasks, the paper describes post-processing to prevent solution leakage but does not quantify how many generated problem statements were removed or flagged. Since this corpus is released as a training resource, a small leakage audit would strengthen the contribution.","section":"§3.6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the filtering arithmetic inconsistency. If the production filter is actually more permissive than the validated configurations, then the released dataset may contain a much higher fraction of underspecified tasks than the 22% implied by Table 1. This is fixable by reporting the actual deployment and recalibrating, but as written the central quality-assurance claim is not credible. The setup ablation and diagnostic study are solid and should be preserved in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this paper as the one releasing 32k executable SWE tasks in 20 languages with pre-built images, plus a 120k PR-derived companion corpus. That resource is real and fills a genuine gap: multilingual RL training pools are scarce. The setup-synthesis ablation (Table 2) is properly grounded in manual references, and the dual-pass full-suite validation is the right protocol. Credit where due.\n\nThe soft spot is the issue-clarity filter, and it is load-bearing. The funnel (Table 1) says stage 4 keeps 77.6% of the 41,349 fail-to-pass tasks. Every calibrated operating point in §4.2 has recall at most 0.31 and precision at most 0.88, so the maximum keep fraction—even assuming every input is well-specified—is about 42%. Three-judge consensus would keep far less. So the deployed filter is either a configuration never reported, or the stage bundles large undocumented exclusions. Either way, the abstract's claim that the pipeline 'filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations' does not describe the released data as reported.\n\nTwo smaller issues: the B1-B7 diagnostic metadata is presented as a headline contribution but has no accuracy measurement, and the 120k PR-expansion's anti-leakage filtering is unevaluated. The authors do acknowledge the missing training ablation and the single-container limitation, which is honest.\n\nNone of this makes the corpus worthless. But a consumer cannot currently know the effective quality of the released tasks without re-auditing a sample. The right move is to get the authors to report the actual deployed filter and validate it; the funnel inconsistency is fixable. This deserves a serious referee—the resource is important enough—but it should come back with the filtering story resolved.","headline":"A large, genuinely useful multilingual SWE training corpus, but the LLM-filter validation claim is internally inconsistent with the reported funnel and needs a re-audit.","tokens_in":24099,"tokens_out":5520,"would_cite":true,"duration_ms":48678,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SWE-rebench V2 claims that fully automated mining — with LLM judges calibrated on human-verified SWE-bench labels — can produce 32,079 executable coding tasks across 20 languages, plus 120,000+ more, without per-instance human verification.","keywords":["software engineering agents","reinforcement learning training data","multilingual task collection","automated dataset construction","LLM-based filtering","fail-to-pass test oracles","reproducible Docker environments","issue clarity filtering"],"falsifier":"Re-run stage 4 exactly as described in §3.4: apply the three-judge consensus with the Verified-E prompt to the 41,349 surviving tasks of Table 1 and count survivors. The reported recall of 0.06–0.10 predicts roughly 2,500–4,100 keepers, not the 32,079 shown in the table; a much higher count would show a different, more permissive configuration was deployed, meaning the released set contains tasks the validated rubric would call underspecified. A complementary check is to have human annotators apply the SWE-bench Verified well-specified rubric to a random sample of tasks per language and compar","tokens_in":22841,"feed_emoji":"🤖","tokens_out":22829,"duration_ms":178829,"temperature":0.7,"pith_summary":"The paper sets out to remove the main bottleneck in reinforcement-learning training of software-engineering agents: the scarcity of executable, test-verified task environments. Its five-stage, language-agnostic funnel mines real GitHub pull-request histories, synthesizes per-repository install-and-test procedures with an interactive setup agent, validates every task by running the test suite before and after the fix, filters underspecified issues with a three-judge LLM ensemble calibrated on human-verified SWE-bench labels, and attaches per-instance diagnostic metadata — so no per-instance human verification is needed. The claimed output is 32,079 containerized tasks across 20 languages and 3,617 repositories, plus 120,000+ PR-derived tasks with problem statements generated from pull-request descriptions. If the automated environment construction and LLM-judged filtering hold up, RL training of coding agents moves from a Python-dominated, human-curated regime to a multilingual, machine-built regime at roughly an order of magnitude larger scale.","feed_headline":"Mines 32,000 coding tasks in 20 languages, no per-task human review","feed_subtitle":"A five-stage funnel turns GitHub pull-request history into RL-ready tasks in 20 languages, plus 120,000 more.","key_machinery":"Three components carry the argument. The interactive setup agent — a mini-SWE-agent scaffold driven by Qwen3-Coder-480B-A35B-Instruct — infers, once per repository, the install commands and a verbose per-test test command, packaged as install_config.json and reused for every task mined from that repository; its success rate (pass@1 around 26%, rising toward 63% with ten attempts and longer contexts) sets the ceiling on task yield. Execution-based validation runs the full test suite twice — first with only the test patch applied, then with the solution patch also applied — and keeps an instance only if at least one test flips from fail to pass; this fail-to-pass oracle is what makes a task us","core_discovery":"The central claim is that executable, test-verified software-engineering tasks can be manufactured automatically at a scale and language breadth that previously required human curation. The construction funnel mines real GitHub issue–pull-request pairs, synthesizes repository-specific installation and test procedures once per repository using an interactive setup agent, validates each candidate by running the full test suite before and after applying the fix — keeping only instances with at least one fail-to-pass test — and filters underspecified issues through an ensemble of three LLM judges whose behavior was calibrated against human-verified SWE-bench annotations. The authors report that","pith_inferences":["The most direct validation this corpus will get is behavioral, not archival: train an RL agent from scratch on the released images and measure held-out performance. The paper's limitations section explicitly says no such training ablation is included, so the training-substrate claim is an invitation rather than a demonstrated gain.","Arithmetic check (mine, not the paper's): the funnel drops from 41,349 to 32,079 tasks after clarity filtering — a 78% keep rate — yet the most permissive filter configuration the paper reports keeps at most about 40% of well-specified tasks. Unless stage 4 bundles other exclusions or a different, unreported configuration was deployed, the released set is larger than the validated filter can justi","Cross-language transfer of the clarity filter is an untested assumption in the paper: calibration used a Python-dominated, human-annotated benchmark, while deployment covers 20 languages with different issue-writing conventions. A human re-annotation of a few hundred non-Python issues would show whether the language-agnostic claim is carried by the judges or just by the setup stage.","The pipeline's decoupling of task construction from any particular data source suggests a natural extension the authors do not pursue: the same setup agent, dual-pass validation, and judge ensemble could be pointed at internal monorepo histories or non-GitHub forges, turning each new repository's merged PRs into training tasks under the same contract."],"forward_implications":["Training substrate: 32,079 containerized tasks across 20 languages with pre-built Docker images lets RL agents train and evaluate on Go, Rust, Scala, and long-tail ecosystems rather than Python alone.","Marginal-cost scaling: since installation and test procedures are synthesized once per repository and reused, each further merged PR from an installed repository adds a new task at low cost, and the funnel's yield numbers let practitioners predict that cost.","Curriculum design: per-instance diagnostic tags (clean A-labeled tasks versus B1–B7 confounders) allow filtering into a clean subset for supervised warm-up and a noisy subset for robustness training, without re-running any evaluation.","Decoupling from issue linkage: the 120,000+ PR-derived tasks with generated problem statements bypass the issue-linkage bottleneck that cuts the candidate pool from 29.5M PRs to 805k issue-linked, test-bearing PRs.","Reproducibility contract: pre-built images and synthesized log parsers standardize test execution so the same task runs identically in training and evaluation."],"fun_headline_variants":["32k coding tasks, 20 languages, built automatically from GitHub","Auto-pipeline yields 32k executable coding tasks across 20 languages","From PRs to RL-ready tasks: 32k instances in 20 languages","No per-task human review: 32k tasks across 20 languages"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the three-LLM vote on whether an issue text is clear enough to train on, calibrated on 1,699 human-annotated, mostly Python examples, behaves identically across all 20 languages and is deployed in one of the configurations that were actually tested — yet the funnel keeps 78% of tasks while the most permissive tested configuration keeps at most about 40% of well-specified tasks, so the identity of the deployed filter is the hinge the whole clai","fun_headline_variants_meta":{"raw":{"variants":["32k coding tasks, 20 languages, built automatically from GitHub","Auto-pipeline yields 32k executable coding tasks across 20 languages","From PRs to RL-ready tasks: 32k instances in 20 languages","No per-task human review: 32k tasks across 20 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001022,"raw_usage":{"total_tokens":4170,"prompt_tokens":788,"completion_tokens":3382,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":3302}},"tokens_in":532,"tokens_out":3382,"duration_ms":21188,"temperature":1.0,"reasoning_tokens":3302,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:10:06.523275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run stage 4 exactly as described in §3.4: apply the three-judge consensus with the Verified-E prompt to the 41,349 surviving tasks of Table 1 and count survivors. The reported recall of 0.06–0.10 predicts roughly 2,500–4,100 keepers, not the 32,079 shown in the table; a much higher count would show a different, more permissive configuration was deployed, meaning the released set contains tasks the validated rubric would call underspecified. A complementary check is to have human annotators apply the SWE-bench Verified well-specified rubric to a random sample of tasks per language and compar","supporting_citations":[],"review_version":1}