Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

SWE-rebench V2 claims that fully automated mining — with LLM judges calibrated on human-verified SWE-bench labels — can produce 32,079 executable coding tasks across 20 languages, plus 120,000+ more, without per-instance human verification.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:10 UTC pith:YKFWNTUW

load-bearing objection A large, genuinely useful multilingual SWE training corpus, but the LLM-filter validation claim is internally inconsistent with the reported funnel and needs a re-audit. the 3 major comments →

arxiv 2602.23866 v2 pith:YKFWNTUW submitted 2026-02-27 cs.SE cs.CL

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

Ibragim Badertdinov , Maksim Nekrashevich , Anton Shevtsov , Alexander Golubev This is my paper
classification cs.SE cs.CL
keywords software engineering agentsreinforcement learning training datamultilingual task collectionautomated dataset constructionLLM-based filteringfail-to-pass test oraclesreproducible Docker environmentsissue clarity filtering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to remove the main bottleneck in reinforcement-learning training of software-engineering agents: the scarcity of executable, test-verified task environments. Its five-stage, language-agnostic funnel mines real GitHub pull-request histories, synthesizes per-repository install-and-test procedures with an interactive setup agent, validates every task by running the test suite before and after the fix, filters underspecified issues with a three-judge LLM ensemble calibrated on human-verified SWE-bench labels, and attaches per-instance diagnostic metadata — so no per-instance human verification is needed. The claimed output is 32,079 containerized tasks across 20 languages and 3,617 repositories, plus 120,000+ PR-derived tasks with problem statements generated from pull-request descriptions. If the automated environment construction and LLM-judged filtering hold up, RL training of coding agents moves from a Python-dominated, human-curated regime to a multilingual, machine-built regime at roughly an order of magnitude larger scale.

Core claim

The central claim is that executable, test-verified software-engineering tasks can be manufactured automatically at a scale and language breadth that previously required human curation. The construction funnel mines real GitHub issue–pull-request pairs, synthesizes repository-specific installation and test procedures once per repository using an interactive setup agent, validates each candidate by running the full test suite before and after applying the fix — keeping only instances with at least one fail-to-pass test — and filters underspecified issues through an ensemble of three LLM judges whose behavior was calibrated against human-verified SWE-bench annotations. The authors report that

What carries the argument

Three components carry the argument. The interactive setup agent — a mini-SWE-agent scaffold driven by Qwen3-Coder-480B-A35B-Instruct — infers, once per repository, the install commands and a verbose per-test test command, packaged as install_config.json and reused for every task mined from that repository; its success rate (pass@1 around 26%, rising toward 63% with ten attempts and longer contexts) sets the ceiling on task yield. Execution-based validation runs the full test suite twice — first with only the test patch applied, then with the solution patch also applied — and keeps an instance only if at least one test flips from fail to pass; this fail-to-pass oracle is what makes a task us

Load-bearing premise

The load-bearing premise is that the three-LLM vote on whether an issue text is clear enough to train on, calibrated on 1,699 human-annotated, mostly Python examples, behaves identically across all 20 languages and is deployed in one of the configurations that were actually tested — yet the funnel keeps 78% of tasks while the most permissive tested configuration keeps at most about 40% of well-specified tasks, so the identity of the deployed filter is the hinge the whole clai

What would settle it

Re-run stage 4 exactly as described in §3.4: apply the three-judge consensus with the Verified-E prompt to the 41,349 surviving tasks of Table 1 and count survivors. The reported recall of 0.06–0.10 predicts roughly 2,500–4,100 keepers, not the 32,079 shown in the table; a much higher count would show a different, more permissive configuration was deployed, meaning the released set contains tasks the validated rubric would call underspecified. A complementary check is to have human annotators apply the SWE-bench Verified well-specified rubric to a random sample of tasks per language and compar

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training substrate: 32,079 containerized tasks across 20 languages with pre-built Docker images lets RL agents train and evaluate on Go, Rust, Scala, and long-tail ecosystems rather than Python alone.
  • Marginal-cost scaling: since installation and test procedures are synthesized once per repository and reused, each further merged PR from an installed repository adds a new task at low cost, and the funnel's yield numbers let practitioners predict that cost.
  • Curriculum design: per-instance diagnostic tags (clean A-labeled tasks versus B1–B7 confounders) allow filtering into a clean subset for supervised warm-up and a noisy subset for robustness training, without re-running any evaluation.
  • Decoupling from issue linkage: the 120,000+ PR-derived tasks with generated problem statements bypass the issue-linkage bottleneck that cuts the candidate pool from 29.5M PRs to 805k issue-linked, test-bearing PRs.
  • Reproducibility contract: pre-built images and synthesized log parsers standardize test execution so the same task runs identically in training and evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The most direct validation this corpus will get is behavioral, not archival: train an RL agent from scratch on the released images and measure held-out performance. The paper's limitations section explicitly says no such training ablation is included, so the training-substrate claim is an invitation rather than a demonstrated gain.
  • Arithmetic check (mine, not the paper's): the funnel drops from 41,349 to 32,079 tasks after clarity filtering — a 78% keep rate — yet the most permissive filter configuration the paper reports keeps at most about 40% of well-specified tasks. Unless stage 4 bundles other exclusions or a different, unreported configuration was deployed, the released set is larger than the validated filter can justi
  • Cross-language transfer of the clarity filter is an untested assumption in the paper: calibration used a Python-dominated, human-annotated benchmark, while deployment covers 20 languages with different issue-writing conventions. A human re-annotation of a few hundred non-Python issues would show whether the language-agnostic claim is carried by the judges or just by the setup stage.
  • The pipeline's decoupling of task construction from any particular data source suggests a natural extension the authors do not pursue: the same setup agent, dual-pass validation, and judge ensemble could be pointed at internal monorepo histories or non-GitHub forges, turning each new repository's merged PRs into training tasks under the same contract.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SWE-rebench V2, an automated pipeline for harvesting executable, containerized software-engineering tasks from GitHub pull-request histories. The pipeline has five stages: mining and filtering PRs, synthesizing repository-level setup scripts via an interactive agent, dual-pass execution-based validation, LLM-judge filtering for issue clarity, and metadata enrichment. The authors release 32,079 issue-linked tasks across 20 languages and 3,617 repositories, plus more than 120,000 PR-derived tasks with generated problem statements. They also report ablations of setup synthesis and issue-clarity filtering, and a diagnostic study of 300 tasks across seven models, producing instance-level labels for confounders such as test-suite coupling and implicit naming.

Significance. If the pipeline works as described, this is a substantial and timely resource: a multilingual, executable training substrate for RL-based SWE agents, with reproducible Docker images, automated setup, and instance-level diagnostics. The setup-synthesis ablation with pass@k against manual reference setups, the dual-pass full-suite validation, and the public release of code and datasets are concrete strengths. However, the central claim that the released tasks were filtered by an LLM-judge ensemble validated on human-verified SWE-bench annotations is not supported by the reported numbers, so the quality assurance of the released dataset is not yet established.

major comments (3)
  1. [Table 1, §3.4, §4.2] The reported stage-4 keep rate is arithmetically incompatible with every calibrated filter configuration. Table 1 shows 32,079/41,349 = 77.6% of F2P tasks surviving 'Issue text based filtering'. For a classifier with precision P and recall R, the keep fraction is at most R/P (when all pre-filter tasks are well-specified). Using the reported operating points — Verified-E (P=0.83, R=0.10), MIXED(CONSENSUS) (P=0.88, R=0.06), and the most permissive MIXED-AVG (P=0.73, R=0.31) — the maximum keep rates are about 12%, 7%, and 42%, respectively. Even the most permissive point cannot yield 77.6%. Either the deployed filter differs from every reported configuration, or stage 4 bundles a substantial undocumented exclusion (e.g., sensitive-information removal). The paper must report the actual deployed filter, the number of tasks removed by each sub-step, and a calibration measurement on the deploye
  2. [§3.4 vs §4.2] There is no description of the exact production configuration. §3.4 states that an instance is retained only when 'all three judges rate the specification as adequate' (a consensus rule). §4.2 states that 'we use the Verified-E configuration throughout the pipeline'. But Tables 3–5 do not include a row for Verified-E combined with a three-judge consensus (or any ensemble). Verified-E is a prompt variant, not an ensembling strategy, and the tables vary prompts, models, and ensembles but never the exact combination used in production. Consequently, the headline claim that the pipeline 'filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations' is not backed by a measured operating point for the actual filter.
  3. [§4.2, Dataset languages] The clarity filter is validated exclusively on the 1,699 SWE-bench Verified instances, which are Python-based. The released dataset spans 20 languages, including long-tail ecosystems with different issue-writing conventions and code representations. The paper provides no evidence that the judge scores transfer across languages, nor does it report per-language keep rates. At minimum, the authors should report the distribution of filter outcomes by language and, ideally, a manual or held-out precision sample for a few non-Python languages. Without this, the 'language-agnostic' filtering claim is unsubstantiated.
minor comments (4)
  1. [Appendix A.3.2] The setup-synthesis prompt is titled 'PROMPT FOR SETUP SYNTHESIS' but begins 'You are a non-interactive build-and-test agent', which seems inconsistent with §3.2's description of an interactive agent. Please clarify whether this is the prompt for the interactive agent or a separate non-interactive baseline.
  2. [§3.1] The test-file identification regex (?i)(test(?:ing|s)?|e2e) is quite broad. The paper does not report false-positive/negative rates for test-file identification, even though this is upstream of the F2P oracle extraction. A brief validation or discussion would help.
  3. [Table 5] The first column header is 'SETUP', which appears to be a typo for the ensemble configuration (e.g., 'ENSEMBLE'). Please correct.
  4. [§3.6] For the 120k+ PR-derived tasks, the paper describes post-processing to prevent solution leakage but does not quantify how many generated problem statements were removed or flagged. Since this corpus is released as a training resource, a small leakage audit would strengthen the contribution.

Circularity Check

2 steps flagged

Deployed issue-clarity filter sees the gold patch and is arithmetically incompatible with the reported funnel.

specific steps
  1. self definitional [Section 4.2 / Appendix A.3.8]
    "Second, along with issue description, we provide the model with patch and test patch. We refer to this as Verified-E. ... Since precision is particularly important for our filtering stage, we use the Verified-E configuration throughout the pipeline."

    The target property is whether the issue text alone is well-specified for an engineer who cannot see the solution, but the deployed judge is given the gold patch and test patch. An issue that is vague in isolation can be rated well-specified because the patch and tests reveal the intended behavior. Thus the filter's prediction is not an independent property of the input the agent will see; it is derived from the very solution the task is supposed to elicit. The validation against human issue-clarity labels therefore measures clarity-with-solution, not standalone clarity.

  2. other [Table 1 / Section 3.7 / Section 4.2]
    "Successful tasks w/ F2P 41,349 4,006 / Issue text based filtering 32,079 3,617"

    The reported keep rate is 32,079/41,349 = 77.6%. For Verified-E (P=0.83, R=0.10), the keep fraction is at most max(R, 1-P) = 0.17; for consensus (P=0.88, R=0.06) it is at most 0.12; even the most permissive reported point (MIXED-AVG P=0.73, R=0.31) caps at 0.31. None can remove only 22% of tasks. Hence the released 32,079 tasks cannot be the output of the validated filter as described; the 'validated filtering' claim is asserted without a filtering configuration that can reproduce the funnel.

full rationale

The setup-synthesis ablation is externally grounded (manual install instructions converted into the pipeline format), and the SWE-bench Verified human labels are an independent anchor, so the paper is not wholly circular. However, the filtering stage is load-bearing for the claim that the released tasks are sound, and it has two circular/unsupported features. First, the deployed Verified-E judge receives the gold patch and test patch, so its 'well-specified' rating is conditioned on the target solution; this makes the filter's prediction reduce, by construction, to the availability and clarity of the patch rather than the standalone clarity of the issue. Second, the funnel arithmetic shows that no calibrated configuration can keep 77.6% of tasks, so the production filter is either undocumented or bundled with unreported exclusions; in either case the validation evidence does not cover the released data. The paper also explicitly acknowledges in Section 5 that it does not include ablation studies on agent training with the filtered subsets, which is a missing support for the metadata's utility rather than a circular step. No load-bearing self-citation chain or imported uniqueness theorem is present; the central issue is the self-referential filtering setup and the disconnected validation.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The pipeline rests on a chain of domain assumptions rather than mathematical derivations: GitHub linkage quality, F2P oracle soundness under full-suite execution, transfer of a Python-calibrated LLM clarity filter to 20 languages, correctness of external reference setups, representativeness of a 5-language diagnostic sample, and single-container reproducibility. No free parameter is fitted to data in the scientific sense, but several hand-set thresholds (star/issue cutoffs, retry budget of 1, consensus rule, filename regex) materially shape the corpus. No new physical or conceptual entities are postulated; the B1-B7 diagnostic labels are a new taxonomy whose accuracy is unvalidated.

free parameters (4)
  • Repository filtering thresholds = 25 stars / 15 closed issues (high-resource); 10 stars / 1 closed issue (long-tail)
    Chosen by hand in §3.1 to trade setup compute against task yield; directly shapes the repository pool and therefore the final task distribution, and is the first major selection bias in the funnel.
  • Setup retry budget = 1 setup run per repository in the main pipeline
    §3.7 and §4.1: single run chosen as a cost-yield trade-off; with best pass@1 = 25.8%, this caps repository-level yield near 20% and biases the corpus toward easy-to-set-up repositories, as the paper acknowledges.
  • Issue-clarity keep rule = 3/3 judges must rate the specification adequate (consensus); or Verified-E prompt
    §3.4/§4.2: the paper states both rules in different places. Neither reported operating point (consensus Prec 0.88/Rec 0.06; Verified-E Prec 0.83/Rec 0.10) is consistent with the 77.6% keep rate in Table 1; see red flag.
  • Test-file identification regex = (?i)(test(?:ing|s)?|e2e)
    §3.1: a filename heuristic splits the PR diff into solution patch and test patch. Misclassification risk is partially mitigated by the B7 inline-test metadata tag, but the oracle extraction depends on this regex at scale.
axioms (6)
  • domain assumption GitHub Archive PR/issue linkage is complete and correct for identifying resolution PRs.
    §3.1: candidate tasks require joining PRs to the issues they resolve via titles and descriptions; wrong or missing linkage silently drops or corrupts tasks, and no audit of linkage accuracy is reported.
  • domain assumption A fail-to-pass test obtained by full-suite dual-pass execution is a sound oracle for the issue.
    §3.3: inherited from SWE-bench. Because the full suite is run, the oracle includes tests unrelated to the issue (acknowledged as B1 test-suite coupling), so oracle soundness is probabilistic, not guaranteed per instance.
  • domain assumption LLM-judge agreement calibrated on SWE-bench Verified (Python-dominated, 1,699 instances) transfers across 20 languages and repository distributions.
    §4.2: the only human-verified calibration set is SWE-bench Verified; no per-language precision/recall is reported, yet the filter is applied corpus-wide.
  • domain assumption The manually written setup instructions from SWE-bench, SWE-bench-multilingual, and Multi-SWE-Bench used as ground truth in the 103-repo ablation are correct for the checked-out commits.
    §4.1: setup success is measured as an exact F2P-set match to these references; if any reference is stale or wrong for the pinned commit, the ablation misstates pipeline quality.
  • domain assumption Seven frontier models' trajectories on 300 tasks in 5 languages are representative of pathologies across all 20 languages.
    §4.3: the B1-B7 taxonomy and the metadata meta-prompt derive from Python/JS/Go/Rust/Scala only; long-tail ecosystems are not sampled, yet the tags are applied corpus-wide.
  • domain assumption Single-container Dockerization with offline execution reproduces project behavior faithfully.
    §3.2-3.3, §5: reproducibility rests on network-free execution in one container; projects requiring external databases or multi-service infrastructure are excluded by design, a limitation the paper states.
invented entities (1)
  • B1-B7 diagnostic metadata schema independent evidence
    purpose: Per-instance tags shipped with all tasks (test-suite coupling, implicit naming, external dependency, ambiguous spec, patch artifacts, implicit knowledge, inline test) intended to enable curriculum filtering and stratified training.
    The tags are released with the dataset on HuggingFace, so a third party can audit them against tasks and trajectories, which is a falsifiable handle. However, the paper reports no accuracy, agreement, or precision/recall study for the gpt-oss-120b-generated tags, so their reliability is unmeasured.

pith-pipeline@v1.3.0-alltime-deepseek · 22482 in / 21991 out tokens · 207401 ms · 2026-08-02T20:10:06.523275+00:00 · methodology

0 comments
read the original abstract

Software engineering agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of benchmarks have emerged, datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource language ecosystems. We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE agents across diverse languages and repositories.

Figures

Figures reproduced from arXiv: 2602.23866 by Alexander Golubev, Anton Shevtsov, Ibragim Badertdinov, Maksim Nekrashevich.

Figure 1
Figure 1. Figure 1: Temporal and language distributions in the benchmark corpus. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Issue-type mix and patch-size distribution. C. Model Performance Results Across Languages C.1. Per-language Performance with Confidence Intervals (a) Go: pass@1, SEM, 95% CI, and pass@3 (60 tasks). Model pass@1 SEM CI95% pass@3 GLM-4.7 17.22% 4.38% [8.65%, 25.80%] 23.33% MiniMax-M2.1 15.56% 4.15% [7.42%, 23.69%] 23.33% Opus-4.5 15.00% 3.92% [7.33%, 22.67%] 25.00% DeepSeek-V3.2 12.22% 3.88% [4.62%, 19.82%] … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Dockerless: Environment-Free Program Verifier for Coding Agents

    cs.SE 2026-06 unverdicted novelty 7.0

    Dockerless uses agentic repository exploration to verify patches without execution, enabling SFT and RL training of coding agents that reach 62.0/50.0/35.2% resolve rates on SWE-bench Verified/Multilingual/Pro while m...

  2. KAT-Coder-V2.5 Technical Report

    cs.SE 2026-07 conditional novelty 6.0

    An end-to-end agentic post-training stack (AutoBuilder, KwaiClawEnv, harness-randomized asymmetric PPO, multi-teacher distillation) yields a coding agent that tops PinchBench and ranks second on SWE-Bench Pro.

  3. A Framework for Evaluating Agentic Skills at Scale

    cs.SE 2026-06 unverdicted novelty 6.0

    The authors developed an evaluation framework that generates 1000 tasks from 500 real-world agent skills, applies instruction-following and goal-completion rubrics, and benchmarks 19 proprietary and open-source model ...

  4. Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

    cs.SE 2026-06 accept novelty 6.0

    Coding benchmarks conflate the model with the harness, environment, and verifier into a single end-to-end score, which is misaligned with agentic software engineering.

  5. Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

    cs.SE 2026-06 unverdicted novelty 4.0

    Coding benchmarks misalign with agentic software engineering because they conflate model and harness, grade against single references, and provide no component-level iteration signals.

  6. Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development

    cs.SE 2026-05 unverdicted novelty 4.0

    Agentic Agile-V uses Agile-V as backbone and a Specify-Constrain-Orchestrate-Prove-Evolve-Verify loop to convert AI agent conversations into traceable engineering artifacts with acceptance evidence.

Reference graph

Works this paper leans on

25 extracted references · cited by 5 Pith papers

  1. [1]

    install project dependencies,

  2. [2]

    build/compile the project (if applicable),

  3. [3]

    Scope and constraints - This is a *base* image used across many repositories, so avoid repository- specific assumptions

    run the project’s test suite. Scope and constraints - This is a *base* image used across many repositories, so avoid repository- specific assumptions. - Prefer official or widely used upstream base images for <LANG_NAME> (e.g., official images on Docker Hub) rather than installing the toolchain from scratch. - Target linux/amd64 unless <LANG_NAME> strongl...

  4. [4]

    Environment variables that place language/package-manager caches under / workspace (so they are writable even for non-root): - Set XDG_CACHE_HOME, XDG_CONFIG_HOME, XDG_DATA_HOME when applicable - Also set language-specific cache/home vars (e.g., for package managers)

  5. [5]

    A suitable FROM line (prefer a stable, commonly used toolchain version; choose an LTS/stable version when possible)

  6. [6]

    Installation of common build and debugging utilities: - git, curl/wget, unzip/zip, ca-certificates - build-essential / compiler toolchain, pkg-config, cmake (if relevant)

  7. [7]

    Installation of native libraries commonly required by packages in this ecosystem (keep this conservative)

  8. [9]

    Create a non-root user (but keep USER optional, as a commented line)

  9. [10]

    Output requirements - Return only the Dockerfile contents, and nothing else (no explanation, no markdown)

    WORKDIR /workspace. Output requirements - Return only the Dockerfile contents, and nothing else (no explanation, no markdown). - Keep the Dockerfile reasonably short and readable. - Clean apt caches (rm -rf /var/lib/apt/lists/ *) to reduce image size. Example style (Elixir) [The following is only an example of the expected style and structure; do not copy...

  10. [11]

    Discover how to install/build the project and run its tests

  11. [12]

    Execute the minimal steps to make tests runnable

  12. [13]

    install": ordered list of the **actual** successful shell commands required for this repo. -

    Generate a correct ‘install_config.json‘ with exactly: - "install": ordered list of the **actual** successful shell commands required for this repo. - "test_cmd": a list of shell commands to run the full test suite ( **per-test verbose**, no ANSI color if possible) that is **general** (no file-, directory -, or target-specific selectors). Should also outp...

  13. [14]

    github/workflows/*‘, ‘.travis.yml‘, etc.)-but stop as soon as you find enough information to install the project or run the tests

    Discover: read top-level docs (‘README *‘, ‘docs/ *‘, ‘CONTRIBUTING *‘), automation files (‘Makefile‘, ‘package.json‘, ‘pyproject.toml‘, ‘go.mod‘, ‘ Cargo.toml‘, ‘pom.xml‘, ‘build.gradle *‘, ‘tox.ini‘, etc.), and CI configs (‘. github/workflows/*‘, ‘.travis.yml‘, etc.)-but stop as soon as you find enough information to install the project or run the tests

  14. [15]

    Plan minimal install: **identify the project root first ** (subdir if monorepo); then project deps/build via the repo’s package manager ( **lockfile-aware commands** preferred)

  15. [16]

    **Do not ** include commands that failed or were not needed

    Execute iteratively: after each **successful and necessary ** command, add it ( in order) to ‘"install"‘. **Do not ** include commands that failed or were not needed

  16. [17]

    test_cmd

    Choose a single, **per-test verbose ** and **no-color** test runner for ‘" test_cmd"‘, and it must be **general** (no file-/target-specific selection). If ‘make test‘ is presented **extract the underlying test command and add per- test verbosity **. Should use JunitXML output if possible

  17. [18]

    install": [ ... ],

    Produce the install_config.json file if you have a working ‘test_cmd‘ (even if some tests fail): ‘‘‘bash cat > install_config.json <<’JSON’ { "install": [ ... ], "test_cmd": [ ... ] } JSON cat install_config.json ‘‘‘ You should create install_config.json file only if you successfully ran tests or if you have a working ‘test_cmd‘ that runs the full suite (...

  18. [19]

    Extract intent from ISSUE_TEXT: expected behavior, observed behavior, reproduction, acceptance criteria

  19. [20]

    Check test alignment: Do assertions verify stated requirements or introduce new ones?

  20. [21]

    Scan for B-category signals: - B1: Tests span unrelated modules - B2: String assertions for names not in issue - B3: URLs to external docs/specs - B4: Vague or missing acceptance criteria - B5: Patch modifies unrelated files that are then required by tests - B6: Tests assert specific approaches not mentioned - B7: Inline tests appear in the non-test patch

  21. [22]

    critical_bug

    Classify: A if clean, B[1-7] for primary issue ## Rubric: PR Category Choose one or more categories that best describe the PR. Allowed categories: 18 SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale ‘‘‘ { "critical_bug", "major_bug", "minor_bug", "regression_bug", "edge_case_bug", " performance_bug", "security_bug", "integration_feat", "core...

  22. [23]

    Identify new functions or classes

  23. [24]

    Identify functions or classes whose signatures have changed significantly ( arguments changed, not just indentation)

  24. [25]

    PROMPT FORFILTERING BYISSUECLARITY We have a dataset of GitHub issues from various open-source Python repositories

    Use the test patch to see which of them, if any, are explicitly called in the new test code Ignore as interfaces: - Purely internal helpers (for example names starting with "_" and not used directly in tests) - Type aliases, enums, simple data or config holders - Constructors or trivial overrides that only support other documented interfaces - Constants, ...

  25. [2025]

    -hosts localhost

    Notion Blog. Mhatre, S., Bajpai, Y ., Gulwani, S., Murphy-Hill, E., and Soares, G. Swe-sharp-bench: A reproducible benchmark for c# software engineering tasks, 2025. URL https: //arxiv.org/abs/2511.02352. Oliva, G. A., Rajbahadur, G. K., Bhatia, A., Zhang, H., Chen, Y ., Chen, Z., Leung, A., Lin, D., Chen, B., and Hassan, A. E. Spice: An automated swe-ben...

    arXiv 2025