Pith. sign in

REVIEW 3 major objections 7 minor 57 references

About one in seven SWE-bench Verified tasks pairs a pull request with the wrong issue scope, and a three-phase agent checker can catch most of those defects at scale.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 02:58 UTC pith:MULX42CS

load-bearing objection Solid empirical SE paper: real misalignment rate on Verified plus a practical multi-agent filter; the stress-test about rubric self-consistency is real but secondary to the construction finding. the 3 major comments →

arxiv 2607.28587 v1 pith:MULX42CS submitted 2026-07-30 cs.SE cs.AI

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Manyi Wang , Junjielong Xu , Pinjia He This is my paper
classification cs.SE cs.AI
keywords SWE-benchPR-Issue misalignmentbenchmark constructionmulti-agent systemsLLM evaluationsoftware engineering agentsdataset quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Benchmarks built like SWE-bench turn GitHub issues into problem statements and the linked pull-request patches into oracles. That pipeline assumes each PR fully and only solves the stated issue. Manual review of all 500 SWE-bench Verified tasks finds that 13.6% break the assumption, across five misalignment families and eleven concrete scenarios, and that never-solved tasks are heavily enriched for these defects. The paper argues the defect is construction-level, not just hard coding, and that it quietly distorts agent rankings. To make future benchmarks cleaner, it introduces PAIChecker: specialized agents first hunt known misalignment patterns in issue and PR text, a coordinator synthesizes their clues (including an open “Others” bucket) and drops unsupported labels, and a final agent checks the surviving claims against the actual code. On held-out SWE-Gym and multilingual suites the system leads strong prompting and agent baselines across four model backbones, reaching roughly 92% and 92% binary accuracy at best.

Core claim

PR–Issue misalignment is a systematic construction defect in SWE-bench-style benchmarks: 13.6% of SWE-bench Verified instances are misaligned under a five-pattern, eleven-scenario taxonomy, misalignment rate falls as more agents solve an instance, and excluding the misaligned set reshuffles a large share of leaderboard ranks. PAIChecker’s three-phase multi-agent design—pattern-specific text agents, cross-agent label synthesis with an open category, then code veto—detects and labels these defects more accurately than monolithic prompting or general agents, up to 92.12% binary accuracy and 84.66% exact match on SWE-Gym and 91.67% / 90.67% on SWE-bench Multilingual.

What carries the argument

PAIChecker’s text-driven, code-validation pipeline: Phase I runs three focused subagents (issue analyzer for incomplete specs; PR-scope analyzer for scope creep and unspecified literals; PR-connection analyzer for defective and follow-up PRs); Phase II’s coordinator only vetoes or assigns “Others” from cross-agent clues; Phase III’s code validator only drops labels that the diff does not support. Predefined labels can be born only in Phase I.

Load-bearing premise

The method assumes that what the PR and issue say in natural language is usually enough to judge alignment, and that code is mainly a check that can reject bad labels—not the first place silent extra changes hide.

What would settle it

Find a large set of real PR–issue pairs where the patch silently implements work never mentioned in PR text, discussions, or review, yet PAIChecker still labels the pair aligned; or show that removing the three specialized Phase I agents (keeping only a single general agent plus code check) matches full PAIChecker accuracy on the same held-out labels.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • New SWE-bench-style train and test sets can be filtered or relabeled automatically before release instead of relying only on human spot checks.
  • Leaderboards recomputed after dropping misaligned instances will shift ranks and pass rates, so published standings on unfiltered sets overstate some agents.
  • Training data built from the same PR–issue scrape will inject noisy supervision unless misaligned pairs are removed or corrected.
  • The open “Others” path plus code veto gives a practical way to catch accidental tracker linkages and other defects outside the fixed taxonomy.
  • Live repository sampling with maintainer confirmation suggests the same checker can flag production PR–issue hygiene problems, not only benchmark archives.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If construction noise explains a large share of “unsolvable” tasks, progress curves on these benchmarks partly measure dataset cleanup rather than pure agent skill.
  • Benchmark builders may need dual oracles: one for the stated issue and one for the full PR, instead of treating a single linked patch as ground truth.
  • The residual silent-diff gap points to a natural next detector that starts from code ownership and review diffs rather than PR prose.
  • Similar misalignment audits likely apply to any issue–patch dataset used for repair, localization, or agent training outside the SWE-bench family.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper argues that SWE-bench-like benchmarks systematically suffer from PR–Issue misalignment: linked PRs often do not exclusively and completely implement the stated issue. Manual open coding of all 500 SWE-bench Verified instances finds 13.6% misaligned, organized into five patterns and eleven scenarios (SC, DP, IS, FP, UL). Misalignment correlates with lower agent resolution rates and shifts 64.1% of leaderboard ranks when excluded. The authors propose PAIChecker, a three-phase multi-agent checker (pattern-specific subagents, coordinator synthesis with an Others path, and code-level veto validation) guided by a text-driven, code-validation principle. On dual-annotated SWE-Gym and SWE-bench Multilingual, PAIChecker outperforms prompting and agent baselines across four LLM backbones, reaching up to 92.12% binary accuracy / 84.66% exact match (SWE-Gym) and 91.67% / 90.67% (Multilingual). Ablations, phase-wise IC>CI label corrections, a live GitHub pilot (16/17 maintainer confirmations), and a public artifact release support the systems claim.

Significance. If the results hold, this is a practically important contribution to LLM-for-SE evaluation infrastructure. SWE-bench-style construction is now the de facto template for issue-resolution benchmarks and training data; documenting construction-level pairing defects, quantifying their prevalence and leaderboard impact, and releasing an automated checker directly improve benchmark hygiene. Strengths include full-corpus open coding of Verified, dual annotation with high Cohen’s κ (0.91/0.86), multi-backbone and multi-baseline evaluation, ablations that isolate Phase I as the detection backbone and II/III as refiners, phase-wise correction statistics, a real-world maintainer confirmation pilot, and public data/code. The work is incremental relative to concurrent quality analyses (OpenAI’s Verified critique, SPICE) but targets a distinct construction axis and provides a usable tool rather than only a diagnosis.

major comments (3)
  1. [§4.1, §5.1, §6, §8] §4.1 and §5.1: Detection metrics are measured against human labels produced under the same taxonomy and text-driven, code-validation principle that define PAIChecker’s agents (IS requires an explicit maintainer request; SC requires PR-side textual claims of extra scope; UL is a three-way text/patch condition; Phase III only vetoes). High BA/EM therefore partly measures fidelity to the authors’ operationalization rather than recovery of an external oracle of “harmful construction defect.” The dual-annotator κ and held-out corpora reduce subjectivity but do not fully separate taxonomy consistency from detection power. Please add a dedicated limitations subsection that (i) states this co-definition explicitly, (ii) reports agreement between PAIChecker and an alternative labeling protocol (e.g., SPICE-style criteria already compared qualitatively in §8, or a code-first pass that ignores PR t
  2. [§2.2.2, Figure 2–3] §2.2.2 RQ2: The misalignment–resolution correlation and leaderboard re-rank are presented as impact evidence, but the paper correctly notes they are not causal. Still, the Finding language (“construction defects add extraneous difficulty and distort standings”) and Abstract framing risk over-reading a confound: hard issues may attract incomplete specs or multi-issue PRs. Strengthen the claim by reporting at least one controlled contrast—e.g., resolution-rate differences within the same repository/difficulty band for aligned vs misaligned instances, or pass-rate change when only high-confidence SC/UL (oracle-mismatch) labels are removed versus DP/FP (gold-patch quality) labels. Without that split, the evaluation-reliability claim remains suggestive rather than load-bearing.
  3. [§3, §6] §3 and §6 (PR description–code misalignment): The central design premise is that PR/issue NL is the primary alignment signal and code is only a veto. The Discussion acknowledges silent unrelated code changes that pass review, calling them “exceedingly rare,” but provides no measurement. Given that Phase I/II cannot surface code-only extras and Phase III cannot add labels, false negatives of this class are structurally invisible to both the system and the current ground truth. Please quantify residual risk on a sample (e.g., manual audit of N aligned instances for code changes unsupported by any PR/issue text) or weaken the claim from “checking PR-Issue misalignment” to “checking text-evidenced PR-Issue misalignment under the stated taxonomy.”
minor comments (7)
  1. [Table 4, §5.1.2] Table 4: Per-label F1 for Others on SWE-Gym remains low (11.1–46.5% depending on backbone) despite being a stated motivation for Phase II. Discuss failure modes of Others more concretely; 16 instances is thin for strong claims about beyond-taxonomy generalization.
  2. [Table 2, Table 4] Table 2 / Multilingual: FP (2) and Others (1) counts are too small for stable per-label comparison; the paper omits some F1s but still reports Macro metrics that include these labels. Consider macro-averaging only over labels with adequate support, or mark Multilingual macro scores as descriptive.
  3. [Title, Abstract] Naming inconsistency: title/abstract use “PAIChecker” / “PaiChecker” / “PAIchecker” interchangeably. Normalize to one spelling throughout.
  4. [Figure 1] Figure 1 panels recycle the same multi-pattern collage with different callouts; readability suffers in print. Consider one clean exemplar per pattern with full issue/PR text in an appendix.
  5. [§4.2, Tables 3–4] §4.2: Claude Code and Codex baselines are restricted to “officially supported models,” breaking the four-backbone grid used for other methods. State this limitation next to Table 3–4 rather than only in the method list.
  6. [Front matter] ACM Reference Format lists a 10.1145 DOI that appears placeholder-like relative to the arXiv stamp; verify before camera-ready.
  7. [§8] §8 comparison with SPICE is useful but the 644-instance overlap analysis would be clearer as a small contingency table (TP/FP-style co-detection) in the main text rather than prose counts only.

Circularity Check

0 steps flagged

No derivation-by-construction circularity; mild residual risk only that eval labels share the authors’ text-first taxonomy operationalization.

full rationale

This is an empirical systems/SE paper, not a first-principles derivation. The load-bearing claims are (i) a manual taxonomy and 13.6% prevalence on SWE-bench Verified and (ii) PAIChecker’s detection/categorization accuracy on held-out SWE-Gym and SWE-bench Multilingual. Taxonomy discovery and evaluation corpora are separated; eval labels come from dual independent annotators (κ=0.91/0.86) following an explicit guideline, not from PAIChecker’s own outputs. Baselines receive the same taxonomy and artifacts, so relative gains are not self-scored. Impact analysis uses external leaderboard resolution data. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no equation that equals its inputs by construction. The only mild concern—shared with most taxonomy papers—is that human ground truth and the agent design both operationalize the same text-driven, code-validation principle, so high BA/EM partly measures fidelity to that rubric; that is construct-validity risk, not circular reduction. Score 1 reflects that residual co-definition without elevating it to a circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

This is an empirical SE/AI-systems paper, not a formal derivation. The central claims rest on qualitative coding standards, the text-first alignment principle, LLM-as-judge agent reliability, and human labels as oracle—not on fitted physical constants. Free parameters are mostly engineering thresholds and prompt/taxonomy choices rather than numeric curve fits.

free parameters (4)
  • Misalignment taxonomy boundary criteria (SC/DP/IS/FP/UL/Others) = 5 patterns / 11 scenarios from 500 Verified instances
    Pattern definitions and multi-label rules are author-designed from open coding; different boundaries would change prevalence and EM scores.
  • IS sufficiency rule (maintainer must request missing bug-report components)
    Incomplete specification is flagged only under maintainer clarification requests, not pure annotator judgment—an explicit policy choice that sets precision/recall tradeoffs.
  • UL triple condition (hardcoded test literal + new in patch + absent from issue)
    Operational definition of unspecified literals is hand-specified and directly controls UL labels.
  • LLM decoding / agent prompt configurations = Four API backbones as evaluated
    Detection accuracy depends on backbone choice and task-specific prompts; not a single reported temperature schedule, but performance is backbone-sensitive (e.g., Qwen weaker).
axioms (6)
  • domain assumption Standard SWE-bench construction pairs a PR to issues via regex/reference extraction and treats issue text as problem_statement and PR patch as oracle.
    Stated in Introduction; defines the pipeline whose failures are studied.
  • ad hoc to paper PR-side natural language is abstractly comparable to issue text and is a more reliable first signal of intent than raw diffs alone.
    Core ‘text-driven, code-validation’ principle in §§1–3; motivates Phase I/II before code veto.
  • domain assumption A well-formed bug report includes reproduction, actual behavior, and expected behavior; maintainer requests for missing pieces indicate incompleteness.
    Issue Analyzer operationalization cites standard bug-report format literature and project templates.
  • domain assumption Human dual annotation under the authors’ guideline is an adequate ground truth for alignment labels.
    Evaluation Design §4.1; κ reported but still a social oracle assumption.
  • domain assumption LLM agents with GitHub/API and repo file tools can recover cross-PR temporal links and validate textual claims against diffs.
    Method §§3.1.3–3.3; required for DP/FP and Phase III.
  • standard math Open coding / grounded qualitative SE methods yield a transferable taxonomy of process-level pairing failures.
    Study design cites qualitative SE practice; treated as standard methodology rather than a new formal axiom.
invented entities (2)
  • PAIChecker three-phase multi-agent architecture (Issue Analyzer, PR Scope Analyzer, PR Connection Analyzer, Coordinator, Code Validator) independent evidence
    purpose: Automate detection and multi-label categorization of PR–Issue misalignment with veto-only later phases and an Others escape hatch.
    Primary system contribution; existence is demonstrated by implementation and experiments, not postulated as a physical entity.
  • PR–Issue misalignment taxonomy (SC, DP, IS, FP, UL + Others; 11 scenarios) independent evidence
    purpose: Name construction failure modes that break the issue-as-spec / PR-as-oracle assumption.
    Derived from manual coding of Verified; Others admits incompleteness of the five families.

pith-pipeline@v1.2.0-daily-grok45 · 30587 in / 3776 out tokens · 72734 ms · 2026-07-31T02:58:50.590381+00:00 · methodology

0 comments
read the original abstract

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

Figures

Figures reproduced from arXiv: 2607.28587 by Junjielong Xu, Manyi Wang, Pinjia He.

Figure 1
Figure 1. Figure 1: Examples of four misalignment patterns [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-agent rank change after excluding 68 mis [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Motivating Example iterative__dvc-4872: PR #4872 is accidentally linked to issue #4796 but actually fixes issue #4504. Backbone: Gemini-3.1-Pro Preview. Mini-SWE-Agent and CoT force-classify as SC; PaiChecker’s coordinator correctly drops SC and assigns Others by synthesizing suspicious clues from two subagents. We observe a monolithic approach struggles when all artifacts and detection rules are packed in… view at source ↗
Figure 5
Figure 5. Figure 5: Workflow of PaiChecker. Phase I performs specific misalignment identification through three specialized subagents. Phase II synthesizes their outputs into a preliminary label and rationale. Phase III validates the preliminary result against code-level evidence and supplementary GitHub context to produce the final decision. and Phase III validates the textual judgment against code-level evidence. Across all… view at source ↗
Figure 6
Figure 6. Figure 6: Binary Accuracy and Exact Match across pipeline [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 15 linked inside Pith

  1. [1]

    SWE-bench Annotation Instructions

    2024. SWE-bench Annotation Instructions. https://cdn.openai.com/introducing- swe-bench-verified/swe-b-annotation-instructions.pdf accessed:2026-03-06

    2024

  2. [2]

    SWE-bench Verified Official Experiments Result

    2024. SWE-bench Verified Official Experiments Result. https://github.com/SWE- bench/experiments accessed:2026-03-06

    2024

  3. [3]

    SWE-bench Verified Official Leaderboards

    2024. SWE-bench Verified Official Leaderboards. https://www.swebench.com/ index.html accessed:2026-03-06

    2024

  4. [4]

    Issue Template of Astropy

    2025. Issue Template of Astropy. https://github.com/astropy/astropy/tree/main/ .github/ISSUE_TEMPLATE Accessed: 2026-03-25

    2025

  5. [5]

    Issue Template of Pytest

    2025. Issue Template of Pytest. https://github.com/pytest-dev/pytest/tree/main/ .github/ISSUE_TEMPLATE Accessed: 2026-03-25

    2025

  6. [6]

    Issue Template of Sphinx

    2025. Issue Template of Sphinx. https://github.com/sphinx-doc/sphinx/tree/ master/.github/ISSUE_TEMPLATE Accessed: 2026-03-25

    2025

  7. [7]

    Issue Template of Xarray

    2025. Issue Template of Xarray. https://github.com/pydata/xarray/tree/main/ .github/ISSUE_TEMPLATE Accessed: 2026-03-25

    2025

  8. [8]

    Pull Request Template of Astropy

    2025. Pull Request Template of Astropy. https://github.com/astropy/astropy/ blob/main/.github/PULL_REQUEST_TEMPLATE.md Accessed: 2026-03-25

    2025

  9. [9]

    Pull Request Template of Scikit-learn

    2025. Pull Request Template of Scikit-learn. https://github.com/scikit-learn/ scikit-learn/blob/main/.github/PULL_REQUEST_TEMPLATE.md Accessed: 2026- 03-25

    2025

  10. [10]

    Pull Request Template of Sympy

    2025. Pull Request Template of Sympy. https://github.com/sphinx-doc/sphinx/ tree/master/.github/ISSUE_TEMPLATE Accessed: 2026-03-25

    2025

  11. [11]

    Claude Code

    2026. Claude Code. https://claude.com/product/claude-code accessed:2026-03- 06

    2026

  12. [12]

    2026. Codex. https://openai.com/codex/ accessed:2026-03-06

    2026

  13. [13]

    https://github.com/manyifire/PAIChecker

    2026.PAIChecker. https://github.com/manyifire/PAIChecker

    2026

  14. [14]

    Why SWE-bench Verified no longer measures frontier coding capa- bilities

    2026. Why SWE-bench Verified no longer measures frontier coding capa- bilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench- verified/ accessed:2026-03-06

    2026

  15. [15]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. SWE-Bench+: Enhanced Coding Benchmark for LLMs. doi:10.48550/arXiv.2410.06992

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.06992 2024

  16. [16]

    Anthropic. 2025. Claude Sonnet 4.6. https://www.anthropic.com/news/claude- sonnet-4-6 Accessed: 2026-03-25

    2025

  17. [17]

    Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. 2025. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents.arXiv preprint arXiv:2505.20411(2025)

    arXiv 2025

  18. [18]

    Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023. Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages7, OOPSLA1 (2023), 85–111

    2023

  19. [19]

    Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering (SIGSOFT ’08/FSE-16). Association for Computing Machinery, New York, NY, USA, 308–318. doi:10.1145/1453101.1453146

    arXiv 2008

  20. [20]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

  21. [21]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  22. [22]

    Mouxiang Chen, Lei Zhang, Yunlong Feng, Xuwu Wang, Wenting Zhao, Ruisheng Cao, Jiaxi Yang, Jiawei Chen, Mingze Li, Zeyao Ma, Hao Ge, Zongmeng Zhang, Zeyu Cui, Dayiheng Liu, Jingren Zhou, Jianling Sun, Junyang Lin, and Binyuan Hui. 2026. SWE-Universe: Scale Real-World Verifiable Environments to Millions. doi:10.48550/arXiv.2602.02361 arXiv:2602.02361 [cs]

    doi:10.48550/arxiv.2602.02361 2026

  23. [23]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe- bench-verified/ accessed:2026-03-06

    2024

  24. [24]

    Mariana Coutinho, Lorena Marques, Anderson Santos, Marcio Dahia, Cesar França, and Ronnie de Souza Santos. 2024. The role of generative ai in software development productivity: A pilot case study. InProceedings of the 1st ACM International Conference on AI-Powered Software. 131–138

    2024

  25. [25]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941(2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Google. 2025. Gemini 3.1 Pro. https://blog.google/innovation-and-ai/models- and-research/gemini-models/gemini-3-1-pro/ Accessed: 2026-03-25

    2025

  27. [27]

    Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories?arXiv preprint arXiv:2507.12415(2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https://arxiv.org/abs/2403.07974

    Pith/arXiv arXiv 2024

  29. [29]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  30. [31]

    OpenAI. 2025. Introducing GPT-5.3 Codex. https://openai.com/index/ introducing-gpt-5-3-codex/ Accessed: 2026-03-25

    2025

  31. [32]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139(2024)

    Pith/arXiv arXiv 2024

  32. [33]

    Jorge Pérez, Jessica Díaz, Javier Garcia-Martin, and Bernardo Tabuenca. 2020. Systematic literature reviews in software engineering—enhancement of the study selection process using Cohen’s Kappa statistic.Journal of Systems and Software 168 (2020), 110657

    2020

  33. [34]

    Qwen Team. 2025. Qwen 3.5. https://qwen.ai/blog?id=qwen3.5 Accessed: 2026-03-25

    2025

  34. [35]

    Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buch- holz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al . 2025. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents.arXiv preprint arXiv:2504.08703(2025)

    Pith/arXiv arXiv 2025

  35. [36]

    Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. InProceedings of PaiChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks ASE ’26, October 12–16, 2026, Munich, Germany the 38th International conference on software engineering. 120–131

    2016

  36. [37]

    Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, and Haoli Bai. 2026. SWE-Lego: Pushing the Limits of Super- vised Fine-tuning for Software Issue Resolving. doi:10.48550/arXiv.2601.01426 arXiv:2601.01426 [cs]

    doi:10.48550/arxiv.2601.01426 2026

  37. [38]

    Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling. doi:10.48550/arXiv.2507. 23370

    doi:10.48550/arxiv.2507 2025

  38. [39]

    Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. 2025. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. doi:10.48550/arXiv.2512.18470

    Pith/arXiv arXiv doi:10.48550/arxiv.2512.18470 2025

  39. [40]

    Lilin Wang, Lucas Ramalho, Alan Celestino, Phuc Anthony Pham, Yu Liu, Umang Kumar Sinha, Andres Portillo, Onassis Osunwa, and Gabriel Maduekwe

  40. [41]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations, Vol. 2025. 65882–65919

    2025

  41. [42]

    Solved Issues

    You Wang, Michael Pradel, and Zhongxin Liu. 2025. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study. icse 2026. doi:10.1145/ 3744916.3764576

    arXiv 2025

  42. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2023

  43. [44]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agent- less: Demystifying LLM-based Software Engineering Agents. doi:10.48550/ARXIV. 2407.01489

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.01489 2024

  44. [45]

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen

  45. [46]

    Junjielong Xu, Ying Fu, Shin Hwei Tan, and Pinjia He. 2025. Aligning the Objective of LLM-based Program Repair. Inicse 2025. icse 2025. doi:10.48550/arXiv.2404. 08877 arXiv:2404.08877 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2404 2025

  46. [47]

    InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.)

    SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 1123–

    2025

  47. [48]

    John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khand- pur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

  48. [49]

    Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang, Yingwei Ma, Yanhao Li, Yue Liu, Zhenxing Hu, Kaitai Zhang, Shuyi Wang, Huarong Chen, Flood Sung, Yang Liu, Yang Gao, Zhilin Yang, and Tianyu Liu. 2025. Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents. iclr 2026. doi:10.48550/arXiv.2509.23045

    doi:10.48550/arxiv.2509.23045 2025

  49. [50]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InThe Thirty-eighth Annual Con- ference on Neural Information Processing Systems. https://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  50. [51]

    Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. 2025. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench. arXiv:2506.09289 [cs.SE] https: //arxiv.org/abs/2506.09289

    Pith/arXiv arXiv 2025

  51. [52]

    Swe-smith: Scaling data for software engineering agents.arXiv preprint arXiv:2504.21798(2025)

    Pith/arXiv arXiv 2025

  52. [53]

    Lei Zhang, Jiaxi Yang, Min Yang, Jian Yang, Mouxiang Chen, Jiajun Zhang, Zeyu Cui, Binyuan Hui, and Junyang Lin. [n. d.]. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner. ([n. d.])

  53. [54]

    Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand. 2026. SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. arXiv:2603.00520 [cs.SE] https: //arxiv.org/abs/2603.00520

    arXiv 2026

  54. [56]

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. 2025. Swe-bench goes live!arXiv preprint arXiv:2505.23419(2025)

    Pith/arXiv arXiv 2025

  55. [58]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Vienna Austria, 1592–1604. doi:10.1145/3650212.3680384

    arXiv 2024

  56. [1139]

    doi:10.18653/v1/2025.findings-acl.62

    doi:10.18653/v1/2025.findings-acl.62 2025

  57. [2025]

    doi:10.48550/arXiv

    SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories. doi:10.48550/arXiv. 2512.17419

    doi:10.48550/arxiv.2512.17419