REVIEW 5 major objections 7 minor 66 references
Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports
T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The strongest coding agents fix only 20% of bugs they must discover without issue reports.
desk verdict A genuinely new proactive-fixing benchmark, but the headline 20% resolved rate conflates task infeasibility with agent incapability until the authors validate that bugs are actually discoverable from code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery that carries the argument is the benchmark construction pipeline and its dual-track evaluation protocol. Each task removes the issue report and instead supplies a unified bug-fixing template plus the set of files pending review, extracted from the reference patch of a GitHub pull request; simple tasks take one recorded bug, and hard tasks merge temporally adjacent pull requests so one snapshot contains multiple recorded bugs. The recorded-bug track scores localization through hunk-level recall and precision against the reference patch and scores repair through a test-driven Resolved indicator; the potential-bug track asks the agent to generate tests reproducing each bug it claims to have fixed, then checks fail-to-pass behavior and semantic association with a judge. A taxonomy-driven curation step with multi-LLM consensus filters out non-bug pull requests, and an automated Docker setup agent makes each task executable and reproducible.
What would settle it
Have a sample of the recorded-bug tasks attempted by human engineers who see only the files pending review and the bug-taxonomy template, with the issue report withheld; if humans cannot locate a substantial fraction of the recorded bugs, then those tasks are not discoverable from code and the benchmark's resolved rates reflect infeasibility rather than agent capability.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that proactive bug fixing – locating and repairing bugs from code inspection alone – is a distinct and largely unsolved capability for state-of-the-art coding agents. Across 400 curated tasks, the strongest model resolves 20.0% of recorded bugs, and the reactive-versus-proactive comparison on a shared subset shows the same models falling from around 59% to 26% resolved when the issue text is removed. Agents that locate the bug are far more likely to fix it, yet localization recall stays below 30% even for the best model, so the paper concludes that accurate localization is the critical precondition for successful repair. In the hard multi-bug setting, full resolution is almost never achieved (0.4% for the best open-source model reported), and agents often fix only a subset of the recorded bugs. On the potential-bug track, agents can surface and validate additional bugs with generated tests at higher rates (up to 75.0% revealed), but invalid test generation and partial coverage keep these results from translating into full repair capability.
Load-bearing premise
The benchmark's central bet is that the bugs it mines from GitHub pull requests are discoverable from the code alone once the issue text is removed; if a bug's existence is only knowable through external intent or the issue report itself, the task is infeasible and the low resolved rates would measure data quality, not agent ability.
Editorial extensions
If this is right
- Bug localization, not patch generation, is the binding constraint; improving how agents explore repositories should raise resolved rates more than improving code editors.
- Reactive benchmarks overstate real-world agent capability by roughly a factor of two, so issue-report-based leaderboards should be read with that gap in mind.
- Multi-bug scenarios are a qualitatively harder regime; evaluation that only tests single-bug fixes misses the failure mode where agents stop after the first fix.
- Test generation is a workable validation mechanism for agent-discovered bugs, but only if tests are fail-to-pass and cover every claimed bug; both failure modes occur often.
- Agent scaffold and model size both materially change proactive performance, so the harness is part of the capability, not just the model.
Reading between the lines
- I would predict that injecting minimal hints – such as file names mentioned in commit messages or recently changed tests – will close much of the reactive-to-proactive gap, because the paper shows localization is the bottleneck rather than repair.
- The potential-bug track could transfer directly to security-oriented code auditing, since it needs no gold patch and only requires a valid test that reproduces the discovered flaw.
- The hard-setting design may understate genuine multi-bug capability: some temporally adjacent pull requests fix regressions introduced by intervening code, so the earlier snapshot may not contain a discoverable fault.
- A human baseline (expert engineers attempting the same report-free tasks) would separate 'benchmark infeasibility' from 'agent weakness' – the paper does not report one.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Active-SWE, a benchmark for evaluating coding agents on proactive bug fixing, i.e., discovering and repairing bugs in a repository without an issue report. The benchmark contains 1,663 tasks across six bug categories and eight programming languages, with a curated 400-task subset used for the main experiments. The construction pipeline mines GitHub PRs, uses LLM annotators to filter for taxonomy-valid bug fixes, sets up executability, and then discards issue text to create simple (single-bug) and hard (multi-bug) instances. Evaluation has two tracks: recorded bugs, scored by localization recall/precision and a Resolved indicator against human-written reference patches and tests, and potential bugs, scored by generated-test validity and an LLM judge's Revealed metric. Experiments on 17 state-of-the-art LLMs report that the best model, Claude Opus 4.8, resolves only 20.0% of recorded bugs, with most models well below that, and the paper concludes that current coding agents struggle substantially in proactive settings.
Significance. If the benchmark is valid, Active-SWE fills a genuine gap: existing SWE benchmarks assume issue reports are available, whereas real-world bug discovery is often proactive and report-free. The benchmark's scale, multilingual coverage, and dual-track evaluation (recorded plus potential bugs) are useful contributions, and the decoupled localization and repair metrics are a sensible way to attribute failure modes. The headline result—20.0% Resolved for the best model—would be an important indicator that today's agents are far from autonomous bug discovery and repair. The main weaknesses are evaluation validity: no feasibility check that recorded bugs are discoverable from code alone, a potential-bug track whose judge is an evaluated model without human validation, and single-run results without variance estimates. These issues need to be addressed before the quantitative claims can be taken at face value.
major comments (5)
- [§3.2–§3.3] The construction pipeline never validates that recorded bugs are discoverable from the repository snapshot and the file list alone, despite the task requiring exactly that. Eq. (3) labels PRs using the issue report s_i, and Eq. (5) extracts the review scope from the reference patch; the issue text is then discarded in §3.3. The case studies in Figs. 13 and 14 show bugs whose intended behavior (quarters support in DateTime.diff; zero-row label-distribution normalization) is supplied by the issue report, not derivable from the code alone. Without an oracle or upper-bound study on the same instances, or a per-instance discoverability filter, the headline 20.0% Resolved in Table 2 conflates task feasibility with agent capability.
- [§3.4.2] The potential-bug track is self-validating and lacks human validation. In Eq. (9), the same evaluated LLM L that produced the patch also generates the reproducing tests, and in Eq. (10) the judge J is instantiated as Qwen3.5-397B (§4.1), which is itself one of the evaluated baselines in Table 2. No human-annotated subset, inter-annotator agreement, or independent-judge comparison is reported. For the Qwen3.5-397B row, the model is effectively judging its own patch-test pairs. This can inflate TV/Revealed and should be addressed by a human-validated sample or by removing the judge model from the baseline set.
- [Tables 2, 8] All reported scores come from a single run with no variance estimates. On the 400-instance curated subset used in Table 2, the difference between the top Resolved values (20.0% vs. 18.5%) is within a few percentage points of binomial standard error, and the scaffold comparison in Table 8 shows differences of up to 48 points (e.g., Claude Opus 4.8 Revealed 73.0 under Claude Code vs. 25.0 under SWE-agent). Without multiple seeds, bootstrap confidence intervals, or standard errors, the cross-model and cross-scaffold rankings are not statistically grounded. Please report variance or at least per-model standard errors.
- [Eq. (11)] The Test Validity metric TV = I[|t_i|=|f_i|] requires the number of generated tests to equal the number of fail-to-pass tests, so any extra or duplicate test that does not exhibit fail-to-pass behavior makes TV=0 and, because Revealed is built on TV=1, also makes Revealed=0. This penalizes correct bug identification whenever the agent writes a redundant test, conflating test-set parsimony with bug validity. A partial-credit metric (e.g., the fraction of generated tests that are fail-to-pass, or Revealed conditioned on the existence of at least one matching F2P test per bug) would better separate coverage from over-generation.
- [§3.3, Eq. (6)] Hard-instance integration is validated only through the condition f_i = ∪_j f_j, but the union of two PR patches applied to the earliest snapshot may fail to apply cleanly, conflict, or produce a repository state that was never tested. No conflict rate or environment-rebuild statistics are reported, and the window size is fixed to 2 (§4.1) without ablating M. Since 100 of the 400 evaluation tasks are hard instances, a nontrivial integration failure rate would directly affect the hard-setting numbers in Table 7.
minor comments (7)
- [§3.2, Eq. (4)] The number of annotators K and the voting rule V(·) are unspecified; the implementation mentions three models, but it is unclear whether the final label requires unanimity or a majority.
- [Eq. (7)] The hunk matching relation ∼ is not defined; please specify the matching criterion (e.g., exact context match or a similarity threshold).
- [Eq. (11)] The symbol f_i is reused: in §3.2 it denotes the reference fail-to-pass tests, while in Eq. (11) it denotes the subset of generated tests exhibiting fail-to-pass behavior. Rename one of them to avoid confusion.
- [§3.3] The hard setting says the review scope involves at least M bugs, but M is never explicitly stated; §4.1 says the window size is 2, yet the threshold M should be defined in the formulation.
- [Fig. 5] The 'Located if LR≥0.5' threshold is introduced without justification; please state how this threshold was chosen and whether the qualitative conclusions are sensitive to it.
- [Table 7] The column 'Sub Res.' is undefined; add a caption or footnote explaining what it measures.
- [Abstract / §4.1] The abstract states 1,663 tasks while the main experiments use a curated 400-task subset; the distinction between Active-SWE and Active-SWE-Extend should be clarified in the main text, not only in the appendix.
Circularity Check
Potential-bug 'Revealed' metric is self-referential: the evaluated LLM writes its own validation tests and the judge is an evaluated baseline; the recorded-bug track is externally grounded.
-
other
[Section 3.4.2 (Eqs. 9-12) and Section 4.1 Implementation Details]
"ti = L(ri, ci, Fi, TS) (9) ... Mi = J(ri, ci, ti, TJ) (10) ... Revealed = I[∀j ∈ {1, . . . , |bi|}, ∃k ∈ {1, . . . , |fi|}: Mi[j, k] = 1] (12) ... For potential bug evaluation, we adopt one of the most popular open-source models, i.e., Qwen3.5-397B, as the LLM-based judge agent"
The 'Revealed' metric is supposed to validate potential bugs, but the evidence t_i is generated by the same evaluated model L after it has produced its own patch c_i (Eq. 9). The judge J in Eq. 10 that decides whether tests match bugs is Qwen3.5-397B, which is itself one of the baselines scored in Table 2. Thus a model can increase its own Revealed score by writing tests that assert its own behavioral changes and having an evaluated LLM judge accept them. The metric therefore measures self-consistency between a model's patch and its self-authored tests, rather than independent validity of the discovered bugs; this is a self-referential evaluation step rather than externally grounded evidence.
full rationale
The recorded-bug track is not circular: gold tests t_i are human PR test patches inherited from real GitHub PRs, and Eq. (8) requires the model patch to pass all of them. The headline result that the best model resolves only 20.0% of recorded bugs is therefore an externally verifiable finding, not a fitted parameter renamed as a prediction. The taxonomy curation and difficulty-aware task formulation also do not define correctness in terms of the evaluated models, and the citations to prior SWE benchmarks are background context rather than load-bearing self-citations. The one real circular element is the potential-bug track: Eq. (9) lets the evaluated LLM generate the tests that validate its own patch, and Eq. (10) uses an evaluated baseline (Qwen3.5-397B) as the judge for all models, so the Revealed metric is partially self-referential. The lack of a feasibility check that recorded bugs are discoverable from code inspection alone (Sec. 3.3) is a task-validity threat, not circularity, and is therefore not counted in the score beyond the partial circularity already noted.
Assumptions & free parameters
free parameters (4)
- hard-setting window size =
2
- number of LLM annotators K =
3
- curated subset size =
400 (300 simple, 100 hard)
- located threshold for LR =
0.5
assumptions (6)
- domain assumption GitHub PRs linked to issues provide ground-truth bug-fixing patches and tests.
- domain assumption Consensus among three LLM annotators yields valid bug taxonomy labels.
- domain assumption The setup agent can construct executable docker environments correctly within the turn limit.
- domain assumption Fail-to-pass tests, when passed after patch application, certify bug resolution.
- domain assumption The LLM judge agent can reliably map each revealed bug to generated tests.
- domain assumption Recorded bugs are discoverable from code inspection alone without the issue report.
Cite this review
Pith. "Pith review of Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports." pith.science (2026). https://pith.science/paper/UDFFON6U
@misc{pith2026260804682,
author = {Pith},
title = {Pith review of: Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports},
year = {2026},
howpublished = {\url{https://pith.science/paper/UDFFON6U}},
note = {Machine review of arXiv:2608.04682}
}
read the original abstract
Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase. However, existing SWE benchmarks typically assume that high-quality issue reports with detailed information are always available, which is easily violated in practice due to the complexity of report acquisition and curation. To address this, we introduce Active-SWE, a benchmark for evaluating coding agents on proactively discovering and fixing multiple bugs without report guidance, covering 1,663 tasks across six bug categories and eight languages. Beyond shifting the focus from existing reactive bug fixing to proactive bug fixing, Active-SWE enables a more in-depth evaluation by expanding the scope from fixing a specific recorded bug to multiple-bug fixing and potential bug discovery scenarios. To construct Active-SWE, we propose a novel difficulty-aware task formulation pipeline with a dual-track evaluation framework, facilitating comprehensive evaluation of proactive bug-fixing capability. Extensive experiments reveal that most state-of-the-art coding agents struggle with proactive bug-fixing tasks, demonstrating limited performance in locating and resolving recorded bugs, handling multiple bug fixing scenarios, and discovering valid potential bugs.
Figures
Figures from the paper (14 more)
Reference graph
Works this paper leans on
-
[1]
Swe-bench: Can language models resolve real-world github issues? , author=. ICLR , year=
-
[3]
arXiv preprint arXiv:2302.13971 , year=
Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=
-
[4]
ACM transactions on intelligent systems and technology , year=
A survey on evaluation of large language models , author=. ACM transactions on intelligent systems and technology , year=
-
[5]
Frontiers of Computer Science , year=
A survey of large language models , author=. Frontiers of Computer Science , year=
-
[6]
ICSE-FoSE , year=
Large language models for software engineering: Survey and open problems , author=. ICSE-FoSE , year=
-
[8]
External Technical Root Cause Analysis — Channel File 291 , author =. 2024 , howpublished =
work page 2024
-
[9]
Helping our customers through the CrowdStrike outage , author =. 2024 , howpublished =
work page 2024
-
[10]
CrowdStrike to Cost Fortune 500 \ 5.4b; Insured Loss Range of \ 0.54b - \ 1.08b , author =. 2024 , howpublished =
work page 2024
Show all 66 references
-
[11]
2026 , howpublished =
Why SWE-bench Verified no longer measures frontier coding capabilities , author =. 2026 , howpublished =
2026
-
[12]
2026 , howpublished =
Separating signal from noise in coding evaluations , author =. 2026 , howpublished =
2026
-
[13]
NeurIPS , year=
Swe-bench goes live! , author=. NeurIPS , year=
-
[15]
NeurIPS , year=
Multi-swe-bench: A multilingual benchmark for issue resolving , author=. NeurIPS , year=
-
[16]
arXiv preprint arXiv:2410.06992 , year=
Swe-bench+: Enhanced coding benchmark for llms , author=. arXiv preprint arXiv:2410.06992 , year=
-
[17]
ACL , year=
Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation , author=. ACL , year=
-
[21]
NeurIPS , year=
Swt-bench: Testing and validating real-world bug-fixes with code agents , author=. NeurIPS , year=
-
[23]
Advances in Neural Information Processing Systems , volume=
Swe-agent: Agent-computer interfaces enable automated software engineering , author=. Advances in Neural Information Processing Systems , volume=
-
[24]
ACL , year=
Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges , author=. ACL , year=
-
[25]
ICLR , year=
Openhands: An open platform for ai software developers as generalist agents , author=. ICLR , year=
-
[26]
ACL , year=
Mapcoder: Multi-agent code generation for competitive problem solving , author=. ACL , year=
-
[27]
NeurIPS , year=
Swe-smith: Scaling data for software engineering agents , author=. NeurIPS , year=
-
[30]
2026 , howpublished =
System Card: Claude Sonnet 4.6 , author =. 2026 , howpublished =
2026
-
[32]
2026 , howpublished =
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author =. 2026 , howpublished =
2026
-
[35]
Qwen3.5: Accelerating Productivity with Native Multimodal Agents , howpublished =
Qwen Team , year =. Qwen3.5: Accelerating Productivity with Native Multimodal Agents , howpublished =
-
[36]
2026 , howpublished =
Gemini 3.1 Pro Model Card , author =. 2026 , howpublished =
2026
-
[37]
2026 , howpublished =
System Card: Claude Opus 4.7 , author =. 2026 , howpublished =
2026
-
[38]
2026 , howpublished =
System Card: Claude Opus 4.8 , author =. 2026 , howpublished =
2026
-
[42]
2026 , howpublished =
2026
-
[43]
System card: Claude sonnet 4.6
Anthropic . System card: Claude sonnet 4.6. https://www-cdn.anthropic.com/78073f739564e986ff3e28522761a7a0b4484f84.pdf, 2026 a
2026
-
[44]
System card: Claude opus 4.8
Anthropic . System card: Claude opus 4.8. https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80fef830a37ea.pdf, 2026 b
2026
-
[45]
A survey on evaluation of large language models
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 2024
2024
-
[46]
The minimax-m2 series: Mini activations unleashing max real-world intelligence
Aili Chen, Aonian Li, Baichuan Zhou, Bangwei Gong, Binyang Jiang, Boji Dan, Changqing Yu, Chao Wang, Cheng Ma, Cheng Zhong, et al. The minimax-m2 series: Mini activations unleashing max real-world intelligence. arXiv preprint arXiv:2605.26494, 2026
2026 arXiv
-
[47]
External technical root cause analysis — channel file 291
CrowdStrike . External technical root cause analysis — channel file 291. https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf, 2024
2024
-
[48]
Deepseek-v4: Towards highly efficient million-token context intelligence
DeepSeek-AI . Deepseek-v4: Towards highly efficient million-token context intelligence. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf, 2026
2026
-
[49]
Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025
Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025
2025 arXiv
-
[50]
Large language models for software engineering: Survey and open problems
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. In ICSE-FoSE, 2023
2023
-
[51]
Gemini 3.1 pro model card
Google DeepMind . Gemini 3.1 pro model card. https://deepmind.google/models/model-cards/gemini-3-1-pro/, 2026
2026
-
[52]
Agentcoder: Multi-agent-based code generation with iterative testing and optimisation
Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023
2023 arXiv
-
[53]
Mapcoder: Multi-agent code generation for competitive problem solving
Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. In ACL, 2024
2024
-
[54]
Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024
2024
-
[55]
From llms to llm-based agents for software engineering: A survey of current, challenges and future
Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479, 2024
2024 arXiv
-
[56]
Ling and ring 2.6 technical report: Efficient and instant agentic intelligence at trillion-parameter scale
Ang Li, Ben Liu, Bin Han, Bin Hu, Bin Jing, Binbin Hu, Bing Li, Cai Chen, Caizhi Tang, Changxin Tian, et al. Ling and ring 2.6 technical report: Efficient and instant agentic intelligence at trillion-parameter scale. arXiv preprint arXiv:2606.15079, 2026
2026 arXiv
-
[57]
Testexplora: Benchmarking llms for proactive bug discovery via repository-level test generation
Steven Liu, Jane Luo, Xin Zhang, Aofan Liu, Hao Liu, Jie Wu, Ziyang Huang, Yangyu Huang, Yu Kang, and Scarlett Li. Testexplora: Benchmarking llms for proactive bug discovery via repository-level test generation. arXiv preprint arXiv:2602.10471, 2026
2026
-
[58]
Helping our customers through the crowdstrike outage
Microsoft . Helping our customers through the crowdstrike outage. https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/, 2024
2024
-
[59]
u ndler, Mark N M \
Niels M \"u ndler, Mark N M \"u ller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents. In NeurIPS, 2024
2024
-
[60]
Why swe-bench verified no longer measures frontier coding capabilities
OpenAI . Why swe-bench verified no longer measures frontier coding capabilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, 2026 a
2026
-
[61]
Separating signal from noise in coding evaluations
OpenAI . Separating signal from noise in coding evaluations. https://openai.com/index/separating-signal-from-noise-coding-evaluations/, 2026 b
2026
-
[62]
Crowdstrike to cost fortune 500 \ 5.4b; insured loss range of \ 0.54b - \ 1.08b
Parametrix . Crowdstrike to cost fortune 500 \ 5.4b; insured loss range of \ 0.54b - \ 1.08b. https://www.parametrixinsurance.com/in-the-news/crowdstrike-to-cost-fortune-500-5-4-billion-insured-loss-range-of-540-million-to-1-08-billion, 2024
2024
-
[63]
Qwen3.7 : The agent frontier, May 2026
Qwen Team . Qwen3.7 : The agent frontier, May 2026. URL https://qwen.ai/blog?id=qwen3.7
2026
-
[64]
Swe atlas: Benchmarking coding agents beyond issue resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, Johannes Baptist Mols, Gautam Anand, Cole McCollum, Edgar Arakelyan, Vijay Bharadwaj, Andrew Park, et al. Swe atlas: Benchmarking coding agents beyond issue resolution. arXiv preprint arXiv:2605.08366, 2026
2026 arXiv
-
[65]
Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complexity. arXiv preprint arXiv:2607.00248, 2026
2026 arXiv
-
[66]
Openai gpt-5 system card
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025
2025 arXiv
-
[67]
Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026
2026 arXiv
-
[68]
Qwen3.5: Accelerating productivity with native multimodal agents
Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents. https://qwen.ai/blog?id=qwen3.5, 2026
2026
-
[69]
Tencent Hy Team . Hy3 . https://huggingface.co/tencent/Hy3, 2026
2026
-
[70]
Openhands: An open platform for ai software developers as generalist agents
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In ICLR, 2025
2025
-
[71]
Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025
Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. Live-swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025
2025
-
[72]
Swe-agent: Agent-computer interfaces enable automated software engineering
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37: 0 50528--50652, 2024 a
2024
-
[73]
Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024 b
John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024 b
-
[74]
Swe-smith: Scaling data for software engineering agents
John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. In NeurIPS, 2026
2026
-
[75]
React: Synergizing reasoning and acting in language models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022
2022 arXiv
-
[76]
Multi-swe-bench: A multilingual benchmark for issue resolving
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. In NeurIPS, 2026
2026
-
[77]
Glm-5: from vibe coding to agentic engineering
Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026
2026 arXiv
-
[78]
Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In ACL, 2024
2024
-
[79]
Swe-bench goes live! In NeurIPS, 2026
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! In NeurIPS, 2026
2026
-
[80]
A survey of large language models
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Zican Dong, Yupeng Hou, Beichen Zhang, Yingqian Min, Junjie Zhang, Peiyu Liu, et al. A survey of large language models. Frontiers of Computer Science, 2026
2026
-
[81]
Featurebench: Benchmarking agentic coding for complex feature development
Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, et al. Featurebench: Benchmarking agentic coding for complex feature development. arXiv preprint arXiv:2602.10975, 2026
2026
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.