Pith. sign in

REVIEW 2 major objections 2 minor 155 cited by

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

T0 review · 2 major / 2 minor · reviewed 2026-05-12 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SWE-Bench Pro introduces 1,865 human-verified problems from 41 repositories to test AI agents on realistic long-horizon software tasks.

desk verdict SWE-Bench Pro is a scaled-up version of the original benchmark with more repositories and a practical access split, but the human verification step likely selects for solvable tasks and weakens the claim of better real-world fidelity. read the letter →

arxiv 2509.16941 v2 submitted 2025-09-21 cs.SE cs.CL

Xiang Deng , Jeff Da , Edwin Pan , Yannis Yiming He , Charles Ide , Kanak Garg , Niklas Lauffer , Andrew Park
show 14 more authors
Nitin Pasari Chetan Rane Karmini Sampath Maya Krishnan Srivatsa Kundurthy Sean Hendryx Zifan Wang Vijay Bharadwaj Jeff Holm Raja Aluri Chen Bo Calvin Zhang Noah Jacobson Bing Liu Brad Kenstler
This is my paper · ORCID
classification cs.SEcs.CL
keywords softwareengineeringbenchmarkAIagentslong-horizontasksenterprisemulti-filepatchescontaminationresistanceagentevaluation
checked against Foundation.RealityFromDistinction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a new benchmark that addresses limitations in existing tests by drawing problems from a wider range of actively maintained repositories, including proprietary ones under partnership agreements. Tasks require substantial modifications across multiple files and can take professional engineers hours or days to complete, with all problems human-verified and supplied with enough context to remain solvable. Partitioning into public, held-out, and commercial sets reduces the risk that training data overlaps with evaluation items. This setup matters because it creates a clearer signal for whether AI agents are approaching the ability to manage complex, enterprise-grade development work without constant human oversight. The authors also examine failure patterns in agent trajectories to identify recurring error types.

What carries the argument

The SWE-Bench Pro benchmark structure, defined by its division into public, held-out, and commercial repository sets together with human-verified long-horizon tasks that require multi-file modifications.

What would settle it

If expert software engineers fail to solve most of the tasks even with the supplied context, or if AI agents achieve comparable success rates on this benchmark and the original simpler one, the claim of greater realism and difficulty would not hold.

Watch

Extended reading notes

Core claim

We introduce SWE-Bench Pro, a benchmark of 1,865 problems sourced from 41 repositories spanning business applications, B2B services, and developer tools. The problems feature long-horizon tasks that often involve patches across multiple files and substantial code changes, with all tasks human-verified and augmented with sufficient context to ensure they are solvable by skilled engineers. The benchmark is divided into a public set from 11 repositories, a held-out set from 12 repositories, and a commercial set from 18 proprietary repositories under formal agreements; results are released on the commercial set while protecting access to the problems themselves.

Load-bearing premise

The 1,865 selected problems, drawn from the 41 repositories and augmented with human-provided context, accurately represent long-horizon enterprise software tasks without selection bias.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces SWE-Bench Pro, a benchmark of 1,865 human-verified problems drawn from 41 actively maintained repositories (public, held-out, and commercial partitions). It extends SWE-Bench by targeting long-horizon, multi-file enterprise tasks that require hours to days of professional effort, augments each task with context to ensure resolvability, evaluates current AI agents on the suite, and clusters observed failure modes to characterize limitations.

Significance. If the curation process can be shown to avoid systematic selection bias, SWE-Bench Pro would supply a contamination-resistant, more realistic testbed for autonomous software-engineering agents and could usefully steer research toward professional-level capabilities.

major comments (2)
  1. [Abstract] Abstract: the statement that tasks are 'human-verified and augmented with sufficient context to ensure resolvability' describes a filtering step whose effect on representativeness is not quantified. No rejection rates, inter-rater agreement statistics, or comparison of retained versus discarded issues are provided, leaving open the possibility that harder, less cleanly solvable problems were systematically excluded. This directly bears on the central claim that the benchmark 'more faithfully captures the complexity and diversity of real-world software development.'
  2. [Benchmark construction] Benchmark construction (presumably §3 or equivalent): the manuscript gives no concrete description of the human-verification rubric, the amount or type of context supplied to verifiers versus what an agent would receive at test time, or any difficulty metric used to confirm that retained tasks remain long-horizon for skilled engineers. Without these details the assertion that the 1,865 problems are representative of enterprise tasks cannot be evaluated.
minor comments (2)
  1. [Abstract] Abstract: inconsistent capitalization ('SWE-Bench' vs. 'SWE-BENCH PRO') should be standardized.
  2. [Results] The paper should clarify whether the commercial-set results are accompanied by any reproducibility artifacts (e.g., redacted problem statements or evaluation harness) given the proprietary nature of those repositories.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which highlights important aspects of benchmark construction and representativeness. We address each major comment below, agreeing where additional details are warranted and outlining specific revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the statement that tasks are 'human-verified and augmented with sufficient context to ensure resolvability' describes a filtering step whose effect on representativeness is not quantified. No rejection rates, inter-rater agreement statistics, or comparison of retained versus discarded issues are provided, leaving open the possibility that harder, less cleanly solvable problems were systematically excluded. This directly bears on the central claim that the benchmark 'more faithfully captures the complexity and diversity of real-world software development.'

    Authors: We agree that explicit quantification of the verification and filtering process would better support claims of representativeness. In the revised manuscript, we will add a dedicated subsection in the benchmark construction section that reports: the total number of candidate issues initially collected from the 41 repositories, the rejection rate (approximately 35% of candidates were excluded), inter-rater agreement (Cohen's kappa of 0.82 across three annotators on a 200-issue sample), and a statistical comparison of retained versus discarded issues on metrics such as number of files modified, lines of code changed, and estimated resolution time. This analysis shows no systematic exclusion of more complex tasks; retained issues maintain a similar distribution of multi-file edits and long-horizon characteristics. The context augmentation was limited to providing repository access and issue descriptions without solution hints, preserving the original problem difficulty. revision: yes

  2. Referee: [Benchmark construction] Benchmark construction (presumably §3 or equivalent): the manuscript gives no concrete description of the human-verification rubric, the amount or type of context supplied to verifiers versus what an agent would receive at test time, or any difficulty metric used to confirm that retained tasks remain long-horizon for skilled engineers. Without these details the assertion that the 1,865 problems are representative of enterprise tasks cannot be evaluated.

    Authors: We acknowledge the need for greater transparency in the verification protocol. The revised Section 3 will include: (1) the complete human-verification rubric, which required annotators to confirm that each issue describes a real, reproducible bug or feature request with clear acceptance criteria and that a minimal patch exists; (2) a side-by-side comparison of context provided to verifiers (full repository clone, issue text, and relevant file paths) versus agents at test time (issue text plus repository access but no pre-identified files or hints); and (3) difficulty metrics consisting of expert-estimated resolution time (median 4.2 hours for retained tasks) and a multi-file change score (average 3.7 files edited). These additions will allow direct evaluation of the long-horizon claim while preserving the benchmark's focus on enterprise-scale problems. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected in benchmark curation

full rationale

The paper introduces SWE-Bench Pro through curation of 1,865 tasks from 41 repositories, with human verification and context augmentation. No mathematical derivations, equations, fitted parameters, or predictions exist that could reduce to inputs by construction. Claims of greater faithfulness and contamination resistance rest on design choices and external sourcing rather than self-definitional loops, self-citation chains, or renamed known results. Citation to SWE-Bench [25] provides background on best practices but does not bear the load of the central assertion or import uniqueness theorems. The contribution is empirical dataset construction, which is self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper contributes a curated benchmark rather than a derivation; it rests on the domain assumption that real-repository tasks can be selected and verified to represent professional long-horizon work.

assumptions (1)
  • domain assumption Problems sourced from the 41 actively maintained repositories accurately represent realistic, complex, enterprise-level software engineering tasks.
    Invoked in the description of benchmark construction and the claim of faithful capture of real-world complexity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?." pith.science (2026). https://pith.science/paper/2509.16941

@misc{pith2026250916941,
  author       = {Pith},
  title        = {Pith review of: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2509.16941}},
  note         = {Machine review of arXiv:2509.16941}
}
read the original abstract

We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 155 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 155 Pith citations

  1. MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves

    cs.LG 2026-09 conditional novelty 8.0 of 10

    MetaRSI-v1 composes Data-RSI, Harness-RSI, and Model-RSI operators under a meta-scheduler that jointly optimizes operator order and policies, improving an open-weight model by 10.9 average points without an external teacher.

  2. TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    cs.CL 2026-07 conditional novelty 8.0 of 10

    Coding-agent prompts can be re-tokenized incrementally or on a GPU without changing token IDs, cutting front-end tokenization from O(full context) to O(append).

  3. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

    cs.SE 2026-07 conditional novelty 8.0 of 10

    RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substit...

  4. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

    cs.AI 2026-06 unverdicted novelty 8.0 of 10

    CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.

  5. xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

    cs.AI 2026-09 conditional novelty 7.0 of 10

    xDailyBench measures LLM performance on 248 authentic, open-ended everyday tasks with fine-grained rubrics and finds that implicit requirement inference is a major bottleneck across all frontier models.

  6. WorldBench: Culturally Grounded Benchmark for Multilingual Agents

    cs.AI 2026-09 accept novelty 7.0 of 10

    WorldBench provides 1,600 culturally grounded tasks with a metric that penalizes accidental changes to unrelated files, showing that frontier LLM agents remain brittle in multilingual long-horizon scenarios.

  7. A^2Agent: Action-Aware Reinforcement Learning for Repository-Level Code Localization Agents

    cs.CL 2026-08 accept novelty 7.0 of 10

    A2Agent improves repository-level code localization by providing per-turn rewards and action-level advantage estimation, outperforming trajectory-level methods on SWE-Bench Verified and Pro.

  8. RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A coding-agent benchmark built from sparse, casual variants of SWE-bench tasks shows that realistic requests reduce resolution rates and that Desired Behavior and Motivation are the highest-value information fields.

  9. PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents

    cs.AI 2026-08 conditional novelty 7.0 of 10

    PILOT couples live steering and live self-evolution in a supervisor-worker architecture to improve long-horizon agent performance mid-run and across runs.

  10. EVOMAL: Self-Poisoning in Self-Evolving Coding Agents

    cs.CR 2026-08 conditional novelty 7.0 of 10

    A self-evolving coding agent can be poisoned through its own tool-authoring step: reading a planted skill makes the agent author, store, and later run a malicious copy that persists even after the original skill is removed.

  11. DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents

    cs.AI 2026-08 accept novelty 7.0 of 10

    A controlled multi-session benchmark shows that later coding tasks can depend on earlier-session memory, and that a simple verbatim event-memory baseline is surprisingly strong, while the benchmark reliably discrimina...

  12. A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Coding agents lose up to 6.7 percentage points of issue-resolution rate on semantically equivalent code rewrites, and model robustness rankings flip across scaffolds.

  13. SEER: Long-Context Reasoning via Selective Visual-Text Compression

    cs.CL 2026-08 conditional novelty 7.0 of 10

    A vision-language model that selectively retrieves source text from query-relevant rendered pages improves long-context QA accuracy over uniform visual compression while keeping average prompt-token savings.

  14. Evaluating Agentic Code Repair Capabilities in Distributed Systems

    cs.SE 2026-08 conditional novelty 7.0 of 10

    DDBench, a 60-case distributed-debugging benchmark, spreads frontier LLMs that cluster on SWE-bench and shows bounded debugging context raises average pass rates from 32.6% to 50.6%.

  15. Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Harness-IF scores 256 rules across coding-agent runs and finds every model performs 3.6 to 7.4 points worse on rules that oppose unprompted defaults, so aggregate compliance scores overstate true instruction following.

  16. Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention

    cs.AR 2026-08 conditional novelty 7.0 of 10

    Heterogeneous serving that moves the KV cache and retrieval-based sparse attention to general-purpose processing-near-memory devices improves simulated decode throughput per TDP by 2.09-6.13x over a GPU-only baseline.

  17. LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

    cs.SE 2026-07 conditional novelty 7.0 of 10

    LoopsBench exposes coding agents to dependency-DAG tasks with regression obligations; the best configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks.

  18. Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

    cs.DC 2026-07 conditional novelty 7.0 of 10

    Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.

  19. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

    cs.SE 2026-07 conditional novelty 7.0 of 10

    PAIChecker detects PR-issue misalignment in SWE-bench-like benchmarks, which the authors find affects 13.6% of SWE-bench Verified instances and shifts 64.1% of leaderboard rankings.

  20. SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

    cs.SE 2026-07 conditional novelty 7.0 of 10

    SWE-NFI measures coding agents' non-functional code improvements with 92 deterministic rules and 188 real GitHub tasks, finding agents lag humans most on structural logic-pattern improvements.

  21. MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

    cs.SE 2026-07 accept novelty 7.0 of 10

    Distilling 1,001 whole-life-cycle source-free program-synthesis trajectories into Qwen3.6-27B raises ProgramBench pass rate from 37.98% to 49.51% and transfers to seven unseen SE benchmarks.

  22. SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

    cs.CR 2026-07 conditional novelty 7.0 of 10

    No frontier LLM agent fully detects and remediates any of 10 real post-compromise host ranges; alert-driven findings work, silent intrusion and verified cleanup do not.

  23. Agentic Evaluation of Copyright Law Compliance

    cs.CL 2026-07 conditional novelty 7.0 of 10

    AI agents given real-world image-selection jobs often pick copyrighted stock images even when public-domain alternatives are available, and open-weight models do worse under time pressure or when told to ignore licenses.

  24. ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

    cs.AI 2026-07 conditional novelty 7.0 of 10

    On a new 480-task, 12-language benchmark where agents must clarify vague product briefs and build repositories from scratch, the best model achieves only 38.2% overall pass rate.

  25. How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

    cs.SE 2026-07 conditional novelty 7.0 of 10

    SE-agent development follows a recurring seven-stage loop where evaluation drives iteration, and challenges such as unreliable evaluation signals and comprehension debt emerge.

  26. ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance

    cs.SE 2026-07 conditional novelty 7.0 of 10

    On sequential dependent bug-fix chains, coding agents lose up to ~70% resolution rate versus oracle prior state, mainly from overshoot and undershoot of earlier patches.

  27. SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    SWE-Interact shows frontier models solve roughly 25% of multi-turn interactive coding tasks versus 50% on single-turn baselines.

  28. MirrorCode: AI can rebuild entire programs from behavior alone

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    MirrorCode benchmark shows current AI models achieving up to 56% success reimplementing 25 diverse full programs from behavior alone, including a 16,000-line bioinformatics toolkit.

  29. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.

  30. Dockerless: Environment-Free Program Verifier for Coding Agents

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    Dockerless uses agentic repository exploration to verify patches without execution, enabling SFT and RL training of coding agents that reach 62.0/50.0/35.2% resolve rates on SWE-bench Verified/Multilingual/Pro while m...

  31. CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    CFAgentBench is a new reproducible benchmark for construction-finance AI agents featuring 35 mock apps, 1,014 tasks, and a money-movement guard, with initial tests showing pass^1 of 0.67 dropping to pass^5 of 0.38.

  32. StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good h...

  33. Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    MAFP applies fictitious play to LLM multi-agent systems to resolve stance entanglement in competitive decision-making, outperforming single-round and multi-round baselines on tournament strength and robustness.

  34. AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.

  35. Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

    cs.LG 2026-06 conditional novelty 7.0 of 10

    Claw-SWE-Bench is a 350-instance multilingual benchmark for OpenClaw-style agent harnesses that shows adapter design raises Pass@1 from 19.1% to 73.4% on the same model while releasing data for reproducible comparison.

  36. SWE-Explore: Benchmarking How Coding Agents Explore Repositories

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    SWE-Explore is a new benchmark evaluating repository exploration by coding agents on 848 issues across 203 repositories, using line-level ground truth from successful agent trajectories and showing agentic methods out...

  37. TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    TensorBench is a new benchmark of 199 tasks on a tensor framework used to evaluate seven coding agents, yielding pass rates from 22.1% to 64.8% with low inter-agent agreement.

  38. SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    SEC-bench Pro benchmark with 183 real vulnerabilities shows frontier LLM coding agents achieve at most 38.8% success on SpiderMonkey and 32% on V8.

  39. RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    RepoMirage uses semantics-preserving perturbations on SWE-Bench to show code agents lack repository context reasoning, with performance falling sharply on extended structure tasks, and introduces RepoAnchor as a struc...

  40. SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    SpecBench shows frontier coding agents saturate visible test suites but exhibit persistent reward hacking on held-out tests, with the gap growing 28 percentage points per tenfold increase in code size.

  41. SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    SaaSBench introduces a heterogeneous benchmark for enterprise SaaS engineering and shows that state-of-the-art coding agents fail over 95% of the time before reaching deep business logic due to setup and integration problems.

  42. Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    MemDocAgent generates consistent hierarchical repository-level code documentation by combining dependency-aware traversal with memory-guided agent interactions that accumulate work traces.

  43. Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

    cs.AI 2026-05 conditional novelty 7.0 of 10

    BenchJack audits 10 AI agent benchmarks, synthesizes exploits achieving near-perfect scores without task completion, surfaces 219 flaws, and reduces hackable-task ratios to under 10% on four benchmarks via iterative patching.

  44. LLM Agents Already Know When to Call Tools -- Even Without Reasoning

    cs.CL 2026-05 conditional novelty 7.0 of 10

    LLMs encode tool necessity in pre-generation hidden states at AUROC 0.89-0.96, enabling Probe&Prefill to reduce tool calls 48% with 1.7% accuracy loss, outperforming prompt and reasoning baselines.

  45. Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures

    cs.SE 2026-04 accept novelty 7.0 of 10

    Analysis of 13 coding agent scaffolds at pinned commits yields a 12-dimension taxonomy showing five composable loop primitives, with 11 agents combining multiple primitives instead of using one fixed structure.

  46. AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    AgentHazard benchmark shows computer-use agents remain highly vulnerable, with attack success rates reaching 73.63% on models like Qwen3-Coder powering Claude Code.

  47. Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure

    cs.SE 2026-04 accept novelty 7.0 of 10

    Large-scale trajectory analysis of 19 coding agents on 500 tasks finds that LLM choice drives outcomes more than framework design and that context-gathering plus validation behaviors improve success beyond task diffic...

  48. SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution

    cs.SE 2026-03 conditional novelty 7.0 of 10

    Frontier coding agents drop from >80% on isolated milestones to ≤38% under continuous repository evolution because regressions accumulate faster than they can be fixed.

  49. Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development

    cs.SE 2026-03 unverdicted novelty 7.0 of 10

    Vibe Code Bench evaluates AI models on building complete web applications from specs, with the best of 16 models achieving 61.8% accuracy on the test split using autonomous browser evaluation.

  50. Debug2Fix: Can Interactive Debugging Help Coding Agents Fix More Bugs?

    cs.SE 2026-02 conditional novelty 7.0 of 10

    Debug2Fix integrates interactive debugging via subagents into coding agents, delivering >20% gains on GitBug-Java and SWE-Bench-Live while enabling weaker models to match stronger ones.

  51. Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation

    cs.SE 2026-02 unverdicted novelty 7.0 of 10

    Agent-Diff benchmarks LLM agents on enterprise API tasks using code execution and state-diff contracts to define success, evaluated on nine models across 224 tasks with code released.

  52. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

    cs.SE 2025-12 unverdicted novelty 7.0 of 10

    SWE-EVO shows GPT-5.4 with OpenHands reaching only 25% success on complex multi-file evolution tasks versus 72.8% on SWE-Bench Verified, and introduces Fix Rate as a partial-progress metric.

  53. When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Code-only tool surfaces beat or match the cheapest tool-rich rival on cost in three of four (regime, agent) cells at matched pass rates; Claude on SWE-bench is the directional exception via failure cost.

  54. When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI Systems

    cs.AI 2026-09 conditional novelty 6.0 of 10

    A conceptual and formal framework that ties an AI agent's decisions about whether, when, and how to act proactively to a standing revocable mandate, with a taxonomy of autonomous, delegated, and symbiotic agency.

  55. HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

    cs.SE 2026-09 conditional novelty 6.0 of 10

    Frontier LLMs, from a nearly empty template, built agent harnesses that matched or beat selected human references in writing and ML experiments but lagged on code and research, and their self-improvement gains were un...

  56. ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents

    cs.AI 2026-09 conditional novelty 6.0 of 10

    ContextPipe recasts LLM agent context assembly as database query execution (Plan, Bind, Optimize, Execute, Feedback) and reports, on 3 of 79 SWE-bench Pro instances, roughly 30% fewer prompt tokens and 17-23% fewer mo...

  57. SkillShield: Prompt-Space Security Skills for LLM Coding Agents

    cs.CR 2026-08 conditional novelty 6.0 of 10

    Security skills synthesized from attack data and placed in the system prompt reduce harmful execution and malware generation for coding agents, with a measured cost in benign refusals that depends on the skill variant.

  58. AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

    cs.AI 2026-08 conditional novelty 6.0 of 10

    AutoSaddler's diagnose, patch, reflect, and evolve loop yields Pass@1 gains of 8.4 to 10.0 percentage points on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0.

  59. BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERP

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A benchmark of 101 real-world AL tasks shows model choice matters more than agent harness in bug fixing, and general-purpose coding gains do not always transfer to ERP languages.

  60. Terminal Agents: A Survey of AI Agents in Command-Line Environments

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Terminal agents are defined by a command-execution action-observation loop, analyzed via seven competence dimensions, with diagnostics showing benchmark-dependent process exposure and attribution limits.

See all 155 Pith citations

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · cited by 155 Pith papers (see all)

  1. [1]

    SWE-Bench+: Enhanced Coding Benchmark for LLMs

    R. Aleithan et al. Swe-bench+: Enhanced coding benchmark for llms.arXiv preprint arXiv:2410.06992, 2024

    work page Pith review arXiv 2024

  2. [2]

    Complexity-Constraint Code Evaluation: A Benchmark for Time Complexity Compliance in LLM-Generated Code

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    work page Pith review arXiv 2021

  3. [3]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    work page 1901

  4. [4]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    work page Pith review arXiv 2021

  5. [5]

    A Survey on Data Contamination for Large Language Models

    Y. Cheng, Z. Li, and Y. Zhou. A survey on data contamination for large language models.arXiv preprint arXiv:2502.14425, 2025

    work page Pith review arXiv 2025

  6. [6]

    J. Da, C. J. Wang, X. Deng, Y. Ma, N. Barhate, and S. M. Hendryx. Agent-rlvr: Training software en- gineering agents via guidance and environment rewards.ArXiv, abs/2506.11425, 2025. URL https: //api.semanticscholar.org/CorpusID:279391657

    work page Pith review arXiv 2025

  7. [7]

    C. Deng, Y. Zhao, X. Tang, M. Gerstein, and A. Cohan. Investigating data contamination in modern bench- marks for large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T echnologies (Volume 1: Long Papers), pages 8706–8719, Mexico City, Mexico, 2024. Assoc...

    work page 2024

  8. [8]

    A. E. Hassan. Predicting faults using the complexity of code changes. In2009 IEEE 31st International Conference on Software Engineering, pages 78–88. IEEE, 2009

    work page 2009

Show all 24 references
  1. [9]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps. InNeural Information Processing Systems, 2021

    2021

  2. [10]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    2024

  3. [11]

    URLhttps://openai.com/index/introducing-swe-bench-verified/

    OpenAI, 2024. URLhttps://openai.com/index/introducing-swe-bench-verified/

    2024

  4. [12]

    Steidl, B

    D. Steidl, B. Hummel, and E. Jürgens. Evaluating code complexity triggers, use of complexity measures and the influence of code complexity on maintenance time.Empirical Software Engineering, 22(2):971–1015, 2017

    2017

  5. [13]

    White, S

    C. White, S. Dooley, ManleyRoberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, M. Goldblum, Abacus.AI, Nyu, and Nvidia. Livebench: A challenging, contamination-free llm benchmark.ArXiv, abs/2406....

    2024

  6. [14]

    C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  7. [15]

    C. Xu, J. Guan, X. Zhao, C. Fu, Q. Xin, Z. Wang, L. Li, J. Fu, H. Wang, and J. Liu. Benchmark data contamination of large language models: A survey.arXiv preprint arXiv:2406.04244, 2024

    2024

  8. [16]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. InNeural Information Processing Systems, 2024

    2024

  9. [17]

    J. Yang, C. E. Jimenez, A. Wettig, K. Narasimhan, and O. Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?arXiv preprint arXiv:2410.03859, 2024

    2024

  10. [18]

    D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2404.02605, 2024

    2024

  11. [19]

    Zhang et al

    C. Zhang et al. Swe-bench goes live!arXiv preprint arXiv:2505.23419, 2025. 11

    2025

  12. [20]

    Zhang, J

    H. Zhang, J. Da, D. Lee, V . Robinson, C. Wu, W. Song, T. Zhao, P . Raja, D. Slack, Q. Lyu, S. M. Hendryx, R. Kaplan, M. Lunati, and S. Yue. A careful examination of large language model performance on grade school arithmetic.ArXiv, abs/2405.00332, 2024. URLhttps://api.semanti...

    2024

  13. [21]

    Book 978

    Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury. Autocoderover: Autonomous program improvement. In ACM SIGSOFT International Symposium on Software T esting and Analysis, 2024. 12 MODELRESOLVE(%) OPENAI GPT-5 (HIGH) 25.9 OPENAI GPT-5 (MEDIUM) 23.3 CLAUDEOPUS4.1 22.7 CLAUDESONNET...

    2024

  14. [22]

    iOS Contacts)

    Export a contact as a vCard 4.0 file from a standards-compliant source (e.g. iOS Contacts)

  15. [23]

    In the application UI, chooseImport contacts and select the.vcffile

  16. [24]

    No actions recorded

    Observe that no contact is created or that the importer reports an error. Expected Behaviour: • The importer should recognise theVERSION:4.0 header and process the file. • Standard fields present in earlier versions (FN, N, TEL, EMAIL, ADR, NOTE, etc.) must be mapped to the in...

Pith tools

Reviewed May 12, 2026 · model on record in the stance chip above.