MetaRSI-v1 composes Data-RSI, Harness-RSI, and Model-RSI operators under a meta-scheduler that jointly optimizes operator order and policies, improving an open-weight model by 10.9 average points without an external teacher.
super hub Canonical reference
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Canonical reference. 71% of citing Pith papers cite this work as background.
139
Pith papers citing it
2
external citations · Pith
Background
71% of classified citations
abstract
We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.
hub tools
citation-role summary
background 16
dataset 7
baseline 1
citation-polarity summary
representative citing papers
RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substitution and answer leakage that change how agent benchmarks must be audited.
CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
PDEAgent-Bench is the first multi-metric, multi-library benchmark for AI-generated PDE solvers, evaluating executability, numerical accuracy, and efficiency across DOLFINx, Firedrake, and deal.II.
HWE-Bench is the first repository-level benchmark for LLM agents on real hardware bug repair, where the best agent fixes 70.7% of 417 tasks but drops below 65% on complex SoC projects.
SQD disaggreagtes LLm inference so that subquadratic attention and the FFN run on SRAM-only accelerators while quadratic attention colocates with prefill on GPUs, improving tokens/J by 31-56% over GPU-only baselines.
ZipBench is a low-cost benchmark compression method that uses six anchor LLMs and synthetic pseudo-logs to build faithful compact benchmarks across over 100 text, multimodal, and agent tasks.
xDailyBench measures LLM performance on 248 authentic, open-ended everyday tasks with fine-grained rubrics and finds that implicit requirement inference is a major bottleneck across all frontier models.
WorldBench provides 1,600 culturally grounded tasks with a metric that penalizes accidental changes to unrelated files, showing that frontier LLM agents remain brittle in multilingual long-horizon scenarios.
A2Agent improves repository-level code localization by providing per-turn rewards and action-level advantage estimation, outperforming trajectory-level methods on SWE-Bench Verified and Pro.
A coding-agent benchmark built from sparse, casual variants of SWE-bench tasks shows that realistic requests reduce resolution rates and that Desired Behavior and Motivation are the highest-value information fields.
PILOT couples live steering and live self-evolution in a supervisor-worker architecture to improve long-horizon agent performance mid-run and across runs.
A self-evolving coding agent can be poisoned through its own tool-authoring step: reading a planted skill makes the agent author, store, and later run a malicious copy that persists even after the original skill is removed.
A controlled multi-session benchmark shows that later coding tasks can depend on earlier-session memory, and that a simple verbatim event-memory baseline is surprisingly strong, while the benchmark reliably discriminates memory-bearing conditions from no memory.
Coding agents lose up to 6.7 percentage points of issue-resolution rate on semantically equivalent code rewrites, and model robustness rankings flip across scaffolds.
A vision-language model that selectively retrieves source text from query-relevant rendered pages improves long-context QA accuracy over uniform visual compression while keeping average prompt-token savings.
DDBench, a 60-case distributed-debugging benchmark, spreads frontier LLMs that cluster on SWE-bench and shows bounded debugging context raises average pass rates from 32.6% to 50.6%.
Harness-IF scores 256 rules across coding-agent runs and finds every model performs 3.6 to 7.4 points worse on rules that oppose unprompted defaults, so aggregate compliance scores overstate true instruction following.
Heterogeneous serving that moves the KV cache and retrieval-based sparse attention to general-purpose processing-near-memory devices improves simulated decode throughput per TDP by 2.09-6.13x over a GPU-only baseline.
SWE-Interact shows frontier models solve roughly 25% of multi-turn interactive coding tasks versus 50% on single-turn baselines.
On 108 long-horizon real-world computer workflows, frontier agents complete at most 20.6% of tasks and fail mainly by losing hidden state, not by basic GUI control.
Dockerless uses agentic repository exploration to verify patches without execution, enabling SFT and RL training of coding agents that reach 62.0/50.0/35.2% resolve rates on SWE-bench Verified/Multilingual/Pro while matching environment-based results.
CFAgentBench is a new reproducible benchmark for construction-finance AI agents featuring 35 mock apps, 1,014 tasks, and a money-movement guard, with initial tests showing pass^1 of 0.67 dropping to pass^5 of 0.38.
StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good harnesses.
citing papers explorer
-
MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves
MetaRSI-v1 composes Data-RSI, Harness-RSI, and Model-RSI operators under a meta-scheduler that jointly optimizes operator order and policies, improving an open-weight model by 10.9 average points without an external teacher.
-
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substitution and answer leakage that change how agent benchmarks must be audited.
-
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
-
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation
PDEAgent-Bench is the first multi-metric, multi-library benchmark for AI-generated PDE solvers, evaluating executability, numerical accuracy, and efficiency across DOLFINx, Firedrake, and deal.II.
-
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
HWE-Bench is the first repository-level benchmark for LLM agents on real hardware bug repair, where the best agent fixes 70.7% of 417 tasks but drops below 65% on complex SoC projects.
-
Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
SQD disaggreagtes LLm inference so that subquadratic attention and the FFN run on SRAM-only accelerators while quadratic attention colocates with prefill on GPUs, improving tokens/J by 31-56% over GPU-only baselines.
-
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
ZipBench is a low-cost benchmark compression method that uses six anchor LLMs and synthetic pseudo-logs to build faithful compact benchmarks across over 100 text, multimodal, and agent tasks.
-
xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
xDailyBench measures LLM performance on 248 authentic, open-ended everyday tasks with fine-grained rubrics and finds that implicit requirement inference is a major bottleneck across all frontier models.
-
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
WorldBench provides 1,600 culturally grounded tasks with a metric that penalizes accidental changes to unrelated files, showing that frontier LLM agents remain brittle in multilingual long-horizon scenarios.
-
A^2Agent: Action-Aware Reinforcement Learning for Repository-Level Code Localization Agents
A2Agent improves repository-level code localization by providing per-turn rewards and action-level advantage estimation, outperforming trajectory-level methods on SWE-Bench Verified and Pro.
-
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
A coding-agent benchmark built from sparse, casual variants of SWE-bench tasks shows that realistic requests reduce resolution rates and that Desired Behavior and Motivation are the highest-value information fields.
-
PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents
PILOT couples live steering and live self-evolution in a supervisor-worker architecture to improve long-horizon agent performance mid-run and across runs.
-
EVOMAL: Self-Poisoning in Self-Evolving Coding Agents
A self-evolving coding agent can be poisoned through its own tool-authoring step: reading a planted skill makes the agent author, store, and later run a malicious copy that persists even after the original skill is removed.
-
DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents
A controlled multi-session benchmark shows that later coding tasks can depend on earlier-session memory, and that a simple verbatim event-memory baseline is surprisingly strong, while the benchmark reliably discriminates memory-bearing conditions from no memory.
-
A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
Coding agents lose up to 6.7 percentage points of issue-resolution rate on semantically equivalent code rewrites, and model robustness rankings flip across scaffolds.
-
SEER: Long-Context Reasoning via Selective Visual-Text Compression
A vision-language model that selectively retrieves source text from query-relevant rendered pages improves long-context QA accuracy over uniform visual compression while keeping average prompt-token savings.
-
Evaluating Agentic Code Repair Capabilities in Distributed Systems
DDBench, a 60-case distributed-debugging benchmark, spreads frontier LLMs that cluster on SWE-bench and shows bounded debugging context raises average pass rates from 32.6% to 50.6%.
-
Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
Harness-IF scores 256 rules across coding-agent runs and finds every model performs 3.6 to 7.4 points worse on rules that oppose unprompted defaults, so aggregate compliance scores overstate true instruction following.
-
Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
Heterogeneous serving that moves the KV cache and retrieval-based sparse attention to general-purpose processing-near-memory devices improves simulated decode throughput per TDP by 2.09-6.13x over a GPU-only baseline.
-
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
SWE-Interact shows frontier models solve roughly 25% of multi-turn interactive coding tasks versus 50% on single-turn baselines.
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
On 108 long-horizon real-world computer workflows, frontier agents complete at most 20.6% of tasks and fail mainly by losing hidden state, not by basic GUI control.
-
Dockerless: Environment-Free Program Verifier for Coding Agents
Dockerless uses agentic repository exploration to verify patches without execution, enabling SFT and RL training of coding agents that reach 62.0/50.0/35.2% resolve rates on SWE-bench Verified/Multilingual/Pro while matching environment-based results.
-
CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents
CFAgentBench is a new reproducible benchmark for construction-finance AI agents featuring 35 mock apps, 1,014 tasks, and a money-movement guard, with initial tests showing pass^1 of 0.67 dropping to pass^5 of 0.38.
-
StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns
StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good harnesses.
-
Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play
MAFP applies fictitious play to LLM multi-agent systems to resolve stance entanglement in competitive decision-making, outperforming single-round and multi-round baselines on tournament strength and robustness.
-
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.
-
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
Claw-SWE-Bench is a 350-instance multilingual benchmark for OpenClaw-style agent harnesses that shows adapter design raises Pass@1 from 19.1% to 73.4% on the same model while releasing data for reproducible comparison.
-
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
SWE-Explore is a new benchmark evaluating repository exploration by coding agents on 848 issues across 203 repositories, using line-level ground truth from successful agent trajectories and showing agentic methods outperform classical retrieval on coverage and ranking.
-
TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
TensorBench is a new benchmark of 199 tasks on a tensor framework used to evaluate seven coding agents, yielding pass rates from 22.1% to 64.8% with low inter-agent agreement.
-
RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
RepoMirage uses semantics-preserving perturbations on SWE-Bench to show code agents lack repository context reasoning, with performance falling sharply on extended structure tasks, and introduces RepoAnchor as a structure-first fix.
-
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
SaaSBench introduces a heterogeneous benchmark for enterprise SaaS engineering and shows that state-of-the-art coding agents fail over 95% of the time before reaching deep business logic due to setup and integration problems.
-
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
BenchJack audits 10 AI agent benchmarks, synthesizes exploits achieving near-perfect scores without task completion, surfaces 219 flaws, and reduces hackable-task ratios to under 10% on four benchmarks via iterative patching.
-
LLM Agents Already Know When to Call Tools -- Even Without Reasoning
LLM agents encode tool necessity in pre-generation hidden states with high linear decodability (AUROC 0.89-0.96); Probe&Prefill uses this to reduce tool calls 48% with 1.7% accuracy loss.
-
Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
Goal clarifications lose nearly all value after 10% of execution while input clarifications retain value until roughly 50%, and asking any type past mid-trajectory hurts performance more than never asking.
-
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
LLM agents exhibit constraint decay with assertion pass rates dropping substantially as structural requirements increase in multi-file backend code generation across web frameworks.
-
Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution
TEBench is a new project-level benchmark for test evolution showing coding agents achieve only 45-49% F1 on identifying tests needing changes, with stale tests hardest due to reliance on execution failures.
-
ProgramBench: Can Language Models Rebuild Programs From Scratch?
ProgramBench introduces 200 tasks where models must reconstruct full programs like FFmpeg or SQLite from docs alone; none of 9 evaluated LMs fully solve any task and the best passes 95% tests on only 3% of tasks while favoring monolithic code.
-
SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
SkillFlow benchmark shows lifelong skill evolution yields modest gains for some models like Claude Opus 4.6 but limited or negative utility for others despite high skill usage.
-
Problem Reductions at Scale: Agentic Integration of Computationally Hard Problems
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
-
HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?
HiL-Bench shows frontier AI agents fail to ask for help on incomplete tasks, recovering only a fraction of full-information performance, but RL training on Ask-F1 reward improves judgment and transfers across domains.
-
Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures
Analysis of 13 coding agent scaffolds at pinned commits yields a 12-dimension taxonomy showing five composable loop primitives, with 11 agents combining multiple primitives instead of using one fixed structure.
-
AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
AgentHazard benchmark shows computer-use agents remain highly vulnerable, with attack success rates reaching 73.63% on models like Qwen3-Coder powering Claude Code.
-
Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
Large-scale trajectory analysis of 19 coding agents on 500 tasks finds that LLM choice drives outcomes more than framework design and that context-gathering plus validation behaviors improve success beyond task difficulty predictions.
-
Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development
Vibe Code Bench evaluates AI models on building complete web applications from specs, with the best of 16 models achieving 61.8% accuracy on the test split using autonomous browser evaluation.
-
Debug2Fix: Can Interactive Debugging Help Coding Agents Fix More Bugs?
Debug2Fix integrates interactive debugging via subagents into coding agents, delivering >20% gains on GitBug-Java and SWE-Bench-Live while enabling weaker models to match stronger ones.
-
Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
Agent-Diff benchmarks LLM agents on enterprise API tasks using code execution and state-diff contracts to define success, evaluated on nine models across 224 tasks with code released.
-
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
SWE-EVO shows GPT-5.4 with OpenHands reaching only 25% success on complex multi-file evolution tasks versus 72.8% on SWE-Bench Verified, and introduces Fix Rate as a partial-progress metric.
-
When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI Systems
A conceptual and formal framework that ties an AI agent's decisions about whether, when, and how to act proactively to a standing revocable mandate, with a taxonomy of autonomous, delegated, and symbiotic agency.
-
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Frontier LLMs, from a nearly empty template, built agent harnesses that matched or beat selected human references in writing and ML experiments but lagged on code and research, and their self-improvement gains were unstable and executor-specific.
-
ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents
ContextPipe recasts LLM agent context assembly as database query execution (Plan, Bind, Optimize, Execute, Feedback) and reports, on 3 of 79 SWE-bench Pro instances, roughly 30% fewer prompt tokens and 17-23% fewer model calls versus append-only assembly, with fresh (uncached) tokens roughly…