KVMem virtualizes KV cache across GPU, host, and NVMe to support agent workspaces up to 1M tokens on a consumer laptop, outperforming compaction-based context management on long-context agent benchmarks.
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
abstract
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
citation-role summary
background 1
dataset 1
citation-polarity summary
years
2026 5representative citing papers
ContextWeave turns real multi-month human work histories into executable agent tasks and shows that memory of prior tasks improves downstream task outcomes across all five tested base models.
The paper organizes recursive self-improvement into an L1-L5 autonomy hierarchy, adds a Headroom-Closed Index to measure capability gaps, and reviews academic plus industrial systems against that ladder.
A multilingual refactoring benchmark with 170 large-scale tasks defeats frontier coding agents, whose best resolve rate is 41.2%.
VibeLifeBench is a 200-task benchmark for long-horizon, proactive life agents in a simulated living world; the strongest frontier model reaches only 32.5 average score.
citing papers explorer
-
KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
KVMem virtualizes KV cache across GPU, host, and NVMe to support agent workspaces up to 1M tokens on a consumer laptop, outperforming compaction-based context management on long-context agent benchmarks.
-
ContextWeave: A Real-World Workflow Benchmark
ContextWeave turns real multi-month human work histories into executable agent tasks and shows that memory of prior tasks improves downstream task outcomes across all five tested base models.
-
The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement
The paper organizes recursive self-improvement into an L1-L5 autonomy hierarchy, adds a Headroom-Closed Index to measure capability gaps, and reviews academic plus industrial systems against that ladder.
-
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
A multilingual refactoring benchmark with 170 large-scale tasks defeats frontier coding agents, whose best resolve rate is 41.2%.
-
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
VibeLifeBench is a 200-task benchmark for long-horizon, proactive life agents in a simulated living world; the strongest frontier model reaches only 32.5 average score.