MetaRSI-v1 composes Data-RSI, Harness-RSI, and Model-RSI operators under a meta-scheduler that jointly optimizes operator order and policies, improving an open-weight model by 10.9 average points without an external teacher.
super hub Mixed citations
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mixed citation behavior. Most common role is background (65%).
187
Pith papers citing it
Background
65% of classified citations
abstract
AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .
hub tools
citation-role summary
background 21
dataset 7
baseline 2
other 1
citation-polarity summary
claims ledger
- abstract AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the bench
authors
co-cited works
years
2026 187representative citing papers
A framework that converts recorded agent trajectories into reusable terminal environments with auto-generated tasks, boosting agent benchmark performance through supervised fine-tuning.
Matched execution scores can hide the difference between command-generation errors and transport damage, as QuoteBench's fixed-reply replay across eight model configurations demonstrates.
RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substitution and answer leakage that change how agent benchmarks must be audited.
CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
A user study with over 100 participants shows humans rarely spot AI agents sabotaging code during extended collaborative tasks, even with a safety monitor present.
The Meta-Agent Challenge shows frontier AI models rarely match human-engineered agent baselines when tasked with autonomous development, with proprietary models succeeding most often and some exhibiting cheating under pressure.
HalluWorld is a controlled benchmark using explicit reference world models to automatically label and disentangle hallucinations in LLMs across synthetic environments with varying complexity and observability.
A new native-runtime benchmark reveals that current frontier AI agents succeed on at most 62 percent of realistic long-horizon CLI tasks.
Agent-ValueBench is the first dedicated benchmark for agent values, showing they diverge from LLM values, form a homogeneous 'Value Tide' across models, and bend under harnesses and skill steering.
HWE-Bench is the first repository-level benchmark for LLM agents on real hardware bug repair, where the best agent fixes 70.7% of 417 tasks but drops below 65% on complex SoC projects.
OccuBench is a new benchmark for AI agents on real-world occupational tasks via LLM-driven simulators, showing no model dominates all industries, implicit faults are hardest, and larger models with more reasoning perform better.
SlopCodeBench shows coding agents degrade in structural quality and verbosity across iterative extensions, with no agent solving any problem completely and agent code 2x more eroded than human code.
SQD disaggreagtes LLm inference so that subquadratic attention and the FFN run on SRAM-only accelerators while quadratic attention colocates with prefill on GPUs, improving tokens/J by 31-56% over GPU-only baselines.
ZipBench is a low-cost benchmark compression method that uses six anchor LLMs and synthetic pseudo-logs to build faithful compact benchmarks across over 100 text, multimodal, and agent tasks.
SemSpot connects agent task semantics with transient MaaS capacity through task-conditioned spot offers and explicit recovery continuations.
HarnessLens evolves LLM agent harnesses by selecting behavior-relevant verification tasks and gating acceptance on attributable evidence, improving held-out performance by 7.6-13.6% with lower budget.
SWE Refactor Bench grades coding agents on 20 whole-repository stack migrations with a three-stage protocol, and finds 5.4% of 520 runs passed all stages, with 13 of 20 tasks unsolved.
Agentic workloads are heavyweight, heterogeneous, and shift bottlenecks in ways conventional LLM serving assumptions miss; characterization-guided optimizations cut latency by up to 40%, memory by 4.6x, and redundant search calls by 35%.
Harness-IF scores 256 rules across coding-agent runs and finds every model performs 3.6 to 7.4 points worse on rules that oppose unprompted defaults, so aggregate compliance scores overstate true instruction following.
The workbook time machine reverse-engineers real workbooks into creation tasks, and the resulting benchmark shows pivot tables remain nearly unsolved by all tested frontier models.
AOSpec co-speculates actions and observations in LLM agents, using expected-value decoding and joint action-state verification to hide tool execution latency, achieving 11.8-32.5% end-to-end latency savings in trace replay.
On 108 long-horizon real-world computer workflows, frontier agents complete at most 20.6% of tasks and fail mainly by losing hidden state, not by basic GUI control.
LLM agents often fail to abstain at the right time in uncertain multi-turn tasks, and the CONVOLVE context engineering method raises timely abstention rates on WebShop from 26.7 to 57.4 without parameter updates.
citing papers explorer
-
MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves
MetaRSI-v1 composes Data-RSI, Harness-RSI, and Model-RSI operators under a meta-scheduler that jointly optimizes operator order and policies, improving an open-weight model by 10.9 average points without an external teacher.
-
Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments
A framework that converts recorded agent trajectories into reusable terminal environments with auto-generated tasks, boosting agent benchmark performance through supervised fine-tuning.
-
QuoteBench: How Matched Scores Can Hide Command-Path Failures
Matched execution scores can hide the difference between command-generation errors and transport damage, as QuoteBench's fixed-reply replay across eight model configurations demonstrates.
-
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substitution and answer leakage that change how agent benchmarks must be audited.
-
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
-
Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?
A user study with over 100 participants shows humans rarely spot AI agents sabotaging code during extended collaborative tasks, even with a safety monitor present.
-
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?
The Meta-Agent Challenge shows frontier AI models rarely match human-engineered agent baselines when tasked with autonomous development, with proprietary models succeeding most often and some exhibiting cheating under pressure.
-
HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models
HalluWorld is a controlled benchmark using explicit reference world models to automatically label and disentangle hallucinations in LLMs across synthetic environments with varying complexity and observability.
-
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
A new native-runtime benchmark reveals that current frontier AI agents succeed on at most 62 percent of realistic long-horizon CLI tasks.
-
Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values
Agent-ValueBench is the first dedicated benchmark for agent values, showing they diverge from LLM values, form a homogeneous 'Value Tide' across models, and bend under harnesses and skill steering.
-
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
HWE-Bench is the first repository-level benchmark for LLM agents on real hardware bug repair, where the best agent fixes 70.7% of 417 tasks but drops below 65% on complex SoC projects.
-
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation
OccuBench is a new benchmark for AI agents on real-world occupational tasks via LLM-driven simulators, showing no model dominates all industries, implicit faults are hardest, and larger models with more reasoning perform better.
-
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
SlopCodeBench shows coding agents degrade in structural quality and verbosity across iterative extensions, with no agent solving any problem completely and agent code 2x more eroded than human code.
-
Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
SQD disaggreagtes LLm inference so that subquadratic attention and the FFN run on SRAM-only accelerators while quadratic attention colocates with prefill on GPUs, improving tokens/J by 31-56% over GPU-only baselines.
-
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
ZipBench is a low-cost benchmark compression method that uses six anchor LLMs and synthetic pseudo-logs to build faithful compact benchmarks across over 100 text, multimodal, and agent tasks.
-
Bridging Agent Semantics with Spot Capacity: An Elastic and Recoverable Service Model
SemSpot connects agent task semantics with transient MaaS capacity through task-conditioned spot offers and explicit recovery continuations.
-
Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification
HarnessLens evolves LLM agent harnesses by selecting behavior-relevant verification tasks and gating acceptance on attributable evidence, improving held-out performance by 7.6-13.6% with lower budget.
-
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
SWE Refactor Bench grades coding agents on 20 whole-repository stack migrations with a three-stage protocol, and finds 5.4% of 520 runs passed all stages, with 13 of 20 tasks unsolved.
-
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
Agentic workloads are heavyweight, heterogeneous, and shift bottlenecks in ways conventional LLM serving assumptions miss; characterization-guided optimizations cut latency by up to 40%, memory by 4.6x, and redundant search calls by 35%.
-
Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
Harness-IF scores 256 rules across coding-agent runs and finds every model performs 3.6 to 7.4 points worse on rules that oppose unprompted defaults, so aggregate compliance scores overstate true instruction following.
-
Back to the Future: A workbook time machine for spread sheet creation benchmarks
The workbook time machine reverse-engineers real workbooks into creation tasks, and the resulting benchmark shows pivot tables remain nearly unsolved by all tested frontier models.
-
AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving
AOSpec co-speculates actions and observations in LLM agents, using expected-value decoding and joint action-state verification to hide tool execution latency, achieving 11.8-32.5% end-to-end latency savings in trace replay.
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
On 108 long-horizon real-world computer workflows, frontier agents complete at most 20.6% of tasks and fail mainly by losing hidden state, not by basic GUI control.
-
Agentic Abstention: Do Agents Know When to Stop Instead of Act?
LLM agents often fail to abstain at the right time in uncertain multi-turn tasks, and the CONVOLVE context engineering method raises timely abstention rates on WebShop from 26.7 to 57.4 without parameter updates.
-
CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?
CyberChainBench is a new benchmark evaluating LLM agents on vulnerability detection, exploit generation, and patch synthesis using 541 real-world DeFi incidents with on-chain historical evaluation, showing peak performance of 43.7% on exploitation.
-
GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
A matched benchmark shows GUI computer-use agents at 59.1% full pass rate versus 48.2% for original-skill CLI agents, rising to 69.3% with verifier-guided augmentation, indicating modality-specific execution bottlenecks.
-
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
CLI-Universe synthesizes a verified 6K dataset of terminal-agent tasks that, when used to fine-tune Qwen3-32B, reaches 33.4% on Terminal-Bench 2.0 and sets a new open-source SOTA for models at or below 32B parameters.
-
StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns
StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good harnesses.
-
Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play
MAFP applies fictitious play to LLM multi-agent systems to resolve stance entanglement in competitive decision-making, outperforming single-round and multi-round baselines on tournament strength and robustness.
-
One Goal, Many Commands: Characterizing Denylist Fragility in AI Agents
ShellSieve, an LLM-driven pipeline, detects command denylist fragility in terminal AI agents and finds 69.0-98.6% of 1,709 GitHub-collected denylists to be bypassable.
-
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Introduces the first community-governed unified JSON schema and crowdsourced repository for AI evaluation results, with converters and a database spanning 22,235 models and 2,273 benchmarks.
-
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.
-
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
SciAgentArena is a new interactive benchmark for AI agents on scientific tasks that finds agents handle clear data-analysis workflows but struggle with novel insights, self-directed exploration, and open-ended questions.
-
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Workflow-GYM is a new benchmark for long-horizon professional GUI agent tasks where state-of-the-art models reach only slightly above 30% success.
-
ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer
ADK Arena evaluates 51 Python ADKs by having an LLM learn each framework's API, write and repair agent code, and run on benchmarks, finding 57% success rate, 5.6x cost variation, no dominant framework, and substitutable information sources.
-
AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
AutoMedBench evaluates AI agents on long-horizon medical workflows across five stages and finds validation and submission as dominant failure points based on thousands of runs.
-
Stateful Online Monitoring Catches Distributed Agent Attacks
A clustering-based stateful online monitor detects distributed multi-agent cyberattacks that evade standard per-transcript monitors, catching them 30% earlier in large-scale simulated traffic with low overhead.
-
SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills
SkillEvolBench is a new diagnostic benchmark that evaluates the transition from episodic experience to procedural skills in LLM agents using role-conditioned task families and frozen deployment tests.
-
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
EvoCode-Bench shows that single-round success rates for coding agents exceed multi-turn persistent execution rates by 22-40 points, with performance dropping below half of round-1 levels by round 5 across 13 evaluated agents.
-
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.
-
Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
Terminal-World is a skill-based synthesis pipeline that generates 5,723 training environments and produces Terminal-World-32B which outperforms baselines on Terminal-Bench 2.0 using only 1.2% of the data.
-
WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games
WebGameBench is a new benchmark that evaluates coding agents on building browser-native games from frozen specifications, with runtime browser evaluation showing best agents reach 76.9% usable rate but only 20.2% excellent rate.
-
Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench
ConsumerSimBench evaluates 13 LLMs on reconstructing crowd reactions from 1,553 Chinese social-media topics using 23,122 auditable yes-no criteria, finding maximum coverage of 47.8% by Gemini-3.1-Pro.
-
Do Coding Agents Understand Least-Privilege Authorization?
Coding agents struggle to infer least-privilege file permissions by omitting needed accesses while granting unused or sensitive ones, but Sufficiency-Tightness Decomposition improves sensitive-task success by up to 15.8% and reduces attacks.
-
ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
ClawForge is a generator framework that creates reproducible executable benchmarks for command-line agents under state conflict, with ClawForge-Bench showing frontier models reach at most 45.3% strict accuracy and that state inspection drives most performance gaps.
-
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
Collider-Bench is a new benchmark showing that current LLM agents cannot reliably reproduce LHC analyses at the level of a physicist-in-the-loop.
-
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
AgentLens reveals 10.7% of passing SWE-agent trajectories exhibit Lucky Pass behaviors and introduces a process-level evaluation framework with a new annotated dataset of 1,815 trajectories.
-
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
BenchJack audits 10 AI agent benchmarks, synthesizes exploits achieving near-perfect scores without task completion, surfaces 219 flaws, and reduces hackable-task ratios to under 10% on four benchmarks via iterative patching.
-
LLM Agents Already Know When to Call Tools -- Even Without Reasoning
LLM agents encode tool necessity in pre-generation hidden states with high linear decodability (AUROC 0.89-0.96); Probe&Prefill uses this to reduce tool calls 48% with 1.7% accuracy loss.
-
MDGYM: Benchmarking AI Agents on Molecular Simulations
MDGYM benchmark shows AI agents achieve low success rates on molecular dynamics tasks, with distinct failure modes including unstable simulations and premature abandonment.