Pith. sign in

REVIEW 3 major objections 3 minor 204 cited by

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

T0 review · 3 major / 3 minor · reviewed 2026-05-11 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark of 89 realistic terminal tasks shows frontier AI agents complete fewer than 65 percent of them.

desk verdict Terminal-Bench ships a usable new set of 89 CLI tasks with public harness and shows frontier agents still below 65%, but the realism claim needs tighter evidence on how the tasks were chosen. read the letter →

arxiv 2601.11868 v1 submitted 2026-01-17 cs.SE cs.AI

Mike A. Merrill , Alexander G. Shaw , Nicholas Carlini , Boxuan Li , Harsh Raj , Ivan Bercovich , Lin Shi , Jeong Yeon Shin
show 77 more authors
Thomas Walshe E. Kelly Buchanan Junhong Shen Guanghao Ye Haowei Lin Jason Poulos Maoyu Wang Marianna Nezhurina Jenia Jitsev Di Lu Orfeas Menis Mastromichalakis Zhiwei Xu Zizhao Chen Yue Liu Robert Zhang Leon Liangyu Chen Anurag Kashyap Jan-Lucas Uslu Jeffrey Li Jianbo Wu Minghao Yan Song Bian Vedang Sharma Ke Sun Steven Dillmann Akshay Anand Andrew Lanpouthakoun Bardia Koopah Changran Hu Etash Guha Gabriel H. S. Dreiman Jiacheng Zhu Karl Krauth Li Zhong Niklas Muennighoff Robert Amanfu Shangyin Tan Shreyas Pimpalgaonkar Tushar Aggarwal Xiangning Lin Xin Lan Xuandong Zhao Yiqing Liang Yuanli Wang Zilong Wang Changzhi Zhou David Heineman Hange Liu Harsh Trivedi John Yang Junhong Lin Manish Shetty Michael Yang Nabil Omi Negin Raoof Shanda Li Terry Yue Zhuo Wuwei Lin Yiwei Dai Yuxin Wang Wenhao Chai Shang Zhou Dariush Wahdany Ziyu She Jiaming Hu Zhikang Dong Yuxuan Zhu Sasha Cui Ahson Saiyed Arinbjörn Kolbeinsson Jesse Hu Christopher Michael Rytting Ryan Marten Yixin Wang Alex Dimakis Andy Konwinski Ludwig Schmidt
This is my paper · ORCID
classification cs.SEcs.AI
keywords AIagentscommandlinebenchmarksterminalenvironmentslong-horizontasksfrontiermodelsagentevaluation
checked against reality_from_one_distinctionCost.FunctionalEquation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Terminal-Bench 2.0 supplies 89 distinct command-line problems drawn from actual developer and system workflows. Each task includes its own isolated environment, a verified human solution, and automated tests that confirm success or failure. Frontier models and agents reach success rates below 65 percent across the full set. The authors include an error breakdown that points to recurring failures in long-horizon planning and tool coordination.

What carries the argument

Terminal-Bench 2.0 dataset of 89 curated tasks, each pairing a terminal environment with a human solution and test suite that together evaluate autonomous completion of long-horizon command-line work.

What would settle it

An agent that solves more than 80 percent of the 89 tasks on the first attempt, using only general capabilities and without any task-specific fine-tuning or example leakage.

Watch

Extended reading notes

Core claim

The paper introduces Terminal-Bench 2.0, a collection of 89 terminal-based tasks modeled on real workflows, each equipped with a unique environment, human-written solution, and comprehensive verification tests, on which current frontier models and agents achieve success rates below 65 percent.

Load-bearing premise

The 89 selected tasks represent a fair sample of valuable real-world long-horizon workflows and the tests correctly measure whether an agent has solved them.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript introduces Terminal-Bench 2.0, a benchmark of 89 tasks in command-line interface environments, each with a unique setup, human-written solution, and comprehensive tests. Tasks are described as carefully curated and inspired by real-world workflows. Frontier models and agents are evaluated and reported to score below 65%, accompanied by an error analysis. The dataset and evaluation harness are released publicly at https://www.tbench.ai/.

Significance. If the curation process ensures representativeness of valuable long-horizon CLI workflows and the tests provide unbiased, complete verification, the benchmark would offer a meaningful advance over existing agent evaluations by focusing on realistic, difficult tasks where current systems fall short. The public release of data and harness is a clear strength that enables reproducibility and community use.

major comments (3)
  1. [Section 3] Section 3 (Benchmark Construction): The central claim that the 89 tasks are 'carefully curated' and 'inspired by problems from real workflows' is load-bearing for both the 'realistic' and 'hard' assertions, yet the manuscript provides no quantitative validation such as inter-rater agreement scores, coverage statistics across workflow categories, expert realism ratings, or comparison against production logs. This leaves the representativeness assumption untested.
  2. [Section 4] Section 4 (Experiments and Evaluation Protocol): The reported <65% scores for frontier models rely on the human-written tests being comprehensive and free of bias or incompleteness. The manuscript does not report metrics on test coverage, inter-test consistency, or handling of partial/edge-case successes, making it difficult to confirm that low scores reflect agent limitations rather than test design.
  3. [Section 5] Section 5 (Error Analysis): While an error analysis is performed, it is not systematically linked back to task characteristics (e.g., horizon length, command diversity, or environment complexity), limiting its utility for identifying targeted improvements and for validating that the benchmark isolates the intended capabilities.
minor comments (3)
  1. [Title and Abstract] The title refers to 'Terminal-Bench' while the abstract and text use 'Terminal-Bench 2.0'; clarify the versioning and relation to any prior version in the introduction.
  2. [Section 2] Add explicit comparison in the related work section to other CLI or terminal benchmarks (e.g., those based on SWE-bench or terminal-specific agent suites) to better position the novelty of the 89-task scale and test design.
  3. [Section 4] Figure captions and axis labels in the results section could be expanded to include exact model names, number of runs, and confidence intervals for the reported success rates.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their detailed and constructive feedback on our manuscript. We appreciate the opportunity to clarify and strengthen our presentation of Terminal-Bench 2.0. Below, we provide point-by-point responses to the major comments and indicate the revisions we will make in the updated version.

read point-by-point responses

  1. Referee: [Section 3] Section 3 (Benchmark Construction): The central claim that the 89 tasks are 'carefully curated' and 'inspired by problems from real workflows' is load-bearing for both the 'realistic' and 'hard' assertions, yet the manuscript provides no quantitative validation such as inter-rater agreement scores, coverage statistics across workflow categories, expert realism ratings, or comparison against production logs. This leaves the representativeness assumption untested.

    Authors: We agree that additional quantitative validation would strengthen the claims regarding the realism and representativeness of the tasks. The curation process involved the authors, who have extensive experience with CLI environments, selecting tasks based on common real-world scenarios encountered in software development, system administration, and data processing. To address this concern, we will revise Section 3 to include: (1) a breakdown of tasks by category (e.g., file operations, networking, scripting), (2) statistics on task complexity such as average number of commands in human solutions, and (3) a description of the selection criteria used to ensure diversity. While we did not conduct formal inter-rater agreement studies or access proprietary production logs, we believe the public release of the benchmark will allow the community to provide further validation. We will also add a limitations section noting this. revision: partial

  2. Referee: [Section 4] Section 4 (Experiments and Evaluation Protocol): The reported <65% scores for frontier models rely on the human-written tests being comprehensive and free of bias or incompleteness. The manuscript does not report metrics on test coverage, inter-test consistency, or handling of partial/edge-case successes, making it difficult to confirm that low scores reflect agent limitations rather than test design.

    Authors: We acknowledge the importance of demonstrating test quality. Each task includes a set of tests written by humans that verify the final state of the environment against the expected outcome, rather than step-by-step matching. To improve transparency, we will add to Section 4 details on how tests were constructed, including examples of test cases that cover different aspects (e.g., file existence, content verification, process status). We will also clarify that success requires all tests to pass, which handles partial successes by treating them as failures. While we do not have automated coverage metrics like code coverage (since tests are not code-based in the traditional sense), we will report the average number of tests per task and note that tests were manually reviewed for completeness. revision: yes

  3. Referee: [Section 5] Section 5 (Error Analysis): While an error analysis is performed, it is not systematically linked back to task characteristics (e.g., horizon length, command diversity, or environment complexity), limiting its utility for identifying targeted improvements and for validating that the benchmark isolates the intended capabilities.

    Authors: We agree that linking the error analysis more explicitly to task characteristics would enhance its value. In the revised manuscript, we will expand Section 5 to include breakdowns of error types by task features, such as correlating failure rates with the length of the human-written solution (as a proxy for horizon length), the diversity of commands used, and the number of files or processes involved. This will help identify whether certain capabilities (e.g., long-horizon planning) are particularly challenging. We will also discuss how these insights point to specific areas for agent improvement. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmark with no derivations or self-referential predictions

full rationale

The paper presents Terminal-Bench 2.0 as a curated set of 89 tasks with human-written solutions and tests, published externally for independent use. No equations, parameter fitting, predictions derived from inputs, or load-bearing self-citations appear in the abstract or described structure. The work is self-contained as benchmark creation, allowing external falsification via the released harness and data. No steps reduce by construction to the paper's own inputs.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is an empirical benchmark paper whose central claims rest on domain assumptions about task realism and test validity rather than new theoretical constructs or fitted parameters.

assumptions (2)
  • domain assumption The 89 tasks are hard and inspired by real workflows.
    Stated in the abstract as the basis for the benchmark's value; no independent verification details provided.
  • domain assumption The comprehensive tests accurately verify task completion.
    Required for the reported scores to be meaningful; assumed but not detailed in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces." pith.science (2026). https://pith.science/paper/2601.11868

@misc{pith2026260111868,
  author       = {Pith},
  title        = {Pith review of: Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2601.11868}},
  note         = {Machine review of arXiv:2601.11868}
}
read the original abstract

AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Foundation/RealityFromDistinction reality_from_one_distinction unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    We present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments... frontier models and agents score less than 65% on the benchmark

  • IndisputableMonolith/Cost/FunctionalEquation washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    Each task features a unique environment, human-written solution, and comprehensive tests for verification.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Showing 60 of 204 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 204 Pith citations

  1. MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves

    cs.LG 2026-09 conditional novelty 8.0 of 10

    MetaRSI-v1 composes Data-RSI, Harness-RSI, and Model-RSI operators under a meta-scheduler that jointly optimizes operator order and policies, improving an open-weight model by 10.9 average points without an external teacher.

  2. Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments

    cs.AI 2026-09 accept novelty 8.0 of 10

    A framework that converts recorded agent trajectories into reusable terminal environments with auto-generated tasks, boosting agent benchmark performance through supervised fine-tuning.

  3. QuoteBench: How Matched Scores Can Hide Command-Path Failures

    cs.AI 2026-08 accept novelty 8.0 of 10

    Matched execution scores can hide the difference between command-generation errors and transport damage, as QuoteBench's fixed-reply replay across eight model configurations demonstrates.

  4. Fantastic Adaptive Taxonomies and How to Use Them

    cs.SE 2026-07 conditional novelty 8.0 of 10

    AdaMAST induces a compact, evidence-grounded failure taxonomy from an agent's own traces and shows it improves agent-system search, runtime reflection, and trajectory selection over free-text and fixed-taxonomy baselines.

  5. WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

    cs.CL 2026-07 conditional novelty 8.0 of 10

    WILDTRACE evaluates long-context models on 481 natural multi-hop evidence trails from 214 real documents, with top systems at 75.3% and geometry-specific weaknesses.

  6. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

    cs.SE 2026-07 conditional novelty 8.0 of 10

    RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substit...

  7. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

    cs.AI 2026-06 unverdicted novelty 8.0 of 10

    CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.

  8. Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?

    cs.AI 2026-06 unverdicted novelty 8.0 of 10

    A user study with over 100 participants shows humans rarely spot AI agents sabotaging code during extended collaborative tasks, even with a safety monitor present.

  9. The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?

    cs.AI 2026-06 unverdicted novelty 8.0 of 10

    The Meta-Agent Challenge shows frontier AI models rarely match human-engineered agent baselines when tasked with autonomous development, with proprietary models succeeding most often and some exhibiting cheating under...

  10. Heimdall: Formally Verified Automated Migration of Legacy eBPF Programs to Rust

    cs.CR 2026-05 unverdicted novelty 8.0 of 10

    Heimdall automates translation of eBPF C programs to Rust with formal equivalence proofs for 94.1% of 102 tested programs using LLMs, static analysis, and Z3-based checking.

  11. HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models

    cs.CL 2026-05 conditional novelty 8.0 of 10

    HalluWorld is a controlled benchmark using explicit reference world models to automatically label and disentangle hallucinations in LLMs across synthetic environments with varying complexity and observability.

  12. WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

    cs.CL 2026-05 unverdicted novelty 8.0 of 10

    A new native-runtime benchmark reveals that current frontier AI agents succeed on at most 62 percent of realistic long-horizon CLI tasks.

  13. Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values

    cs.AI 2026-05 unverdicted novelty 8.0 of 10

    Agent-ValueBench is the first dedicated benchmark for agent values, showing they diverge from LLM values, form a homogeneous 'Value Tide' across models, and bend under harnesses and skill steering.

  14. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

    cs.SE 2026-03 unverdicted novelty 8.0 of 10

    SlopCodeBench shows coding agents degrade in structural quality and verbosity across iterative extensions, with no agent solving any problem completely and agent code 2x more eroded than human code.

  15. Bridging Agent Semantics with Spot Capacity: An Elastic and Recoverable Service Model

    cs.DC 2026-08 conditional novelty 7.0 of 10

    SemSpot connects agent task semantics with transient MaaS capacity through task-conditioned spot offers and explicit recovery continuations.

  16. Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification

    cs.AI 2026-08 conditional novelty 7.0 of 10

    HarnessLens evolves LLM agent harnesses by selecting behavior-relevant verification tasks and gating acceptance on attributable evidence, improving held-out performance by 7.6-13.6% with lower budget.

  17. SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

    cs.CL 2026-08 conditional novelty 7.0 of 10

    SWE Refactor Bench grades coding agents on 20 whole-repository stack migrations with a three-stage protocol, and finds 5.4% of 520 runs passed all stages, with 13 of 20 tasks unsolved.

  18. K-Bench: measuring model performance on real scientific agent requests

    cs.AI 2026-08 conditional novelty 7.0 of 10

    On K-Bench 01, a benchmark built from real user requests with attachments, no frontier scientific agent clears the acceptance threshold under all three LLM judges, and every model scores higher on communication than o...

  19. From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

    cs.OS 2026-08 conditional novelty 7.0 of 10

    Agentic workloads are heavyweight, heterogeneous, and shift bottlenecks in ways conventional LLM serving assumptions miss; characterization-guided optimizations cut latency by up to 40%, memory by 4.6x, and redundant ...

  20. Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Harness-IF scores 256 rules across coding-agent runs and finds every model performs 3.6 to 7.4 points worse on rules that oppose unprompted defaults, so aggregate compliance scores overstate true instruction following.

  21. Back to the Future: A workbook time machine for spread sheet creation benchmarks

    cs.AI 2026-08 conditional novelty 7.0 of 10

    The workbook time machine reverse-engineers real workbooks into creation tasks, and the resulting benchmark shows pivot tables remain nearly unsolved by all tested frontier models.

  22. GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Rule hybridization makes agent self-evolution gains attributable; the best evolved configuration reaches 67.07% against a 91.6% fully-informed oracle ceiling.

  23. AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

    cs.LG 2026-08 conditional novelty 7.0 of 10

    AOSpec co-speculates actions and observations in LLM agents, using expected-value decoding and joint action-state verification to hide tool execution latency, achieving 11.8-32.5% end-to-end latency savings in trace replay.

  24. LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

    cs.SE 2026-07 conditional novelty 7.0 of 10

    LoopsBench exposes coding agents to dependency-DAG tasks with regression obligations; the best configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks.

  25. Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

    cs.DC 2026-07 conditional novelty 7.0 of 10

    Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.

  26. ORCA-bench: How Ready Are Language Model Agents for Oncall?

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Frontier coding agents achieve at most 25.3% RCA accuracy on realistic medium oncall tasks and 10% on hard ones in a production-fidelity telemetry-plus-code benchmark.

  27. SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

    cs.CR 2026-07 conditional novelty 7.0 of 10

    No frontier LLM agent fully detects and remediates any of 10 real post-compromise host ranges; alert-driven findings work, silent intrusion and verified cleanup do not.

  28. Is Progressive Disclosure All You Need for Long-Context Agents?

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Progressive disclosure is redundant for a strong agent reading one book, but decisive when an agent must navigate a 20-book library; one flat routing level beats deeper hierarchies.

  29. Interactive Task Alignment as a POMDP

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Under ambiguous user requests, current LLMs recover the intended task only 22–32% of the time, well below human accuracy of 48%, and post-training only partially closes the gap.

  30. Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning

    cs.CL 2026-07 conditional novelty 7.0 of 10

    On a new 615-question business-case benchmark graded by AI against instructor rubrics, frontier LLMs score 87-88% partial credit but complete only about half the questions.

  31. ArchEval: Measuring AI Agents as Computer Architects

    cs.AR 2026-07 conditional novelty 7.0 of 10

    LLM agents beat architecture baselines with full simulator harnesses, but only one configuration stays above baseline without feedback, and performance modeling remains weak.

  32. GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

    cs.SE 2026-07 conditional novelty 7.0 of 10

    On 110 scoped Unreal Engine C++ tasks from nine real game repos, the best coding-agent setup reaches only 55.5% pass@1 and 31 tasks remain unsolved by all twelve configurations.

  33. MOSAIC: Knowledge-Guided CLI Command Composition Attack in LLM Coding Agents

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Individually benign CLI commands compose via shared OS state into high-success attacks on real LLM coding agents; MOSAIC systematically generates them from CVE/PoC knowledge at 96.59% ASR.

  34. MirrorCode: AI can rebuild entire programs from behavior alone

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    MirrorCode benchmark shows current AI models achieving up to 56% success reimplementing 25 diverse full programs from behavior alone, including a 16,000-line bioinformatics toolkit.

  35. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.

  36. Agentic Abstention: Do Agents Know When to Stop Instead of Act?

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    LLM agents often fail to abstain at the right time in uncertain multi-turn tasks, and the CONVOLVE context engineering method raises timely abstention rates on WebShop from 26.7 to 57.4 without parameter updates.

  37. CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    CyberChainBench is a new benchmark evaluating LLM agents on vulnerability detection, exploit generation, and patch synthesis using 541 real-world DeFi incidents with on-chain historical evaluation, showing peak perfor...

  38. GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    A matched benchmark shows GUI computer-use agents at 59.1% full pass rate versus 48.2% for original-skill CLI agents, rising to 69.3% with verifier-guided augmentation, indicating modality-specific execution bottlenecks.

  39. CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    CLI-Universe synthesizes a verified 6K dataset of terminal-agent tasks that, when used to fine-tune Qwen3-32B, reaches 33.4% on Terminal-Bench 2.0 and sets a new open-source SOTA for models at or below 32B parameters.

  40. StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good h...

  41. Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    MAFP applies fictitious play to LLM multi-agent systems to resolve stance entanglement in competitive decision-making, outperforming single-round and multi-round baselines on tournament strength and robustness.

  42. One Goal, Many Commands: Characterizing Denylist Fragility in AI Agents

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    ShellSieve, an LLM-driven pipeline, detects command denylist fragility in terminal AI agents and finds 69.0-98.6% of 1,709 GitHub-collected denylists to be bypassable.

  43. Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Introduces the first community-governed unified JSON schema and crowdsourced repository for AI evaluation results, with converters and a database spanning 22,235 models and 2,273 benchmarks.

  44. AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    AgentBeats implements agentified evaluation of diverse AI agents through standardized interfaces, validated at scale in a five-month competition with 298 judges and 467 subjects plus a coding case study.

  45. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    SciAgentArena is a new interactive benchmark for AI agents on scientific tasks that finds agents handle clear data-analysis workflows but struggle with novel insights, self-directed exploration, and open-ended questions.

  46. Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Workflow-GYM is a new benchmark for long-horizon professional GUI agent tasks where state-of-the-art models reach only slightly above 30% success.

  47. Self-Harness: Harnesses That Improve Themselves

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Self-Harness lets LLM agents autonomously refine their interaction harnesses through weakness mining, proposal generation, and validation, raising held-out pass rates on Terminal-Bench-2.0 from 40.5% to 61.9%, 23.8% t...

  48. ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    ADK Arena evaluates 51 Python ADKs by having an LLM learn each framework's API, write and repair agent code, and run on benchmarks, finding 57% success rate, 5.6x cost variation, no dominant framework, and substitutab...

  49. AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

    cs.AI 2026-06 conditional novelty 7.0 of 10

    AutoMedBench evaluates AI agents on long-horizon medical workflows across five stages and finds validation and submission as dominant failure points based on thousands of runs.

  50. Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Sakura is a multi-agent system that generates structurally complex tests from NL descriptions, achieving 50-78% higher compilability and 38-66% higher coverage overlap than baselines on 1,464 scenarios from 20 Apache ...

  51. Stateful Online Monitoring Catches Distributed Agent Attacks

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    A clustering-based stateful online monitor detects distributed multi-agent cyberattacks that evade standard per-transcript monitors, catching them 30% earlier in large-scale simulated traffic with low overhead.

  52. SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    SkillEvolBench is a new diagnostic benchmark that evaluates the transition from episodic experience to procedural skills in LLM agents using role-conditioned task families and frozen deployment tests.

  53. EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    EvoCode-Bench shows that single-round success rates for coding agents exceed multi-turn persistent execution rates by 22-40 points, with performance dropping below half of round-1 levels by round 5 across 13 evaluated agents.

  54. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.

  55. Terminal-World: Scaling Terminal-Agent Environments via Agent Skills

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Terminal-World is a skill-based synthesis pipeline that generates 5,723 training environments and produces Terminal-World-32B which outperforms baselines on Terminal-Bench 2.0 using only 1.2% of the data.

  56. WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    WebGameBench is a new benchmark that evaluates coding agents on building browser-native games from frozen specifications, with runtime browser evaluation showing best agents reach 76.9% usable rate but only 20.2% exce...

  57. Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    ConsumerSimBench evaluates 13 LLMs on reconstructing crowd reactions from 1,553 Chinese social-media topics using 23,122 auditable yes-no criteria, finding maximum coverage of 47.8% by Gemini-3.1-Pro.

  58. Do Coding Agents Understand Least-Privilege Authorization?

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    Coding agents struggle to infer least-privilege file permissions by omitting needed accesses while granting unused or sensitive ones, but Sufficiency-Tightness Decomposition improves sensitive-task success by up to 15...

  59. ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    ClawForge supplies a generator that turns scenario templates into reproducible command-line tasks testing state conflict handling, where the strongest frontier model scores only 45.3 percent strict accuracy.

  60. CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editing

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    CRANE merges Instruct and Thinking model checkpoints via constrained nullspace editing to improve code agent reasoning and benchmark performance without retraining.

See all 204 Pith citations

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · cited by 204 Pith papers (see all)

  1. [1]

    TextArena

    URLhttps://arxiv.org/abs/2504.11442. Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. Swe-perf: Can language models optimize code performance on real-world repositories?arXiv preprint arXiv:2507.12415, 2025. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama...

    work page Pith review arXiv 2025

  2. [2]

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.50. URL https://aclanthology.org/2024.acl-long.50/. Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphin...

    work page doi:10.18653/v1/2024.acl-long.50 2024

  3. [3]

    URLhttps://openreview.net/forum?id=ryTr83DxRq. OpenAI. GPT-5 System Card. Technical report, OpenAI, August 2025. URLhttps://openai. com/index/gpt-5-system-card/. Technical report describing the GPT-5 model family including GPT-5, GPT-5-Mini, and GPT-5-Nano. OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URLhttps://arxiv.org/abs/ 2508.10925. Davide P...

    work page Pith review arXiv 2025

  4. [4]

    Romera-Paredes, M

    URLhttps://openreview.net/forum?id=fp6t3F669F. Melissa Z Pan, Mert Cemri, Lakshya A Agrawal, Shuyi Yang, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica. Why do multiagent systems fail? InICLR 2025 Workshop on Building Trust in Language Models and Applicatio...

    work page doi:10.48550/arxiv.2507.15887 2025

  5. [5]

    Establishing Best Practices for Building Rigorous Agentic Benchmarks

    General Language Model 4.6, a large mixture-of-experts language model with extended context, enhanced reasoning and coding capabilities. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A real- istic web environment for building autonomou...

    work page Pith review arXiv 2024

  6. [6]

    18 APPENDIXTABLE OFCONTENTS A Detailed Results 21 A.1 Comprehensive Results

    URLhttps://openreview.net/forum?id=YrycTjllL0. 18 APPENDIXTABLE OFCONTENTS A Detailed Results 21 A.1 Comprehensive Results . . . . . . . . . . . . . . . . . . . . . . . . 21 A.2 Agent Run Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 21 A.3 Task-Level Results . . . . . . . . . . . . . . . . . . . . . . . . . . 21 B Quality Control 21 B.1 Ta...

    work page

  7. [7]

    EXACT SPECIFICATION MISMATCHES: Tests expect specific parameter names, file formats, function signatures, or return values NOT precisely specified in task instructions -> INSUFFICIENT INSTRUCTIONS

    work page

  8. [8]

    IMPLICIT EXPECTATIONS: Tests check for behavior/format/structure requiring agent assumptions about implementation details -> INSUFFICIENT INSTRUCTIONS

    work page

Show all 20 references
  1. [9]

    HARDCODED TEST EXPECTATIONS: Tests look for exact strings, parameter names, or data structures not documented in task instructions -> INSUFFICIENT INSTRUCTIONS

  2. [10]

    outcome":

    EXAMPLES vs REQUIREMENTS: Task provides examples but doesn’t explicitly state they’re REQUIRED -> tests expecting exact example matching indicate INSUFFICIENT INSTRUCTIONS For MULTIPLE TRIALS: Compare failure patterns across trials. If all trials fail on the same tests with si...

    2025

  3. [11]

    Outcome Validity.Most criteria are satisfied, with the following exceptions. 1.I.d.2: Measures quality of test cases using objective metrics (e.g., code coverage, cyclomatic complexity control).Test case quality is not assessed via objective, automated metrics. Instead, the be...

  4. [12]

    Independent inspection of the released tasks con- firmed that they are well-specified and largely free of ambiguity or underspecification

    Task Validity.We found no major issues. Independent inspection of the released tasks con- firmed that they are well-specified and largely free of ambiguity or underspecification

  5. [13]

    held-out

    Benchmark Reporting.Two checklist items warrant comment: 1.III.3: Includes measures to prevent data contamination at the time of benchmark release, such as a private, held-out test set.All tasks are hosted on a public GitHub repository. While a canary string is systematically ...

    2025

  6. [14]

    Analyse the input and corresponding outputs - What type of command is being executed? What type of output is being produced?

  7. [15]

    We’re looking for indications that running the command has failed - carefully analyse the outputs to determine this

    Identify if any output failures that occur. We’re looking for indications that running the command has failed - carefully analyse the outputs to determine this. # Response format requirements - Always use valid JSON in your outputs. Only output JSON - The schema must contain a...

  8. [16]

    Carefully analyse the taxonomy, understand all categories and subcategories in the taxonomy

  9. [17]

    Look through the information provided by the user, analyse this carefully

  10. [18]

    You are about to hand off your work to another agent. Please summarize what you’ve done so far

    Classify the CLI failures / errors into the error taxonomy. This is the taxonomy: {{taxonomy}} Tips: * Read through the taxonomy to find the most applicable categories and subcategories of errors. * Select the most applicable error category and subcategory. * If there is no fa...

  11. [19]

    The negligible correlation demonstrates that the number of interaction rounds has no meaningful relationship with task success. The analysis reveals negligible correlation (r=-0.028, p=0.916) between episode count and success rate, demonstrating that the number of interaction ...

  12. [20]

    Results: X Y Z

    The weak negative correlation (r=-0.170, p=0.515) indicates that verbosity does not significantly predict task success. generates the most tokens by far ( 60k average) yet achieves only 8% success, indicating that ex- cessive verbosity can reflect inefficient reasoning rather ...

Pith tools

Reviewed May 11, 2026 · model on record in the stance chip above.