TUA-Bench provides 120 manually designed terminal tasks across five families with execution-based scoring; the top agent reaches 65.8% success.
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 representative, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve. Data and code are available at https://github.com/EuniAI/TerminalWorld.
years
2026 3verdicts
UNVERDICTED 3representative citing papers
HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.
OpenRath introduces Session as a first-class, branchable runtime value that unifies fragmented state in multi-agent systems and makes fork, merge, and replay explicit operations.
citing papers explorer
-
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
TUA-Bench provides 120 manually designed terminal tasks across five families with execution-based scoring; the top agent reaches 65.8% success.
-
HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning
HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.
-
OpenRath: Session-Centered Runtime State for Agent Systems
OpenRath introduces Session as a first-class, branchable runtime value that unifies fragmented state in multi-agent systems and makes fork, merge, and replay explicit operations.