SemSpot connects agent task semantics with transient MaaS capacity through task-conditioned spot offers and explicit recovery continuations.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
abstract
AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
citation-role summary
background 1
citation-polarity summary
years
2026 5verdicts
CONDITIONAL 5roles
background 1polarities
unclear 1representative citing papers
Terminal agents are defined by a command-execution action-observation loop, analyzed via seven competence dimensions, with diagnostics showing benchmark-dependent process exposure and attribution limits.
CalibForge generates terminal-agent training tasks by revising candidates until a strong solver passes and a weak solver fails, and students trained on the resulting 5,431 tasks gain up to 30 points on held-out benchmarks.
A contextual multidimensional IRT model improves within-scenario prediction of LLM performance on unseen questions, while cross-scenario transfer remains weak.
ERASE improves training throughput by up to 9.51% on a CTR model by using Forward-Forward-style stop-gradient detachment to launch each block's backward pass early on separate CUDA streams.
citing papers explorer
-
Bridging Agent Semantics with Spot Capacity: An Elastic and Recoverable Service Model
SemSpot connects agent task semantics with transient MaaS capacity through task-conditioned spot offers and explicit recovery continuations.
-
Terminal Agents: A Survey of AI Agents in Command-Line Environments
Terminal agents are defined by a command-execution action-observation loop, analyzed via seven competence dimensions, with diagnostics showing benchmark-dependent process exposure and attribution limits.
-
CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
CalibForge generates terminal-agent training tasks by revising candidates until a strong solver passes and a weak solver fails, and students trained on the resulting 5,431 tasks gain up to 30 points on held-out benchmarks.
-
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model
A contextual multidimensional IRT model improves within-scenario prediction of LLM performance on unseen questions, while cross-scenario transfer remains weak.
-
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems
ERASE improves training throughput by up to 9.51% on a CTR model by using Forward-Forward-style stop-gradient detachment to launch each block's backward pass early on separate CUDA streams.