{"id":"27285748-e85c-4661-9273-2182f5ff7d43","arxiv_id":"2601.11868","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Terminal-Bench 2.0 is a new benchmark of 89 realistic terminal tasks on which frontier AI agents score below 65%.","lead":"This paper introduces Terminal-Bench 2.0, a benchmark of 89 hard tasks for AI agents in command-line terminal environments drawn from real workflows. A smart generalist should read it to see how current frontier models perform on practical, long-horizon computing tasks and what gaps remain.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Representativeness of the 89 tasks for real long-horizon workflows rests on unvalidated curation claims without external metrics or expert review.","rationale":"The reader's weakest assumption directly matches the load-bearing point. With full text now accessible, the curation and validation details remain high-level and self-referential, so the concern stands without new evidence from the manuscript. No internal inconsistency found in the reported scores or setup, but external validation is missing.","tokens_in":1656,"tokens_out":300,"duration_ms":22166,"concrete_test":"Sample 15 tasks; have 5 independent domain experts (sysadmins/devops) rate each on 1-5 scales for realism to real workflows and completeness of tests; if mean realism <4.0 or >20% of tasks flagged for missing edge cases, rerun the frontier model evaluation on the flagged subset to measure score change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the tasks are both hard (frontier <65%) and representative of valuable real-world CLI workflows. The paper states tasks are 'carefully curated' and 'inspired by problems from real workflows' with human-written solutions and tests, but provides no quantitative validation (e.g., inter-rater agreement on difficulty/realism, coverage of workflow categories, or comparison to production logs). This leaves open whether selection bias or incomplete test suites could inflate perceived hardness or understate agent capability.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces Terminal-Bench 2.0, a benchmark of 89 tasks in command-line interface environments, each with a unique setup, human-written solution, and comprehensive tests. Tasks are described as carefully curated and inspired by real-world workflows. Frontier models and agents are evaluated and reported to score below 65%, accompanied by an error analysis. The dataset and evaluation harness are released publicly at https://www.tbench.ai/.","tokens_in":1746,"tokens_out":684,"duration_ms":49407,"significance":"If the curation process ensures representativeness of valuable long-horizon CLI workflows and the tests provide unbiased, complete verification, the benchmark would offer a meaningful advance over existing agent evaluations by focusing on realistic, difficult tasks where current systems fall short. The public release of data and harness is a clear strength that enables reproducibility and community use.","major_comments":[{"comment":"Section 3 (Benchmark Construction): The central claim that the 89 tasks are 'carefully curated' and 'inspired by problems from real workflows' is load-bearing for both the 'realistic' and 'hard' assertions, yet the manuscript provides no quantitative validation such as inter-rater agreement scores, coverage statistics across workflow categories, expert realism ratings, or comparison against production logs. This leaves the representativeness assumption untested.","section":"Section 3"},{"comment":"Section 4 (Experiments and Evaluation Protocol): The reported <65% scores for frontier models rely on the human-written tests being comprehensive and free of bias or incompleteness. The manuscript does not report metrics on test coverage, inter-test consistency, or handling of partial/edge-case successes, making it difficult to confirm that low scores reflect agent limitations rather than test design.","section":"Section 4"},{"comment":"Section 5 (Error Analysis): While an error analysis is performed, it is not systematically linked back to task characteristics (e.g., horizon length, command diversity, or environment complexity), limiting its utility for identifying targeted improvements and for validating that the benchmark isolates the intended capabilities.","section":"Section 5"}],"minor_comments":[{"comment":"The title refers to 'Terminal-Bench' while the abstract and text use 'Terminal-Bench 2.0'; clarify the versioning and relation to any prior version in the introduction.","section":"Title and Abstract"},{"comment":"Add explicit comparison in the related work section to other CLI or terminal benchmarks (e.g., those based on SWE-bench or terminal-specific agent suites) to better position the novelty of the 89-task scale and test design.","section":"Section 2"},{"comment":"Figure captions and axis labels in the results section could be expanded to include exact model names, number of runs, and confidence intervals for the reported success rates.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript aligns with the scope of software engineering venues focused on empirical evaluation of AI tools, but the absence of external validation for task curation is a recurring pattern in benchmark papers; the authors could be encouraged to add a small expert review appendix if revising."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their detailed and constructive feedback on our manuscript. We appreciate the opportunity to clarify and strengthen our presentation of Terminal-Bench 2.0. Below, we provide point-by-point responses to the major comments and indicate the revisions we will make in the updated version.","responses":[{"response":"We agree that additional quantitative validation would strengthen the claims regarding the realism and representativeness of the tasks. The curation process involved the authors, who have extensive experience with CLI environments, selecting tasks based on common real-world scenarios encountered in software development, system administration, and data processing. To address this concern, we will revise Section 3 to include: (1) a breakdown of tasks by category (e.g., file operations, networking, scripting), (2) statistics on task complexity such as average number of commands in human solutions, and (3) a description of the selection criteria used to ensure diversity. While we did not conduct formal inter-rater agreement studies or access proprietary production logs, we believe the public release of the benchmark will allow the community to provide further validation. We will also add a limitations section noting this.","revision_made":"partial","referee_comment":"[Section 3] Section 3 (Benchmark Construction): The central claim that the 89 tasks are 'carefully curated' and 'inspired by problems from real workflows' is load-bearing for both the 'realistic' and 'hard' assertions, yet the manuscript provides no quantitative validation such as inter-rater agreement scores, coverage statistics across workflow categories, expert realism ratings, or comparison against production logs. This leaves the representativeness assumption untested."},{"response":"We acknowledge the importance of demonstrating test quality. Each task includes a set of tests written by humans that verify the final state of the environment against the expected outcome, rather than step-by-step matching. To improve transparency, we will add to Section 4 details on how tests were constructed, including examples of test cases that cover different aspects (e.g., file existence, content verification, process status). We will also clarify that success requires all tests to pass, which handles partial successes by treating them as failures. While we do not have automated coverage metrics like code coverage (since tests are not code-based in the traditional sense), we will report the average number of tests per task and note that tests were manually reviewed for completeness.","revision_made":"yes","referee_comment":"[Section 4] Section 4 (Experiments and Evaluation Protocol): The reported <65% scores for frontier models rely on the human-written tests being comprehensive and free of bias or incompleteness. The manuscript does not report metrics on test coverage, inter-test consistency, or handling of partial/edge-case successes, making it difficult to confirm that low scores reflect agent limitations rather than test design."},{"response":"We agree that linking the error analysis more explicitly to task characteristics would enhance its value. In the revised manuscript, we will expand Section 5 to include breakdowns of error types by task features, such as correlating failure rates with the length of the human-written solution (as a proxy for horizon length), the diversity of commands used, and the number of files or processes involved. This will help identify whether certain capabilities (e.g., long-horizon planning) are particularly challenging. We will also discuss how these insights point to specific areas for agent improvement.","revision_made":"yes","referee_comment":"[Section 5] Section 5 (Error Analysis): While an error analysis is performed, it is not systematically linked back to task characteristics (e.g., horizon length, command diversity, or environment complexity), limiting its utility for identifying targeted improvements and for validating that the benchmark isolates the intended capabilities."}],"tokens_in":1341,"tokens_out":798,"duration_ms":61985,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a fresh benchmark of 89 terminal tasks, each with its own environment, a human-written solution, and tests. They release the full dataset and evaluation harness, and they report that current frontier models and agents top out under 65 percent, plus a basic error breakdown. That package is immediately useful for anyone testing long-horizon CLI agents.","headline":"Terminal-Bench ships a usable new set of 89 CLI tasks with public harness and shows frontier agents still below 65%, but the realism claim needs tighter evidence on how the tasks were chosen.","tokens_in":2616,"tokens_out":159,"would_cite":true,"duration_ms":22231,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A benchmark of 89 realistic terminal tasks shows frontier AI agents complete fewer than 65 percent of them.","keywords":["AI agents","command line","benchmarks","terminal environments","long-horizon tasks","frontier models","agent evaluation"],"falsifier":"An agent that solves more than 80 percent of the 89 tasks on the first attempt, using only general capabilities and without any task-specific fine-tuning or example leakage.","tokens_in":2542,"feed_emoji":"💻","tokens_out":402,"duration_ms":21043,"temperature":0.7,"pith_summary":"Terminal-Bench 2.0 supplies 89 distinct command-line problems drawn from actual developer and system workflows. Each task includes its own isolated environment, a verified human solution, and automated tests that confirm success or failure. Frontier models and agents reach success rates below 65 percent across the full set. The authors include an error breakdown that points to recurring failures in long-horizon planning and tool coordination.","feed_headline":"AI agents solve under 65% of 89 realistic terminal tasks","feed_subtitle":"Benchmark built from real command-line workflows reveals current models still struggle with autonomous execution.","key_machinery":"Terminal-Bench 2.0 dataset of 89 curated tasks, each pairing a terminal environment with a human solution and test suite that together evaluate autonomous completion of long-horizon command-line work.","core_discovery":"The paper introduces Terminal-Bench 2.0, a collection of 89 terminal-based tasks modeled on real workflows, each equipped with a unique environment, human-written solution, and comprehensive verification tests, on which current frontier models and agents achieve success rates below 65 percent.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Agents under 65% on 89 terminal tasks","Frontier models below 65% on hard terminal benchmark","Terminal-Bench 2.0: AI under 65% success rate","89 CLI tasks: AI agents succeed under 65%"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The 89 selected tasks represent a fair sample of valuable real-world long-horizon workflows and the tests correctly measure whether an agent has solved them.","fun_headline_variants_meta":{"raw":{"variants":["Agents under 65% on 89 terminal tasks","Frontier models below 65% on hard terminal benchmark","Terminal-Bench 2.0: AI under 65% success rate","89 CLI tasks: AI agents succeed under 65%"]},"model":"grok-4.3","cost_usd":0.00858,"raw_usage":{"total_tokens":3733,"prompt_tokens":548,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":85803000,"prompt_tokens_details":{"text_tokens":548,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3118,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":548,"tokens_out":67,"duration_ms":36764,"temperature":1.0,"reasoning_tokens":3118,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-11T03:32:11.518359+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An agent that solves more than 80 percent of the 89 tasks on the first attempt, using only general capabilities and without any task-specific fine-tuning or example leakage.","supporting_citations":[],"review_version":1}