Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents

T0 review · 2 major / 1 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TUA-Bench evaluates terminal agents on 120 general tasks and finds the top model at 65.8 percent.

desk verdict TUA-Bench adds a terminal benchmark covering routine and expert tasks, but the 65.8% result depends on unvalidated manual task selection. read the letter →

arxiv 2606.28480 v1 pith:2ZP3DHFD submitted 2026-06-26 cs.SE cs.AI

Shoufa Chen , Luyuan Wang , Xuan Yang , Zhiheng Liu , Yuren Cong , Yuanfeng Ji , Feiyan Zhou , Xiaohui Zhang
show 2 more authors
Fanny Yang Belinda Zeng
This is my paper
classification cs.SEcs.AI
keywords terminal-useagentscomputer-usebenchmarkTUA-Benchgeneral-purposeexecution-basedevaluationdocumenteditingscientificworkflowsClaudeOpus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TUA-Bench to measure general-purpose terminal-use agents on tasks that extend past coding into everyday digital work. It supplies 120 manually designed tasks spanning document editing, email management, web information seeking, and PhD-level scientific workflows, each executed in a real terminal with a deterministic setup script. Performance is measured by an execution-based scoring protocol rather than subjective review. The evaluation shows the strongest agent reaches only 65.8 percent overall, with clear shortfalls across task families. This matters because it gives a concrete way to track progress toward agents that can handle diverse terminal environments reliably.

What carries the argument

TUA-Bench benchmark of 120 manually designed tasks with deterministic terminal setups and execution-based scoring protocol.

What would settle it

A new agent that scores above 90 percent on the 120 tasks yet fails on similar but unseen terminal tasks outside the benchmark set would indicate the tasks do not generalize.

Watch

Extended reading notes

Core claim

TUA-Bench consists of 120 real-world tasks across five families for terminal-use agents. The tasks address routine activities such as document editing and email management together with scientific and engineering workflows co-designed with domain experts. Each task runs in a real terminal under a deterministic setup script and receives an execution-based score. The strongest frontier agent, Claude Code powered by Claude Opus 4.8 at maximum reasoning effort, reaches 65.8 percent overall performance with substantial gaps across both routine and expert tracks.

Load-bearing premise

The 120 manually designed tasks and the execution-based scoring protocol together provide a representative and unbiased measure of general-purpose terminal-use agent capabilities.

Editorial extensions

If this is right

  • Terminal agents must close large performance gaps before they can be considered reliable for general use.
  • Routine tasks such as document editing and email management expose limitations not addressed by coding-only benchmarks.
  • Scientific workflows that require specialized software demand agent capabilities beyond standard shell commands.
  • Execution-based scoring offers an objective alternative to human judgment for terminal agent evaluation.
  • Future agent development can target measurable improvement on both routine and expert task families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark could be extended with tasks that involve live state changes or multi-turn user interactions to test robustness further.
  • The observed gaps suggest current agents would benefit from training data that emphasizes non-programming terminal workflows.
  • Pairing TUA-Bench results with existing GUI benchmarks would allow direct comparison of terminal versus graphical computer-use performance.
  • A score of 65.8 percent implies that unsupervised deployment of these agents in varied digital environments remains premature.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces TUA-Bench, a benchmark for general-purpose terminal-use agents (TUAs) consisting of 120 manually designed tasks across five families. Tasks span routine activities (document editing, email, web search) and expert co-designed scientific/engineering workflows, each with deterministic setup scripts and execution-based scoring. The central empirical result is that the strongest frontier agent (Claude Code with Claude Opus 4.8 at max reasoning effort) reaches 65.8% overall success, with substantial gaps across tracks; the benchmark is positioned as more general than prior GUI or shell-focused suites.

Significance. If the task set is representative, TUA-Bench fills a documented gap between GUI-centric computer-use benchmarks and narrow coding/shell benchmarks, supplying a reproducible, execution-scored evaluation that could guide development of broader terminal agents. The deterministic setups and execution-based protocol are explicit strengths that support reproducibility and reduce scoring ambiguity.

major comments (2)
  1. [Abstract] Abstract: the claim that the 65.8% result and observed gaps constitute a 'broad and realistic evaluation' of general-purpose TUA capabilities is load-bearing on the representativeness of the 120 tasks; the abstract supplies no information on task selection process, coverage metrics, inter-rater reliability for task design, or external validation against real-world terminal usage distributions or expert consensus, leaving open the possibility that performance ceilings reflect curation choices rather than intrinsic limitations.
  2. [Section 3] Task construction description (Section 3 / benchmark design): while the five task families and PhD co-design are described, the manuscript provides no quantitative validation (e.g., diversity statistics, overlap with usage logs, or comparison to existing terminal corpora) that would confirm the tasks are unbiased relative to the general-purpose claim; this directly affects whether the 65.8% ceiling can be interpreted as a frontier measurement.
minor comments (1)
  1. [Abstract] Abstract: the model identifier 'Claude Opus 4.8' is not standard; a brief clarification of the exact model/version and reasoning-effort parameterization would improve precision.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for highlighting the importance of justifying the representativeness of TUA-Bench tasks. We address each major comment below and will revise the manuscript to strengthen the description of task construction while being transparent about limitations.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the 65.8% result and observed gaps constitute a 'broad and realistic evaluation' of general-purpose TUA capabilities is load-bearing on the representativeness of the 120 tasks; the abstract supplies no information on task selection process, coverage metrics, inter-rater reliability for task design, or external validation against real-world terminal usage distributions or expert consensus, leaving open the possibility that performance ceilings reflect curation choices rather than intrinsic limitations.

    Authors: We agree that the abstract would benefit from more context on task selection. In the revision we will expand the abstract to state that the 120 tasks were manually designed to cover five families spanning routine digital activities and PhD-co-designed scientific workflows, with deterministic setups and execution-based scoring. Inter-rater reliability metrics were not computed because task design was an iterative collaborative process rather than independent rating. We will also note that external validation against usage distributions was not performed. These additions will make the scope and limitations of the 'broad and realistic' phrasing clearer without overstating the evidence. revision: partial

  2. Referee: [Section 3] Task construction description (Section 3 / benchmark design): while the five task families and PhD co-design are described, the manuscript provides no quantitative validation (e.g., diversity statistics, overlap with usage logs, or comparison to existing terminal corpora) that would confirm the tasks are unbiased relative to the general-purpose claim; this directly affects whether the 65.8% ceiling can be interpreted as a frontier measurement.

    Authors: We will revise Section 3 to include quantitative diversity statistics, such as the breakdown of tasks by family, command categories, and estimated complexity. We will also add a limitations paragraph explicitly discussing the absence of direct comparison to public terminal usage logs or corpora. The task set was constructed through author expertise supplemented by PhD-level domain experts to target both everyday and specialized terminal activities; however, no suitable public corpora existed for quantitative overlap analysis. This revision will allow readers to evaluate the general-purpose claim more precisely while preserving the benchmark's contribution. revision: partial

standing simulated objections not resolved
  • Direct quantitative overlap or comparison against real-world terminal usage distributions or existing corpora, as no appropriate public datasets were identified for such validation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmark with direct measurements

full rationale

The paper introduces TUA-Bench as a new set of 120 manually designed tasks with deterministic setup scripts and execution-based scoring. No equations, fitted parameters, or predictions are derived from prior quantities; reported performance (e.g., 65.8% for Claude Code) consists of direct empirical measurements on the introduced tasks. No self-citations serve as load-bearing premises for any result, and the construction does not reduce any claimed outcome to its own inputs by definition. The work is self-contained as a benchmark proposal.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Benchmark introduction paper containing no mathematical derivations, fitted parameters, or postulated entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents." pith.science (2026). https://pith.science/paper/2ZP3DHFD

@misc{pith2026260628480,
  author       = {Pith},
  title        = {Pith review of: TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZP3DHFD}},
  note         = {Machine review of arXiv:2606.28480}
}
read the original abstract

As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Across 1,794 CLI agent trajectories, failures are mostly epistemic, start by median step 7, and often stay silent until after lock-in.

Reference graph

Works this paper leans on

90 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    International Conference on Learning Representations , volume=

    Openhands: An open platform for ai software developers as generalist agents , author=. International Conference on Learning Representations , volume=

  2. [2]

    Advances in Neural Information Processing Systems , volume=

    Swe-agent: Agent-computer interfaces enable automated software engineering , author=. Advances in Neural Information Processing Systems , volume=

  3. [3]

    The twelfth international conference on learning representations , year=

    Swe-bench: Can language models resolve real-world github issues? , author=. The twelfth international conference on learning representations , year=

  4. [4]

    The Fourteenth International Conference on Learning Representations , year=

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. The Fourteenth International Conference on Learning Representations , year=

  5. [5]

    Advances in Neural Information Processing Systems , volume=

    macosworld: A multilingual interactive benchmark for gui agents , author=. Advances in Neural Information Processing Systems , volume=

  6. [6]

    2026 , url=

    Hongrui Jia and Jitong Liao and Xi Zhang and Haiyang Xu and Tianbao Xie and Chaoya Jiang and Ming Yan and Si Liu and Wei Ye and Fei Huang , booktitle=. 2026 , url=

    2026

  7. [7]

    The Twelfth International Conference on Learning Representations , year=

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author=. The Twelfth International Conference on Learning Representations , year=

  8. [8]

    2022 , month = nov, howpublished =

    Introducing ChatGPT , author =. 2022 , month = nov, howpublished =

    2022

Show all 90 references
  1. [9]

    2026 , month = apr, day =

    Introducing. 2026 , month = apr, day =

    2026

  2. [10]

    2025 , month = may, howpublished =

    Introducing Codex , author =. 2025 , month = may, howpublished =

    2025

  3. [11]

    2026 , howpublished =

    Claude Code , author =. 2026 , howpublished =

    2026

  4. [12]

    2026 , howpublished =

    Introducing Claude Opus 4.7 , author =. 2026 , howpublished =

    2026

  5. [13]

    2026 , howpublished =

    Gemini 3.1 Pro: A smarter model for your most complex tasks , author =. 2026 , howpublished =

    2026

  6. [14]

    2026 , howpublished =

    OpenCode: A Powerful AI Coding Agent Built for the Terminal , author =. 2026 , howpublished =

    2026

  7. [15]

    2023 , month = mar, howpublished =

    Introducing Claude , author =. 2023 , month = mar, howpublished =

    2023

  8. [16]

    2026 , howpublished =

    2026

  9. [17]

    2025 , month = jun, howpublished =

    Gemini CLI: Your Open-Source AI Agent , author =. 2025 , month = jun, howpublished =

    2025

  10. [18]

    2021 , month = jun, howpublished =

    Introducing GitHub Copilot: Your AI Pair Programmer , author =. 2021 , month = jun, howpublished =

    2021

  11. [19]

    2026 , howpublished =

    Cursor , author =. 2026 , howpublished =

    2026

  12. [20]

    2025 , month = mar, howpublished =

    Manus , author =. 2025 , month = mar, howpublished =

    2025

  13. [21]

    2026 , howpublished =

    OpenClaw: Personal AI Assistant , author =. 2026 , howpublished =

    2026

  14. [22]

    The UNIX

    Kernighan, Brian W and Mashey, John R , journal=. The UNIX. 1979 , publisher=

    1979

  15. [23]

    2014 , publisher=

    Data science at the command line: Facing the future with time-tested tools , author=. 2014 , publisher=

    2014

  16. [24]

    Gigascience , volume=

    Tools and techniques for computational reproducibility , author=. Gigascience , volume=. 2016 , publisher=

    2016

  17. [25]

    GitHub repository , howpublished =

    OpenCLI Contributors , title =. GitHub repository , howpublished =. 2026 , publisher =

    2026

  18. [27]

    Advances in Neural Information Processing Systems , volume=

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume=

  19. [28]

    2026 , howpublished =

    lark-cli: The Official Lark/Feishu CLI Tool , author =. 2026 , howpublished =

    2026

  20. [29]

    2026 , howpublished =

    Podman: The Best Free and Open Source Container Tools , author =. 2026 , howpublished =

    2026

  21. [30]

    International Conference on Machine Learning , pages=

    World of bits: An open-domain platform for web-based agents , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  22. [31]

    International Conference on Learning Representations , year=

    Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration , author=. International Conference on Learning Representations , year=

  23. [32]

    Advances in Neural Information Processing Systems , volume=

    Mind2web: Towards a generalist agent for the web , author=. Advances in Neural Information Processing Systems , volume=

  24. [33]

    2025 USENIX Annual Technical Conference (USENIX ATC 25) , pages=

    The Koala benchmarks for the shell: characterization and implications , author=. 2025 USENIX Annual Technical Conference (USENIX ATC 25) , pages=

    2025

  25. [36]

    LLM-supported natural language to bash translation , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  26. [38]

    International Conference on Learning Representations , volume=

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery , author=. International Conference on Learning Representations , volume=

  27. [41]

    The Fourteenth International Conference on Learning Representations , year=

    The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution , author=. The Fourteenth International Conference on Learning Representations , year=

  28. [44]

    Windows Agent Arena: Evaluating Multi-Modal

    Rogerio Bonatti and Dan Zhao and Francesco Bonacci and Dillon Dupont and Sara Abdali and Yinheng Li and Yadong Lu and Justin Wagle and Kazuhito Koishida and Arthur Bucker and Lawrence Keunho Jang and Zheng Hui , booktitle=. Windows Agent Arena: Evaluating Multi-Modal. 2025 , url=

    2025

  29. [45]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Webvoyager: Building an end-to-end web agent with large multimodal models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  30. [47]

    Introducing claude

    Anthropic . Introducing claude. https://www.anthropic.com/news/introducing-claude, March 2023

    2023

  31. [48]

    Claude code

    Anthropic . Claude code. https://www.anthropic.com/claude-code, 2026 a

    2026

  32. [49]

    Introducing claude opus 4.7

    Anthropic . Introducing claude opus 4.7. https://www.anthropic.com/news/claude-opus-4-7, 2026 b

    2026

  33. [50]

    Anysphere . Cursor. https://cursor.com/, 2026

    2026

  34. [51]

    Setupbench: Assessing software engineering agents' ability to bootstrap development environments

    Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing software engineering agents' ability to bootstrap development environments. arXiv preprint arXiv:2507.09063, 2025

    2025

  35. [52]

    Terminal wrench: A dataset of 331 reward-hackable environments and 3,632 exploit trajectories

    Ivan Bercovich, Ivgeni Segal, Kexun Zhang, Shashwat Saxena, Aditi Raghunathan, and Ziqian Zhong. Terminal wrench: A dataset of 331 reward-hackable environments and 3,632 exploit trajectories. arXiv preprint arXiv:2604.17596, 2026

    2026 arXiv

  36. [53]

    Windows agent arena: Evaluating multi-modal OS agents at scale

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Keunho Jang, and Zheng Hui. Windows agent arena: Evaluating multi-modal OS agents at scale. In Forty-second International C...

    2025

  37. [54]

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representa...

    2025

  38. [55]

    Terminalworld: Benchmarking agents on real-world terminal tasks

    Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O'Hearn, Earl T Barr, Mark Harman, Federica Sarro, et al. Terminalworld: Benchmarking agents on real-world terminal tasks. arXiv preprint arXiv:2605.22535, 2026

    2026 arXiv

  39. [56]

    OpenCLI : Make any website your CLI --- an AI -native runtime for browser automation and dynamic web data extraction

    OpenCLI Contributors. OpenCLI : Make any website your CLI --- an AI -native runtime for browser automation and dynamic web data extraction. https://github.com/jackwener/opencli, 2026

    2026

  40. [57]

    Osuniverse: Benchmark for multimodal gui-navigation ai agents

    Mariya Davydova, Daniel Jeffries, Patrick Barker, Arturo M \'a rquez Flores, and Sin \'e ad Ryan. Osuniverse: Benchmark for multimodal gui-navigation ai agents. arXiv preprint arXiv:2505.03570, 2025

    2025

  41. [58]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36: 0 28091--28114, 2023

    2023

  42. [59]

    Introducing github copilot: Your ai pair programmer

    GitHub . Introducing github copilot: Your ai pair programmer. https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/, June 2021

    2021

  43. [60]

    GitHub CLI Documentation

    GitHub . GitHub CLI Documentation . https://docs.github.com/en/github-cli, 2026

    2026

  44. [61]

    Gemini cli: Your open-source ai agent

    Google . Gemini cli: Your open-source ai agent. https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemini-cli-open-source-ai-agent/, June 2025

    2025

  45. [62]

    gcloud CLI Overview

    Google Cloud . gcloud CLI Overview . https://docs.cloud.google.com/sdk/gcloud, 2026

    2026

  46. [63]

    Terminus-2: Harbor's High-Performance Reference Agent Implementation

    Harbor . Terminus-2: Harbor's High-Performance Reference Agent Implementation . https://www.harborframework.com/docs/agents/terminus-2, 2026

    2026

  47. [64]

    Harbor: A framework for evaluating and optimizing agents and models in container environments , January 2026

    Harbor Framework Team . Harbor: A framework for evaluating and optimizing agents and models in container environments , January 2026. https://github.com/harbor-framework/harbor

    2026

  48. [65]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2024

  49. [66]

    O'Reilly Media, Inc

    Jeroen Janssens. Data science at the command line: Facing the future with time-tested tools. " O'Reilly Media, Inc.", 2014

    2014

  50. [67]

    OSW orld- MCP : Benchmarking MCP tool invocation in computer-use agents

    Hongrui Jia, Jitong Liao, Xi Zhang, Haiyang Xu, Tianbao Xie, Chaoya Jiang, Ming Yan, Si Liu, Wei Ye, and Fei Huang. OSW orld- MCP : Benchmarking MCP tool invocation in computer-use agents. In The Fourteenth International Conference on Learning Representations, 2026. https://op...

    2026

  51. [68]

    Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023

    2023

  52. [69]

    The unix programming environment

    Brian W Kernighan and John R Mashey. The unix programming environment. Software: Practice and Experience, 9 0 (1): 0 1--15, 1979

    1979

  53. [70]

    Process-level trajectory evaluation for environment configuration in software engineering agents

    Jiayi Kuang, Yinghui Li, Xin Zhang, Yangning Li, Di Yin, Xing Sun, Ying Shen, and Philip S Yu. Process-level trajectory evaluation for environment configuration in software engineering agents. arXiv preprint arXiv:2510.25694, 2025

    2025

  54. [71]

    The koala benchmarks for the shell: characterization and implications

    Evangelos Lamprou, Ethan Williams, Georgios Kaoukis, Zhuoxuan Zhang, Michael Greenberg, Konstantinos Kallas, Lukas Lazarek, and Nikos Vasilakis. The koala benchmarks for the shell: characterization and implications. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), p...

    2025

  55. [72]

    lark-cli: The official lark/feishu cli tool

    LarkSuite . lark-cli: The official lark/feishu cli tool. https://github.com/larksuite/cli, 2026

    2026

  56. [73]

    The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution

    Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig, and Junxian He. The tool de...

    2026

  57. [74]

    Reinforcement learning on web interfaces using workflow-guided exploration

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018. https://openreview.net/forum?id=ryTp3f-0-

    2018

  58. [75]

    Mcp-universe: Benchmarking large language models with real-world model context protocol servers

    Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. Mcp-universe: Benchmarking large language models with real-world model context protocol servers. arXiv preprint arXiv:2508.14704, 2025

    2025

  59. [76]

    Manus . Manus. https://manus.im/, March 2025

    2025

  60. [77]

    Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, ...

    2026

  61. [78]

    Kimi Code : Next-gen ai code agent

    Moonshot AI . Kimi Code : Next-gen ai code agent. https://www.kimi.com/code, 2026

    2026

  62. [79]

    Introducing chatgpt

    OpenAI . Introducing chatgpt. https://openai.com/index/chatgpt/, November 2022

    2022

  63. [80]

    Introducing codex

    OpenAI . Introducing codex. https://openai.com/index/introducing-codex/, May 2025

    2025

  64. [81]

    Introducing GPT-5.5

    OpenAI . Introducing GPT-5.5 . https://openai.com/index/introducing-gpt-5-5/, April 2026

    2026

  65. [82]

    Openclaw: Personal ai assistant

    OpenClaw . Openclaw: Personal ai assistant. https://openclaw.ai/, 2026

    2026

  66. [83]

    Opencode: A powerful ai coding agent built for the terminal

    OpenCode . Opencode: A powerful ai coding agent built for the terminal. https://github.com/opencode-ai/opencode, 2026

    2026

  67. [84]

    Gdpval: Evaluating ai model performance on real-world economically valuable tasks

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Sim \'o n Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. arXiv preprint arXi...

    2025 arXiv

  68. [85]

    Tools and techniques for computational reproducibility

    Stephen R Piccolo and Michael B Frampton. Tools and techniques for computational reproducibility. Gigascience, 5 0 (1): 0 s13742--016, 2016

    2016

  69. [86]

    Podman: The best free and open source container tools

    Podman Container Tools . Podman: The best free and open source container tools. https://podman.io/, 2026

    2026

  70. [87]

    Qwen Code : An open-source ai coding agent that lives in your terminal

    QwenLM . Qwen Code : An open-source ai coding agent that lives in your terminal. https://github.com/QwenLM/qwen-code, 2026

    2026

  71. [88]

    World of bits: An open-domain platform for web-based agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135--3144. PMLR, 2017

    2017

  72. [89]

    Slack CLI

    Slack . Slack CLI . https://docs.slack.dev/tools/slack-cli/, 2026

    2026

  73. [90]

    Gemini 3.1 pro: A smarter model for your most complex tasks

    The Gemini Team . Gemini 3.1 pro: A smarter model for your most complex tasks. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/, 2026

    2026

  74. [91]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2...

    2025

  75. [92]

    Officebench: Benchmarking language agents across multiple applications for office automation

    Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation. arXiv preprint arXiv:2407.19056, 2024

    2024

  76. [93]

    Llm-supported natural language to bash translation

    Finnian Westenfelder, Erik Hemberg, Stephen Moskal, Una-May O'Reilly, and Silviu Chiricescu. Llm-supported natural language to bash translation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human...

    2025

  77. [94]

    MiMo Code : Scaling coding agents to long-horizon tasks

    Xiaomi MiMo Team . MiMo Code : Scaling coding agents to long-horizon tasks. https://mimo.xiaomi.com/coder, 2026

    2026

  78. [95]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  79. [96]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37: 0 50528--50652, 2024

    2024

  80. [97]

    macosworld: A multilingual interactive benchmark for gui agents

    Pei Yang, Hai Ci, and Mike Zheng Shou. macosworld: A multilingual interactive benchmark for gui agents. Advances in Neural Information Processing Systems, 38: 0 134014--134056, 2026 a

    2026

  81. [98]

    Cli-anything: Towards agent-native computer use

    Yuhao Yang, Tianyu Fan, and Chao Huang. Cli-anything: Towards agent-native computer use. arXiv preprint arXiv:2606.03854, 2026 b

    2026 arXiv

  82. [99]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learni...

    2024

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.