Pith. sign in

REVIEW 1 major objections 7 minor 1 cited by

UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

T0 review · 1 major / 7 minor · reviewed 2026-07-10 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Framework Choice Beats Model Choice for Real-World AI Agents

desk verdict Solid benchmark, but the headline framework-vs-model claim is not supported by the tables read the letter →

arxiv 2607.08768 v1 pith:3URZWC37 submitted 2026-07-09 cs.CL

Zhekai Chen , Chengqi Duan , Kaiyue Sun , Bohao Li , Yuqing Wang , Manyuan Zhang , Xihui Liu This is my paper
classification cs.CL
keywords agentagentscapabilitiesreal-worldbenchmarkmodelmodelsuniclawbench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a benchmark built around five distinct capabilities—skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination—rather than the usual scenario categories. It runs 400 bilingual tasks inside live Docker containers with real browsers and GUI applications, replacing static answer keys with fine-grained checkpoints scored by a hidden supervisor agent. A three-role closed loop pairs the executor agent with a hidden supervisor and a user simulator, allowing multi-turn feedback without leaking grading criteria. The central finding is that agent framework architecture—how execution trajectories are organized and context is preserved—consistently shapes performance more than the underlying model choice, and that long-context and multimodal tasks emerge as the major bottlenecks for current systems.

What carries the argument

A three-role closed-loop evaluation system comprising an executor agent operating in a live Docker container, a hidden supervisor agent that scores against fine-grained checkpoint rubrics, and a user simulator agent that generates natural feedback based only on visible trajectory and a coarse progress signal. An information firewall isolates the hidden grading criteria from the executor, enabling multi-turn interaction without answer leakage.

What would settle it

If the hidden supervisor's scoring systematically diverges from human judgment on certain task categories or if the checkpoint rubrics fail to capture nuanced correctness, the central claim about framework impact versus model impact could be an artifact of evaluation bias rather than a property of the agents themselves.

Watch

Extended reading notes

Core claim

When proactive agents are evaluated on dynamic, real-world tasks organized by capability rather than scenario, framework design exerts a stronger influence on success than base model choice, while long-context reasoning and multimodal understanding remain the primary failure modes.

Load-bearing premise

The benchmark assumes that an automated LLM-based supervisor can reliably and objectively evaluate complex, multi-step agent trajectories using checkpoint rubrics, and that its scoring fidelity is sufficient to support cross-framework and cross-model comparisons across all 400 tasks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The manuscript introduces UniClawBench, a capability-driven benchmark for proactive agents comprising 400 bilingual real-world tasks organized along five capability dimensions (Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, Cross-Platform Coordination). The benchmark departs from prior work in two main respects: (i) it evaluates agents in live Docker containers with real browsers and GUI applications rather than sandboxed mirrors, and (ii) it introduces a three-role closed-loop evaluation strategy (executor, hidden supervisor, user simulator) that supports multi-turn feedback while maintaining an information firewall between grading criteria and the agent under test. The authors evaluate ten models under a single framework (OpenClaw) and three models across three frameworks (OpenClaw, EDICT, Nanobot), reporting Pass Rate and checkpoint-based Average Score. A 50-sample human agreement study (92% pass/fail agreement, Pearson r=0.71 for scores) is provided to validate the LLM-based supervisor. Code and benchmark are publicly released.

Significance. The benchmark design addresses genuine gaps in the existing literature: the capability-oriented taxonomy is a useful diagnostic lens compared to scenario-based organization, and the three-role closed-loop evaluation with an information firewall is a thoughtful solution to the leakage problem in multi-turn evaluation. The release of code, task definitions, and the Docker-based runtime is a concrete contribution to reproducibility. The reliability study, while small, provides falsifiable evidence for the supervisor's scoring fidelity. The cross-framework comparison is a relatively underexplored axis in agent benchmarking and is welcome. However, the paper's central comparative claim (framework vs. model impact) is not well-supported by the presented data, which limits the significance of the empirical findings.

major comments (1)
  1. The headline claim that 'framework choice consistently impacts capability performance more than model choice' (Figure 1 caption; §4.4) is not supported by the raw data in Tables 1 and 2. In Table 1 (cross-model, all under OpenClaw), overall PR ranges from 0.152 (GPT-4.1) to 0.475 (Claude Opus-4.8), a spread of ~0.32. In Table 2 (cross-framework), the framework spread for each of the three tested models is only 0.084–0.117. Even restricting to the same three models used in the cross-framework study under OpenClaw (GPT-5.4: 0.407, Claude Opus-4.8: 0.475, Kimi-2.6: 0.362), the model spread (~0.11) is comparable to—not smaller than—the framework spread. Per-capability, the discrepancy is starker: for Long Context, the model spread is ~0.38 while the framework spread is 0.075–0.100. The paper provides no variance decomposition, ANOVA, or statistical test to justify 'consistently more.' This结论
minor comments (7)
  1. §4.2: The reliability study reports 92% agreement on pass/fail and Pearson r=0.71 for scores, but does not report inter-annotator agreement among the three human experts. Please report this to contextualize the supervisor's performance.
  2. Table 1: Claude Opus-4.8 has the highest overall PR (0.475) but not the highest overall AS; GPT-5.4 has AS 0.774 vs. Claude's 0.702. This is worth a sentence of discussion.
  3. §4.4: The characterization of EDICT's 'coordination friction' and Nanobot's 'extreme token efficiency' trade-off is qualitative. Quantitative evidence (e.g., handoff failure rates) would make these claims more falsifiable.
  4. Figure 1: The bar chart in the bottom panel visually suggests the framework-vs-model conclusion, but the normalization and grouping are not clearly specified. Please ensure the figure accurately reflects the data.
  5. §3.1: The statement that each task is assigned to the category that 'best captures its primary capability bottleneck' is subjective. Please report inter-annotator agreement on task categorization (or describe how disagreements were resolved).
  6. Appendix D: The end-to-end case study is helpful but cherry-picked (a successful pass). A failed example would strengthen the evaluation pipeline's credibility.
  7. The paper acknowledges limitations (400 tasks, live-environment instability, LLM-based evaluation bias) but does not discuss potential mitigation strategies. Please address this concern.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark metrics are defined by an independent rubric and validated against human judgment; no prediction reduces to its inputs by construction.

full rationale

UniClawBench is an empirical benchmark paper, not a derivation chain. The metrics (Pass Rate, Average Score) are defined by a hidden supervisor applying task-specific checkpoint rubrics (eval_rule.md, Appendix A.3), and the supervisor's reliability is validated against a 50-sample human agreement study (92% pass/fail agreement, Pearson r=0.71, §4.2). The cross-model and cross-framework comparisons (Tables 1-2) use the same evaluation pipeline across all conditions, so no result is forced by a fitted parameter or self-definitional reduction. The headline claim that 'framework choice consistently impacts capability performance more than model choice' (§4.4, Figure 1 caption) is an empirical observation from the data, not a prediction derived from a self-cited theorem or an ansatz. While the skeptic correctly notes that the raw PR spreads in Tables 1-2 do not obviously support the 'consistently more' claim without formal variance decomposition, this is a correctness/evidential concern, not circularity: the claim is not equivalent to its inputs by construction. No step in the paper's methodology reduces to a self-citation chain, a fitted-input-renamed-as-prediction, or a self-definitional identity. The derivation is self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The ledger captures the key design choices that structure the benchmark. The free parameters are thresholds that define pass/fail boundaries. The axioms are domain assumptions about LLM evaluation and task categorization that underpin the benchmark's validity.

free parameters (3)
  • success_threshold = 0.90
    The minimum supervisor score required to pass a task, set as a default in the task YAML template.
  • timeout_seconds = 1200-1800
    Per-turn and global time limits chosen to bound execution, affecting pass rates on complex tasks.
  • max_user_followups = 2
    The maximum number of follow-up cycles allowed, directly impacting the multi-turn recovery opportunity and final pass rate.
assumptions (3)
  • domain assumption LLM-based evaluation fidelity
    The benchmark assumes that an LLM (GPT-5.4) can reliably grade complex agent trajectories against a rubric. This is validated on a small sample (50 tasks) but assumed to hold across all 400 tasks.
  • domain assumption Capability independence
    The taxonomy assumes tasks can be cleanly assigned to a single primary capability bottleneck, though the paper acknowledges auxiliary operations appear naturally.
  • domain assumption Live environment stability
    The evaluation assumes that live web content and APIs remain sufficiently stable during the evaluation window for the checkpoint rubrics to remain valid.
invented entities (1)
  • Three-role closed-loop evaluation independent evidence
    purpose: To simulate multi-turn human-agent interaction without leaking grading criteria.
    The information firewall design is a novel architectural contribution, validated by the end-to-end case study in Appendix D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks." pith.science (2026). https://pith.science/paper/3URZWC37

@misc{pith2026260708768,
  author       = {Pith},
  title        = {Pith review of: UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3URZWC37}},
  note         = {Machine review of arXiv:2607.08768}
}
read the original abstract

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

Figures

Figures reproduced from arXiv: 2607.08768 by the authors.

Figure 1
Figure 1. Overview of UniClawBench. UniClawBench consists of 400 bilingual real-world tasks spanning 5 core capabilities: multimodal, long-context, skill usage, exploration, and cross-platform. We propose a three-role closed-loop evaluation framework, where an executor agent performs tasks in real environments, a supervisor evaluates trajectories and artifacts using hidden rubrics, and a user simulator provides natural feedba… view at source ↗
Figure 2
Figure 2. Three-role Closed-loop Evaluation Strategy. A task is executed by an agent (Executor) within a real-world environment using only public inputs (e.g., tools, files, and web access). The resulting observable trajectory and produced artifacts are assessed by a hidden Supervisor, which ap￾plies fine-grained checkpoint-based rubrics and private references to compute a structured evaluation state (e.g., pass / fail / cont… view at source ↗
Figure 3
Figure 3. Diversity and composition statistics of UniClawBench. The heatmaps detail (a) the broad [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Token Usage and Performance Progression by Cycle. (a) and (b) display the token usage on the OpenClaw system, categorised by model and task dimension, (c) demonstrates the performance progression by cycle motivated by the user follow-ups. The central orchestrator dispa…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A dual Task/Process benchmark with turn-level rubrics filters lucky agent successes and improves post-training when used as a trajectory filter.

Reference graph

Works this paper leans on

81 extracted references · 81 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pinchbench: Real-world benchmarks for ai coding agents

    Kilo AI. Pinchbench: Real-world benchmarks for ai coding agents. https://github.com/ pinchbench/skill, 2026

    work page 2026

  2. [2]

    Kimi k2.6: From code to creation, from one to many

    Kimi AI. Kimi k2.6: From code to creation, from one to many. https://www.kimi.com/ ai-models/kimi-k2-6, 2026

    work page 2026

  3. [3]

    Introducing claude opus 4.6

    Anthropic. Introducing claude opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6, 2026

    work page 2026

  4. [4]

    Introducing claude sonnet 4.6

    Anthropic. Introducing claude sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6, 2026

    work page 2026

  5. [5]

    Openclaw web-search skill

    Billyutw. Openclaw web-search skill. https://clawhub.ai/billyutw/web-search, 2026

    work page 2026

  6. [6]

    Openclaw edict.https://github.com/cft0808/edict, 2026

    cft0808. Openclaw edict.https://github.com/cft0808/edict, 2026

    work page 2026

  7. [7]

    Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588, 2022

    work page Pith review arXiv 2022

  8. [8]

    The BrowserGym Ecosystem for Web Agent Research

    De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F Xu, Siva Reddy, Quentin Cappart, et al. The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467, 2024

    work page Pith review arXiv 2024

Show all 81 references
  1. [9]

    Duckduckgo search openclaw

    Cscsxx606. Duckduckgo search openclaw. https://github.com/cscsxx606/ duckduckgo-search-openclaw, 2026

    2026

  2. [10]

    Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    2024 arXiv

  3. [11]

    Codeplot-cot: Mathematical visual reasoning by thinking with code-driven images

    Chengqi Duan, Kaiyue Sun, Rongyao Fang, Manyuan Zhang, Yan Feng, Ying Luo, Yufang Liu, Ke Wang, Peng Pei, Xunliang Cai, et al. Codeplot-cot: Mathematical visual reasoning by thinking with code-driven images. InProceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2026

  4. [12]

    Gemini 3 flash.https://deepmind.google/models/gemini/flash/, 2025

    Google. Gemini 3 flash.https://deepmind.google/models/gemini/flash/, 2025

    2025

  5. [13]

    Gemini 3.1 flash lite.https://deepmind.google/models/gemini/flash-lite/, 2026

    Google. Gemini 3.1 flash lite.https://deepmind.google/models/gemini/flash-lite/, 2026

    2026

  6. [14]

    Gemini 3.1 pro.https://deepmind.google/models/gemini/pro/, 2026

    Google. Gemini 3.1 pro.https://deepmind.google/models/gemini/pro/, 2026

    2026

  7. [15]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14281–14290, 2024

    2024

  8. [16]

    Macagentbench: Benchmark agents where they actually work — on macos

    JetAstra. Macagentbench: Benchmark agents where they actually work — on macos. https://github.com/JetAstra/MacAgentBench, 2026. 10

    2026

  9. [17]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. InProceedings of the 62nd Annual Meeting of the Associatio...

    2024

  10. [18]

    Agent browser openclaw

    Vercel Labs. Agent browser openclaw. https://github.com/vercel-labs/ agent-browser, 2026

    2026

  11. [19]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    2026 arXiv

  12. [20]

    Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  13. [21]

    Liveclawbench: Benchmarking llm agents on complex, real-world assistant tasks.arXiv preprint arXiv:2604.13072, 2026

    Xiang Long, Li Du, Yilong Xu, Fangcheng Liu, Haoqing Wang, Ning Ding, Ziheng Li, Jianyuan Guo, and Yehui Tang. Liveclawbench: Benchmarking llm agents on complex, real-world assistant tasks.arXiv preprint arXiv:2604.13072, 2026

    2026 arXiv

  14. [22]

    Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362, 2024

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362, 2024

    2024

  15. [23]

    Skillclaw: Let skills evolve collectively with agentic evolver.arXiv preprint arXiv:2604.08377, 2026

    Ziyu Ma, Shidong Yang, Yuxiang Ji, Xucong Wang, Yong Wang, Yiming Hu, Tongwen Huang, and Xiangxiang Chu. Skillclaw: Let skills evolve collectively with agentic evolver.arXiv preprint arXiv:2604.08377, 2026

    2026 arXiv

  16. [24]

    Desktop control openclaw

    Matagul. Desktop control openclaw. https://clawhub.ai/matagul/desktop-control, 2026

    2026

  17. [25]

    Clawmark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents.arXiv preprint arXiv:2604.23781, 2026

    Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, et al. Clawmark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents.arXiv preprint arXiv:2604.23781, 2026

    2026 arXiv

  18. [26]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  19. [27]

    Hermes agent: An agent that grows with you

    NousResearch. Hermes agent: An agent that grows with you. https://github.com/ nousresearch/hermes-agent, 2026

    2026

  20. [28]

    Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/, 2025

    OpenAI. Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/, 2025

    2025

  21. [29]

    Introducing gpt-5.4

    OpenAI. Introducing gpt-5.4. https://openai.com/index/introducing-gpt-5-4/ , 2026

    2026

  22. [30]

    Introducing gpt-5.4 mini and nano

    OpenAI. Introducing gpt-5.4 mini and nano. https://openai.com/zh-Hans-CN/index/ introducing-gpt-5-4-mini-and-nano/, 2026

    2026

  23. [31]

    Art: Automatic multi-step reasoning and tool-use for large language models.arXiv preprint arXiv:2303.09014, 2023

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models.arXiv preprint arXiv:2303.09014, 2023

    2023 arXiv

  24. [32]

    Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

    Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

    2022 arXiv

  25. [33]

    Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    2023 arXiv

  26. [34]

    Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024. 11

    2024 arXiv

  27. [35]

    Nanobot: The ultra-lightweight personal ai agent

    Xubin Ren et al. Nanobot: The ultra-lightweight personal ai agent. https://github.com/ HKUDS/nanobot, 2026

    2026

  28. [36]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

    2023

  29. [37]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180, 2023

    2023

  30. [38]

    Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    2023

  31. [39]

    Openclaw.https://github.com/openclaw/openclaw, 2025

    Peter Steinberger. Openclaw.https://github.com/openclaw/openclaw, 2025

    2025

  32. [40]

    Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    2026 arXiv

  33. [41]

    From assistant to double agent: Formalizing and bench- marking attacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412, 2026

    Yuhang Wang, Feiming Xu, Zheng Lin, Guangyu He, Yuzhe Huang, Haichang Gao, Zhenxing Niu, Shiguo Lian, and Zhaoxiang Liu. From assistant to double agent: Formalizing and bench- marking attacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412, 2026

    2026

  34. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  35. [43]

    Intelligent agents: Theory and practice.The knowledge engineering review, 10(2):115–152, 1995

    Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice.The knowledge engineering review, 10(2):115–152, 1995

    1995

  36. [44]

    Metaclaw: Just talk–an agent that meta-learns and evolves in the wild.arXiv preprint arXiv:2603.17187, 2026

    Peng Xia, Jianwen Chen, Xinyu Yang, Haoqin Tu, Jiaqi Liu, Kaiwen Xiong, Siwei Han, Shi Qiu, Haonian Ji, Yuyin Zhou, et al. Metaclaw: Just talk–an agent that meta-learns and evolves in the wild.arXiv preprint arXiv:2603.17187, 2026

    2026

  37. [45]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Sy...

    2024

  38. [46]

    Theagentcompany: benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    2024 arXiv

  39. [47]

    Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  40. [48]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    2022

  41. [49]

    tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  42. [50]

    Claw-eval: Toward trustworthy evaluation of autonomous agents.arXiv preprint arXiv:2604.06132, 2026

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Toward trustworthy evaluation of autonomous agents.arXiv preprint arXiv:2604.06132, 2026

    2026 arXiv

  43. [51]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025

    2025

  44. [52]

    Clawbench: Can ai agents complete everyday online tasks?arXiv preprint arXiv:2604.08523, 2026

    Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, et al. Clawbench: Can ai agents complete everyday online tasks?arXiv preprint arXiv:2604.08523, 2026. 12

    2026 arXiv

  45. [53]

    Scaling Laws for Neural Language Models

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. 13 A Task Definition Format A UniC...

    2023 arXiv

  46. [54]

    You must find the correct paper yourself and confirm the correct page/figure number; do not recreate the wrong chart. 19

  47. [55]

    The figure should preserve the original axis meanings, logarithmic x-axis, three curves for different model sizes, legend, and overall style

  48. [56]

    draw the chart, compare it with the original, then revise

    Perform at least one "draw the chart, compare it with the original, then revise" iteration instead of producing it in one pass

  49. [57]

    Save the final deliverables under`/tmp_workspace/results/`, including at least: -`figure5_aspect_ratio_recreated.png` -`figure5_aspect_ratio_recreated.py` -`notes.md`(briefly state which paper and figure you confirmed, and what revisions you made) task_22_met_art_room_match Pl...

  50. [58]

    Claude Monet -- https://www.metmuseum.org/art/collection/search/437133

  51. [59]

    Vincent van Gogh -- https://www.metmuseum.org/art/collection/search/436535

  52. [60]

    Caspar David Friedrich -- https://www.metmuseum.org/art/collection/search/438417 Please do the following:

  53. [61]

    If a single screenshot cannot fit both, save an additional supplemental screenshot for the same artwork and mark that clearly in the filename or notes

    Save at least 1 webpage screenshot for each painting to `/tmp_workspace/results/artwork_screenshots/`; the screenshot must include both the main image and a clear metadata area. If a single screenshot cannot fit both, save an additional supplemental screenshot for the same art...

  54. [62]

    Read`/tmp_workspace/clawbench/sources/my_room_style.json`

  55. [63]

    Generate the following in`/tmp_workspace/results/`: -`artworks_metadata.csv` -`visual_analysis.md` -`best_choice_for_my_room.md` 4.`artworks_metadata.csv`must organize at least these fields: - artwork_url, title, artist, object_date, medium, dimensions 5.`visual_analysis.md`mu...

    2025

  56. [64]

    Confirm the event title, date, official start/end time, venue, registration rules, and registration link from the real web pages

  57. [65]

    The title must include `Rust Global: Tokyo`

    Create an event in the real calendar GUI. The title must include `Rust Global: Tokyo`

  58. [66]

    Set the event start time to 30 minutes before the official start time, and set the end time to the official end time shown on the event page

  59. [67]

    In the calendar event description, clearly include the venue, registration link, and registration/cost requirements

  60. [68]

    An ICS import preview, a screen where no calendar has been selected, or a dialog where import/save has not been clicked does not count as a created event

    Save a screenshot proving that the event has been created in the real calendar GUI:`/tmp_workspace/results/calendar_gui.png`; if the calendar supports export, also save 21 `/tmp_workspace/results/rust_global_tokyo.ics`. An ICS import preview, a screen where no calendar has bee...

  61. [69]

    Record the calendar application used, event title, start/end time, timezone, venue, registration link, and evidence file paths

    Also write an audit JSON file to `/tmp_workspace/results/calendar_evidence.json`. Record the calendar application used, event title, start/end time, timezone, venue, registration link, and evidence file paths. If the event is still only pending import, explicitly write`pending...

  62. [70]

    Available Images

    Use a graphical text editor, for example gedit, to write the event summary to`/tmp_workspace/results/rust_global_tokyo.md`. Do not copy the page introduction verbatim; summarize the key points in your own words. C Role Prompts Three prompts drive every cycle: a session wrapper...

  63. [71]

    Read`references/eval_rule.md`as the primary judging spec

  64. [72]

    Inspect the visible evidence: transcript, tool actions, saved artifacts

  65. [73]

    could be more thorough

    Classify the attempt state and assign a score. ## Strict scoring discipline (no rule-invention) You MUST score using ONLY the rubric lines and Section 6 score caps that appear in`references/eval_rule.md`. Do NOT invent additional checkpoints, quality bars, or quality complaint...

  66. [74]

    Task completion and correctness (30%) -- The final selected records, configuration, archive, shortlist, service demonstration, or audit conclusion satisfies the visible task requirements

  67. [75]

    Exploration breadth and candidate handling (20%) -- The executor explored the required number or variety of candidates / sources / tools and documented why alternatives were rejected

  68. [76]

    Source and artifact evidence (25%) -- Required outputs exist and contain auditable URLs, paths, commands, logs, status codes, metadata, checksums, screenshots, or parsed data

  69. [77]

    Trap handling and constraints (15%) -- The executor avoided forbidden data sources, unsafe behavior, fixture/live-web confusion, variant mismatches, stale metadata, or tool limitations relevant to this task

  70. [78]

    Now let me research the LOC API to understand how to find items with clear rights and accessible images

    Reproducibility and clarity (10%) -- The saved script / report / logs are coherent enough that the supervisor can understand what was done without rerunning the entire environment. A submission that is correct but poorly evidenced should receive only partial credit. A submissi...

  71. [79]

    Cycle 1 has the right pipeline and the right data; only the selection rigor and the exclusion-file consistency are insufficient

    The supervisor catches recoverable failure modes rather than rejecting them outright. Cycle 1 has the right pipeline and the right data; only the selection rigor and the exclusion-file consistency are insufficient. Returning continue (not fail) at score = 0.65 preserves the ex...

  72. [80]

    Hidden judging assets are never mounted into the user simulator’s workspace, and a deterministic rewriter further sanitizes the simulator’s candidate text

    The information firewall is structural, not stylistic.The supervisor returns a seven-field decision but only four fields cross to the user simulator (Appendix D.5). Hidden judging assets are never mounted into the user simulator’s workspace, and a deterministic rewriter furthe...

  73. [81]

    UniClawBench scores reflect both the model’s first-pass and its recovery capabilities

    Multi-turn recovery is a core capability.The cycle-1 score of 0.65 would have been a fail under any single-turn paradigm; cycle 2 reaches 0.96 not by patching the saved files but by rebuilding the pipeline from item-level JSON, in the process uncovering two additional issues t...

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.