Pith. sign in

REVIEW 1 major objections 7 minor 81 references

Framework Choice Beats Model Choice for Real-World AI Agents

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 01:29 UTC pith:3URZWC37

load-bearing objection Solid benchmark, but the headline framework-vs-model claim is not supported by the tables the 1 major comments →

arxiv 2607.08768 v1 pith:3URZWC37 submitted 2026-07-09 cs.CL

UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

Zhekai Chen , Chengqi Duan , Kaiyue Sun , Bohao Li , Yuqing Wang , Manyuan Zhang , Xihui Liu This is my paper
classification cs.CL
keywords agentagentscapabilitiesreal-worldbenchmarkmodelmodelsuniclawbench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a benchmark built around five distinct capabilities—skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination—rather than the usual scenario categories. It runs 400 bilingual tasks inside live Docker containers with real browsers and GUI applications, replacing static answer keys with fine-grained checkpoints scored by a hidden supervisor agent. A three-role closed loop pairs the executor agent with a hidden supervisor and a user simulator, allowing multi-turn feedback without leaking grading criteria. The central finding is that agent framework architecture—how execution trajectories are organized and context is preserved—consistently shapes performance more than the underlying model choice, and that long-context and multimodal tasks emerge as the major bottlenecks for current systems.

Core claim

When proactive agents are evaluated on dynamic, real-world tasks organized by capability rather than scenario, framework design exerts a stronger influence on success than base model choice, while long-context reasoning and multimodal understanding remain the primary failure modes.

What carries the argument

A three-role closed-loop evaluation system comprising an executor agent operating in a live Docker container, a hidden supervisor agent that scores against fine-grained checkpoint rubrics, and a user simulator agent that generates natural feedback based only on visible trajectory and a coarse progress signal. An information firewall isolates the hidden grading criteria from the executor, enabling multi-turn interaction without answer leakage.

Load-bearing premise

The benchmark assumes that an automated LLM-based supervisor can reliably and objectively evaluate complex, multi-step agent trajectories using checkpoint rubrics, and that its scoring fidelity is sufficient to support cross-framework and cross-model comparisons across all 400 tasks.

What would settle it

If the hidden supervisor's scoring systematically diverges from human judgment on certain task categories or if the checkpoint rubrics fail to capture nuanced correctness, the central claim about framework impact versus model impact could be an artifact of evaluation bias rather than a property of the agents themselves.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The manuscript introduces UniClawBench, a capability-driven benchmark for proactive agents comprising 400 bilingual real-world tasks organized along five capability dimensions (Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, Cross-Platform Coordination). The benchmark departs from prior work in two main respects: (i) it evaluates agents in live Docker containers with real browsers and GUI applications rather than sandboxed mirrors, and (ii) it introduces a three-role closed-loop evaluation strategy (executor, hidden supervisor, user simulator) that supports multi-turn feedback while maintaining an information firewall between grading criteria and the agent under test. The authors evaluate ten models under a single framework (OpenClaw) and three models across three frameworks (OpenClaw, EDICT, Nanobot), reporting Pass Rate and checkpoint-based Average Score. A 50-sample human agreement study (92% pass/fail agreement, Pearson r=0.71 for scores) is provided to validate the LLM-based supervisor. Code and benchmark are publicly released.

Significance. The benchmark design addresses genuine gaps in the existing literature: the capability-oriented taxonomy is a useful diagnostic lens compared to scenario-based organization, and the three-role closed-loop evaluation with an information firewall is a thoughtful solution to the leakage problem in multi-turn evaluation. The release of code, task definitions, and the Docker-based runtime is a concrete contribution to reproducibility. The reliability study, while small, provides falsifiable evidence for the supervisor's scoring fidelity. The cross-framework comparison is a relatively underexplored axis in agent benchmarking and is welcome. However, the paper's central comparative claim (framework vs. model impact) is not well-supported by the presented data, which limits the significance of the empirical findings.

major comments (1)
  1. The headline claim that 'framework choice consistently impacts capability performance more than model choice' (Figure 1 caption; §4.4) is not supported by the raw data in Tables 1 and 2. In Table 1 (cross-model, all under OpenClaw), overall PR ranges from 0.152 (GPT-4.1) to 0.475 (Claude Opus-4.8), a spread of ~0.32. In Table 2 (cross-framework), the framework spread for each of the three tested models is only 0.084–0.117. Even restricting to the same three models used in the cross-framework study under OpenClaw (GPT-5.4: 0.407, Claude Opus-4.8: 0.475, Kimi-2.6: 0.362), the model spread (~0.11) is comparable to—not smaller than—the framework spread. Per-capability, the discrepancy is starker: for Long Context, the model spread is ~0.38 while the framework spread is 0.075–0.100. The paper provides no variance decomposition, ANOVA, or statistical test to justify 'consistently more.' This结论
minor comments (7)
  1. §4.2: The reliability study reports 92% agreement on pass/fail and Pearson r=0.71 for scores, but does not report inter-annotator agreement among the three human experts. Please report this to contextualize the supervisor's performance.
  2. Table 1: Claude Opus-4.8 has the highest overall PR (0.475) but not the highest overall AS; GPT-5.4 has AS 0.774 vs. Claude's 0.702. This is worth a sentence of discussion.
  3. §4.4: The characterization of EDICT's 'coordination friction' and Nanobot's 'extreme token efficiency' trade-off is qualitative. Quantitative evidence (e.g., handoff failure rates) would make these claims more falsifiable.
  4. Figure 1: The bar chart in the bottom panel visually suggests the framework-vs-model conclusion, but the normalization and grouping are not clearly specified. Please ensure the figure accurately reflects the data.
  5. §3.1: The statement that each task is assigned to the category that 'best captures its primary capability bottleneck' is subjective. Please report inter-annotator agreement on task categorization (or describe how disagreements were resolved).
  6. Appendix D: The end-to-end case study is helpful but cherry-picked (a successful pass). A failed example would strengthen the evaluation pipeline's credibility.
  7. The paper acknowledges limitations (400 tasks, live-environment instability, LLM-based evaluation bias) but does not discuss potential mitigation strategies. Please address this concern.

Circularity Check

0 steps flagged

No circularity: benchmark metrics are defined by an independent rubric and validated against human judgment; no prediction reduces to its inputs by construction.

full rationale

UniClawBench is an empirical benchmark paper, not a derivation chain. The metrics (Pass Rate, Average Score) are defined by a hidden supervisor applying task-specific checkpoint rubrics (eval_rule.md, Appendix A.3), and the supervisor's reliability is validated against a 50-sample human agreement study (92% pass/fail agreement, Pearson r=0.71, §4.2). The cross-model and cross-framework comparisons (Tables 1-2) use the same evaluation pipeline across all conditions, so no result is forced by a fitted parameter or self-definitional reduction. The headline claim that 'framework choice consistently impacts capability performance more than model choice' (§4.4, Figure 1 caption) is an empirical observation from the data, not a prediction derived from a self-cited theorem or an ansatz. While the skeptic correctly notes that the raw PR spreads in Tables 1-2 do not obviously support the 'consistently more' claim without formal variance decomposition, this is a correctness/evidential concern, not circularity: the claim is not equivalent to its inputs by construction. No step in the paper's methodology reduces to a self-citation chain, a fitted-input-renamed-as-prediction, or a self-definitional identity. The derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The ledger captures the key design choices that structure the benchmark. The free parameters are thresholds that define pass/fail boundaries. The axioms are domain assumptions about LLM evaluation and task categorization that underpin the benchmark's validity.

free parameters (3)
  • success_threshold = 0.90
    The minimum supervisor score required to pass a task, set as a default in the task YAML template.
  • timeout_seconds = 1200-1800
    Per-turn and global time limits chosen to bound execution, affecting pass rates on complex tasks.
  • max_user_followups = 2
    The maximum number of follow-up cycles allowed, directly impacting the multi-turn recovery opportunity and final pass rate.
axioms (3)
  • domain assumption LLM-based evaluation fidelity
    The benchmark assumes that an LLM (GPT-5.4) can reliably grade complex agent trajectories against a rubric. This is validated on a small sample (50 tasks) but assumed to hold across all 400 tasks.
  • domain assumption Capability independence
    The taxonomy assumes tasks can be cleanly assigned to a single primary capability bottleneck, though the paper acknowledges auxiliary operations appear naturally.
  • domain assumption Live environment stability
    The evaluation assumes that live web content and APIs remain sufficiently stable during the evaluation window for the checkpoint rubrics to remain valid.
invented entities (1)
  • Three-role closed-loop evaluation independent evidence
    purpose: To simulate multi-turn human-agent interaction without leaking grading criteria.
    The information firewall design is a novel architectural contribution, validated by the end-to-end case study in Appendix D.

pith-pipeline@v1.1.0-glm · 31869 in / 1931 out tokens · 195168 ms · 2026-07-10T01:29:07.253637+00:00 · methodology

0 comments
read the original abstract

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

Figures

Figures reproduced from arXiv: 2607.08768 by Bohao Li, Chengqi Duan, Kaiyue Sun, Manyuan Zhang, Xihui Liu, Yuqing Wang, Zhekai Chen.

Figure 1
Figure 1. Figure 1: Overview of UniClawBench. UniClawBench consists of 400 bilingual real-world tasks spanning 5 core capabilities: multimodal, long-context, skill usage, exploration, and cross-platform. We propose a three-role closed-loop evaluation framework, where an executor agent performs tasks in real environments, a supervisor evaluates trajectories and artifacts using hidden rubrics, and a user simulator provides natu… view at source ↗
Figure 2
Figure 2. Figure 2: Three-role Closed-loop Evaluation Strategy. A task is executed by an agent (Executor) within a real-world environment using only public inputs (e.g., tools, files, and web access). The resulting observable trajectory and produced artifacts are assessed by a hidden Supervisor, which ap￾plies fine-grained checkpoint-based rubrics and private references to compute a structured evaluation state (e.g., pass / f… view at source ↗
Figure 3
Figure 3. Figure 3: Diversity and composition statistics of UniClawBench. The heatmaps detail (a) the broad [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Token Usage and Performance Progression by Cycle. (a) and (b) display the token usage on the OpenClaw system, categorised by model and task dimension, (c) demonstrates the performance progression by cycle motivated by the user follow-ups. The central orchestrator dispatches tasks based on discrete state polling (e.g., Kanban boards) but lacks real-time supervision over downstream sub-agents. If a sub-agent… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 81 canonical work pages · 19 internal anchors

  1. [1]

    Pinchbench: Real-world benchmarks for ai coding agents

    Kilo AI. Pinchbench: Real-world benchmarks for ai coding agents. https://github.com/ pinchbench/skill, 2026

    work page 2026

  2. [2]

    Kimi k2.6: From code to creation, from one to many

    Kimi AI. Kimi k2.6: From code to creation, from one to many. https://www.kimi.com/ ai-models/kimi-k2-6, 2026

    work page 2026

  3. [3]

    Introducing claude opus 4.6

    Anthropic. Introducing claude opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6, 2026

    work page 2026

  4. [4]

    Introducing claude sonnet 4.6

    Anthropic. Introducing claude sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6, 2026

    work page 2026

  5. [5]

    Openclaw web-search skill

    Billyutw. Openclaw web-search skill. https://clawhub.ai/billyutw/web-search, 2026

    work page 2026

  6. [6]

    Openclaw edict.https://github.com/cft0808/edict, 2026

    cft0808. Openclaw edict.https://github.com/cft0808/edict, 2026

    work page 2026

  7. [7]

    Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  8. [8]

    The BrowserGym Ecosystem for Web Agent Research

    De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F Xu, Siva Reddy, Quentin Cappart, et al. The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Duckduckgo search openclaw

    Cscsxx606. Duckduckgo search openclaw. https://github.com/cscsxx606/ duckduckgo-search-openclaw, 2026

    work page 2026

  10. [10]

    WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks?arXiv preprint arXiv:2403.07718, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Codeplot-cot: Mathematical visual reasoning by thinking with code-driven images

    Chengqi Duan, Kaiyue Sun, Rongyao Fang, Manyuan Zhang, Yan Feng, Ying Luo, Yufang Liu, Ke Wang, Peng Pei, Xunliang Cai, et al. Codeplot-cot: Mathematical visual reasoning by thinking with code-driven images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9586–9596, 2026

    work page 2026

  12. [12]

    Gemini 3 flash.https://deepmind.google/models/gemini/flash/, 2025

    Google. Gemini 3 flash.https://deepmind.google/models/gemini/flash/, 2025

    work page 2025

  13. [13]

    Gemini 3.1 flash lite.https://deepmind.google/models/gemini/flash-lite/, 2026

    Google. Gemini 3.1 flash lite.https://deepmind.google/models/gemini/flash-lite/, 2026

    work page 2026

  14. [14]

    Gemini 3.1 pro.https://deepmind.google/models/gemini/pro/, 2026

    Google. Gemini 3.1 pro.https://deepmind.google/models/gemini/pro/, 2026

    work page 2026

  15. [15]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14281–14290, 2024

    work page 2024

  16. [16]

    Macagentbench: Benchmark agents where they actually work — on macos

    JetAstra. Macagentbench: Benchmark agents where they actually work — on macos. https://github.com/JetAstra/MacAgentBench, 2026. 10

    work page 2026

  17. [17]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881–905, 2024

    work page 2024

  18. [18]

    Agent browser openclaw

    Vercel Labs. Agent browser openclaw. https://github.com/vercel-labs/ agent-browser, 2026

    work page 2026

  19. [19]

    SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

    Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [20]

    AgentBench: Evaluating LLMs as Agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks

    Xiang Long, Li Du, Yilong Xu, Fangcheng Liu, Haoqing Wang, Ning Ding, Ziheng Li, Jianyuan Guo, and Yehui Tang. Liveclawbench: Benchmarking llm agents on complex, real-world assistant tasks.arXiv preprint arXiv:2604.13072, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [22]

    Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362, 2024

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362, 2024

    work page 2024

  23. [23]

    SkillClaw: Let Skills Evolve Collectively with Agentic Evolver

    Ziyu Ma, Shidong Yang, Yuxiang Ji, Xucong Wang, Yong Wang, Yiming Hu, Tongwen Huang, and Xiangxiang Chu. Skillclaw: Let skills evolve collectively with agentic evolver.arXiv preprint arXiv:2604.08377, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  24. [24]

    Desktop control openclaw

    Matagul. Desktop control openclaw. https://clawhub.ai/matagul/desktop-control, 2026

    work page 2026

  25. [25]

    ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

    Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, et al. Clawmark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents.arXiv preprint arXiv:2604.23781, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  26. [26]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  27. [27]

    Hermes agent: An agent that grows with you

    NousResearch. Hermes agent: An agent that grows with you. https://github.com/ nousresearch/hermes-agent, 2026

    work page 2026

  28. [28]

    Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/, 2025

    OpenAI. Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/, 2025

    work page 2025

  29. [29]

    Introducing gpt-5.4

    OpenAI. Introducing gpt-5.4. https://openai.com/index/introducing-gpt-5-4/ , 2026

    work page 2026

  30. [30]

    Introducing gpt-5.4 mini and nano

    OpenAI. Introducing gpt-5.4 mini and nano. https://openai.com/zh-Hans-CN/index/ introducing-gpt-5-4-mini-and-nano/, 2026

    work page 2026

  31. [31]

    ART: Automatic multi-step reasoning and tool-use for large language models

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models.arXiv preprint arXiv:2303.09014, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [32]

    TALM: Tool Augmented Language Models

    Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  33. [33]

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Nanobot: The ultra-lightweight personal ai agent

    Xubin Ren et al. Nanobot: The ultra-lightweight personal ai agent. https://github.com/ HKUDS/nanobot, 2026

    work page 2026

  36. [36]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

    work page 2023

  37. [37]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180, 2023

    work page 2023

  38. [38]

    Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    work page 2023

  39. [39]

    Openclaw.https://github.com/openclaw/openclaw, 2025

    Peter Steinberger. Openclaw.https://github.com/openclaw/openclaw, 2025

    work page 2025

  40. [40]

    Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  41. [41]

    From assistant to double agent: Formalizing and bench- marking attacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412, 2026

    Yuhang Wang, Feiming Xu, Zheng Lin, Guangyu He, Yuzhe Huang, Haichang Gao, Zhenxing Niu, Shiguo Lian, and Zhaoxiang Liu. From assistant to double agent: Formalizing and bench- marking attacks on openclaw for personalized local ai agent.arXiv preprint arXiv:2602.08412, 2026

    work page arXiv 2026

  42. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    work page 2022

  43. [43]

    Intelligent agents: Theory and practice.The knowledge engineering review, 10(2):115–152, 1995

    Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice.The knowledge engineering review, 10(2):115–152, 1995

    work page 1995

  44. [44]

    Metaclaw: Just talk–an agent that meta-learns and evolves in the wild.arXiv preprint arXiv:2603.17187, 2026

    Peng Xia, Jianwen Chen, Xinyu Yang, Haoqin Tu, Jiaqi Liu, Kaiwen Xiong, Siwei Han, Shi Qiu, Haonian Ji, Yuyin Zhou, et al. Metaclaw: Just talk–an agent that meta-learns and evolves in the wild.arXiv preprint arXiv:2603.17187, 2026

    work page arXiv 2026

  45. [45]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    work page 2024

  46. [46]

    TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

    Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [48]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    work page 2022

  49. [49]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  50. [50]

    Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Toward trustworthy evaluation of autonomous agents.arXiv preprint arXiv:2604.06132, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  51. [51]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025

    work page 2025

  52. [52]

    ClawBench: Can AI Agents Complete Everyday Online Tasks?

    Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, et al. Clawbench: Can ai agents complete everyday online tasks?arXiv preprint arXiv:2604.08523, 2026. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  53. [53]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. 13 A Task Definition Format A UniClawBench task is generally defined by two files: a single task declaration writt...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  54. [54]

    You must find the correct paper yourself and confirm the correct page/figure number; do not recreate the wrong chart. 19

    work page

  55. [55]

    The figure should preserve the original axis meanings, logarithmic x-axis, three curves for different model sizes, legend, and overall style

    work page

  56. [56]

    draw the chart, compare it with the original, then revise

    Perform at least one "draw the chart, compare it with the original, then revise" iteration instead of producing it in one pass

    work page

  57. [57]

    Save the final deliverables under`/tmp_workspace/results/`, including at least: -`figure5_aspect_ratio_recreated.png` -`figure5_aspect_ratio_recreated.py` -`notes.md`(briefly state which paper and figure you confirmed, and what revisions you made) task_22_met_art_room_match Please open the collection pages on The Met website for the following three paintings:

    work page

  58. [58]

    Claude Monet -- https://www.metmuseum.org/art/collection/search/437133

    work page

  59. [59]

    Vincent van Gogh -- https://www.metmuseum.org/art/collection/search/436535

    work page

  60. [60]

    Caspar David Friedrich -- https://www.metmuseum.org/art/collection/search/438417 Please do the following:

    work page

  61. [61]

    If a single screenshot cannot fit both, save an additional supplemental screenshot for the same artwork and mark that clearly in the filename or notes

    Save at least 1 webpage screenshot for each painting to `/tmp_workspace/results/artwork_screenshots/`; the screenshot must include both the main image and a clear metadata area. If a single screenshot cannot fit both, save an additional supplemental screenshot for the same artwork and mark that clearly in the filename or notes

    work page

  62. [62]

    Read`/tmp_workspace/clawbench/sources/my_room_style.json`

    work page

  63. [63]

    Generate the following in`/tmp_workspace/results/`: -`artworks_metadata.csv` -`visual_analysis.md` -`best_choice_for_my_room.md` 4.`artworks_metadata.csv`must organize at least these fields: - artwork_url, title, artist, object_date, medium, dimensions 5.`visual_analysis.md`must analyze each painting's: - Main color palette and warm/cool relationship - Co...

    work page 2025

  64. [64]

    Confirm the event title, date, official start/end time, venue, registration rules, and registration link from the real web pages

    work page

  65. [65]

    The title must include `Rust Global: Tokyo`

    Create an event in the real calendar GUI. The title must include `Rust Global: Tokyo`

    work page

  66. [66]

    Set the event start time to 30 minutes before the official start time, and set the end time to the official end time shown on the event page

    work page

  67. [67]

    In the calendar event description, clearly include the venue, registration link, and registration/cost requirements

    work page

  68. [68]

    An ICS import preview, a screen where no calendar has been selected, or a dialog where import/save has not been clicked does not count as a created event

    Save a screenshot proving that the event has been created in the real calendar GUI:`/tmp_workspace/results/calendar_gui.png`; if the calendar supports export, also save 21 `/tmp_workspace/results/rust_global_tokyo.ics`. An ICS import preview, a screen where no calendar has been selected, or a dialog where import/save has not been clicked does not count as...

    work page

  69. [69]

    Record the calendar application used, event title, start/end time, timezone, venue, registration link, and evidence file paths

    Also write an audit JSON file to `/tmp_workspace/results/calendar_evidence.json`. Record the calendar application used, event title, start/end time, timezone, venue, registration link, and evidence file paths. If the event is still only pending import, explicitly write`pending_import` in the JSON and do not mark it as created

    work page

  70. [70]

    Available Images

    Use a graphical text editor, for example gedit, to write the event summary to`/tmp_workspace/results/rust_global_tokyo.md`. Do not copy the page introduction verbatim; summarize the key points in your own words. C Role Prompts Three prompts drive every cycle: a session wrapper that frames every role call, thesupervisorprompt that judges the executor’s att...

    work page

  71. [71]

    Read`references/eval_rule.md`as the primary judging spec

    work page

  72. [72]

    Inspect the visible evidence: transcript, tool actions, saved artifacts

    work page

  73. [73]

    could be more thorough

    Classify the attempt state and assign a score. ## Strict scoring discipline (no rule-invention) You MUST score using ONLY the rubric lines and Section 6 score caps that appear in`references/eval_rule.md`. Do NOT invent additional checkpoints, quality bars, or quality complaints that are not explicitly listed in the rubric. In particular: - If a deliverabl...

    work page

  74. [74]

    Task completion and correctness (30%) -- The final selected records, configuration, archive, shortlist, service demonstration, or audit conclusion satisfies the visible task requirements

    work page

  75. [75]

    Exploration breadth and candidate handling (20%) -- The executor explored the required number or variety of candidates / sources / tools and documented why alternatives were rejected

    work page

  76. [76]

    Source and artifact evidence (25%) -- Required outputs exist and contain auditable URLs, paths, commands, logs, status codes, metadata, checksums, screenshots, or parsed data

    work page

  77. [77]

    Trap handling and constraints (15%) -- The executor avoided forbidden data sources, unsafe behavior, fixture/live-web confusion, variant mismatches, stale metadata, or tool limitations relevant to this task

    work page

  78. [78]

    Now let me research the LOC API to understand how to find items with clear rights and accessible images

    Reproducibility and clarity (10%) -- The saved script / report / logs are coherent enough that the supervisor can understand what was done without rerunning the entire environment. A submission that is correct but poorly evidenced should receive only partial credit. A submission that is well documented but reaches the wrong target or violates source restr...

    work page

  79. [79]

    Cycle 1 has the right pipeline and the right data; only the selection rigor and the exclusion-file consistency are insufficient

    The supervisor catches recoverable failure modes rather than rejecting them outright. Cycle 1 has the right pipeline and the right data; only the selection rigor and the exclusion-file consistency are insufficient. Returning continue (not fail) at score = 0.65 preserves the executor’s progress while flagging the remediable gap

    work page

  80. [80]

    Hidden judging assets are never mounted into the user simulator’s workspace, and a deterministic rewriter further sanitizes the simulator’s candidate text

    The information firewall is structural, not stylistic.The supervisor returns a seven-field decision but only four fields cross to the user simulator (Appendix D.5). Hidden judging assets are never mounted into the user simulator’s workspace, and a deterministic rewriter further sanitizes the simulator’s candidate text. The user simulator’s follow-up reads...

    work page

Showing first 80 references.