Pith. sign in

REVIEW 4 major objections 5 minor 20 references

State-of-the-art LLM agents lose substantial accuracy when workplace workflows mix languages across reasoning, tools, and final outputs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 01:36 UTC pith:PPBLWL3R

load-bearing objection Solid hand-curated multilingual agent bench with careful multi-harness reporting; the abstract’s mono-vs-multi degradation claim runs ahead of the controlled evidence in §4. the 4 major comments →

arxiv 2607.06008 v2 pith:PPBLWL3R submitted 2026-07-07 cs.AI cs.CL

PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

Hongliang Li , Yijin Liu , Zhiwei Zhang , Zihe Liu , Xinyue Lou , Jinan Xu , Fandong Meng , Kaiyu Huang This is my paper
classification cs.AI cs.CL
keywords LLM agentsmultilingual benchmarkslong-horizon workflowstool useworkplace taskshybrid evaluationcross-lingual reasoningstructured outputs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most agent benchmarks test long-horizon planning and tool use in a single language, while most multilingual benchmarks test static questions without multi-step execution. Real workplace work often mixes languages inside one workflow: instructions in one language, source documents in another, and deliverables in a third. This paper builds PolyWorkBench—67 hand-authored tasks across commerce, knowledge work, legal analysis, localization, and manufacturing—so agents must keep meaning aligned while retrieving, reasoning, calling tools, and producing structured artefacts. With hybrid scoring that checks structure, executable correctness, and semantic quality, the authors find that even strong models leave large room for failure, with uneven drops by domain and language. The practical claim is that language variation is not a side detail of inputs; it compounds along the agent’s trajectory, so agent evaluation has to treat multilinguality and procedural decision-making together.

Core claim

On multilingual long-horizon workplace workflows, current LLM agents suffer significant performance degradation relative to monolingual settings. Multilinguality introduces compounding effects across reasoning and execution steps—not only comprehension errors but also failures of planning stability, tool reliability, and cross-lingual coordination—so language variation and multi-step decision-making must be evaluated jointly rather than as separate axes.

What carries the argument

PolyWorkBench: 67 end-to-end workplace tasks (five domains, ten languages) that embed language variation into the full execution trajectory, scored by a hybrid framework of structural Grade, executable Pytest checks, and LLM-as-Judge semantic assessment, with Pass@1 defined as mean Grade over all tasks.

Load-bearing premise

That the measured shortfalls are mainly caused by multilingual trajectory coupling, rather than by large harness effects, hand-built tasks with planted anchors, uneven language coverage, or the lack of a controlled same-task monolingual ablation in the main results.

What would settle it

Rerun the same 67 tasks under identical models, harnesses, timeouts, and scoring, but force instruction, sources, and required outputs into a single language; if Pass@1 and domain/language gaps do not close toward strong monolingual agent performance, the multilingual-compounding claim is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent leaderboards that ignore harness choice will mis-rank models, because the same model can shift by roughly 8–21 Pass@1 points across scaffolds.
  • Overall Pass@1 will overstate reliability for enterprise work that demands strict end-to-end numerical or schema correctness, especially commerce-style reconciliation tasks.
  • Mid-tier models can recover large Grade gains via multi-sample best-of-N, while top models are already near saturation on Pass@1.
  • Deterministic graders and LLM judges measure different axes; reporting Grade, Pytest, and Judge together is required to catch both functional failure and fluent-but-wrong or structure-correct-but-semantically-poor outputs.
  • Future agent evaluation should treat cross-lingual consistency as part of the trajectory, not only as input translation or final-output language choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training and scaffolding that only optimize monolingual tool loops will systematically under-prepare agents for mixed-language enterprise pipelines even if static multilingual QA scores look strong.
  • Commerce-like workflows with unforgiving arithmetic and schema checks may be a sharper stress test for agent reliability than long-form legal or knowledge writing that awards partial structural credit.
  • Balancing rare languages and adding matched monolingual twins of each task would turn the benchmark into a cleaner causal test of trajectory coupling versus base capability.
  • Harness-invariant agent interfaces may matter as much as model scale for closing the multilingual gap the paper reports.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PolyWorkBench, a 67-task benchmark spanning five workplace domains (commerce, knowledge, legal, localization, manufacturing) for evaluating LLM agents on multilingual long-horizon workflows. Tasks require heterogeneous multilingual inputs, multi-step tool use, and structured outputs, with language variation embedded across instruction, source, and output roles (88% of tasks involve three or more languages). Evaluation combines structural Grade, executable Pytest suites, and LLM-as-Judge. Across 18 model×harness entries, the best Pass@1 is 0.921 (Claude Opus 4.8 + ClaudeCode); most models fall below 0.77. The authors report large harness effects (8–21 Pass@1 points on the same model), a systematic Commerce dip, non-uniform per-language performance, weak Grade–Judge correlation (r=0.18), and sampling headroom that grows for mid-tier models. They conclude that multilinguality introduces compounding effects across reasoning and execution and that agents degrade relative to monolingual counterparts.

Significance. If the comparative claim holds, the work would fill a genuine gap between monolingual agent benchmarks (WebArena, OSWorld, SWE-bench, OdysseyBench) and static multilingual suites (MGSM, M-MMLU, FLORES) by treating language variation as a trajectory-level factor rather than an input attribute. The hybrid Grade/Pytest/Judge design, multi-harness matrix, Pass@1 vs Pass@3 analysis, and honest disclosure of Judge bimodality and harness sensitivity are concrete methodological contributions that other agent benchmarks can reuse. Even without a clean monolingual ablation, a carefully curated, executable multilingual workplace suite of this kind is useful for the community. The significance of the causal story about “cross-lingual trajectory coupling,” however, depends on isolating multilinguality from task hardness, schema strictness, and scaffolding—something the current experiments only partially achieve.

major comments (4)
  1. [Abstract, §1, §4] Abstract and §1 claim that SOTA agents “suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts” and that multilinguality introduces compounding trajectory effects. Section 4 reports only within-PolyWorkBench variation (domain, language, harness, sampling). No controlled same-task monolingual ablation—identical tools, schemas, step counts, and back-injected anchors, with instruction/source/output collapsed to one language—appears in the main results. Without that contrast, low Pass@1 and domain/language drops cannot be cleanly attributed to cross-lingual trajectory coupling rather than numerical/schema strictness (Table 2), harness scaffolding (Fig. 6), or hand-authored difficulty. This is load-bearing for the paper’s central comparative claim and should be added or the claim should be narrowed to within-benchmark multilingual dif
  2. [§3.3, Fig. 2(b), Appendix A.2] Language coverage is highly imbalanced (Fig. 2(b); Appendix A.2): English touches 66 tasks while Arabic has n=1 (uniform Grade 0.850 by construction). Per-language means for RU/ES/DE are therefore informative, but the “ten languages” framing and any claim of broad multilingual generalization overstate coverage. Either expand low-resource languages or report primary analyses only on languages with adequate task counts and treat AR as a pilot case.
  3. [§4.1–4.2, Table 1, Fig. 6] Harness choice moves Pass@1 by 0.08–0.21 on the same model (Fig. 6; Table 1), and ClaudeCode is best or tied-best whenever available. The paper correctly treats harness as a first-class variable, but the leaderboard and abstract-level “agent” claims still risk being read as model rankings. The primary reported comparison should either fix one harness for all models or report model-level scores only after harness-normalized aggregation, with harness effects relegated to a sensitivity analysis rather than mixed into the main ranking narrative.
  4. [§3.4–3.5, §4.3, Fig. 5] The hybrid evaluation axiom—that Grade + Pytest + Judge jointly capture functional correctness and linguistic consistency—is only partially supported. Grade and Pytest align well (r=0.85), but Judge is weakly correlated (r=0.18 overall; r=−0.04 when Grade≥0.5) and heavily bimodal (§4.3, Fig. 5(a), Appendix A.3–A.4). Pass@1 is defined as mean Grade alone. If linguistic consistency is a core claim of the benchmark, either (i) define a composite metric that includes Judge under conditions where it is reliable, or (ii) demote Judge to a diagnostic secondary signal and revise claims about “linguistic consistency” accordingly. The current design honestly discloses the gap but still markets a three-axis framework whose third axis does not rank models.
minor comments (5)
  1. [§3.3, Fig. 2(c)] Estimated step counts (mean 8.54) and difficulty scale 3–6 are used throughout §3.3 and Fig. 2(c) but are not operationally defined (tool calls? human annotation? agent traces?). A short definition or measurement protocol would help reproducibility.
  2. [§2, References] Several cited “2026” arXiv preprints (Claw-Eval, WildClawBench, CoffeeBench, MAPS, etc.) are contemporaneous or future-dated relative to the paper’s July 2026 date. Ensure citation status and availability are accurate at camera-ready time.
  3. [Fig. 1–3] Figure 1 and Figure 3 are dense overview diagrams; axis labels and small text in the language polar plot (Fig. 2(b)) may not reproduce well in print. Consider simplifying or enlarging key panels.
  4. [Table 1, Fig. 6] Table 1 lists only one Codex entry; the harness comparison for Codex is underpowered relative to ClaudeCode/OpenClaw/Hermes. Note this limitation explicitly when discussing harness ordering.
  5. [Throughout] Typographical consistency: “artefacts” vs “artifacts,” and occasional spacing anomalies around decimals (e.g., “0 .921”) appear in the compiled text and should be cleaned.

Circularity Check

0 steps flagged

No derivation circularity: PolyWorkBench is an empirical benchmark whose scores are external measurements, not quantities recovered from their own fitted inputs.

full rationale

This paper does not present a first-principles derivation, uniqueness theorem, or fitted law that is then re-sold as a prediction. Its load-bearing content is (i) hand-authored multilingual workplace tasks with back-injected verifiable anchors, (ii) a hybrid Grade / Pytest / LLM-as-Judge evaluation stack, and (iii) Pass@1 measurements of external model×harness systems on those fixed tasks. Grade and Pytest are defined against task-specific structural and executable criteria that do not include the models under test; the reported numbers are therefore not forced by construction. Self-citations in Related Work are ordinary prior-art pointers (WebArena, SWE-bench, MGSM, etc.), not load-bearing uniqueness results by the same authors. LLM-as-Judge is used only as a secondary semantic axis and is explicitly shown to correlate weakly with Grade (r = 0.18), so ranking is not tautologically driven by a model family judging itself. The abstract’s comparative claim of degradation versus monolingual counterparts is an evidence-gap / unsupported-contrast issue (no controlled same-task monolingual ablation in §4), not a circular reduction of a prediction to its inputs. Per the analyzer rules, that concern belongs under correctness risk, not circularity. Honest finding: no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

As a benchmark paper, load-bearing content is definitional and methodological rather than physical axioms. The central claims rest on how tasks and scores are constructed (hand curation, trilingual role assignment, Grade as primary metric, hybrid axes), on treating agent harnesses as comparable scaffolds, and on interpreting score gaps as multilingual trajectory effects. No free physical constants; free design choices are the scoring and sampling conventions that define the leaderboard.

free parameters (4)
  • Pass@1 primary metric = mean Grade over 67 tasks
    Leaderboard ranking is defined by average structural Grade, not Pytest or Judge; partial-credit weights inside each task's grade() are task-specific design choices that determine who ranks first.
  • Estimated workflow step counts and difficulty scale (mean 8.5 steps; scale 3–6)
    Pool splits (baseline 29 / stress 38) and complexity claims depend on author-estimated steps and a 1–6 difficulty scale, not an external validated measure of horizon length.
  • Timeout budget 1800s per task; n_runs up to 3 for Pass@3
    Success rates and sampling headroom depend on these evaluation protocol choices; different budgets could reorder mid-tier models.
  • LLM-as-Judge prompt and score aggregation in [0,1]
    Semantic axis is defined by a standardized but author-chosen judge prompt; the paper itself shows the resulting scores are highly bimodal and weakly correlated with Grade.
axioms (4)
  • domain assumption Hand-authored tasks with back-injected ground-truth anchors and no machine-translated source files are a valid sample of real multilingual workplace workflows.
    Section 3.2 construction pipeline; generalization of 'real-world multilingual agents' rests on this sampling assumption.
  • ad hoc to paper Structural Grade plus Pytest plus LLM-as-Judge jointly capture functional correctness and linguistic consistency of long-horizon agent trajectories.
    Section 3.4–3.5 defines the hybrid framework as the evaluation truth; primary ranking uses Grade alone.
  • domain assumption Differences across ClaudeCode, OpenClaw, Hermes, and Codex primarily reflect harness design rather than uncontrolled implementation bugs.
    Section 4.1–4.3 treats harness as a first-class experimental factor when interpreting model scores.
  • standard math Standard agent-benchmark and multilingual-NLP evaluation practices (tool-using agents, Pass@k, LLM-as-judge) are appropriate background methods.
    Related Work and evaluation design inherit these community conventions without re-deriving them.
invented entities (3)
  • PolyWorkBench (67-task multilingual workplace agent suite) no independent evidence
    purpose: Provide end-to-end tasks that embed language variation into the full agent trajectory rather than only into static inputs.
    Core contribution of the paper; existence and difficulty profile are defined by the authors' curation.
  • Hybrid Grade / Pytest / LLM-as-Judge evaluation pipeline with Pass@1 = mean Grade no independent evidence
    purpose: Score heterogeneous structured outputs for both deterministic correctness and semantic quality.
    Primary leaderboard signal and secondary metrics are paper-defined composites, not external standards.
  • Cross-lingual trajectory coupling (as explanatory construct) no independent evidence
    purpose: Name the hypothesized mechanism by which multilinguality compounds errors across planning, tool use, and generation.
    Introduced in the introduction/analysis as the phenomenon the benchmark is meant to expose; not independently measured outside score patterns.

pith-pipeline@v1.1.0-grok45 · 20430 in / 3684 out tokens · 46803 ms · 2026-07-11T01:36:03.480897+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks implicitly assume a monolingual setting, where the entire execution process, including reasoning, tool invocation, and output generation, is conducted within a single language. In contrast, real-world applications often involve multilingual inputs and outputs within a unified workflow, yet the interaction between multilinguality and agentic execution remains underexplored. In this work, we introduce PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows. PolyWorkBench consists of 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing, where agents must process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. To enable comprehensive evaluation, we propose a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment. This design allows us to capture both functional correctness and linguistic consistency across complex workflows. Empirical results show that state-of-the-art LLM agents suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts. Our analysis suggests that multilinguality introduces compounding effects across reasoning and execution steps, highlighting the importance of jointly modeling language variation and procedural decision-making in agent evaluation.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 10 internal anchors

  1. [1]

    Mle-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. InInternational Conference on Learning Representations, volume 2025, pages 50466–50494,

    work page 2025

  2. [2]

    WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

    Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, et al. Wildclawbench: A benchmark for real-world, long-horizon agent evaluation.arXiv preprint arXiv:2605.10912,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems

    Robert Friel, Masha Belyi, and Atindriyo Sanyal. Ragbench: Explainable benchmark for retrieval-augmented generation systems.arXiv preprint arXiv:2407.11005,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  5. [5]

    Maps: A multilingual benchmark for agent performance and security

    Omer Hofman, Jonathan Brokman, Oren Rachmil, Shamik Bose, Vikas Pahuja, Toshiya Shimizu, Trisha Starostina, Kelly Marchisio, Seraphina Goldfarb-Tarrant, and Roman Vainshtein. Maps: A multilingual benchmark for agent performance and security. InFindings of the Association for Computational Linguistics: EACL 2026, pages 821–845,

    work page 2026

  6. [6]

    Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157,

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157,

    work page 2024

  7. [7]

    Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback

    Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Yansong Feng and Els Lefever, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demo...

    work page 2023

  8. [8]

    Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-demo.28.https://aclanthology.org/2023.emnlp-demo.28/. Chenxin Li, Zhengyang Tang, Mingxin Huang, Yunlong Lin, Shijue Huang, Shengyuan Liu, Bowen Ye, Rang Li, Lei Li, Benyou Wang, et al. Claw-eval-live: A live agent benchmark for evolving real-world workflows.arXiv preprint arXiv:2604.28139,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2023.emnlp-demo.28.https://aclanthology.org/2023.emnlp-demo.28/ 2023

  9. [9]

    Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation

    Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008–6018,

    work page 2020

  10. [10]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InInternational Conference on Learning Representations, volume 2024, pages 52989–53046,

    work page 2024

  11. [11]

    ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

    Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, et al. Clawmark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents.arXiv preprint arXiv:2604.23781,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InInternational Conference on Learning Representations, volume 2024, pages 9025–9049,

    work page 2024

  13. [13]

    Language Models are Multilingual Chain-of-Thought Reasoners

    11 Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners.arXiv preprint arXiv:2210.03057,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Coffeebench: Benchmarking long-horizon llm agents in heterogeneous multi-agent economies.arXiv preprint arXiv:2606.16613,

    Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, and Takashi Ishida. Coffeebench: Benchmarking long-horizon llm agents in heterogeneous multi-agent economies.arXiv preprint arXiv:2606.16613,

    work page arXiv

  15. [15]

    Mirage-bench: Automatic multilingual benchmark arena for retrieval-augmented generation systems

    Nandan Thakur, Suleman Kazi, Ge Luo, Jimmy Lin, and Amin Ahmad. Mirage-bench: Automatic multilingual benchmark arena for retrieval-augmented generation systems. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 274–298,

    work page 2025

  16. [16]

    OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows

    Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Rajmohan. Odysseybench: Evaluating llm agents on long-horizon complex office application workflows.arXiv preprint arXiv:2508.09124,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation

    Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation.arXiv preprint arXiv:2407.19056,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Towards trustworthy evaluation of autonomous agents.arXiv preprint arXiv:2604.06132,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, volume 2024, pages 15585–15606,

    work page 2024