Pith. sign in

REVIEW 3 major objections 4 minor 47 references

X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that current LLM web agents fall more than 20% behind English-only performance when tested across 14 non-English languages, and that the gap is a language-alignment problem, not a reasoning problem.

desk verdict Useful multilingual agent benchmark resource, but the headline claim of a >20% language gap is not controlled: no English baseline is run inside the same environment. read the letter →

arxiv 2505.15372 v1 pith:SZHU2RJQ submitted 2025-05-21 cs.CL

Peng Wang , Ruihan Tao , Qiguang Chen , Mengkang Hu , Libo Qin This is my paper
classification cs.CL
keywords multilingualagentswebagentbenchmarkcross-lingualpromptinglanguagealignmentShopinteractiveenvironmenttaskscoreGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that language agents built and evaluated in English do not transfer to other languages: when instructions and the shopping environment are translated into 14 languages, task scores drop by more than 20% relative to English-only results. To make this measurable, it constructs X-WebAgentBench, an interactive e-commerce benchmark with 2,800 multilingual instructions and 589,946 products, translated from WebShop. Across five LLMs and four agent strategies, no configuration reaches a satisfactory score; even GPT-4o, the strongest model tested, averages 37.11 with the base agent and 37.94 with cross-lingual prompting. The paper argues the bottleneck is language alignment rather than reasoning logic, since strong reasoning models do not close the gap.

What carries the argument

The load-bearing instrument is X-WebAgentBench itself: an interactive web-shopping environment built by taking 500 WebShop instructions, translating them into 14 languages with Google Translate to form 2,800 instructions, and translating product data with GPT-4 to form 589,946 products, each instruction paired with about 211 products to keep the context window manageable. The four agent strategies compared are BaseAgent, Translate-en (external translation to English), Self-Translate-en (model self-translation), and CLP (cross-lingual prompting that first aligns the observation in English and then answers in the local language). The metric is the WebShop task score, which rewards buying the item that matches the instruction's attributes; the benchmark also records action steps, input/output tokens, and error types to diagnose where the gap comes from.

What would settle it

Run the same BaseAgent and CLP methods on the English version of X-WebAgentBench's 500 instructions in its 211-product environment. If the resulting English average task score is near or below the reported multilingual averages (around 37), the language-gap claim would lose its empirical basis; if English stays near the original WebShop level (roughly 50 or higher), the multilingual gap is confirmed.

Watch

Extended reading notes

Core claim

The central discovery is that multilingual interactive environments expose a persistent performance gap in current agentic systems. Under the WebShop-style task score, GPT-4o reaches only 37.11 on average across the 14 languages with the base agent and 37.94 with cross-lingual prompting, and no tested model or alignment method approaches the reported English-only level, which the paper takes from the original WebShop results. The paper attributes the gap mainly to language alignment (Section 5.5): DeepSeek-R1, a strong reasoning model, performs about as well as GPT-4o and still stays below 50, so additional reasoning alone does not fix the problem. It also reports that cross-lingual self-alignment helps only larger multilingual-capable models, that small open models improve when an external translator converts the environment to English, and that interaction failures concentrate in no-output, click-error, and action-error categories.

Load-bearing premise

The headline claim that multilingual performance lags English by more than 20% assumes that English task scores reported for the original WebShop benchmark are comparable to scores measured here in a simplified, translated environment; the paper never runs an English control inside X-WebAgentBench.

Editorial extensions

If this is right

  • Multilingual e-commerce agents will need explicit language-alignment components; English-centric training and prompting alone will not serve non-English users adequately.
  • Cross-lingual self-alignment should be reserved for models with strong multilingual capacity, while smaller models should be paired with external translation tools rather than asked to self-translate.
  • Because the paper finds token-cost unfairness driven by script rather than training-resource level, changing tokenizers is a concrete lever for making multilingual agents affordable.
  • Engineering more reasoning into agents will not close the language gap; improvements in multilingual understanding are the binding constraint.
  • The gap is measured even in a multi-agent framework (BOLAA), so it is a general property of current agentic systems, not an artifact of one prompting style.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves implicit is an English control inside X-WebAgentBench; running the same BaseAgent and CLP on the English version would determine whether the reported >20% gap is robust or partly due to comparing with an external English benchmark.
  • The observation that agents never use an offered translation tool suggests that tool availability is not enough; future agents need a meta-decision rule for when to invoke translation, a design implication beyond the paper's stated findings.
  • If token-cost unfairness is script-driven, then tokenizers trained on more balanced script data could disproportionately improve low-resource language performance without changing model size, a hypothesis worth testing directly.
  • The paper's own Limitations section concedes that Google Translate plus quality checks cannot fully match experienced human translators, so translation noise bounds how precisely the benchmark can attribute score differences to agent capability rather than to translation artifacts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces X-WebAgentBench, a multilingual interactive web agent benchmark built by translating and adapting the WebShop environment into 14 languages. The benchmark contains 2,800 instructions (200 per language) and roughly 590,000 product records, with a documented pipeline of machine translation, GPT-4-based quality scoring, and human rechecking (reported kappa 0.8). The authors evaluate GPT-4o, GPT-3.5-turbo, Qwen2-7B, Mistral-7B, and Llama3-8B under four baseline strategies (BaseAgent, Translate-en, Self-Translate-en, CLP), and report additional analyses of token consumption, action steps, in-context demonstrations, reasoning models, and error types. The paper's headline finding is that multilingual performance lags English-only performance by over 20%, and that even GPT-4o reaches only about 37-38 average task score. It also argues that the main bottleneck is language alignment rather than reasoning logic.

Significance. If the construction holds, X-WebAgentBench would be a genuinely useful resource: it is the first interactive web agent benchmark of this scale to combine multilingual instructions with a multilingual environment, and its quality-control pipeline (onboarding test, LLM-based translation scoring, manual recheck with kappa 0.8) is described in unusual detail. The paper also contains reproducible, falsifiable empirical claims, e.g., that external translation-to-English helps small open models while self-alignment (CLP) only helps GPT-series models, and that token costs in agentic settings are driven by script rather than training-resource level. These are concrete strengths. However, the central quantitative claim of a persistent 'over 20%' multilingual gap is not established by the experiments as reported, because no English control is run inside the new benchmark. The resource and several analysis findings may still be valid, but the headline comparative claim requires either a same-environment English baseline or a substantial reframing.

major comments (3)
  1. [Section 1, Figure 1] The claim that 'multilingual performance lags behind English-only performance by over 20%' is not supported by a controlled comparison. The English task scores in Figure 1 are taken from Yang et al. (2023) on the original WebShop benchmark, while the multilingual scores come from X-WebAgentBench. The two settings differ in at least three load-bearing ways: Section 2.1 simplifies the environment to an average of 211 products per instruction; Section 2.4 filters the instruction set to 200 quality-checked items per language; and Appendix E uses an AgentBench-style prompt scaffold rather than the WebShop/ReAct protocol used by Yang et al. Any of these differences can shift absolute task scores by more than the size of the claimed gap. The paper never runs the same model and baseline on an English-language version of its own 200-instruction set. Without that same-environment English control, the central 'over 20%' lag is confounded with environment, prompt, and instruction-selection differences. I would ask the authors to add an English control inside X-WebAgentBench for at least GPT-4o and GPT-3.5-turbo under BaseAgent and CLP, and then either report the corrected gap or revise the framing to say that the benchmark reveals low multilingual performance without quantifying the lag against English.
  2. [Section 5.5, Figure 8] The conclusion that 'the main bottleneck of X-WebAgentBench still lies in language alignment, rather than reasoning logic' is not established by the reported experiments. DeepSeek-R1 and DeepSeek-R1-Distill-Llama-70B are evaluated only on multilingual tasks and compared only with multilingual GPT-4o scores. There is no same-environment English baseline for the reasoning models. If DeepSeek-R1 also scores near 37 on the English version of X-WebAgentBench, then the bottleneck could be task complexity or environment difficulty rather than language alignment. The dichotomy between 'language alignment' and 'reasoning logic' is only meaningful if English-in-this-environment performance is substantially higher for at least one strong model. The authors should add English controls for the reasoning models, or soften the causal attribution accordingly.
  3. [Section 4.3, Figure 5] The analysis that 'performance degradation occurs earlier than in English-based tasks' in multilingual scenarios uses an external English planning-boundary estimate (Chen et al., 2024a) rather than a measured English control in the same environment. Figure 5(b) shows reward distributions only for multilingual runs of GPT-3.5-turbo, so the comparison to an English 5-10 step boundary is not operationalized. This is the same missing-control issue as the headline gap, but it is load-bearing for one of the paper's stated analysis findings. A straightforward fix is to run the same BaseAgent method on the English subset of X-WebAgentBench and compare the step-reward distribution directly.
minor comments (4)
  1. [Section 2.4] The text says the LLM check results in 'approximately 250 instructions and associated products' per language and then manual recheck yields 200, but the reason for dropping from 250 to 200 is not explained. Please state whether the same 250 items were used for all languages and what criteria removed the 50.
  2. [Section 5.4] The statement that 'across all 14 languages, the LLMs never utilize the translation function' is presented without any table, log count, or other supporting evidence. Since this is a strong overconfidence claim, please report the number of attempts to use the tool or provide a representative trace.
  3. [Figure 3] Figure 3(a) is titled 'Distribution of 15 languages on the world map,' but the benchmark covers 14 languages plus English as a reference point. The title and caption should clarify whether English is included in the language count.
  4. [Section 4.2] The claim about less-spoken languages requiring more actions is based on Table 3, but the thresholds for 'far exceeds the average' (e.g., 'far above the average of 1-2 searches') are stated informally. Adding mean/median action counts with standard deviations would make the comparison more rigorous.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark scores are empirical measurements, and the paper's self-citations are not load-bearing for the central claims.

full rationale

No circular step can be exhibited from the paper's own equations or construction. X-WebAgentBench is a newly constructed benchmark, and the reported task scores are measured outcomes of running agent systems in that environment. The CLP baseline (Qin et al., 2023) and other self-citations are used as methods or baselines to be evaluated, not as evidence that determines the measured scores. The paper's central gap claim compares English task scores from Yang et al. (2023) on the original WebShop benchmark with multilingual scores from X-WebAgentBench, and the paper does not run an English control inside its own environment; however, this is a comparison-validity and confounding concern, not a circular reduction in which a fitted parameter or defined quantity is renamed as a prediction. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is merely renamed. The self-citations are numerous but minor and non-load-bearing, so the appropriate circularity score is 1 rather than a higher score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and rests on construction choices and comparability assumptions rather than on derived quantities. The listed free parameters are hand-chosen design thresholds that shape which tasks survive and how scores are computed; the axioms are the fidelity and comparability assumptions the benchmark's interpretation depends on. No invented physical or conceptual entities are introduced.

free parameters (4)
  • Instructions per language = 200
    The benchmark keeps 200 instructions per language after quality filtering; this size is hand-chosen and determines the statistical resolution of per-language scores in Table 2.
  • LLM check inclusion threshold = 8 out of 10
    Translations are kept only if GPT-4 scores them 8 or higher (Section 2.4). This threshold is arbitrary and affects which instructions and products survive.
  • Language set = 14 languages
    The 14 languages are selected to cover 7 families following Conneau et al. (2018); the choice affects the benchmark's scope and the external validity of the findings.
  • Product context size = 211 products average per instruction
    Section 2.1 limits each task to an average of 211 products to manage context length; this simplification changes difficulty relative to the original WebShop and is not calibrated to preserve task equivalence.
assumptions (4)
  • domain assumption Machine-translated instructions and product pages preserve the semantic content and task difficulty of the original English WebShop tasks.
    The benchmark claims to measure agent multilingual ability, but if translations introduce ambiguity or errors, scores reflect translation quality rather than agent skill. Section 2.4 discusses polysemy (e.g., Chinese conflating 'trousers' and 'shorts'), and the Limitations section concedes translation quality cannot fully match experienced translators.
  • domain assumption English reference scores from Yang et al. (2023) are comparable to scores from X-WebAgentBench.
    Figure 1 and Section 3.1 compare external English WebShop scores against multilingual scores from a simplified environment (211 products per task). No English control is run in X-WebAgentBench, so environment differences are not accounted for.
  • domain assumption The WebShop task score rewards the correct product selection and attribute match in all languages.
    The task score is adopted from Yao et al. (2022) without re-validation in the multilingual setting; if reward semantics degrade under translation, absolute scores are not comparable across languages.
  • standard math A kappa coefficient of 0.8 indicates acceptable annotator agreement for the manual recheck.
    Section 2.4 uses the conventional Landis and Koch (1977) threshold; this is a standard statistical interpretation but a chosen cutoff.

how reviews work

0 comments
Cite this review

Pith. "Pith review of X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System." pith.science (2026). https://pith.science/paper/SZHU2RJQ

@misc{pith2026250515372,
  author       = {Pith},
  title        = {Pith review of: X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZHU2RJQ}},
  note         = {Machine review of arXiv:2505.15372}
}
read the original abstract

Recently, large language model (LLM)-based agents have achieved significant success in interactive environments, attracting significant academic and industrial attention. Despite these advancements, current research predominantly focuses on English scenarios. In reality, there are over 7,000 languages worldwide, all of which demand access to comparable agentic services. Nevertheless, the development of language agents remains inadequate for meeting the diverse requirements of multilingual agentic applications. To fill this gap, we introduce X-WebAgentBench, a novel multilingual agent benchmark in an interactive web environment, which evaluates the planning and interaction performance of language agents across multiple languages, thereby contributing to the advancement of global agent intelligence. Additionally, we assess the performance of various LLMs and cross-lingual alignment methods, examining their effectiveness in enhancing agents. Our findings reveal that even advanced models like GPT-4o, when combined with cross-lingual techniques, fail to achieve satisfactory results. We hope that X-WebAgentBench can serve as a valuable benchmark for multilingual agent scenario in real-world applications.

Figures

Figures reproduced from arXiv: 2505.15372 by the authors.

Figure 1
Figure 1. Comparison of performance in English and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The construction of X-WebAgentBench includes four stages: (a) Data Preparation, (b) Multilingual Instruction Construction, (c) Multilingual Environment Construction, and (d) Quality Check. This workflow figure refers to M3CoT (Chen et al., 2024b). 2 X-WebAgentBench Construction This section describes the construction of X-WebAgentBench, including Data Prepara￾tion (§ 2.1), Multilingual Instruction Construction (§ 2.… view at source ↗
Figure 3
Figure 3. The distribution of languages and product category in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Statistics of average input token and output [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Statistics of action reward for BaseAgent method by GPT-3.5-turbo. Method de ru tr vi bg hi ReAct + En-ICL 12.81 8.77 15.09 12.06 8.78 2.15 ReAct + De-ICL 19.65 8.21 10.61 9.07 13.50 2.91 ReAct + L-ICL 19.65 ↑ 18.58 ↑ 1.60 ↓ 9.34 ↓ 14.65 ↑ 5.79 ↑ [PITH_FULL_IMAGE:figu…
Figure 7
Figure 7. Figure 7: The performance of ReAct and BOLAA with L-ICL. Task score in English is from Liu et al. (2024b). 5.3 Multilingual gaps are widespread in current English-centric agentic systems To comprehensively evaluate the multilingual lim￾itation of current English-centric strong a…
Figure 9
Figure 9. Figure 9: The diagram of these baseline methods. Fig [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Performance comparison of different-scale LLMs on [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Statistics of action step (a), reward (b), and in [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 13 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  3. [3]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567

    arXiv 2025

  4. [4]

    Qiguang Chen, Libo Qin, Jiaqi WANG, Jingxuan Zhou, and Wanxiang Che. 2024 a . https://openreview.net/forum?id=pC44UMwy2v Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  5. [5]

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.446 M ^3 C o T : A novel benchmark for multi-domain multi-step multi-modal chain-of-thought . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8199--8221, Bangko...

    doi:10.18653/v1/2024.acl-long.446 2024

  6. [6]

    Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021 a . https://doi.org/10.18653/v1/2021.emnlp-main.343 W eb SRC : A dataset for web-based structural reading comprehension . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173--4185, Online and Punta Cana, Domi...

    doi:10.18653/v1/2021.emnlp-main.343 2021

  7. [7]

    Xingyu Chen, Zihan Zhao, Lu Chen, Jiabao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021 b . Websrc: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173--4185

    work page 2021

  8. [8]

    Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Wanxiang Che, Hang Yan, Kai Chen, and Dahua Lin. 2024 c . What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. arXiv preprint arXiv:2409.01893

    arXiv 2024

Show all 47 references
  1. [9]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods ...

    2018 doi

  2. [10]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    2024

  3. [11]

    Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2024. https://doi.org/10.18653/v1/2024.naacl-short.46 Do multilingual language models think better in E nglish? In Proceedings of the 2024 Conference of the North American Chapter of the Associ...

    2024 doi

  4. [12]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  5. [13]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J \"u rgen Schmidhuber. 2024. https://openreview.net/forum?id=VtmBAGCN7o Metagpt...

    2024

  6. [14]

    Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. 2024 a . Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. arXiv preprint arXiv:2408.09559

    2024 arXiv

  7. [15]

    Mengkang Hu, Yao Mu, Xinmiao Chelsey Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. 2024 b . https://openreview.net/forum?id=Glcsog6zOe Tree-planner: Efficient close-loop task planning with large language models . In The Twelfth Intern...

    2024

  8. [16]

    Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, and Dongmei Zhang. 2024 c . https://doi.org/10.48550/arXiv.2408.00764 Agentgen: Enhancing planning abilities for large language model based agent via environment and task genera...

    2024 doi arXiv

  9. [17]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  10. [18]

    Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. https://doi.org/10.18653/v1/2024.findings-acl.108 The impact of reasoning step length on large language models . In Findings of the Association for Computational Linguis...

    2024 doi

  11. [19]

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2025. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pag...

    2025

  12. [20]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159--174

    1977

  13. [21]

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations

    2018

  14. [22]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2024 a . https://openreview.net/forum?id=zAdUB0aCTQ Agentbench: Evaluating LLM s as agents . In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh R N, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil L Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. 2024 b . https://openreview.net/forum?id=BUa5ekiHlQ BOLAA : BENCHMARKING...

    2024

  16. [24]

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2024. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36

    2024

  17. [25]

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. 2024. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373

    2024 arXiv

  18. [26]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/74bb24dca8334adce292883b4b651eda-Paper-Conference.pdf Language model tokenizers introduce unfairness between languages . In Advances in Neural Inf...

    2023

  19. [27]

    Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. 2024 a . https://openreview.net/forum?id=REVdYKGcfb What factors affect multi-modal in-context learning? an in-depth exploration . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  20. [28]

    Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024 b . Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819

    2024 arXiv

  21. [29]

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695--2709

    2023

  22. [30]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns, 6(1)

    2025

  23. [31]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. https://openreview.net/forum?id=fR3wGCk-IXp Language models are multilingual chain-of-thought reasone...

    2023

  24. [32]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf Reflexion: language agents with verbal reinforcement learning . In Advances...

    2023

  25. [33]

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. Meta-gui: Towards multi-modal conversational agents on mobile gui. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6699--6712

    2022

  26. [34]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  27. [35]

    Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2105.13231

    2021 arXiv

  28. [36]

    Peng Wang, Wenpeng Lu, Chunlin Lu, Ruoxi Zhou, Min Li, and Libo Qin. 2025. https://doi.org/10.26599/BDMA.2024.9020090 Large language model for medical images: A survey of taxonomy, systematic review, and future trends . Big Data Mining and Analytics, 8(2):496--517

    2025

  29. [37]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Proces...

    2024

  30. [38]

    Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. 2021. https://doi.org/10.18653/v1/2021.naacl-main.80 Grounding open-domain instructions to automate web support tasks . In Proceedings of the 2021 Conference of the North American C...

    2021 doi

  31. [39]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  32. [40]

    Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224

    2023 arXiv

  33. [41]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference.pdf Webshop: Towards scalable real-world web interaction with grounded language agents . In Advances i...

    2022

  34. [42]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  35. [43]

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484--5505

    2023

  36. [44]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  37. [45]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. https://openreview.net/forum?id=oKn9c6ytLx Webarena: A realistic web environment for building autonomous agents ....

    2024

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.