{"id":"7b95c16a-ade7-4139-8ae7-dc221d8b89f4","arxiv_id":"2505.15372","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"X-WebAgentBench translates the WebShop e-commerce agent task into 14 languages and shows that current agents, including GPT-4o, perform substantially worse in multilingual settings than in English.","lead":"The paper introduces X-WebAgentBench, a web-agent benchmark that translates the English WebShop shopping task into 14 languages, including translated instructions and product pages. It reports that strong models such as GPT-4o score far lower in non-English settings, and that small models only improve when an external translator converts pages to English.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'over 20% language gap' is computed against an English WebShop baseline from a different environment, prompt, and instruction subset; without an English control inside X-WebAgentBench the central comparative claim is unestablished.","rationale":"Reading the paper in good faith, the contribution is a plausible first multilingual interactive web benchmark with a documented construction pipeline, pre-validation, GPT-4 checks, and manual rechecks. The absolute multilingual scores in Table 2 are informative and the per-language analyses are descriptive. However, the headline comparative claim depends on an English baseline that is never measured in the authors' own environment. The reader's weakest-assumption analysis identifies exactly this point, and I agree. The concern is load-bearing because the paper's abstract and Section 5.5 make the size of the language gap and the 'language alignment, not reasoning' diagnosis central to the narrative. The proposed English control is a single, feasible run: use the same 200 English instructions, the untranslated English environment, the same prompts, and the same scoring function. This would settle whether the gap is linguistic or partly an artifact of environment and pipeline differences. The paper also currently offers no code release, pinned version, or error bars; those are additional reasons to keep the verdict conditional, but they are secondary. No adjustment to the reader's verdict is needed: CONDITIONAL remains the right call, with the English control as the key required revision.","tokens_in":17588,"tokens_out":4121,"duration_ms":36485,"concrete_test":"Run GPT-4o with BaseAgent and CLP on the 200 English source instructions in the English version of the X-WebAgentBench environment, using the Appendix E prompts and the Table 2 scoring procedure, for at least three independent runs each. If the English average task score is not significantly higher than the 37.11/37.94 multilingual averages (for example, within one standard error), the >20% language-gap claim in Section 1 and the Section 5.5 alignment-bottleneck interpretation are not supported; if the English score is substantially higher, the original claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most load-bearing quantitative claim—that multilingual performance lags English by more than 20% (Abstract; Section 1; Figure 1)—rests on comparing GPT-4o's 37.11/37.94 average task scores in X-WebAgentBench (Table 2) with English scores taken from Yang et al. (2023) on the original WebShop benchmark. That comparison is uncontrolled. Section 2.1 narrows the environment to an average of 211 products per instruction, Section 2.1/2.4 filters the instruction set down to 200 informative, quality-checked items per language, and Appendix E uses an AgentBench-style prompt scaffold rather than the WebShop/ReAct protocol. Any of these factors can change absolute task scores by more than the size of the claimed gap. The paper never runs the same BaseAgent or CLP agent on the English-language version of its own 200 instructions, so the gap between 'multilingual' and 'English' is confounded with environment, prompt, and selection differences. This same missing control affects Section 5.5's conclusion that the bottleneck is language alignment rather than reasoning logic: DeepSeek-R1 is compared only with multilingual scores, not with an English baseline inside the same environment. If English scores in X-WebAgentBench are also near 37, the central 'language gap' narrative is not supported, although the benchmark itself may still be a valid resource.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces X-WebAgentBench, a multilingual interactive web agent benchmark built by translating and adapting the WebShop environment into 14 languages. The benchmark contains 2,800 instructions (200 per language) and roughly 590,000 product records, with a documented pipeline of machine translation, GPT-4-based quality scoring, and human rechecking (reported kappa 0.8). The authors evaluate GPT-4o, GPT-3.5-turbo, Qwen2-7B, Mistral-7B, and Llama3-8B under four baseline strategies (BaseAgent, Translate-en, Self-Translate-en, CLP), and report additional analyses of token consumption, action steps, in-context demonstrations, reasoning models, and error types. The paper's headline finding is that multilingual performance lags English-only performance by over 20%, and that even GPT-4o reaches only about 37-38 average task score. It also argues that the main bottleneck is language alignment rather than reasoning logic.","tokens_in":17812,"tokens_out":4862,"duration_ms":42148,"significance":"If the construction holds, X-WebAgentBench would be a genuinely useful resource: it is the first interactive web agent benchmark of this scale to combine multilingual instructions with a multilingual environment, and its quality-control pipeline (onboarding test, LLM-based translation scoring, manual recheck with kappa 0.8) is described in unusual detail. The paper also contains reproducible, falsifiable empirical claims, e.g., that external translation-to-English helps small open models while self-alignment (CLP) only helps GPT-series models, and that token costs in agentic settings are driven by script rather than training-resource level. These are concrete strengths. However, the central quantitative claim of a persistent 'over 20%' multilingual gap is not established by the experiments as reported, because no English control is run inside the new benchmark. The resource and several analysis findings may still be valid, but the headline comparative claim requires either a same-environment English baseline or a substantial reframing.","major_comments":[{"comment":"The claim that 'multilingual performance lags behind English-only performance by over 20%' is not supported by a controlled comparison. The English task scores in Figure 1 are taken from Yang et al. (2023) on the original WebShop benchmark, while the multilingual scores come from X-WebAgentBench. The two settings differ in at least three load-bearing ways: Section 2.1 simplifies the environment to an average of 211 products per instruction; Section 2.4 filters the instruction set to 200 quality-checked items per language; and Appendix E uses an AgentBench-style prompt scaffold rather than the WebShop/ReAct protocol used by Yang et al. Any of these differences can shift absolute task scores by more than the size of the claimed gap. The paper never runs the same model and baseline on an English-language version of its own 200-instruction set. Without that same-environment English control, the central 'over 20%' lag is confounded with environment, prompt, and instruction-selection differences. I would ask the authors to add an English control inside X-WebAgentBench for at least GPT-4o and GPT-3.5-turbo under BaseAgent and CLP, and then either report the corrected gap or revise the framing to say that the benchmark reveals low multilingual performance without quantifying the lag against English.","section":"Section 1, Figure 1"},{"comment":"The conclusion that 'the main bottleneck of X-WebAgentBench still lies in language alignment, rather than reasoning logic' is not established by the reported experiments. DeepSeek-R1 and DeepSeek-R1-Distill-Llama-70B are evaluated only on multilingual tasks and compared only with multilingual GPT-4o scores. There is no same-environment English baseline for the reasoning models. If DeepSeek-R1 also scores near 37 on the English version of X-WebAgentBench, then the bottleneck could be task complexity or environment difficulty rather than language alignment. The dichotomy between 'language alignment' and 'reasoning logic' is only meaningful if English-in-this-environment performance is substantially higher for at least one strong model. The authors should add English controls for the reasoning models, or soften the causal attribution accordingly.","section":"Section 5.5, Figure 8"},{"comment":"The analysis that 'performance degradation occurs earlier than in English-based tasks' in multilingual scenarios uses an external English planning-boundary estimate (Chen et al., 2024a) rather than a measured English control in the same environment. Figure 5(b) shows reward distributions only for multilingual runs of GPT-3.5-turbo, so the comparison to an English 5-10 step boundary is not operationalized. This is the same missing-control issue as the headline gap, but it is load-bearing for one of the paper's stated analysis findings. A straightforward fix is to run the same BaseAgent method on the English subset of X-WebAgentBench and compare the step-reward distribution directly.","section":"Section 4.3, Figure 5"}],"minor_comments":[{"comment":"The text says the LLM check results in 'approximately 250 instructions and associated products' per language and then manual recheck yields 200, but the reason for dropping from 250 to 200 is not explained. Please state whether the same 250 items were used for all languages and what criteria removed the 50.","section":"Section 2.4"},{"comment":"The statement that 'across all 14 languages, the LLMs never utilize the translation function' is presented without any table, log count, or other supporting evidence. Since this is a strong overconfidence claim, please report the number of attempts to use the tool or provide a representative trace.","section":"Section 5.4"},{"comment":"Figure 3(a) is titled 'Distribution of 15 languages on the world map,' but the benchmark covers 14 languages plus English as a reference point. The title and caption should clarify whether English is included in the language count.","section":"Figure 3"},{"comment":"The claim about less-spoken languages requiring more actions is based on Table 3, but the thresholds for 'far exceeds the average' (e.g., 'far above the average of 1-2 searches') are stated informally. Adding mean/median action counts with standard deviations would make the comparison more rigorous.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction is careful and the resource is potentially valuable, but the central comparative claim needs an English control inside X-WebAgentBench. The paper also cites the authors' own prior work (Qin et al., 2023, 2025) heavily; most citations are relevant, but the novelty framing should be checked so that the 'first multilingual interactive benchmark' claim is not overstated. The missing English control is fixable within the scope of the paper, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you lean on the 'multilingual lag' number. First, the benchmark is a real contribution: the first interactive web agent benchmark I know of that translates both the instructions and the environment into 14 languages, with a careful pipeline and quality checks. Second, the headline claim that multilingual performance trails English by over 20% is not actually measured here — it is an apples-to-oranges comparison against English WebShop scores from Yang et al. (2023).\n\nWhat is genuinely new: 2,800 instructions (200 per language), 589,946 translated products, a construction pipeline with pre-validation, GPT-4-based scoring, annotator onboarding, manual recheck, and a kappa of 0.8. That is solid work. The per-language scores in Table 2 are informative, and the descriptive findings — small models benefit from external translation, self-alignment only helps stronger models, agents overuse actions in lower-resource languages — are grounded in the actual runs.\n\nThe soft spot is the comparative claim. Section 1 and Figure 1 say multilingual performance lags English by over 20%, but the English numbers come from a different environment. X-WebAgentBench limits each instruction to an average of 211 products, filters to 200 quality-checked items per language, and uses an AgentBench-style prompt (Appendix E). Any of these can shift task scores by more than the size of the claimed gap. The paper never runs the same BaseAgent or CLP agent on the English version of its own 200 instructions, so the 'language gap' is confounded with environment, prompt, and selection differences. This same missing control weakens Section 5.5's conclusion that the bottleneck is language alignment rather than reasoning: DeepSeek-R1 has no English baseline either.\n\nSmaller issues: no error bars or significance tests on the per-language differences, the code/data link is a placeholder, and 'satisfactory' is never defined. None of these are fatal, but they need attention.\n\nWho this is for: anyone working on multilingual agents or cross-lingual evaluation. The resource deserves serious refereeing, and the central flaw is fixable — run an English control in the same environment and recalculate or qualify the gap. I would accept it for peer review with that expectation.","headline":"Useful multilingual agent benchmark resource, but the headline claim of a >20% language gap is not controlled: no English baseline is run inside the same environment.","tokens_in":18419,"tokens_out":2151,"would_cite":false,"duration_ms":19339,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current LLM web agents fall more than 20% behind English-only performance when tested across 14 non-English languages, and that the gap is a language-alignment problem, not a reasoning problem.","keywords":["multilingual agents","web agent benchmark","cross-lingual prompting","language alignment","WebShop","interactive environment","task score","GPT-4o"],"falsifier":"Run the same BaseAgent and CLP methods on the English version of X-WebAgentBench's 500 instructions in its 211-product environment. If the resulting English average task score is near or below the reported multilingual averages (around 37), the language-gap claim would lose its empirical basis; if English stays near the original WebShop level (roughly 50 or higher), the multilingual gap is confirmed.","tokens_in":17316,"feed_emoji":"🌐","tokens_out":7364,"duration_ms":57797,"temperature":0.7,"pith_summary":"The paper sets out to show that language agents built and evaluated in English do not transfer to other languages: when instructions and the shopping environment are translated into 14 languages, task scores drop by more than 20% relative to English-only results. To make this measurable, it constructs X-WebAgentBench, an interactive e-commerce benchmark with 2,800 multilingual instructions and 589,946 products, translated from WebShop. Across five LLMs and four agent strategies, no configuration reaches a satisfactory score; even GPT-4o, the strongest model tested, averages 37.11 with the base agent and 37.94 with cross-lingual prompting. The paper argues the bottleneck is language alignment rather than reasoning logic, since strong reasoning models do not close the gap.","feed_headline":"Even GPT-4o scores only 37 on multilingual web tasks","feed_subtitle":"A 14-language, 2,800-task benchmark shows the bottleneck is language alignment, not reasoning.","key_machinery":"The load-bearing instrument is X-WebAgentBench itself: an interactive web-shopping environment built by taking 500 WebShop instructions, translating them into 14 languages with Google Translate to form 2,800 instructions, and translating product data with GPT-4 to form 589,946 products, each instruction paired with about 211 products to keep the context window manageable. The four agent strategies compared are BaseAgent, Translate-en (external translation to English), Self-Translate-en (model self-translation), and CLP (cross-lingual prompting that first aligns the observation in English and then answers in the local language). The metric is the WebShop task score, which rewards buying the item that matches the instruction's attributes; the benchmark also records action steps, input/output tokens, and error types to diagnose where the gap comes from.","core_discovery":"The central discovery is that multilingual interactive environments expose a persistent performance gap in current agentic systems. Under the WebShop-style task score, GPT-4o reaches only 37.11 on average across the 14 languages with the base agent and 37.94 with cross-lingual prompting, and no tested model or alignment method approaches the reported English-only level, which the paper takes from the original WebShop results. The paper attributes the gap mainly to language alignment (Section 5.5): DeepSeek-R1, a strong reasoning model, performs about as well as GPT-4o and still stays below 50, so additional reasoning alone does not fix the problem. It also reports that cross-lingual self-alignment helps only larger multilingual-capable models, that small open models improve when an external translator converts the environment to English, and that interaction failures concentrate in no-output, click-error, and action-error categories.","pith_inferences":["A direct test the paper leaves implicit is an English control inside X-WebAgentBench; running the same BaseAgent and CLP on the English version would determine whether the reported >20% gap is robust or partly due to comparing with an external English benchmark.","The observation that agents never use an offered translation tool suggests that tool availability is not enough; future agents need a meta-decision rule for when to invoke translation, a design implication beyond the paper's stated findings.","If token-cost unfairness is script-driven, then tokenizers trained on more balanced script data could disproportionately improve low-resource language performance without changing model size, a hypothesis worth testing directly.","The paper's own Limitations section concedes that Google Translate plus quality checks cannot fully match experienced human translators, so translation noise bounds how precisely the benchmark can attribute score differences to agent capability rather than to translation artifacts."],"forward_implications":["Multilingual e-commerce agents will need explicit language-alignment components; English-centric training and prompting alone will not serve non-English users adequately.","Cross-lingual self-alignment should be reserved for models with strong multilingual capacity, while smaller models should be paired with external translation tools rather than asked to self-translate.","Because the paper finds token-cost unfairness driven by script rather than training-resource level, changing tokenizers is a concrete lever for making multilingual agents affordable.","Engineering more reasoning into agents will not close the language gap; improvements in multilingual understanding are the binding constraint.","The gap is measured even in a multi-agent framework (BOLAA), so it is a general property of current agentic systems, not an artifact of one prompting style."],"supporting_citations":[{"why":"Supplies the original WebShop environment, product data, and the task-score metric that X-WebAgentBench adapts.","marker":"Yao et al. (2022)"},{"why":"Provides the 500 WebShop test instructions and the ReAct-style action loop the benchmark and BaseAgent rely on.","marker":"Yao et al. (2023)"},{"why":"Source of the English-only task scores used in Figure 1 to compute the multilingual performance gap.","marker":"Yang et al. (2023)"},{"why":"Defines the BaseAgent evaluation setting and prompt format used as the main baseline.","marker":"Liu et al. (2024a)"},{"why":"Contributes CLP, the cross-lingual alignment prompting method tested as the strongest variant for GPT-4o.","marker":"Qin et al. (2023)"},{"why":"Underlies the Translate-en external-translation baseline and the cross-lingual demonstration experiments.","marker":"Shi et al. (2023)"},{"why":"Introduces the self-translation idea used by the Self-Translate-en baseline.","marker":"Etxaniz et al. (2024)"},{"why":"Guides the 14-language selection across language families for broad coverage.","marker":"Conneau et al. (2018)"},{"why":"Provides DeepSeek-R1, the reasoning model used to argue that the bottleneck is language alignment rather than complex reasoning.","marker":"Guo et al. (2025)"}],"fun_headline_variants":["Multilingual web tasks stump GPT-4o and all tested agents","GPT-4o hits 37 on multilingual benchmark, far below English level","Language alignment, not reasoning, is the real blocker for web agents","X-WebAgentBench: 14 languages, 2,800 tasks, no agent cracks the code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline claim that multilingual performance lags English by more than 20% assumes that English task scores reported for the original WebShop benchmark are comparable to scores measured here in a simplified, translated environment; the paper never runs an English control inside X-WebAgentBench.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual web tasks stump GPT-4o and all tested agents","GPT-4o hits 37 on multilingual benchmark, far below English level","Language alignment, not reasoning, is the real blocker for web agents","X-WebAgentBench: 14 languages, 2,800 tasks, no agent cracks the code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3848,"prompt_tokens":911,"completion_tokens":2937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2852}},"tokens_in":527,"tokens_out":2937,"duration_ms":19157,"temperature":1.0,"reasoning_tokens":2852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:18:36.369487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same BaseAgent and CLP methods on the English version of X-WebAgentBench's 500 instructions in its 211-product environment. If the resulting English average task score is near or below the reported multilingual averages (around 37), the language-gap claim would lose its empirical basis; if English stays near the original WebShop level (roughly 50 or higher), the multilingual gap is confirmed.","supporting_citations":[],"review_version":1}