{"id":"2d74e38b-74bb-4067-99de-3370e710df2a","arxiv_id":"2501.05478","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Arabic instructions on the R2R navigation task preserve much of GPT-4o mini's success rate but push Phi-3 and Jais to zero, indicating model capability rather than language is the decisive factor.","lead":"This paper translates 100 indoor navigation instructions from English to Arabic and runs four language models inside the NavGPT framework, finding that the model's multilingual robustness matters more than the input language. A generalist reader might care because it shows what works and where the main bottlenecks are for Arabic-speaking users of navigation agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated machine-translated Arabic R2R instructions confound the core language-robustness claim: observed Arabic performance drops could reflect translation or parsing artifacts rather than model reasoning ability.","rationale":"The reader's weakest assumption and my concern coincide: the Arabic dataset is generated by one machine-translation API with no validation, and the Limitations section explicitly concedes translation errors. I agree with the CONDITIONAL verdict because this is a fixable but central issue. The central empirical claim is comparative: 'the robustness of a model's reasoning and planning capabilities mattered more than the language itself.' That inference requires that the English and Arabic versions of each trajectory present the same navigation problem; otherwise language differences are confounded with translation fidelity and prompt/parsing differences. I considered other weaknesses—unequal completed trajectories, no confidence intervals, model-specific prompts—and they are real, but the translation-fidelity issue is the one that most directly threatens the comparison between languages. I also credit the paper for releasing code, using standard VLN metrics, and stating limitations, and the qualitative examples do show some genuine Arabic planning. But those positives do not substitute for a validation step. Since the reader already flagged exactly this assumption and chose CONDITIONAL, my stress-test does not move the verdict; it strengthens the rationale for conditioning acceptance on the proposed check.","tokens_in":10519,"tokens_out":3513,"duration_ms":36922,"concrete_test":"Back-translate a random sample of 50 Arabic instructions and observations from the released dataset to English using two independent native Arabic speakers or a high-quality commercial translator, and annotate each for semantically significant deviations in spatial relations (left/right, front/behind, landmark names, ordering). If any sample trajectory has a spatial error, rerun the 100-trajectory NavGPT evaluation on human-corrected Arabic translations for GPT-4o mini and Phi-3. If GPT-4o mini's Arabic SR remains near 20 and Phi-3 still fails completely, the translation confound is ruled out; if Arabic SRs shift materially, the article must either report corrected numbers or soften the attribution of Arabic failures to model capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the unvalidated machine translation of the R2R dataset, described in 'Dataset Translation' and acknowledged in 'Limitations and Future Work' ('Machine translation... can introduce errors or ambiguities'). The Arabic instructions, observations, and object lists were generated by Groq's Llama-3.2-90B-text-preview with no reported human validation, back-translation, or spot-check. Since R2R instructions encode spatial relations (turns, landmarks, room types, ordering) and NavGPT reasons over textual descriptions of observations, a mistranslated direction or object name directly changes the planning problem. If the Arabic dataset is systematically noisier than the English one, then the lower Arabic success rates of Phi-3 (SR 7.32 to 0, with only 18/100 trajectories completed) and Jais (0 in both languages, so less relevant) are confounded: they may reflect translation quality or Arabic-script parsing failures rather than 'inherent limitations' in Arabic reasoning. The paper's central claim—that model robustness matters more than input language—rests mainly on GPT-4o mini's similar English/Arabic SR (21 vs 20), but a robust model could also be resilient to translation noise, so parity does not by itself prove the translated dataset is faithful. Additionally, Arabic outputs were prompted in Arabic while NavGPT's parser/action format was designed for English (Experimental Setup: 'different prompts were used depending on the model'), so parsing issues are entangled with language effects. This is fixable, but as reported the strongest claim is not fully secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates four language models (GPT-4o mini, Llama 3 8B, Phi-3 medium 14B, and the Arabic-centric Jais 30B) in a zero-shot Vision-and-Language Navigation setting using the NavGPT framework on 100 R2R trajectories. The English R2R instructions, observation descriptors, and object lists are machine-translated into Arabic via Groq's Llama-3.2-90B-text-preview, and the models are compared on standard VLN metrics in English, Arabic, and a mixed Arabic-input/English-reasoning condition. The authors report that GPT-4o mini performs comparably in English and Arabic, while Phi-3 and Jais fail in Arabic, and conclude that model robustness and planning ability matter more than input language. The paper includes a qualitative analysis of success, parsing, and failure cases, and the code is made publicly available.","tokens_in":10731,"tokens_out":3116,"duration_ms":33325,"significance":"If the central claim holds, the paper provides a useful early data point for multilingual VLN: it benchmarks zero-shot navigation in Arabic, a language with very few VLN resources, and it compares small multilingual models against a dedicated Arabic LLM. The public code and the translated R2R subset are potentially reusable assets for the community. However, the empirical basis is narrow: 100 trajectories, no statistical inference, no validation of the translated inputs, and model-specific prompt/parsing configurations. The paper is therefore best read as an exploratory benchmark rather than a conclusive demonstration that input language is irrelevant. The central claim is plausible and testable, but the current evidence is not sufficient to rule out translation quality and pipeline artifacts as drivers of the observed Arabic performance drop.","major_comments":[{"comment":"The Arabic dataset was produced by automatic translation with Groq's Llama-3.2-90B-text-preview, and the manuscript reports no human validation, back-translation, or even spot-checking. The Limitations section itself acknowledges that machine translation 'can introduce errors or ambiguities.' This is load-bearing because R2R instructions encode spatial relations (turn directions, landmarks, room types, ordering), and NavGPT reasons over textual observations; a mistranslated direction or object name changes the planning problem. The Phi-3 Arabic collapse (SR 0.00 with only 18 of 100 trajectories completed) and the smaller Arabic drops for other models are therefore confounded with translation quality and Arabic-script parsing. The authors should either add a translation validation protocol (e.g., human evaluation on spatial-relation fidelity, back-translation agreement, or comparison against an existing Arabic VLN resource) or explicitly reframe the conclusions as being about this specific translated dataset, not about Arabic as a language.","section":"Dataset Translation / Limitations and Future Work"},{"comment":"The Experimental Setup states that 'different prompts were used depending on the model' and that Arabic inputs were paired with Arabic thought outputs, while the NavGPT parser and action format were designed for English. This changes at least three variables at once: the instruction language, the prompt template, and the output language/parsing expectations. Consequently, observed differences between English and Arabic, and between models, cannot be attributed solely to the models' Arabic reasoning ability. For example, Phi-3's Arabic failure is described as 'non-multilingual nature,' but it could equally reflect an incompatibility between the Arabic outputs and the English-designed parser. The authors should report the exact prompts and parser behavior for each language/model condition, and ideally hold the prompt template and output format fixed across languages or provide an ablation that isolates parsing failures from reasoning failures.","section":"Experimental Setup"},{"comment":"The quantitative results rest on 100 trajectories with no confidence intervals, significance tests, or paired per-trajectory analysis, while completion counts differ strongly across models (e.g., Phi-3 completes 41/100 in English but 18/100 in Arabic; Jais 95/100 and 82/100). The claim that GPT-4o mini's English and Arabic performance is 'comparable' is based on SR 21.0 vs 20.0, a difference that is well within sampling noise at this sample size, and the same is true for the 'approximately three times' and 'approximately five times' comparisons (7 vs 21 and 4 vs 21). Moreover, the central argument that 'model robustness matters more than language' is supported by exactly one model exhibiting language parity; parity could also mean that GPT-4o mini is robust to translation noise. The authors should provide per-trajectory paired results, confidence intervals or a simple significance test, and should temper the strength of the headline conclusion accordingly.","section":"Table 1 / Results and Discussion"}],"minor_comments":[{"comment":"There are several typos and formatting inconsistencies, including 'Evaulation Metrics' in the section heading and inconsistent use of 'Llama 3 8B' vs 'Llama 3'; a final proofread is needed.","section":"Throughout"},{"comment":"Several references are incomplete or not citable in their current form: 'Vidhyashree, A. 2024' appears to be a blog post, 'Team, S. 2024. Silma.' lacks a title and venue, and the Jais reference is missing its arXiv or publication details.","section":"References"},{"comment":"The qualitative examples in Figures 3–6 are referenced extensively but are not part of the submitted text; please ensure these figures are present, legible, and include the actual English and Arabic prompts, model outputs, and parser responses so the reader can verify the qualitative claims.","section":"Figures 3–6"},{"comment":"The 'Succ.' column is not formally defined in the metrics section; it appears to be the number of trajectories for which a valid prediction was produced, and this should be stated explicitly, along with how trajectories with no valid prediction are handled in the metric averages.","section":"Table 1"},{"comment":"The paper does not describe the Arabic translation prompt or any post-processing of Arabic text (e.g., normalization, diacritics, right-to-left handling); a brief description would help reproducibility, especially since the code repository is promised as the reference implementation.","section":"Dataset Translation"}],"recommendation":"major_revision","confidential_remarks":"The paper is an exploratory empirical benchmark with a plausible but not yet fully supported central claim. The main risks are the unvalidated translation and the language/prompt/parser confound, both of which are fixable within the scope of a revision. I would not recommend rejection, but the authors should either add validation and controlled comparisons or substantially soften the 'language is not the bottleneck' conclusion. The contribution is modest but potentially useful for the Arabic VLN community, especially given the public code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a new Arabic-language VLN benchmark (translated R2R subset) and a clean evaluation of four models in NavGPT. The headline result—GPT-4o mini performs about the same in English and Arabic (SR 21 vs 20), while weaker models collapse in both—is a useful signal that model robustness matters more than input language. But the translation is machine-generated and unvalidated, so the Arabic-specific drops (especially Phi-3) could be partly translation or parsing artifacts rather than pure reasoning limits. The paper admits this in the limitations section but doesn't address it.\n\nWhat's good: the dataset artifact is new (RxR notably excludes Arabic), the code is on GitHub, and the evaluation is straightforward and honest. The qualitative failure analysis (parsing errors, Jais failing) is illustrative. The literature review is adequate, and self-citations are not an issue.\n\nSoft spots: (1) No human validation, back-translation, or spot-check of the Arabic instructions. Since R2R instructions encode turns, landmarks, and ordering, a mistranslation directly changes the planning problem. (2) Unequal denominators: Phi-3 evaluated on 18/100 Arabic trajectories vs 41 English; SR 0% on 18 is not comparable to 7.32% on 41. (3) Different prompts per model confound prompt compatibility with reasoning ability. (4) No error bars or significance tests; 21 vs 20 is well within noise. (5) The qualitative claim of 'capable of high-level planning' rests on a couple of examples while quantitative success is 21/100 at best. That's an overstatement, though not a fatal one.\n\nThe central thesis probably survives these issues—Jais scores 0 in both languages, so language alone isn't the driver—but the evidence is weaker than the abstract suggests. The translation validation gap is the load-bearing soft spot and must be fixed before publication.\n\nWho this is for: researchers working on multilingual VLN, Arabic NLP, or deploying navigation agents in Arabic-speaking regions. It deserves a serious referee; the benchmark and code are worth engaging with, and the claims are falsifiable. I'd recommend peer review with major revisions: human evaluation of a translation sample, per-trajectory counts, confidence intervals, and at least a prompt-consistency check across models.\n\nMy verdict: conditional accept, with translation validation as the gate.","headline":"A modest first Arabic VLN benchmark with an honest but under-validated translation step; the central robustness-vs-language claim is plausible but not fully secured.","tokens_in":11351,"tokens_out":4107,"would_cite":true,"duration_ms":34509,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in zero-shot robot navigation from language instructions, the model's own reasoning and output-formatting strength determines success more than whether the instruction is in English or Arabic.","keywords":["Vision-and-Language Navigation (VLN)","Small Language Models (SLMs)","Large Language Models (LLMs)","Arabic language","Jais","NavGPT","R2R dataset","zero-shot navigation planning"],"falsifier":"Run the same 100 trajectories with human-verified Arabic translations, or with back-translation quality checks, and compare the success rates; if Phi-3 or Jais scores rise on Arabic to near their English level, then the observed Arabic failure was caused by translation quality, not by the model's Arabic capability.","tokens_in":10267,"feed_emoji":"🧭","tokens_out":7093,"duration_ms":63209,"temperature":0.7,"pith_summary":"The paper tackles a practical question: can a robot navigate from Arabic instructions as well as from English, or does the model's own reasoning ability dominate? It adapts the NavGPT text-only navigation agent to accept Arabic, translates 100 Room-to-Room trajectories into Arabic, and runs four language models—GPT-4o mini, Llama 3 8B, Phi-3 medium 14B, and the Arabic-centric Jais 30B—zero-shot, with no navigation-specific training. The central claim is that model robustness, not instruction language, decides success: a strong multilingual model performs comparably in both languages, while a model with weak Arabic exposure fails on Arabic and an Arabic-centric model can still score zero in both. If the claim holds, improving general planning, parsing, and multilingual reasoning is more useful for Arabic navigation than language-specific fine-tuning.","feed_headline":"Model strength, not instruction language, decides Arabic navigation success","feed_subtitle":"Four language models tested zero-shot on English and Arabic Room-to-Room routes; GPT-4o mini leads by 3-5x.","key_machinery":"NavGPT is the central mechanism: a pure LLM navigation agent that converts visual observations into textual descriptions and lets the language model choose the next viewpoint from a predefined navigation graph. At each step the Prompt Manager combines the instruction, current observation, navigation history, and candidate viewpoints into a structured prompt, and the model must output a thought and an action in a fixed format; a history buffer summarizes past observations. Because the agent is text-only and zero-shot, any language effect is isolated in the model's reasoning and parsing, which is exactly the comparison the paper makes between English and Arabic runs.","core_discovery":"The paper establishes, for the first time in this setting, that Arabic instructions can drive high-level navigation planning in a pure language-model agent, and that the bottleneck is the model, not the language. GPT-4o mini reaches a success rate of 21.0 on English and 20.0 on Arabic; Llama 3 manages 4.0 and 3.12; Phi-3 medium drops from 7.32 on English to 0.00 on Arabic; and Jais 30B, despite being Arabic-centric, scores 0.00 on both. The authors attribute Phi-3's Arabic collapse to limited Arabic training and formatting or parsing failures, and Jais's overall failure to a lack of instruction-following fine-tuning for navigation. The conclusion is that the architecture and multilingual training of the language model matter more than the input language, and that Arabic-centric pretraining alone does not produce navigation competence.","pith_inferences":["The paper's Arabic results should be read as evidence about machine-translated instructions; if human-validated Arabic translations were used, the Phi-3 and Jais gaps could change, because translation artifacts such as altered spatial relations were not controlled.","The same protocol could be applied to other low-resource languages, and to Arabic instructions written natively rather than translated from English, which would test whether spatial language itself, not translation, affects planning.","Because the framework is text-only, the ranking might shift once a vision encoder is added; the claim is most directly about the language-and-planning layer, not about full multimodal navigation.","A cheap improvement path suggested by the data is prompt-structure hardening: constraining output schemas and retrying on parse failures could raise multiple models without any additional language training."],"forward_implications":["A strong multilingual SLM can handle English and Arabic navigation instructions in a zero-shot text-only pipeline, so language support does not necessarily require training a separate Arabic model.","Arabic-centric models should be instruction-tuned for navigation and output formatting before being deployed; Arabic training data by itself did not help Jais reach any goal.","Parsing and format adherence are first-class problems: several trajectory failures came from invalid viewpoint IDs or missing action outputs, not from misunderstanding the route.","The Arabic-augmented R2R data and the NavGPT evaluation recipe give later work a direct baseline for comparing newer Arabic models and translation methods.","Improvements to general multilingual reasoning could transfer directly to VLN, since the measured gap within a capable model across languages is small."],"supporting_citations":[{"why":"Supplies the NavGPT framework and the prompt-based zero-shot action prediction pipeline that all experiments run through.","marker":"Zhou, Hong, and Wu 2023"},{"why":"Provides the Room-to-Room dataset, the 100 val-unseen trajectories, and the SR, SPL, and NE metrics used in both languages.","marker":"Anderson et al. 2018"},{"why":"Defines Jais, the Arabic-centric 30B model whose failure in both languages is a key result.","marker":"Sengupta et al. 2023"},{"why":"Defines Llama 3 8B, one of the multilingual SLMs compared.","marker":"Dubey 2024"},{"why":"Defines Phi-3 medium 14B, the model whose Arabic failure is attributed to non-multilingual training.","marker":"Abdin et al. 2024"},{"why":"Documents GPT-4o mini, the model that performs best in both languages.","marker":"Vidhyashree 2024"}],"fun_headline_variants":["Arabic robot navigation: model strength beats language skill","GPT-4o mini tops Arabic and English navigation; Jais zero","First Arabic VLN evaluation: model capability trumps input language","Model limits, not Arabic, explain navigation failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Arabic instructions produced by automatic machine translation are faithful enough to the English originals that any performance gap can be blamed on the model's reasoning rather than on translation errors or ambiguities.","fun_headline_variants_meta":{"raw":{"variants":["Arabic robot navigation: model strength beats language skill","GPT-4o mini tops Arabic and English navigation; Jais zero","First Arabic VLN evaluation: model capability trumps input language","Model limits, not Arabic, explain navigation failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001455,"raw_usage":{"total_tokens":5877,"prompt_tokens":987,"completion_tokens":4890,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":4823}},"tokens_in":603,"tokens_out":4890,"duration_ms":35899,"temperature":1.0,"reasoning_tokens":4823,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:44:18.983733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 100 trajectories with human-verified Arabic translations, or with back-translation quality checks, and compare the success rates; if Phi-3 or Jais scores rise on Arabic to near their English level, then the observed Arabic failure was caused by translation quality, not by the model's Arabic capability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Jais, the Arabic-centric 30B model whose failure in both languages is a key result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents GPT-4o mini, the model that performs best in both languages."}],"review_version":1}