{"id":"3ba1bfab-7373-4042-ad35-b62aaf08293e","arxiv_id":"2507.20395","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A new maze-navigation benchmark claims LLM spatial reasoning is language-dependent, with O3 exceptional and other models failing by looping, but the looping result is an artifact of the termination rule.","lead":"MazeEval tests eight LLMs on coordinate-only maze navigation in English and Icelandic. It reports that only OpenAI's O3 navigates large mazes reliably, that most models loop catastrophically beyond 9x9, and that Icelandic performance is worse, but key findings are built into the benchmark rules.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cross-linguistic significance test is invalid because the stopping rule makes the paired observations incomplete; the reported p<0.001 likely depends on unmeasured sizes being imputed as zeros.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern I do: the Wilcoxon signed-rank test is not valid as described because the stopping rule leaves unmeasured success rates for larger maze sizes. This is the most load-bearing issue because the paper's two headline contributions are the catastrophic looping failure and the cross-linguistic degradation. The looping claim is operationally tied to the termination threshold, but the paper reports no move-budget failures, so it is not purely an artifact; it is a real behavioral pattern though less surprising. The O3 30x30 claim is based on a single maze and would weaken the paper's secondary claim about exceptional performance, but it is not the central novelty. The cross-linguistic claim, however, is the abstract's main inference: 'spatial reasoning in LLMs emerges from linguistic patterns rather than language-agnostic mechanisms.' That inference is supported only by the significance test, and the test's paired structure is broken by the stopping rule. Without knowing whether unmeasured sizes were excluded or imputed, the p<0.001 cannot be verified. The paper could be fixed by running a complete paired protocol, reporting only measured sizes, or clearly stating and justifying an imputation strategy. Since the reader already rejected on this basis and my analysis confirms it, the verdict should remain unchanged.","tokens_in":13109,"tokens_out":6181,"duration_ms":73017,"concrete_test":"Ask the authors to release the per-run data for every model, language, and maze size. Reproduce the Wilcoxon signed-rank test two ways: (1) using only sizes where both English and Icelandic were actually tested (no imputation), and (2) using all 11 sizes with unmeasured rates set to 0. If the aggregate p-value fails to reach significance under (1), or if the effect size drops below the reported d=0.50, the cross-linguistic conclusion is an artifact of the stopping rule. The paper should state which analysis it used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline cross-linguistic claim depends on a Wilcoxon signed-rank test (Section 3.7) that is invalid under its own stopping rule. Section 3.3 and the Figure 3 caption state that testing stops at the first maze size where a model fails all five attempts. Section 3.7 says the test treats success rates at each of the 11 sizes (5x5 to 15x15) as paired observations. For any size beyond the stop point in one language, no success rate was measured. To include those sizes, the authors must either exclude the pair (reducing the effective sample size well below 11 per model) or silently impute 0. The reported W=273, p<0.001 across all models and the individual p-values are not reproducible without disclosing which choice was made. Since the abstract's conclusion that spatial reasoning emerges from linguistic patterns rests on this significance test, the central claim is not supported as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MazeEval, a benchmark in which language models navigate coordinate-based mazes from 5x5 to 15x15 by calling a move function with only current coordinates, distances to walls, and a visit history; eight models are evaluated on identical mazes in English and Icelandic. The paper reports that OpenAI's O3 solves all tested mazes and a single 30x30 maze, that other models fail catastrophically beyond roughly 9x9, that 100% of failures are excessive-looping terminations, and that a Wilcoxon signed-rank test shows significantly lower success in Icelandic, leading to the conclusion that spatial reasoning in LLMs emerges from linguistic patterns rather than language-agnostic mechanisms.","tokens_in":13400,"tokens_out":8983,"duration_ms":96275,"significance":"The benchmark design has useful properties: deterministic maze generation with fixed seeds, identical maze layouts across languages, and a function-calling interface that avoids parsing ambiguities. If the data were analyzed with a protocol that respects its own stopping rule, the benchmark could provide a solid descriptive picture of LLM navigation ability. However, the headline results as reported are not supportable because the loop-failure statistic is definitionally enforced, the key significance test is applied to censored paired data, and the O3 and cross-linguistic size-drop claims exceed the evidence. The central contribution is therefore currently a benchmark in need of corrected analysis rather than a validated set of findings.","major_comments":[{"comment":"The claim that 100% of failures are due to excessive looping is an artifact of the evaluation protocol. Section 3.3 imposes a per-cell visit cap of 10 and a move budget of 3n^2; Section 4.5 reports that no run ever exceeded the move budget. Under these rules every terminating failure is, by construction, a loop-rule termination. The abstract and Section 5.3 therefore overinterpret this statistic as evidence of a \"universal failure mode\" or a lack of spatial memory. To make an empirical looping claim, the authors should report the distribution of maximum visit counts and the number of runs terminated by each rule, and should avoid presenting the 100% figure as an empirical discovery.","section":"§3.3, §4.5"},{"comment":"The Wilcoxon signed-rank test is not valid on the data as collected. The Figure 3 caption and Section 3.3 state that testing stops at the first maze size where a model fails all five attempts, so for each language there are sizes beyond the stopping point with no measured success rate. Section 3.7 nonetheless treats the success rate at each of the 11 sizes as a paired observation. The reported aggregate W=273, p<0.001 and the per-model p-values are not reproducible unless the authors specify whether missing sizes were excluded or silently imputed as zero. Both choices change the result, and silent zero-imputation would bias the test toward the reported significance. The paper should restrict the paired test to sizes measured in both languages, report the effective number of pairs, or use a censoring-aware method.","section":"§3.7, Fig. 3 caption"},{"comment":"The abstract's statement that O3 achieves \"perfect navigation for mazes up to size 30x30\" is not supported by the reported data. The systematic evaluation covers sizes 5x5 through 15x15, and the 30x30 result in Section 4.3 comes from \"additional experimentation\" with a single maze in each language. Perfect navigation up to a size normally implies all sizes and multiple instances. The authors should report how many 30x30 mazes were attempted, how many were solved, and should state the claim as \"a single 30x30 maze was solved in both languages,\" with the 40x40 failure also based on a single maze.","section":"§4.3, Abstract"},{"comment":"The conclusion that models \"typically\" solve mazes 3-4 sizes smaller in Icelandic is not supported by Table 1. In that table only Claude Opus 4 (11 to 8), Claude Sonnet 4 (12 to 8), and Gemini 2.5 Pro (11 to 8) show a drop of 3 or 4 sizes; Gemini 2.5 Flash and GPT-4o show no drop, GPT-4.1-mini drops one size, and GPT-4o-mini has no measurable maximum in either language. The aggregate language effect may still be present, but the size-drop claim should be stated descriptively with per-model counts rather than as the typical pattern.","section":"§4.3, Table 1, §6"}],"minor_comments":[{"comment":"The Results section says \"We evaluated seven state-of-the-art language models\" but Section 3.6 and Table 1 list eight models; the count should be corrected.","section":"§4"},{"comment":"There is a typo in \"we did not purse five repeated evaluations\" which should read \"pursue.\"","section":"§4.3"},{"comment":"The limitations section should explicitly state that the stop-at-first-failure rule censors observations for large maze sizes and that the loop-failure category is defined by the 10-visit threshold; neither is currently acknowledged.","section":"§5.4"},{"comment":"The phrase \"repeated experiments would surely provide the power to detect smaller language performance differences\" is speculative; a power analysis or a more cautious phrasing is needed.","section":"§4.3"},{"comment":"The paper does not include a data or code release link for the benchmark or the raw per-run results; given the reproducibility claims in Section 3.1, the authors should provide the raw data and scripts.","section":"§3.1"},{"comment":"The claim that \"no dataset of this specific type ... has been publicly released before\" is a strong contamination-related assertion that needs a citation or a clear statement of the benchmark's release date; otherwise it is unverifiable.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"I would not accept the paper in its current form. The headline claims, especially the 100% looping statistic and the cross-linguistic p-value, are likely to be cited and need to be corrected before publication. The benchmark itself is potentially useful, so I support a major revision rather than outright rejection, provided the authors are willing to re-analyze the data under a protocol consistent with their stopping rule and to release the raw results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the MazeEval paper. My short take: the benchmark idea is real and worth having, but the two headline claims—100% looping failures and the Icelandic degradation—are not supported as stated. The first is definitional, the second rests on a statistical test that the paper's own stopping rule invalidates.\n\nWhat's new: a coordinate-only maze navigation task with function calling, tested in English and Icelandic on eight models. That's a sensible minimal-sensory-agent setup, and the observation that most models collapse around 9x9 while O3 sails through is a real empirical pattern, even if it's presented with more confidence than the data allow. The related work is fair and the prompt design is transparent.\n\nThe soft spots are load-bearing. The '100% of failures due to excessive looping' is guaranteed by the protocol: runs terminate at 10 visits to a cell, and the move budget never binds. So that number is a definition, not a discovery. The cross-linguistic result depends on a Wilcoxon signed-rank test that treats success rates at 11 maze sizes as paired observations, but the protocol stops testing at the first size where a model fails all five mazes. Beyond that point, the English and Icelandic observations are missing. To get W=273, p<0.001, the authors had to either exclude those sizes (which would shrink n and make the p-value implausible) or impute zeros without saying so. Either way, the reported test is not reproducible from the text. There's also the O3 30x30 claim resting on a single maze, no repeats.\n\nI don't think the paper is dishonest; I think it's overconfident. The benchmark is useful and the cross-linguistic degradation is plausible. But as it stands, the evidence doesn't support the abstract's strong conclusion that spatial reasoning emerges from linguistic patterns. A revised version with released code/data, a properly paired analysis (e.g., compare only sizes where both languages were measured, or do a mixed model), and an honest restatement of what the failure-mode analysis can and cannot say could change my mind.\n\nWho should read it: people building multilingual agent benchmarks and anyone deploying LLM navigators in low-resource languages. It deserves a serious referee—the topic matters and the benchmark deserves scrutiny—but it needs major revision before publication.\n\nRegards.","headline":"A useful benchmark undercut by a definitional failure claim and a statistically invalid cross-linguistic test.","tokens_in":13772,"tokens_out":2979,"would_cite":false,"duration_ms":29775,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MazeEval, a coordinate-only maze navigation benchmark, finds that most large language models fail on mazes larger than 9×9, with every failure caused by excessive looping, and that the same models solve mazes three to four sizes smaller…","keywords":["spatial reasoning","agent evaluation","multilingual benchmarks","maze navigation","function calling","sequential decision-making","looping behavior","Icelandic"],"falsifier":"Re-run the full English and Icelandic maze battery without early stopping—measure all eleven sizes from $5\\times 5$ to $15\\times 15$ for every model—and recompute the Wilcoxon signed-rank test using only the actually measured paired differences; if the effect no longer reaches the Bonferroni-corrected threshold, the language-dependence claim collapses. A second decisive check: add a few English-style navigation examples to the Icelandic prompt and see whether the three-to-four-size gap disappears, which would show the effect is about instruction-following rather than spatial reasoning.","tokens_in":12894,"feed_emoji":"🧭","tokens_out":10405,"duration_ms":96924,"temperature":0.7,"pith_summary":"This paper introduces MazeEval, a benchmark that tests whether large language models can navigate a maze using only coordinate feedback and distance-to-wall information, with no visual input. Across a set of state-of-the-art models running identical mazes in English and Icelandic, the benchmark finds a sharp tiered split: one model, OpenAI's O3, solves mazes up to $30\\times 30$ perfectly, while all other models collapse beyond roughly $9\\times 9$ grids, and 100 percent of failures are traced to a single behavior—revisiting the same cell at least ten times. The paper also reports that models solve mazes three to four sizes smaller in Icelandic than in English, which the author reads as evidence that spatial reasoning in LLMs is shaped by linguistic patterns and training-data availability rather than being language-agnostic. The practical stakes are that agents deployed in robotics and embodied AI without visual cues may inherit these looping and language-dependent limitations.","feed_headline":"Most AI models loop in circles on mazes over 9×9","feed_subtitle":"New benchmark finds every failure is a model revisiting the same cell; Icelandic drops performance by 3-4 maze sizes.","key_machinery":"The carrying object is the MazeEval task itself: a function-calling interface in which the model must call a `move` function choosing north, south, east, or west, receiving only its current $(x,y)$ coordinates, the distance to walls in each cardinal direction, the goal coordinates, and a full history of its visited positions with the visibility distances observed there. Two constraints do the work of turning raw behavior into a diagnosis: a cap of ten visits to any cell and a movement budget of $3n^2$ steps for an $n\\times n$ maze. The ten-visit cap defines the paper's central failure category, 'excessive looping,' and the paired English-Icelandic design with identical maze configurations is what lets the language comparison be attributed to the linguistic framing rather than maze difficulty.","core_discovery":"The central claim is that current large language models do not possess a language-agnostic spatial reasoning ability: when navigation is stripped to coordinates and wall distances, most models cannot maintain a spatial state beyond a small maze, and their competence degrades measurably when the same task is posed in Icelandic rather than English. Concretely, the paper reports perfect performance by O3 on all tested sizes up to $30\\times 30$ and failure at $40\\times 40$, while every other evaluated model falls off steeply at or before $9\\times 9$ mazes. In the failure analysis, every observed failure is attributed to excessive looping—a cell visited at least ten times—rather than to step-budget exhaustion or wall collisions. The cross-linguistic comparison shows a consistent English advantage, with models solving mazes three to four sizes smaller in Icelandic, and the aggregated Wilcoxon signed-rank test is reported as significant ($W=273$, $p<0.001$, Cohen's $d=0.50$). The author interprets these results as evidence that spatial intelligence in LLMs is a language-dependent skill acquired from linguistic patterns, not a universal mechanism.","pith_inferences":["The ten-visit cap may be doing more work than the paper credits: a model that would eventually escape is terminated early, so the '100% looping' statistic conflates persistent revisiting with failure to escape within the allowed visits; varying the cap would separate the two.","The reported language effect could be partly an instruction-following effect: Icelandic prompts may be less familiar to the models than the spatial task itself, so a control condition with English instructions but Icelandic direction words (or vice versa) would clarify whether spatial reasoning or language comprehension drives the gap.","If the language dependence is real, benchmark suites for embodied AI should be per-language rather than English-only, and a model's deployment readiness should be certified on the language it will actually operate in.","The distance-to-wall feedback gives models partial visibility that a human navigating blind would not have; comparing MazeEval against a binary wall-detection variant would quantify how much of the difficulty comes from integrating distance information."],"forward_implications":["Robotic and embodied agents that rely on LLMs with only coordinate or distance feedback will likely enter loops on spaces larger than about $8\\times 8$, so deployment will need external loop-breaking or memory mechanisms.","Multilingual deployments should expect systematically worse spatial performance in lower-resource languages like Icelandic, with implications for safety-critical navigation tasks outside English.","O3's perfect navigation up to $30\\times 30$ shows the failure is not a hard architectural limit; training, post-training, or architecture changes can in principle remove the looping failure.","Because MazeEval is the first public benchmark of its kind, the results provide an unpolluted measure of spatial reasoning that can serve as a baseline for future models."],"supporting_citations":[{"why":"Supplies the prior StepGame finding that LLMs struggle with multi-hop spatial reasoning, which motivates MazeEval's design and the claim that spatial reasoning did not emerge spontaneously.","marker":"(Li et al., 2024)"},{"why":"Documents ChatGPT's performance drop on non-English languages, the comparative baseline for MazeEval's Icelandic degradation result.","marker":"(Lai et al., 2023)"},{"why":"Shows large performance gaps on African languages in LLM benchmarks, reinforcing the paper's training-data-availability explanation for the English-Icelandic gap.","marker":"(Adelani et al., 2025)"},{"why":"Previous benchmark on spatial-temporal path planning that MazeEval extends by removing visual input and using coordinate-only feedback.","marker":"(Aghzal et al., 2023)"},{"why":"BabyAI grid-world navigation benchmark, one of the prior spatial task families MazeEval distinguishes itself from by not providing rich instructions or visual context.","marker":"(Chevalier-Boisvert et al., 2018)"},{"why":"The 'illusion of thinking' framework used to interpret O3's performance ceiling at $40\\times 40$ mazes, supporting the claim that even strong models fail beyond their complexity distribution.","marker":"(Shojaee*† et al., 2025)"}],"fun_headline_variants":["MazeEval: LLMs loop to failure past 9×9 mazes","Spatial reasoning in LLMs is a language skill, not universal","O3 solves 30×30 mazes; other AIs loop at 9×9","Icelandic mazes trip up LLM navigation by 3–4 sizes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The statistical comparison assumes that English and Icelandic success rates can be paired at every maze size, but the protocol stops testing a model at the first size where it fails all five mazes, so the reported significance depends on treating the unmeasured larger sizes as paired observations with zero success.","fun_headline_variants_meta":{"raw":{"variants":["MazeEval: LLMs loop to failure past 9×9 mazes","Spatial reasoning in LLMs is a language skill, not universal","O3 solves 30×30 mazes; other AIs loop at 9×9","Icelandic mazes trip up LLM navigation by 3–4 sizes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1773,"prompt_tokens":1075,"completion_tokens":698,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":621}},"tokens_in":691,"tokens_out":698,"duration_ms":7247,"temperature":1.0,"reasoning_tokens":621,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:32:36.038749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full English and Icelandic maze battery without early stopping—measure all eleven sizes from $5\\times 5$ to $15\\times 15$ for every model—and recompute the Wilcoxon signed-rank test using only the actually measured paired differences; if the effect no longer reaches the Bonferroni-corrected threshold, the language-dependence claim collapses. A second decisive check: add a few English-style navigation examples to the Icelandic prompt and see whether the three-to-four-size gap disappears, which would show the effect is about instruction-following rather than spatial reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows large performance gaps on African languages in LLM benchmarks, reinforcing the paper's training-data-availability explanation for the English-Icelandic gap."}],"review_version":1}