Pith. sign in

REVIEW 4 major objections 6 minor 21 references

MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MazeEval, a coordinate-only maze navigation benchmark, finds that most large language models fail on mazes larger than 9×9, with every failure caused by excessive looping, and that the same models solve mazes three to four sizes smaller…

desk verdict A useful benchmark undercut by a definitional failure claim and a statistically invalid cross-linguistic test. read the letter →

arxiv 2507.20395 v1 pith:QLTVUHNX submitted 2025-07-27 cs.AI

Hafsteinn Einarsson This is my paper · ORCID
classification cs.AI
keywords spatialreasoningagentevaluationmultilingualbenchmarksmazenavigationfunctioncallingsequentialdecision-makingloopingbehaviorIcelandic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MazeEval, a benchmark that tests whether large language models can navigate a maze using only coordinate feedback and distance-to-wall information, with no visual input. Across a set of state-of-the-art models running identical mazes in English and Icelandic, the benchmark finds a sharp tiered split: one model, OpenAI's O3, solves mazes up to $30\times 30$ perfectly, while all other models collapse beyond roughly $9\times 9$ grids, and 100 percent of failures are traced to a single behavior—revisiting the same cell at least ten times. The paper also reports that models solve mazes three to four sizes smaller in Icelandic than in English, which the author reads as evidence that spatial reasoning in LLMs is shaped by linguistic patterns and training-data availability rather than being language-agnostic. The practical stakes are that agents deployed in robotics and embodied AI without visual cues may inherit these looping and language-dependent limitations.

What carries the argument

The carrying object is the MazeEval task itself: a function-calling interface in which the model must call a `move` function choosing north, south, east, or west, receiving only its current $(x,y)$ coordinates, the distance to walls in each cardinal direction, the goal coordinates, and a full history of its visited positions with the visibility distances observed there. Two constraints do the work of turning raw behavior into a diagnosis: a cap of ten visits to any cell and a movement budget of $3n^2$ steps for an $n\times n$ maze. The ten-visit cap defines the paper's central failure category, 'excessive looping,' and the paired English-Icelandic design with identical maze configurations is what lets the language comparison be attributed to the linguistic framing rather than maze difficulty.

What would settle it

Re-run the full English and Icelandic maze battery without early stopping—measure all eleven sizes from $5\times 5$ to $15\times 15$ for every model—and recompute the Wilcoxon signed-rank test using only the actually measured paired differences; if the effect no longer reaches the Bonferroni-corrected threshold, the language-dependence claim collapses. A second decisive check: add a few English-style navigation examples to the Icelandic prompt and see whether the three-to-four-size gap disappears, which would show the effect is about instruction-following rather than spatial reasoning.

Watch

Extended reading notes

Core claim

The central claim is that current large language models do not possess a language-agnostic spatial reasoning ability: when navigation is stripped to coordinates and wall distances, most models cannot maintain a spatial state beyond a small maze, and their competence degrades measurably when the same task is posed in Icelandic rather than English. Concretely, the paper reports perfect performance by O3 on all tested sizes up to $30\times 30$ and failure at $40\times 40$, while every other evaluated model falls off steeply at or before $9\times 9$ mazes. In the failure analysis, every observed failure is attributed to excessive looping—a cell visited at least ten times—rather than to step-budget exhaustion or wall collisions. The cross-linguistic comparison shows a consistent English advantage, with models solving mazes three to four sizes smaller in Icelandic, and the aggregated Wilcoxon signed-rank test is reported as significant ($W=273$, $p<0.001$, Cohen's $d=0.50$). The author interprets these results as evidence that spatial intelligence in LLMs is a language-dependent skill acquired from linguistic patterns, not a universal mechanism.

Load-bearing premise

The statistical comparison assumes that English and Icelandic success rates can be paired at every maze size, but the protocol stops testing a model at the first size where it fails all five mazes, so the reported significance depends on treating the unmeasured larger sizes as paired observations with zero success.

Editorial extensions

If this is right

  • Robotic and embodied agents that rely on LLMs with only coordinate or distance feedback will likely enter loops on spaces larger than about $8\times 8$, so deployment will need external loop-breaking or memory mechanisms.
  • Multilingual deployments should expect systematically worse spatial performance in lower-resource languages like Icelandic, with implications for safety-critical navigation tasks outside English.
  • O3's perfect navigation up to $30\times 30$ shows the failure is not a hard architectural limit; training, post-training, or architecture changes can in principle remove the looping failure.
  • Because MazeEval is the first public benchmark of its kind, the results provide an unpolluted measure of spatial reasoning that can serve as a baseline for future models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ten-visit cap may be doing more work than the paper credits: a model that would eventually escape is terminated early, so the '100% looping' statistic conflates persistent revisiting with failure to escape within the allowed visits; varying the cap would separate the two.
  • The reported language effect could be partly an instruction-following effect: Icelandic prompts may be less familiar to the models than the spatial task itself, so a control condition with English instructions but Icelandic direction words (or vice versa) would clarify whether spatial reasoning or language comprehension drives the gap.
  • If the language dependence is real, benchmark suites for embodied AI should be per-language rather than English-only, and a model's deployment readiness should be certified on the language it will actually operate in.
  • The distance-to-wall feedback gives models partial visibility that a human navigating blind would not have; comparing MazeEval against a binary wall-detection variant would quantify how much of the difficulty comes from integrating distance information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces MazeEval, a benchmark in which language models navigate coordinate-based mazes from 5x5 to 15x15 by calling a move function with only current coordinates, distances to walls, and a visit history; eight models are evaluated on identical mazes in English and Icelandic. The paper reports that OpenAI's O3 solves all tested mazes and a single 30x30 maze, that other models fail catastrophically beyond roughly 9x9, that 100% of failures are excessive-looping terminations, and that a Wilcoxon signed-rank test shows significantly lower success in Icelandic, leading to the conclusion that spatial reasoning in LLMs emerges from linguistic patterns rather than language-agnostic mechanisms.

Significance. The benchmark design has useful properties: deterministic maze generation with fixed seeds, identical maze layouts across languages, and a function-calling interface that avoids parsing ambiguities. If the data were analyzed with a protocol that respects its own stopping rule, the benchmark could provide a solid descriptive picture of LLM navigation ability. However, the headline results as reported are not supportable because the loop-failure statistic is definitionally enforced, the key significance test is applied to censored paired data, and the O3 and cross-linguistic size-drop claims exceed the evidence. The central contribution is therefore currently a benchmark in need of corrected analysis rather than a validated set of findings.

major comments (4)
  1. [§3.3, §4.5] The claim that 100% of failures are due to excessive looping is an artifact of the evaluation protocol. Section 3.3 imposes a per-cell visit cap of 10 and a move budget of 3n^2; Section 4.5 reports that no run ever exceeded the move budget. Under these rules every terminating failure is, by construction, a loop-rule termination. The abstract and Section 5.3 therefore overinterpret this statistic as evidence of a "universal failure mode" or a lack of spatial memory. To make an empirical looping claim, the authors should report the distribution of maximum visit counts and the number of runs terminated by each rule, and should avoid presenting the 100% figure as an empirical discovery.
  2. [§3.7, Fig. 3 caption] The Wilcoxon signed-rank test is not valid on the data as collected. The Figure 3 caption and Section 3.3 state that testing stops at the first maze size where a model fails all five attempts, so for each language there are sizes beyond the stopping point with no measured success rate. Section 3.7 nonetheless treats the success rate at each of the 11 sizes as a paired observation. The reported aggregate W=273, p<0.001 and the per-model p-values are not reproducible unless the authors specify whether missing sizes were excluded or silently imputed as zero. Both choices change the result, and silent zero-imputation would bias the test toward the reported significance. The paper should restrict the paired test to sizes measured in both languages, report the effective number of pairs, or use a censoring-aware method.
  3. [§4.3, Abstract] The abstract's statement that O3 achieves "perfect navigation for mazes up to size 30x30" is not supported by the reported data. The systematic evaluation covers sizes 5x5 through 15x15, and the 30x30 result in Section 4.3 comes from "additional experimentation" with a single maze in each language. Perfect navigation up to a size normally implies all sizes and multiple instances. The authors should report how many 30x30 mazes were attempted, how many were solved, and should state the claim as "a single 30x30 maze was solved in both languages," with the 40x40 failure also based on a single maze.
  4. [§4.3, Table 1, §6] The conclusion that models "typically" solve mazes 3-4 sizes smaller in Icelandic is not supported by Table 1. In that table only Claude Opus 4 (11 to 8), Claude Sonnet 4 (12 to 8), and Gemini 2.5 Pro (11 to 8) show a drop of 3 or 4 sizes; Gemini 2.5 Flash and GPT-4o show no drop, GPT-4.1-mini drops one size, and GPT-4o-mini has no measurable maximum in either language. The aggregate language effect may still be present, but the size-drop claim should be stated descriptively with per-model counts rather than as the typical pattern.
minor comments (6)
  1. [§4] The Results section says "We evaluated seven state-of-the-art language models" but Section 3.6 and Table 1 list eight models; the count should be corrected.
  2. [§4.3] There is a typo in "we did not purse five repeated evaluations" which should read "pursue."
  3. [§5.4] The limitations section should explicitly state that the stop-at-first-failure rule censors observations for large maze sizes and that the loop-failure category is defined by the 10-visit threshold; neither is currently acknowledged.
  4. [§4.3] The phrase "repeated experiments would surely provide the power to detect smaller language performance differences" is speculative; a power analysis or a more cautious phrasing is needed.
  5. [§3.1] The paper does not include a data or code release link for the benchmark or the raw per-run results; given the reproducibility claims in Section 3.1, the authors should provide the raw data and scripts.
  6. [§1] The claim that "no dataset of this specific type ... has been publicly released before" is a strong contamination-related assertion that needs a citation or a clear statement of the benchmark's release date; otherwise it is unverifiable.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: cross-linguistic comparison is self-contained; the lone self-citation is background, and the 100%-looping statistic is an operational definition rather than a derived prediction.

full rationale

The paper's central claim—that success rates drop in Icelandic relative to English—rests on measured success rates and a Wilcoxon signed-rank test; no parameter is fitted and then relabeled as a prediction. The termination rule in Section 3.3 (10 visits per cell or 3n^2 moves) defines the failure taxonomy, and Section 4.5's '100% of failures were due to excessive cell visits' is a report of which protocol-defined termination condition bound empirically, not a quantity derived from the protocol, so it is not a circular reduction. The only self-citation (Matheus Gauy et al., 2018) is used as background neuroscience in Section 5.5 and is not load-bearing for the benchmark results. The Wilcoxon paired-observation concern raised by the stopping rule is a statistical validity and reproducibility question, not a circularity, and does not change this finding. Thus no circular step is present; score is 2 only for the minor non-load-bearing self-citation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical claims rest on experimental design choices (loop threshold, stop rule, prompt translation) that are not independently validated. The only mathematical axiom is the standard DFS maze property. No invented entities are introduced.

free parameters (2)
  • loop termination threshold = 10 cell visits
    Chosen by hand in Section 3.3; this threshold defines the failure condition, so the claim that 100% of failures are loop-triggered is an artifact of this design choice.
  • movement budget = 3n^2 steps
    Chosen by hand in Section 3.3; it is generous enough that runs are almost always stopped by the loop threshold first, which is why no failures are attributed to budget exhaustion.
assumptions (3)
  • standard math DFS-generated mazes have exactly one solution path
    Section 3.1; this is a standard property of depth-first search maze generation when no cycles are introduced.
  • domain assumption Identical maze configurations in English and Icelandic ensure direct comparability
    Section 3.4; assumes the translation does not change task difficulty or introduce confounds such as instruction-following differences.
  • domain assumption The function-calling interface measures the model's intended actions without parsing error
    Section 3.3; assumes every model handles the function-calling format equally and that the chosen move reflects the model's navigation decision.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models." pith.science (2026). https://pith.science/paper/QLTVUHNX

@misc{pith2026250720395,
  author       = {Pith},
  title        = {Pith review of: MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QLTVUHNX}},
  note         = {Machine review of arXiv:2507.20395}
}
abstract

As Large Language Models (LLMs) increasingly power autonomous agents in robotics and embodied AI, understanding their spatial reasoning capabilities becomes crucial for ensuring reliable real-world deployment. Despite advances in language understanding, current research lacks evaluation of how LLMs perform spatial navigation without visual cues, a fundamental requirement for agents operating with limited sensory information. This paper addresses this gap by introducing MazeEval, a benchmark designed to isolate and evaluate pure spatial reasoning in LLMs through coordinate-based maze navigation tasks. Our methodology employs a function-calling interface where models navigate mazes of varying complexity ($5\times 5$ to $15\times 15$ grids) using only coordinate feedback and distance-to-wall information, excluding visual input to test fundamental spatial cognition. We evaluate eight state-of-the-art LLMs across identical mazes in both English and Icelandic to assess cross-linguistic transfer of spatial abilities. Our findings reveal striking disparities: while OpenAI's O3 achieves perfect navigation for mazes up to size $30\times 30$, other models exhibit catastrophic failure beyond $9\times 9$ mazes, with 100% of failures attributed to excessive looping behavior where models revisit a cell at least 10 times. We document a significant performance degradation in Icelandic, with models solving mazes 3-4 sizes smaller than in English, suggesting spatial reasoning in LLMs emerges from linguistic patterns rather than language-agnostic mechanisms. These results have important implications for global deployment of LLM-powered autonomous systems, showing spatial intelligence remains fundamentally constrained by training data availability and highlighting the need for architectural innovations to achieve reliable navigation across linguistic contexts.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [1]

    As LLMs increasingly serve as the foundation for autonomous agents (Duan et al., 2022), understanding their capacity for spatial rea- soningbecomescrucial

    Introduction Spatial reasoning and navigation represent funda- mentalcognitiveabilitiesthathumansemployeffort- lessly in daily life. As LLMs increasingly serve as the foundation for autonomous agents (Duan et al., 2022), understanding their capacity for spatial rea- soningbecomescrucial. WhileLLMsexcelatmany language understanding tasks, their ability to ...

    arXiv 2022

  2. [2]

    Related Work The evaluation of LLMs as autonomous agents has emerged as a critical research area, with several benchmarks addressing different aspects of agent capabilities. 2.1. General Agent Benchmarks Recent work has produced comprehensive bench- marks for evaluating LLM agents. AgentBench (Liu et al., 2023a) provides a diverse evaluation suite testing...

    work page 2022

  3. [3]

    We focus on the configurationthatprovidesthemostchallengingyet fair assessment of spatial reasoning capabilities

    Methods Our evaluation framework consists of three core components: maze generation, LLM interaction interface, and evaluation metrics. We focus on the configurationthatprovidesthemostchallengingyet fair assessment of spatial reasoning capabilities. 3.1. Maze Generation We generate mazes using a depth-first search (DFS) algorithm, ensuring each maze has e...

    work page 2023

  4. [4]

    The results re- veal significant variations in spatial reasoning capa- bilities and provide insights into how these abilities transfer across languages

    Results and Analysis We evaluated seven state-of-the-art language mod- els across maze sizes ranging from5× 5to15 × 15 grids in both English and Icelandic. The results re- veal significant variations in spatial reasoning capa- bilities and provide insights into how these abilities transfer across languages. 4.1. Overall Performance Figure 2 presents the h...

    work page

  5. [5]

    illusion of thinking

    Discussion Our results reveal a striking dichotomy in spatial reasoning capabilities among current LLMs. While O3demonstratesthatnear-perfectmazenavigation is achievable, the steep performance degradation observed in other models highlights fundamental challenges in spatial cognition that persist despite advances in language understanding. 5.1. The Spatia...

    work page 2024

  6. [6]

    Conclusion We presented MazeEval, a comprehensive bench- mark for evaluating spatial reasoning and sequen- tial decision-making in Large Language Models through coordinate-based maze navigation. Our evaluation of eight state-of-the-art models across varying maze complexities in both English and Ice- landic reveals fundamental limitations in current LLMs’ ...

    work page

  7. [7]

    Bibliographical References David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, God- son Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yue...

    work page 2025

  8. [11]

    Can large language models be good path planners? a benchmark and investigation on spatial-temporal reasoning.arXiv preprint arXiv:2310.03249. Kabir Ahuja, Harshita Diddee, Rishav Hada, Mil- licent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evalu...

    arXiv

Show all 21 references
  1. [13]

    Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al

    Dialectical language model evaluation: An initial appraisal of the commonsense spa- tial reasoning abilities of llms.arXiv preprint arXiv:2304.11164. Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla E...

    2018 arXiv

  2. [14]

    In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18500–18507

    Advancing spatial reasoning in large lan- guage models: An in-depth evaluation and en- hancement using the stepgame benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18500–18507. Peggy Li, Linda Abarbanell, Lila Gleitman, and Anna Pa...

    2011 arXiv

  3. [15]

    InProceedings of the IEEE conferenceoncomputervisionandpatternrecog- nition, pages 8494–8502

    Virtualhome: Simulating household activ- ities via programs. InProceedings of the IEEE conferenceoncomputervisionandpatternrecog- nition, pages 8494–8502. Laria Reynolds and Kyle McDonell. 2021. Prompt programmingforlargelanguagemodels: Beyond the few-shot paradigm. InExtended...

    2021 arXiv

  4. [17]

    At (0, 0) [saw: 1E, 0W, 0S, 0N] - Moved east to position (1, 0),→

  5. [18]

    At (1, 0) [saw: 0E, 1W, 2S, 0N] - Moved south to position (1, 1),→

  6. [19]

    At (1, 1) [saw: 0E, 0W, 1S, 1N] - Moved south to position (1, 2),→

  7. [20]

    At (1, 2) [saw: 0E, 1W, 0S, 2N] - Moved west to position (0, 2),→

  8. [21]

    Choose a direction: north, south, east, or west

    At (0, 2) [saw: 1E, 0W, 4S, 1N] - Moved south to position (0, 3),→ Use the move function to navigate. Choose a direction: north, south, east, or west. ,→ ,→ WARNING: If you visit the same cell 10 times, the evaluation will be terminated. Avoid getting stuck in loops! ,→ ,→ ,→ ...

  9. [2002]

    Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Ar- mand Joulin, and Tomas Mikolov

    Human spatial representation: Insights from animals.Trends in cognitive sciences, 6(9):376–382. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Ar- mand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prereq...

    2015 arXiv

  10. [2018]

    TextWorld (Côté et al., 2018) offers text-based navigation but in richly described environments that provide substantial contextual cues

    provides grid-world navigation tasks but focuses on instruction following rather than au- tonomous navigation. TextWorld (Côté et al., 2018) offers text-based navigation but in richly described environments that provide substantial contextual cues. The bAbI tasks (Weston et al...

    2018

  11. [2021]

    presents an alternative pathway for improv- ing navigation performance. Advanced prompt- ing strategies that explicitly scaffold spatial rea- soning, such as encouraging models to maintain explicit coordinate maps or systematically plan routesbeforeexecution,couldpotentiallyov...

    2018

  12. [2023]

    demonstrated that LLMs show degraded per- formance compared to monolingual tasks. Our inclusion of Icelandic evaluation provides insights into whether spatial reasoning capabilities transfer across languages, particularly for a morphologi- cally rich language with relatively l...

  13. [2024]

    In Proceedings of the IEEE/CVF Conference on ComputerVisionandPatternRecognition, pages 14455–14465

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on ComputerVisionandPatternRecognition, pages 14455–14465. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thie...

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.