Before We Trust Them: Decision-Making Failures in Navigation of Foundation Models
Pith reviewed 2026-05-16 16:34 UTC · model grok-4.3
The pith
High success rates on navigation tasks do not guarantee reliable decision making by foundation models.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
High success rates on navigation-related tasks do not necessarily translate into reliable decision making by foundation models. Evaluations on six diagnostic tasks in complete, incomplete, and safety-relevant spatial settings show persistent failures including structural collapse, hallucinated reasoning, constraint violations, and unsafe decisions. GPT-5 reaches 93 percent success in path planning with unknown cells yet exhibits fundamental limits in structural spatial understanding, while Gemini-2.5 Flash scores only 67 percent on emergency evacuation compared to 100 percent by Gemini-2.0 Flash.
What carries the argument
Six diagnostic tasks spanning reasoning under complete spatial information, incomplete spatial information, and safety-relevant information.
If this is right
- Standard success metrics overlook fundamental limits in spatial structure and safety reasoning.
- Newer foundation models are not guaranteed to outperform earlier versions on navigation reliability.
- Models can produce constraint violations and unsafe choices even at high overall success rates.
- Fine-grained failure analysis is needed to identify and correct specific decision-making breakdowns.
Where Pith is reading between the lines
- Benchmarks that reward only final success may mask models that remain brittle when deployed in dynamic environments.
- Similar diagnostic tasks could be applied to other agent domains such as manipulation or multi-agent coordination to surface parallel weaknesses.
- Adding explicit verification steps for spatial consistency might reduce the observed hallucinations and constraint violations.
Load-bearing premise
The six diagnostic tasks and three settings accurately capture the critical decision-making limitations that foundation models would encounter in actual navigation scenarios.
What would settle it
A controlled navigation simulation in which models that pass all six tasks still generate unsafe or spatially incoherent decisions, or in which models that fail the tasks perform reliably in practice.
read the original abstract
High success rates on navigation-related tasks do not necessarily translate into reliable decision making by foundation models. To examine this gap, we evaluate current models on six diagnostic tasks spanning three settings: reasoning under complete spatial information, reasoning under incomplete spatial information, and reasoning under safety-relevant information. Our results show that the current metrics may not capture critical limitations of the models and indicate good performance, underscoring the need for failure-focused analysis to understand model limitations and guide future progress. In a path-planning setting with unknown cells, GPT-5 achieved a high success rate of 93%; Yet, the failed cases exhibit fundamental limitations of the models, e.g., the lack of structural spatial understanding essential for navigation. We also find that newer models are not always more reliable than their predecessors on this end. In reasoning under safety-relevant information, Gemini-2.5 Flash achieved only 67% on the challenging emergency-evacuation task, underperforming Gemini-2.0 Flash, which reached 100% under the same condition. Across all evaluations, models exhibited structural collapse, hallucinated reasoning, constraint violations, and unsafe decisions. These findings show that foundation models still exhibit substantial failures in navigation-related decision making and require fine-grained evaluation before they can be trusted.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that high success rates on navigation-related tasks do not guarantee reliable decision-making in foundation models. It evaluates models on six diagnostic tasks across three settings (complete spatial information, incomplete spatial information, and safety-relevant information), reporting concrete results such as GPT-5 achieving 93% success in unknown-cell path planning yet showing structural limitations, Gemini-2.5 Flash reaching only 67% on emergency evacuation (underperforming its predecessor), and widespread issues including structural collapse, hallucinated reasoning, constraint violations, and unsafe decisions. The work concludes that current metrics are insufficient and calls for failure-focused analysis to guide progress.
Significance. If the empirical observations hold after methodological clarification, the work is significant for exposing the gap between aggregate success metrics and actual decision reliability in foundation models applied to navigation. It provides direct evidence of specific failure modes (e.g., lack of structural spatial understanding) that standard benchmarks may overlook, which could inform the creation of more robust evaluation protocols and safer deployment practices in robotics and autonomous systems. The direct observation of model behaviors on defined tasks is a strength.
major comments (3)
- [Abstract] Abstract: The manuscript reports precise performance figures (GPT-5 at 93% success, Gemini-2.5 Flash at 67% on emergency evacuation) and failure modes but provides no information on task construction details, number of trials or sample sizes, statistical controls, or the criteria used to classify model outputs as failures. This information is load-bearing for verifying whether the data support the central claim that high success rates mask critical limitations.
- [Diagnostic tasks] Diagnostic tasks section: The six tasks are asserted to capture critical decision-making limitations relevant to navigation, yet no explicit mapping is given from observed failures (e.g., structural collapse in unknown-cell path planning) to real-world error modes such as dynamic obstacles or sensor noise, nor is there comparison against established navigation benchmarks. This leaves open whether the reported collapses generalize or are artifacts of the specific task formulations.
- [Results] Model comparison results: The observation that newer models are not always more reliable (Gemini-2.5 Flash underperforming Gemini-2.0 Flash) is presented without details on experimental controls such as prompt consistency, temperature settings, or output parsing methods. These controls are necessary to establish that the reliability differences are attributable to model capabilities rather than confounding factors.
minor comments (1)
- [Abstract] Abstract: The sentence beginning 'Yet, the failed cases' uses nonstandard capitalization after a semicolon; revise to lowercase 'yet' for consistency with academic style.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed feedback. The comments highlight important areas for improving clarity and rigor, and we have revised the manuscript to address them fully while preserving the core contributions.
read point-by-point responses
-
Referee: [Abstract] Abstract: The manuscript reports precise performance figures (GPT-5 at 93% success, Gemini-2.5 Flash at 67% on emergency evacuation) and failure modes but provides no information on task construction details, number of trials or sample sizes, statistical controls, or the criteria used to classify model outputs as failures. This information is load-bearing for verifying whether the data support the central claim that high success rates mask critical limitations.
Authors: We agree that these details are essential for reproducibility and to substantiate the central claims. In the revised manuscript we have added a new 'Experimental Setup' subsection that specifies task construction (including grid sizes, obstacle distributions, and prompt templates), the number of trials (100 independent runs per model per task), sample sizes, statistical controls (including variance reporting and significance testing), and explicit failure classification criteria (e.g., structural collapse defined as invalid path topology, constraint violations as explicit rule breaches, and unsafe decisions as actions that would cause harm in the simulated environment). Full prompts and parsing code are now provided in the appendix. revision: yes
-
Referee: [Diagnostic tasks] Diagnostic tasks section: The six tasks are asserted to capture critical decision-making limitations relevant to navigation, yet no explicit mapping is given from observed failures (e.g., structural collapse in unknown-cell path planning) to real-world error modes such as dynamic obstacles or sensor noise, nor is there comparison against established navigation benchmarks. This leaves open whether the reported collapses generalize or are artifacts of the specific task formulations.
Authors: We appreciate this point and acknowledge that the original submission did not make the real-world connections explicit enough. In the revision we have added a dedicated paragraph in the 'Diagnostic Tasks' section that maps each observed failure mode to concrete real-world navigation errors (structural collapse to failures under dynamic obstacles or partial observability; hallucinated reasoning to sensor noise or map inaccuracies; unsafe decisions to collision or evacuation risks). We also include a new comparison table relating our results to aggregate metrics from established benchmarks such as ALFRED and Habitat, showing that our tasks surface limitations that standard success-rate evaluations overlook. revision: yes
-
Referee: [Results] Model comparison results: The observation that newer models are not always more reliable (Gemini-2.5 Flash underperforming Gemini-2.0 Flash) is presented without details on experimental controls such as prompt consistency, temperature settings, or output parsing methods. These controls are necessary to establish that the reliability differences are attributable to model capabilities rather than confounding factors.
Authors: We agree that these controls must be documented to support the model-comparison claims. The revised manuscript now states that all experiments used temperature = 0 for deterministic generation, identical base prompt templates (with only model-specific formatting adjustments), and a two-stage output parsing procedure (rule-based extraction followed by manual review of ambiguous cases). We additionally report standard deviation across three independent prompt phrasings to confirm that the observed performance gaps (including Gemini-2.5 Flash vs. Gemini-2.0 Flash) persist under controlled conditions. revision: yes
Circularity Check
No circularity: purely empirical evaluation of model behaviors on defined tasks
full rationale
The paper conducts direct empirical testing of foundation models across six diagnostic navigation tasks in three settings. It reports observed success rates (e.g., GPT-5 at 93% on path planning) and failure modes (structural collapse, hallucinations, constraint violations) without any derivations, equations, fitted parameters renamed as predictions, or self-referential definitions. All claims rest on explicit task definitions and model outputs rather than reducing to inputs by construction. No self-citation chains or uniqueness theorems are invoked as load-bearing elements.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The diagnostic tasks accurately probe the intended reasoning capabilities under complete, incomplete, and safety-relevant information.
Forward citations
Cited by 2 Pith papers
-
Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses
The survey organizes over 400 papers on embodied AI safety into a multi-level taxonomy and flags overlooked issues such as fragile multimodal fusion and unstable planning under jailbreaks.
-
Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control
LLMs for robotic health attendant control violate safety rules in 54.4% of harmful scenarios on average, with proprietary models at 23.7% median violation versus 72.8% for open-weight models, indicating they are not y...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.