pith. sign in

arxiv: 2604.22409 · v3 · pith:CQZNRNRNnew · submitted 2026-04-24 · 💻 cs.CV

SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments

Chih-Ting Liao , Xi Xiao , Chunlei Meng , Zhangquan Chen , Yitong Qiao , Weilin Zhou , Tianyang Wang , Xu Zheng
show 1 more author
Xin Cao
This is my paper

Pith reviewed 2026-05-08 12:29 UTC · model grok-4.3

classification 💻 cs.CV
keywords spatial reasoningembodied environmentsvision-language modelsspatial memorybenchmarkdynamic belief updateperception integration
0
0 comments X

The pith

Vision-language models maintain spatial beliefs only when given text histories and collapse without them in changing environments.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SpaMEM, a benchmark built from millions of images across 25,000 action sequences in procedurally generated houses, to test how multimodal models update spatial beliefs when objects are spawned, placed, or removed. It defines a three-level progression: basic perception from single views, temporal reasoning supplied with oracle text, and full end-to-end maintenance from raw visual streams alone. Benchmark results on representative open-source models show consistent failure at coordinate grounding and a steep drop when text scaffolding is withheld, revealing that models can follow symbolic state updates but cannot sustain visual memory across long horizons. If this diagnosis holds, progress in embodied AI will require new mechanisms that keep spatial relations coherent without external bookkeeping. A reader should care because real robots and agents must track changing layouts from egocentric camera streams, exactly the setting the benchmark isolates.

Core claim

SpaMEM formalizes embodied spatial reasoning as a three-level hierarchy over action-conditioned transformations. Level 1 measures atomic perception, Level 2 adds oracle textual histories to remove perceptual noise, and Level 3 demands belief maintenance from raw RGB, depth, and segmentation streams under identical task dimensions. Evaluations across open-source VLM families identify a stacked bottleneck: coordinate-consistent grounding remains difficult, and performance collapses from Level 2 to Level 3, indicating that models succeed via text-based bookkeeping but cannot sustain robust visual memory.

What carries the argument

The three-level task hierarchy (atomic perception, oracle-text temporal reasoning, and raw-visual end-to-end belief maintenance) that isolates perception-memory integration across spawn-place-remove action sequences.

Load-bearing premise

The procedurally generated houses, action sequences, and task levels isolate genuine spatial belief evolution without introducing simulation artifacts that would not appear in physical settings.

What would settle it

Deploy the identical Level-3 tasks on a physical robot that performs the same spawn-place-remove actions in a real room and measure whether spatial reconstruction accuracy matches or exceeds the simulated Level-3 scores.

Figures

Figures reproduced from arXiv: 2604.22409 by Chih-Ting Liao, Chunlei Meng, Tianyang Wang, Weilin Zhou, Xin Cao, Xi Xiao, Xu Zheng, Yitong Qiao, Zhangquan Chen.

Figure 1
Figure 1. Figure 1: Overview of the SpaMEM benchmark. SpaMEM evaluates spatial reason￾ing under dynamic scene evolution. Scenes evolve through action-conditioned transfor￾mations (spawn, place, remove) over long temporal horizons. The benchmark organizes evaluation into three hierarchical levels: L1 atomic spatial perception from single ob￾servations, L2 symbolic temporal reasoning with textual state descriptions, and L3 full… view at source ↗
Figure 2
Figure 2. Figure 2: SpaMEM evaluation framework (Update and Answer Modes). view at source ↗
Figure 3
Figure 3. Figure 3: High-level diagnostic syntheses under SpaMEM. view at source ↗
Figure 4
Figure 4. Figure 4: Fine-grained analysis of Semantic Recognition Performance (T1_F1) condi￾tioned on receptacle types. The results highlight a persistent performance gap between salient open surfaces and occluding containers view at source ↗
Figure 5
Figure 5. Figure 5: Object-wise Semantic Recognition Performance (F1) for InternVL and Qwen families. Both families show consistent improvement in grounding mid-sized objects across generations view at source ↗
Figure 6
Figure 6. Figure 6: Comparison across different VLM architectures and SOTA leaders. The results highlight the persistent resolution bottleneck for thin objects like pencils and forks across all leading models view at source ↗
Figure 7
Figure 7. Figure 7: Temporal stability analysis of the InternVL family. While SOR-M (Perception) remains consistent due to textual grounding, CSR (Integration) exhibits a sharp decay as the event sequence length increases. This "memory entropy" phenomenon suggests that as the history grows, the cumulative logic required to maintain a consistent world model exceeds the model’s coherent reasoning capacity. As shown in view at source ↗
Figure 8
Figure 8. Figure 8: Correlation analysis between Perception (F1) and Integration (CSR). The re￾sults show that while perception anchors addition events, removal events are entirely decoupled from perceptual grounding. – Background Bias: Large interactive objects like DiningTable suffer from high failure rates. Models often misclassify these as static environmental geometry rather than dynamic interactable entities. – Resoluti… view at source ↗
Figure 9
Figure 9. Figure 9: Diagnostic analysis of text-aided episodic memory (Level 2). view at source ↗
Figure 10
Figure 10. Figure 10: Temporal stability decay in visual-only episodic memory. view at source ↗
Figure 11
Figure 11. Figure 11: Causal analysis of the perception-integration link. view at source ↗
Figure 12
Figure 12. Figure 12: Fragility and Grounding Death in Level 3. view at source ↗
Figure 13
Figure 13. Figure 13: Cross-level diagnostic comparison demonstrating symbolic depen view at source ↗
Figure 14
Figure 14. Figure 14: SpaMEM Dataset Statistics. (Left) Action distribution showing the bal￾ance between scene population and manipulation. (Right) Top-8 receptacle interaction frequency, highlighting the dominance of occluding containers like Drawers. (Bottom) Top-15 object frequency distribution across the 103 unique categories, demonstrating semantic and scale diversity view at source ↗
read the original abstract

Multimodal large language models (MLLMs) have advanced static visual--spatial reasoning, yet they often fail to preserve long-horizon spatial coherence in embodied settings where beliefs must be continuously revised from egocentric observations under environmental change. We introduce SpaMEM (Spatial Memory from Action Sequences), a large-scale diagnostic benchmark that isolates the mechanics of spatial belief evolution via action-conditioned scene transformations (spawn, place, remove) over long interaction horizons. SpaMEM is built on a physically grounded dataset with 10,601,392 high-fidelity images across four modalities (RGB, depth, instance, semantic segmentation), collected from 25,000+ interaction sequences in 1,000 procedurally generated houses. We formalize embodied spatial reasoning as a three-level hierarchy with 15 diagnostic tasks: Level 1 measures atomic spatial perception from single observations; Level 2 probes temporal reasoning with oracle textual state histories to factor out perceptual noise; and Level 3 requires end-to-end belief maintenance from raw visual streams under the same task dimensions. We further evaluate both short-term (step-wise) updates and long-term (episodic) reconstruction. Benchmarking representative open-source VLM families reveals a consistent stacked bottleneck: coordinate-consistent grounding remains a hard ceiling, and the sharp collapse from Level 2 to Level 3 exposes a pronounced symbolic scaffolding dependency, where models succeed with text-based bookkeeping but struggle to sustain robust visual memory. SpaMEM provides a granular diagnostic standard and motivates explicit mechanisms for state representation, belief revision, and long-horizon episodic integration. A subset of SpaMEM is publicly available at https://huggingface.co/datasets/mill-ct-liao/SpaMEM.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The manuscript introduces SpaMEM, a large-scale benchmark for dynamic spatial reasoning in embodied environments. It consists of 10,601,392 images across RGB, depth, instance, and semantic modalities collected from 1,000 procedurally generated houses and over 25,000 interaction sequences using spawn, place, and remove actions. The benchmark defines a three-level task hierarchy with 15 diagnostic tasks: Level 1 for atomic spatial perception from single observations, Level 2 for temporal reasoning using oracle textual state histories, and Level 3 for end-to-end belief maintenance from raw visual streams. Evaluations of open-source VLM families show consistent performance collapse from Level 2 to Level 3, which the authors attribute to a symbolic scaffolding dependency where models rely on text-based bookkeeping but fail to sustain visual memory.

Significance. If the central claims hold after addressing potential confounds, the work is significant as a granular diagnostic benchmark that quantifies limitations in current VLMs for long-horizon spatial coherence and belief revision under environmental change. The dataset scale (over 10 million images from 1,000 houses) and structured hierarchy provide a reproducible standard that can drive progress on state representation, belief revision, and episodic integration mechanisms. This is a clear strength for the embodied AI and multimodal reasoning community.

major comments (3)
  1. [Abstract] The abstract's claim that the sharp collapse from Level 2 to Level 3 'exposes a pronounced symbolic scaffolding dependency' is load-bearing for the main conclusion. However, this attribution assumes the only relevant difference between oracle text and raw visuals is perceptual noise, without controls or analysis to rule out that procedural simulation properties (perfectly consistent lighting, discrete object placements, absence of naturalistic sensor noise) systematically increase visual state-tracking difficulty independently of memory mechanisms.
  2. [§4 (Benchmarking Results)] The performance claims of a 'consistent stacked bottleneck' and 'hard ceiling' on coordinate-consistent grounding (abstract and §4) are not accompanied by error bars, statistical significance tests, or variance analysis across the 1,000 houses and 25,000+ sequences. This weakens support for the cross-level and cross-model generalizations.
  3. [§3 (Benchmark Construction)] The three-level hierarchy is presented as cleanly isolating spatial belief evolution, but §3 does not provide explicit verification that the fixed action set and procedural house generation do not introduce task-specific biases or simulation artifacts that affect Level 3 more than Level 2 beyond the intended perceptual noise factor.
minor comments (3)
  1. [Abstract] The abstract reports '25,000+' sequences but gives an exact image count; ensure numerical consistency and precise reporting of sequence counts in the main text and tables.
  2. A summary table listing all 15 diagnostic tasks with their level, input type (text vs. visual), and evaluation metric would improve clarity of the task hierarchy.
  3. [Figures] Figure captions describing the modalities and example action sequences could be expanded for readers unfamiliar with the simulation setup.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thoughtful and constructive comments. We address each major comment below with clarifications and commitments to revisions that strengthen the manuscript without altering its core claims.

read point-by-point responses

  1. Referee: [Abstract] The abstract's claim that the sharp collapse from Level 2 to Level 3 'exposes a pronounced symbolic scaffolding dependency' is load-bearing for the main conclusion. However, this attribution assumes the only relevant difference between oracle text and raw visuals is perceptual noise, without controls or analysis to rule out that procedural simulation properties (perfectly consistent lighting, discrete object placements, absence of naturalistic sensor noise) systematically increase visual state-tracking difficulty independently of memory mechanisms.

    Authors: We appreciate this observation on the attribution. The benchmark design holds the environment, actions, house generation, and task definitions fixed across levels, with the sole controlled difference being the input (oracle text histories in Level 2 versus raw visual streams in Level 3). This isolates the visual memory component. The idealized simulation is a deliberate choice to diagnose reasoning failures without sensor confounds, consistent with standard embodied AI benchmarks. In revision we will expand the abstract and add a dedicated paragraph in §4 (and a limitations subsection) explicitly discussing this design rationale and noting that future extensions could incorporate naturalistic noise. revision: partial

  2. Referee: [§4 (Benchmarking Results)] The performance claims of a 'consistent stacked bottleneck' and 'hard ceiling' on coordinate-consistent grounding (abstract and §4) are not accompanied by error bars, statistical significance tests, or variance analysis across the 1,000 houses and 25,000+ sequences. This weakens support for the cross-level and cross-model generalizations.

    Authors: We agree that quantitative rigor requires statistical support. In the revised manuscript we will augment all tables and figures in §4 with error bars (standard deviation computed across the 1,000 houses and 25,000+ sequences) and include paired t-test results (with p-values) comparing performance across levels and models to substantiate the reported bottlenecks and generalizations. revision: yes

  3. Referee: [§3 (Benchmark Construction)] The three-level hierarchy is presented as cleanly isolating spatial belief evolution, but §3 does not provide explicit verification that the fixed action set and procedural house generation do not introduce task-specific biases or simulation artifacts that affect Level 3 more than Level 2 beyond the intended perceptual noise factor.

    Authors: We thank the referee for this point on verification. Section 3 already specifies that the action vocabulary, procedural house generator, and 15 task dimensions are identical across levels. To make this explicit, we will insert a new verification subsection in §3 that (a) confirms instance-level matching of tasks between levels, (b) reports per-house variance statistics demonstrating consistency, and (c) includes qualitative examples illustrating that any simulation artifacts are shared and do not differentially impact Level 3. These additions will be supported by supplementary per-house breakdowns. revision: partial

Circularity Check

0 steps flagged

No circularity: pure empirical benchmark with direct evaluations

full rationale

The paper introduces a new benchmark (SpaMEM) with procedurally generated data, defines a three-level task hierarchy, and reports model performance on direct evaluations across modalities and horizons. No derivations, equations, fitted parameters, or predictions appear; claims about bottlenecks and scaffolding dependency are interpretive summaries of observed results rather than reductions to self-defined inputs or self-citations. The work is self-contained as an empirical diagnostic standard without load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the domain assumption that the defined task hierarchy and procedural environments measure intended spatial belief capabilities without major confounds.

axioms (1)
  • domain assumption Embodied spatial reasoning can be formalized as a three-level hierarchy that isolates atomic perception, temporal reasoning with oracle textual histories, and end-to-end belief maintenance from raw visual streams.
    This formalization is presented directly in the abstract as the basis for the 15 diagnostic tasks.

pith-pipeline@v0.9.0 · 5608 in / 1245 out tokens · 71138 ms · 2026-05-08T12:29:34.022871+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning

    cs.AI 2026-05 unverdicted novelty 7.0

    MentalMap benchmark identifies a universal L3 reasoning cliff in LLMs' text-based spatial reasoning that persists across languages, scales, and prompting, and is replicated in human evaluations.

  2. Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

    cs.CV 2026-06 unverdicted novelty 5.0

    VIGIL is a counterfactual RL alignment method that reduces visual hallucinations in MLLMs by enforcing visual grounding via masked attention penalties, outperforming baselines with 25% of the data and showing emergent...